Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model
Authors: Jincheng Zhong
XiangCheng Zhang
Jianmin Wang
Mingsheng Long
Venue: ICLR 2025 Poster (2025)
Keywords: transfer learning
diffusion models
fine-tuning
guidance
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: This paper proposes a conditional generation perspective for the transfer of diffusion models and derives a simple approach named Domain Guidance to enhance transfer learning.

Abstract: Recent advancements in diffusion models have revolutionized generative modeling. However, the impressive and vivid outputs they produce often come at the cost of significant model scaling and increased computational demands. Consequently, building personalized diffusion models based on off-the-shelf models has emerged as an appealing alternative. In this paper, we introduce a novel perspective on conditional generation for transferring a pre-trained model. From this viewpoint, we propose *Domain Guidance*, a straightforward transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain. Domain Guidance shares a formulation similar to advanced classifier-free guidance, facilitating better domain alignment and higher-quality generations. We provide both empirical and theoretical analyses of the mechanisms behind Domain Guidance. Our experimental results demonstrate its substantial effectiveness across various transfer benchmarks, achieving over a 19.6\% improvement in FID and a 23.4\% improvement in FD$_\text{DINOv2}$ compared to standard fine-tuning. Notably, existing fine-tuned models can seamlessly integrate Domain Guidance to leverage these benefits, without additional training. Code is available at this repository: https://github.com/thuml/DomainGuidance.

Published as a conference paper at ICLR 2025
DOMAIN GUIDANCE: A SIMPLE TRANSFER APPROACH FOR A PRE-TRAINED DIFFUSION MODEL
Jincheng Zhong, Xiangcheng Zhang, Jianmin Wang, Mingsheng Long" School of Software, BNRist, Tsinghua University, China {zjc22, xc- zhang21] @mail s... t nghua .dd {j mwan, mi mingsheng) 8t @t.singhu..... edu.cn cn
ABSTRACT
Recent advancements in diffusion models have revolutionized generative modeling. However, the impressive and vivid outputs they produce often come the cost significant model scaling and increased computational demands. Consequently, building personalized diffusion models based on off-the-shelf models has emerged appealing alternative. In this paper, introduce novel perspective on conditional generation for transferring pre-trained model. From this viewpoint, we propose Domain Guidance, straightforward transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain. Domain Guidance shares a formulation similar to advanced classifier-free guidance, facilitating better domain alignment and higher-quality generations. We provide both empirical and theoretical analyses of the mechanisms behind Domain Guid- ance. Our experimental results demonstrate its substantial effectiveness across various transfer benchmarks, achieving over 19.6% improvement in FID and a 23.4% improvement in FDDINOv2 compared to standard fine-tuning. Notably, existing fine-tuned models can seamlessly integrate Domain Guidance to leverage these benefits, without additional training. Code is available at this repository: https::/git wwtom//hul///// htt.//Dmmaniiaaaammmmmmm
INTRODUCTION
Diffusion models have significantly advanced the state the art across various generative tasks, such as image synthesis (Ho et al. (2220). video generation (Ho et al. 2022). and cross-moda generation (Saharia et al., 2022: Rombach et al., 2022). Concurrently, advancements in guidance techniques (Dhariwal & Nichol 2021 Ho & Salimans 2022) have significantly enhanced mode consistency and generation quality, becoming indispensable components of contemporary diffusion models (Esser et al. 2024 Peebles & Xie, 2023). However, generating high-quality samples frequently requires substantial computational resources to scale foundational diffusion models. In practical settings, transfer learning, especially fine-tuning, proves vital for personalized generative scenarios.
Recent research has yielded promising outcomes in fine-tuning scaled pre-trained models. Despite diverse motivations, these efforts converge on common objective: efficient fine-tuning with minimal parameter adjustment, a group of methods termed parameter-efficient transfer learning (PEFT) (Houlsby et al., 20199] Zaken al. 2021] Xie al., 2023). Nevertheless, PEFT introduces significant optimization challenges, including the necessity for considerably higher learning rates-an often an order of magnitude greater than typical-whie may precipitate spikes in loss. An effective transfer strategy that capitalizes on the intrinsic properties of diffusion models remains largely unexplored.
In this paper, we introduce a novel perspective on conditional generation for fine-tuning. We conceptualize the transfer of a pre-trained model to a downstream domain as conditioning the sampling process on the target domain, relative to the pre-trained data distribution. From this viewpoint, we incorporate guidance principles (Dhariwal & Nichol 2021 Ho & Salimans, 2022) and introduce Domain Guidance (DoG) as a general transfer method to enhance model transfer. DoG involves fine-tuning the pre-trained model specifically for the new domain to create domain conditional branch, while simultaneously maintaining the original model as an unconditional guiding counterpart. At each sampling step, the domain conditional and the pre-trained guiding model executed once

---

Published as a conference paper at ICLR 2025
each, with the fine-tuned results being further extrapolated from the pre-trained base using DoG factor hyperparameter. This method not only offers a general guidance strategy for transferring pre-trained models but also seamlessly integrates models fine-tuned in the classifier-free guidance (CFG) style by simply excluding the unconditional component, without necessitating additional training. This streamlined approach significantly improves domain alignment and generation quality.
To further explore the mechanism behind DoG, we provide both empirical and theoretical analyses. Firstly, we employ a mixture of Gaussian synthetic examples and perform a theoretical analysis of DoG behaviors, which reveal that DoG effectively leverages the pre-trained domain knowledge, improving domain alignment. In contrast, standard CFG with a fine-tuned model often suffers from catastrophic forgetting, eroding valuable pre-trate pre-trained knowledge. Furthermore, we observe that limited training resources and a low-data regime typically challenge the unconditional guiding component's ability to fit the target domain, leading to diitrii-tttiiitoinn (OOD) samples and exacerbating sampling errors. DoG effectively mitigates these issues, reducing overall errors and enhancing the generation quality.
Experimentally, we evaluate DoG across seven well-established transfer learning benchmarks, pro- viding quantitative and qualitative evidence to substantiate its efficacy. Our comprehensive ablation study further underscores its superiority in the transfer of pre- pre-trained diffusion models.
Overall, our contributions can be summarized follows:
We introduce novel conditional generation perspective for transferring pre-trained models and present Domain Guidance (DoG) as a streamlined, effective transfer learning approach that leverages the principles of CFG to enhance domain alignment and generation quality.
We delve into the mechanisms behind DoG's improvements, offering both empirical and theoretical evidence that underscores how DoG enhances domain alignment by harnessing pre-trained knowledge. We also highlight how standard CFG approaches with fine-tuned guiding models often face challenges from poor fitness, which can exacerbate guidance performance issues due to increased variance in OOD samples. Conversely, DoG effectively addresses these concerns.
We validate DoG across various benchmarks, confirming its effectiveness. Our quantitative assessments show marked improvements in generated image distributions, as measured by FID (Heusel al., 2017) and FDDINOV2 (Stein et al. (024b) metrics, and reveal that existing fine-tuned models can benefit from DoG without additional training.
2 RELATED WORK
Diffusion models. Diffusion-based generative models (Ho et al. 2020 Song & Ermon 2019
Song et al. 2020b Karras et al., 2022) transform pure noise into high-quality samples through an iterative denoising process. This gradual transformation stabilizes the training process but also imposes substantial computational demands for sampling. Recent improvements in diffusion models have primarily addressed noise schedules (Nichol & Dhariwal 2021 Karras et al., 2022), training objectives (Salimans & Ho 202T Karras et al. 2022), efficient sampling techniques (Song et al. 2020a), controllable generation (Ho & Salimans 2022 Zhang al. 2023 Dhariwal & Nichol 2021 and model architectures (Peebles & Xie| (023). Current state-of the-rar art models benefit significantly from sealing up training parameters and datasets, necessitating considerable resources. In this work, we examine efficient transfer learning strategies for pre-trained diffusion models.
Guidance techniques for diffusion models. The notable successes recent applications (Dhariwal & Nichol 2021; Blatmaan et al. 2023 Esser et al. 2024) using diffusion models can largely be attributed to advances in guidance techniques, which ensure that model outputs align closely with human preferences. Prior studies have developed various methods for effectively modeling conditional control information. Dhariwal & Nichol (202I) introduced classifier guidance, which enhances conditional generation through an additional trained classifier. Subsequently, classifier-free guidance (CFG), proposed by Ho & Salimans (2022). has emerged as the de facto standard in modern diffusion models due to its robust performance. Recently, line of works Kynkäääniemi et al. (2024); Karras iet al. (2024) investigates how to utilize guidance techniques to enhance the generation performance in Elucidating Diffusion Models (EDM) Karras et al. (2022). Our work identifies challenges in the
2

---

Published as a conference paper at ICLR 2025
underperformance of fine-tuned diffusion models within standard CFG frameworks and investigates novel guidance strategies for adaptation.
Transfer learning. Transfer learning seeks to leverage existing knowledge to facilitate learning in a new domain (Pan & Yang 2009), typically through fine-tuning a pre-trained model (Yosisski et al.. 2014). Previous research has aimed to refine standard fine-tuning techniques to address issues such as catastrophic forgetting (Zhong et al. 2024 Li & Hoiem 2017). negative transfer (Chen et al. 2019). and overfitting (Dubey al.) 2018). With the recent significant expansion in model scales, the focus has shifted to research area known as parameter-efficien transfer learning (Houlsby et al., 2019) Zaken al. (202T) which aims adjust as few parameters possible minimize memory usage and computational demands on gradient calculations. In this work, we reframe transfer learning in the context of domain conditional generation and propose streamlined and effective approach.
METHOD
3.1 BACKGROUND
Diffusion formulation. Before demonstrating our method, we briefly revisit the basic concepts in diffusion models. Gaussian diffusion models are defined by forward process that gradually adds noise to original samples: x, = va,Xa + vI RyE, where X ~ denotes the original samples, (o0, I) denotes the noise signal, and constants ar hyperparameters that determine the level of noise infusion.
The training of diffusion models typically involves learning a parameterized function f that predicts the noise added to a sample, formalized by the loss function:
L( Ex,xo,e [uy ||e fo (Vaixo + vI axe, 1)]
where wy = 1 is set by default, following the simple setting used in prior studies (Ho et al. 2020). Sampling from diffusion models fo then follows a Markov chain, iteratively denoising from I) back to Xg-
Classifier-free guidance. In various complex real-world scenarios, aligning the outputs of diffusion models with human preferences is crucial. Classifier-free guidance (CFG) has become an essential tool for enhancing the outputs of practically all mmage-generating diffusion models (Ho & Salimans) Karras et al. 2024b). CFG is formalized follows:
2022 Esser al. 2024
Vx, log Pw
CFG (x/ = Vx, log le) + (w 1) (Vx, logp(x;/c) Xx logp(x;)).
Here, w is the guidance factor, typically set greater than 1, modulating the influence between the outputs of the conditional and unconditional models to achieve the desired guiding effect.
Practically, CFG is implemented by constructing both conditional model Vx, log (egpe(x, (x+/c) and an unconditional guiding model Vx, (x()) within a shared-weight network So. The combined training loss is described as:
L( = E,, Ex.xo.e.c [||c So (Va,X + v1 - o.C, t, (r)p)ut;(c))]1?
(3)
where the dropout ratio 5 is typically set 10%, as endorsed by recent studies (Peebles & Xie 2023
Esser et al. 2024).
3.2 DOMAIN GUIDANCE
Fine-tuning existing checkpoints for target domains has become a prevalent practice in transfer learning. In this section, introduce novel perspective transferring generative model through the lens of conditional generation, bridging the commonly used classifier-free guidance (CFG) into transfer learning develop our method, named Domain Guidance (DoG).
3

---

Published as a conference paper at ICLR 2025
Cassfforrrree Guidance
Ground Trush
(0) Scandrrd Chaster-ree Guldarce tueike Semwine
(e) Appling Dorrhh Gatfarce
Oc.
(e(
[J] Domsin Gurdence
c]
Fo(Me.sle.))
(4,,0)
(K..))
(o 12G000--1edd) wicle.s1d)
Figure 1: Conceptual comparisons between Domain Guidance and standard classifier-fres guidance. (a) shows standard CFG modeling both conditional density and unconditional guiding signals for the target domain simultaneously. (b) illustrates the proposed Domain Guidance, which focuses building conditional density and guides the sampling process from the pre-trained model to the target domain. (c) (e) depict conceptual examples of the mechanism differences between CFG and DoG, highlighting how DoG leverages pre-trained knowledge to enhance generation for the target domain.
The domain conditional generation perspective of transfer. The primary goal in training a generative model on a target domain is to accurately capture its distribution. When fine-tuning pre-trained generative model, we start with models that have learned the distribution of the pre-trained data. Ideally, it should leverage the distribution knowledge from the pre-trained data, effectively modeling the conditional distribution given this pre-trained context. However, the relationship (xx**|DDsc) is often compromised due catastrophic forgetting, as the model loses access to the pre- pre-trained data. Without adequate regularization from these pre-trained datasets, the model tends to converge solely the marginal target distribution 2 (D's+) through empirical risk minimization with Equation 3 This convergence results in the loss of valuable pre-trained domain knowledge, limiting the standard fine-tuning effectiveness in modeling the relationship of the domain conditional generation.
Guiding generations to the target domain via domain guidance. Building on the domain conditional generation viewpoint, we introduce Domain Guidance (DoG), which utilizes the original pre-trained model as an unconditional guiding model. This approach leverages pre-trained knowledge to direct the generative process towards the target domain, outlined below:
c DoG (x D co (x( + (w
1) (xx(x) co, (x))
(4)
where eø (x() represents the output of the fine-tuned model specific to the target domain, and eo, (x) denotes the output from the original pre-trained model, with On marking the weights prior to fine-tuning. The guidance factor, DoG adjusts the influence of this guidance, where values greater than 1 typically emphasize traits of the target domain. Specifically, DoG reduces to the standard fine-tuned model output, <x (D*tt), when w and to the pre-trained model cp, (x), when wo 0.
DoG serves as versatile mechanism for the transfer of pre-trained model and can be directly expanded to a variety of transfer scenarios involving both conditional signals e and domains Dist enhancing its applicability. The formulation of DoG in these contexts is given by:
DoG DoGG(xe (x/c, Dist)= = =0(xxc, c, D + (w 1) (=o(xlc, Dist) 6 6,, (x))
(5)
For practical implementation, inputs are concatenated with the conditional signal c, while the domain signal D'st is implicitly integrated during fine-tuning on the target domain. The dropout ratio 5 in the standard CFG setup (Equation [ can be set to 0, eliminating the need to fine-tune the unconditional guiding model and thereby simplifying the fitting process. Moreover, models that have been previously fine-tuned using the CFG approach can seamlessly transition to benefit from DoG by merely substituting the unconditional guiding component. This adjustment allows existing models to leverage pre-trained knowledge more effectively, enhancing their adaptability and performance in new domain settings.
Comparision with CFG. We conceptually compare DoG with CFG in Figure [ illustrated within a general transfer scenario involving conditional signals c and domains d. Both approaches exhibit distinct behaviors in transfer settings. The jointly fine-tuning with Equation 3 and performing CFG
4

---

Published as a conference paper at ICLR 2025
-22
(4) Ground Truch
(b) CFG
(c) DOG
(d) Score Field
Figure 2: A mixture of Gaussians synthetic dataset with different colored dots represent modes of different classes. In (a), the target domain is defined by the orange area, while the pre-training distribution forms the blue background. Green and red dots represent two classes, with filled dots indicating in-domain real datalSnmpling results from these classes after model fine-tuning are denoted by circles with corresponding color. (b) illustrates how CFG leads to out-of-domain samples by disregarding pre-trained knowledge, while (c) demonstrates how DoG maintains domain consistency by effectively utilizing pre-trained data, (d) contrasts the directional guidance provided by DoG (red arrows) against CFG (blue arrows) for intermediate samples Xmid, showing how DoG steers samples towards the Nomain-speciffe regions, unlike CFG which may lead samples towards outliers.
on the fine-tuned model is the standard practice (e.g., (Esser et al. 2024 Xie et al. 2023 Zhang
eet al. 2023). as shown in Figure aa) and (d)). This method uses the target data construct weight-sharing network that models both conditional and unconditional densities simultaneously. Applying CFG through two forward passes can steer generation towards nowttemperature conditional areas, thus enhancing generation quality and improving conditional consistency. However, CFG fails to leverage pre-trained knowledge due to catastrophic forgetting resulting from the inaccessibility of pre-trained data. As result, directly performing CFG guides the generation to rely only on the limited support of the target domain, leading to high variance in density fitness and aggravating the OOD samples (as shown in Figure d)). In contrast, DoG addresses these limitations by integrating the original pre-trained model as the unconditional guiding model, as depicted in Figure Tbb) and (e). DoG leverages the entire pre-trained distribution, which is typically more extensive than the target domain's 's distribution, to guide the generative process.
Remarkably, DoG can be implemented by executing both the fine-tuned model and the pre-trained model once each, thus not introducing additional computational costs compared to CFG during the sampling process. Unlike CFG, DoG separates the unconditional reference from the fine- tuned networks, allowing for more focused optimization on fitting the conditional density and reducing conflicts associated with competing objectives from the unconditional model. This strategic decoupling enhances the model's ability to harness pre-trained knowledge without the interference of unconditional training dynamics, leading improved stability and effectiveness in generating high-quality, domain-consistent samples.
3.3 EMPIRICAL AND THEORETICAL INSIGHTS BEHIND DOMAIN GUIDANCE
We provide both empirical and theoretical evidence to demonstrate why Domain Guidance (DoG) significantly outperforms CFG when paired with standard fine-tuning. The advantages of DoG are primarily twofold: 1) DoG leverages pre-trained knowledge to guide generation within the target domain, achieving enhanced domain alignment, and 2) the unconditional guiding model in CFG often suffers from high variance due to underfitting in conditions of insufficient training and low-data availability in the target domain. This leads to an increased frequency of out-of-distribution samples.
DoG leverages the pre-trained knowledge. Building upon the conceptual differences illustrated in Figure [] we analyze 2D Mixture Gaussian synthetic dataset as a concrete example (Figure 2 This dataset features hundreds of modes, where a subset is designated as the target domain and the remainder as the source domain (shown in Figure This example consists of mixture of Gaussian distributions with hundreds of modes, where a subset of modes is selected for the target domain, and others remain as the source domain (As shown in Figure 2 a)). We pre-train small diffusion model on the source domain and subsequently fine-tune it on the target domain, observing distinct behaviors between CFG and DoG. Figure b)) reveals that CFG biases sampling paths away from high-density
5

---

Published as a conference paper at ICLR 2025
centers, leading to outlier generations and loss of domain consistency. Conversely, as shown in Figure 2[c), DoG leverages dense pre-trained data to guide samples accurately toward high-density areas of the target domain, thereby enhancing generation quality. Details of the setup are provided in Appendix C
Theoretical insights into DoG. Beyond empirical observation, we present theoretical insights into DoG, conceptualizing it as an augmentation of classifier guidance (Dhariwal & Nichol 2021) to the conventional CFG sampling steps:
Proposition 1.
xx, logru DoG (x, Diet) Vx, logPu Pw
CFG (xele, Dist) + (w 1)Vx, log p( xx)
(6)
The details can be found in Appendix B This adjustment means that the DoG sampling distribution Pw pDoG is tuned to discourage sampling from cuito--distriuution areas, effectively using the pre-trained domain knowledge to regularize the process and improve domain consistency:
DoG
CCC x p(Dist(x.) (x,) << For X2 E Dist
highlighting how DoG steers the sampling process toward the core of the target domain manifold, thereby avoiding low low-probale low-probability regions and reducing outlier generations. Figure d]d) visually illustrates the stark guiding distinctions between CFG and DoG, underscoring the effectiveness of DoG.
The poor fitness of the guiding model. The second reason that limits the performance of CFG is the bad fitness of the guiding model. The low-data regime of the target domain, the conflicts arising from unconditional objectives, along with only a small slice of the training budget, result in poor fitness of the fine-tuned model [Chn et al. (2023); Zhhan et al. (2024). The visual quality difference is obvious we simply inspect the unconditional images generated by the fine-tuned model. Furthermore, the unconditional case tends to work so poorly that the corresponding quantitative numbers are hardly ever reported. For example, the fine-tuned DiT-XL/2 with Stanford Car exhibits an FID of 6.57 in conditional settings versus 22.8 unconditionally.
Theorem 1. Denote the mariginal distribution at timestep t conditioning on N data samples D } as pe(x)) = Ei=1 q(x, Xo y:). with yi ~ p(y). Denote the true marginal distribution at P (xx)) = Sy p()) = y). The forward process is defined as g(xx) X y) = N (xelvang) 3rI). Consider the expected estimation ervor between PO and Pł with respect to all the datasets D p(D), we have for all x,:
EDDp(D) ((pr(x)) - pł (x+)1l S
Proof. See Appendix B
Ramark. Given that Nigt < Nate, it can be assumed that across most of the manifold, ea, (x) offers better approximation to the ground truth marginal distribution, particularly areas outside the target domain. In scenarios where a pre-trained model is transferred to domain lacking in training samples, fine-tuning enhances performance on in-distribution targets but often falters on out-of-distribution samples Kumar et al. (2022). This propensity leads to a sequence of errors in diffusion model sampling, further compounding the issues faced during the guidance process. By incorporating DoG, we mitigate these errors, steering the model away from out-of-domain areas and substantially improving sampling accuracies.
4 EXPERIMENTS
We evaluate DoG on seven well-established fine-grained downstream datasets, comparing generation quality against standard fine-tuning with CFG. Additionally, conduct comprehensive experi- ments to analyze the specific properties of each component within DoG. Detailed implementation information be found in Appendix A

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons on downstream tasks with pre- pre-trained DiT-XL-2-256x256 FID 1
Dataset
CUB
Stanford Car
Average
Food
SUN
Caltech
DF-20M ArtBench
Method
Bird
FID
Fine-tuning (w/o guidance)
16.04
21.41
31.34
9.81
11.29
17.92
22.76
18.65
Classifier- free guidance
10.93
14.13
23.84
5.37
6.32
15.29
19.94
13.69
Domain guidance
9.25
11.69
23,05
3,52
4.38
12.22
16.76
11,55
Relative promotion
15.36%
17.27%
3.31%
34.45%
30.70%
20.08%
15.95%
19.59%
Table Comparisons on downstream tasks with pre-trained DiT-X1-2-256x2566 DD
Dataset
CUB
Stanford Car
Average
Food
SUN
Caltech
DF-20M ArtBench
Method
Bird
FDDINOv2
Fine-tuming (w/o guidance)
626.90 796.77
551.69
421.29
351.97
594.50
337.87
501.48
+ Classifier-free guidance
423.90 653.19 416.78
198.12
219.25
326.77
291.23
363.58
+ Domain guidance
351.93 620.58
392.92
140.00
134.15
151.39
257.39
292.62
Relative promotion
20.0% 5.0%
5.7%
29.3%
38.8%
53.7%
11.62%
23.4%
Setup. Fine-tuning a pre-trained diffusion model to a target downstream domain is a fundamental task in transfer learning. We utilize the publicly available pre-trained model DiT-XL/2" (Peebles)
& 2023), which is pre-trained on ImageNet at resolution of 256 x 256 for million training
steps, achieving a Fréchet Inception Distance (FID) of 2.27 (Heusel al. 2017). Our benchmark
setups include 7 fine-grained downstream datasets: Food101 (Bossard et al. 2014), SUN397 (Xiao)
ct al. 2010), DF20-Mini (Picek et al.) 2022). Caltech101 (Grifin et al. 2007), ) 2UB-200-2011 (Wah
al. I. ArtBench-10 10 (Liao al. 2022), and Stanford Cars (Krause et al. 2013). Most of these
datasets are selected from CLIP downstream tasks except ArtBench- 10 and DF-20M. DF-20M has
no overlap with ImageNet, while ArtBench-10 10 features a distribution that is completely distinct from ImageNet. This diversity allows for a more comprehensive evaluation of DoG in scenarios where pre- pre-trained data are significantly different from the target domain. We perform fine-tuning for 24,000 steps with a batch size of 32 at 256 x 256 resolution for all benchmarks. The standard fine-tuned models are trained in CFG style, with a label dropout ratio of 10%. Each fine-tuning task is executed on single NVIDIA A100 40GB GPU over approximately 6 hours. Following prior evaluation protocols (Peebles& & Xie 2023 Xie al. 2023). we generate 10,000 images with 50 sampling steps per benchmark, setting the guidance weights for both CFG and DoG to 1.5. We calculate metries between the generated images and test set, reporting the widely used FIL (Heusel et al. 2017) and the more recent FDDINO (Stein et al. 2024a for richer evaluation. More detailed results of precision and recall can be found in Appendix D
The FID results are summarized in Table while the FDDINO,2 results are presented in T Our results indicate that standard CFG is crucial for class-conditional generation, despite its inherent challenges. In contrast, the proposed DoG consistently improves all transfer tasks, effectively addressing the limitations faced by CFG and significantly enhancing generation quality- in a relative FID improvement of 19.59% compared to CFG. Notably, the last two columns for DF20-Mini and ArtBench- 10 exhibit a significant discrepancy from the pre-trained domain. Despite this challenge, DoG performs well, showcasing its robustness in various transfer scenarios. Even when the guided pre-trained model is considerably distant from the target domain, DoG effectively steers the generation process, enhancing both domain alignment and overall generation quality. This capability underscores DoG's utility in bridging substantial gaps between pre-trained and target domains, ensuring consistent performance across diverse settings.
Results
4.1 ABLATION STUDY AND DISCUSSION
Discussion on unconditional guiding models. Building the discussion of unconditional guiding models presented in Section 3.3 a pertinent question arises: Can extending the training budget
7

---

Published as a conference paper at ICLR 2025
Table 3: Results of CFG and DoG on
Table 4: Results of DoG on varying training strategies. FID +
varying sampling steps. FID +
CUB bird
SUN
Training steps
Steps
Dropout
ArtBench
Caltech
DF20M
CFG
DoG
CFG
DoG
25
9.69
4.60
24.34
19,87
24,000
16.76
23.05
12.22
50
5.37
3.52
14.13
11.69
21,600
16.33
22.93
11.83
100
4.27
3.35
10.07
8.71
24,000
x
16.13
22.44
11.60
Rassaree
Forgetting
FID
3.5
Stags
5000 1000)
15000
30000
25000
2.0
2.2
2.4
(3) Unconditional model with increasing training
(b) Sensilivily of FID to the guidance parameter
Figure 3: Component analysis of DoG. (a) illustrates that separately fine-tuned unconditional guiding model degrades generation performance as training steps increase. (b) shows the sensitivity of FID to guidance parameters in DoG.
for a separate unconditional model address this issue? To explore this, we conducted an analysis using the CUB dataset. We fine-tuned separate unconditional guiding models with varying numbers of fine-tuning steps and employed them to guide the fine-tuned conditional model. As illustrated in Figure aa), this approach is fine-tuning steps increases. This counterintuitive outcome can be attributed to catastrophic forgetting and overfitting, where the model loses valuable pre-trained knowledge and becomes overly focused on the target domain in a low-data regime, thereby diminishing the effectiveness of the guidance.
actually deteriorates as the number of
Discarding the unconditional training. As previously noted, DoG focuses exclusively modeling the class conditional density without the necessity of jointly fitting an unconditional guiding model. To illustrate this, present comparison of different training strategies in Table 3 using the Caltech101, DF20M, and ArtBench datasets. The dropout ratio is set to 10% when applicable; otherwise, it is set O, indicating no unconditional training involved. The first row displays the results from standard fine-tuning in CFG style (with DoG results reported in TT. The second row corresponds the same 21,600 conditional training steps (90% of the standard fine-tuning). The third row demonstrates that, under the same computational budget, DoG yields superior results. The improvement observed the second row suggests that eliminating conflicts arising from the multi-task training of the unconditional guiding model is advantageous, along with achieving 10% reduction training costs.
Sensitivity of the guidance weight factor. Figure B(b) probs the sensitivity guidance weight factor in DoG across various datasets. Our best results are typically achieved with values w ranging from 1.4 2, indicating relatively narrow search space for this parameter. To ensure fair comparisons with limited resources, all other results reported in this paper are fixed at w 1.5.
Sampling steps. We also evaluate DoG using various sampling parameters, typically halving and doubling the default 50 sampling steps employed in iDDPM (Nichol & Dhariwal 202T). Table B shows a consistent improvement performance across these configurations. Notably, the guidance signal from DoG demonstrates more significant enhancement with fewer steps, suggesting that the guidance becomes more precise due to the reduced variance provided by DoG.
QUALITATIVE RESULTS
Figure 4 showcases examples of generated images for fine-tuned downstream tasks listed in Table
I These examples demonstrate that both CFG and DoG enhance the perceptual quality images, with clearer outcomes the guidance weight increases. However, CFG, hampered by its insufficient
8

---

Published as a conference paper at ICLR 2025
CFG
DoG
Increasing guidance weight
Figure 4: Qualitative showcases for DoG across downstream tasks. Best viewed zoomed in. Each nine-grid case compares CFG (left column) and DoG (right column), with the middle column blending the two. Rows increase guidance weights from (2,3,4g 4).
utilization of pre-trained knowledge and the limitations of its poor unconditional guiding model, often directs the sampling process toward mnttrf-distributiin (OOD) outliers. This misdirection can result in noticeable distortions or blurring in the generated images. In contrast, DoG effectively counters these challenges, steering the generative process towards more accurate and visually appealing representations. notable example is seen in the depiction of an airplane in the middle-left panel of the figure. Under CFG, the airplane's 's fuselage appears fragmented, and this distortion intensifies as the guidance weight increases. DoG, on the other hand, maintains the integrity of the airplane's structure, producing a coherent and detailed image without the distortions observed with CFG.
4.3 APPLYING DoG TO STABLE DIFFUSION WITH LoRAS
To evaluate the adaptability of DoG across broader range of models and in conjunction with LoRA fine-tuning, we conduct experiments using off-the-shelf LoRAs of the SDXL model available in the Huggingface community. Specifically, we employ the SDXL modef" and select two off-the- shelf LoRA adapters: Chalkboard style and Yarn art style our implementation of DoG, we compute the text-conditional output using the LoRA adapters while disabling the LoRA during the computation the unconditional output. We adopt a default guidance scale of 5.0 for each
1.0

---

Published as a conference paper at ICLR 2025
Prompt:
CFG
DoG
"Giraffe ChaltBoardDrawingg
"A ChalkBoardDrwwing with many flowers"
"cat, yarn art style"
Figure 5: Qualitative showcases for LoRA-based transfer tasks with SDXL. The first two rows feature showcases in Chalkboard style transfer tasks, and the bottom row shows the Yarn art style transfer task. (A default guidance scale of 5.0 for each generation)
generation. The qualitative results, as showcased in Figure indicate that DoG can produce more vivid and contextually enriched generations compared to CFG. To quantitatively assess our method, we calculate the CLIP score between the generated images and the prompts in the fine-tuning dataset. In the ChalkboardDrawing Style task, the CLIP Score increases from 27.23 with CFG to 35.24 with DoG. the Yarn Art style, the CLIP Score increases from 34.89 to 35.03. These results demonstrate that DoG seamlessly adapts to pre-trained text-to-image models and integrates effectively with fine-tuning. Additional qualitative comparisons can be found in Appendix
5 CONCLUSION AND FUTURE WORKS
In this paper, we provide novel conditional generation perspective for the transfer of pre-trained diffusion models. Based on this viewpoint, we introduce domain guidance, a simple transfer approach in similar format of classifier-free guidance, improving the transfer performance significantly. We provide both empirical and theoretical evidence revealing that the effectiveness of the DoG stems from leveraging the knowledge of the pre-trained model to improve domain consistency and reduce OOD accumulated error in the sampling process. Given the promising results in this paper, potential future work could further explore the compositional guiding model for transfer learning study a general aaage-scale pre-trained model serving as a unified guiding model, improving the transfer performance in arbitrary downstream tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Project, and the National Engineering Research Center for Big Data Software.
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arXiv:231..1512) 2023.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 m0n-innng discriminative compo- nents with random forests. In ECCV, 2014.
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data, 2023. URL https
//aaxii..rg//bb//3000 07194
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. NeurlPS, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
Abhimanyu Dubey, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Maximum Maximum-entropy entropy fine grained classification. In NewrIPS, 2018.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equilibrium. NewrlPS, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In NeurlPS, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, 2019.
Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. NeuriPS, 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. arXiv preprint arXiv:2406.02507, 2024a.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024b. 2024b.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma. and Percy Liang, Fine-tuning can distort pretrained features and underperform ofttof--distibutiinn arXiv preprint arXiv:2202.1000 10054, 2022.
11

---

Published as a conference paper at ICLR 2025
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Tuomas Kynkäääniemii, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance in limited interval improves sample and distribution quality in diffusion models. arXiv preprint arXiv:2404.07724 2024.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on patlern analysis and machine intelligence, 40(12):2935-290 2017.
Peiyuan Liao, Xiuyu Li, Xihui Liu, and Kurt Keutzer. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint arXiv:2206. 11420, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICMI,, 2021.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10))1345-1355 2009.
William Peebles and Saining Xie. Scalable diffusson models with transformers. In ICCV, 2023.
Lukáš Picek, Milan Sule, Jiří Matas, Thomas Jeppesen, Jacob Callmann-CClasseed Thomas Læssse, and Tobias Frøslev. Danish fungi 2020-not just another image recognition dataset. In WACV, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxema, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NewrIPS, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2020b.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganen Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024a.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganena Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024b.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jianxiong Xiao. James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
12

---

Published as a conference paper at ICLR 2025
Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaogiang Liu, Jiawei Li, and Zhenguo Li. Difffit: Unlocking transferability large diffusion models via simple parameter-efficiin fine-tuning, In ICCV, 2023.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NeurIPS, 2014.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-modeks arXiv preprint arXiv:2 2021.
Kaihong Zhang, Caitlyn H. Yin, Feng Liang, and Jingbo Liu. Minimax optimality of score-based diffusion models: Beyond the density lower bound assumptions, 2024. URL htttss:::://
erq/abs/ 2402. 15602
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Jincheng Zhong, Xingzhuo Guo, Jiaxiang Dong, and Mingsheng Long. Diffusion tuning: Transferring diffusion models via chain of forgetting. In NewrlPS, 2024.
A IMPLEMENTATION DETAILS
In this section we provide the details of our experiments. All of our experiments are inplemented using PyTorch and conducted on NVIDIA A100 40G GPUs.
A. BENCHMARK DESCRIPTION
This section describes the benchmarks used for finetuning.
Food101 (Bossard et al. 2014) This dataset consists of 101 food categories with total of 101,000 images. For each class, 750 training images preserving some amount noise and 250 manually reviewed test images are provided. All images were rescaled to have maximum side length of 512 pixels.
SUN397 (iaoo et al., 2010 The SUN397 dataset contains 108,753 images of 397 well-sampled categories from the origin Scene UNderstanding (SUN) database. The number of images varies across categories, but there at least 100 images per category. We finetune domain model on a random partition of the whole dataset with 76, 128 training images, 10,875 validation images and 21,750 test images.
DF20M (Picek ef al. 2022) Danish Fungi 2020 (DF20) is a new fine-grained dataset and benchmark featuring highly accurate class labels based the taxonomy of observations submitted to the Danish Fungal Atlas. The dataset has a well-defined class hierarchy and rich observational metadata. It characterized by a highly imbalanced long-tailed class distribution and a negligible error rate. Importantly, DF20 has no intersection with ImageNet, ensuring unbiased comparison of models fine-tuned from ImageNet checkpoints.
Caltech101 (Griffin et al., 2007) The Caltech 101 dataset comprises photos of objects within 101 distinct categories, with roughly 40 to 800 images allocated to each category. The majority of the categories have around 50 images. Each image is approximately 300x200 pixels in size.
CUB-200-201 GGifffnn al. 2007) CUB-200-2011 (Caltech-UCSe Birds-200-2011 1) is an expansion of the CUB- 200 dataset by approximately doubling the number of images per category and adding new annotations for part locations. The dataset consists of 11,788 images divided into 200 categories.
ArtBench10 (Liao et al. 2022) ArtBench- 10 is class-balanced, standardized dataset comprising 60,000 high- quality images of artwork annotated with clean and precise labels. It offers several advantages over previous artwork datasets including balanced class distribution, high-quality images, and standardized data collection and pre-prscs procedures. contains 5,000 training images and 1, 1,000 testing images per style.
13

---

Published as a conference paper at ICLR 2025
Oxford Flowers Nilsback & Zisserman 2008) The Oxford 102 Flowers Dataset contains high quality images of I02 commonly occurring flower categories in the United Kingdom. The number of images per category range between 40 and 258. This extensive dataset provides an excellent resource for various computer vision applications, especially those focused on flower recognition and classification.
Stanford Cars (Krause et al., 2013) In the Stanford Cars dataset, there are 16,185 images that display 196 distinct classes cars. These images are divided into training and testing set: 8,144 images for training and 8,041 images for testing. The distribution of samples among classes is almost balanced. Each class represents a specific make, model, and year combination, e.g., the 2012 Tesla Model S or the 2012 BMW M3 coupe.
A.2 EXPERIMENT DETAILS
For all our experiments, we use the ImageNet pre-trained DiT-XL/2 (Peebles & Xie, 2023) as the unconditional model to provide the guidance in DoG. For finetuning domain, provide the hyperparameter configuration below:
Table 5: Hyperparameter of domain transfer experiments
Hyperparameter
Configuration
Backbone
DiT-XL/2
Image Size
256
Batch Size
32
Learning Rate
le-4
Optimizer
Adam
Training Steps
24,000
Validation Interval
24,000
Sampling Steps
50
B PROOFS OF THEORETICAL EXPLANATION IN SECTION 3.3
Theorem (Full version of Proposition D. Denote
Xx, log Pw DoG
(xele, D'st) = Vx, log pxxel, D' (w 1) (Vx, log pxg D(st) Vx, logp(x:))
as the underlying score function corresponding to domain guidunce, and let
Vxi logPw Pw (xelc, D'st) = Xx, log p(xelc, Distt + (w 1) (Vx. log p(xele, Dist) ) Vxi log (Dtst)
CFG
denote the score function of CFG. Then domain guidance is equivalent to applying classifier guidance to the target domain:
Xx, log p (xrle le, D'st)= = Vx, log
DoG
CFG (x,lc, Mak/,,DeEe D'stt + (w 1VV 1
(8)
Proof. Since
Xx, log Pe (xelc, D Vx, log Pw (x, D'sst =(w (Vx. logp(x;|D'st) Vx, log p(x:)
DoG
Using Bayes' rule, we have:
(Digt
p(Dist|/x))
v(xx)
Thus we have:
Xx, (Dts(
Vx. logp(x;) Vx. (D'st x.)
14

---

Published as a conference paper at ICLR 2025
Proof of Theorem ! We denote Pe(x)) Ei= {>x/// ys) be the marginal distribu-
tion at time conditioning on the dataset samples = {urh== Yi ~ p(y) and p (x+) = fy (()) Xo - y) is the marginal distribution of the ground truth. Then we have:
ED-D(D) lp (xx) (x(x)))]
S BDwD(D) (pT( (x
For a dataset D (yil Vi ~ p(y), we have p(D) = II p(y:). Thus we have:
EDwp(D) (pi(xx) {c(xx)))
= -/ Ju Mi 1=1
II p(1s)
(xx =0)) (xx))
N
{ JINI 1.
II I(Vs))
N (g(x; -/:) -p² (x+))
1=1
p(yx)) N2 (x(xx yi) (xx))
p(y))p(1s) N2 (a(xe|x0 y.) pr (x.)) (a(x.(x0 yi) 22 (xx))
U::V:
N
p(y:) (a(xe/xo = -1)-1i (xx)))
(9)
N2
1=1
1
(10)
N
where Eq 9
is because Sy. p(y)) (a(x,/x0 y.) (x)(x,) = 0, and Eq 10 is because
ly. p(v:) (g(x:|x0- y:) - (x.)))) < 1. result:
EDwp(D) (xx) - (x(x))l S VN
C DETAILS OF THE 2D TOY EXAMPLE
We randomly generated 100 Gaussians M, = {Ki, U;, Σj} as the pre-train data distribution and generated Gaussians M, in a selected area as the distribution of the target domain. We divide the Gaussians into two classes Mai and M, each occupying a selected area as the two class conditions. Given above, we can write the data density as:
Source density Po P=(x) Σ
@.V(2 E,),
ieM,
target density pr(x: Σ
(xr E,),
iEM.
class-conditional target density p. (x(xc)) =
Σ
d:N(zlu,, E.).
tEM.
where the multivariate Gaussian distribution is defined as:
((2xT² det ( exp (- (x---)) /)/²²² - -11)
N(xlp, Σ,)
15

---

Published as a conference paper at ICLR 2025
We implement the denoising network 4-layer fully connected ReLU network with hidden feature dimension 64. We use sinusoidal positional embeddings for time conditioning as in (Ho et al. 2020), and we add the time embedding every intermediate layer. Unlike traditional CFG where use dropout ratio to learn the non-conditinal distribution, here we train separate unconditional model on (x)) and ps (a:) for guidance, similar (Karras et al. 2024b). We parameterize the network to explicitly output the score function. The pre-train model converges after 10000 Adam steps with
batch size of 128 and learning rate le-3. For fine-tuning on the target domain, the model converges after 1000 steps.
For training, we used the DDPM noise schedule from Ho et (2020). We used the DDIM sampler Song et al. (2020a) for 20 sampling steps to generate our samples. For all of our experiments, we set the CHG weight and the DoG weight at 2.
D ADDITIONAL EXPERIMENT RESULTS
Here we provide the additional results for the Precision and Recall metrics (Kynkäänniemi al. 2019). Notably, DoG enhances precision without compromising recall, indicating overall improvement in generation quality.
Table 6: Precision Comparisons downstream tasks with pre-trained
Dataset
CUB
Stanford Car
Average Precision
Food
SUN
Caltech
DF-20M ArtBench
Method
Bird
Fine-tuning (wlo guidance)
00.376 0.583 0.536
0.143
0.331
0.502
0.821
0.470
Classifier free guidance
0.455 0.590
0.668
0.331
0,501
0.537
0.831
0.559
Domain guidance
0.533 0.601 0.715
0.431
0.631
0.708
0.901
0.646
Table Recall 1 Comparisons on downstream tasks with pre-trained DIT-XL-
Dataset
CUB
Stanford Car
Average Recall
Food
SUN
Caltech
DF-20M ArtBench
Method
Bird
Fine-tuning (w/o guidance)
0.652 0.326 0.650
0.960
0,840
0.712
0.212
0.621
+ Classifier-free guidance
0.640 0.370
0.548
0.890
0.840
0.711
0.230
0.604
+ Domain guidance
0.651 0.370 0.546
0.860
0.840
0.638
0.230
0.590
We conducted experiments applying DoG to off-the-shelf LoRAs of the SDXL. model available in the Huggingface community. The results Table show that DoG can enhance the CLIP Score of the fine-tuned model significantly.
Table 8: CLIP Score 1 Enhance the transfer of the SDXL model with the off-the-shell LoRAs
CLIP Score f
Chalkboard Drawing Style f
Yarn Art Style 1
Real data
36.02
33.88
Off thetthe-shef LoRAs with CFG
27.23
34.89
Off- the-shelf LoRAs with DoG
35.24
35.03
Table FID J. Transferring pre-trained DIT-XI----1125122 to Food 512x512 dataset
Food (512x512)
FID
Relative Promotion 1
Fine-tuning with CFG
13.56
0.0%
Fine-tuning with DoG
11.05
18.5 %
16

---

Published as a conference paper at ICLR 2025
Table 10: FID &. Comparision with different pairs of pre-trained and guiding models
CUB 256x256 FID /
Fine-tum tune DiT-L/2-300M |
Fine-tune DiT-XL/2-700M 1
Standard CFG
15.20
5.32
DoG with DiT-L-2-300M
6.56
3,85
DoG with DIT-XL-2-700M
8.80
3.52
Table 11: FID 4 Incorporate with DiffFit
Transfer DiT-XL/2 to CUB 256x256
FID 4
DiffFit with CFG DiffFit with DoG
4.98
3.66
E RELATIONS WITH AUTOGUIDANCE
Autoguidance (Karras et al. 2024a) focuses on guiding models from suboptimal outputs to better generations, typically using less-traine trained version of the model for guidance, equipped with guid- ance interval techniques (Kykkäänniemi et al. 2024). This method has demonstrated remarkable performance in EDM group methods.
DoG and Autoguidance originate from different contexts: generation within the same domain, whereas DoG is designed adapt pre-trained models to domains outside their original training context. It is inappropriate to consider the original pre-trained model in DoG suboptimal version of the fine-tuned model.
Autoguidance focuses on improving
Furthermore, pre-trained models are usually optimized on extensive datasets with distinct training strategies, whereas fine-tuned models are trained on different downstream domains with limited data points. This difference undermines the same degradation hypothesis posited by Autoguidance.
Additionally, the pre-trained model is generally well-trained model with rich knowledge, while fine-tuning to a small dataset often leads to catastrophic forgetting and poor fitness of the target domain. It is not accurate state that the pre-trained model is a bad version the fine-tuned one.
As demonstrated in Table I0 using a better DiT-XL/2 to guide the model fine-tuned from DiT-L/2 conceptually achieves the transfer gain.
Autoguidance is mainly proved in EDM context, whereas DoG mainly provides evidence in DiT-based model and text2img stable diffusion models. Exploring general understanding of guidance is indeed an valuable avenue for future work. We believe it is valuable to formally establish the properties required for a unified unconditional guide model and to develop a general guidance model beneficial to all diffusion tasks.
17

---

Published as a conference paper at ICLR 2025
F ADDITIONAL QUALITATIVE SAMPLES WITH OFF-THE-SHELF LORAS
Prompt:
CFG
DoG
"A ChalkBoarDrrawing of a house" "
ChaltBoardDrawing of a crazy clown"
"A cute puppy ChalkBoardDrawing
Figure 6: Qualitative showcases of the Chalkboard style transfer task, utilizing default guidance scale of 5.0 for each generation.
18

---

Published as a conference paper at ICLR 2025
Prompt:
CFG
DoG
"Kermit the frog, yarn art style" "
"A clown, yarn art style" "
"A castle cake, yarn art style" "
Figure 7: Qualitative showcases of the Yarn style transfer task. utilizing default guidance scale of 5.0 for each generation.
19


Reviews:
[{"review_id": "59WqgSBJS6", "reviewer": "Reviewer_CbgC", "summary": "The paper introduces \"Domain Guidance\" (DoG), a new transfer approach for pre-trained diffusion models aimed at enhancing domain alignment and generating high-quality outputs. DoG builds on classifier-free guidance (CFG) but adapts it to transfer learning by incorporating pre-trained model knowledge to guide the generative process. Through empirical and theoretical analysis, the authors demonstrate DoG’s effectiveness in maintaining pre-trained knowledge, improving image fidelity, and reducing out-of-distribution errors. The results indicate substantial improvements in several metrics (FID and FDDINOv2) across various downstream tasks compared to standard fine-tuning and CFG.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. DoG’s design as a CFG variant is well-motivated, with clear delineation of the benefits over standard CFG. The illustrations of DoG’s guidance process effectively highlight its domain-alignment benefits.\n2. Quantitative results show that DoG significantly outperforms CFG, particularly on datasets with substantial domain shifts. Qualitative results are also compelling, with DoG generating visually consistent images even with increased guidance weights.\n3. The paper provides a theoretical foundation for DoG, showing how it maintains domain alignment and mitigates OOD sampling errors, supported by an illustrative analysis using Gaussian mixtures.", "weaknesses": "1. The paper could benefit from more discussion on DoG’s adaptability, particularly in domains vastly different from the pre-trained model’s source domain.\n2. The influence of guidance weights is explored, but further sensitivity analysis could strengthen the understanding of DoG’s stability across diverse settings.\n3. While diverse datasets are used, further tests on varying resolutions and different architectures could better illustrate DoG’s flexibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GN8K8dHQ2K", "reviewer": "Reviewer_CXsS", "summary": "This submission proposed a new conditioning method to transfer a diffusion model\n to a new domain named Domain Guidance (DoG).\nThe idea behind DoG is to pull the generation process toward the target domain\nfrom the pre-trained domain.\nGiven a pretrained model and fine-tuned version of it, domain guidance guides the process\nto the direction that the fine-tuned one indicates and against the pretrained ones's direction.\nTheoretical analyses show that DoG mitigates sampling from out-of-distribution area of the target domain.\nExperiments show that DoG outperforms CFG in combination with diffusion models finetuned in relatively small\ndatasets.", "questions": "Please see Weaknesses 1.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The motivation and idea of DoG is clearly presented, and the paper is overall easy to follow.\n- Theoretical analyses and toy examples well depict the advantage of DoG.\n- DoG outperformed CFG, a de facto standard method for diffusion guidance with a margin.", "weaknesses": "- Range of applicable models is not discussed well:\nThe experiments are conducted with ImageNet-pretrained  DiT-XL/2 and \nbut is it mandatory to use the same architecture and initialization weights to perform DoG?\nFurthermore, transfering diffusion models to small datasets are often done with LoRA [a] rather than full finetuning.\nIs DoG applicable to LoRA-based transfer?\n[a] LoRA: Low-Rank Adaptation of Large Language Models \n\n\n- Relationship with Autoguidance:\nA similar idea is presented in a preprint [b] which uses less-trained version of the model for guidance, \ninstead of unfinetuned version in DoG.\nDiscussing this may be useful, while it it not mandatory due to its unpublished state and missing it should not be penalized.\n[b] Guiding a Diffusion Model with a Bad Version of Itself, NeurIPS 2024 to appear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YOD4iPxpj", "reviewer": "Reviewer_Pu4u", "summary": "This work leverages pre-trained knowledge as the domain guidance to guide the model toward the target domain. This work thinks that domain guidance facilitates better domain alignment and higher-quality generations, and provides correlation theoretical analyses.", "questions": "it is the same as Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The idea of using the pre-trained model as a domain guidance to guide transfer learning is innovative.\n\nQuality and Clarity: The quality of this is satisfactory and the analysis is thoughtful. The paper is well-written, with a thorough and comprehensive ablation study.\n\nSignificance. This work provides a contribution to the transfer learning of diffusion learning. In particular, this work can be seamlessly integrated into the existing fine-tuning models without additional training.", "weaknesses": "1. Is there a domain gap between personalized scenarios and general scenarios?\n2. During the transfer learning process, does all the knowledge from the pre-trained model need to be effectively utilized?\n3. How can the impact of the knowledge in the pre-trained model be assessed in the target domain?\n4. For DF-20M (which has no overlap with ImageNet) and ArtBench-10 (whose feature distribution is completely distinct from ImageNet), why can guidance from the model pre-trained on ImageNet lead to better generation?\n5. Is the distribution of the downstream domain a subset of the distribution of the pre-trained domain?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)