Efficient Distillation of Classifier-Free Guidance using Adapters
Authors: Cristian Perez Jensen
Seyedmorteza Sadat
Venue: Accepted by TMLR ()

Abstract: While classifier-free guidance (CFG) is essential for conditional diffusion models, it doubles the number of neural function evaluations (NFEs) per inference step. To mitigate this inefficiency, we introduce adapter guidance distillation (AGD), a novel approach that simulates CFG in a single forward pass. AGD leverages lightweight adapters to approximate CFG, effectively doubling the sampling speed while maintaining or even improving sample quality. Unlike prior guidance distillation methods that tune the entire model, AGD keeps the base model frozen and only trains minimal additional parameters ($\sim$2%) to significantly reduce the resource requirement of the distillation phase. Additionally, this approach preserves the original model weights and enables the adapters to be seamlessly combined with other checkpoints derived from the same base model. We also address a key mismatch between training and inference in existing guidance distillation methods by training on CFG-guided trajectories instead of standard diffusion trajectories. Through extensive experiments, we show that AGD achieves comparable or superior FID to CFG across multiple architectures with only half the NFEs. Notably, our method enables the distillation of large models ($\sim$2.6B parameters) on a single consumer GPU with 24 GB of VRAM, making it more accessible than previous approaches that require multiple high-end GPUs. We will publicly release the implementation of our method.

Published in Transactions on Machine Learning Research (10/2025)
Efficient Distillation of Classifier-Free Guidance using Adapters
Cristian Perez Jensen" ETH Zürich
ejense
Scyedmorteza Sadat" ETH Zürich
ssadat@ttha.ch
Reviewed on OpenReview: https: openreview........ net/ per/fo uM28F44
Abstract
While classifier-free guidance (CFG) is essential for conditional diffusion models, it doubles the number of neural function evaluations (NFEs) inference step. To miligale this inelficiency, we introduce adapter guidance distillation (AGD), a novel approach that simulates CFG in a single forward pass. AGD leverages lightweight adapters to approximate CFG, effectively doubling the sampling speed while maintaining or even improving sample quality. Unlike prior guidance distillation methods that tune the entire model, AGD keeps the base model frozen and only trains minimal additional parameters (~2%% to significantly reduce the resource requirement of the distillation phase. Additionally, this approach preserves the original model weights and enables the adapters to be seamlessly combined with other checkpoints derived from the same base model. We also address a key mismatch between training and inference in existing guidance distillation methods by training on CFG-guided trajectories instead of standard diffusion trajectories. Through extensive experiments, we show that AGD achieves comparable or superior FID to CFG across multiple architectures with only half the NFEs. Notably, our method enables the distillation of large models (~2.6B parameters)
single consumer GPU with 24 GB of VRAM, making it more accessible than previous approaches that require multiple high-end GPUs. We will publicly release the implementation of our method.
1 Introduction
Score-based diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020b) are a family of generative models that learn the data distribution by reversing a forward process that progressively corrupts the data until becomes indistinguishable from pure Gaussian noise. Theoretically, running the reverse diffusion process should enable accurate sampling from the data distribution, assuming access to the ground truth score function. However, in practice, unguided sampling from diffusion models often produces low-quality images that fail to align well with the given input condition due optimization errors (Karras et al., 2024a). Accordingly, CFG (Ho & Salimans, 2022) has become a crucial technique in modern conditional diffusion models to enhance both generation quality and aligument to conditioning signals though this comes at the expense of reduced sample diversity (Ho & Salimans, 2022; Sadat et al., 2024).
CFG an inference method that enhanoes generation quality by leveraging the difference between conditional and unconditional model predictions each inference step. This difference serves as an update direction to improve both quality and alignment with the target condition. However, CFG requires two forward passes per inference step, resulting in twice the sampling cost compared to unguided sampling. This increased cost
These authors contributed equally this work.

---

Published in Transactions on Machine Learning Research (10/2025)
AGD
Stable Diffusion XI.
Stable Diffusion 2.1
Diffusion Transformer
Figure 1: Generated samples using adapter guidance distillation (AGD) applied to various models. By efficiently baking classifier-fis guidance (CFG) into the base diffusion model, AGD generates high-quality samples with only single forward pass per inference step, which results in doubling the sampling speed compared to standard CFG.
introduces a significant computational overhead, especially when sampling from large-scale diffusion models or employing these pretrained modcls for tasks such as score distillation (Poolc et al., 2023).
In this paper, we aim double the sampling speed of CFG by training small set of adapters to integrate the CFG behavior directly into the model. Our method, called AGD, learns to replicate the CFG-guided output each inference step using single forward pass while preserving the original diffusion model weights. These lightweight adapters add only 1-5% more parameters to the base model and introduce negligible latency overhead during inference. Since the base model remains frozen during training, and only the adapter parameters are updated, AGD is computationally efficient and can be trained on a single consumer GPU with 24 GB of VRAM, even for large models like Stable Diffusion XL (SDXL). Furthermore, AGD allows the trained adapters be seamlessly integrated with other checkpoints originating from the same base model, such IP-adapters (Ye ct al., 2023). We demonstrate that our approach maintains or improves generation quality compared to standard CFG and outperforms existing methods such as guidance distillation (GD) (Meng et al, 2023), all while significantly reducing resource requirements during training.
Morcover, we identify and address mismatch between training and inference trajectories in prior guidance distillation methods. We argue that effective guidance distillation requires training on CFG-guided Irajectoiie computed by running the sampling process with CFG, as these differ significantly from standard diffusion trajectories obtained by adding noise the training data. Furthermore, training on guided trajectories eliminates the need to load teacher model during distillation, thus reducing memory requirements when training AGD. Another advantage is that the distillation can be performed entirely on the synthetic data
2

---

Published in Transactions on Machine Learning Research (10/2025)
Xt-1
X
X,, 6,C
Base model
+ é
X1, Xt, C,w
Base + adapter
+ - L X1, x,,t,c,w
Base adapter +
Xx, 0
é
(a) Trajectory collection.
(b) Training on collected trajectories.
(c) Inference with adapters.
Figure 2: High-level overview of AGD components. (a) Instead of training on diffusion trajectories, we first. run the sampling process with classifier- free guidance (CFG) and use the resulting guided trajectories (í.e. intermediate prodictions at cach time step t) as our training dataset. (b) We then introduce small adapters to the base model and train them to replicate the CFG-guided predictions from (a) while keeping the base model frozen. (c) During inference, the base model combined with the trained adapter produces guided predictions in a single forward pass, effectively doubling the sampling speed compared to CFG.
generated by the teacher model without needing any real dataset in advance. Our experiments demonstrate that training CFG-guided trajectories enhances performance compared with training diffusion trajectories.
Figure 2 gives an overview of different components in AGD. In summary, our primary contributions are:
We introduce AGD, an efficient method for simulating CFG in single forward pass by training lightweight adapters alongside frozen base diffusion model, eliminating the need to fine-tune the entire model.
We propose training AGD on CFG-guided trajectories instead of diffusion trajectories, reducing the mismatch between training and inference and improving performance.
3. We demonstrate the resource efficiency of AGD by successfully distilling SDXL (2.6B parameters) on a single RTX 4090 GPU with 24 GB of VRAM.
4. Through extensive experiments, we show that AGD matches surpasses CFG in performance across various models such as Diffusion Transformer and Stable Diffusion while doubling the sampling speed compared to CFG.
2 Related work
Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Song el al., 2020a;b; Ho et al., 2020) have emerged as leading approach for generative modeling across various domains, including images (Ramesh et al., 2022; Rombach et al., 2022; Dai et al., 2023; Saharia et al., 2022), text (Hoogeboom et al., 2021; Li et al., 2022; Austin al., 2021), audio (Evans et al., 2024), and molecular generation (Hoogeboom 2022). Since the introduction of DDPM (Ho et al., 2020), significant progress has been made in multiple aspects, such as refining network architectures (Hoogeboom et al., 2023; Karras et al., 2024b; Peebles & Xie, 2023; Dhariwal Nichol, 2021), developing more efficient sampling methods (Song et al., 2020a; Karras et al., 2022; Liu et al., 2022; Ta al., 2022; Salimans & Ho, 2022b), and introducing novel training strategies (Nichol & Dhariwal, 2021; Song et al., 2020b; Salimans & Ho, 2022a; Rombach ct al., 2022; Karras al., 2022; 2024b). Nevertheless, various guidance techniques (Dhariwal & Nichol, 2021; Ho & Salimans, 2022; Karras et al, 2024a; Sadat et al., 2025b) have remained essential for enhancing generation quality and the alignment between conditioning inputs and generated outputs (Nichol et al., 2022), though they lead to increased sampling time (Ho & Salimans, 2022), reduced diversity (Sadat et al., 2024; Kynkäänniemi et al., 2024), and excessive oversaturation (Sadat et al., 2025a).
al.,
Several works have recently explored modifying the weight schedule of CFG by applying guidance only certain sampling steps (Castillo et al., 2023; Wang al., 2024: Kynkäänniemi al., 2024), primarily balance diversity and quality in generation. However, these mot still require two NFEs for most steps
3

---

Published in Transactions on Machine Learning Research (10/2025)
and therefore cannot fully double the inference speed. Additionally, our approach is orthogonal to these methods, as the distilled model can be used for the steps where CFG is applied in the above works.
Alternatively, Meng et al. (2023) introduced guidance distillation (GD), which fine-tunes diffusson model to generate guided predictions in a single forward pass. However, fully fine-tuning the base model is often inefficient and unstable for large models, overwrites the original model weights, and demands high-end GPUs with substantial memory for training. To improve efficiency of GD,, Hsiao et al. (2024) introduced plug- and-play diffusion distillation (PPDD), which injects residuals via auxiliary model. However, their most accurate variant (requiring approximately 42% additional parameters) still underperforms CFG by roughly 14% in FID. To address these limitations, we propose more efficient distillation method using adapters, which achieves substantially better performance while remaining significantly more parameter-efficinn. Moreover, both Meng et al. (2023) and Hsiao et al. (2024) trained their models on standard diffusion trajectories, which we show to be less effective than training the distillation process on CFG-guided trajectories.
Finally, adapters (Houlsby et al., 2019) have emerged as a parameter-effiienl solution for fine-tuning large- scale diffusion models, mainly for integrating image conditions into pretrained text-to-image models (Mou et al., 2024; Ye et al., 2023). In contrast, we leverage adapters to inject guided predictions directly into the model's forward pass. Notably, we demonstrate that adapters not only require significantly fewer training resources but also slightly outperform full fine-tuning.
3 Background
In this section, we provide a concise overview of diffusion models. Consider a data point ~ Pdata and noise e (N, I), and let the forward diffusion process be defined as x, = x + (t)e, where noise is gradually introduced over time e [0, T]. The function (t) serves as the noise schedule, determining the extent of perturbation at each step, with (0) = 0 and (T) = maxx As shown by Karras et al. (2022), this process can be described by the following ordinary differential equation (ODE):
dx, = = Xx, Po
(1)
where Pt is the marginal distribution of the noisy data at time step t, transitioning from the original data distribution Po = Pdata to a Gaussian prior pr = NO of I)
Assuming access to the time-dependent score function Vx, log);(X)), one can solve this ODE in reverse, Le., from = T to t = O, to generate new samples from Pdata- The unknown score function Vx, log (x,,) is typically learned using a neural denoiser ()) which is trained to recover clean samples from their noisy counterparts Xr, Additionally, conditional generation can be achieved by extending the denoiser to (x,,,, f, c), where e represents auxiliary conditioning information, such as class labels or text.
Training Following (Ho et al., 2020), the denoiser Do b) is commonly parameterized as Dx(x,, t) x, (1) t),
(2)
and is trained by predicting the added noise e in X,, that is by solving
0" E argmin Ex.e.t [x)eo(x,,) t) - =ll?].
(3)
After training, the score function can be approximated via
Vx, lx,) 2 Do(x,, Xt
(0(x1,1) o(t)
(4)
a(t)
Classifier-free guidance CFG is an inference technique aimed at improving the quality of generated samples by blending the outputs of a conditional and an unconditional model (Ho & Salimans, 2022). Specifically, CFG adjusts the denoiser's s output at each sampling step according to
Eo(XI, c, web (xe, (,) (w 1) 1,0),
(5)
4

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 1 Trajectory collection for AGD.
Require: Set of conditions C, Guidance scale range [Wiins Wmax], Pretrained diffusion model cæ
1: Ω = 0
2: for CEC C do
3:
w Unif (Wiin (mak))
4: Run the reverse diffusion process from Equation (1) using co with w and N steps Cache (x,, l,c,w) and the CFG-guided prediction éo (x,, L c, c,w) at each sampling step:
Ω + QU {(x,, C, Ex(x,, i, w))
end for
Algorithm Adapter training for AGD.
Require: Trajectory dataset St from Algorithm 1, Model with adapters cj0,0): Loss function while not converged do k,,,t,c.,,k Unif([)) 3: L= L= (0, [0)
4: Update vb with gradient step on
5: end while
where w = 1 corresponds to unguided sampling, and c 0 represents the unconditional prediction. The unconditional model (x,,,, 0) is typically trained by randomly replacing the conditioning input with 0 during training. Alternatively, dedicated denoiser can be trained separately to approximate the unconditional score (Karras et al., 2024b). CFG is known significantly improve generation quality, though it comes at the cost of doubling the sampling time (Ho & Salimans, 2022).
4 Adapter guidance distillation
We now introduce our method, adapter guidance distillation (AGD), for doubling the sampling speed of CFG. As shown in Figure 2, AGD consists of two main components: (1) training on CFG-guided trajectories instead of standard diffusion trajectories, and (2) training lightweight adapters to distill CFG instead of fine-tuning the full model. Below, discuss each component in detail, with Algorithms and 2 also providing the training details of AGD.
4.1 Training CFG-guided trajectories
Prior guidance distillation mothods are trainod on standard diffusion trajectorics, where noise added to the training data, and the CFG prediction matched at each inference step (Meng el al., 2023). However, since CFG modifies the reverse process of diffusion models, guided trajectories differ significantly from standard diffusion trajectories, as shown in Figure 3. We argue that training directly on CFG-guided trajectories enhances guidance distillation by exposing the model to regions in space that the guided reverse process will follow. To bridge the gap between training and inference, we thus train AGD directly on CFG-guided trajectories. We generate guided trajectories as outlined in Algorithm 1, which are then used to train AGD, Since these trajectories can be cachod, the teacher model does not need to be loaded during training, freeing up VRAM. Moreover, because this method only depends on generated samples from the teacher model, it does not require external dataset for training. Additionally, the trajectory dataset only needs to be collected once, enabling efficient hyperparameter tuning for the adapters.
4.2 Efficient guidance distillation with adapters
For more efficient training, AGD only uses small learnable modules, or adapters (Houlsby al., 2019), replicate the effect of CFG. Unlike tuning the whole diffusion network in GD (Meng al., 2023), we freeze the original model weights, ensuring that the base model is still available after training. This also allows
5

---

Published in Transactions on Machine Learning Research (10/2025)
T 1b]
head
Traénable +b
(a) Visualizing the marginal densities of the diffusion pro- cess with a mixture of Gaussians as the data distribution.
adapter
block
Prosen
adapter
block
(b) Illustration the distribution mismatch between conditional diffusion (blue) and classifier-free guidance (red) trajectories during the reverse diffusion process.
L.C.
One-dimensional illustration the mis- match between standard diffusion trajectories used for training in existing guidance distillation methods (such as GD (Meng al., 2023)) and the actual CFG-guided trajectories followed during inference.
Figure 3:
Figure 4: Visual illustration of the trainable adapters alongside the frozen base model. The adapters are typically integrated with attention layers (either self- attention cross-attention), and their outputs added to those of the frozen attention blocks.
to use the learned adapters with other checkpoints that obtained from the same base model, such IP-adapters (Ye et al., 2023). The details of the adapters used in AGD are given below.
Adapter formulation Let fo denote an intermediate layer in the network with Z E RLxd being its upstream imput. Further, fe receives the time step t and the condition embedding as input. An adapter with parameters &b is a layer that combines So with encodings of the guidance scale w, and the input conditions (t, c) via residual connection:
fe (Z, w,,, So fo 4, (o (2,,ttc))
(6)
This architecture is illustrated in Figure 4. During training, the model weights @ are kept frozen and only the adapter parameters &b are optimized to match the CFG step based on the trajectory dataset, introduced in Section 4.1, d.e.,
l, / E argmin B[E(E(e (xt, e, w), (0(x,,,,,
(7)
+
where ěØ (x,,,,,,, c, (,w) denotes CFG step with guidance scale w, c (x,, c,w) ) the output of the model with the adapters, and is the loss function.
Adapter architecture We mainly experiment with two adapter architectures: (1) cross-attention adapters, and (2) offset adapters. Let C [c....., cc] represent the matrix containing all conditioning embeddings (e.g., guidance scale, prompt embeddings, elc.), linearly projected to the same dimensionality via learned projection. Akin to IP-adapter (Ye et al., 2023), the cross-attention adapter formulates Ie as
QK
I (Z w. f. Softmax
V
(8)
Vd
where ZW, K CW and V CW.. e. The offset adapter formulates I&p
C
IN (Z, w, = MLP
Σ
c;
(9)

---

Published in Transactions on Machine Learning Research (10/2025)
We found that offset adapters perform better for simpler models like DiT, whereas cross-attention adapters are more effective for text-to-image models. Several ablations other adapter design space are provided in Appendix B.
Implementation details We embed the guidance scale w via a Fourier feature encoder (Tancik et al,, 2020) followed by na multi-layer perceptron (MLP). We also extract the text or class embeddings from the base model (e.g., CLIP embeddings) and lincarly project them into the same dimensionality as the guidance scale embedding. In DiT (Peebles & Xie, 2023), we place adapter in each transformer block after the sclf-attention mechanism. For text-to-image models such Stable Diffusion 2.1 (SD2.1) (Rombach al., 2022) and SDXL (Podell et al., 2024), place the adapters in conjunction with the cross-attention layers, since the text prompt is only used in these blocks.
Efficiency Since the adapters introduce only 1 5% additional parameters relative to the base model, their computational overhead remains negligible during both training and inference. Furthermore, unlike CFG, which requires two forward passes per diffusion step, our approach performs only one, effectively reducing the NFEs by half. Consequently, our method achieves twice the speed of CFG when generating samples from pretrained diffusion models.
5 Experiments and results
Setup We evaluate AGD on class-conditional generation using 256x256 DIT-XL/2 (Peebles & Xie, 2023), and text-to-image generation using 768x768 Stable Diffusion 2.1 (SD2.1) (Rombach et al., 2022) and 1024x 1024 Stable Diffusion XL (SDXL) (Podell et al., 2024). All experiments are conducted on a single RIX 4090 GPU GB of VRAM). Training is performed using the Adam optimizer (Kingma & Ba, 2014) without weight decay, where the learning rate follows a linear warm-up to I x 10-4 over the first 10% of steps, after which it decays via a cosine annealing schedule (Loshchilov & Hutter, 2016). For training adapters on DiT, trajectories are sampled with guidance scales ~ Unif((1,6)) 6)), with four trajectories per class label of ImageNet (Deng al., 2009). For text-to-image models, we randomly select 500 captions from the COCO-2017 training set (Lin et al., 2014), generating single trajectory per caption with guidance scales Uni 12)). Please refer to Appendix D for additional details regarding the experiments.
Evaluation metrics We mainly use FID (Heusel et al., 2017) to mcasure the quality and diversity of generated images, since it closely aligns with human perception. Given FID's sensitivity to implementation details, we evaluate all models under identical conditions to ensure consistency. Additionally, we report precision as a measure of quality and recall as an indicator of diversity (Kynkäšnniemi et al., 2019).
5.1 Qualitative results
We evaluate the qualitative performance of AGD and CFG in Figure 5, generating samples using the same random seeds for both methods. Our results indicate that AGD produces images structurally similar to CFG while being more visually appealing across multiple models and resolutions. Thus, AGD retains the quality benefits of CFG while achieving twice the sampling speed per image.
5.2 Quantitative results
The quantitative evaluation of AGD and CFG is shown in Table 1. We observe that AGD achieves metrics comparable CFG, with both methods significantly outperforming the unguided sampling baseline. This confirms that AGD enhances generation quality similarly CFG while requiring only half the NFEs. Notably, AGD even slightly outperforms CFG for the DiT model.
5.3 Comparing AGD with guidance distillation
We next compare our method to guidance distillation (GD) (Meng et al., 2023), which fine-tunes the entire diffusion model to replicate guided outputs. We train AGD and GD under the same training setup using DiT
7

---

Published in Transactions on Machine Learning Research (10/2025)
CFG
AGD (Ours)
(a)
Diffusion Transformer (256x250)
CFG
AGD (Ours)
(b) Stable Diffusion 2.1 (768x768)
CFG
AGD (Ours)
(c) Stable Diffusion XL (1024x1024]
Figure 5: Qualitative comparison between AGD and CFG. AGD produces samples with comparable quality CFG while achieving twice the inference speed by requiring only single forward pass through the model. Additionally, AGD samples maintain structural similarity to CFG but often have better visual coherence.
Table 1: Quantitative comparison between AGD and CFG. AGD outperforms CFG in class-condiiional generation using DiT and performs similarly for text-to-image models (SD2.1 and SDXL). The throughput is computed with batch size 8.
Model
Guidance
FID
Prec. t
Recall f
NFEs
Steps/s
Unguided
12.57
0.67
0.74
250
15.84
DiT (Peebles & Xie, 2023)
CFG
5.30
0.83
0.66
500
8.83
AGD (Ours)
5.03
0.80
0.68
250
15.50
Unguided
49.94
0.39
0.63
50
1.81
SD2.1 (Rombach et al., 2022)
CFG
20.94
0.67
0.55
100
0.91
AGD (Ours)
21.09
0.66
0.55
50
1.79
Unguided
60.30
0.35
0.54
50
0.66
SDXL (Podell et al., 2024)
CFG
22.82
0.66
0.52
100
0.33
AGD (Ours)
22.98
0.67
0.52
50
0.64
as the base diffusion model for class-conditional ImageNet generation. Table 2 shows that AGD outperforms GD in FID while having significantly less trainable parameters. Thus, we conclude that GD can be made significantly more efficient by keeping the base model frozen and only training the adapters. Moreover,
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Comparing AGD and GD (Meng et al., 2023) Table 3: Importance of training on guided trajec- using DiT under the same training setup. AGD slightly torics. AGD performs best when trained on CFG- outperforms GD while only training the adapters instead guided trajectories instead of the standard diffusion of tuning the full model.
trajectories used in Meng et al. (2023).
Method
Params
FID
Prec.
Recall f
Method
FID |
Prec. t Recall
GD
676 M
5.66
0.80
0.67
CFG
5.30
0.83
0.66
AGD (Ours)
16M
5.03
0.80
0.68
AGD (Diffusion)
5.54
0.80
0.68
AGD (Trajectory)
5.03
0.80
0.68
Image prompt.
CFG
AGD (Ours)
GD (Meng et al., 2023)
AGD (Ours)
||
3
Figure 6: Comparison of AGD with guidance distil- lation (GD) (Meng et al., 2023) for unseen guidance scales. While GD fails completely for oft-of-domain guidance scales, AGD contimues to generate mcan- ingful images. The models in this experiment were trained for w E 6].
Figure 7: Using AGD with IP-adapter (Ye el al., 2023) and ControlNet (Zhang et al., 2023) for SDXL. AGD can be integrated with other checkpoints de- rived from the same base model, achieving the bene- fits of both modules.
Figure 6 shows that GD completely fails when used with guidance scales outside the domain during finc-tuning. In contrast, AGD remains robust to this issue, demonstrating better gencralization across guidance scales.
5.4 Importance of training on guided trajectories
In order to validate our claim that training on CFG-guided trajectories is beneficial, we compared AGD trained on standard diffusion trajectories with AGD trained on guided trajectories. As shown in Table 3, training on guided trajectorics leads a substantial improvement over training on standard diffusion trajectorics. Hence, conclude that bridging the train-inference gap by aligning these trajectories enhances performance, it focuses training regions of the space that important for CFG.
5.5 Combining AGD with other fine- innettned checkpoints
Figure 7 shows samples generated by combining AGD with IP-Adapter (Ye et al., 2023) and ControlNet (Zhang et al., 2023). As shown, AGD maintains high sample quality when integrated with other checkpoints derived from the same base model, enabling controllable generation at twice the sampling speed.
5.6 Training efficiency
Table compares the training speed and VRAM usage of AGD and GD (Meng al., 2023). We note that for larger networks like SDXL, AGD successfully distill the model using consumer GPU with 24 GB
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Comparing the memory requirements and training speed of AGD and GD. AGD enables the distillation of large models on an RTX 4090 with 24 GB of VRAM while also being significantly faster each training iteration.
Model
Method
VRAM (GB)
It/s
GD
17.67
0.52
DiT (Peebles & Xie, 2023)
AGD (Ours)
16.79
2.36
GD
Out-of-memory
SD2.1 (Rombach et al., 2022)
AGD (Ours)
23.83
2.05
GD
Out-of-memory
SDXL (Podell et al., 2024)
AGD (Ours)
22.77
3.94
FID
Precision
Recall
15
0.9
0.7
0.6
0.8
10
0.5
0.7
5
0.4
1.5
2
2.5
1.5
2
2.5
3
1.5
2.5
3
CFG
AGD (Ours)
Figure The performance of DiT with AGD as we increase the guidance scale. Compared to CFG, AGD offers better trade-off between precision and recall resulting in better FID for most guidance scales.
VRAM, whereas GD encounters oft-or-memory issues. Even when VRAM is not a constraint, each training step of AGD remains significantly more efficient than GD (~4.5x faster for DIT).
5.7 AGD and guidance scale
Figure 8 shows how the performance of AGD varies we increase the guidance scale w. We observe that the FID curve for CFG is more peaked, whereas the curve for AGD is relatively flatter, making it less sensitive to the exact guidance value at inference for good performance. Additionally, we note that AGD have a more favorable trade-off between precision and recall compared to CFG, resulting in better FID scores for most guidance scales.
5.8 Changing the scheduler at inference time
Next, we demonstrate that AGD is robust to the choice of scheduler for generating guided trajectories. In Figure 9, we show samples from the DDPM algorithm Ho et al. (2020) using adapters trained on DDIM trajectories (Song el al., 2020a). Even when a different. scheduler is used at inference, AGD consistently produces high- quality images. Morcover, Table reports comparable FID scores for both DDIM and DDPM. Note that the slight FID advantage of DDIM reflects its efficiency at lower sampling steps (e.g., 50), rather than any degradation in AGD.
Table 5: Compatibility of AGD with dif- ferent diffusion samplers at inference.
Sampler
FID
Prec. f Recall
DDIM DDPM
21.09
0.66
0.55
22.15
0.67
0.51
10

---

Published in Transactions on Machine Learning Research (10/2025)
DDIM
DDPM
Figure 9: Samples SD2.1 generated with AGD using the DDPM sampler, with adapters trained on DDIM trajectories. AGD successfully generates high-quality samples when used with different schedulers inference.
5.9 Training and inference wall-clock times
To further demonstrate the computational efficiency of our method, we compared the wall-clock training and inference times of AGD and GD on DiT using an RTX 4090 GPU. For training, GD requires 160.3 minutes to complete, whereas AGD achieves comparable performance in just 33.9 minutes (i.e. a 4.7x speedup). During inference, both AGD and unguided DiT require 6.0 seconds per sample, indicating that the adapter introduces negligible computational overhead. This confirms that the core advantage of AGD-roducing forward passes by half compared to CFG-translalee translates directly into inference speedup without incurring any additional per-step costs. The combination of faster training, reduced memory usage, and accelerated inference makes AGD compelling alternative to both standard CFG and existing guidance distillation approaches.
6 Conclusion
This paper introduced adapter guidance distillation (AGD), efficient approach to achieving the benefits of classifier-fr -frse guidance at half the sampling cost. By training lightweight adapters to estimate guided outputs and training on CFG-guided trajectories, we address both the computational overhead and the train-inference mismatch of prior guidance distillation methods. Through extensive experiments, we showed that AGD matches or surpasscs CFG's performance, remains robust previously unscen guidance scales, and can be trained on single consumer GPU even for large models such as SDXL. Thus, we believe that AGD offers efficient and flexible alternative to prior guidance distillation methods while eliminating the sampling overhead of classifier-free guidance. Future research directions could explore integrating AGD with enhanced guidance algorithms (Kynkäänniemi et al., 2024; Karras et al., 2024a; Sadat et al., 2025a) and leveraging adapters for other distillation techniques, e.g., adversarial distillation (Sauer et al., 2024; 2025), to further reduce the sampling time of diffuuson models.
References
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 17981-17993, 2021.
Angela Castillo, Jonas Kohler, Juan C Pérez, Juan Pablo Pérez, Albert Pumarola, Bernard Ghanem, Pablo Arbeláez, and Ali Thabet. Adaptive guidance: Training-fres acceleration of conditional diffusion models. arXiv preprint arXiv:2312.12487, 2023.
Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint ar Xrx::2330 15807, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEFE conference on computer vision and pattern recognition, pp. 248- 2882.5.. Icce, 2009.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural informalion processing systems,
2021.
Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast timing- coniingconddttooeed latent audio diffusion. In Forty-first International Conference on Machine Learning, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1250 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840- 118800-6511, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34:12454 -111445--1265, 2021.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867 8887. PMLR, 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high URL https://doi.or:
resolution images. CoRR, abs/2301.11098, 2023. doi: 235500/rXXiv2300 10. 00.445500/rXiv.. wttpxxivvv....11111 0301.11093.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference machine learning, pp. 2790-2799. PMLR, 2019.
Yi-Ting Hsiao, Siavash Khodadadeh, Kevin Duarte, Wei-An Lin, Hui Qu, Mingi Kwon, and Ratheesh Kalarot. Plug-and-play diffusion distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13743-13752. 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565-26577 2022.
Tero Karras, Miika Aittala, Thomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with bad version of itself. ar preprint arXiv:2406.0m 02500007,77 2024a.
Tero Karras, Milk Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6960 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Tuomas Kynkäännicmi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance a limited interval improves sample and distribution quality in diffusion models. arXiv preprin! arXiv:2404007724, 2024.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori Hashimoto. Diffusion-lm improves controllable text generation. Advances in neural information processing systems, 35:4328 -4448, 2022.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrencce Zitnick. Microsoft coco: Common objects in context. In Computer Vision Vision-ECCV ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Luping Liu, Yi Ren, Zhijie Lin, and Zhon Zhao. Pseudo numerical methods for diffusion models on manifolds. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.neg 2022. URL https://openrevieet http:/pparrriiieeeeeeerrrrrraaap mttt/toruuurrrid-ppddd22
lya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ar preprint arXiv: 03983, 2016.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NewrIPS, 2022. URL http://papers. nips cc/ wetprrffleepppppappeee http://aa///////aa// he0ae www.aaaaaaaaaaaaaeeeeeeeeeeeoeee -btpreeeeeecceeebaaaatattt----oe Abstract-Conferennce html.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2iadapterr: Learning adapters dig more controllable ability for text-lo-image diffusion models. In Proceedings of the AAAT conference artificial intelligence, volume 38, pp. 4296-4304 2024.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. PMLR, 2021. URL http://proceedingg mlr. wwtppp/ewe//nieeeeeee cho121a. html.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranay Shyam, Pamola Mishkin, Bob McGrew, Πlya Sutskever, and Mark Chen. GLIDE: towards photorealistie image generation and editing with text- guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16784 16804. html.
PMLR, 2022. URL pttp://///coccco....mmmm mmr. wwt.semw......s..eeeeeeeeeee prress/vi62nnich///
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Dustin Podell, Zion English, Kyle Laccy, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improvvin latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevier forum2id-d1522A8xet
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference Learning Representations, 2023. URL https: //openreview. http//orummid--Fyyyssee
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Cascy Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint Xrxi::2204.00 061122 1(2):3, 2022.
Robin Rombach, Andreas Blattmamn, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Negar Rostamzadch, Emily Denton, and Linda Petrini. Ethics and creativity in computer vision. CoRR, abs/2111.03111, 2021. URL https://arxiss http://aawwvv..g/aaa///aaa/ org//bs/////////aoe/a 03111.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Un- leashing the diversity of diffusion models through condition-annealed sampling. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevier: wwt
http:/ppweweieeeeeeeeeooo/odddda net/fforumiiddzMMorraj....
Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating oversaturation and artifacts of high guidance scales in diffusion models. In The Thirteenth International Conference on Learning Represenlations, 2025a. URL
weet/forumidd==2DNBBB
Seyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, and Romann M. Weber. No training, no problem: Rethinking classifier-free guidance for diffusion models. In The Thirleenth International Conference Learning Representations, 2025b. URL attps://openrevieer www.eeewweeeffrrreeeeee net//Eoruuiidee300cooo
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479 36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint ar arXiv:2202.00512 2022a..
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffussio models. In The Tenth International Conference on Learning Represenlations, TCLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1 11, 2024.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2025.
Jascha Sohl-Dickstein, Eric Weiss, Niru Mahoswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256-22662222 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar preprinl arXiv:2010. 02002, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, A Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXxv:201..1346, 2020b.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier fcatures let networks Iearn high frequency functions in low dimensional domains. Advances in neural information processing systems, 63:7537-75417 2020.
Xi Wang, Nicolas Dufour, Nefeli Andreou, Marie-Paule CANI, Victoria Fernandez Abrevaya, David Picard, and Vicky Kalogeiton. Analysis of classifier-free guidance weight schedulers. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreviex http://eee/oooommtttttttttt
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-lo-image diffusion models. arXw preprinl arXin:2308.06721, 2023.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
14

---

Published in Transactions on Machine Learning Research (10/2025)
A Broader impact
Our method accelerates guided sampling in diffusion models, broadening accessibility to large-scale text- lo-image or class-conditional generative syslems. This can reduce energy consumplion and compulational barriers to use Al-generated content for various creative applications. However, while advancements in Al-generated content have the potential to improve efficiency and stimulate creativity, it is essential to consider the associated ethical implications. For a more in-depth exploration of ethics and creativity in computer vision, we refer readers to Rostamzadeh al. (2021)
B Ablation studies
This section presents our ablation studies. Unless otherwise specified, all experiments are conducted using the DiT model for class-condine conditionn generation. We use FID as the primary metric to determine the adapter configuration used in the main experiments.
( Adapter architecture We first examine various design choices for the adapter architecture Oe (Z, 4. c). Let C= (c cc] represent the matrix containing all conditioning embeddings. The cross-attention and offset adapter architectures are formalized in Equations (8) and (9) respectively. We further experimented with a gating architecture defined as
g wt,,tt = (z(Zv) MLP(Z))
(10)
33 EC c.), vector independently. Lastly, also considcred positional encoding adapter architecture given by
where 2, - 23
a is the sigmoid function, and o : RT x Rxx + RTxd scales each d-dimensional
Iw (Z, wt,,c) = MLP(C),
(11)
where ēÿ = Øj. Ei Ci and Øj encodes the j-th attention time step. Specifically, ej is computed by a Fourier feature encoder (Tancik et al., 2020), followed by an MLP. The performance of these architectures using the DiT model are given in Table 6a. Note that for the DiT model, the offset architecture works the best. However, as shown Table 6b, the cross-attention adapter works better for text-to-image models such Stable Diffusion (Rombach et al., 2022). Hence, we used the offset architecture for class-conditional generation, and the cross-attention adapter for more complex text-to-iman to-image models. We also experimented with using dropout in the offset MLPs for further regularization but found that the model performs best without using any dropout (see Table 7).
Adapter initialization While adapters typically initialized with zero values such that C(0,v) = Co at initialization (Houlsby et al,, 2019), Table shows that Xavier initialization yields better results for guidance distillation. Therefore, we recommend avoiding zero initialization of the adapters for AGD.
Dimensionality of the adapter We now examine the impact of adapter dimensionality in Table 9, Our results show that increasing the hidden dimension initially improves FID but eventually leads to degradation, likely due to overfitting Therefore, we recommend designing adapters with fewer than 5% additional parameters w.r.t. the base model.
Training loss functions We also explored various loss functions for training AGD. Specifically, we experimented with yx, ||x yll and a weighted (x(x,)) = A(t)]xx - yllz. where A(t) is a weighting function depending on the time step. As shown in Table 10, the simple lg loss with A(t) = 1 performs best.
C
Details of the evaluation samples for qualitative comparison
DiT The samples used guidance scale of 4.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: Ablation study on adapter architectures across different baseline models. We observe that offset adapters perform best for class-conditional models, while cross-attention adapters are more effective for text-to-image generation.
DiT.
(b) SD2.1
Architecture
FID +
Prec. f
Recall t
Architecture
FID +
Prec. 1
Recall
Cross-attention
5.49
0.83
0.65
Cross-attention
21.09
0.66
0.55
Offset
5.03
0.80
0.68
Offset
22.05
0.63
0.54
Positional encoding
5.25
0.81
0.66
Gating
5.54
0.83
0.66
Table 7: Ablation on the dropout rate using DiT.
Table 8: Ablation on the initialization type of the adapter layers.
Dropout
FID
Prec. f
Recall f
Init. scheme
FID +
Prec. 1
Recall
0%
5.22
0.80
0.67
10%
5.27
0.82
0.67
Zero Xavier
5.24
0.81
0.68
20%
5.39
0.83
0.66
5.03
0.80
0.68
50%
5.69
0.85
0.63
Table 9: Ablation the hidden dimensionality of the adapters.
Table 10: Effect of using different loss functions for distillation.
Dim.
Params
FID +
Prec.
Recall t
Loss
Weight X(t)
FID
64
0.8%
3.33
0.81
0.67
9.91
128
2.5%
5.03
0.80
0.68
82
5.03
256
6.1%
5.22
0.80
0.67
Eg
(t))
5.30
512
17.2%
5.26
0.81
0.67
2 1
L(Eo, Ea)
6.64
SD2.1
The samples used guidance scale of 10. From left to right, the prompts used in Figure 5b were:
1. A cat on the flower."
2. "A close-up of blooming flower."
3. "A A quiet beach at sunset with gentle waves."
4. "A calm lake reflecting the blue sky."
SDXL The samples used guidance scale of 12. Further, the prompts used in Figure 5c were:
"A modern reinterpretation of a classical Renaissance painting, where futuristic elements and digital motifs merge with traditional portraiture."
" A fantastical scene of celestial garden floating in space, featuring luminous, otherworldly flora against backdrop of swirling galaxies."
"A hyper-realistic digital painting of a futuristic metropolis at sunset, with neon lights reflecting off rain-soaked streets and towering holograms."
4. "A cozy winter scene of remote mountain village, with softly glowing windows, snow-covered rooftops, and star-filled night sky."
16

---

Published in Transactions on Machine Learning Research (10/2025)
D Additional implementation details
Section 4.2 provides the main implementation details. The DiT-XL/2 model was trained with a batch size of 64 for 5000 gradient steps, the SD2.1 model with a batch size of 8 for 5000 gradient steps, and the SDXL model with a batch size of 1 for 20000 gradient steps. These settings were selected based on the maximum batch size that fits within 24,GB of VRAM. For all quantitative experiments, we set the guidance scale to the value that achieved the best FID for each method. The AGD implementation will be publicly released support further research guidance distillation.
The FID scores for class-conditional models were computed using 10k generated samples and the entire ImageNet training set. For text-to-image models, we used the full COCO-2017 validation set as the real data. All metrics were computed using the ADM evaluation code base (Dhariwal & Nichol, 2021) to ensure fairness across experiments.
E
Additional visual samples
Figures 10 to 21 provide additional visual samples comparing AGD and CFG across various models used in this work. Similar to our main findings, AGD samples consistently match or surpass CFG samples in both quality and diversity. The results are best seen when zoomed in.
CFG
AGD (Ours)
ertwoo J0cr
Figure 10: Uncurated samples using DLT-XL/2. Class label: "Space shuttle" (812); guidance scale: 2.
CFG
AGD (Ours)
Figure 11: Uncurated samples using DiT-XL/2. Class label: "Golden retriever" (207), guidance scale: 3,
17

---

Published in Transactions on Machine Learning Research (10/2025)
CFG
AGD (Ours)
Figure 12: Uncurated samples using DiT-XL/2. Class lable: "Macaw" (88), guidance scale: 3.
CFG
AGD (Ours)
Figure 13: Uncurated samples using DiT-XL/2. Class label: "Arctic fox" (279), guidance scale: 4.
CFG
AGD (Ours)
Figure 14: Uncurated samples using DIT-XT./2. Class lable: "Red panda" (387), guidance scale: 5
18

---

Published in Transactions on Machine Learning Research (10/2025)
CFG
AGD (Ours)
Figure 15: Uncurated samples using DIT-XL/2. Class label: "Dog sled" (537), guidance scale: 6.
CFG
AGD (Ours)
Figure 16:
Uncurated samples using SD2.1. Prompt: AA close up of a clear vase with flowers.", guidance
scale: 10.
CFG
AGD (Ours)
Figure 17: Uncurated samples using SD2.1. Prompt: "A set plush toy teddy bears sitting a sled.",
guidance scale: 10.
19

---

Published in Transactions on Machine Learning Research (10/2025)
CFG
AGD (Ours)
Figure 18: Uncurated samples using SD2.1. Prompt: "People flying kites in park on windy day.", guidance
scale: 10.
CFG
AGD (Ours)
Figure 19: Uncurated samples using SDXL. Prompt: "Two stuffed animals posed together in black and white."
guidance scale: 12.
CFG
AGD (Ours)
Figure 20: Uncurated samples using SDXL Prompt: "A capybara made of lego sitting in a realistic, natural
field.", guidance scale: 12.
20

---

Published in Transactions on Machine Learning Research (10/2025)
CFG
AGD (Ours)
Figure 21: Uncurated samples using SDXL Prompt: "A close-up of a fire spitting dragon, cinematic shot.",
guidance scale: 12.
21


Reviews:
[{"review_id": "PiK5P3egRX", "reviewer": "Reviewer_QNzb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper proposes a method based on distillation to substitute classifier-free guidance (CFG) in diffusion models. In particular, CFG requires at each diffusion step two calls to a denoising model, one to the conditioned model and one to the unconditioned one. Here, the model is trained to reproduce CFG denoising trajectories by simply adding a small model, called adapter, which purpose is to steer the base denoising model. The main differences with previous work (Meng et al.) are that (i) the small adapters add negligible memory and runtime overhead, and (ii) the adapters are trained on CFG-guided trajectories instead of unconditional diffusion trajectories, which offers a boost in the generation quality. The method is validated three base diffusion models, using FID as the main quantitative metric. \n\n\n**Strengths:**\n\n- The paper is well-written and easy to follow, even for people outside of the subfield. The method, albeit simple, is explained very clearly.\n- The presented results seem to be promising, as the proposed AGD method offers quality comparable to CFG while halving the calls to the denoising models. \n- Moreover, AGD offers better generation quality and lower memory requirements compared to guidance distillation (Meng et al.).\n\n**Weaknesses:**\n\n- For the larger diffusion models, ADG seems to produce slightly worse FID scores compared to CFG, which might limit the applicability of the method when high-quality images are required. This, however, is quite expected, and not a major issue. \n- In Table 1, it would be interesting to report inference times, as reporting solely the calls to the denoiser is not that informative. In Section 5.9 the times for CFG are not reported, which is a bit suspicious. \n- The code is not available to reviewers. \n\nUnfortunately, I remark that this paper is significantly outside my area of expertise, so I could not check the details of the paper thoroughly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method is validated on multiple models and several ablation studies are presented. Therefore, I find the experimental evidence quite convincing.", "audience_yn": "Yes", "audience_explanation": "I believe that a large portion of the diffusion models subfield will find this paper very interesting, including but not limited to TMLR's audience.", "requested_changes": "- Please provide inference times in Table 1. I find this quite critical for assessing the paper's contribution. \n- Also, please provide the code to reviewers (and make it public after acceptance!).", "broader_impact_concerns": "Section A in the Appendix is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mORp6pZKuy", "reviewer": "Reviewer_87Q3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the authors introduce the method of adapter guidance distillation (AGD) as a substitute of classifier-free guidance (CFG) methods for conditional diffusion models. While CFG remains a critical method for increasing the output quality of diffusion models, it is known to be very costly due to the inherent need to double the sampling cost as guidance. AGD is proposed as an alternative technique that simulates CFG while greatly reducing the additional cost of sampling, while maintaining the improved level of output quality.\n\nThe paper's main contribution is the aforementioned introduction of AGD, which utilizes 'adapters', lightweight modules that augments a frozen base model, to greatly reduce the cost of inference and simulate CFG with only a single pass without needing to fine-tune the entire, possibly complex model. Additionally, they identify a previously unrecognized observation that CFG and similar methods introduce a significantly differing training trajectory than classical diffusion, and thus proposes training on CFG-guided trajectories as opposed to classical ones. They claim that AGD can be deployed and trained on small-scale processing units such as common household ones and do not require much processing power.\n\nThe authors perform comprehensive experimental evaluations of AGD against traditional CFG as well as guidance distillation (GD) without adapters. They claim that the experimental results showcase that AGD matches and possibly improves on the quality of the generated output, while greatly improving the efficiency and cost of training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made in the submission are sound and supported by evidence to my best knowledge.\n\nThe authors claim that AGD produces outputs that are \"more visually appealing\" and \"often have better visual coherence\", which is not exactly simple to quantify. The other claims, including that AGD produces images of comparable quality and that AGD trains more efficiently, is supported by experimental evidence.\n\nI am not familiar with diffusion models and their theoretical foundations, and as such cannot claim for certain that the evidences and analyses provided are completely reasonable; however I am able to follow the intuitions behind the authors' ideas and tend to believe that their claims are sound.", "audience_yn": "Yes", "audience_explanation": "Diffusion models and their application in text/image generation is a vital and vigorous topic in machine learning and large language models. Accompanied with the interests are a variety of concerns and critiques of their ethical concerns and environmental impacts. This work addresses these interests and concerns, by proposing a novel technique that can help diffusion models deploy on smaller scale hardware that consumes significantly less energy and resources, both material and time-wise. As a result, I think the contributions of this paper would be of interest to TMLR's audience.", "requested_changes": "None. Unfortunately I do not believe myself informed enough about the topic beyond a general audience level to propose reasonable inputs.", "broader_impact_concerns": "Ethical concerns on large language models, generative artificial intelligence, and diffusion models mostly focus on (1) the energy consumption and environmental impact of training and deploying these models, and (2) the fair sourcing and usage of training data and output. The authors' work intrinsically addresses the first point. While I do not believe it precisely the authors' responsibility to address the second, I would appreciate some additional sources and citations that discuss this aspect in more depth; The reference provided is published in 2021, while the landscape of AI and diffusion models have altered significantly since then.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DGOPRd63cz", "reviewer": "Reviewer_j5zt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on a potential issue in the classifier-free guidance (CFG) ; that is, it requires two forward passes during the inference step. To solve this issue, this paper proposes the adapter guidance distillation (AGD), an adapter-based approach to mimic the behavior of the CFG procedure. Also, this paper proposes an approach to train this AGD and empirically verifies the performance of AGD.\n\nStrength:\n\nI believe the proposed approach has (at least partially) addressed the issue mentioned in the beginning of this paper. The AGD doesn't require to run two forward passes to follow the guidance. \n\nWeakness:\n\nHowever, the proposed method has several fundamental issues: (1) This paper doesn't consider the additional time cost for training the adapter. (2) I would also be concerned about the generalization ability of the adapter for those prompts that are not distilled.", "claims_and_evidence_yn": "No", "claims_explanation": "Here I further explain the two weaknesses I mentioned above, as the current results can only partially support the claims made in this paper. \n\n1. As I commented above, trainning the adapter requires additional time or training resources. These cost should also be considered when cmpared with the CFG method. As the claim made in this paper is that the AGD will save the computational overhead in the second paragraph (page 1), (from my perspective), it should be the entire computational overhead. So, the current result is not enough to support this claim. \n\n2. The original diffusion model typically was trained on a huge dataset containing millions of prompts, whereas distillation training typically relies on only hundreds or at most a few thousand captions. Then the evaluation is made over similar captions or prompts. This result is not sufficient to support the effectiveness of the proposed AGD method as it is hard to tell how the adapter will perform when it recieves a prompt that has not been used during the distillation training procedure. The adapter will perform significantly different from CFG as it has not been \"trained\" to be the same.  \n\nAs a result, I would suggest add some additional experiments to support these claims (see Requested Changes).", "audience_yn": "Yes", "audience_explanation": "I don't work on the diffusion model but I am sure that the method proposed in this paper would be interesting for those who are working in this field. Because in traditional methods (e.g. CFG), it requires two forward passes, this paper successfully reduces the number from 2 to 1. This result might be surprising for people in this field as it will violate the common intuition.", "requested_changes": "1. I suggest add the inference time cost of both CFD and AGD method (for example, CFD takes 2 seconds and AGD takes 1 second). Also, add the time cost of training the adapter (e.g. 100 seconds). Then the result tells that after generating 100 batches of images, using AGD will alway lead to positive gains.  \n\n2. I suggest evaluate the generalization ability in a larger dataset. For example, evaluate the DiT for the whole ImageNet using CFG. But train AGD on a few class of it. When the training data contains the whole ImageNet, I would expect to see the performance would be similar to the CFG. But when it only uses a few classes, the performance might be slightly worse. \n\nAlso, the experiment section is not very clear. There are multiple sizes of DiT models. Which one are you using?", "broader_impact_concerns": "I don't have any ethical concern on this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)