Gradient-Aligned Calibration for Post-Training Quantization of Diffusion Models
Authors: Dung Anh Hoang
Cuong Pham
Trung Le
Jianfei Cai
Thanh-Toan Do
Venue: ICLR 2026 Poster (2026)
Keywords: Quantization
Diffusion
Primary area: generative models
TL;DR: A paper that propose adaptive sample weight to address gradient conflict problem of diffusion quantization

Abstract: Diffusion models have shown remarkable performance in image synthesis by progressively estimating a smooth transition from a Gaussian distribution of noise to a real image. Unfortunately, their practical deployment is limited by slow inference speed, high memory usage, and the computational demands of the noise estimation process. Post-training quantization (PTQ) emerges as a promising solution to accelerate sampling and reduce the memory overhead of diffusion models.  Existing PTQ methods for diffusion models typically apply uniform weights to calibration samples across timesteps, which is sub-optimal since data at different timesteps may contribute differently to the diffusion process. Additionally, due to varying activation distributions and gradients across timesteps, a uniform quantization approach is sub-optimal. Each timestep requires a different gradient direction for optimal quantization, and treating them equally can lead to conflicting gradients that degrade performance. In this paper, we propose a novel PTQ method that addresses these challenges by assigning appropriate weights to calibration samples. 
    Specifically, our approach learns to assign optimal weights to calibration samples to align the quantized model’s gradients across timesteps, facilitating the quantization process.  Extensive experiments on CIFAR-10, LSUN-Bedrooms, and ImageNet datasets demonstrate the superiority of our method compared to other PTQ methods for diffusion models.

Published as a conference paper at ICLR 2026
GRADIENT-ALIGNED CALIBRATION FOR POST- TRAINING QUANTIZATION OF DIFFUSION MODELS
Hoang Anh Dung
Cuong Pham
Trung Le
Jianfei Cai
Thanh-Toan Do
Department of Data Science and AI, Monash University, Australia (hoang. dung, cuong. pham1,
trunglm, jianfei. cai, toan. do) @monash.
ABSTRACT
Diffusion models have shown remarkable performance in image synthesis by progressively estimating a smooth transition from a Gaussian distribution of noise real image. Unfortunately, their practical deployment is limited by slow inference speed, high memory usage, and the computational demands of the noise estimation process. Post-training quantization (PTQ) emerges promising solution to accelerate sampling and reduce the memory overhead diffusion models. Existing PTQ methods for diffusion models typically apply uniform weights to calibration samples across timesteps, which is sub-optimal since data at different timesteps may contribute differently to the diffusion process. Additionally, due to varying activation distributions and gradients across timesteps, uniform quantization approach sub-optimal. Each timestep requires a different gradient direction for optimal quantization, and treating them equally can lead to conflicting gradients that degrade performance. this paper, we propose novel PTQ method that addresses these challenges by assigning appropriate weights to calibration samples. Specifically, our approach learns to assign optimal weights to calibration samples align the quantized model's gradients across timesteps, facilitating the quantization process. Extensive experiments on CIFAR-10, LSSUN-Bedrood and ImageNet datasets demonstrate the superiority of method compared to other PTQ methods for diffusion models.
INTRODUCTION
In recent years, diffusion models have become prominent framework for high-quality image synthe- sis (Ho et al. 2020: Dhariwal Nichol 2021 Rombach et al. 2022).. Despite their effectiveness, the practical deployment of diffusion models is hindered by the substantial computational cost associated with the sampling procedure, which typically involves hundreds of iterative denoising steps. Fur- thermore, the noise estimation networks employed in these models often composed of complex architectures with a large number of parameters, making them resource-intensive and difficult to deploy devices with limited computational memory capacity.
Quantization has become a widely adopted approach for reducing the computational and memory cost of deep neural networks Dung et al. (2024); ; [in al. (2023): Jeon al. (2023): Pham al. (024).. More recently, this paradigm has been extended to diffusion models, where approximating network weights and activations using reduced--reeiisoon representations (Li al. (2023) ; He et al. (2023b); Huang et al. (2024); Shang et al. (2023): Wang et al. (2024a) enables substantial reductions in memory footprint and computational overhead, with only minor performance degradation. In particular, PTQ stands out as practical method for adapting diffusion models to low-resource settings, as allows for model compression without revisiting the training process or relying on the original dataset.
In the context of post-training quantization for diffusion models, calibration data plays a critical role in guiding the quantization process and is typically collected from various stages of the denoising tra- jectory. For instance, Q-Diffusion (Li et al. 2023) adopts fixed-interval selection strategy, collecting samples uniformly across the entire set of denoising steps. On the other hand, in PTQ4DM (Shang ct al. 2023),, a number of timesteps are sampled from a Gaussian distribution, and the images gen- erated these timesteps are then used calibration data. Building upon this, TFMQ-DM (Huang

---

Published as a conference paper at ICLR 2026
et al. 2024) follows the same sampling strategy Q-Diffusion and further introduces method designed to preserve temporal feature consistency during quantization. A common assumption in
existing quantization methods for diffusion models (Shang al., 2023 Li et al. 2023 Huang et al. 2024) is that all calibration samples contribute equally to the quantization process. However, recent research on diffusion models challenges this notion by demonstrating that sample importance varies significantly across timesteps. For example, (Xie et al., (024) shows that the loss gradient norms of samples are highly dependent on their associated timesteps, introducing systematic bias in influence estimation. Samples corresponding to timesteps with larger gradient norms tend to exert disproportionately higher impact on the model. Similarly, (Wang et al., 2024b) empirically categorize timesteps into acceleration, deceleration, and convergence phases based on process increments, each contributing differently to the model's learning dynamies.
On the other hand, since activations and gradients vary significantly across timesteps, calibration data from different timesteps can be interpreted as representing distinct tasks with divergent gradient dynamics. Prior works diffusion model training have highlighted the challenge of gradient conflict, which arises when optimization directions across timesteps interfere with each other. For example, (Hang et al. 2023) frames diffusion model training multi task problem, showing that optimizing the denoising objective at a specific noise level can degrade performance at others. Similarly, (Go al.) (202)) observes that negative transfer can occur due conflicting gradients across timesteps. In the context of quantization, this challenge becomes more pronounced due to the discrete nature of the parameter space, Quantized models with binary constraints lack the flexibility to represent intermediate values, forcing parameters to take discrete values such as 0 or 1. Unlike full-precision models, which can mitigate conflicting gradient signals by adjusting parameters incrementally, quantized models cannot resolve such conflicts effectively. As result, when gradients from different timesteps compete, the model may incur large losses in directions where no suitable quantized value exists, leading uneven performance across timesteps. Consequently, improving performance at one timestep may inherently degrade performance at others due to representational trade-offs.
To this end, we propose novel meta-learming-baseee -based approach that dynamically assigns importance weights to calibration samples during the quantization process. Our goal is to calibrate the quantized model using weighted sample set that not only achieves strong validation performance but also promotes alignment between gradients from different timesteps. We formulate this as a bi-level optimization problem, learning sample weights such that the calibrated model maintains gradient consistency and improves adaptability during the quantization process. By aligning gradient directions and emphasizing samples that contribute most effectively, and overall quantization quality. We validate our proposed approach on the widely used CIFAR- 10 (Krizhevsky Hinton 2009). LLUN- Bedrooms (Yu et al. 2015) and ImageNet (Deng et al. 2009) datasets with various noise estimation network architectures under different bit-width settings The extensive experiments demonstrate that our method outperforms the state-of of-the-art PTQ methods for diffusion models. The contributions of this work can be summarized as follows:
method enhances gradient propagation
We are the first to identify the issue of gradient conflict during post-training quantiza- tion of diffusion models, where calibration samples from different timesteps may induce inconsistent optimization directions.
We introduce the first PTQ framework for diffusion models that leverages gradient alignment to leam sample-wise importance weights for calibration data. By emphasizing samples with coherent gradient directions across timesteps, our method enhances quantization effectiveness.
Extensive experiments on CIFAR- 10, -eUd-eedo and ImageNet demonstrate that our approach consistently achieves superior FID scores compared to prior PTQ techniques for diffusion models.
RELATED WORKS
Post-training quantization on diffusion Models. Diffusion models (Ho et al. 2020 Song et al.
2021b) have emerged as a powerful generative framework, capable of producing high quality images through iterative refinement of noisy inputs. Despite their impressive results, the sheer number of inference steps required in the denoising trajectory poses a substantial bottleneck for real-world
deployment. While acceleration techniques (Lu et al. 2022 Song et al. 2021a Zhao et al. 2023)
2

---

Published as a conference paper at ICLR 2026
have been introduced reduce inference time, these methods often remain resource-intensive due to the size and complexity of the underlying noise estimation networks. To alleviate these overhead, model compression techniques, particularly model quantization (Li et al. 2023; He et al.) 2023b Wang et al. 2024a Huang et al. 2024, Shang et al., 2023; He et al. 2023a), offer a promising solution for diffusion models, by minimizing both computational and memory footprints. Among these, the post-training quantization (PTQ) et al. 2023 Huang et al., (2024) has gained much attention as practical approach that does not require full model retraining.
Data optimization for diffusion model quantization. PTQ methods for difffusion models typically rely on generated calibration data and effective quantization strategies. Recent efforts in this direction have primarily focused sampling strategies for calibration, aiming to select optimal calibration data for the quantization process. For instance, APQ-DM (Wang et al., 2024a) adopts a principled time- step selection strategy rooted in structural risk minimization to guide the generation of calibration inputs. On the other hand, PTQ4DM et al. 2023) demonstrates that calibrating quantized models with samples generated from the denoising process leads superior results compared to using samples from the forward process. Building on this idea, Q-Diffusion (Li et al. (023) samples intermediate results at fixed intervals and introduces a novel shortcut-aware quantization technique to improve the quantized model's 's performance across different benchmarks.
While existing PTQ methods typically assign equal weight to all calibration samples, this overlooks important characteristies of diffusion models. Prior studies (Nichol & Dhariwal 2021; Zhang et al., 2022) have shown that different timesteps contribute unequally to the generative process. For instance, later timesteps tend to capture higher-level semantic structures, while earlier ones focus more on denoising low-level details. Treating all timesteps uniformly may therefore dilute the influence of more impactful samples, leading suboptimal quantization.
Moreover, samples from different timesteps follow distinct distributions and can be seen as separate subtasks with divergent learning dynamics. Uniformly optimizing over the entire training set may induce conflicting gradient signals across timesteps. resulting in performance trade-offs where improvements in certain timesteps degrade others. To address these challenges, introduce meta-learning-basee baasd framework that learns sample-wise importance weights, promoting calibration samples that yield coherent and stable gradient directions across timesteps. This improves the overall quantization quality by guiding optimization in a more coherent direction.
3 PRELIMINARY ANALYSIS
To investigate the gradient dynamics induced by calibration samples from different timesteps during the quantization process, we analyze how gradients of the quantized model vary across timesteps. Specifically, we compute gradient vectors of the quantization loss evaluated on calibration samples drawn from different timesteps with respect to model parameters, using 256 samples per timestep for CIFARIO dataset. We then measure the pairwise cosine distance between these gradient vectors to construct gradient dissimilarity matrix. As shown in the heatmap in Figure La the cosine distance between gradients varies across timesteps. In particular, while gradients from earlier timesteps exhibit higher consistency, those from later stages of the denoising process tend to diverge more noticeably. This observation indicates that calibration data from different timesteps induce distinct gradient signals. Ignoring these variations during quantization may result in gradient misalignment, which can hinder effective optimization and reduce generalization across timesteps, causing the model to perform well to certain timesteps while underperforming on others.
Additionally, we visualize the loss of the quantized model after calibrated, using timestep-specific calibration subsets, each corresponding to specific timestep. The results, shown in Figure [b reveal significant variation in loss across timesteps, indicating that the quantized model struggles to generalize across the full diffusion process.
PROPOSED METHOD
4.1 PROBLEM DEFINITION
Before the quantization process starts, we construct the calibration set following the procedure
outlined in Q-Diffusion (Li al. 2023 by selecting generated samples at fixed intervals across the
3

---

Published as a conference paper at ICLR 2026
1.0
0.6
Denalsing Timestep
Averspo Loss
5.6
1:2
95
Denoising Timestep
Timestep
(a) Gradient dissimilarity across timesteps.
(b) Timestep-wise loss of the quantized model
Figure 1: Timestep-wise behavior in quantized diffusion models. (a) Gradient dissimilarity matrix constructed by computing pairwise cosine distance between gradient vectors of the quantization loss, with respect model parameters, for calibration samples drawn from different timesteps. Higher value indicates higher divergent between timesteps. (b) Quantization loss evaluated separately for calibration samples grouped by timestep, highlighting the uneven performance of the quantized model across different timesteps.
denoising timesteps. Each calibration sample the training set X = = x represented (x) E x(T), where xT() is generated sample at the timestep. A validation set (V) X is constructed by generating equal number of synthetic samples x, from each timestep f. In our method, each calibration sample (x. (T) ((")) is assigned a learnable weight wi, which reflects its influence on the quantized model's performance. The complete of weights for all
training samples is denoted by w [w:)|x 1x Our goal is to dynamically optimize training sample weights, such that the resulting quantized model 0g obtained after quantized using the weighted samples, achieves strong performance on the validation set. Given the full-pree pele-iiciioonn model 6pp and the initial quantized model lq, the optimization objective for is defined as follows:
arg min Lv AL. (F
(x
@CMSE(fo OFP, x())
s.t: (w) 0o
wi
00g
(I)
where J.| signifies the cardinality of a given set; (MSE(-) denotes the quantization loss (MSE loss) that matches the outputs of the full-precision model OFp and the quantized model 0q: Ly AL (.) denotes our optimization objective, to help the model achieves strong performance on the validation set. (w) denotes the final quantized model after calibrated over the training set; denotes the learning rate.
4.2 CALIBRATION DATA OPTIMIZATION
Beside model performance on the validation set, since all timesteps in diffusion model shares the same quantized model weight 0g. for consistent quantization process, we aim to align the gradients of 07 across different timestep-specific validation set promoting smoother optimization across subsequent quantization stages. In practice, we would divide timesteps into different groups, as adjacent timesteps often exhibit similar gradient behavior, as illustrated in Figure Ia] However, for simplicity, we assume each group consists single timestep and use T to denote both the number of timesteps and the number of groups throughout the algorithmic description.
4

---

Published as a conference paper at ICLR 2026
Our validation loss Lv (L) is defined as:
Lv (l 8pp, x( (as x + Luss Opp, x(O),
(2)
where the gradient matching loss LGM for gradients w.r.t the model weights is defined as:
Lom x(0) = T (T-
2
Σ
s.t: Gog DLMSE(OGg X(")
002
Regarding the loss LMSE in Eq. I..
The reconstruction loss LMSE. commonly employed in
prior quantization methods, is defined as follows:
LMSE(0Q.0Pm) X
OFp.
X(T)
[[((0Fp I(0Q, ( 2:)11,
(4)
X
where f(0rr, s:)) and f(0g, (4) respectively denote the outputs of the full-precision and quantized models given the input sample T;--
Directly optimizing Eq. ( is challenging due to the involvement of the third-order term in the gradient of (.M(-) with respect to the sample weights w. To address this, we propose a more efficient algorithm in Algorithm 2 and prove that the proxy objective optimized by this algorithm serves as a faithful surrogate for the original loss in Eq. (T). Theorem 4.1 formalizes this relationship, demonstrating that optimization via Algorithm induces the minimization the original objective in Eq. I..
Theorem 4.1. The optimization in Algorithm implicitly lead the minimization of the target objective (-Al(-) in Eq. 7
In order prove main result, present two lemmas that will be instrumental in the proof of the theorem.
Lemma 4.2. Let us denotes Gu.t (C
The second gradient matching loss
dw
CMM for gradients w.r. the sample weight is defined as:
2 (02, = = (T-11) Em.
2
(5)
T
The minimization of F (.) will implicitly lead to the minimization of Laas()), in the sense that minimizer of F corresponds to a minimizer of the target loss LGM-
We begin by leveraging Lemma 4.2 to show that minimizing the surrogate gradient matching loss C in Eq. 115 Jimplies the minimization of the original gradient matching loss LGM (.) in Eq. G0. Therefore, minimizing the surrogate validation objective (()) (.) in Eq. (8) leads to the minimization of the true validation loss Lv At.) in Eq. (2). establish Theorem 4.1 thus suffices 2
to show that Algorithm minimizes c(2) (.), which is stated by Lemma 4.3
Lemma 4.3. Let define a second validation loss:
Cl 0Fr, x(V) = (22) (o x(() + LMsE((f) x(V)..
(6)
(2) The Algorithm will minimize L (0), 0pp: in Eq.
Therefore, combining Lemma 4.3 and Lemma 4.2 we conclude that Theorem 4.1 holds. Please see
the Supplementary for the proof of our Lemmas and 4.2
Based on Theorem 4.1 we can optimize the sample weights in Eq. I implicitly using the Algorithm 2 in the Appendix.
5

---

Published as a conference paper at ICLR 2026
4.3 OVERALL OPTIMIZATION FRAMEWORK
At the beginning of training, a synthetic dataset is constructed by sampling from the full-precision diffusion model across multiple timesteps. This set is then divided into fimestep-balanced validation set and a training set. The training sample weight w are initialized uniformly as
exp(s:/T)
Wi
Σ, exp(s;/T)
where we initialize s; = 32
S |X denotes the temperature ayper-parameter.
During the training process, model calibration is performed in a block-wise fashion, with sample weights updated at each transition to a new block. A summary of the proposed model weight calibration procedure is provided in Algorithm
Algorithm Diffusion Model Quantization with Sample Weights
1: Input: Full- preciiioo model 0pp; number of layers L; number of timesteps T; the training set X(O;; the validation set (
2: Initialize sample weights w
3: Initialize quantized model 0q < 0FP
4: for L = 1 to L do
5: Use Algorithm to update w
6: Update quantized model le by calibrating layer l with training set ( and the updated sample weights w.
7: end for
8: Return: Quantized model Bq
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models and datasets. To assess the effectiveness of our approach, we conduct experiments on popular diffusion architectures. Specifically, we provide performance on DDPM (Ho al. 2020), for unconditional generation, and I.DM (Rombach et al.) 2022), which utilizes latent space and supports both unconditional and class-conditional generation tasks.
Our evaluation spans multiple standard datasets, including CIFAR-10 10 at a resolution of 32 x 32(Krizhevsky et al.] 2010)),
at 256 256 al. 2015), and ImageNet at
256 x 256 (Deng et al. 2009).
Implementation details. Our approach aligns with the current state-of-the- techniques in post- training quantization (PTQ) applied to diffusion models (Shang et al. 2023] Huang et al.) 2024), targeting both model weights and activations. In practical scenarios, post- trainirainin quantization (PTQ) for diffusion models typically addresses weight and activation quantization separate processes. For activations, TFMQ-DM (Huang et al. 2024) has shown that adopting sophisticated quantization techniques tends introduce high computational cost while yielding only limited performance gains. As a result, we employ the lightweight activation quantization method from TFMQ-DM, which estimates activation ranges using exponential moving average (EMA)(acob et al. 2018) mini-batches. On the other hand, we quantize the model weights using the AdaRound algorithm (Nagel al., 2220)) in conjunction with block-wise reconstruction (Li al. (021). to efficiently quantize the noise- estise-st network. We adopt AdaRound as it is the standard weight-quantization scheme used in prior diffusion PTQ methods (Huang et al. 2024 He et al. (003b), ensuring fair and consistent comparison. We begin by generating the calibration data through inference on the pretrained full-precision diffusion model, as outlined in Q-Diffusion (Li et al. 2023). To ensure consistency with prior post-training quantization efforts (Shang et al. 2023 Li et al. | 2023 Huang et al. 2024). we adopt the LAPQ method (Nahshan et al. 2022 to iitiiaiie the quantized model parameters 9g using weights from the original full-precision network. Additionally,
integrate
the temporal feature preservation strategy proposed in TFMQ-DM (Huang et al.) 2024) to better maintain generative quality. Regarding the validation set, small subset of the generated data

---

Published as a conference paper at ICLR 2026
the validation set. Since algorithm aligns gradients across timesteps and adjacent timesteps tend to exhibit similar behavior, we partition the validation data into 5 groups, each corresponding to consecutive range of timesteps. Each group is treated separate task in algorithm. We optimize the sample weights using the Adam optimizer with learning rate of 5 x 10-6 for 1500 iterations per update. For each quantization block, apply 20000 optimization iterations, in line with existing approaches (Shang et al., 2023] Li et al., 2023) Huang et al., 2024). Optimization of the sample weights carried out using the Adam optimizer (Kingma & Ba 2015), ) with fixed learning rate of 4 x 10-5, We employ the higher library to enable gradient based meta-optimization.
Evaluation metrics. To ensure alignment with established benchmarks (Shang et al. 2023 Li
et al. 2023) Huang et al. 2024). assess the generative quality of diffusion models using both the Fréchet Inception Distance (Heus et al.] 2017) and spatial Fréchet Inception Distance (sFID)(Saliman) al., 20T6). These metrics provide complementary insights into visual fidelity and spatial coherence. Specifically, FID evaluates the discrepancy between real and generated image distributions by comparing their high-level Inception features, while sFID focuses on mid-level features to better capture localized structural patterns in the images. Following common practice, we generate and evaluate 50,000 synthetic samples compute these scores, for a fair comparison with prior quantization studies.
Table Quantization results for unconditional image generation with DDIM on CIFAR- 10 32 32.
CIFAR-10 32
32
Methods
W/A
FID|
sFID
W/A
FID|
sFID]
PTQ4DM (Shang et al. 2023
5.14
O-Diffusion eL al. 2023
4.98
4.98
5.68
TFMQ-DM (Hang al. 2024
4/32
4.73
4/8
4.78
Ours
4.28
4.56
4.32
4.61
Table Quantization results for image generation with LDM-4 on LSUN-Bedrooms and ImageNet at resolution 256 x 256. We report FID, sFID, Precision, and Recall for each dataset.
LSUN-Bedrooms
ImageNet
Methods
Bits (W/A)
FIDJ
sFIDI Precisiont Recallf FID. sFID] Precisionf Recall
Full Prec.
32/32
2.98
7.09
10.91
7.67
PTQ4DM (Shang al.
2023
4/32
4.83
7,94
Q-Diffusion (Li et 2024
4/32
4.20
7.66
11.87
8.76
PTOD He al. 202
4/32
4.42
7.88
11.65
9.06
TFMQ-DM Huang
2024
4/32
3.60
7.61
65.92
44.88
10.50
7.98
92.91
30.24
Ours
4/32
3.14
7.22
66.11
45.50
10.17
7.40
93.02
30.97
PTQ4DM Shang et al.)
2023
4/8
20.72
$4,30
O-Diffusion (ii et al. 2023
4/8
6.40
17.93
10.68
14.85
PTOD (ee et at. 20236
4/8
5,94
15.16
10.40
12.63
TFMQ-DM (Huang al.
2024
4/8
3.68
7.65
65.89
44.99
10.29
7.35
92.53
30.98
Ours
4/8
3.26
7.40
66.05
45.20
9.96
7.55
92.75
30.71
5.2 COMPARISON WITH THE STATE-OF-THE-ART QUANTIZATION METHODS FOR DIFFUSION MODELS
To validate the effectiveness of our method, we benchmark it against the current state-of-the- thetth post- training quantization techniques developed for diffusion models, such as TFMQ-DM (Huang et al. 2024), PTQD (He et al. [0236)PTQ4DM (Shang et al. (023) and Q-Diffusion (Li et al.) 2023). For consistency and comparability, the baseline results are directly adopted from the TFMQ-DM study. Our evaluation spans a diverse set of datasets: CIFAR-10 (32 x 32). LSUN-Bedroomss (256 x 256)
for unconditional generation tasks, as well as ImageNet (256 256) for class-conditional image synthesis. All experiments adhere to the evaluation protocols established in prior work (Huang et al. 2024

---

Published as a conference paper at ICLR 2026
Unconditional image generation. To evaluate the effectiveness of method under extreme low-bit quantization, we conduct comprehensive experiments on both low-resolution and high- nigh-resolution generative models. Specifically, we test DDPM on CIFAR-10 32 32 resolution, and LDM-4 on two high-resolution datasets: LSSN--Bedrooss at 256 x 256. For consistency and fair comparison, employ the DDIM sampler (Song et al. 2021a) with 100 steps for CIFAR-10 and 200 steps for the two high-resolution datasets.
Across all three benchmarks, our method consistently outperforms existing state-of-the-art techniques, particularly in low-bit regimes. As reported in Table we achieve state-of-the FID scores in all quantization settings. Specifically, on CIFAR-10, our approach surpasses the compared method method (TFMQ-DM) with FID improvements of 0.45 and 0.46 in W4A32 and W4A8 settings, respectively. Similar improvements are observed on (W4A32) and 0.42 (W4A8). In summary, our approach demonstrates consistent and meaningful improvements over prior methods.
with FID gains of 0.46
Class-conditional image generation. We further evaluate our method on the high-resolution ImageNet dataset using LDM-4 and the DDIM sampler (Song et al.) 2022aa with 20 denoising steps. To ensure fair comparison, baseline results are adopted directly from the TFMQ-DM paper (Huang et al. 2024). As summarized in Table 2 our approach consistently surpasses existing methods across all quantization configurations.
In particular, the performance gain in the W4A32 setting is notable: our method reduces FID by 0.33 and sFID by 0.58 compared to TFMQ-DM, one of the current state-of-the-art PTQ methods for diffusion model. These results highlight our method's ability to preserve generation quality under aggressive quantization, even large-scale and visually complex datasets such as ImageNet.
5.3 ABLATION STUDIES
Ablation studies for the temperature parameter т. Please refer to the Appendix for more ablation studies of our method. We vary the value of from 0.2 to 2 and evaluate the model's performance on the CIFAR-10 dataset under the W4A32 setting, as shown in Table 3b As shown, the performance will degrade if we T smaller values.
Table 3: Ablation studies CIFAR-10 dataset.
(a) Effect of validation set size.
(b) Effect temperature parameter
2%
5%
10%
20%
0.2
0.5
2
FID.
4.55
4.32
4.59
4.75
FID1
4.85
4.55
4.28
4.32
sFID
4.71
4.61
4.38
4.51
sFID
4.74
4.67
4.56
4.61
Ablation studies for the number of timesteps. To evaluate the effectiveness of our method in the extreme case of very few timesteps, we test it on the ImageNet dataset (4/32 setting) using the DDIM sampler with 5, 10, and 20 timesteps. As shown in Table our method remains highly effective, outperforms the baseline TFMQ-DM under these challenging conditions.
Table Ablation studies for the number of inference timesteps. The results demonstrate that our method remains effecttve when the number of timesteps is small.
Methods
Timestep
FID |
sFID |
TFMQ-DM Huang et al.
2024
20
10.50
7.98
Ours
20
10.17
7.40
TFMQ-DM Huang et al.
2024
10
9.01
12.75
Ours
10
8.73
11.26
TFMQ-DM Huang et al
2024
5
19.10
38.69
Ours
5
18.22
35.05
8

---

Published as a conference paper at ICLR 2026
Grodlant Alg
08
06
04
02
0.
23
Figure 2: Visualization of the correlation between optimized sample weights and gradient alignments. All samples are sorted in descending order ot their sample weights, and divided uniformly into 50 groups. The blue line represents the average sample weight per group, while the red line indicates the average gradient alignment between samples in each group and the validation set. This demonstrates the positive correlation between gradient alignment and sample weight.
Ablation studies for the validation set size. In practice, we only use a small part of the training set as the validation set ( 5%), therefore the total number of images that our methods use are equal to that of the baseline method (TFMQ-DM), M.M...0 To assess sensitivity, we evaluate the performance using various validation sizes below Table 3a We observe that the method performs reliably across all sizes, with 5% achieving the best FID and competitive sFID. Notably, increasing the validation size beyond this point does not consistently improve performance, possibly due to increased sample diversity making it harder to optimize the reweighting under fixed calibration cost.
Visualization of sample weight. We visualize the distribution of sample weights alongside their corresponding average gradient similarity scores across groups in the validation set. For each training sample, the gradient similarity score is computed by assessing the alignment of its gradient with those of each task, and then averaging the similarity scores across tasks. As illustrated in Figure 2 our method assigns higher weights to samples exhibiting stronger gradient alignment, which facilitates more consistent optimization across the diffusion process by prioritizing samples that reduce gradient conflicts and improve the overall convergence across timesteps
The comparison of the computation cost and hardware efficiency. Although our method intro- duces additional computational overhead during training process. it remains competitively efficient. (256 x 256) under the W4A8 setting, TFMQ-DM (Huang et al., 2024) and Q-Difussion (Li et al. 2023) requires 2.32 and 5.29 GPU hours for training cost, respectively. In comparison, our approach takes around 3.5 GPU hours, representing moderate increase by hour over TFMQ-DM, but still more efficient than Q-Diffusion. Crucially, this modest increase in training cost yields consistently superior FID results across all evaluated settings, demonstrating an effective trade-off between performance and computational burden. It's also important to note that the added complexity is confined to the training stage. During inference, our method shares the same model structure and quantization format as TFMQ-DM, leading to identical hardware efficiency and latency at test time.
On
6 CONCLUSION
In this work, we address the overlooked issue of gradient conflict during post-training quantization (PTQ) of diffusion models, which arises from treating calibration samples across timesteps as equally important. We propose meta-weighting framework that dynamically learns sample-wise importance by promoting gradient alignment across timesteps. This approach enables more effective calibration under quantization constraints. Extensive experiments CIFAR-10. and ImageNet demonstrate consistent improvements over existing PTQ methods, highlighting the significance of timestep-aware sample weighting.

---

Published as a conference paper at ICLR 2026
REFERENCES
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
Hoang Anh Dung, Cuong Pham, Trung Le, Jianfei Cai, and Thanh-Toan Do. Sharpness-aware data generation for zero-shot quantization. In Proceedings of the 41st International Conference Machine Learning, (CML'24. JMLR.org, 2024.
Weilun Feng, Haotong Qin, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang, Renshuai Tao, Yongjun Xu, and Michele Magno. Mpq-dm: Mixed precision quantization for extremely low bit diffusion models. ArXiv, abs/2412. 11549, 2024. URL https://aaap
semanticscholin htt.:/oo......ooooooraaaa22722::
Hyojun Go, Jinyoung Kim, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Seungtaek Choi. Addressing negative transfer in diffusion models. ArXiv, abs/2306.00354, 2023.
URL httph:httppp..s.ott semant escholar org/ Io:p:/oor:pssSSia 258
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Bain- ing Guo. Efficient diffusion training via min-snr weighting strategy. 2023 IEEE/CVF In-
ternational Conference on Computer Vision (ICCV), pp. 7407-7417, 2023. URL https:
7/api. htapp..smmnntccccollrrrorr.o/./
Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization-
aware fine-tuning of low-bit diffusion models. ArXiv, abs/2310.03277, 2023a. URL https
7/api. s
Yefei He, Luping Liu, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. PTQD: Accurate post-training quantization for diffusion models. In NeurIPS, 2023b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In NeurIPS, 2020.
Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. TFMQ-DM: Temporal feature maintenance quantization for diffusion models. In CVPR, 2024.
Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-aribhneticonlly inference. In CVPR, 2018.
Yongkweon Jeon, Chungman Lee, and Ho-young Kim. Genie: Show me the data for quantization. In CVPR, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2010.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-Diffusion: Quantizing diffusion models. In ICCV, 2023.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. ICLR, 2021.
10

---

Published as a conference paper at ICLR 2026
Chen Lin, Bo Peng. Zheyang Li, Wenming Tan, Ye Ren, Jun Xiao, and Shiliang Pu. Bit-shrinking: Limiting instantancous sharpness for improving post- training quantization. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16196- 16205, 2023. doi: 10.1 (009/CPP557299.02..0155
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. Machine Learning, 1 110(11-12): 12652,262, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
Cuong Pham, Hoang Anh Dung, Cuong C. Nguyen, Trung Le, Dinh Q. Phung, Gustavo
Carneiro, and Thanh-Toan Do. Metaaug: Meta-data augmentation for post-training quantization.
ArXiv, abs/2407. 14726, 2024. URL httpps://aaaat semanticscholatt www.schooaaa.oo/oooosuss.e...ees org/CorpusID 271328964
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, In NeurlPS, 2016.
Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization diffusion models. CVPR, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In JCLR, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 202
Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang. Jie Zhou, and Jiwen Lu. Towards accurate post-training quantization for diffusion models. In CVPR, 2024a.
Kai Wang, Yukun Zhou, Mingjia Shi, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, and Yang You. A closer look time steps is worthy of triple speed-up for diffusion model training. ArXiv, abs/2405. 17403, 2024b. URL hhttps://aap semanti ar. org/ DDrpusID:2700630
Tong Xie, Haoyu Li, Andrew Bai, and Cho-Jui Hsieh. Data attribution for diffusion models: Timestep-induced bias in influence estimation. ArXiv, abs/2401.09033) .09031, 2024. URL https
//api. . sema nt www.ticcchhlaaa.....ccccccccccce wrt.aaarrrroooooo...D. ID: 2D:227227616
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction
large-scale image dataset using deep learning with humans in the loop. CoRR, abs/1506.03365. 2015.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.0552 055664 2022.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. UniPC: A unified predictor- corrector framework for fast sampling of diffusion models. In NeurlPS, 2023.
11

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.I THE USE OF LARGE LANGUAGE MODELS
We used a large language model (ChatGPT) to help with editing this paper. It was only used for simple tasks such fixing typos, rephrasing sentences for clarity, and improving word choice. All ideas, experiments, and analyses were done by the authors, and the use of LLMs does not affect the reproducibility of our work.
A.2 THEORETICAL PROOFS
Lemma A.1 (Restated from Lemma 4.3). Let us define second validation loss:
AV App, x()) ((2) (0 x + LusE((% 0pp, x()),.
(8)
The Algorithm will minimize (39) (0) 0pp: X(V) in Eq. 8).
the gradient w.r. w( when evaluated on the validation set at the t-th iteration, (1-1) denotes the samples weight after the (i-1)-tt iteration and before the i-th iteration optimized with Algorithm 1, (0)) denotes the initial sample weights. We have:
Proof of Lemma I.I. Let denote _ @CMee(92.0EP..X
Bail-1]
( w
(9)
Using First Order Taylor approximation around (0) and combine with Eq. 9 we have:
g Ri ( + HHw,(( w0)gg
(/. n/HLw
g.9 O(1),
(10)
where Hurt denotes the Hessian matrix of the model loss with respect to the sample weights w. evaluated on the validation data at the t-th iteration.
Replace ( gli) O()) Vi 1,2, .... we have:
G. G. - nHH
g
m/Hw
(c O(r)))
G. MH..
(g + O(7²)
(11)
For any i < j. because at each iteration and j we randomly sample validation data of a timestep. the iteration indexes are interchangeable, i.c. E EL , Therefore, have:
ag (6) T ((i) BiH... , (()) Hu..
dw
2B(
(g(()) g(t))
Ho.1
(g
dw
(12)
12

---

Published as a conference paper at ICLR 2026
Combine Eq. 11 and Eq.
12
we have:
G RJ 9
n/Hwit (
7 1-1
7
dw
==1
(13)
Combine Eq. 9 and Eq.
13
we have:
w
00)
0g(1)
(g
)),
w11
dw
(g(1) c
9 7
Σ
aw
ISiOjT
0L MLM.E(0G OFP, X
nI 1) OL
(0 X(())
dw
8w
(T( 1) dC GM 0L (0g OFP,
(0 x(
4
dw
(14)
According to Eq. 14). the update of Algorithm corresponds to a gradient descent step on composite loss function comprising Lass(()) and (.). This indicates that the optimization of Algorithm effectively minimizes the combined validation loss AL (0% 0pr, X(V) as defined in Equation
Lemma A.2 (Restated from Lemma 4.2). Let us denotes Gw,t (8 x(()) tu
The second
gradient matching loss c (.) for gradients w.r. the sample weight defined as:
C x(V)
Ew.esw.k
(15)
T 1)
The minimization of (.) will implicitly lead to the minimization of LGas in the sense that minimizer of ( corresponds to minimizer the target loss LGM-
Proof of Lemma 1.2. To demonstrate that minimizing c() (.) leads to the minimization of LaM (-), we show that under sufficiently small learning rate 17, the optimality C (-) implies the optimality of LaM(-) (-). Let (
yT denotes the set of gradient w.r.t the intial quantized model's parameters 02 when evaluated on timestep-specific training sets (x(") Similarly, we define {ggg IT 11=1
as the set of gradient w.r.t the meta model's parameters 6% when evaluated T timestep-speciff training subsets. We aim to prove that, under sufficiently small learning rate 17, for any 1 < i < j $ T,
cos(Gw.tr Gu.j) = 1,
then it follows that
cos Gog.s = 1.
13

---

Published as a conference paper at ICLR 2026
We begin by expressing the gradients using the chain rule:
Gw,t dLMSæ (0q, 0F,, )
dw
0LMsæ(0p OFP, x 7
202
00%
dw
@L.sæ(02. OFP, x
7 002
00%
dw
8(0 - 11 1
ac 0 ppp.a.
Wi
80g
G
Ow
Tf =-19og.
CMSk(Pg.0pp.r)
Cuswilg MEP.E (
26g
B0
30Q
(16)
Let us define matrix C' size T x N where Or, = cos(
denoting the
86
cosine similarity between the meta model's gradient evaluated the validation data timestep t and the original model's 's gradient when evaluated on jth training samples.
From Eq. (16), for any 1 iijjsT if cos(Sw,s; Gw.s) 1, then have:
Ct.1 C22
CLN > Cj.N
wk E (1....,,N) N) with CJ.b # 0
C₁ C₁.2
(17)
Supposed 1, 2, then cos Gw.2) and we assume that cos Gug.2) = < we will prove that this lead to contradiction. Using First-Order Taylor approximation around 0g we have:
Gog.1
G
(X and x has similar distribution)
Geg..1
1
((T)
%
(ggg + (02- 0q)
c
0
(
ML (0g Orr x: (T)
(?)
(9
wi
Hi).
(18)
(T)
80g
where Hi denotes the Hessian matrix w.r.t the model weight Og when evaluated on the training set ((T) for the timestep
90g.1
G0g.2
Let B
we have
G0g.1
Seg.21
:2
Gog.1
Gog.2 2
gog.1
Gug.2
||
+
=
(19)
Gog
Gog
Therefore, multiply both size of Eq.
(18) with have::
Goz
(x(
aLmsn(Og, OFF, (T))
Goz,1
B
(9
7
wi
00Q
(Hi)
G0g.1
x
(CCsE(OQ,,00 (3)
( y
TB+ +
Σ
H2)
wi
00Q
90g.1
(x(2))
aLMsr OFp, x
Hi TT
11
wi
d0g
0
(20)
14

---

Published as a conference paper at ICLR 2026
8C MSME ( 0 @Fp.a! ) 06g
Let us denote ( max (m))
Because E, = 1 and ||]]|
according
to Eq. 19
we then have:
Wi
0LMse(0g O (ao,,(0,,00r.000 2i (3)
d0g
-
9
77
|Hill2
X
wi
0CMSE (6q 0Fr, x
d0g
(T))
[B]
NOT)
- 2vy H11l2 1l
(21)
where || 1/11122 l2 denotes the spectral norm of Hi. Combining Eq. (1
and Eq. 20 we have:
(X(T)
DCMse(0a (00,000,,00000 0pp, x (T)
(
(g( 1-y
1 B
+
Wi
BBQ
H11²² DT B
(1 (1- 13
NTTT)
1-7 22n1
for rj
((T)
2/ ((TT)
(22)
Therefore, for sufficiently small learning 17, have (g0g.1 TT 0. This implies that there exist at least a single training sample z (T) such that
dL @CMsn(Oo, Opp.2/ (T))
B > 0
00Q
@L OFp, F6
(
Goz.1
Gez.2
(23)
BAQ
Gmg.1
G0z.2|
CL.E C2.6
Similarly, there exists training sample 2i
(T) such that C2, CT This leads to a contradiction,
because by assumption, C1.k C.. > 0 for all k such that C2,K # 0 and C2. + 0. If C2, = 0, C2,k C2.
then it must be that C₁ 0 as well; contradicting C1 C2 Therefore, if cos(Gw. Gu.j) = it follows that cos Goz, (.g.)) 1 as well.
A.3 VISUALIZATION GENERATED IMAGES
We visualize sample images generated from the full-precision model, well as from quantized models obtained using the O-Diffusion (Li et al. 2023) method, the TFMQ (Huang et al. 2024) method, and proposed method with the W4A32 setting, all initialized with fixed random seed. As shown in Figure 3 our proposed method generates images that closely match those of the pull-precision models, demonstrating the effectiveness of our approach.
15

---

Published as a conference paper at ICLR 2026
Full precision.
(b) Q-Diffusion (W4A32).
(c) TFMQ-DM (W4A32).
(d) Our proposed method (W4A32).
Figure 3: Generated samples from (a) full full-precision I.MM-, (b) Q-Diffusion (W4A32), (c) TFMQ- DM (W4A32), and (d) our proposed method (W4A32) on 256 256 dataset with fixed random seed.
16

---

Published as a conference paper at ICLR 2026
A.4 FINAL ALGORITHM
Algorithm 2 Sample Weights Optimization for Diffusion Quantization
1: Input: Full-preed precision model ppp; number of timesteps T; samples per timestep B; learning rate for model quantization y,. learning rate for sample weights yus total iterations /
2: Initialize quantized model 0Q OFP ViE (1,..,,
3: Initialize sample weights
4: ())))
5: for iteration i = to I do
{L...., T} 7: 86 = 0 - 11 Σ j=1 Wj Vos (fs 8FP, X(T)
6: Sample timestep t
II randomly choose a timestep Il one-step-ale ahead model
8:
for = =110 to B do
W j (1-1)
n v uy OFP, x! x(
II pseudo update w with X x-V)
9:
10: end for
if mod T = 0 then
12:
for = 1 to B do
13:
Wj - wj
(0)
+ nw
+ [wj -
Il final update of w
14:
end for
15: (0)
16: end if
17:
end for Return:
18:
A.5 ADDITIONAL RESULTS
Ablation studies for the number of timestep groups. In practice, uniformly divide timesteps into 5 groups for simplicity across all datasets. To investigate the effectiveness our method under different timestep groupings, evaluate the CIFAR-10 dataset using the W4A32 setting. Table presents ablation results for varying numbers of groups. We observe that increasing the number of groups does not significantly improve performance but may introduce additional computational overhead.
Table 5: Ablation studies for the number of timestep groups The results are on the CIFAR-10 10 dataset with the W4A32 setting.
Group FID J
2
5
10
20
4.63
4.57
4.28
4.25
4.26
sFID 1
4.61
4.63
4.56
4.57
4.56
Evaluation on extreme bit settings To further investigate the effectiveness of our method under extreme low-bit settings, we report additional results on the ImageNet 256 x 256 dataset using the LDM-DM model with 3/6 and 2/4 configurations. The results summarized in Table 6 As shown, our method achieves substantial improvements over baseline approaches, even under these highly constrained quantization regimes.
17

---

Published as a conference paper at ICLR 2026
Table Additional results low-bit settings for unconditional image generation with LDM-4 on ImageNet x56x256. Results are taken from (Feng et al. 2024)
Methods
Bits (W/A)
FID +
sFID +
Precision 1
Full Prec.
32/32
9.36
8.67
PTQD (He et al. TFMQ-DM (Huang et al. 2024
2023b
17.98
57.31
63.13
3/6
15.90
40.63
67.42
Ours
10.65
11.94
72.57
PTQD (He et al.
2023b
336.57
288.42
0.01
TFMQ-DN -MMQ (Huang et al.
2024
2/4
300.03
272.64
0.03
Ours
226.27
102.83
0.08
A.6 VISUALIZATION THE CHANGE LOSS ACROSS GROUP.
We visualize the difference in training loss between the quantized diffusion model trained with our sample-weighting strategy and one trained with uniform sample weights, using the CIFAR-10 dataset under the 4/32 quantization setting. The comparison is made across groups of timesteps sorted by ascending loss. shown in Figure 4 our method consistently reduces the loss in groups that tend to be under-optimized when uniform weights are used. This supports our motivation to assign sample weights that mitigate gradient conflicts, preventing the model from over-optimizing certain timesteps at the expense of others.
Per-Group Loss Change
1.2
1.0
Change in Loss
0.8
0.4
0.2
0.0
Group 1
Group 2
Group 3
Group 4
Group
Figure 4: Visualization of loss differences across timestep groups on the CIFAR-10 calibration set (4/32 setting) for quantized models trained with and without our sample-weighting method. Data are grouped by timesteps into five categories during sample weight optimization and are shown in ascending order of loss. Orange bars indicate loss reduction with method, while green bars indicate an increase. The results demonstrate that our approach effectively reduces loss for under- optimized timesteps, addressing the gradient conflict issue that leads the neglect of certain timestep
We visualize the difference in training loss between the quantized diffusion model trained with our sample-weighting strategy and one trained with uniform sample weights, using the CIFAR-10 dataset under the 4/32 quantization setting. The comparison is made across groups of timesteps sorted by
18

---

Published as a conference paper at ICLR 2026
ascending loss. As shown in Figure our method consistently reduces the loss in groups that tend to be under-optimized when uniform weights are used. This supports our motivation to assign sample weights that mitigate gradient conflicts, preventing the model from over-optimizing certain timesteps at the expense of others.
19


Reviews:
[{"review_id": "mjiZYlyOHn", "reviewer": "Reviewer_Qghd", "summary": "The paper proposes a novel post-training quantization (PTQ) method for diffusion models to improve inference speed and reduce memory consumption without retraining. Existing PTQ approaches treat all timesteps and calibration samples equally, which leads to suboptimal results because different timesteps contribute unequally and require distinct gradient directions during quantization. To address this, the authors introduce a timestep-aware weighting strategy that learns to assign optimal weights to calibration samples, aligning gradients across timesteps for better quantization. Experiments on CIFAR-10, LSUN-Bedrooms, and ImageNet demonstrate that this approach outperforms previous PTQ methods for diffusion models.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper identifies a key limitation in existing PTQ methods: uniform treatment of all timesteps. It then proposes a principled weighting mechanism that learns optimal calibration weights, effectively aligning gradients across timesteps.\n- The proposed method is evaluated on multiple benchmark datasets (CIFAR-10, LSUN-Bedrooms, and ImageNet), consistently outperforming prior PTQ approaches.", "weaknesses": "- As of 2025, most state-of-the-art diffusion models are built upon the DiT architecture. However, this submission does not include experiments on such models, which limits the generalizability and relevance of the findings to current diffusion frameworks.\n\n- The experimental evaluation is primarily conducted on small-scale datasets (e.g., CIFAR) with low-resolution images (e.g., 32×32). While these settings are useful for preliminary validation, they do not sufficiently demonstrate the scalability or robustness of the proposed method on more challenging benchmarks.\n\n- The study focuses mainly on bit-width configurations (e.g., W4A8, W4A32), but the results do not show clear improvements over existing baselines such as TFMQ-DM (2024). A more comprehensive comparison and discussion of potential advantages (e.g., efficiency, training stability, lower bit-width, or qualitative sample quality) would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALSVnCKHsv", "reviewer": "Reviewer_bPXm", "summary": "Many existing post-training quantization (PTQ) methods for diffusion models assume calibration data should be treated uniformly across timesteps. This paper shows that, during PTQ, the quantization-loss gradients do not align across timesteps; treating timesteps equally can therefore cause degrade performance. To address this, the authors assign a learnable weight to each calibration sample to reflect its contribution to the gradient update, and they optimize these weights as a proxy objective to align gradient directions across timesteps.", "questions": "- In Figure 2, is the x-axis ordered by sample timesteps? If so, it’s hard to strictly verify the claim that samples with stronger gradient alignment receive higher emphasis; a different visualization might make this clearer.\n- FID and sFID indicate improved fidelity. However, because the method reweights contributions per calibration sample, I’m concerned about possible effects on sample diversity. Table 6 reports Precision; could you also report Recall?\n- Similarly, for the main results, consider adding diversity-aware metrics (e.g., Precision/Recall curves) alongside FID/sFID.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Improvement of FID and sFID is verified by experiment. \n- Provides a theoretical justification for the proxy objective with approximation, and reports optimization trends consistent with the theory.", "weaknesses": "- Limited preliminaries on the specific techniques used (e.g., AdaRound) and related design choices.\n- Evaluation metrics lean heavily on FID, leaving diversity aspects less explored in the main tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SfJFzoLs4y", "reviewer": "Reviewer_DoWQ", "summary": "This work presents an interesting phenomenon that the quantization of diffusion models would raise grad conflict when training with different samples. With this in mind, a train-able weight is added to each sample for quantization training, to try to align the direction of grads when training quantization for different de-noising timesteps across different samples. Significant performance boosts are observed and detailed theoretical proof are offered to support the proposed motivation, as well as the methods to avoid such dis-alignment.", "questions": "1. Can this method also be extended to diffusion model's training? Have the authors tested whether diffusion's training from scratch / fine-tuning would encounter similar phenomenon? If so, I think this can be a great extension to the manuscript. (but I don't expect the authors to add this during the short rebuttal period, and this does not negatively influence on my evaluations.)\n2. While the authors ablate the results on the validation set size, I hope the authors to give some insights on the whole quantization dataset's size: when the whole dataset's size goes up, there might be more weights to learn, which might be a burden. I hope the authors can clarify on this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed motivation is very interesting and important. It starts from the numerical angle, looking into the grad dis-alignment when training quantization for different de-noising steps, which I believe opens up an important area to explore.\n2. The proposed method is well-designed. Both intuitions and theoretical proofs are provided, making it very clear to me.\n3. The proposed method achieves significant performance boost on the quantization task, and theoretical analysis proves that such improvements come from solving the proposed grad misalignment problem.", "weaknesses": "1. Minor issue: All citations are not in the correct format, which makes reading sometimes hard. My recommendation is: the authors should check them in the next version.\n2. The visualizations could be further refined to make it more impressive: While Fig. 1(a) presents the interesting grad conflict phenomenon very clearly, Fig. 2 looks not as straight-forward as Fig. 1. I recommend the authors to re-make Fig. 2 in the form of Fig. 1, to make it more impressive and more comparable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)