Modulated Diffusion:  Accelerating Generative Modeling with Modulated Quantization
Authors: Weizhi Gao
Zhichao Hou
Junqi Yin
Feiyi Wang
Linyu Peng
Xiaorui Liu
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models; Efficiency; Quantization; Caching Method
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Diffusion models have emerged as powerful generative models, but their high computational cost in iterative sampling remains a significant bottleneck. In this work, we present an in-depth and insightful study of state-of-the-art acceleration techniques for diffusion models, including caching and quantization, and reveal their limitations in computation error and generation quality. To break these limits, this work introduces Modulated Diffusion (MoDiff), an innovative, rigorous, and principled framework that accelerates generative modeling through modulated quantization and error compensation. MoDiff not only inherits the advantages of existing caching and quantization methods but also serves as a general framework to accelerate all diffusion models. The advantages of MoDiff are supported by solid theoretical insight and analysis. In addition, extensive experiments on CIFAR-10 and LSUN demonstrate that MoDiff significantly reduces activation quantization from 8 bits to 3 bits without performance degradation in post-training quantization (PTQ). Our code implementation is available at https://github.com/WeizhiGao/MoDiff.

Modulated Diffusion:
Accelerating Generative Modeling with Modulated Quantization
Weizhi Gao Zhichao Hou
Junqi Yin
Feiyi Wang? Linyu Peng Xiaorui Liu
Abstract
computational costs (Liu et al., 2022: Li et al., 2023). The iterative nature of the sampling process, which requires multiple inferences through neural networks, makes these models computationally expensive and time-intensive. This limitation restricts their scalability and accessibility.
Diffusion models have emerged as powerful gen- erative models, but their high computational cost in iterative sampling remains a significant bot- tleneck. In this work, we present an in-depth and insightful study of ofatc-of-the-art accelera- tion techniques for diffusion models, including caching and quantization, and reveal their limita- tions in computation error and generation quality. To break these limits, this work introduces Modu- lated Diffusion (MoDiff), an innovative, rigorous, and principled framework that accelerates gener- ative modeling through modulated quantization and error compensation. MoDiff not only inherits the advantages of existing caching and quantiza- tion methods but also serves general frame- work to accelerate all diffusion models. The ad- vantages of MoDiff are supported by solid theoret- ical insight and analysis. In addition, extensive ex- periments on CIFAR-10 10 and LSUN demonstrate that MoDiff significantly reduces activation quan- tization from 8 bits to 3 bits without performance degradation in post-training quantization (PTQ). Our code implementation is available at https: / /i t hub. /We zh Gao /MoDi ff.
Existing work aims to enhance the efficiency of the sam- pling process in diffusion models through several strategies. Caching methods, for example, accelerate diffusion models by reusing intermediate computations (Ma et al., 2024b; Wimbauer et al., 2024). These methods exploit significant similarities between features at nearby time steps, enabling the skipping of redundant computations by directly using cached results. Additionally, quantization techniques reduce inference costs by converting model weights and activations into integers using scaling factors (Nagel et al., 2021; Yang et al., 2019). Among these, post-training quantization (PTQ) stands out as a promising approach since estimates scaling factors a training-free free manner, making it broadly applica- ble to pre-trained models (Li et al., 2021). Another line of work focuses on efficient sampling strategies with solvers or samplers, such as denoising diffusion implicit models (DDIMs), which significantly reduce the number of sam- pling steps required in diffusion models, and speed up the process (Song et al., 2021a).
Our preliminary studies reveal that while caching and PTQ methods have achieved notable success in accelerating the sampling process, they also introduce significant limitations. First, our analysis reveals that caching methods can lead to reuse errors that accumulate throughout the generation process, particularly when reuse schedules are not carefully designed with respect to the time step and reused compo- nents. For instance, when following the reuse strategy of DeepCache (Ma et al., 2024b), but slightly modifying the reused components, we observe that the relative lg distance between the features of standard diffusion model and those of caching methods increases significantly throughout the generation process, reaching 40% in the final step, even when the cache is updated every three steps. On the other hand, our studies show that diffusion models exhibit sig- nificant outliers activations and variations activation ranges across time steps, leading substantial quantization errors under low-bit activation quantization.
1. Introduction
Diffusion models have emerged as powerful generative mod- els for producing high-quality data samples, ranging from images to audio and beyond (Ho et al., 2020; Song et al., 2021a;b). These models work by iteratively transforming simple noise distribution into complex, structured out- puts, guided by learned reverse diffusion process. Despite their effectiveness, diffusion models come with significant
'Department of Computer Science, North Carolina State Uni- versity NNational Center for Computational Science, Oak Ridge National Lab ' Department of Mechanical Engineering, Keio Uni- versity. Correspondence to: Xiaorui Liu <xliu96@ ncw.//6@ccu..u.....
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
In this work, we propose Modulated Diffusion (MoDiff), an innovative, rigorous, and principled framework that ac- celerates the diffusion sampling process while addressing the limitations of existing methods. Specifically, we propose modulated computation to significantly reduce activation quantization error by leveraging the computation redun- dancy across the diffusion time steps. Moreover, we further introduce a novel eтrоr compensation modulation address error accumulation. Furthermore, we provide theoretical analyses to explain why the temporal difference results in lower quantization error and how error compensation effec- tively eliminates accumulated errors. Our extensive experi- ments validate the effectiveness of this framework, demon- strating that MoDiff pushes the activation quantization limit of PTQ methods from 8 bits as low as 3 bits without any performance degradation, all within a training-free manner.
2. Related Work
Diffusion Models. Diffusion models have become a corner- stone of generative modeling, achieving remarkable success across diverse domains such as image synthesis, data distilla- tion, and molecular modeling (Ho et al., 2020; Hoogeboom et al., 2022; Su et al., 2024). These models operate on an iterative framework that involves adding noise in the for- ward process and learning to remove it during the reverse process (Dhariwal & Nichol, 2021). However, the iterative nature of the sampling process makes generation compu- tationally expensive (Song et al., 2021b; Ho al., 2020). To address this efficiency bottleneck, a line of research has focused improving the sampling process by optimizing the variance schedule or employing more advanced ODE solvers (Song et al., 202 Nichol & Dhariwal, 2021; Liu et al., 2022). For example, Denoising Diffusion Implicit Models (DDIMs) introduce a Markovinn formulation for the diffusion process, significantly reducing the number of sampling steps required (Song al., a))
The proposed MoDiff framework inherits the advantages of existing acceleration methods while addressing their limi- tations. It significantly generalizes the caching techniques through modulated computation but reduces upprpoximation and accumulated error. Additionally, from a quantization perspective, MoDiff significantly reduces the quantization error of existing PTQ methods, enabling the use of much lower activation bit- widht without sacrificing performance. Notably, MoDiff is agnostic to quantization algorithms and can be generally applied across different methods, mak- ing its contribution orthogonal to existing PTQ techniques. Furthermore, MoDiff imposes no constraints on samplers, ensuring compatibility with solver-based acceleration meth- ods. In summary, our main contributions are as follows:
Caching Methods. Caching methods for accelerating dif- fusion models aim to reduce redundant computations dur- ing the generative process by reusing intermediate results, thereby improving efficiency (Ma et al., 2024b; Wimbauer et al., 2024; Ma et al., 2024a). These strategies address the high computational cost by selectively storing intermedi- ate states from the reverse diffusion process for reuse in subsequent steps. For example, DeepCache reuses cached upsampled features every N time steps (Ma et al., 2024b). However, it can accumulate errors in the generation process with the reusing technique. Existing works rely on heuristic approaches to determine N, which limits its generalizabil- ity. Some methods also attempt to preserve model perfor- mance by fine-tuning diffusion models, but this approach can be computationally expensive (Wimbauer al., 2024; Ma et al., 2024a; Chen et al., 2024). In contrast to caching methods, our proposed MoDiff introduces a novel and prin- cipled framework to leverage the computation redundancy between sampling steps through modulated computing.
We present an in-depth and insightful preliminary study that reveals the limitations of existing acceleration tech- niques for diffusion models, such as caching and quan- tization methods, highlighting issues like error accu- mulation and high approximation error.
We propose MoDiff, a novel, rigorous, and principled framework that accelerates diffusion models through modulated quantization and error compensation. MoD- iff not only inherits the advantages of existing methods but also overcomes their limitations, enabling signifi- cantly more aggressive activation quantization.
Post-Training Quantization. Quantization aims to reduce inference costs by converting floating-pone numbers into low-bit integers (Nagel et al., 2021: Yang al., 2019) us- ing scaling factors. Post-training quantization (PTQ) has emerged as a powerful approach due to its training-free na- ture, making it suitable for pre-trained models (Li et al., 2021). Several studies have explored the application of PTQ techniques to diffusion models (Li et al., 2023; Wang et al., 2024; Huang et al., 2024; Shang et al., 2023; He et al., 2023b: Zhao et al., 2025). For example, Q-Diffusion intro- duces a time-step- nism tailored for diffusion models, achieving strong perfor- mance with 8-bit activations. However, a common issue is that existing methods struggle to quantize activations to
We provide theoretical analyses of quantization error and the error compensation mechanism in MoDiff, demonstrating that our approach can significantly re- duce the required activation bit precision in PTQ
Extensive experiments on CIFAR-10, 10, LSUN-CCurches, and
calibration data sampling mecha-
show that MoDiff' enables state- of-the-art quantization techniques to reduce activation precision from 8 bits to as low as 3 bits without any performance degradation in a training-free manner.
2

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
W/o reuse
5 stop
Activation
Temporal Difference
Relative 1, Error
0.8
3 stop
10 step
Activation Range
100
0.6
50
0.4
0.2
0.0
100
90
85
80
75
70
65
55
45 Time Step
40
35
30
25
20
10
0
Time Step
(a)
(h)
Figure 1. A preliminary study using DDIM on CIFAR-10 with 100 generation steps. (a) The relative & distance between the cached and standard diffusion features middle block, initialized from the same noise. As the reuse frequency increases, error accumulation hecomes significant. (b) The distribution of activations and their temporal differences across different diffusion time steps. The blue violin plots show that activation ranges fluctuate over time and exhibit outliers with long-tailed distributions. In contrast, the orange violin plots demonstrate more consistent ranges and concentrated distributions.
low bitwidths due to the presence of outlier values with large dynamic ranges (Xiao et al., 2023; Feng al., 2024). One related method is PTQD (He et al., 2023b), which post processes quantized models to reduce the quantization enror.
tribution step by step with a denoising network:
p= |xx) N (x; 1e f), o;
(2)
where (x t) is predicted by neural network, while or is typically set to 8t- With this parametrization, the sampling process can be expressed as:
We pose the detailed comparison to PTQD in Appendix E.
Another line of research focuses on quantization-awanl train- ing (QAT), which integrates the quantization process into the training phase, enabling model parameters adapt to quantization (Nagel al., 2022). These methods effectively address the challenges of low-bit quantization in diffusion models (Feng et al., 2024; He et al., 2023a). However, QAT approaches require costly retraining of diffusion models, which orthogonal to but not the focus of this work. In contrast, the proposed MoDiff framework can seam- lessly integrated state-of-the-an PTQ methods reduce activation bit-widths without additional training.
B. X = -I- X1 VI- - (x((xx)))
+ o,2, (3)
where ã₂ = -1 Be), and c0 () represents a U-Net that predicts the noise. However, since generating samples requires predicting noise across T steps, the diffusion pro- cess is computationally expensive for practical applications.
Existing approaches reuse historical computations to accel- erate the sampling process by exploiting the similarities between features at adjacent time steps in diffusion mod- els (Ma et al., 2024b; Wimbauer et al., 2024; Ma et al., 2024a). However, these caching methods directly reuse past information, which often cause approximation errors and deviate from the standard generation path of diffusion models. This discrepancy introduces errors each reuse step, which accumulate over multiple iterations. As result, these techniques require careful design of reuse schedules and even rely retraining to tune models, necessitating expensive hyperparameter search.
3. Preliminary Study
In this section, we introduce the fundamental concepts of diffusion models and quantization. Additionally, we ex- amine the challenges associated with existing caching and quantization methods with preliminary experiments.
3.1. Diffusion Models and Cache Reusing
Diffusion models consist of two processes: forward pro- cess and backward process, operating over I'' steps. Using Denoising Diffusion Probabilistic Models (DDPMs) example (Ho et al., 2020), the forward process incrementally adds noise to the image at each step, gradually transforming the data distribution into standard Gaussian distribution:
To illustrate the impact of reuse schedules, we conduct a preliminary study, where we apply caching to the residual connections of the U-Net in DDIM on CIFAR-10 10 without tuning following Ma et al. (2024b). Specifically, we reuse the activations from the previous time step for N - steps and update them at every N-th step. We compare the rela- tive &2 distance between standard diffuusion models and the variant with reused caching in middle block. As shown in Figure la, the relative eπтor increases significantly as the number of reuse steps and the time steps grows.
gxx x+-) = N(x; /1 11-xxx-1,,811
(1)
Mcanwhile, the reverse process progressively denoises the Gaussian distribution, reconstructing the original image dis-
3

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
T-1
A
1
A())
A4)
layer
f
1-th
C
Q
ar
Q
U-Net
T
U-Net
t=
(a) Standard Quantization
(b) Quantization w/ MoDift
Figure (a) Standard PTQ methods: The computations at different time steps are independent, with the raw activation a serving directly as the input to the quantizer. (b) Quantization with our MoDiff: For each linear operator, such as linear layers and convolutional layers, we cache the output from the previous time step, (0)) and input the temporal difference (1) a." into the quantizer. The final output obtained by aggregating the current computation results of A' with the eached output from the previous step o!
3.2. Post- post-Training Quantization
low-bit activation quantization, as shown in Figure 2. We introduce high-level motivation Section 4.1. To ease the understanding, first present equivalent reformulation of the diffusion process to reduce the quantization error in Section 4.2. Then, we propose novel crror-compensee modulation to address accumulated error across the diffu- sion steps in Section 4.3, Computation and memory costs are discussed Section 4.4, followed by theoretical error analyses of our MoDiff in Section 4.5.
Quantization an effective technique to reduce the infer- ence cost of deep learning models by utilizing low-precision integers (Nagel et al., 2021). Given x, we denote the integer representation Xint and the dequantization vector Q(x):
int clamp( X
+ z2; -1),
(4)
((x)= s(im
z),
(5)
where b represents the quantization bandwidth, and clamp()) enforces value cut-offs between two integer bounds. The parameters and z correspond to the scale factor and zero point, respectively. PTQ (Shang et al., 2023) dynamically estimates the parameters s and z or derives them using calibration datasets with a pre-trained model. Due to its simplicity and efficiency, PTQ is widely adopted.
4.1. High-Level Motivation
While the heuristic design of caching methods exhibit sig- nificant and accumulated computation errors, the motivation to leverage computation in previous time steps reduce computation in future time steps is still of great interest. In- spired by the similarity of activation patterns across adjacent time steps, we measure the temporal differences between (!)
The major challenge for PTQ methods in diffusion models is to use low-bit quantization. First, the activation tensor ranges vary significantly across different time steps, as illus- trated by the height of the blue violin plot in Figure 1b. This variation makes it difficult for a shared scaling parameter s to handle all ranges. Second, significant outlier values exist within the activations at each time step. In Figure 1b, the width of the violin plot represents the distribution of activa- tion values for a specific time step. These outliers make it challenging to select a scaling parameter s that minimizes both clipping error and rounding error simultaneously.
activation values over the diffusion process as a!" a where (0) represents the activation at time step / for the 1-th layer of the denoising network. The distribution of these differences is visualized in Figure 1b in orange color. Com- pared to the activations, their temporal differences exhibit a much smaller and more consistent range across time steps. Moreover, their distribution is more concentrated, effec- tively reducing the presence of outliers. These interesting observation and analyses suggest that strong motivation to leverage this temporal stability with quantized computing to alleviate the computation errors in existing approaches.
In a nutshell, both caching and PTQ methods face their own inherent challenges. highlighting the need for a more effective strategy that can incorporate historical information while also mitigating the effects of activation distributions.
4.2. Modulated Quantization
Notation. Let (-) denote the I-th linear operator in the denoising network, such as the linear and convolutional lay- ers, where E 2..,..... L}. We denote the input and output activations for (1) at time step as a and of = (a("), respectively. Note that we focus on accel-
4. Modulated Diffusion
In this section, we propose Modulated Diffusson (MoDiff), a novel framework to accelerate all diffusion models with
4

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
erating the computation of linear operators, such as linear and convolutional layers, since they are the most costly op- erations in neural networks and account for the majority of computation during data generation (Zhao et al., 2024).
4.3. Brror-Compensated Modulation
While the modulated computation and quantization in Eq. (7) can reduce the activation quantization errors, com- paring with the full full-precision computation in Eq. (6), the computation error o o, will be carried over across the diffusion time steps and cause large accumulated errors. In this section, we introduce a novel erro--comeenstted modu- lation to address the error accumulation, which leads to the complete MoDiff framework as follows:
Motivated by the insights from Section 4.1, we propose novel modulated computation reformulate the computa- tion of each 1-th linear layer the denoising network in diffusion models as follows:
(a
2 TT- = (a - (a)
(AT-1 - a + +oT
år (arr)
(8)
oσ = A(âr)
(9)
(6)
o = (a! = (a)) - - + +
âr -I = Q( âr) + år
(10)
or = A(âr-1) = A ( 1 - ar)) + or
(11)
of (al)) (a a (0)
(12)
à, Q(ai å(+1) + +a+11
(13)
where the output of (1) puted by incrementally refining the output 0 computed in previous time step ++1 with the modulated computation A (a - a. Specifically, the second equality in each equation holds because of the linearity of the operator A
in time step & can be equivalently com-
6, = A(ā,) = A A(((a: - -a,+1) + 0
(14)
(15)
å = Q(ai - -2) +
(16)
6 = A(ai) = A - - + +
(17)
A (a. (1) = A (1) (a. ( - A (a/f1) + A (2)2)
Specifically, we construct an intermediate variable å to store the activation that is actually computed through quan- tization, which keeps track of the quantization errors:
A
(1) a
(1) 0+1'
We further propose apply quantizer Q to approximate the temporal difference before quantized computation':
e; = -(r ae+1) 1) Q( ac+1)
(18)
= (a: âc+1) (å. ae+1) a -a.
ôr A(Q(ar))
A(ar)
where the second equation comes from Eq. (13). Since we do not have access to the accurate o, but only its approx- imation o,, the incremental refinement will be on top of 6,. Given that 6 = A ((x) is feature tranformation of å, the residual should be computed based on á instead of a., which will compensate the errors and avoid error accumu- lation. Note that as shown in Eq. (14), o, = A(āa)) only represents their relation, but the actual quantized computa- tion is the following: o, - A A(Q(a, - à(+1)) + ó A slight rewrite of this update clearly illustrates how quantiza- tion error is compensated across the diffusion time steps:
-ar) Ôr-1- = -AQQar-1- ar)) + ôr A(aT-1)
d: =A(Q(a) - alt1)) + + 2 A(ai)
6, = -A(Q(ai -az)) +02 = A(an)
Since the temporal difference a." and concentrated range as discussed in Section 4.1, its quan- tization will incur much smaller quantization errors. Remark 4. When the input range falls bellow a tolerable threshold due to significant computation redundancy, MoD- iff allows assigning 0-bit representation in the quantizer, which skips the computation. This behavior subsumes exist- ing heuristic caching strategies (Ma et al., 2024b) as special cases within a generalizable and principled framework, al- lowing more flcxible control over caching.
-a(a(") a44 has a much smaller
(1)
o = A((((a: a++1 c1t) + oc+1 A(cr+1),
where the previous time step misses the computation of A(C++1) but will be compensated in the next time step by adding e1+1 into the input activations.
Remark 4.2. The proposed MoDiff framework is agnostic to quantization methods and can be applied to existing PTQ methods. Therefore, it is orthogonal to the contributions of prior works in this area. Moreover, we argue that MoDiff is not limited to quantization. It can be extended to other techniques, such as sparse techniques (Han et al., 2015), further demonstrating its generality and versatility.
Note that the proposed modulated computation will be inde- pendently applied every costly linear neural layer, we omit the superscript index (I) in the rest of the paper for simplicity.
5

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
4.4. Computation and Memory Costs
The quantization error is bounded in terms of the quantiza- tion scaling factor which depends on the range of x and the bandwidth b. Specifically, we have:
Computation Cost. We categorize the computing opera- tions into three main types: matrix multiplication, matrix addition, and quanization//qquanizzatoo For matrix mul- tiplication, our method maintains integer-only operations, identical to standard quantization techniques. Additionally, by reducing quantization errors, our approach enables the use of a lower bandwidth for activations, potentially reduc- ing the computation cost. For matrix addition, our approach introduces two additional operations in a, â and 0 For Quantization and Dequantization, only dequantization on Q(a, - 1 is an additional step introduced by our approach. Modern quantization techniques (Nagel et al., 2021) indicate that matrix multiplication is the dominant computational cost during inference. Since our method in- troduces only a minimal number of additional additions and cuantizatio/ddequanizaatoo operations, their overhead is negligible in comparison to matrix multiplication. Conse- quently, our approach do not increase or even decrease the computation cost compared to existing PTQ methods.
- (() s (max(xx - min(x)) (23) (2 1)2
The proof is provided in Appendix A.I. Theorem 4.3 estab- lishes that quantization error is directly influenced by the input range and quantization bandwidth. Specifically, for smaller input range, lower-bit quantization can achieve the same error bound. Our preliminary results show that the residuals exhibit significantly reduced activation range, more than 10x smaller, which suggests that activation bit precision can be lowered by at least 3 bits while maintaining comparable quantization error.
To illustrate how error-compensated modulation eliminates error accumulation, we assume that the inputs are indepen- dent for simplicity. The following theorem demonstrates that it reduces error accumulation at exponential rate:
Memory Consumption. One limitation of our method is that it requires additional memory to store the intermediate variable a, and outputs or for each layer. However, as demonstrated in Section 5.3, this memory overhead remains negligible compared to the model size when using small batch sizes and low-bit activation quantization. Furthermore, we can locally select the layers that use MoDiff, allowing for a trade-off between performance and memory efficiency.
Theorem 4.4. Let A(-) be a linear operator and consider sequence of inputs ar, ar-1..... a1, with corresponding outputs or, or....... 01. Given a quantization operator Q we estimate the outputs using standard modulation:
o, A(Q(a: a(+1)) + +1+11
(24)
ör = A(ar),
(25)
where 1 T- 2, Similarly, we estimate the outputs using ernor-compensated modulation:
4.5. Theoretical Error Analysis
The proposed MoDiff framework enables quantization with low quantization error while mitigating error accumulation through error compensation. theoretical analyses to formally justify these advantages. The following theorem establishes the relationship between input magnitude and quantization error. For simplicity, we use dynamic quantizers, which determine the scaling pa- rameter based on the input values to avoid clipping errors, and we consider vector inputs instead of assuming a specific distribution for the input data.
6, A(Q(a; - -a++1) + ôc+1,
(26)
à = Q(ai - .a+1)) + a(+1.
(27)
this section, we provide
ôT A(ar), âr = aT:
(28)
where t T 1,...,,, 1. Suppose the quantization opera- tor Q satisfies the following error bound:
|xx - Q(x)12 S cl x
0 0<c<1.
(29)
Then, the estimation errors are bounded as follows:
Theorem 4.3 (Quantization Error). Lei Rdd be vector, and let the quantization bandwidth be b E N. Define the max-min dynamic quantizer as follows:
T-1
||0z - 5rl13
<<
2T
d|A1|1laa a 12,
(30)
k=1
8 max (x) min (x)
26_ 1
(19)
||0, 1
6ol13
<
Σ (1a)T*x-11A113 km
a 1 (31)
min
z =
(20)
x + z, 0, 2 1).
The proof is provided in Appendix A.2. Here, we assume (21) that the quantization error is bounded by the input magni- tude with a coefficient smaller than 1/2, which is a direct corollary of Theorem 4.3 with appropriate as shown in Ap- (22) pendix A.3. Theorem 4.4 provides two key insights. First,
Xint clamp(
The corresponding dequantization is given by:
Q(x) = =x z).
6

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table The IS, FID, sFID, and GBOPs for CIFAR-10 10 with DDIM under different precisions. The hest performance is bolded.
Methods
Bits (W/A)
GBops
1
FID
sFID 4
Bits (W/A)
GBops
FID 4
sFID 4
Full Prec. (Act)
8/32
1636
9.00
4.24
4.41
4/32
818
8.78
5.09
5.19
Q-Diff
9.48
3.75
4.49
9.12
4.93
5.03
O-Diff+MoDiff (Ours)
9.10
4.10
4.39
9.08
5.13
5.18
8/8
409
4/8
204
LCQ
9.01
4.21
4.41
8,80
4,96
4.94
LCQ+MoDiff (Ours)
9.10
4.10
4.39
9.08
4,95
4.95
Q-Diff'
8.76
29.16
13.81
28.60
15.09
D-DifffMoDiff (Ours)
9.38
4.19
4.32
8.85
5.62
4,93
8/6
307
4/6
153
LCQ
9.24
4.15
4.61
9.01
4,49
4.94
LCQ+MeDiff (Ours)
9.01
4.21
4.40
8.80
5,01
4.92
Q-Diff
2.19
332.75
100.37
2.47
325.76
92.84
O-Dif-MMoiif(Ouus)
9.71
13.41
11.25
9.60
13.62
11.94
8/4
205
4/4
T.CQ
10.01
24.09
13.07
9.72
22.50
12.95
LOQ+MeDiff (Ours)
9.08
4.31
4.38
8.82
5,10
4.94
Q-Diff
1.19
457,35
165.79
1.19
457.35
165.79
M-Diff+MoDiff (Ours)
5.19
90.34
41.26
7.34
47.35
13.87
8/3
153
4/3
777
LCQ
4.06
143.39
33.97
3.86
146.29
33.56
L.Q++Moiff (Ours)
9.02
4.14
4.38
8.79
4.98
4.95
Q-Diff
1.19
457.34
165.79
1.19
457.34
165.79
O-Diff+MoDiif (Ours)
266.68
75.88
1.36
387.75
168,38
8/2
102
4/2
51
LCQ
1.20
429.59
146.46
1.20
430.26
146.91
LCQ+M (Ours)
8,94
15.85
8.42
8,63
18.10
11.02
MoDiff without error compensation accumulates error more than linearly over the generation steps, making its perfor- mance highly dependent on the quantization parameter Second, eroo--compensation modulation in MoDiff ensures that errors from previous time steps are reduced exponen- tially, preventing error accumulation.
of 4 and 8, referred to LDM-4 (Bedrooms) and LDM-8 (Churches), respectively (Rombach et al., 2022). We use 500 sampling steps for LDM-4 and 200 steps for LDM-8. To demonstrate the generalization capability of MoDiff across datasets and architecture, we also conduct experiments on Stable Diffusion and Transformer-basee models (Peebles & Xie, 2023) on MS-COCO (Lin al., 2014) and Ima- geNet (Russakovsky al., 2015), respectively. Additional details and results are provided in Appendix C.I and C.2.
Finally, we note that Theorem 4.4 assumes independent in- puts. However, in diffusion models, is computed layer by layer using o,, which can further accumulate errors. As result, error compensation has greater meaning in the appli- cation of diffusion models compared to standard modulation, as counteracting error propagation more indispensable.
We assess generation quality using Inception Score (IS) (Salimans et al., 2016), Fréchet Inception Distance (FID) (Heusel al., 2017), and Sliced Fréchet Inception Distance (sFID) (Salimans et al., 2016) for CIFAR-10 10, and FID and sFID for the LSUN, IS is ric for datasets that significantly deviate from ImageNet categories. All metrics are computed based on 50,000 gener- ated images. Additionally, we provide precision and recall measurements (Sajjadi et al., 2018) in Appendix C.3.
5. Experiments
reliable met-
In this section, we first introduce the experimental setup. We then evaluate our method across different quantization precisions, demonstrating its ability to significantly reduce activation bit requirements compared to existing methods across multiple datasets. Additionally, we conduct com- prehensive ablation studies and present visualizations of generated images to assess the effectiveness of MoDiff.
Quantization Methods. We use dynamic channel-wise quantization and Q-Diffusion as the base quantization meth- ods and apply MoDiff to both (Dettmers al., 2022; Li et al., 2023). We also present results using dynamic tensor- wise quantization in Appendix D.2. Additionally, we include re- sults with full-precision activation (32 bits), for comparison. For weight quantization, we adopt the MSE reconstruction method, following the Q-Diffusion checkpoints. For ac- tivation quantization, dynamic channel-win quantization determines the scaling factor based on the channel-wise min-max range of the input. Due to its dynamic nature, we directly apply MoDiff to this method. In contrast, Q- Diffussion optimizes the scaling factor by minimizing the
5.1. Experiment Settings
Datasets, Models, and Evaluation. We majorly evaluate the effectiveness of our MoDiff the CIFAR-10 (32 32), LSUN-Bedrooms (256 x 256), and LSUN-Church-Outdon (256 x 256) datasets (Krizhevsky et al., 2009; Yu et al., 2015). For CIFAR-10, we use DDIM models with 100 de- noising steps (Song et al., 2021a). For the LSUN datasets, we use Latent Diffusion Models with downsampling factors

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table 2. The IS, FID, sFID, and GBOPs for LSUN-Church with
Table 3. The IS, FID, sFID, and GBOPs for -SU--UNdrooo-nn with LDM-4 under different precisions.
LDM-8 8 under different precisions,
Methods
Bits (W/A)
GBops
FID J
sFID 4
Methods
Bits (W/A)
GBops
FID L.
sFID
Full Prec. (Act)
8/32
5015
4.03
10.89
Full Prec.
8/32
25560
3.45
8.45
Q-Diff
4.24
10.57
LCQ
3.61
8.65
LCQ
Q-Diff+MoDiff (Ours)
8/8
1254
4.02
3,85
11.53
10.82
LCQ+MoDiff (Ours)
8/8
6390
3,57
8.44
L.QQMoDiff (Qurs)
3.99
10.06
LCQ
8/6
4609
64.17
63.18
L.Q++MoDiff (Ours)
3.57
6.53
Q-Diff
55.13
30.98
LCQ
M-Diff+MoDifl (Ours)
8/6
940
4.50
5.43
12.90
13.41
LCQ
L.Q+MMoDiff (Ours)
8/4
3195
372. 30
27.88
262.11
77.85
LCQ+MoDiff (Ours)
3,89
10.12
Q-Diff
355.85
187.56
to page limitations, we only present results for 8-bit weight quantization on the LSSUN datasets here. For results with 4-bit weight quantization, please refer to Appendix D.I. We highlight the best performance in bold. Based on these results, we draw the following conclusions.
O-Diff+MoDiff (Ours)
3.97
11.16
8/4
627
LCQ
198.37
161.03
LCQ+MoDiff (Ours)
34,02
10.59
Q-Diff
367.51
354.59
O-Diff+MoDiff (Ours)
5.40
13.81
LOQ
8/3
470
341.62
407.68
LCQ+MeDiff (Ours)
12.05
35.29
Generation Quality. Our method preserves high generation quality and significantly outperforms the base quantization approach when using lower activation precision across dif- ferent quantization methods and datasets. Specifically, with LCQ+MoDiff, activation precision in dynamic quantization can be reduced to bits for CIFAR- 10 without sacrificing generation quality. In contrast, the base quantization method experiences significant performance drop even at 6- bit ac- tivation precision. For example, the sFID score of Q-Diff on CIFAR- degrades from 4.49 to 13.81. Furthermore, even at 2-bit activation precision, our method maintains an sFID of 8.42 on CIFAR-10.
MSE reconstruction loss using calibration datasets across different time steps. To apply MoDiff to Q-Diffusion, we calibrate the activation quantizers by inputting the calibra- tion datasets into our MoDiff and learn the scaling factors with the residual. Additional implementation details can be found in Appendix B. We also perform few-step gener- ation experiments using MixDQ (Zhao et al., 2024) as the baseline, with results provided in Appendix D.4.
For quantization hyperparameters, we select weight bit widths from {) and activation bit widths from (2,3, 4, 6, 8). For notation simplicity, we use the format "W/A", where "W" represents the weight precision and "A" represents the activation precision. We refer to Q-Diffusion and dynamic channel-wise quantization as LCQ and Q- respectively. Our implementations based on them are de- noted as LCQ+MoDiff and O-Diff-MoDifr We denote full-precision activation models as Full Prec. (Act).
Generality. Our method is generalizable across different datasets and various quantization methods. For both Q-Diff and LCQ, our approach consistently improves their perfor- mance. However, we observe that quantizing activations to extremely low-bit precision becomes increasingly challeng- ing for higher-resolution datasets, even with our method. As shown in Tables 2 and the FID and sFID scores increase significantly at 3-bit precision for the Churches dataset and become notably high at 4-bit precision for the Bedrooms dataset. This challenge arises because LDMs are deeper and contain higher-dimensional hidden embeddings, making it more difficult to minimize quantization error.
Remark 5.1. The primary objective of this paper is to demon- strate the effectiveness of our method. We do not report the real accelcration metrics, such as running time. Following existing works (Li et al., 2023; Wang et al., 2024), we evalu- ate efficiency by measuring the number of binary operations (Bops) per denoising step for single image with the help of DeepSpeed (Song et al., 2023). Implementing acceleration on specialized hardware is beyond the scope of this work, but will be promising future direction, which is plausi- ble given the increasing hardware support for low- low-precision formats such as 4-bit integers (Dave et al., 2019).
Efficiency. As shown in Tables and 3. our method con- sistently reduces binary operations (Bops) for generation. For full-precision activation model, one denoising step in CIFAR- 10 requires 1636 GBops. In contrast, LCQ+MoDDin completes inference with only 154 GBops without any per- formance degradation, achieving over 10x computational savings. However, the generation quality (sFID) begins to degrade at 8/6-bit precision, which requires 307 GBops.
5.2. Main Results on CIFARIO and LSUN
We conducted experiments to generate images using quan- tized diffusion models and measure their quality. The IS, FID, and sFID scores for CIFAR-10, Churches, and Bed- rooms are presented in Tables 1, 2, and 3, respectively. Due
5.3. Ablation Study
In this section, we conduct ablation studies to analyze the effectiveness of MoDiff. First, we evaluate the impact of
8

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table FID on CIFAR-10 using the DDIM sampler in the ablation study of error compensation. "EC" denotes error compensation and the best performance bolded.
Table 5. FID and sFID L.UU--eedrooss using the DDPM sam- pler under different quantization precisions with LCQ. The best performance is bolded.
MCQ+MoDiff w/o EC
Methods
Bits (W/A)
FID L
sFID 4
Bits (W/A)
LCQ
w/ EC
LCQ (DDPM)
3.61
8.65
8/8
8/8
4.61
4.41
4,40
LCQ+MoDiff (DDPM)
3.39
8.02
8/6
13.07
10.21
4.38
LCQ (DDPM)
50.17
52.18
8/4
33.97
25.42
4.38
LCQ+MoDiff (DDPM)
8/6
12.60
13.71
I.CQ (DDDPM)
102.16
104.18
8/4
eroor-compensation modulation. Next, we demonstrate the compatibility of MoDiff with different samplers. Addition- ally, we examine how our method balances the trade-off between memory and computational efficiency. Finally, we present visualization results further illustrate the effec- tiveness of MoDiff in Appendix F. To further demonstrate the generalization of MoDiff, we also include results on few- step diffusion models, as presented in the Appendix D.4.
LCQ+MoDill (DDPM)
34.25
30.12
indicate that MoDiff enhances the generation quality of DDPM, particularly at lower activation bit widths. However, the improvement is less pronounced compared to DDIM. This is because the DDPM sampler introduces random noise at each step, increasing the difference between adjacent fea- tures. Consequently, this leads to larger residual magnitudes, which in turn amplify quantization errors. We provide ad- ditional experiments on PLMS (Liu et al., 2022) and DPM solver (Lu et al., 2022) in Appendix D.3.
3-bit
3-bit EC
0.6
4-bit
4-bit EC
Relative ba Error
6-bit
6-bit w/ EC
Memory Consumption. As discussed in Section 4, our method reduces quantization error at the cost of slightly increased memory usage. Table 6, we demonstrate that MoDiff significantly reduces Bops at lower bit precision while maintaining manageable memory overhead. The re- sults show that the memory overhead is minimal-no more than 4 MB, For more details, we refer to Appendix D.6.
0.4
0.2
0.0
100
80
60
40
20
0
Time Step
Table 6. The relationship between BOPs and memory usage of method using DDIM on CIFAR- 10 for generating a single image.
Figure 3. The relative &a distance between the features in the stan- dard diffusion model and the quantized model in middle block. "w/ EC" denotes the use of the compensatnnnn technique.
Measurement
W8A2
W8A4
W8A8
W8A32
sFID)
8.42
4.38
4.39
4.41
GBops
102
205
409
1636
Effects of error compensation We demonstrate how the error compensation technique mitigates error accumulation by comparing both generation quality and the quantization error, using DDIM on CIFAR-10 with LCQ. FIDs are shown in Table 4. To quantify quantization error, we compute the relative lg distance between the features of middle block in the standard diffusion model and the quantized model, both initialized from the same noise. As shown in Figure 3, the dashed line represents the relative error without error compensation, while the solid line represents the error with compensation. Without error compensation, error accumula- tion becomes significant below 6-bit precision. In contrast, with error compensation, the error remains minimal, even at 3-bit precision. In summary, our technique effectively avoids error accumulation in modulated computing.
Memory (Mb)
35.28
36.4
38.89
35.09
6. Conclusion
In this paper, we propose MoDiff, a principled framework for accelerating generative modeling. Our preliminary stud- ies reveal the challenges in caching and PTQ methods. To address these, we introduce modulated quantization and error compensation, which reduce quantization eгror and mitigate error accumulation. We provide theoretical anal- yses demonstrating the effectiveness of our approach. Ex- perimental results show that MoDiff significantly enhances activation quantization, enabling PTQ methods to operate at bit-widths as low bits without performance degradation. One limitation is that MoDiff reduces computation at the cost of increased memory usage. Additionally, we evaluate acceleration based on theoretical computational complexity rather than real-world hardware speedup. We leave hard- ware implementation and further memory optimizations of our MoDiff for future work.
Different Samplers We demonstrate that MoDiff is com- patible with different samplers in diffusion models. While DDIM is used as the sampler in our main experiments, we also evaluate our method with DDPM on LCQ using the dataset, as shown in Table 5. Our results

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Acknowledgments
He, Y., Liu, L.,, Liu, J., Wu, W., Zhou, H., and Zhuang, B, PTOD: Accurate post-training quantization for diffusion models. Advances in Neural Information Processing Systems, 2023b.
Weizhi Gao, Zhichao Hou, and Dr. Xiaorui Liu are sup- ported by the National Science Foundation (NSF) Award un- der grant number 11S-2443182, NSF National AI Research Resource Pilot Award, NCSU Data Science Academy Seed Grant Award, and NCSU Faculty Research and Professional Development Award.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Impact Statement
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Ethical Considerations. This work introduces modulated diffusion models accelerate the generation process of diffusion models. We do not foresee any significant ethical concerns associated with this approach.
Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867 8887. PMLR, 2022.
Societal Impact. Enhancing the efficiency of diffusion models can facilitate the broader adoption of generative AI, benefiting both Al research and hardware development. This advancement has the potential contribute to more efficient and accessible Al-driven solutions.
Huang, Y. Gong, R., Liu, J., Chen, T., and Liu, X. TFMQ- DM: Temporal feature maintenance quantization for dif- fusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7362-7371, 2024.
References
Chen, P., Shen, M., Ye, P., Cao, J., Tu, C., Bouganis, C.-S., Zhao, Y., and Chen, T. A-DiT: A training-five acceler- ation method tailored for diffusion transformers. arXiv preprint arXiv:2406.01125. 2024.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Li, X., Liu, Y., Lian, L.,, Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-Diffusion: Quantizing dif- fusion models. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 17535-17545, 2023.
Dave, S., Hao, w. Milind, K., and Niall, E. Int4 preci- sion for ai inference. Technical report, NVIDIA, 2019. https: /ttp:::sss//eeellpperre. www.dooeeer.iiiiiiaae.ee. wwwiiiiiiia...mm cww/boom//oloooooooooooooooooooe citoor--aiiifffeenc..
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8() 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-30333 2022.
Li, Y., Gong, R., Tan, X., Yang. Y., Hu, P., Zhang, Q., Yu, F. Wang, W., and Gu, S. BRECQ: Pushing the limit of post-training quantization by block reconstruction. Inter- national Conference on Learning Representations, 2021.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, $4:8780-8794, 2021.
Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. In European Con- ference on Computer Vision, pp. 740-755, 2014.
Feng, W., Qin, H., Yang, C., An, Z., Huang, L., Diao, B., Wang, F., Tao, R., Xu, Y., and Magno, M. MPQ-DM: Mixed precision quantization for extremely low bit diffu- sion models. arXiv preprint arXiv:2412. 1549, 2024.
Liu, L., Ren, Y., Lin, Z., and Zhao, Z. Pseudo numerical methods for diffusion models on manifolds. International Conference on Learning Representations, 2022.
Han, S., Mao, H., and Dally, W. Deep Compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv: /5/.000//99 2015.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. DPM-Solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
He, Y.. Liu, J., Wu, W., Zhou, H., and Zhuang, B. EF- ficientDM: Efficient iuantization-aware fine-tuning of low-bit diffusion models. International Conference on Learning Representations, 2023a.
Ma, X., Fang, G., Bi Mi, M., and Wang, X. Learning- to-cache: Accelerating diffusion transformer via layer caching. Advances in Neural Information Processing Systems, 37:133282-133300 2024a
10

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Ma, X., Fang, G., and Wang, X. Deepcache: Acceler- ating diffusion models for free. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15762-15772, 2024b.
DeepSpeed4Science Initiative: Enabling large-scale sci- entific discovery through sophisticated ai system tech- nologies. NeurIPS 2023 for Science Workshop, 2023.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole. B. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021b.
Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., Van Baalen, M., and Blankevoort, T. A white pa- per on neural network quantization. arXiv preprint arXiv:2106.08200 2021.
Su, D., Hou, J., Gao, W., Tian, Y., and Tang, B, D M: Dataset distillation via disentangled diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5809-5818, 2024.
Nagel, M., Fournarakis, M.. Bondarenko, Y., and Blankevoort, T. Overcoming oscillations in quantization- aware training. In International Conference Machine Learning, pp. 16318-16333 16330. PMLR, 2022.
Wang, C., Wang, 7., Xu, X., Tang, Y., Zhou, J., and Lu, J. Towards accurate post-training quantization for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16026- 16035, 2024.
Nichol, A. Q. and Dhariwal, Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-81111111 8171. PMLR, 2021.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Wimbauer, F., Wu, B., Schoenfeld, E., Dai, X.. Hou, J., He, Z.., Sanakoyeu, A., Zhang, P., Tsai, S., Kohler, J., al. Cache Me If You Can: Accelerating diffusion models through block caching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6211-62222 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffuuson models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp- 10684-10695, 2022.
Wu, J., Wang. H., Shang, Y., Shah, M., and Yan, Y. PTQ4DiT: Post-training quantization for diffusion trans- formers. arXiv preprint arXiv:2405. 16005, 2024.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg. A. C., and Fei-Fei, L.. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211-555) 2015. doi: 10.1007/
Xiao, G., Lin, J.. Seznee, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post- training quantization for large language models. In In- ternational Conference on Machine Learning, pp. 38087- 38099. PMLR, 2023.
s1 11126-015--816-..
Yang, J., Shen, X., Xing, J., Tian, X., Li, H., Deng, B., Huang, J., and Hua, X.s. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7308-7316, 2019.
Sajjadi, M. S., Bachem, O., Lucic, M., Bousquet, O., and Gelly, S. Assessing generative models via precision and recall. Advances in Neural Information Processing Sys- tems, 31, 2018.
Yu, F,, Zhang, Y., Song, S., Seff, A., and Xiao, J. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506 03365, 2015.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
Zhao, T., Ning, X., Fang, T., Liu, E., Huang, G., Lin, Z. Yan, S., Dai, G., and Wang, Y. MixDQ: Memory- efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. In Eu- ropean Conference on Computer Vision, pp. 285-302, 2024.
Shang, Y., Yuan, Z. Xie, B., Wu, B., and Yan, Y. Post- training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1972-1981, 2023.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. International Conference on Learning Representations, 2021a
Zhao, T., Fang. T., Huang, H., Liu, E.,, Wan, R., Soedarmadji, W., Li, S., Lin, Z., Dai, G., Yan, S., et al. ViDiT-Q: Elli- cient and accurate quantization of diffusion transformers for image and video generation. International Conference on Learning Representations, 2025.
Song. S. L., Kruft, B., Zhang, M., Li, C., Chen, S., Zhang, C., Tanaka, M., Wu, X., Rasley, J., Awan, A. A., et al.

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
A. Proof
In this section, we provide proofs for the theorems presented the main paper.
A.1. Proof of Theorem 4.3
Theorem A.1 (Restated, 4.3). Let x E Rdd be a vector, and let the quantization bandwidth be b E N. Define the max-min dynamic quantizer as follows:
max(x) min(x)
(32)
min(x)
2
(33)
Xint clamp(
x
+ 2, -1).
(34)
The corresponding dequantization given by:
Q(x) = s(Xiji z).
(35)
The quantization error is bounded in terms of the quantization scaling factor s, which depends on the range of x and the bandwidth b. Specifically, we have:
(max(x) min (xin(x)))
||x Q(x)) s2d
(36)
(2 1)²
Proof: For any i E (1,2....., ..,.. the value x; satisfies the following property:
0 < (x) min(x)
max(xx) xx
min(x)
max(x) min(x)
+
+
(37)
From condition (32), we have:
0 S Tint S 2 1.
This ensures that the clipping error is zero, meaning only need to consider the rounding error. Thus, we obtain:
- Q(x); =
T
(38)
T;
S
(39)
Therefore, applying this to the lg- fg-m error bound, we derive:
||x
Q(x)
Σ
(2. - Q(x),)²
(40)
im
2
(41)
2d.
(42)
12

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
A.2. Proof of Theorem 4.4
Theorem A.2 (Restated, 4.4). Let A(-) be linear operator and consider a sequence of inputs ar ar-1.... as with corresponding outputs or, -y...... O1. Given quantization operator Q, estimate the outputs using standard modulation:
o = A(Q(a: ac+1)) + +0+1,
(43)
ÖT = A(ar),
(44)
where ,,...,22 2, 1. Similarly, we estimate the outputs using error-compensatedd modulation:
6, A(Q(aa å(+1)) + 01+1,
(45)
å Q(a å.+1) + ar+1:
(46)
ôr = A(ar), år =ar.
(47)
where t = T -1,... ,,...,2. 2, 1. Suppose the quantization operator Q satisfies the following error bound:
||x 2 0 0<c< 2
(48)
Then, the estimation ervors are bounded as follows:
||o, - S
ax+112. (20) || - a
(49)
T
|1o - 6,
S
(50)
A=
Proof: Denote the error for standard modulation in Equation (43) é and for error-compensation modulation in Equation (44) as é, at time step t. We first compute the error for standard modulation:
è
|lo - 0.112 -
(51)
|oo A(QQ(ar ar+1)) o.+ 113
(52)
1|,- - 10+1 AlQQaa ar+ 1)) + (0++1 - oz+1) 112
(53)
||A(ar a - A(Q(a: a)- + (0r+1 o 1|
(54)
= ||A(a; a++1 - -Q(a: a++1)) + (0 0
(55)
< 2|A(a, a++1 Q(a: - + 2| 6
(56)
Since [|(0 0,+1)l1 represents the error from the previous time step, applying the submultiplicative inequality yields:
E² ||o, - o
(57)
< 2| ar+1 - Q(a; a(+1)11 + 2c a 112 2e?
(58)
(59)
Accumulating the error from time T' to f, we obtain Equation (49).
For the crror-compensation modulation, we compute:
e² |oo 0z112
(60)
= |lo, - A(Q(a, a(+1)) - 6
(61)
= ||A(ar) - A(Q((a, å - A A(a
(62)
||A(a, - Q(ar a
(63)
5 cl/A1|l1e | å 11
(64)
13

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Next, we expand a, A
a a
2 aa Q(a ax+2) àr+2112
(65)
= =laa - a++1 - Q(a++1 - å(+2) + ac+1 à
(66)
- 2/ a.+1l12 + + - åo42) + act1 2z+2 12
(67)
- 2 -
a 112 + 2c|lac+1 å,42llá
(68)
Substituting this into Equation (64), we complete the proof.
A.3. Proof of Corollary
Corollary A.3. Let x Id be vector, and let the quantization bandwidth be b N. Define the max-min dynamic quantizer as follows:
(x) - mmnnx))
8
(69)
2h - 1
min(x)
(70)
Xint clamp(
2, 0, 1).
(71)
The corresponding dequantization is given by:
Q(x) = s(xinr z).
(72)
For any 0 < <<< 1, we can revise Q with a new bandwidth satisfying:
|x - (1)) < </1x1
(73)
Proof: From Theorem 4.3, we have:
(max(x) min(x))
||x - Q(x)l1
(74)
(26 - 12
4||x 13.
(75)
(2 1)²
4||x| Ed
(76)
(20 1)2
To satisfy the desired bound, we choose b such that:
4d +1)
b > logg
(77)
Thus, the proof is complete.
B. Implementation details
In this section, we talk about the hyperparameters our experiments and the implementation details of our MoDiff.
Baselines. For the implementation of baselines, we follow the existing codebase. Specifically, we conduct Q-Diffusion experiments by directly using their provided code (Li et al., 2023). We also utilize the calibration datasets they provide to quantize the models at different bit levels. For LCQ, we follow the BRECQ framework and adopt channel-wise quantization (Li et al., 2021).
MoDiff. For our MoDiff implementation, we incorporate several key techniques:
14

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
We remove all bias terms from layers that apply MoDiff. This is necessary because our method, as described in Equation (13), requires layers to be bias-free to prevent unwanted accumulation of bias terms.
Bias Removal:
Warm-up: We apply warm-up at the first step, where we use full activation for computation. More detailed analysis shown in Appendix D.5.
Calibration Dataset Reconstruction: We reconstruct the calibration dataset for Q-Diff + MoDiff, ensuring it captures nearby information. During calibration, we store the inputs and outputs of MoDiff rather than the raw activations.
Layer-wise Reconstruction: Instead of reconstructing entire blocks, we reconstruct each layer individually, as we find this approach leads to more stable performance.
Hyperparameter Consistency: We do not fine-tune the calibration hyperparameters, as optimizing them is not the primary focus of our work.
C. Additional Main Results
C.1. Results on Stable Diffusion
To demonstrate that our method generalizes to larger-scale datasets and higher resolutions, we conduct experiments on MS-COCO 2014 (Lin et al., 2014) using Stable Diffusion v1.4 with DPM solvers(Lu et al., 2022). We apply tensor- wise dynamic quantization and evaluate the quantized models within the Q-Diffusion framework. A total of 30,000 images generated using 50 sampling steps. As shown in Table 7, the resulting FID scores confirm that MoDiff consistently performs well on large-scale diffusion models.
Table The FID and sFID on MS-COCO with Stable Diffusion using PLMS solver under different precisions. The best performance bolded.
Methods
Bits (W/A)
FID
sFID |
LTQ
12.15
19.05
8/8
LTQ+MoDiff (Ours)
12.14
19.05
LTQ
71.38
59.74
8/6
LTQ+MoDiff (Ours)
13.21
20.07
LTQ
408.42
199.59
8/4
LTQfMoDiff (Ours)
225.22
104.12
C.2. Results on Transformer-Based Models
To evaluate the generalizability of MoDiff across different architectures, we conduct experiments on the Diffusion Trans- former (Peebles & Xie, 2023). Following PTQ4DiT (Wu et al., 2024), we use DiT-XL/2 as the baseline model. The experiments are performed on the ImageNet 256x256 dataset (Russakovsky al., 2015) using tensor-wise dynamic quan- tization. We generate 10,000 images using 50 sampling steps for evaluation. As shown in Table 8, MoDiff consistently enhances generation quality under low activation bit widths.
Table 8. The IS, FID, and sFID for ImageNet 256x256 with DIT-XL/2 under different precisions.
The best performance is bolded.
Methods
Bits (W/A)
FID J
sFID
PTQ4DIT
36.91
54.80
89.60
8/8
PTQ4DiT+MoDiff (Ours)
37.37
53.76
89.53
PTQ4DiT
3.41
200.26
373.71
8/6
PTQ4DiT+MoDiff (Ours)
36.74
54.74
88.49
PTQ4DiT
1.45
271.87
207.59
8/4
PTQ4DiT+MoDiff (Ours)
17.23
90.91
102.07
15

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
C.3. More Measurements Generation Quality
In the main paper, we evaluate the quality of generated outputs using Inception Score (IS), Fréchet Inception Distance (FID), and sFID. Here, further assess the performance of our method using precision and recall.
The results are presented in Table 9, Table 10, and Table 11. These results demonstrate that MoDiff effectively preserves precision and recall even at low activation bit levels. For instance, on CIFAR-IO, LCQ+MoDiff achieves precision of 0.58 and recall of 0,50, whereas LCQ alone results in for both metrics.
Table The Precision and Recall for CIFAR-10 with DDIM under different Bits. The best performance is bolded.
Methods
Bits (W/A)
Precision
Recall
|| Bits (W/A)
Precision
Recall
Full Prec. (Act)
8/32
0.65
0.55
4/32
0.64
0.56
Q-Diff'
0.65
0.55
0.66
0.58
O-Diff+MDDif' (Ours)
0.65
0,56
0.65
0.58
8/8
4/8
LCQ
0.67
0.59
0.67
0.57
LCQ+MoDiff (Ours)
0.66
0.59
0.67
0.55
Q-Diff
0.46
0.47
0.47
0.44
Q-Diff+MoDif' (Ours)
0.66
0.57
0.65
0.59
8/6
4/6
LCQ
0.67
0.58
0.67
0.57
LCQ+MoDiff (Ours)
0.66
0.58
0.67
0.56
Q-Diff
0.08
0.00
0.05
0.00
O-Diff-MDDiffi(uurs)
0.54
0,53
0.53
0.55
8/4
4/4
LCQ
0.47
0,44
0.48
0.43
LCQ+MoDiff' (Ours)
0.67
0.59
0.67
0.57
Q-Diff
0.00
0.00
0.00
0.00
O-Diff+MoDin' (Ours)
0.45
0.39
0.33
0.32
8/3
4/3
L.QQ
0.33
0.08
0.35
LCQ+MoDiff (Ours)
0.66
0.59
0.67
0.57
Q-Diff
0.00
0.00
0.00
0.00
O-Diff+MoDif' (Ours)
0.00
0.00
0.14
0.00
8/2
4/2
LCQ
0.00
0.00
0.00
0.00
LCQ+MoDiff (Ours)
0.58
0.50
0.58
0.47
Table 10. The Precision and Recall for Church with LDM-8 under different Bits.
The best performance is bolded.
Methods
Bits (W/A)
Precision
Recall
Bits (W/A)
Precision
Recall
Full Prec. (Act)
8/32
0.63
0.51
4/32
0.63
0.52
LCQ
0.62
0.47
0.62
0.46
8/8
4/8
LCQ+MoDiff (Ours)
0.63
0.53
0.63
0.53
LCQ
0.59
0.46
0,59
0.45
8/6
4/6
LCQ+MoDiff (Ours)
0.63
0.53
0.63
0.53
LCQ
0.03
0.14
0.02
0.07
8/4
4/4
LCQ+MoDiff (Ours)
0.63
0.53
0.63
0.5
LCQ
0.00
0.00
0.00
0.00
8/3
4/3
L.Qff (Ours)
0.61
0.34
0.60
0.34
D. Ablation Study
D.I. Results on Other Weight Precision
In the main paper, we present results for 8-bit weight quantization on LSUN-Churches and limitation. In this section, we extend our analysis to 4-bit weight quantization and observe consistent conclusions. As shown in Table 12 and Table 13, our method successfully maintains generation quality at 4/3 bits for Churches and 4/4 bits for Bedrooms. In contrast, LCQ experiences a significant performance drop.
for the page
16

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table 11. The Precision and Recall for Bedroom with LDM-4 under different Bits.
The hest performance bolded.
Methods
Bits (W/A)
Precision
Recall
Bits (W/A)
Precision
Recall
Full Prec. (Act)
8/32
0.65
0.45
4/32
0.66
0.41
LCQ
0.65
0,45
0.68
0.41
8/8
4/8
LCQ+MoDiff (Ours)
0.60
0.51
0.62
0.47
LCQ
0.17
0.13
0.63
0.43
8/6
4/6
LCQ+MoDiff (Ours)
0.59
0.51
0.62
0.47
T.CQ
0.00
0.00
0.00
8/4
4/4
LCQ+MoDiff (Ours)
0.40
0.17
0.46
0.22
Table 12. The FID, sFID, and GBOPs for LSUN-Church with LDM under 4-bit weight quantization. The hest performance is bolded.
Methods
Bits (W/A)
GBops
FID
sFID 4
Full Prec. (Act)
8/32
5015
4.03
10.89
LCQ
4.02
11.53
8/8
1254
LCQ+MoDiff (Ours)
3.99
10.06
LCQ
4.50
12.90
8/6
940
LCQ+MoDiff (Ours)
3.89
10.12
LCQ
198.37
161.03
8/4
627
LCQ+MoDiff (Ours)
34.02
10.59
LCQ
341.62
407.68
8/3
470
LCQ+MoDiff (Ours)
12.05
35.29
Table 13. The 18, FID, sFID, and GBOPs for -eU--oddo with LDM under 4-bit weight quantization. bolded.
The best performance is
Methods
Bits (W/A)
GBops
FID |
sFID
Full Prec.
8/32
25560
3.45
8.45
LCQ
3.61
8.65
8/8
LCQ+MoDiff (Ours)
3.57
8.44
LCQ
64.17
63.18
8/6
4609
LCQ+MoDiff' (Ours)
3.57
6.53
LCQ
372.30
262.11
8/4
3195
LCQ+MoDiff' (Ours)
27.88
77.85
D.2. Results on Tensor- Wise Quantization
In our main experiments, we present results using dynamic channel-wise quantization (LCQ). this section, we extend our analysis dynamic tensor-wise quantization (LTQ), which is more hardware-friende friendly. We conduct experiments on CIFAR- 10 using DDIM, while continuing to use Q-Diffusion checkpoints for weight quantization. As shown in Table 14, MoDiff framework is also effective for L.Q. However, the minimum activation bit-width achievable with LTQ is higher than that of LCQ. This is because tensor-wise quantization operates on higher-dimensional data, making accurate quantization more challenging.
D.3. Results on More Samplers
In the main paper, we demonstrate that MoDiff generalizes to the DDPM sampler. Here, we further show its applicability to additional solvers. Specifically, we perform tensor-wise dynamic quantization using DPM--olver-2 (Lu et al., 2022) on CIFAR-10 with sampling steps. Additionally, we evaluate MoDiff with the PLMS solver using 50 steps on Stable Diffusion with the MS-COCO 2014 dataset (Liu et al., 2022). As shown in Table 15 and Table 7, MoDiff consistently
17

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table 14. The IS, FID, sFID, and GBOPs for CIFAR-IU 10 with DDIM using tensor wise quantization under different precisions. The best performance is bolded.
Methods
Bits (W/A)
IS f
FID +
sFID 4
Bits (W/A)
IS f
FID +
sFID
Full Prec. (Act)
8/32
9.00
4.24
4.41
4/32
8.78
5.09
5.19
LTQ
9.08
4.19
4.40
8.80
5.02
5.21
8/8
4/8
LTQ+MoDifT (Ours)
9.04
4.21
4.37
8.76
5.05
5.16
LTQ
8.98
9.93
8.69
8.89
9.96
8.07
8/6
4/6
LTQ+MoDiff (Ours)
9.09
4.00
4.27
8,80
5.04
4.42
LTQ
2,27
306.06
94.28
2.37
294.88
90.91
8/4
4/4
LTQ+MoDiff(Uus)
8.37
28.19
19.90
8.35
26.17
18.94
LTQ
1.19
457.25
165.85
1.19
457.11
165.61
4/2
LTQ+MoDiff (Ours)
4.26
186.04
86.73
3.29
146.52
87.78
improves FID) scores across different solvers.
Table 15. The FID on CIFAR-10 with DDIM using DPM solver under different precisions. The best performance is bolded.
Methods
Bits (W/A)
FID
DPM
3.92
8/8
DPM+MoDiff (Ours)
3.91
DPM
10.82
8/6
DPM+MoDiff (Ours)
3.91
DPM
299.72
8/4
DPM+MoDiff (Ours)
26.54
D.4. Results on Fewer Generation Steps
To demonstrate that MoDiff remains effective with fewer generation steps, we conduct experiments on CIFAR-10 using the DDIM sampler with only 20 steps. Tensor-wise dynamic quantization is applied throughout. As shown in Table 16, MoDiff maintains strong performance even under this reduced-step setting.
Table 16. FID CIFAR-10 using the DDIM sampler in the ablation study of fewer steps.
Methods
Bits (W/A)
FID 1
LTQ
6.93
8/8
LTQ+MoDiff (Ours)
6.90
LTQ
20.28
8/6
LTQ+MoDiff (Ours)
6.75
LTQ
297.21
8/4
MoDiff (Ours)
22.12
line of research has focused on distilling diffusion models into few-step variants, which can achieve comparable generation quality within significantly fewer sampling steps. To evaluate the generalizability of MoDiff in this setting, conduct experiments with MixDQ (Zhao et al., 2024), method specifically designed for few-step diffusion. We use SDXL-'Turbo as the backbone and apply 2, 4, and 8 sampling steps for image generation on the MS-COCO 2014 dataset (Lin et al., 2014). generating 10,000 images for FID computation. As shown Table 17, our method is compatible with MixDQ and further improves performance in the few-step diffusion regime. The performance indicates that is more challenging to lower the activation bit for SDXL-Turbo.
18

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table 17. FID MS-COCO using SurrooTurho and MixDQ across different generation steps.
The best performance bolded.
Step
Bits(W/A)
MixDQ
MixDQ+MoDiff
8/8
46.48
46.30
2
8/6
318.68
193.17
8/4
304.77
192.65
8/8
44.29
44.74
4
8/6
318.57
191.59
325.68
192.74
8/8
44.61
43.30
8
8/6
347.75
210.38
8/4
348.75
212.68
D.5. Results on Warm-up
To verify that warm-up is not the primary source of improvement, we conduct an ablation study by applying warm-up to the baseline and removing it from MoDiff. The experiments are performed using the DDIM sampler on CIFAR-10 10 with LCQ. As shown in Table 18, MoDiff consistently outperforms the baseline under fair comparison, indicating that the observed performance gains are not attributable to the warm-up mechanism.
Table 18. FID on CIFAR-10 using the DDIM sampler in the ablation study of warm-up. The best performance is bolded.
Bits (W/A)
LCQ w/o warmup
LCQ w/ warmup
LCQ+MoDiff w/o warmup
LCQ+MoDiff w/ warmup
8/8
4.19
4.19
4.22
4.21
8/6
9.93
9.53
4.25
4.00
8/4
306.06
299.96
31.22
28.19
Moreover, as indicated by Theorem 4.4, warm-up can be achieved by repeatedly inputting ay- This process converges to the full-precision activation due to the contraction of the quantization error. As demonstrated in our experiments, approximately 4 to 5 steps are sufficient reduce the quantization error negligible level on CIFAR-10 using 4-bit precision.
D.6. Analysis on Memory Consumption
In the main paper, we present the trade-off analysis between computation cost and memory cost for MoDiff when generating single image on CIFAR-10 with DDIM. In this section, we extend our analysis to larger batch sizes selected from {2, 4, 8). The results are shown in Tables 19, 20, and 21. The results, shown in Tables 19, 20, and 21, demonstrate that MoDiff significantly reduces computation cost while incurring only minimal increase in memory usage.
Table The relationship between BOPs and memory usage of our method using DDIM on CIFAR-10 generation with batch size
Measurement
W8A2
W8A4
W8A8
W8A32
GBops
204
410
918
3272
Memory (Mb)
36.49
38.89
43.69
36.09
Table 20, The relationship between BOPs and memory usage of our method using DDIM on CIFAR-10 for generation with batch size 4.
Measurement
W8A2
W8A4
W8A8
W8A32
GBops
408
820
1836
6544
Memory (Mb)
38.89
43.69
53.28
38.09
19

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Table 21. The relationship between ROPs and memory usage of method using DDIM CIFAR-10 for generation with batch size 8.
Measurement
W8A2
W8A4
W8A8
W8A32
GBops
906
1640
3672
13088
Memory (Mb)
43.69
53.28
72.47
42.09
E. Compared to PTQD
Post-Training Quantization for Diffusion Models (PTQD) aims reduce quantization error by post-processing quantized models, sharing similar objective with our work. this section, we highlight the key differences between MoDiff and PTQD. Compared to PTQD, MoDiff is (1) more general and flexible, (2) free from strong assumptions about error distribution, and (3) significantly more effective in low-precision scenarios.
(1) PTQD requires solver-specific adaptations to address variance and bias, while MoDiff can be applied across solvers without modification. Moreover, PTQD is restricted to standard diffusion models, whereas MoDiff also supports cached diffusion models by compensating for reuse errors in cached components.
(2) PTQD relies on strong assumptions about error distribution, specifically that quantization errors follow Gaussian distribution after imput rescaling. This assumption introduce inaceuracies in error estimation. In contrast, MoDiff leverages the widely observed similarity between timesteps, which is well-supported by prior works (Ma et al., 2024b).
(3) MoDiff performs well in low-precision activation settings, whereas PTQD fails entirely. To demonstrate this, we evaluate both methods on CIFAR-10 10 with W8A4 quantization. PTQD yields an FID of 397.12 and fails to produce meaningful images, while MoDiff achieves a much lower FID of 13.41.
F. Comprehensive Visualization Results
In this section, present visualization results for CIFAR-IO. LSUN-Churches, LSUN-Bedroom, and MS-COCO-2014, These results illustrate the performance that MoDiff can achieve. For instance, as shown in Figure 5, LCQ+MoDDif closely aligns with full-precision generation at W8A4, whereas LCQ only captures the image textures. Additionally, LCQ+MoDiff can still generate recognizable images at W8A3, albeit with some loss of detail.
LTQ+M W8A8
LTQ W8A8
LTQ+MoDiff W8A6
LTQ W8A6
L.TO+MDiff W8A4
LTQ W8A4
Figure 4, Visualization of MS-COCO-2014 generated using LTQ and LTQ+MoDiff under 8-bit weight quantization precisions on Stable
Diffusion v..4.
20

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Full Precision (Act) W8A32
MoO+M W8A8
LCQ W8A8
LCQ+MoDiff W8A6
LCQ W8A6
LCQ+MoDiff W8A4
LCQ W8A4
LCQ+MoDiff W8A3
LCQ W8A3
Figure Visualization LSUN-Churches 256 256 generated using LCQ and LCQ+MoDiff under 8-bit weight quantization precisions.
21

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Full Precision (Act) W4A32
MoDif W4A8
LCQ W4A8
LCQ+MoDiff W4A6
LCQ W4A6
LCQ+MoDiff W4A4
LCQ W4A4
LCQ+MoDiff W4A3
LCQ W4A3
Figure Visualization LSUN-Churches 256 256 generated using LCQ and LCQ+MoDiff 4-bit weight quantization precisions.
22

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Full Precision (Act) W8A32
MCQ+M W8A8
LCQ WSAS
L.Q+MoDiff W8A6
LCQ W8A6
LCQ+MoDiff W8A4
LCQ W8A4
Figure Visualization L.SUN-bedrooms. generated using LCQ and LCQ+MoDiff under 8-bit weight quantization precisions.
23

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Full Precision (Act) W4A32
MoQ+f W4A8
LCQ W4A8
L.Q+MoDiff W4A6
LCQ W4A6
LCQ+MoDiff W4A4
LCQ W4A4
Figure Visualization L.SUN-bedrooms generated using LCQ and LCQ+MoDiff under 4-bit weight quantization precisions.
24

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Full Precision (Act) W8A32
LCQ+MoDiff W8A8
LCQ W8A8
O-Diff+MoDif W8A8
Q-Diff W8A8
LCQ+MoDiff W8A6
LCQ W8A6
O-Diff+MoDiff W8A6
Q-Diff W8A6
LCQ+MoDiff W8A4
LCQ W8A4
O-Diff+MoDiff W8A4
Q-Diff W8A4
LCQ+MoDiff W8A3
LCQ W8A3
O-Diff+MoDiff W8A3
Q-Diff W8A3
LCQ+MoDiff W8A2
I.CQ W8A2
Q-Diff+MoDif W8A2
Q-Diff W8A2
Figure 9, Visualization of LSUN bedroomss 256 x 256 generated using LCQ, LCQ+MoDiff, Q-Diff, and Q-Diff+MoDif under 8-bit
weight quantization precisions.
25

---

Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Full Precision (Act) w4A32
LCQ+MoDiff w4A8
LCQ w4A8
O-Diff+MoDif w4A8
Q-Diff w4A8
LCQ+MoDiff w4A6
LCQ w4A6
Q-Diff+MoDif w4A6
Q-Diff w4A6
L.Q++MoDff w4A4
LCQ w4A4
Q-Diff+MoDif w4A4
Q-Diff w4A4
LCQ+MoDiff w4A3
L.Q w4A3
O-Diff+MoDif w4A3
Q-Diff w4A3
L.Q++oDiff w4A2
L.Q w4A2
O-Diff+MoDif w4A2
Q-Diff w4A2
Figure 10. Visualization of LSUN- bedrooms 256 x 256 generated using LCQ, LCQ+MoDiff, Q-Diff, and Q-Diff+MoDiff under 4-bit
weight quantization precisions.
26


Reviews:
[{"review_id": "dSqQ3GfHxp", "reviewer": "Reviewer_EG1n", "summary": "This paper proposed a method for accelerating diffusion model sampling by modulated quantitation and a carefully designed error compensation mechanism, the method is able to significantly reduce the accumulative error of previous methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty, effectiveness, and clarity of our paper. We are glad to address your questions.\n\n**1. In Table 1, FID column, some values are incorrectly bolded, e.g., 4.21**\n\nThank you for pointing out the incorrect bold formatting. We will revise it in a new version.\n\n**2. In the proof of Theorem 4.2, is there a factor of two missing in equation (56)?**\n\nThanks for pointing out the missing constant scalar. Here is the revised proof, which does not affect the conclusions:\n\n$$\n\\tilde{\\mathbf{e}}\\_t^2 = \\\\|\\mathbf{o}\\_t - \\tilde{\\mathbf{o}}\\_t\\\\|\\_2^2\n$$\n\n$$\n= \\\\|\\mathbf{o}\\_t - \\mathcal{A}(Q(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1})) - \\tilde{\\mathbf{o}}\\_{t+1}\\\\|\\_2^2\n$$\n\n$$\n= \\\\|\\mathbf{o}\\_t - \\mathbf{o}\\_{t+1} - \\mathcal{A}(Q(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1})) + (\\mathbf{o}\\_{t+1} - \\tilde{\\mathbf{o}}\\_{t+1})\\\\|\\_2^2\n$$\n\n$$\n= \\\\|\\mathcal{A}(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1}) - \\mathcal{A}(Q(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1})) + (\\mathbf{o}\\_{t+1} - \\tilde{\\mathbf{o}}\\_{t+1})\\\\|\\_2^2\n$$\n\n$$\n= \\\\|\\mathcal{A}(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1} - Q(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1})) + (\\mathbf{o}\\_{t+1} - \\tilde{\\mathbf{o}}\\_{t+1})\\\\|\\_2^2\n$$\n\n$$\n\\leq 2\\\\|\\mathcal{A}(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1} - Q(\\mathbf{a}\\_t - \\mathbf{a}\\_{t+1}))\\\\|\\_2^2 + 2\\\\|\\mathbf{o}\\_{t+1} - \\tilde{\\mathbf{o}}\\_{t+1}\\\\|\\_2^2\n$$", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The proof of Theorem 4.3 is correct minus some minor mistake that only affect the constant. See below suggestions.", "experimental_designs_or_analyses": "Yes. The experiments are thorough and give strong evidence for the claims.", "relation_to_broader_scientific_literature": "The main contribution lies in the error compensation mechanism designed with modulated quantization. Previous quantization methods have the issue that error accumulates with diffusion steps.", "essential_references_not_discussed": "I am not an expert in quantization methods for diffusion models. The diffusion model literature is properly cited in this work.", "other_strengths_and_weaknesses": "* The paper is well presented and easy to understand with sufficient background materials.\n* The error compensation scheme addresses the issue of error accumulation nicely and is backed by theory. \n* Results on both pixel diffusion and latent diffusion show significant improvement by applying MoDiff to existing diffusion quantization methods Q-Diff and LCQ, especially in the low bit regime. \n\nI don't see much weaknesses except that the experiments can benefit from having more diverse datasets like ImageNet 256x256 and 512x512,", "other_comments_or_suggestions": "Minor suggestions:\n\n* In Table 1, FID column, some values are incorrectly bolded, e.g., 4.21\n* In the proof of Theorem 4.2, is there a factor of two missing in equation (56)?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hiZFhh2bdK", "reviewer": "Reviewer_NH3e", "summary": "This paper introduces Modulated Diffusion (MoDiff), an approach that combines caching and quantization techniques while addressing their limitations. By leveraging the differences in activations across diffusion timesteps for quantization and incorporating an error compensation mechanism, MoDiff effectively mitigates error accumulation. The method's effectiveness is validated through experiments on the CIFAR-10 and LSUN datasets.\n\n## update after rebuttal\n\nI appreciate the authors' efforts of adding new baselines during the rebuttal, that certainly improved the paper. This paper is a borderline paper, However, as the authors didn't fully address the concern on the practical speed up, I tend to keep my original rating, as I find the description of the rating 2 \"leaning towards reject, but could also be accepted\" to best describe my judgement.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Experimental Designs Or Analyses: Q1**\n\nAlthough the goal of this paper is to validate our method, not to benchmark it across all diffusion architectures, we conduct experiments following [1] to address your concern, using DiT on ImageNet. The results demonstrate that our method consistently improves generation quality:\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 54.80 | **53.76** |\n| 8/6 | 200.26 | **54.74** |\n| 8/4 | 271.87 | **90.91** |\n\n[1] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.\n\n**Experimental Designs Or Analyses: Q2**\n\nWe will include the mentioned literature in the revised version. Regarding the papers you cited, MoDiff (1) offers orthogonal contributions and can complement quantization methods; (2) generalizes caching approaches; and (3) reduces quantization error better compared to PTQD.\n\n(1) ViDiT-Q is a quantization method that does not exploit time step similarities or compensate for quantization errors. MoDiff addresses these limitations to reduce quantization error and prevent error accumulation, which is orthogonal to ViDiT-Q. Moreover, MoDiff is a general framework compatible with ViDiT-Q's techniques, including group-wise quantization, channel balance, and mixed-precision computation.\n\n(2) $\\Delta$-ViT and dual feature caching rely on empirical, heuristic designs to cache different components without compensating for quantization error. MoDiff is a general framework that encompasses these methods as special cases. Specifically, MoDiff reduces to them when the cached component is quantized to 0 bits while the remaining components use full precision.\n\n(3) PTQD reduces quantization error by post-processing quantized models based on assumed error distributions. However, it ignores timestep similarities and fails under low-precision activations. To verify this, we evaluate PTQD with W8A4 on CIFAR-10, where it yields an FID of 397.12 and fails to generate meaningful images, while MoDiff achieves a significantly better FID of 13.41.\n\n**Other Strengths And Weaknesses: Q1**\n\nWe believe there are some misunderstandings about the novelty of MoDiff. Specifically, (1) $\\Delta$-DiT only approximates temporal differences without modulating the cache, and (2) it lacks any form of error compensation.\n\nFirst, $\\Delta$-DiT does not approximate temporal differences but instead estimates differences between transformer blocks. Moreover, it directly reuses cached values, which introduces errors, whereas MoDiff applies lightweight modulation to reduce these errors.\n\nSecond, MoDiff explicitly traces quantization errors from the previous timestep and compensates for them in the next with theoretical guarantees. In contrast, caching methods, including $\\Delta$-DiT, simply reuse cached components without addressing accumulated errors. \n\n**Questions For Authors: Q1**\n\nWe want to emphasize that (1) MoDiff is hardware-friendly and benefits from hardware in practice, though the hardware implementation is out of the scope of our study, and (2) MoDiff focuses on improving computational efficiency rather than memory efficiency. In addition, exploring lower-bit activation, even down to 1 bit, is valuable and aligns with ongoing research in quantization [2, 3].\n\nFirst, GPU architectures support 4-bit activation computation [1], but performance degrades significantly at such low precision. MoDiff addresses this limitation by overcoming the performance bottleneck of low-precision activations.\n\nSecond, low-bit activations offer much more speedup over 8-bit activations by reducing FLOPs [1], making them a valuable target for acceleration. \n\nFinally, MoDiff represents a step toward enabling low-bit activations, not the endpoint. This direction could eventually push activation precision down to 1 bit, potentially driving future hardware designs to better support ultra-low-bit computation, such as logic-based operations.\n\n[1] Int4 Precision for AI Inference. https://developer.nvidia.com/blog/int4-for-ai-inference.\n\n[2] BinaryDM: Accurate Weight Binarization for Efficient Diffusion Models. ICLR 2025.\n\n[3] BiDM: Pushing the Limit of Quantization for Diffusion Models. NeurIPS 2024.\n\n**Questions For Authors: Q2**\n\nWe want to emphasize that MoDiff is agnostic to quantization methods. Channel-wise dynamic quantization is used to show the feasibility and potential of MoDiff following the baseline [1]. In addition to channel-wise dynamic quantization, we have also reported the results on Q-Diffusion and tensor-wise quantization (more friendly to hardware) in Table 1 and Table 11 with consistent improvement over measurements. \n\n[1] DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models. ICLR 2025. \n\n**Questions For Authors: Q3**\n\nTo address your concern, we perform tensor-wise quantization using the DPM-Solver-2 on CIFAR-10 with 20 sampling steps. \n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 3.92 | **3.91** |\n| 8/6 | 10.82 | **3.92** |\n| 8/4 | 299.72 | **26.54** |", "claims_and_evidence": "Yes, the claims are well-supported by evidence. Nevertheless, please see the \"Questions\" section for further questions on the quantization scheme.", "methods_and_evaluation_criteria": "The evaluation and methodology design are generally appropriate for the study.", "theoretical_claims": "The claims are correct.\n\n- Theorem 4.2: Quantization error is related to the input range and the number of bits. The small range of temporal differences allows for low-bit quantization (Paragraphs 1-86, 1-97).\n- Theorem 4.3: Error compensation makes the error decay exponentially, avoiding linear accumulation (Paragraphs 1-98, 1-110)", "experimental_designs_or_analyses": "The experimental designs are generally appropriate for the study. However, comparison with existing literature are lacking. The paper would benefit from a more thorough discussion and comparison with existing works in the field.\n1. Limited focus on U-Net-based architectures: While the paper primarily focuses on U-Net-based architectures, diffusion transformer-based architectures (e.g., DiT) have emerged as widely adopted alternatives. Including experiments or analysis on more recent architectures is recommended.\n2. Narrow scope of related work: The paper’s focus on diffusion model caching and quantization. However, only earlier research (e.g., DeepCache, Q-Diffusion, published in early 2023) is discussed. More recent diffusion quantization literature should be discussed in the related work section. Additionally, an analysis of whether the proposed method is applicable to these techniques would strengthen the authors’ claim that \"MoDiff is agnostic to quantization techniques.\"", "relation_to_broader_scientific_literature": "The idea of modulated quantization relates with other diffusion quantization, and caching techniques, such as:\n\n- Diffusion Quantization methods aim to reduce quantization error from alternative perspectives.\n\n[1] He, Yefei et al. “PTQD: Accurate Post-Training Quantization for Diffusion Models.” ArXiv abs/2305.10657 (2023): n. pag.\n\n[2] Zhao, Tianchen et al. “ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation.” ArXiv abs/2406.02540 (2024): n. Pag.\n\n- Diffusion Feature Caching techniques propose a similar idea of representing the residual of a timestep to reduce caching errors.\n\n[3] Chen, Pengtao et al. “Δ-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers.” ArXiv abs/2406.01125 (2024): n. Pag.\n\n[4] Zou, Chang et al. “Accelerating Diffusion Transformers with Dual Feature Caching.” ArXiv abs/2412.18911 (2024): n. pag.", "essential_references_not_discussed": "Some recent related literature related with diffusion quantization and caching are not discussed and compared, refer to the \"relation to broader scientific\" section.", "other_strengths_and_weaknesses": "**Other Strength**\n\nThe paper provides theoretical analyses to support the claims.\n\n**Other Weakness**\n\nI encourage the authors to further discuss the novelty: The key idea of MoDiff aligns with existing research. Specifically, MoDiff proposes to separately quantize the timestep-wise difference and the original computation, leveraging the observation that the temporal difference distribution \"has a smaller but consistent range.\" Similar concepts, such as approximating temporal differences and storing intermediate results for error compensation, have been explored in recent caching methods [1] to reduce caching errors. \n\n[1] Chen, Pengtao et al. “Δ-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers.” ArXiv abs/2406.01125 (2024): n. Pag.", "other_comments_or_suggestions": "No", "questions_for_authors": "I have three questions on the practical efficiency improvements:\n1.  The primary contribution of MoDiff is achieving lower activation bitwidth (e.g., W8A4). However, such bitwidth reductions face challenges in delivering practical savings. On GPUs, activations need to be upcasted to 8-bit for INT8 computation. In most cases, the memory cost of activations is significantly smaller than that of weights, so the memory reduction from quantizing activations to lower bitwidths is limited.\n2. As stated, “For activation quantization, dynamic channel-wise quantization determines the scaling factor based on the channel-wise min-max range of the input.” However, this channel-wise activation scheme can hinder actual hardware acceleration, as discussed in [1], because channels need to be summed together and should share the same quantization parameters to enable efficient integer computation.\n3. Additionally, the evaluation is limited to earlier solvers (e.g., DDIM) with a notably large number of timesteps (100–400). It remains unclear whether MoDiff is applicable to more commonly adopted efficient solvers, such as DPMSolver.\n\n[1] Zhao, Tianchen et al. “ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation.” ArXiv abs/2406.02540 (2024): n. Pag", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M1ZgcEaS0B", "reviewer": "Reviewer_32oq", "summary": "The author introduces MoDiff, a framework designed to accelerate generative modeling by addressing challenges in caching and post-training quantization (PTQ). MoDiff incorporates modulated quantization and error compensation to reduce quantization errors and mitigate error accumulation. Theoretical analysis supports its effectiveness, and experiments on CIFAR-10 and LSUN show that MoDiff enables PTQ to operate at as low as 3-bit activation quantization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty, effectiveness, and clarity of our paper. We believe there are some misunderstandings about our implementation, and we are glad to address your questions.\n\n**Essential References Not Discussed: Q1. Compare with PTQD.**\n\nCompared to PTQD, MoDiff is (1) more general and flexible, (2) free from strong assumptions about error distribution, and (3) significantly more effective in low-precision scenarios.\n\n(1) PTQD requires solver-specific adaptations to address variance and bias, while MoDiff can be applied across solvers without modification. Moreover, PTQD is restricted to standard diffusion models, whereas MoDiff also supports cached diffusion models by compensating for reuse errors in cached components.\n\n(2) PTQD relies on strong assumptions about error distribution, specifically that quantization errors follow a Gaussian distribution after input rescaling. This assumption can introduce inaccuracies in error estimation. In contrast, MoDiff leverages the widely observed similarity between timesteps, which is well-supported by prior works [1].\n\n(3) MoDiff performs well in low-precision activation settings, whereas PTQD fails entirely. To demonstrate this, we evaluate both methods on CIFAR-10 with W8A4 quantization. PTQD yields an FID of 397.12 and fails to produce meaningful images, while MoDiff achieves a much lower FID of 13.41.\n\nWe will include these comparisons in the revision. \n\n[1] Deepcache: Accelerating diffusion models for free. CVPR 2024.\n\n**Other Strengths And Weaknesses: Q1. Lack experiments on large models (Stable Diffusion).**\n\nTo address your concern, we conduct tensor-wise quantization on Stable Diffusion v1.4 using the 50-step PLMS solver on MS-COCO-2014. The resulting FID scores demonstrate that MoDiff consistently performs well on large-scale diffusion models.\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 54.80 | **53.76** |\n| 8/6 | 200.26 | **54.74** |\n| 8/4 | 271.87 | **90.91** |\n\n**Other Strengths And Weaknesses: Q2. MoDiff is restrictive, which does not support norm layer folding and depends on dynamic quantization.**\n\nThere are a few misunderstandings about our paper. We clarify that MoDiff (1) supports norm layer folding and (2) does not depend on dynamic quantization.\n\n(1) MoDiff supports norm layer folding and is practical in use. It applies to any linear operation, not just linear layers. By folding norm layers with other linear components such as convolution layers, MoDiff can be applied to the resulting block due to preserved linearity. In our implementation, we did not perform such merging, following standard practices in the diffusion quantization community [1].\n\n(2) MoDiff is agnostic to the quantization method and not limited to dynamic quantization. As shown in Table 1 of the main paper, it consistently improves performance in low-precision settings with Q-Diffusion, which is static quantization. Additionally, dynamic quantization is well-studied in the literature [2] and supported by certain hardware platforms [3].\n\n[1] https://github.com/Xiuyu-Li/q-diffusion.\n\n[2] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023.\n\n[3] NVIDIA TensorRT Documentation. https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/work-quantized-types.html\n\n**Other Strengths And Weaknesses: Q3. Suboptimal results in some settings.**\n\nFirst, after carefully reviewing all log files, we identified some errors in writing the results for the $W8A8$ and $W4A8$ cases. The revised results are now provided. As shown in the updated table, MoDiff consistently matches both the baseline and full-precision models in terms of FID and sFID.\n\n| Model | FID | sFID |\n|:--|--:|--:|\n| LCQ 4/8 | 4.96 | 4.94 |\n| LCQ + MoDiff 4/8 | 4.95 | 4.95 |\n| LTQ 4/8 | 5.02 | 5.21 |\n| LTQ + MoDiff 4/8 | 5.05 | 5.16 |\n| LTQ 8/8 | 4.19 | 4.40 |\n| LTQ + MoDiff 8/8 | 4.21 | 4.37 |\n\nSecond, our MoDiff is agnostic to quantization methods, where channel-wise dynamic quantization (we follow [1] and agree that it is hardware-unfriendly) is used to show the feasibility and potential of MoDiff. In addition to channel-wise dynamic quantization, we report the results on Q-Diffusion and tensor-wise quantization in Table 1 and 11 with consistent improvement over FID on low-bit activation cases. \n\n[1] DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Models. ICLR 2025. \n\n**Other Comments Or Suggestions: Q1. Missing right parenthesis in Eq (18).**\n\nThank you for pointing it out. We will correct the typo in the revised version.\n\n**Questions For Authors: Q1. If MoDiff needs more training time.**\n\nMoDiff is even more efficient than Q-Diffusion in scaling factor calibration. As noted in our implementation, we skip block-wise quantization for training stability while maintaining the same number of iterations to learn scaling factors. This results in reduced time for post-training calibration.", "claims_and_evidence": "The submission claims that existing caching methods introduce significant computation errors, but leveraging temporal stability in activation patterns can mitigate these issues. This claim is supported by empirical observations and visualizations (Figure 1), which show that temporal differences in activations have a more stable and concentrated distribution, reducing outliers and error accumulation.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are aligned with the problem. MoDiff is tested on relevant benchmark datasets (CIFAR-10, LSUN) using widely accepted evaluation metrics like FID, IS, and SFID. The study also includes various quantization methods and bit-width settings for comprehensive comparison.", "theoretical_claims": "The theoretical claims in the submission are supported by mathematical analysis, particularly Theorems 4.2 and 4.3, which establish the relationship between quantization error, input magnitude, and error accumulation. The proofs suggest that MoDiff effectively reduces quantization error and mitigates error propagation over time.", "experimental_designs_or_analyses": "The experimental design and analysis are well-structured, assessing MoDiff's effectiveness across multiple datasets (CIFAR-10, LSUN) using standard evaluation metrics (IS, FID, sFID). The results show that MoDiff preserves generation quality while significantly reducing computational costs.", "relation_to_broader_scientific_literature": "The proposed MoDiff framework builds on existing diffusion model quantization techniques and introduces modulated quantization with error compensation to reduce quantization error and mitigate error accumulation. Notably, MoDiff is orthogonal to existing quantization methods, meaning it can be integrated with approaches like Q-diffusion to further enhance performance.", "essential_references_not_discussed": "MoDiff introduces a novel quantization approach that is closely related to existing post-training quantization methods like PTQD. PTQD addresses the challenges of quantization noise and its accumulation by introducing bias correction and variance schedule calibration. Similarly, MoDiff aims to mitigate quantization error and error accumulation. A direct comparison with PTQD would help clarify the advantages and distinctions of MoDiff.\n\n[1] PTQD: Accurate Post-Training Quantization for Diffusion Models.", "other_strengths_and_weaknesses": "1、The paper lacks experiments on larger models, such as Stable Diffusion, which would better demonstrate its scalability and real-world applicability.\n\n2、MoDiff's use of modulated quantization and error compensation is highly restrictive, as it can only be applied to linear modules. This requires re-quantization and de-quantization at every linear module, necessitating a redesigned quantization computation graph. As a result, layers like batch normalization cannot be folded, significantly reducing its practical feasibility. Additionally, since MoDiff relies on dynamic quantization, its actual acceleration benefits remain questionable.\n\n3、The performance of LCQ+MoDiff is suboptimal. In Table 1, the results for 4/8 and 4/6 bit quantization are worse than the original LCQ method. Moreover, channel-wise quantization is difficult to implement efficiently on real hardware. Even with tensor-wise quantization, Table 11 shows that LTQ+MoDiff underperforms compared to the original LTQ method at 8/8 and 4/8 bit precision.", "other_comments_or_suggestions": "There is a missing right parenthesis in Equation (18).", "questions_for_authors": "The paper adopts a layer-wise reconstruction training approach, which differs from the traditional block-wise method. This raises the question of whether it results in a longer training duration. A comparison of training efficiency between these approaches would help clarify the potential trade-offs.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v50M3pglJS", "reviewer": "Reviewer_wYCA", "summary": "The paper investigates the shortcomings of current acceleration methods for diffusion models, such as caching and quantization, which suffer from error accumulation and high approximation errors, and introduces MoDiff—a novel framework that accelerates diffusion models through modulated quantization combined with error compensation. The authors support their proposal with theoretical analyses that detail quantization errors and the benefits of the error compensation mechanism, and they validate their method through experiments on datasets like CIFAR-10, LSUN-Churches, and LSUN-Bedroom, demonstrating that activation precision can be reduced from 8 bits to as low as 3 bits without any performance degradation in a training-free manner.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty, effectiveness, and clarity of our paper. We are glad to address your questions.\n\n**Experimental Designs Or Analyses: Q1. All experiments were conducted on small-scale datasets such as CIFAR-10, LSUN-Churches, and LSUN-Bedroom, whereas it is standard practice to validate a research idea on larger datasets like ImageNet—especially since the proposed methods are training-free, making scalability feasible.**\n\nTo address your concern, we conduct experiments following [1], using DiT on ImageNet 256×256 with tensor-wise dynamic quantization. The results demonstrate that our method consistently improves generation quality at low activation bit widths:\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 54.80 | **53.76** |\n| 8/6 | 200.26 | **54.74** |\n| 8/4 | 271.87 | **90.91** |\n\n[1] PTQ4DiT: Post-training Quantization for Diffusion Transformers. NeurIPS 2024.\n\n**Experimental Designs Or Analyses: Q2. Should extend its evaluation to include more recent samplers, such as flow models, EDM, and DPM-solver.**\n\nTo address your concern, we perform tensor-wise quantization using the DPM-Solver-2 on CIFAR-10 with 20 sampling steps. Additionally, we conduct experiments with the PLMS solver using 50 steps on Stable Diffusion with MS-COCO-2014. In both cases, MoDiff consistently improves FID scores across different solvers, even with reduced sampling steps.\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 3.92 | **3.91** |\n| 8/6 | 10.82 | **3.92** |\n| 8/4 | 299.72 | **26.54** |\n\nTable1: DPM on CIFAR-10\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 54.80 | **53.76** |\n| 8/6 | 200.26 | **54.74** |\n| 8/4 | 271.87 | **90.91** |\n\nTable 2: PLMS on MS-COCO\n\n**Experimental Designs Or Analyses: Q3. The ablation study on error compensation would also benefit from employing direct image quality metrics like FID, IS, or P\\&R instead of the L2 metric, which is less convincing for assessing image quality.**\n\nTo address your concern, we conduct experiments to show the sFID measurement with or without error compensation on CIFAR10 with DDIM. The results show that error compensation reduces the error accumulation in low bit activation cases.\n\n| W/A | Baseline | w/o Compensation | w/ Compensation |\n|:-:|--:|--:|--:|\n| 8/6 | 4.61 | 4.41 | **4.40** |\n| 8/4 | 13.07 | 10.21 | **4.38** |\n| 8/3 | 33.97 | 25.42 | **4.38** |\n\n**Experimental Designs Or Analyses: Q4. Table 1 contains an error in the bolding.**\n\nThank you for pointing out the incorrect bold formatting. We will revise it in a new version.\n\n**Essential References Not Discussed: Q1. There are some caching methods missing discussion.**\n\nThank you for mentioning the relevant papers. We will include them in the related works section of the revised version.", "claims_and_evidence": "Yes.\n\nFigure 1 offers an interesting analysis that clearly motivates the use of modulated quantization, while the experimental results robustly validate the authors’ claims by demonstrating that MoDiff works well on various diffusion weight and activation precision.", "methods_and_evaluation_criteria": "Yes. The method part is well-motivated.", "theoretical_claims": "Yes. From my point of view, the proofs in section 4.2 and 4.3 are correct and easy to understand.", "experimental_designs_or_analyses": "The experiments are generally convincing, but they have minor issues. \n\n- All experiments were conducted on small-scale datasets such as CIFAR-10, LSUN-Churches, and LSUN-Bedroom, whereas it is standard practice to validate a research idea on larger datasets like ImageNet—especially since the proposed methods are training-free, making scalability feasible. \n- Additionally, section 5.3 should extend its evaluation to include more recent samplers, such as flow models, EDM, and DPM-solver, as relying solely on DDIM and DDPM is insufficient for the SOTA validation. \n- The ablation study on error compensation would also benefit from employing direct image quality metrics like FID, IS, or P&R instead of the L2 metric, which is less convincing for assessing image quality. \n- Table 1 contains an error in the bolding; in the 8/6 bits quantization setting, LCQ has a lower FID compared to LCQ+MoDiff, indicating an error in the highlighted results.", "relation_to_broader_scientific_literature": "This paper examines the limitations of current caching techniques and post-training quantization (PTQ) methods, and based on these insights, the authors introduce MoDiff. Notable related approaches include DeepCache: Accelerating Diffusion Models for Free (NeurIPS 2024) and Q-Diffusion: Quantizing Diffusion Models (ICCV2023).", "essential_references_not_discussed": "There are some caching methods missing discussion. The latter two are about video caching but it shares some similar ideas to image caching.\n\n- DiTFastAttn: Attention Compression for Diffusion Transformer Models, Neurips2024\n- FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality, ICLR2025\n- Adaptive Caching for Faster Video DiTs, arxiv.", "other_strengths_and_weaknesses": "please see above.", "other_comments_or_suggestions": "No any other suggestions.", "questions_for_authors": "No any other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFhM9ywyPr", "reviewer": "Reviewer_YCE9", "summary": "This work introduces MoDiff, a novel framework for accelerating diffusion models by combining modulated quantization and error compensation. It enhances existing techniques like caching and quantization, offering a more efficient approach without sacrificing generation quality. MoDiff reduces activation quantization from 8 bits to 3 bits in post-training quantization (PTQ) on datasets like CIFAR-10 and LSUN, with no performance loss.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing the novelty of our paper. We believe there are some misunderstandings about our implementation, and we will address your questions with the following experiments.\n\n**Methods And Evaluation Criteria: Q1**\n\nMoDiff focuses on quantization rather than the latest models or datasets. To clarify its generality, we evaluate MoDiff on Stable Diffusion with MS-COCO and DiT on ImageNet using public quantized checkpoints [1,2]. Results show that MoDiff generalizes well across architectures and datasets.\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 54.80 | 53.76 |\n| 8/6 | 200.26 | 54.74 |\n| 8/4 | 271.87 | 90.91 |\n\nTable 1: Stable Diffusion\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 54.80 | 53.76 |\n| 8/6 | 200.26 | 54.74 |\n| 8/4 | 271.87 | 90.91 |\n\nTable 2: DiT\n\n[1] Q-Diffusion. ICCV 2023.\n\n[2] PTQ4DiT. NeurIPS 2024.\n\n**Theoretical Claims: Q1**\n\nThe corollary validates our assumptions and guides design; it's not a strict per-layer rule nor the tightest bound. We aim to extend this to mixed-precision in future work.\n\n**Theoretical Claims: Q2**\n\nWe warm up by running the first timestep in full precision and then using low-bit activations. This adds negligible overhead. We apply warm-up to baselines for fairness, showing the performance gain is not from warm-up.\n\n| W/A | Baseline + Warmup | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 4.19 | 4.21 |\n| 8/6 | 9.53 | 4.28 |\n| 8/4 | 299.96 | 28.19 |\n\n**Experimental Designs Or Analyses: Q1**\n\nFirst, MoDiff is quantization-method agnostic. We follow [1]'s channel-wise dynamic quantization (hardware-unfriendly, but illustrative), and also test Q-Diffusion and tensor-wise quantization (Table 1, 11), showing consistent FID improvements.\n\nSecond, low-bit activation cases are rare due to the large quantization error they introduce. MoDiff breaks the limitations of existing methods by effectively reducing activation quantization error. Similar configurations are also used in prior works [2,3].\n\n[1] DGQ. ICLR 2025.\n\n[2] PikeLPN. CVPR 2024.\n\n[3] WRPN. ICLR 2018.\n\n**Experimental Designs Or Analyses: Q2**\n\nThe baseline does not necessarily outperform MoDiff since MoDiff (1) has smaller quantization errors and (2) has better performance on sFID. Specifically, MoDiff closely matches the full-precision model across all metrics compared to the baseline, meaning its output is nearly the same as the ground truth. Moreover, MoDiff outperforms the baseline in sFID. It is inaccurate to conclude that the baseline is better considering all metrics. \n\n**Experimental Designs Or Analyses: Q3**\n\nTo address your concern, we show how error compensation improves sFID for CIFAR-10 with DDIM.\n\n| W/A | Baseline | w/o Compensation | w/ Compensation |\n|:-:|--:|--:|--:|\n| 8/6 | 4.61 | 4.41 | 4.40 |\n| 8/4 | 13.07 | 10.21 | 4.38 |\n| 8/3 | 33.97 | 25.42 | 4.38 |\n\n**Experimental Designs Or Analyses: Q4**\n\nOur comparison with Q-Diffusion is fair. We only make minimal changes and necessary adaptations to integrate our method. Specifically, we (1) apply modulated computation and remove bias for correctness, (2) use temporal differences instead of raw activations for calibration since they serve as inputs during sampling. The last implementation is (3) omitting block-wise quantization for training stability. To demonstrate that (3) does not account for MoDiff’s performance gains, we applied it to Q-Diffusion with W8A8 on CIFAR-10, yielding FID 4.76 (worse than baseline). Our implementation also reproduces Q-Diffusion results without modulation, confirming consistency.\n\n**Relation To Broader Scientific Literature: Q1**\n\nWe exclude MixDQ due to a lack of public code. EfficientDM uses quantization-aware training, which differs from our post-training quantization focus.\n\n**Other Strengths And Weaknesses: Q1**\n\nLines 237–260 explain how errors accumulate in modulated computation and how our compensation method addresses this. Specifically, $\\hat a\\_t$ represents the activation computed in the previous timestep, while $a\\_t$ is the desired (unquantized) activation. Modulated computation reuses $A(\\hat a\\_t)$ instead of $A(a\\_t)$, so computing temporal difference based on $a_t$ in the next step will introduce errors. Our compensation technique corrects this by computing temporal difference based on $\\hat a_t$.\n\nLines 324–329 emphasize that compensation is more critical in practice than theory suggests. In practice, diffusion models suffer from significant error accumulation as quantization errors build up layer by layer. The severity of error accumulation highlights the necessity of applying error compensation.\n\n**Other Comments Or Suggestions: Q1**\n\nThanks for pointing out the missing right parenthesis. We will correct the typo in the revised version.\n\n**Questions For Authors: Q1**\n\nWe run tensor-wise quantization on CIFAR-10 with 20-step DDIM. MoDiff consistently improves low-bit quality.\n\n| W/A | Baseline | MoDiff |\n|:-:|--:|--:|\n| 8/8 | 6.93 | 6.90 |\n| 8/6 | 20.28 | 6.75 |\n| 8/4 | 297.21 | 22.12 |", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "The models and datasets used in this paper are too old. For example, this paper's latest model LDM [1] was released in 2022 and most of experiments are based on CIFAR 32$\\times$32. More advanced models (e.g., PixArt [2] or FLUX [3] ), metrics (e.g., Image Reward [4] or GenVal [5]), and datasets (e.g., DCI [6] or MJHQ [7]) should be involved.\n\n[1] Rombach R, Blattmann A, Lorenz D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 10684-10695.\n\n[2] Chen J, Yu J, Ge C, et al. Pixart-\n: Fast training of diffusion transformer for photorealistic text-to-image synthesis[J]. arXiv preprint arXiv:2310.00426, 2023.\n\n[3] https://github.com/black-forest-labs/flux\n\n[4] Xu J, Liu X, Wu Y, et al. Imagereward: Learning and evaluating human preferences for text-to-image generation[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[5] Ghosh D, Hajishirzi H, Schmidt L. Geneval: An object-focused framework for evaluating text-to-image alignment[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[6] Urbanek J, Bordes F, Astolfi P, et al. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 26700-26709.\n\n[7] Li D, Kamko A, Akhgari E, et al. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation[J]. arXiv preprint arXiv:2402.17245, 2024.", "theoretical_claims": "1. The assumption in Eq. (29) requires proper bit-width choices, which are constrained by the tensor shape as illustrated in Eq. (78). I suggest the authors discuss the feasible choices for different linear operations in diffusion.\n\n2. The inputs in Eq. (25) and Eq. (28) are not quantized. Does this mean that the authors use the full-precision denoising at $T$?", "experimental_designs_or_analyses": "1. The superior results in the paper are mainly based on LCQ (i.e., dynamic channel-wise activation quantization). I believe that this scenario can not obtain any speedup and is never presented in any CNN quantization works. Moreover, the bit-width settings in this paper are a little bit wired (activation << weight). All these need to be further discussed.\n\n2. w8a8 Q-Diffusion w/o MoDiff in some scenarios outperforms MoDiff, which needs to be further explained.\n\n3. Ablation studies for only employing standard modulate quantization vs. baseline are missing.\n\n4. Experiments related to Q-Diffusion may be not fair. For example, the implemental details in Sec. B are all inconsistent with Q-Diffusion.", "relation_to_broader_scientific_literature": "This paper lacks a lot of advanced baselines, e.g., MixDQ and EfficientDM.  It is better to compare and combine this work with them to show the contribution.\n\n[8] Zhao T, Ning X, Fang T, et al. Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 285-302.\n\n[9] He Y, Liu J, Wu W, et al. Efficientdm: Efficient quantization-aware fine-tuning of low-bit diffusion models[J]. arXiv preprint arXiv:2310.03270, 2023.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n The idea is novel and interesting.\n\nWeaknesses:\n\nSome parts of this paper are hard to read. For example, the expression in L237-260 and L324-329 is very confusing and difficult to understand.", "other_comments_or_suggestions": "Eq. (18) needs to be fixed.", "questions_for_authors": "This paper currently focuses on diffusion with hundreds of denoising steps during inference. However, few-step diffusion models are widely used. Can this work achieve performance improvement when being applied to these models? As shown in Eq. (30) and (31), the errors are the same w/ and w/o error compensation for a 2-step diffusion model.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The Modulated Diffusion (MoDiff) framework accelerates diffusion models through modulated quantization and error compensation, reducing activation precision to 3 bits without significant performance loss, as demonstrated on CIFAR-10, LSUN, Stable Diffusion, and DiT (ImageNet), with strong theoretical support. Despite concerns from reviewers NH3e, 32oq, and YCE9 about hardware efficiency and comparisons, the authors’ rebuttal addressed these with additional experiments (e.g., MixDQ, DPM-Solver) and clarified misunderstandings (e.g., norm folding, single-model use), though some practical applicability questions remain. Given the paper’s innovation, robust rebuttal, and partial unfairness in reviews (e.g., overlooked W4A4 results, YCE9’s score downgrade), I recommend Weak Accept with revisions to enhance comparisons and hardware discussion.
Decision: Accept (poster)