BiDM: Pushing the Limit of Quantization for Diffusion Models
Authors: Xingyu Zheng
Xianglong Liu
Yichen Bian
Xudong Ma
Yulun Zhang
Jiakai Wang
Jinyang Guo
Haotong Qin
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Model Binarization
Low-bit Quantization
Machine Learning
Primary area: diffusion_based_models
TL;DR: In this paper, we propose BiDM, pushing the quantization of the diffusion model to the limit with fully binarized weights and activations.

Abstract: Diffusion models (DMs) have been significantly developed and widely used in various applications due to their excellent generative qualities. However, the expensive computation and massive parameters of DMs hinder their practical use in resource-constrained scenarios. As one of the effective compression approaches, quantization allows DMs to achieve storage saving and inference acceleration by reducing bit-width while maintaining generation performance.  However, as the most extreme quantization form, 1-bit binarization causes the generation performance of DMs to face severe degradation or even collapse. This paper proposes a novel method, namely BiDM, for fully binarizing weights and activations of DMs, pushing quantization to the 1-bit limit. From a temporal perspective, we introduce the Timestep-friendly Binary Structure (TBS), which uses learnable activation binarizers and cross-timestep feature connections to address the highly timestep-correlated activation features of DMs. From a spatial perspective, we propose Space Patched Distillation (SPD) to address the difficulty of matching binary features during distillation, focusing on the spatial locality of image generation tasks and noise estimation networks. As the first work to fully binarize DMs, the W1A1 BiDM on the LDM-4 model for LSUN-Bedrooms 256$\times$256 achieves a remarkable FID of 22.74, significantly outperforming the current state-of-the-art general binarization methods with an FID of 59.44 and invalid generative samples, and achieves up to excellent 28.0 times storage and 52.7 times OPs savings.

BiDM: Pushing the Limit of Quantization for Diffusion Models
Xingyu Zheng', Xianglong Liu Yichen Bian', Xudong Ma', Yulun Zhang",
Jiakai , Jinyang Guo Haotong Qin Beihang University Shanghai Jiao Tong University ZZhngguancun Laboratory "ETH Zürich
(zhengringyu, xiliu, ixlpiiinnnnnuiluuuueeee fyichen. bian biw..wor., poookk..uuunn0000gmaaaaa com wangjk0zgclab.eo edu.. haotong qin0pbl.... ethz.ch
jayangggouu adu....du..a.......... cn
Abstract
Diffusion models (DMs) have been significantly developed and widely used in various applications due to their excellent generative qualities. However, the ex- pensive computation and massive parameters of DMs hinder their practical resource-constrained scenarios. As one of the effective compression approaches, quantization allows DMs achieve storage saving and inference acceleration by reducing bit-width while maintaining generation performance. However, as the most extreme quantization form, 1-bit binarization causes the generation perfor- mance of DMs to face severe degradation or even collapse. This paper proposes novel method, namely BiDM, for fully binarizing weights and activations of DMs, pushing quantization to the 1-bit limit. From a temporal perspective, we introduce the tairestep-friendf Binary Structure (TBS), which uses learnable ac- tivation binarizers and cross- cooss-timestep feature connections to address the highly timestep-correlated activation features of DMs. From a spatial perspective, we propose Space Patched Distillation (SPD) to address the difficulty of matching binary features during distillation, focusing on the spatial locality of image genera- tion tasks and noise estimation networks. As the first work to fully binarize DMs, the WIAI BiDM on the LDM-4 model for L.SU--Bedrue Bedrooms 256 achieves a remarkable FID of 22.74, significantly outperforming the current state-of-the-an general binarization methods with an FID of 59.44 .44 and invalid generative samples, and achieves up excellent 28.0x storage and 52.7x OPs savings.
Introduction
Diffusion models (DMs) [19, 50, 44, 76], type of generative visual model [66, 59, 68]. have garnered impressive attention and applications in various fields, such image [57, 58], speech [42, 45,24) 24], and video [40, 18], because of their high-quality and diverse generative capabilities. The diffusion model can generate data from random noise through up to 1000 denoising steps [19]. Although some accelerated sampling methods effectively reduce the number of steps required for generating tasks [56, 31], the expensive floating-poie computation of each timestep still limits its wide application on resource-constrainee scenarios. Therefore, compression of the diffusion model becomes crucial step for its broader application, and existing compression methods mainly include quantization [30, 54, 47], distillation [53, 36, 41, 73, 11], pruning [7, 12, 14, 13], etc. These compression approaches aim to reduce storage and computation while preserving accuracy.
Quantization considered highly effective model compression technique [70, 9, 64, 21, 10], which quantizes the weights and/or activations to low-bit integers or binaries for compact storage and
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Timestep-friende Binory Structure
Time [+1
Timet
Time b1
Home!
(Iernable)
Monet
(Ieomablee
connaction
CRASE Sintetee caneeclon
Sten0)
Space Patched Distillation
peirs
patch
palch
para
Cun
Eup
Figure I: Overview of BiDM with Timestep-friendly Binary Structure, which improves DM architec- ture temporally, and Space Patched Distillation, which enhances DM optimization spatially.
efficient computation in inference. Some existing works thus apply quantization to compress DMs. aiming to compress and accelerate them while maintaining the quality of generation. Among them, 1-bit quantization, namely binarization, can achieve maximum storage savings for models and has performed well in discriminative models such as CNNs [33, 67, 65]. Furthermore, when both weights and activations are quantized to 1-bit, e.g., fully binarized, efficient bitwise operations such as XNOR and bitcount can replace matrix multiplication, achieving the most efficient acceleration [74].
Some existing works have attempted to quantize DM to 1-bit [77], but their exploration mainly focuses on the weights, which are still far from full binarization. In fact, for generative models like DM, the impact of fully binarizing weights and activations catastrophic: a) As generative models, DMs have rich intermediate representations closely related to timesteps and highly dynamic activation ranges, which are both very limited information when binarized weights and activations are used; b) Generative models like DMs are typically required to output complete images, but the highly discrete parameter and feature space make it particularly difficult for binarized DMs match the ground truth during training. The limited representational capacity, which is hard to match with timesteps dynamically, and the optimization difficulty of generative tasks in discrete space make difficult for the binarized DM to converge or even collapse during the optimization process.
We propose BiDM push diffusion models towards extreme compression and acceleration through complete binarization of weights and activations. It is designed to address the unique properties of DMs' activation features, model structure, and the demands of generative tasks, overcoming the difficulties associated with complete binarization. BiDM consists of two novel techniques: From temporal perspective, we observe that the activation properties of DMs are highly correlated with timesteps. We introduce the Timestep-friendie Binary Structure (TBS), which uses learnable activation binary quantizers match the highly dynamic activation ranges of DMs and designs feature connections across timesteps to leverage the similarity of features between adjacent timesteps, thereby enhancing the representation capacity of the binary model. From a spatial perspective, we note the spatial locality of DMs in generative tasks and the cnrvolution-basede U-Net structure. We propose Space Patched Distillation (SPD), which introduces a full- full-precision model as a supervisor and uses attention-gus gunddd imitation on divided patches to focus on local features, better guiding the optimization direction of the binary diffusion model.
Extensive experiments show that compared to existing SOTA fully binarized methods, BiDM signifi- cantly improves accuracy while maintaining the same inference efficiency, surpassing all existing baselines across various evaluation metrics. Specifically, in pixel space diffusion models. BiDM is the only method that raises the IS 5.18. close to the level of full full-precision models and 0.95 higher than the best baseline method. In LDM, BiDM reduces the FID on -eUN-oedoooms from the SOTA method's 59.44 .4 to an impressive 22.74, while fully benefiting from 28.0x storage and 52.7x OPs savings. As the first fully binarized method for diffusion models, numerous generated samples also demonstrate that BiDM is currently the only method capable of producing acceptable images with fully binarized DMs, enabling the efficient application of DMs in low-resource scenarios.

---

2 Related Work
Diffusion models (DMs) have demonstrated excellent generative capabilities across various tasks [19, 57, 58, 43, 42, 45, 24]. However, their large-scale model architectures and the high computational costs required for multi-step inference limit their practical applications. To address this, methods for accelerating the process at the timestep level have been widely proposed, including sampling acceleration that does not require retraining [56, 31, 34, 35] and distillation methods [53, 36, 41J. A recent method called DeepCache [38] caches high- -igh-dimensional features avoid a lot of redundant computations and compatible with typical sampling acceleration methods. However, these methods cannot overcome the memory bottlenecks and efficiency limits during single-step inference.
Quantization is a widely validated compression technique that compresses weights and activations from the usual 32 bits to 1-8 bits to achieve compression and acceleration [6, 78, 37, 75J. Conse- quently, quantization is being studied for application in diffusion models [15, 4]. These methods generally consider the unique timestep structure and spatial architecture of diffusion models, but due to the significant difficulty of quantizing generative models, most post-training quantization (PTQ) methods can only quantize models to 4 bits or more [29, 54, 22], while more accurate quantization- aware training (QAT) methods face severe performance bottlenecks below bits [30, 55].
Binarization, the most extreme form of quantization, typically expresses weights and activations as #1, allowing the model to achieve maximum compression and acceleration [60, 62]. In computer vision, binarization work has mainly focused on discriminative models like CNNs [49, 33, 46, 48] or ViTs [28, 16], with limited work on generative models. While ResNet VAE and Flow++ [1] have achieved complete binarization for VAEs [26], they do not offer generative performance comparable to current advanced models. Binary Latent Diffusion [61] binarized the latent space LDMs [26] but did not improve the model's spatial footprint or inference efficiency. The latest work, BinaryDM [50], quantized DMs to nearly W1A4, but it did not address activation quantization, leaving room for achieving full binarization and acceleration of DMs.
3 Method
3.1 Binarized Diffusion Model Baseline
Diffusion models. Given a data distribution æ0 ~ g(mr)), the forward process generates a se- quence of random variables "",,,..,,r with transition kernel g(ar, (2;-1)). usually Gaussian perturbation, which can be expressed as
T
q (0)..... ET 1o) = II aax 1z-1), q (w,, 11-1)== =N (wt:v1 B,20.-1 (1)
1=1
where B, E (0, 1) is noise schedule. Gaussian transition kernel allows us to marginalize the joint 3g and ā I at,
distribution, so with a₁ sampling gaussian vector I) and applying the transformation H; Vārzo + vI a,e.
easily obtain a sample of by
The reverse process aims to generate samples by removing noise, approximating the unavailable conditional distribution (xt-1 ax) with a learnable transition kernel po (xx-1 æz)). which can be expressed as
po (x,-1 x2) V (x:-1; (x (),,6),88 B,1).
(2)
The mean igg (xe, 1) and variance 3, could be derived using the reparameterization tricks in [19]:
1 042 co (Itz, (x1,,t) t)): 3. 1 a
ig (z),
Vat
V1
-1-a ae
where Eg is function approximation with the learnable parameter 0, which predicts e given æ;.
For the training of DMs, simplified variant of the variational lower bound is usually applied as the loss function for better sample quality, which can be expressed as
Cum = (tw ex |lee - co (æ 1)11
3

---

Cosin Similarity
0.96
0.94
0.90
Time Step
Time Step
(a) Activation Range
(b) Activation Features
Figure 2: (a) The activation range of the 4th convolutional layer of the full-precision DDIM model on CIFAR-10 varies with the denoising timesteps. (b) The output features are similar at each step of the full-precision LDM-4 model on S-r-oodooo compared to the previous step.
U-Net [51], due to ability to fuse low-level and high-dimensional features, has become the main- stream backhone of Diffusion. The input-output blocks of U-Net can be represented as {Dm m== and { m== J where blocks corresponding to smaller m are more low-level. Skip connections propagate low-level information from Dm (-) U-(.). so the input received by Um is expressed as:
Concat(Dm (+), Um+1(-))
The quantization compresses and accelerates the noise estimation model by discretiz- ing weights and activations to low bit-width. the baseline of the binarized diffusion model, the weights binarized 1-bit [49, 20]:
Binarization.
w 2 0,
th = sign(w) = 1-a, otherwise,
where sign function confine +1 or -1 with O thresholds. is floating-pon point scalar, which is initialized as [wll (n denotes the number of weights) and learnable during training following [49, 33].
n Meanwhile, activations are typically quantized by naive BNN quantizers [23, 32]:
1, if a 2 0, -1, otherwise.
a sign(a)
When both weights and activations are quantized to 1-bit, the computations of the denoising model be replaced by XNOR and bitcount operators, achieving significant compression and acceleration.
3.2 Timestep-friendly Binary Structure
Before delving into the detailed description of the proposed method, we summarize our observation on the properties of DMs:
Observation 1.
The activation range varies significantly across long-ters timesteps, but the activation features are similar in short-term neighbouring timesteps.
Previous works, such TDQ [55] and Q-DM [30], have commonly demonstrated that the activation distribution of DMs largely depends on denoising process, manifesting as similarities between adjacent timesteps while difference between distant ones, shown in Figure 2(a). Therefore, applying a fixed scaling factor to activations across all timesteps can cause significant distortion the activation range. Beyond the distribution range, Deepcache [38] highlights the substantial temporal consistency high-dimensions features across consecutive timesteps, as shown in Figure 2(b).
These phenomena prompt to reexamine existing binary structures. Binaryization, especially the full binaryization weights and activations, results greater loss activation range and precision compared to low-bit quantizations like 4-bit [50]. This makes it more challenging to generate rich activation features. Such deficiencies in activation range and output features significantly harm representation-rich generative models like DMs. Therefore, adopting binary quantizers with more

---

flexible activation ranges for DMs, and enhancing the model's 's overall expressive power by leveraging its feature outputs, are crucial strategies for improving its generative capability after full binaryization.
We first focus on the differences between various timesteps over the long term. Most existing activation quantizers, such as BNN [23] and Bi-Real [32], as shown in Eq. (7), directly quantize activations to [+1,-1}, This approach significantly disrupts activation features and negatively impacts the expressive power of generative models. Some improved activation binary quantizers, such as XNOR+ [2], adopt a trainable scale factor k:
th K sign(a) =
K, i 2 0.
(-K, otherwise,
where the form of K could be either a vector or the product of multiple vectors, but it remains a constant value during inference. Although this approach partially restores the feature expression of activations, does not align well with diffusion models that are highly correlated with timesteps and may still lead to significant performance loss.
We turn our attention the original XNOR, which employs dynamically computed means to construct the activation binary quantizer. Its operation for 2D convolution can be expressed as:
I W (I sign( (W)) (Ka) (Ii @ sign (W)) @ ka))
where I E Rexemxhn W E Rexwxh A 1 a = + || RixIxuxh represents a 2D filter, where Vij kij w * and & indicate convolution with and without multiplication, respectively. This approach naturally preserves the range of activation features and dynamically adapts with the input range across different timesteps. However, due to the rich expression of DM features, local activations exhibit inconsistency in range before and after passing through modules, indicating that the predetermined value of k does not effectively restore the activation representation.
1
Therefore, we make i adjustable and allow to be learned during training to adaptively match the changes in the range of activations hefore and after. The gradient calculation process of our learnable tiny convolution & can be expressed as follows:
dc ak
OC
0(A ka)
(sign(I) @ sign( (W).
(10)
a( W)
Ok
Notably, making & learnable does not add any extra inference burden. The computational cost remains unchanged, allowing for efficient binary operations.
On the other hand, focus the similarity between adjacent timesteps. Deepcache directly extracts high-dimensional features as a cache to skip a large amount of deep computation in U-Net, achieving significant inference acceleration. This process is expressed as:
Feache f- U (+), ())" Ftatee))
(11)
However, this approach does not apply to binarized diffusion models, the information content of each output from a binary network is very limited. For binary diffusion models, which inherently achieve significant compression and acceleration but have limited expressive power, we anticipate that the similarity of features between adjacent timesteps will enhance binary representation, thereby compensating for the representation challenges.
We construct a cross-timestep information enhancement connection to enrich the expression at the current timestep using features from the previous step. This process can be expressed as:
Concat((D (.), (1 ( U (.) + 1 (.)),
(12)
where
4-1 mm+1 is a learnable scaling factor. As shown in Figure 2(b), the similarity of high- -igh-dimensional features varies across different blocks and timesteps DMs. Therefore, we set multiple independent values to allow the model to adaptively learn more effectively during training.
In summary, -rmeste Binary Structure (TBS) includes learnable tiny convolution applied to scaling factors after averaging the inputs and connections across timesteps. Their combined effect adapts to the changes in the activation range of diffusion models over long-range timesteps and leverages the similarity of high-dimensional features between adjacent timesteps to enhance information representation.
5

---

From the perspective of eπror reduc- tion, a visualization of TBS is shown in Figure 3. First, we abstract the out- put of the binary DM under the base- line method as vector B The mis- match in scaling factors creates a sig- nificant difference in length between it and the output vector 1=-1 of the fulll- precision model. Using our proposed scaling factors and learnable tiny con- volutions, B is expanded to L'-1 T is closer to pt but there is still directional difference from the full- precision model. The cross-timestep connection further incorporates the out- puts f of the previous timestep, D³, and L'. The high-dimensional feature similarity between adjacent timesteps means the gap between F*-1 and Fr is relatively small, facilitating the combination of L'-1 and L'. Finally, we obtain the binarized DM's output with TBS applied as 7 (1 a) - + a L', closest to the output F--1 of the full- full-precision model. The learnable tiny convolution k in TBS allows scaling factors to adapt more flexibly to the representation of DM, while connections across timesteps enable the binarized DM to use the previous step's s output information for appropriate information compensation.
Ft
F
B
BC-1
Figure 3: An illustration of TBS. Since the feature space high-dimensional, we illustrate it using schematic diagrams.
3.3 Space Patched Distilation
Due to the nature of generative models, the optimization process of diffusion models exhibits different characteristics from past discriminative models:
Observation 2. Conventional distillation struggles to guide fully binarized DMs align with full-precision DMs, while the features of DM exhibit locality in space during the generation task.
In previous practices, adding distillation loss during the training of quantized models has been common approach. As the numerical space of binary models is limited, directly optimizing them using naive loss leads to difficulties in adjusting gradient update directions and makes learning challenging. Therefore, adding distillation loss to intermediate features can better guide the model's s local and global optimization process.
However, as a generative model, the highly rich feature representation of DMs makes extremely difficult for binary models to finely mimic full-precision models. Although the L2 loss used in the original DM training aligns with the Gaussian noise in the diffusion process, it is not suitable for the distillation matching of intermediate features. During regular distillation, the commonly used L2 loss tends to prioritize optimizing pixels with larger discrepancies, leading to a more uniform and smooth optimization result. This global constraint learning process is challenging for binary models aimed at image generation, as their limited representation capacity makes it difficult for fine fine-graing grained distillation imitation to directly adjust them to fully match the direction of full precisren models.
At the same time, we note that DMs using U-Net as a backbone naturally exhibit spatial locality due to their convolution-based structure and generative task requirements. This is different from past discriminative models, where tasks like classification only require overall feature extraction without low-level requirements, making traditional distillation methods unsuitable for DMs with spatial locality in generative tasks. Additionally, most existing DM distillation methods focus on reducing the number of timesteps and do not address the spatial locality of features required for image generation tasks.
Therefore, given the difficulty in optimizing binary DMs with existing loss functions and the spatial locality of DMs, we propose Space Patched Distillation (SPD). Specifically, we designed a new loss function that partitions features into patches before distillation and then calculates spatial attention- guided loss patch by patch. While conventional 1.2 loss makes it difficult for binary DMs to achieve direct matching, leading to optimization challenges, the attention mechanism allows the distillation

---

FP32
Diff
SPD
Figure 4: Visualization of the last TimeStepBlock's output of the LDM model on LSUN-bedroom dataset. FP32 denotes the full-precision model's output F Diff denotes the difference between the Ours denotes the attention-
output of the full- mull-precision model and the binarized one || F²
guided SPD.
optimization to focus more on critical parts. However, this is still challenging for fully binarized DMs because the highly discrete binary outputs have limited information, making it difficult for the model to capture global information. Therefore, we leverage the spatial locality of DMs by dividing intermediate features into multiple patches and independently calculating spatial attention guioonddgdddd loss for each patch, allowing the binary model to better utilize local information during optimization.
SPD first divides the intermediate features F and FfP E Fbxexxx, output by a block of the binary DM and the full plleciseeiinn DM respectively, into p² patches:
P
Flp
P.j F bnwttw/pputtiii
(13)
Then, attention-guin guided loss is calculated for each patch separately:
A
p.h ph Ar, = Phy Pbu
(14)
After regularization, the losses at corresponding positions are calculated and summed up:
SPD
||4%|l
||
(15)
1=
j=0
where ||
||2 denotes the L2 function. Finally, the total training loss L is computed as:
2d+1
L LDM
2d
Σ
SPD
(16)
where d denotes the number blocks during the upsampling process or downsampling process, resulting in total of 2d + intermediate features, including the middle block. À is a hyperparameter coefficient to balance the loss terms, defaulting set
We visualize the intermediate features and attention guidded SPD mentioned above. As Figure 4 shown, our SPD allows the model to pay more attention local information in each patch.
4
Experiment
We conduct experiments on various datasets, including CIFAR-10 32 x 32 [27], LSUN-
256 256 [72], LSUN-Churcher 256 256 [72] and FFHQ 256 256 [25] over pixel space diffusion models [19] and latent space diffusion models [50]. The evaluation metrics used in our study encompass Inception Score (IS), Fréchet Inception Distance (FID) [17]. Sliding Fréchet Inception Distance (sFID) [52], Precision and Recall. To date, there has been по research that compresses diffusion models such an extreme extent. Therefore, we use classical binarization algorithms [2, 78, 33, 49], the recent SOTA general binarization algorithms [62], and quantization methods suited to generative models [15, 63] as baselines. We extract the outputs EimestepEnbeeBBlokk from the DM to serve as the operating target for our TBS and SPD. And we employ the same shortcut connections in convolutional layers as those used in ReActNet(33) Detailed experiment settings are presented in the Appendix A.

---

4.1
Main Results
Pixel Space Diffusion Models. We first conduct experiments on the CIFAR-10 10 x dataset. As the results presented in Table 1, WIA1 binarization of DM using baseline methods results in substantial degradation. However, BiDM demonstrated significant improvements across all metries, achieving unprecedented restoration of image quality. Specifically, BiDM achieved remarkable enhancements from 4.23 to 5.18 in the IS metric, and reduced 27. 27.9% in the FID metric.
Table Binarization results for DDIM on CIFAR- 10 datasets with 100 steps.
Model
Dataset
Method
#Bits
1St
FID
sFID
Precision
FP
32/32
8.90
5.54
4.46
67.92
XNOR++2]
1/1
2.23
251.14
60.85
44.98
DoReFa[78]
1/1
1.43
397.60
139.97
0.17
CIFAR-10
DDIM
[eActNet[33]
1/1
3.35
231.55
119.80
18.37
x 32
ReSTE[62]
1.26
394.29
125.84
0.18
XNOR[49 [49]
1/1
4.23
113.36
27.67
46.96
BiDM
1/1
5.18
81.65
25.68
52.92
Latent Space Diffusion Models. Our LDM experiments encompass the evaluation of LDM-4 on LSUNNBedroome 256 x 256 and FFHQ 256 256 datasets, along with the assessment of LDM-8 on the LSUN-Churches 256 x 256 dataset. The experiments utilized the DDIM sampler with 200 presented in Table 2. Across these three datasets, method achieved significant improvements over the best baseline methods. In comparison to other binarization algorithms, BiDM outperformed across all metrics. On the LSUN-Bedrooms, L.SUN-Churche and FFHQ datasets, the FID metric of BiDM decreased by 61.7%, 30.7%, and 51.4%, respectively, compared to the best results among the baselines.
steps, and the detailed outcomes
In contrast to XNOR++, its adoption of fixed activation scaling factors in the denoising process results in very limited dynamic range for its activations, making it difficult to match the highly flexible generative representations of DMs. BiDM addressed this challenge by making the tiny convolution & learnable, which acts the dynamically computed scaling factors. This optimization led to substantial improvements exceeding an order of magnitude across all metrics. On the LSUN- Bedrooms and LSUN- C-UN-hhhrcees datasets, the FID metric decreased from 319.66 22.74 and from 292.48 to 29.70, respectively. Additionally, compared to the SOTA binarization method ReSTE, BiDM achieved significant enhancements across multiple metrics, particularly demonstrating notable improvements on the LSUN- -edrrooms dataset. We have supplemented our work with BBCU, binarization method more akin generative models like DMs rather than discriminative models. Experimental results indicate that even as a binarization strategy for generative models. BBCU faces significant breakdowns when applied to DMs, as FID dropped dramatically 236.07 on L.SUN-Bedrooms. As work targeting QAT for DM, EfficientDM is indeed suitable comparison, especially since it designs TALSQ address the variation activation range. The results show that EfficientIM struggles to adapt to the extreme scenario of W1A1, 11, and this may be due to its quantizer having difficulty adapting to binarized DM, and using QALoRA for weight updates might yield suboptimal results compared to full-parameter QAT.
As we mentioned in the TBS section of our manuscript, most existing binarization methods struggle to handle the wide activation range and flexible expression of DMs, further highlighting the necessity of TBS. Their optimization strategies may also not be tailored for the image generation tasks performed by DM, which means they only achieve conventional but suboptimal optimization.
4.2 Ablation Study
We perform comprehensive ablation studies for LDM-4 on the evaluate the effectiveness of each proposed component in BiDM. We evaluate the effectiveness of our proposed SPD and TBS, and the results are presented in Table Upon separately applying our SPD or TBS methods to LDM, we observed significant improvements compared to the original performance. When the TBS method was incorporated, FID and sFID dropped sharply from 106.62 and 56.61 to 35.23 and 25.13 13, respectively. Similarly, when the SPD method was added, FID and sFID decreased
256 x 256 dataset to

---

Table 2: Quantization results for LDM on LSUN- LSUN-Churches and FFHQ datasets.
Model
Dataset
Method
#Bits
FID4
sFID4
Precision
Recall
FP
32/32
2.99
7.08
65.02
47.54
XNOR+
1/1
319.66
184.75
0.00
0.00
BBCU
1/1
236.07
89.66
0.59
5.66
EfficientDM
1/1
194.45
113.24
0.99
9.20
LSUN-Bec NUN-Bedrooms 256 x 256
LDM-4
DoReFa
1/1
188.30
89.28
0.86
0.18
ReActNet
1/1
154.74
61.50
4.63
9.30
ReSTE
1/1
59.44
42.16
12.06
2.92
XNOR
1/1
106.62
56.81
6.82
5.22
BiDM
1/1
22.74
17.91
33.54
19.90
32/32
4.36
16.00
74.64
48.98
XNOR+
1/1
292.48
668.33
0.02
0.00
DoReFa
1/1
162.06
95.37
7.85
0.74
LSUN-Churches
LDM-8
ReActNet
1/1
56.39
54.68
45.13
2.06
256 256
ReSTE
1/1
47.88
52.44
51.98
3.34
XNOR
1/1
42.87
49.24
51.53
4.28
BiDM
1/1
29.70
45.14
55.75
14.80
FP
32/32
4.87
6.96
74.73
50.57
XNOR++
1/1
379.49
320.64
0.00
0.00
DoReFa
1/1
214.06
177.63
2.09
0.00
FFHQ 256 256
LDM-4
ReActNet
1/1
147.88
141.31
3.36
0.69
ReSTE
1/1
144.37
97.43
4.03
0.03
XNOR
1/1
89.37
54.04
31.31
4.11
BiDM
1/1
43.42
32.35
49.44
13.96
LSUN-Bedroomd Bedrooms 256x256
FFHQ 256x256
XNOR+
ReActNat
ReSTE
XNOR
BIDM
XNOR+
RoAcNNt
RoSTE
XNOR
BiDM
Figure 5: Visualization of samples generated by the WIAI baseline and our BiDM. BiDM is the first fully binarized DM method capable of generating viewable images, significantly surpassing advanced binarization methods.
significantly from 106.62 and 56.61 to 40.62 and 31.61, respectively. Other metrics also exhibited substantial improvements. This demonstrates the effectiveness of our approach in continuously approximating the binarized model features to full-precisse precision features during training by introducing learnable factor af and incorporating connections between adjacent time steps. Furthermore, when we combined our two methods and applied them to LDM, we observed an additional improvement compared the individual application of each method. This further substantiates that performing distillation between full-precision and binarized models at the patch level can significantly enhance performance the binarized model. We also conducted additional ablation experiments, and the results presented in the appendix B.

---

Table 3: Ablation result of each proposed component.
Method
#Bits
FID
sFID
Prec.1
Recallt
Vanilla
1/1
106.62
56.81
6.82
5.22
+TBS
1/1
35.23
25.13
26.38
14.32
+SPD
1/1
40.62
31.61
23.87
11.18
BiDM
1/1
22.74
17.91
33.54
19.90
4.3
Efficiency Analysis
Inference Efficiency Analysis. We conducted an analysis of the diffusion model's inference efficiency under complete binarization. During inference, BiDM requires only a very small number of additional floating-point additions for the connections across timesteps compared to the classic binarization work XNOR-Net, and there are differences in the majority of calculations, such as convolutions. Performing a floating-pon point convolution with a depth of for scaling factors requires only small amount of computation, and the overhead for averaging matrix presented in Table 4 reveal that BiDM, while achieving the same 28.0x memory efficiency and 52.7 computational savings as the XNOR baseline, demonstrates significantly superior image generation capabilities, with the FID decreased from 106.62 to 22.74. See Appendix B for more details.
also minimal. The findings
Table 4: Inference efficiency of our proposed BiDM of LDM-4 on LSUN-Bedrooms.
Method
#Bits
Size(MB)
BOPs(x109)
(LOPs(x10)
OPs(x10')
FID,
FP
32/32
1045.4
96.00
96.00
2.99
XNOR
1/1
37.3
92.1
0.38
1.82
106.62
BiDM
1/1
37.3
92.1
0.38
1.82
22.74
Training Efficiency Analysis. We also explored the training efficiency of BiDM, as the overhead required for the QAT of binarized DMs cannot be overlooked. Theoretical analysis and experimental results show that BiDM achieved significantly better generative results than baseline methods under the same training cost, demonstrating that it not only has higher upper limit of generative capability also relatively efficient terms of generative performance. See Appendix for details.
Limitations. The techniques of BiDM increase the training time of DMs compared with the original process, and future efforts may thus focus on the efficient quantization process of DMs.
Conclusion.
In this paper, we present BiDM, novel fully binarized method that pushes the compression of observations activations different timesteps and
diffusion models the limit. Based the characteristics of image generation tasks - we propose the cenestep-friendle Binary Structure (TBS) and Space Patched Distillation (SPD) from temporal and spatial perspectives, respectively. These methods address the severe limitations in representation capacity and the challenges of highly discrete spatial optimization in full binarization. As the first fully binarized diffusion model, BiDM demonstrates significantly better generative performance than the SOTA general binarization methods across multiple models and datasets. On
BiDM achieves an FID of 22.74, greatly surpassing the SOTA method with an FID of 59, 94,44 making it the only method capable of generating visually acceptable samples while achieving up to 28.0x storage savings and 52.7 x OPs savings.
Acknowledgments and Disclosure of Funding
This work was supported by the Beijing Municipal Science and Technology Project (No. 7231 100010323002), the National Natural Science Foundation of China (Nos. 62306025, 92367204), and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Thomas Bird, Friso H Kingma, and David Barber. Reducing the computational cost of deep generative models with binary neural networks. arXiv preprint arXiv:2010.13476, 2020.
[2] Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks.
arXiv preprint arXiv:1909.13860 pages 1-12, 2019.
[3] Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed- mixed-precision neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2349-2358 2020.
[4] Zheng Chen, Haotong Qin, Yong Guo, Xiongfei Su, Xin Yuan, Linghe Kong, and Yulun Zhang. Binarized diffusion model for image super- super-resolution. arXiv preprint arXiv:2406.05722, 2024.
[5] Matthieu Courbariaux, Itay Hubara, Daniel Soudry. Ran El- D--aa,,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1, Xrxiv preprint 1roiv:160..02882 pages 1-11, 2016.
[6] Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. In International Conference on Learning Representations, pages 1-12, 2019.
[7] Gongfan Fang. Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. arXiv preprint arXiv:2305. 10924, 2023.
[8] Lukas Geiger and Plumerai Team. Larq: An open-source library for training binarized neural networks. Journal of Open Source Software, 5(45):1746, 2020.
[9] Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. In Low-Power Com- puter Vision, pages 291-326. Chapman and Hall/CRC, 2022.
[10] Ruihao Gong, Yifu Ding, Zining Wang, Chengtao Lv, Xingyu Zheng, Jinyang Du, Haotong Qin, Jinyang Guo, Michele Magno, and Xianglong Liu. survey of low-bit large language models: Basics, systems, and algorithms. arXiv preprini arXiv:2409. 16694, 2024.
[11] Guangyu Guo, Longfei Han, Le Wang. Dingwen Zhang, and Junwei Han. Semantic- knowledge distillation with parameter-fie free feature uniformization, Visual Intelligence, 1(1):6, 2023.
[12] Jinyang Guo, Jiaheng Liu, and Dong Xu. Jointpruning: Pruning networks along multiple dimensions for efficient point cloud processing. IEEE Transactions on Circuits and Systems for Video Technology, 32(6):3659-366 3672, 2021.
[13] Jinyang Guo, Jiaheng Liu, and Dong Xu. 3d-pruning: A model compression framework for efficient 3d action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):77178872) 2022.
[14] Jinyang Guo, Dong Xu, and Wanli Ouyang. Multidimensional pruning and its extension: A unified framework for model compression. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[15] Yefei He, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Efficientdm: Efficient quantization-awee -ware fine-tuning of low-bit diffusion models. arXiv preprint 231.022310003270. 2023.
[16] Yefci He. Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Bivit: Extremely compressed binary vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5651-5663, 2023.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

---

[18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:221000230:3 2022.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840- 01884066611,, 2020.
[20] Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402. 04291, 2024.
[21] Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lx, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized Ilama3 models? an empirical study. arXiv preprint arXiv:2404. 14047, 2024.
[22] Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. Tfmqq-dm: Temporal feature maintenance quantization for diffusion models. arXiv preprint arXiv:231 166503, 2023,
[23] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in Neural Information Processing Systems, 29:1-9. 9, 2016.
[24] Myconghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint 1XXv:2104.01409 2021.
[25] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 4401-4410, 2019.
[26] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. pages 1-60, 2009.
[28] Phuoc-Hoan Charles Le and Xinlin Li. Binaryvit: pushing binary vision transformers towards convolutional models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[29] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17535-17545 2023.
[30] Yanjing Li, Sheng Xu, Xianbin Cao, Xiao Sun, and Baochang Zhang. Q-dm: An efficient low- bit quantized diffusion model. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[31] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778. 2022.
[32] Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang, Wei Liu, and Kwang- -ing Cheng. Bi-real net: Binarizing deep network towards real-network performance. International Journal of Computer Vision, 128:202-219, 2020.
[33] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Proceedings of the European Conference on Computer Vision, pages 143-159. Springer, 2020.
[34] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dom-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221..01095 2022.
12

---

[36] Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. arXiv preprint 042.222304.02262, 2023.
[37] Chengtao Lv, Hong Chen, Jinyang Guo, Yifu Ding, and Xianglong Liu. Piq4sam: Post-training quantization for segment anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15941-15951, 2024.
[38] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. arXiv preprint arXiv:2312.00085: 2023.
[39] Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. arXiv preprint arXiv:2003. 15535, 2020.
[40] Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9117-9125, 2023.
[41] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[42] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffuuson models. arXiv preprint drXIv:2103.16091, 2021.
[43] Chenhao Niu, Yang Song. Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference Artificial Intelligence and Statistics, pages 4474-4484. PMLR, 2020.
[44] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[45] Vadim Popov, Ivan Vovk. Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599-8608. PMLR, 2021.
[46] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2250-2259, 2020.
[47] Haotong Qin, Xudong Ma. Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xi- anglong Liu, and Michele Magno. Accurate pra-finetuning quantization of Ilms via information retention. arXiv preprint arXiv:2402.05444, 2024.
[48] Haotong Qin, Xiangguo Zhang, Ruihao Gong, Yifu Ding, Yi Xu, and Xianglong Liu. Distribution-sensidv information retention for accurate binary neural network. International Journal of Computer Vision, 131()):26-4,,200 2023.
[49] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision, pages 525-542. Springer, 2016.
[50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention- 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedins, part III 18, pages 234-241. Springer, 2015.
[52] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[53] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0053 2022.
13

---

[54] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pages 1972-1981, 2023.
[55] Junhyuk So, Jungwon Lee, Daehyun Ahn, Hyungjun Kim, and Eunhyeok Park. Temporal dynamic quantization for diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[56] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[57] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[58] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[59] Jia-Mu Sun, Tong Wu, and Lin Gao. Recent advances in implicit representation- based 3d shape generation. Visual Intelligence, 2(1):9, 2024.
[60] Peisong Wang,
Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. Sparsity-inducing binarized neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12192-12199, 2020.
[61] Ze Wang, Jiang Wang. Zicheng Liu, and Qiang Qiu. Binary latent diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22576-22585, 2023.
[62] Xiao-Ming Wu, Dian Zheng, Zuhao Liu, and Wei-Shi Zheng. Estimator meets equilibrium perspective: A rectified straight through estimator for binary neural networks training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17055- 17064, 2023.
[63] Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, and Luc Van Gool. Basic binary convolution unit for binarized image restoration network. arXiv preprint arXiv:2210.00400 2022.
[64] Yisong Xiao, Aishan Liu, Tianyuan Zhang, Haotong Qin, Jinyang Guo, and Xianglong Liu. Robustmq: benchmarking robustness of quantized models. Visual Intelligence, 1(1):30, 2023.
[65] Yixing Xu, Kai Han, Chang Xu, Yehui Tang. Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. Advances Neural Information Processing Systems, 34:25553-2555 2021.
[66] Zhekai Xu, Haohong Shang, Shaoze Yang, Ruiqi Xu, Yichao Yan, Yixuan Li, Jiawei Huang, Howard C Yang, and Jianjun Zhou. Hierarchical painter: Chinese landscape painting restoration with fine- fine-grained styles. Visual Intelligence, 1(1):19, 2023.
[67] Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5198-5208, 2021.
[68] Yichao Yan, Zanwei Zhou, Zi Wang, Jingnan Gao, and Xiaokang Yang. Dialoguenerf: Towards realistic avatar face-to-face conversation video generation. Visual Intelligence, 2(1):24, 2024.
[69] Jiaming Yang, Chenwei Tang, Caiyang Yu, and Jiancheng Lv. Gwq: Group-wise quantization framework for neural networks. In Asian Conference on Machine Learning, pages 1526-1541. PMLR, 2024.
[70] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7308-7316, 2019.
14

---

[71] Kai-Lang Yao and Wu-Jun Li. Full-precision free binary graph neural networks. 2021.
[72] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03360,335, 2015.
[73] Zelong Zeng, Fan Yang. Hong Liu, and Shin ichi Satoh. Improving deep metric learning via self-distillation and online batch diffusion process. Visual Intelligence, 2(1):18, 2024.
[74] Jianhao Zhang, Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. Dabnn: super fast inference framework for binary neural networks on arm devices. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2272-2275, 2019.
[75] Yulun Zhang, Haotong Qin, Zixiang Zhao, Xianglong Liu, Martin Danelljan, and Fisher Yu. Flexible residual binarization for image super-resolution. In Forty-first International Conference on Machine Learning.
[76] Wenliang Zhao, Haolin Wang. Jie Zhou, and Jiwen Lu. De-solver: Improving predictor-correctt diffusion sampler via dynamic compensation. arXiv preprint arXiv:2409.03755. 2024.
[77] Xingyu Zheng, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojie Hao, Jiakai Wang. Zixiang Zhao, Jinyang Guo, and Xianglong Liu. Binarydm: Towards accurate binarization of diffusion model. arXiv preprini arXiv:2404.05666 2024.
[78] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXiv:1606.06/ pages 1-13, 2016.
15

---

A Experiment Settings
We adopt several classic binarization algorithms, including XNOR [49]. XNOR++ [2], DoReFa [78]. and ReActNet [33], along with the SOTA binarization method, ReSTE [62] as baselines. Addi- tionally, we also include the quantization methods designed for generative models, BBCU [63] and EfficientDM [15]. We extract the output features of TinestepEibeeBlocck from the DM to serve as the targets of TBS and SPD operations. For the CIFAR-10 [27] dataset, We add TBS connections to the outputs of the last 2 timestep embedding blocks and set Ciivit to 0.3. The À on CIFAR- 10 is set to 3e-2. For the connections to the outputs of the last 8 timestep embedding blocks and also set Oinit to 0.3. The À on these three datasets is set to le-2.
[72], L.SUN-Churche [72] and FFHQ [25] datasets, We add TBS
Our quantization-avae parameters and latent weights are trained simultaneously. The overall training process relatively consistent with the original training process of DDIM or LDM. For the CIFAR- 10 dataset, we set the learning rate 6c-5 and the batch size 64 during training. The training process consisted of 100k iterations, and during sampling, we used 100 sampling steps. For the -UU-Bedroomss and FFHQ datasets, the learning rate was set to 2e-5 and the batch size during training. The training consisted of 200k iterations, with 200 steps used during denoising phase.
training is based on the pre-trained diffusion model, and the quantizer
We conducted extensive experiments fusion model LDM and the pixel-space diffusion model DDIM. For the DDIM model, we specifically selected the CIFAR-10 10 dataset with resolution of 32 x 32 for experiments. For the LDM model, our experiments spanned multiple datasets, including the LSUN-Bedrooms, LSUN- CUU-Chhuss and the FFHQ dataset, all with a resolution of 256 256. To evaluate the generation quality of the diffusion model, we utilize several evaluation metrics, including Inception Score (IS), Fréchet Inception Dis- tance (FID) [17], Sliding Fréchet Inception Distance (sFID) [52]. and
different types of diffusion models: the latent-space dif-
Red-allll.. After 200,000 iterations of training, we randomly sample and generate 50,000 images from the model and compute the metrics based on reference batches. The reference batches used evaluate FID and sFII) contain all the corresponding datasets. We recorded FID, sFID, and Precision for all tasks and additional for CIFAR-10.
We utilize OPs as metrics for evaluating theoretical inference efficiency. Taking the convolutional unit an example, the BOPs for single computation operation of single convolution are defined as follows mmnkeb,b.. [69, 71.. It is composed of 6.. bits for weights, 6, bits for activation, n input channels, m output channels, and a k & convolutional kernel. For the output feature with width w and height h, BOPs xe whnmk? b As there might also be full-precision modules in the model, the total OPs of the model are summed up as BOPs + FLOPs [3]. All our experiments are conducted on server with NVIDIA A100 40GB GPU.
B Additional Quantitative Results
We conducted more detailed ablation experiments to comprehensively validate our results.
Effects of learnable & in TBS. We apply the proposed learnable i to the XNOR baseline. The experimental results shown in Table 5 indicate that this modification can lead to significant improvement in performance. The model achieved a doubling of improvement in FID, sFID. Their original values were 106.62 and 56.81, respectively, and they decreased 57.62 and 30.46. The negligible degradation in Recall can be overlooked.
Table 5: Solely transforming & into learnable on the XNOR baseline network.
#Bits
FIDI
sFIDI
Prec.
Recallt
Vanilla learnable
1/1 106.62
56.81
6.82
5.22
1/1
57.26
30.46
15.88
5.00
Effects of cross-timestep connection in TBS. We investigated the impact of varying the number of TBS connections. Table 6 illustrates that the introduction of TBS cross-timestep connections consistently outperforms models without such connections(n = 0). This validates the efficacy of our cross-timestep linkage strategy based on the high-dimensional feature similarity of LDM. Among
16

---

the experiments incorporating cross-timestep connections, the models with and 8 connections both achieved equally optimal results. The model with connection demonstrated slightly superior performance in FID and Precision, whereas the model with 8 nodes exhibited marginally better outcomes in sFID and Recall.
Table 6: The number of TBS connections
71
FID
sFID
Prec.
Recallf
0
30.24
28.21
29.77
16.94
24.22
20.94
34.28
18.22
8
22.74
17.91
33.54
19.90
12
23.25
28.31
37.74
18.78
Effects of SPD. As general quantization method, real-to-binary [39] suggests that using attention map-based loss during the distillation of a binary model from a full-precision model achieves better results. In contrast, BinaryDM, the work most closely related BiDM, directly points out that using L2 loss makes it difficult to align and optimize binary features with full-precision features. These studies indicate that the general L2 loss inadequate for meeting the optimization needs of binary scenarios. So we also compare our SPD with the commonly used L2 loss function. As shown in Table 7, by replacing the L2 loss function with patch distillation, the model can achieve better performance.
Table 7: Different distillation strategies
Caistil
FID|
sFID
Prec.1
Recallf
C2 LSPD
26.07
23.26
33.12 12
18.98
22.74
17.91
33.54
19.90
Further Inference Efficiency Analysis. We expand upon the inference process described in Eq.9 and provide a detailed explanation and testing. Since the divisor involved in calculating the mean of AL.i. from To,,, (i.e., the channel dimension c) can be integrated into 1,,1,3 advance, resulting Additionally, a n,1,1,1 derived from Wnn.ch,, can also be computed ahead of inference. Therefore, the actual operations involved during inference are as follows:
in 11,1,333 k...3.3
C
[FP] Original -reciiini convolution:
. (0) Perform convolution between full-precision 14=448, and full-precision W 444844
to obtain the full-precision output O 448,32,33
XNOR ENNO-NNetRal Net/BiDM] The inference process for
involves the following 6 steps:
Sign operation:
(1) Sign operation:
Binary operation:
(2) Perform convolution between the binary I 44,,32233 and the binary W 44448,448, 448,3,3 to obtain the full-precision output O 448,32,32
Full-precision operations:
(3) Sum the full- full-precision , 48,,32332 across channels to obtain A 1,32332
(4) Perform convolution between full- A 1.32,332 and 1,,,,,,,3 3 to obtain 01. 3,3,,2
(5) Pointwise multiply 0f 32,32 by 0 1..,33 to obtain the full-precision output
O 44.322.3
(6) Pointwise multiply 0 32,,322.32 by a 118,1,1 to obtain the final full aall-precision output
O 32,32
17

---

We utilized the general deployment library Larq [8] on Qualcomm Snapdragon 855 Plus to test the actual runtime efficiency of the aforementioned single convolution. The runtime results for single inference are summarized in the Table 8. Due to limitations of the deployment library and hardware, Baseline achieved 9.97x speedup, while XNOR-Net BiDM achieved an 8.07x speedup. Besides, the improvement in generation performance brought by BiDM is even more significant, and we believe that could achieve better acceleration results in a more optimized environment.
Table 8: The actual runtime efficiency of single convolution.
Runtime(Jus /convolution)
Method
(1)+(2)
(3)
FID
FP
176371.0
176371.0
2.99
Baseline (DoReFa) XNOR-Net / BiDM
17695.2
4.3
17699.5
188.30
17695.2
2948.8 1133.3
83.2
4.3
21864.8
22.74
Further Training Efficiency Analysis. BiDM consists of two techniques: TBS and SPD. The time efficiency analysis during training is follows: (1) TBS includes the learnable convolution of scaling factors (Eq.10) 10) and the cross-time step connection (Eq.12). The increase in training time due the convolution of trainable scaling factors is minimal, the depth of the convolution for scaling factors is only 1, and the size of the trainable convolution kernel is only 3 x 3. The cross-time step connection the primary factor for the increase in training time. Since it requires training we introduce this structure during training, so each training sample requires not only noise estimation for 7 but also for Tt, directly doubling the sampling steps. (2) SPD may lead slight increase in training time (an additional 0. 0.18 times), but since we only apply supervision to the larger aaaammmiinimidddllwwwwwssaamppmm
blocks, the increase limited.
The results in Figure align well with the theoretical analysis mentioned above. BiDM achieved significantly better generative results than baseline methods under the same training iterations, demonstrating that it not only has higher upper limit of generative capability but is also relatively efficient when considering generative performance.
We also tested the FID after uniformly training for 0.5 days, and the results in Tabel 9 show: (1) BiDM has the best convergence, even in short training time. (2) No.3 significantly outperforms No.5 because connections across timesteps greatly increase training time, making No.3 converge faster in the early training stages. (3) No.5 slightly outperforms No.7 because LSRD causes slight increase in training time.
We emphasize that the biggest challenge in fully binarizing DM lies in the drop in accuracy. Although BiDM requires a longer training time for the same number iters, it significantly enhances the quality of generated images, as no other method has been able produce effective images.
Table 9: Training speed under different settings, and FID at 0.5 days.
convolution of scaling learnable connections across factors (Eq.9) k
Training Speed (ms/iter)
FID
No.
Ldiorst
timesteps
at 0.5 days
309.8
167.59
2
310.2
121.63
3
340.8
58.55
458.5
93.66
5
V
480.2
70.80
LSPD Eg (MSE)
389.6
86.78
7
V
496.8
71.15
LSPD
547.2
47.11
C Additional Visualization Results
18

---

0.28
0.28
0.26
0.26
Training Loss
Training Loss
0.22
0.22
0.18
0.18
Training Iteration
Training Time (s)
(a) Training loss Iters
(b) Training loss Time
Figure 6: (a) Training iterations and training loss under different settings. (b) Training time and training loss under different settings. The meaning of the numbers the legend corresponds those in Table 9.
XNOR+
ReActNet
ReSTE
XNOR
BIDM
Figure 7: Generation results of BiDM and baselines on the LSUN-Bedrondd dataset.
19

---

+NOR+
ReActNet
ReSTE
XNOR
BiDM
Figure 8: Generation results of BiDM and baselines on the LSUN-Churches dataset.
XNOR++
ReActNet
ReSTE
XNOR
BIDM
Figure 9: Generation results of BiDM and baselines on the FFHQ dataset.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work in section4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: the paper fully discloses all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper provides open access the data and code, with sufficient instructions
to faithfully reproduce the main experimental results, as described the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. wtc/public/guiiee httpa///guiidee/oeeuuubbiieeeee www.sSubmisiiooPoo...
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The paper specifies all the details necessary to understand the results in the section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper ensure the reproducibility of the experiment by fixing random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources in the section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //teurip//eurip.... cctppccic//ceieeeeeeeee wtt.ei//ieecishuieeeelleelleleee ines?
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the original papers of assets used and introduces the details in the section A.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The details of the new assets are introduced in the section 4 and the section A.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "110bz4EkCN", "reviewer": "Reviewer_4HDt", "summary": "This paper aims to fully binarize weights and activations of diffusion models (DMs) to achieve storage saving and inference acceleration. To this end, the paper proposes timestep-friendly binary structure (TBS), which employs learnable activation binarizers and cross-timestep feature connections to capture the correlation of the activation features over the timesteps. In addition, the paper introduces space patched distillation (SPD) to match the spatial locality of binary features with the full-precision ones during training. The method is tested on some common image generation benchmarks such as LSUN and FFHQ 256x256, CIFAR-10.", "questions": "In Table 3, it is not clear why adding SPD alone makes the results worse, whereas combining it with TBS leads to improved performance. Do the authors have any explanation for this phenomenon? This point should be addressed to clarify the underlying reasons for the observed behavior and to provide a deeper understanding of the interaction between SPD and TBS in the proposed method.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper aims to tackle a challenging problem which is to fully binarize both weights and activations of diffusion models.\n- The paper is well-written.\n- The experimental results are promising.\n- The proposed methods of using timestep-friendly binary structure and space patched distillation are well-motivated.", "weaknesses": "- The main weakness of the paper is that the proposed method, BiDM, increases the training time of DMS compared to the original process. This issue should be more precise in the paper. The authors should compare the convergence rates of the proposed method and its competitors in terms of wall-clock time and identify which components contribute to the increased training time by breaking down the complexity of all main components of the proposed model. \n- The chosen baselines are quite weak as all of them were originally designed for discriminative tasks such as image classification. The authors should consider stronger baselines dedicated to generative tasks such as [1] and the baselines therein.\n\nRef:\n\n[1] Xia et al. Basic Binary Convolution Unit for Binarized Image Restoration Network. ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our manuscript and providing valuable suggestions. Here are our responses to some of the concerns you raised:\n\n> Q1: The main weakness of the paper is that the proposed method, BiDM, increases the training time of DMS compared to the original process. This issue should be more precise in the paper. The authors should compare the convergence rates of the proposed method and its competitors in terms of wall-clock time and identify which components contribute to the increased training time by breaking down the complexity of all main components of the proposed model.\n\nThank you for your suggestions. BiDM consists of two techniques: TBS and SPD. The time efficiency analysis during training is as follows:\n\n- TBS includes the learnable convolution of scaling factors (Eq.10) and the cross-time step connection (Eq.12):\n  - The increase in training time due to the convolution of trainable scaling factors is minimal, as the depth of the convolution for scaling factors is only 1, and the size of the trainable convolution kernel is only $3\\times3$.\n  - The cross-time step connection is the primary factor for the increase in training time. Since it requires training $\\alpha$, we introduce this structure during training, so each training sample requires not only noise estimation for $T^{t-1}$ but also for $T^{t}$, directly doubling the sampling steps.\n\n- SPD may lead to a slight increase in training time (an additional 0.18 times), but since we only apply supervision to the larger upsampling/middle/downsampling blocks, the increase is limited.\n\nWe have supplemented the actual training time on an NVIDIA A100 40GB GPU, and the results in Global Rebuttal (3) align well with the theoretical analysis mentioned above. Due to the actual software and hardware frameworks, the actual training time per iter for BiDM did not completely double compared to the baseline.\n\nFollowing your suggestion, we compared the training loss under the same training iterations or training time. BiDM achieved significantly better generative results than baseline methods under the same training cost, demonstrating that it not only has a higher upper limit of generative capability, but is also relatively efficient when considering generative performance. You can refer to Global Rebuttal (3) for a more detailed explanation.\n\nWe also tested the FID after uniformly training for 0.5 days, and the results in Global Rebuttal (3) show:\n\n- BiDM has the best convergence, even in a short training time.\n- No.3 significantly outperforms No.5 because connections across timesteps greatly increase training time, making No.3 converge faster in the early training stages.\n- No.5 slightly outperforms No.7 because $\\mathcal{L}_{SPD}$ causes a slight increase in training time.\n\nWe emphasize that the biggest challenge in fully binarizing DM lies in the drop in accuracy. Although BiDM requires a longer training time for the same number of iters, it significantly enhances the quality of generated images, as no other method has been able to produce effective images. \n\n\n\n> Q2: The chosen baselines are quite weak as all of them were originally designed for discriminative tasks such as image classification. The authors should consider stronger baselines dedicated to generative tasks such as [1] and the baselines therein.\n\nThank you for your suggestion. We have supplemented the results and analysis for BBCU, and we have also included experimental results for other quantization methods suited for generative models, such as EfficientDM. You can refer to the Global Rebuttal (1) for more comprehensive information.\n\nWe will include the above discussion in the revised version of the manuscript. These discussions will further clarify the motivation and necessity behind each component of BiDM.\n\n\n\n> Q3: In Table 3, it is not clear why adding SPD alone makes the results worse, whereas combining it with TBS leads to improved performance. Do the authors have any explanation for this phenomenon? This point should be addressed to clarify the underlying reasons for the observed behavior and to provide a deeper understanding of the interaction between SPD and TBS in the proposed method.\n\nSorry for the confusion, but there might be a misunderstanding. In fact, SPD alone improves the results compared to Vanilla. Specifically, the ablation results in Table 3 can be more clearly restated in the table below. In all metrics, the binarized DM with only SPD outperforms the Vanilla method. For example, FID decreases from 106.62 to 40.62, demonstrating its superiority in optimizing binarized DMs.\n\nWhen SPD and TBS are used together, SPD brings even better generative performance on top of TBS, further reducing the FID from 35.23 to 22.74.\n\n| Method  |   TBS    |   SPD    | FID$\\downarrow$ | sFID$\\downarrow$ | Prec.$\\uparrow$ | Recall$\\uparrow$ |\n| ------- | :------: | :------: | --------------: | ---------------: | --------------: | ---------------: |\n| Vanilla |          |          |          106.62 |            56.81 |            6.82 |             5.22 |\n| +TBS    | $\\surd $ |          |           35.23 |            25.13 |           26.38 |            14.32 |\n| +SPD    |          | $\\surd $ |           40.62 |            31.61 |           23.87 |            11.18 |\n| BiDM    | $\\surd $ | $\\surd $ |           22.74 |            17.91 |           33.54 |            19.90 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MO8cNqyzxJ", "reviewer": "Reviewer_cNoX", "summary": "The manuscript proposes a method for fully binarizing both the weights and activations of diffusion models, named BiDM. Structurally, it introduces an improved XNOR method for scaling factors of activations and high-level feature connections across time steps, based on observations of existing temporal phenomena. For optimization, the manuscript presents a patch-based attention mechanism distillation method, approached from the perspective of spatial features. Quantitative analysis and generated examples demonstrate that BiDM, as the first work to fully binarize diffusion models, surpasses existing binarization methods, showcasing the potential for deploying diffusion models in resource-constrained scenarios.", "questions": "See Weaknesses", "limitations": "The manuscript only addresses diffusion models using a U-Net backbone primarily based on convolution. It appears to be inapplicable to diffusion models like DiT, which are primarily based on transformers and not related to U-Net.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis is the first work to fully binarize diffusion models, preventing them from collapsing in extreme scenarios and demonstrating acceptable generated samples. This is significant for exploring the compression potential of generative models.\n2.\tThe improvement to XNOR is not only applicable to diffusion models (DMs) but also has positive implications for the whole binarization field. Most papers and deployment frameworks, when replicating and implementing XNOR, have consistently followed the approach described in Equations 7 or 8 of this manuscript, rather than the method described in Equation 9. The authors have meticulously examined and adaptively improved the original XNOR approach within the DM field, prompting reconsideration in both DM compression and general binarization fields.\n3.\tThe cross-time step connection appears novel and well-suited to the inference structure of DMs. Unlike the cache design in DeepCache, which aims at inference efficiency, this manuscript uses connections to enhance information due to the inherently efficient nature of binarized models. These connections can be placed at multiple nodes within the model, with learnable coefficient factors providing greater adaptability.\n4.\tSection 3.2 first designs TBS from the perspective of information enhancement and then provides further explanation from the perspective of error reduction in Figure 3, making the overall method design appear cohesive and intuitively clear.\n5.\tThe distillation strategy design is straightforward and effective, approaching patch division based on the inherent requirements of the generation task and the natural locality of the convolution module, and achieving excellent results.", "weaknesses": "1. The description of the cross-time step connection during the training phase is somewhat vague. During the inference phase, the impact of this connection is iteratively attenuated. For example, when α is 0.3, the influence of step T on step T-2 is 0.3×0.3=0.09, rather than 0. However, upon reviewing the source code in the supplementary materials, I found that the authors only consider steps T-1 and T-2 during training. I would like to know if this affects the final accuracy.\n2. The scenario assumed in Figure 3 is overly idealized. When only L^t changes, for instance, when L^t is at the lower right of L^(t-1), the weighted average might result in T^(t-1) being further from F^(t-1). How is this situation explained?\n3. Although BiDM shows significant improvements over other binarization methods, there remains a substantial performance gap compared to full-precision models. This could still hinder its practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your high recognition of our work and the valuable suggestions you provided. Our response is as follows:\n\n> Q1: The description of the cross-time step connection during the training phase is somewhat vague. During the inference phase, the impact of this connection is iteratively attenuated. For example, when α is 0.3, the influence of step T on step T-2 is 0.3×0.3=0.09, rather than 0. However, upon reviewing the source code in the supplementary materials, I found that the authors only consider steps T-1 and T-2 during training. I would like to know if this affects the final accuracy.\n\nFrom the optimization principles of DDPM[1], this does not affect the accuracy of the results. Essentially, our training process is no different from that of DDPM, as both use efficient training to optimize random terms of the usual variational bound on negative log likelihood with stochastic gradient descent. Due to the cross-time step connection in TBS, which requires considering the values from the previous inference step, we included T-2, T-1, and T in our considerations. This approach still adheres to the DDPM training methodology.\n\n[1] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. NeurIPS, 2020.\n\n\n\n> Q2: The scenario assumed in Figure 3 is overly idealized. When only L^t changes, for instance, when L^t is at the lower right of L^(t-1), the weighted average might result in T^(t-1) being further from F^(t-1). How is this situation explained?\n\nYou can refer to the Global Rebuttal (2) for the explanation. Additionally, in practical applications, the learnability of $\\alpha$ ensures the effective collaboration between $L^{t-1}$ and $L^{t}$. Moreover, the ablation study results in Table.6 also demonstrate the effectiveness of connecting $L^{t-1}$ and $L^{t}$ across time steps.\n\n\n\n\n\n> Q3: Although BiDM shows significant improvements over other binarization methods, there remains a substantial performance gap compared to full-precision models. This could still hinder its practical application.\n\nBiDM is the first fully binarized DM method capable of generating viewable images, significantly surpassing advanced binarization methods like ReSTE in quantitative metrics (on LSUN-Bedrooms, BiDM achieves an FID of 22.74, notably better than ReSTE's FID of 59.44). This demonstrates the feasibility of fully binarized DMs, marking a step towards practical applications with great potential.\n\nBesides, as you highlighted in the Strengths section, the improvements to techniques like XNOR will have a positive impact on the entire binarization field. We will further explore this area in future work to achieve broader practical applications.\n\n\n\n> Q4: The manuscript only addresses diffusion models using a U-Net backbone primarily based on convolution. It appears to be inapplicable to diffusion models like DiT, which are primarily based on transformers and not related to U-Net.\n\nThe connections across timesteps in TBS and SPD should be directly applicable to DiT, with SPD likely being even more compatible with DiT's inherently space-patched input. While the convolutional design in TBS isn't suitable for DiT's linear-based architecture, we have observed that finer quantization granularity (such as per-group quantization) is gradually being proposed for architectures like transformers. Using a similar approach of dynamically calculating statistics for scaling factors followed by convolution or linear transformation could also be applicable. We plan to explore these aspects further in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4knNkYWUMr", "reviewer": "Reviewer_AtKg", "summary": "This paper proposes BiDM, which focuses on quantizing both weights and activations of diffusion models (DMs). Specifically, the authors introduce:\n- Cross-timestep feature connection to enhance the accuracy of noise prediction in binarized DMs.\n- Space-patched distillation, a novel variant of distillation loss that emphasizes spatial locality.", "questions": "- Are the Q, K, V in the attention modules binarized during inference?\n\n- During inference, is $A$ in Equation 9 calculated from the inputs?\n\n- In Line 224, the authors claim that \"conventional $l_2$  loss makes it difficult for binary DMs to achieve direct matching.\" Could you provide evidence or reference existing works that have verified this claim?\n\n- Figure 3 is somewhat confusing. Why is $L^{t-1}$ a linear scaling up of $B^{t-1}$? Is this relationship depicted by Equation 12?", "limitations": "This paper includes a discussion of limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Binarization of diffusion models (DMs) represents a promising research area aimed at accelerating sampling processes, and this work appears to be the first to fully binarize DMs to W1A1 to the best of my knowledge.\n\n- The introduction of cross-timestep information connection is innovative in enhancing the performance of binarized DMs.\n\n- The experimental results on LSUN-Bedrooms, LSUN-Churches, and FFHQ effectively demonstrate the effectiveness of BiDM and XNOR techniques.", "weaknesses": "- The experimental results lack sufficient conviction. Tables 1 and 2 compare BiDM primarily with works focused on quantizing discriminative models, which may not be the most appropriate choice of baseline. It would be more informative to include comparisons with quantization methods designed specifically for diffusion models, such as EfficientDM [cite], given its open-sourced nature and compatibility under W1A1 settings. This comparison is crucial for a comprehensive evaluation.\n\n- Figure.4 may not effectively demonstrate the advantages of paying attention to local information. To better illustrate the effectiveness of Space-Patched Distillation (SPD), the authors should compare experimental results with those obtained using vanilla distillation loss (Mean Squared Error between outputs of a full precision model and a binarized model).\n\n- The proposed binarization method introduces additional convolution computations between two matrices in floating-point, which may not be efficient for hardware deployment. Moreover, the dynamic calculation of matrix $A$ during inference introduces memory access overhead that could be significant in practical deployment scenarios.\n\n- During inference, matrix multiplications (MMs) in the attention mechanism typically consume considerable time. It appears that this work retains these MMs in floating-point arithmetic, which could impact efficiency.\n\n**minor issues**\n- Line 93: The citation for BinaryDM is incorrectly linked.\n- In Figure 4, the caption states \"Ours denotes the self-attention,\" but in the figure, it's labeled as \"SPD.\"\n- The term \"self-attention\" in Section 3.3 is confusing; perhaps using an alternative term would be clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our work. Here are our responses to your concerns:\n\n> Q1: The experimental ...\n\nThank you for your suggestion. We have added experiments and discussions on EfficientDM. You can check the Global Rebuttal (1) for more details.\n\n\n\n> Q2: Figure.4 ...\n\nThank you for your suggestion. We have already placed this exploration in the manuscript. Table.7 shows that $L_2$ represents the results using MSE Loss as the distillation loss. $L_{SPD}$, with an FID of 22.74, significantly outperforms MSE Loss, which has an FID of 26.07, demonstrating the suitability of SPD for DM.\n\n\n\n> Q3: The proposed ...\n\nIt should be noted that during inference, BiDM requires only a very small number of additional floating-point additions for the connections across timesteps compared to the classic binarization work XNOR-Net, and there are no differences in the majority of calculations, such as convolutions. XNOR-Net's original paper already stated that it achieves a 58× speedup in convolution operations in its abstract.\n\nWe understand your concerns about efficiency, but performing a floating-point convolution with a depth of 1 for scaling factors requires only a very small amount of computation, and the overhead for averaging matrix $A$ is also minimal. Statistical results show that this portion accounts for only 0.08% of the total OPs in binarized DM. Therefore, BiDM remains efficient during inference.\n\n\n\n> Q4: During inference ...\n\nBiDM fully binarizes components such as convolutions and QKV, and the remaining components have a very minimal impact on computation. Specifically, the computational cost of this part accounts for only 0.38% of the total OPs in the binarized DM.\n\nIn particular, we fully binarized linear/conv1d layers with weights and MMs involving the input x in AttentionBlock, while MMs involving intermediate results (such as proj_out) were performed using floating-point operations. This decision was based on our observation that the latter does not involve weight storage, involves very little computation, and has a relatively noticeable impact on the final results.\n\nThus, we chose not to fully binarize the mentioned MMs, and this trade-off is necessary to achieve a more accurate fully binarized DM.\n\n\n\n> Q5: minor issues:\n>\n> Q5-1: Line 93 ...\n\nSorry for the incorrect link; it should have pointed to reference [54] in the manuscript.\n\n> Q5-2: In Figure 4 ...\n\nHere, we provide a clearer explanation to present the meaning of Fig.4 better:\n\n- The first row labeled `FP32` shows the visualization of the full-precision model output $\\mathcal{F}^{fp}$.\n- The second row labeled `Diff` represents the visualization of the difference between the full-precision model output and the binarized model output, $\\mathcal{F}^{fp} - \\mathcal{F}^{bi}$.\n- The third row labeled `SPD` shows the visualization of the local attention for both the full-precision model and the binarized model calculated using the SPD method: $ \\frac{ A^{fp}}{ ||A^{fp}||_2 } - \\frac{A^{bi}}{ ||A^{bi}||_2 } $.\n\nFrom Figure 4, it can be observed that our SPD method allows the model to pay more attention to local information in each patch.\n\n> Q5-3: The term ...\n\nThank you for your feedback. Using \"attention-guided\" might be a more appropriate term. Our implementation of SPD is inspired by self-attention, involving matrix multiplication with its transpose locally to measure local self-attention. However, as you mentioned, self-attention is a widely accepted technical term, and using it to describe our SPD could lead to conceptual confusion.\n\nWe will update the explanations for these three issues in the revised version of the manuscript. Thank you very much for your detailed review of our work.\n\n\n\n> Q6: Are the Q ...\n\nSure, as we mentioned in our response to `Q4` above, QKV components have all been binarized.\n\n\n\n> Q7: During inference ...\n\nYes, the calculations are dynamic. However, as explained in `Q3`, this part of the computation is identical to that in XNOR-Net and involves very minimal additional calculation, resulting in a very limited extra burden on the inference process.\n\n\n\n> Q8: In Line 224 ...\n\nAs a general quantization method, real-to-binary [1] suggests that using attention map-based loss during the distillation of a binary model from a full-precision model achieves better results. In contrast, BinaryDM, as the work most closely related to BiDM, directly points out that using L2 loss makes it difficult to align and optimize binary features with full-precision features. These studies indicate that the general L2 loss is inadequate for meeting the optimization needs of binary scenarios.\n\nAs a result, Table.7 of our manuscript shows that SPD indeed outperforms L2 loss with MSE.\n\nWe will include a discussion on this section in the revised version of the manuscript to make the motivation behind SPD more clear.\n\n[1] Training binary neural networks with real-to-binary convolutions.\n\n\n\n> Q9: Figure 3 ...\n>\n> (1) Why is Lt−1 ...\n>\n> (2) Is this relationship ...\n\nSorry for the confusion. Here are our explanations for the two issues:\n\n- (1) In reality, although the trainable $k$ results in changes to local elements, this does not imply that the entire vector is linearly scaled. Thus, ${L}^{t-1}$ is not a linear scaling up of ${B}^{t-1}$.\n- (2) The relationship between these two is unrelated to Equation 12. ${L}^{t-1}$ and ${B}^{t-1}$ should be considered as differences in the output features of the model obtained after training under non-trainable/trainable settings of $k$ in Eq.10. This means that the learnable $k$ allows for more flexible and free feature representation.\n\nYou could refer to the Global Rebuttal (2) for a further explanation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Rebuttal

We appreciate all reviewers for their careful reviews and the constructive feedback provided on our work, BiDM. Here is a summary of the main contributions of BiDM:

We propose BiDM, the first method to achieve an accurate fully binarized diffusion model, aiming for extreme compression and inference acceleration. Based on two observations — activations at different timesteps and the characteristics of image generation tasks — we introduce the Timestep-friendly Binary Structure (TBS) and Space Patched Distillation (SPD) from temporal and spatial perspectives, respectively. In TBS, the learnable tiny convolution adapts to the highly dynamic activation range of DMs from the basic binary operator level. It is tightly integrated with cross-timestep connections that leverage the similarity of activation features between adjacent timesteps, forming the structure of BiDM. SPD takes advantage of the spatial locality in both the DM model structure and the image generation tasks it performs, alleviating optimization difficulties brought by general training methods or naive L2 distillation loss and achieving better generative results.

As the first fully binarized DM, BiDM achieves the best generative results, reducing FID by 62% on LSUN-Bedrooms and is currently the only method capable of generating visually acceptable images. At the same time, BiDM ensures considerable efficiency during inference. Compared to the classic binarization method XNOR-Net, it only adds a minimal amount of addition operations for cross-timestep connections, achieving 28.0× storage and 52.7× OPs savings.

We also noticed that the reviewers raised some common questions. We have summarized and responded to them collectively as follows:

(1) Reviewers suggested adding quantization methods more suited to generative models, such as BBCU and EfficientDM, for comparison. We have supplemented this on LSUN-Bedrooms and conducted the following analysis:

- BBCU:
  - We have supplemented our work with BBCU, a binarization method more akin to generative models like DMs rather than discriminative models. We implemented the residual connections in BBCU and used RPReLU. However, since DMs do not have BN layers, we did not incorporate the BN design from BBCU in our adaptation.
  - Experimental results indicate that even as a binarization strategy for generative models, BBCU faces significant breakdowns when applied to DMs.
- EfficientDM:
  - As a work targeting QAT for DM, EfficientDM is indeed a suitable comparison, especially since it designs TALSQ to address the variation in activation range. We adapted DoReFa as the basic operator for its use under W1A1.
  - The results in the table below show that EfficientDM struggles to adapt to the extreme scenario of W1A1, and this may be due to its quantizer having difficulty adapting to binarized DM, and using QALoRA for weight updates might yield suboptimal results compared to full-parameter QAT.

As we mentioned in the TBS section of our manuscript, most existing binarization methods struggle to handle the wide activation range and flexible expression of DMs, further highlighting the necessity of TBS. Their optimization strategies may also not be tailored for the image generation tasks performed by DM, which means they only achieve conventional but suboptimal optimization.

| Method      | #Bits | FID$\downarrow$ | sFID$\downarrow$ |
| :---------- | ----- | --------------: | ---------------: |
| BBCU        | 1/1   |          236.07 |            89.66 |
| EfficientDM | 1/1   |          194.45 |           113.24 |
| BiDM        | 1/1   |           22.74 |            17.91 |

(2) Some reviewers expressed confusion regarding the details of Fig.3. Here is a clearer clarification for Fig.3:

- Since the feature space is very high-dimensional, we could only illustrate it using schematic diagrams. The functions of the two components in TBS are exaggerated for a clearer explanation. Thus, Fig.3 is merely a visual representation of the collaborative function of the two components in TBS. For a detailed explanation, please refer to the end of section 3.2 in our manuscript. In simple terms, the learnable tiny convolution $k$ in TBS allows scaling factors to adapt more flexibly to the representation of DM, while connections across timesteps enable the binarized DM to use the previous step’s output information for appropriate information compensation. Together, these elements enhance the precision of the binarized DM.
- We have adjusted Fig.3 to make it more general. You can view the updated illustration in the attached PDF.

(3) We have also included in the PDF attachment the convergence of training loss over iterations/time for different methods. The results show that BiDM not only achieves the best generative performance with sufficient training time, as stated in the original manuscript but also exhibits the best convergence even under the same number of iterations/time.

We will include the above discussions in the revised version of the manuscript. These discussions will further clarify the motivation and necessity behind each component of BiDM.

For specific questions raised by each reviewer, please refer to our corresponding responses.

Meta-review: This paper presents BiDM, a method for fully binarizing both the weights and activations of diffusion models. Techincally, BiDM consists of a timestep-friendly binary structure (TBS) to capture the correlation of the activation features over the timesteps and a space patched distillation (SPD) to match the spatial locality of binary features with the full-precision ones during training. Results on common image generation benchmarks were reported. The initial concerns were addressed during rebuttal. I recommend to accept, and encourage the authors to incorporate the discussions into the final version.
Decision: Accept (poster)