BiDM: Pushing the Limit of Quantization for Diffusion Models

Authors: xingyu  zheng
xianglong  liu
yichen  bian
xudong  ma
yulun  zhang
jiakai  wang
jinyang  guo
haotong  qin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-28 04:33:04

Abstract: Diffusion models (DMs) have been significantly developed and widely used in various applications due to their excellent generative qualities. However, the expensive computation and massive parameters of DMs hinder their practical use in resource-constrained scenarios. As one of the effective compression approaches, quantization allows DMs to achieve storage saving and inference acceleration by reducing bit-width while maintaining generation performance.  However, as the most extreme quantization form, 1-bit binarization causes the generation performance of DMs to face severe degradation or even collapse. This paper proposes a novel method, namely BiDM, for fully binarizing weights and activations of DMs, pushing quantization to the 1-bit limit. From a temporal perspective, we introduce the Timestep-friendly Binary Structure (TBS), which uses learnable activation binarizers and cross-timestep feature connections to address the highly timestep-correlated activation features of DMs. From a spatial perspective, we propose Space Patched Distillation (SPD) to address the difficulty of matching binary features during distillation, focusing on the spatial locality of image generation tasks and noise estimation networks. As the first work to fully binarize DMs, the W1A1 BiDM on the LDM-4 model for LSUN-Bedrooms 256$\times$256 achieves a remarkable FID of 22.74, significantly outperforming the current state-of-the-art general binarization methods with an FID of 59.44 and invalid generative samples, and achieves up to excellent 28.0 times storage and 52.7 times OPs savings.

Main paper: 
 introduction: 
 Diffusion models (DMs) [19, 50, 44, 76] , as a type of generative visual model [66, 59, 68] , have garnered impressive attention and applications in various fields, such as image [57, 58] , speech [42, 45, 24] , and video [40, 18] , because of their high-quality and diverse generative capabilities. The diffusion model can generate data from random noise through up to 1000 denoising steps [19] . Although some accelerated sampling methods effectively reduce the number of steps required for generating tasks [56, 31] , the expensive floating-point computation of each timestep still limits its wide application on resource-constrained scenarios. Therefore, compression of the diffusion model becomes a crucial step for its broader application, and existing compression methods mainly include quantization [30, 54, 47] , distillation [53, 36, 41, 73, 11] , pruning [7, 12, 14, 13] , etc. These compression approaches aim to reduce storage and computation while preserving accuracy.Quantization is considered a highly effective model compression technique [70, 9, 64, 21, 10] , which quantizes the weights and/or activations to low-bit integers or binaries for compact storage and 38th Conference on Neural Information Processing Systems (NeurIPS 2024). efficient computation in inference. Some existing works thus apply quantization to compress DMs, aiming to compress and accelerate them while maintaining the quality of generation. Among them, 1-bit quantization, namely binarization, can achieve maximum storage savings for models and has performed well in discriminative models such as CNNs [33, 67, 65] . Furthermore, when both weights and activations are quantized to 1-bit, e.g., fully binarized, efficient bitwise operations such as XNOR and bitcount can replace matrix multiplication, achieving the most efficient acceleration [74] .Some existing works have attempted to quantize DM to 1-bit [77] , but their exploration mainly focuses on the weights, which are still far from full binarization. In fact, for generative models like DM, the impact of fully binarizing weights and activations is catastrophic: a) As generative models, DMs have rich intermediate representations closely related to timesteps and highly dynamic activation ranges, which are both very limited in information when binarized weights and activations are used; b) Generative models like DMs are typically required to output complete images, but the highly discrete parameter and feature space make it particularly difficult for binarized DMs to match the ground truth during training. The limited representational capacity, which is hard to match with timesteps dynamically, and the optimization difficulty of generative tasks in discrete space make it difficult for the binarized DM to converge or even collapse during the optimization process.We propose BiDM to push diffusion models towards extreme compression and acceleration through complete binarization of weights and activations. It is designed to address the unique properties of DMs' activation features, model structure, and the demands of generative tasks, overcoming the difficulties associated with complete binarization. BiDM consists of two novel techniques: From a temporal perspective, we observe that the activation properties of DMs are highly correlated with timesteps. We introduce the Timestep-friendly Binary Structure (TBS), which uses learnable activation binary quantizers to match the highly dynamic activation ranges of DMs and designs feature connections across timesteps to leverage the similarity of features between adjacent timesteps, thereby enhancing the representation capacity of the binary model. From a spatial perspective, we note the spatial locality of DMs in generative tasks and the convolution-based U-Net structure. We propose Space Patched Distillation (SPD), which introduces a full-precision model as a supervisor and uses attention-guided imitation on divided patches to focus on local features, better guiding the optimization direction of the binary diffusion model.Extensive experiments show that compared to existing SOTA fully binarized methods, BiDM significantly improves accuracy while maintaining the same inference efficiency, surpassing all existing baselines across various evaluation metrics. Specifically, in pixel space diffusion models, BiDM is the only method that raises the IS to 5.18, close to the level of full-precision models and 0.95 higher than the best baseline method. In LDM, BiDM reduces the FID on LSUN-Bedrooms from the SOTA method's 59.44 to an impressive 22.74, while fully benefiting from 28.0× storage and 52.7× OPs savings. As the first fully binarized method for diffusion models, numerous generated samples also demonstrate that BiDM is currently the only method capable of producing acceptable images with fully binarized DMs, enabling the efficient application of DMs in low-resource scenarios. 
related work: 
 Diffusion models (DMs) have demonstrated excellent generative capabilities across various tasks [19, 57, 58, 43, 42, 45, 24] . However, their large-scale model architectures and the high computational costs required for multi-step inference limit their practical applications. To address this, methods for accelerating the process at the timestep level have been widely proposed, including sampling acceleration that does not require retraining [56, 31, 34, 35] and distillation methods [53, 36, 41] . A recent method called DeepCache [38] caches high-dimensional features to avoid a lot of redundant computations and is compatible with typical sampling acceleration methods. However, these methods cannot overcome the memory bottlenecks and efficiency limits during single-step inference.Quantization is a widely validated compression technique that compresses weights and activations from the usual 32 bits to 1-8 bits to achieve compression and acceleration [6, 78, 37, 75] . Consequently, quantization is being studied for application in diffusion models [15, 4] . These methods generally consider the unique timestep structure and spatial architecture of diffusion models, but due to the significant difficulty of quantizing generative models, most post-training quantization (PTQ) methods can only quantize models to 4 bits or more [29, 54, 22] , while more accurate quantizationaware training (QAT) methods face severe performance bottlenecks below 3 bits [30, 55] .Binarization, the most extreme form of quantization, typically expresses weights and activations as ±1, allowing the model to achieve maximum compression and acceleration [60, 62] . In computer vision, binarization work has mainly focused on discriminative models like CNNs [49, 33, 46, 48] or ViTs [28, 16] , with limited work on generative models. While ResNet VAE and Flow++ [1] have achieved complete binarization for VAEs [26] , they do not offer generative performance comparable to current advanced models. Binary Latent Diffusion [61] binarized the latent space of LDMs [26] but did not improve the model's spatial footprint or inference efficiency. The latest work, BinaryDM [50] , quantized DMs to nearly W1A4, but it did not address activation quantization, leaving room for achieving full binarization and acceleration of DMs.3 Method 
binarized diffusion model baseline: 
 Diffusion models. Given a data distribution x 0 ∼ q(x 0 ), the forward process generates a sequence of random variables x t ∈ {x 1 , • • • , x T } with transition kernel q(x t |x t-1 ), usually Gaussian perturbation, which can be expressed asEQUATIONwhere β t ∈ (0, 1) is a noise schedule. Gaussian transition kernel allows us to marginalize the joint distribution, so with α t := 1 -β t and ᾱt := t i=1 α i , we can easily obtain a sample of x t by sampling a gaussian vector ϵ ∼ N (0, I) and applying the transformationx t = √ ᾱt x 0 + √ 1 -ᾱt ϵ.The reverse process aims to generate samples by removing noise, approximating the unavailable conditional distribution q (x t-1 | x t ) with a learnable transition kernel p θ (x t-1 | x t ), which can be expressed asEQUATIONThe mean μθ (x t , t) and variance βt could be derived using the reparameterization tricks in [19] :EQUATIONwhere ϵ θ is a function approximation with the learnable parameter θ, which predicts ϵ given x t .For the training of DMs, a simplified variant of the variational lower bound is usually applied as the loss function for better sample quality, which can be expressed as EQUATIONConcat(D m (•), U m+1 (•)).(EQUATIONBinarization. The quantization compresses and accelerates the noise estimation model by discretizing weights and activations to low bit-width. In the baseline of the binarized diffusion model, the weights w are binarized to 1-bit [49, 5, 20] :EQUATIONwhere sign function confine w to +1 or -1 with 0 thresholds. σ is a floating-point scalar, which is initialized as ∥w∥ n (n denotes the number of weights) and learnable during training following [49, 33] . Meanwhile, activations are typically quantized by naive BNN quantizers [23, 32] :EQUATIONWhen both weights and activations are quantized to 1-bit, the computations of the denoising model can be replaced by XNOR and bitcount operators, achieving significant compression and acceleration.Before delving into the detailed description of the proposed method, we summarize our observation on the properties of DMs:Observation 1. The activation range varies significantly across long-term timesteps, but the activation features are similar in short-term neighbouring timesteps.Previous works, such as TDQ [55] and Q-DM [30] , have commonly demonstrated that the activation distribution of DMs largely depends on denoising process, manifesting as similarities between adjacent timesteps while difference between distant ones, as shown in Figure 2 (a). Therefore, applying a fixed scaling factor to activations across all timesteps can cause significant distortion in the activation range. Beyond the distribution range, Deepcache [38] highlights the substantial temporal consistency of high-dimensional features across consecutive timesteps, as shown in Figure 2(b) .These phenomena prompt us to reexamine existing binary structures. Binaryization, especially the full binaryization of weights and activations, results in a greater loss of activation range and precision compared to low-bit quantizations like 4-bit [50] . This makes it more challenging to generate rich activation features. Such deficiencies in activation range and output features significantly harm representation-rich generative models like DMs. Therefore, adopting binary quantizers with more flexible activation ranges for DMs, and enhancing the model's overall expressive power by leveraging its feature outputs, are crucial strategies for improving its generative capability after full binaryization.We first focus on the differences between various timesteps over the long term. Most existing activation quantizers, such as BNN [23] and Bi-Real [32] , as shown in Eq. ( 7), directly quantize activations to {+1, -1}. This approach significantly disrupts activation features and negatively impacts the expressive power of generative models. Some improved activation binary quantizers, such as XNOR++ [2] , adopt a trainable scale factor k:EQUATIONwhere the form of K could be either a vector or the product of multiple vectors, but it remains a constant value during inference. Although this approach partially restores the feature expression of activations, it does not align well with diffusion models that are highly correlated with timesteps and may still lead to significant performance loss.We turn our attention to the original XNOR, which employs dynamically computed means to construct the activation binary quantizer. Its operation for 2D convolution can be expressed as:EQUATIONwhereI ∈ R c×win×hin , W ∈ R c×w×h , A = |Ii,:,:| c , α = 1 n ∥W∥ ℓ1 . k ∈ R 1×1×w×h represents a 2D filter, where ∀ij k ij = 1w×h . * and ⊗ indicate convolution with and without multiplication, respectively. This approach naturally preserves the range of activation features and dynamically adapts with the input range across different timesteps. However, due to the rich expression of DM features, local activations exhibit inconsistency in range before and after passing through modules, indicating that the predetermined value of k does not effectively restore the activation representation.Therefore, we make k adjustable and allow it to be learned during training to adaptively match the changes in the range of activations before and after. The gradient calculation process of our learnable tiny convolution k can be expressed as follows:EQUATIONNotably, making k learnable does not add any extra inference burden. The computational cost remains unchanged, allowing for efficient binary operations.On the other hand, we focus on the similarity between adjacent timesteps. Deepcache directly extracts high-dimensional features as a cache to skip a large amount of deep computation in U-Net, achieving significant inference acceleration. This process is expressed as:EQUATIONHowever, this approach does not apply to binarized diffusion models, as the information content of each output from a binary network is very limited. For binary diffusion models, which inherently achieve significant compression and acceleration but have limited expressive power, we anticipate that the similarity of features between adjacent timesteps will enhance binary representation, thereby compensating for the representation challenges.We construct a cross-timestep information enhancement connection to enrich the expression at the current timestep using features from the previous step. This process can be expressed as:EQUATIONwhere α t-1 m+1 is a learnable scaling factor. As shown in Figure 2 (b), the similarity of high-dimensional features varies across different blocks and timesteps in DMs. Therefore, we set multiple independent α values to allow the model to adaptively learn more effectively during training.In summary, Timestep-friendly Binary Structure (TBS) includes learnable tiny convolution applied to scaling factors after averaging the inputs and connections across timesteps. Their combined effect adapts to the changes in the activation range of diffusion models over long-range timesteps and leverages the similarity of high-dimensional features between adjacent timesteps to enhance information representation. From the perspective of error reduction, a visualization of TBS is shown in Figure 3 . First, we abstract the output of the binary DM under the baseline method as vector B t-1 . The mismatch in scaling factors creates a significant difference in length between it and the output vector F t-1 of the fullprecision model. Using our proposed scaling factors and learnable tiny convolutions, B t-1 is expanded to L t-1 . L t-1 is closer to F t-1 , but there is still a directional difference from the fullprecision model. The cross-timestep connection further incorporates the outputs F t of the previous timestep, B t , and L t . The high-dimensional feature similarity between adjacent timesteps means the gap between F t-1 and F t is relatively small, facilitating the combination of L t-1 and L t . Finally, we obtain the binarized DM's output with TBS applied asT t-1 = (1 -α) • L t-1 + α • L t ,closest to the output F t-1 of the full-precision model. The learnable tiny convolution k in TBS allows scaling factors to adapt more flexibly to the representation of DM, while connections across timesteps enable the binarized DM to use the previous step's output information for appropriate information compensation.Due to the nature of generative models, the optimization process of diffusion models exhibits different characteristics from past discriminative models: Observation 2. Conventional distillation struggles to guide fully binarized DMs to align with full-precision DMs, while the features of DM exhibit locality in space during the generation task.In previous practices, adding distillation loss during the training of quantized models has been a common approach. As the numerical space of binary models is limited, directly optimizing them using naive loss leads to difficulties in adjusting gradient update directions and makes learning challenging. Therefore, adding distillation loss to intermediate features can better guide the model's local and global optimization process.However, as a generative model, the highly rich feature representation of DMs makes it extremely difficult for binary models to finely mimic full-precision models. Although the L2 loss used in the original DM training aligns with the Gaussian noise in the diffusion process, it is not suitable for the distillation matching of intermediate features. During regular distillation, the commonly used L2 loss tends to prioritize optimizing pixels with larger discrepancies, leading to a more uniform and smooth optimization result. This global constraint learning process is challenging for binary models aimed at image generation, as their limited representation capacity makes it difficult for fine-grained distillation imitation to directly adjust them to fully match the direction of full-precision models.At the same time, we note that DMs using U-Net as a backbone naturally exhibit spatial locality due to their convolution-based structure and generative task requirements. This is different from past discriminative models, where tasks like classification only require overall feature extraction without low-level requirements, making traditional distillation methods unsuitable for DMs with spatial locality in generative tasks. Additionally, most existing DM distillation methods focus on reducing the number of timesteps and do not address the spatial locality of features required for image generation tasks.Therefore, given the difficulty in optimizing binary DMs with existing loss functions and the spatial locality of DMs, we propose Space Patched Distillation (SPD). Specifically, we designed a new loss function that partitions features into patches before distillation and then calculates spatial attentionguided loss patch by patch. While conventional L2 loss makes it difficult for binary DMs to achieve direct matching, leading to optimization challenges, the attention mechanism allows the distillation SPD first divides the intermediate features F bi and F fp ∈ F b×c×w×h , output by a block of the binary DM and the full-precision DM respectively, into p 2 patches:EQUATIONThen, attention-guided loss is calculated for each patch separately:EQUATIONAfter regularization, the losses at corresponding positions are calculated and summed up:EQUATION)where ∥ • ∥ 2 denotes the L2 function. Finally, the total training loss L is computed as:EQUATIONwhere d denotes the number of blocks during the upsampling process or downsampling process, resulting in a total of 2d + 1 intermediate features, including the middle block. λ is a hyperparameter coefficient to balance the loss terms, defaulting set to 4.We visualize the intermediate features and attention-guided SPD mentioned above. As Figure 4 shown, our SPD allows the model to pay more attention to local information in each patch. 
experiment: 
 We conduct experiments on various datasets, including CIFAR-10 32 × 32 [27] , LSUN-Bedrooms 256 × 256 [72] , LSUN-Churches 256 × 256 [72] and FFHQ 256 × 256 [25] over pixel space diffusion models [19] and latent space diffusion models [50] . The evaluation metrics used in our study encompass Inception Score (IS), Fréchet Inception Distance (FID) [17] , Sliding Fréchet Inception Distance (sFID) [52] , Precision and Recall. To date, there has been no research that compresses diffusion models to such an extreme extent. Therefore, we use classical binarization algorithms [2, 78, 33, 49] , the recent SOTA general binarization algorithms [62] , and quantization methods suited to generative models [15, 63] as baselines. We extract the outputs of TimestepEmbedBlocks from the DM to serve as the operating target for our TBS and SPD. And we employ the same shortcut connections in convolutional layers as those used in ReActNet [33] . Detailed experiment settings are presented in the Appendix A.Pixel Space Diffusion Models. We first conduct experiments on the CIFAR-10 32 × 32 dataset.As the results presented in Table 1 , W1A1 binarization of DM using baseline methods results in substantial degradation. However, BiDM demonstrated significant improvements across all metrics, achieving unprecedented restoration of image quality. Specifically, BiDM achieved remarkable enhancements from 4.23 to 5.18 in the IS metric, and reduced 27.9% in the FID metric. In contrast to XNOR++, its adoption of fixed activation scaling factors in the denoising process results in a very limited dynamic range for its activations, making it difficult to match the highly flexible generative representations of DMs. BiDM addressed this challenge by making the tiny convolution k learnable, which acts on the dynamically computed scaling factors. This optimization led to substantial improvements exceeding an order of magnitude across all metrics. On the LSUN-Bedrooms and LSUN-Churches datasets, the FID metric decreased from 319.66 to 22.74 and from 292.48 to 29.70, respectively. Additionally, compared to the SOTA binarization method ReSTE, BiDM achieved significant enhancements across multiple metrics, particularly demonstrating notable improvements on the LSUN-Bedrooms dataset. We have supplemented our work with BBCU, a binarization method more akin to generative models like DMs rather than discriminative models.Experimental results indicate that even as a binarization strategy for generative models, BBCU faces significant breakdowns when applied to DMs, as FID dropped dramatically to 236.07 on LSUN-Bedrooms. As a work targeting QAT for DM, EfficientDM is indeed a suitable comparison, especially since it designs TALSQ to address the variation in activation range. The results show that EfficientDM struggles to adapt to the extreme scenario of W1A1, and this may be due to its quantizer having difficulty adapting to binarized DM, and using QALoRA for weight updates might yield suboptimal results compared to full-parameter QAT.As we mentioned in the TBS section of our manuscript, most existing binarization methods struggle to handle the wide activation range and flexible expression of DMs, further highlighting the necessity of TBS. Their optimization strategies may also not be tailored for the image generation tasks performed by DM, which means they only achieve conventional but suboptimal optimization.We perform comprehensive ablation studies for LDM-4 on the LSUN-Bedrooms 256 × 256 dataset to evaluate the effectiveness of each proposed component in BiDM. We evaluate the effectiveness of our proposed SPD and TBS, and the results are presented in Table 3 . Upon separately applying our SPD or TBS methods to LDM, we observed significant improvements compared to the original performance. When the TBS method was incorporated, FID and sFID dropped sharply from 106.62 and 56.61 to 35.23 and 25.13, respectively. Similarly, when the SPD method was added, FID and sFID decreased Other metrics also exhibited substantial improvements. This demonstrates the effectiveness of our approach in continuously approximating the binarized model features to full-precision features during training by introducing a learnable factor α t m and incorporating connections between adjacent time steps. Furthermore, when we combined our two methods and applied them to LDM, we observed an additional improvement compared to the individual application of each method. This further substantiates that performing distillation between full-precision and binarized models at the patch level can significantly enhance the performance of the binarized model. We also conducted additional ablation experiments, and the results are presented in the appendix B. Training Efficiency Analysis. We also explored the training efficiency of BiDM, as the overhead required for the QAT of binarized DMs cannot be overlooked. Theoretical analysis and experimental results show that BiDM achieved significantly better generative results than baseline methods under the same training cost, demonstrating that it not only has a higher upper limit of generative capability but is also relatively efficient in terms of generative performance. See Appendix B for details.Limitations. The techniques of BiDM increase the training time of DMs compared with the original process, and future efforts may thus focus on the efficient quantization process of DMs. 
conclusion.: 
 In this paper, we present BiDM, a novel fully binarized method that pushes the compression of diffusion models to the limit. Based on two observations -activations at different timesteps and the characteristics of image generation tasks -we propose the Timestep-friendly Binary Structure (TBS) and Space Patched Distillation (SPD) from temporal and spatial perspectives, respectively. These methods address the severe limitations in representation capacity and the challenges of highly discrete spatial optimization in full binarization. As the first fully binarized diffusion model, BiDM demonstrates significantly better generative performance than the SOTA general binarization methods across multiple models and datasets. On LSUN-Bedrooms, BiDM achieves an FID of 22.74, greatly surpassing the SOTA method with an FID of 59.44, making it the only method capable of generating visually acceptable samples while achieving up to 28.0× storage savings and 52.7× OPs savings. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper specifies all the details necessary to understand the results in the section A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper ensure the reproducibility of the experiment by fixing random seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper cites the original papers of assets used and introduces the details in the section A.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The details of the new assets are introduced in the section 4 and the section A.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We adopt several classic binarization algorithms, including XNOR [49] , XNOR++ [2] , DoReFa [78] ,and ReActNet [33] , along with the SOTA binarization method, ReSTE [62] as baselines. Additionally, we also include the quantization methods designed for generative models, BBCU [63] and EfficientDM [15] . We extract the output features of TimestepEmbedBlocks from the DM to serve as the targets of TBS and SPD operations. For the CIFAR-10 [27] dataset, We add TBS connections to the outputs of the last 2 timestep embedding blocks and set α init to 0.3. The λ on CIFAR-10 is set to 3e-2. For the LSUN-Bedrooms [72] , LSUN-Churches [72] and FFHQ [25] datasets, We add TBS connections to the outputs of the last 8 timestep embedding blocks and also set α init to 0.3. The λ on these three datasets is set to 1e-2. We conducted extensive experiments on two different types of diffusion models: the latent-space diffusion model LDM and the pixel-space diffusion model DDIM. For the DDIM model, we specifically selected the CIFAR-10 dataset with a resolution of 32 × 32 for our experiments. For the LDM model, our experiments spanned multiple datasets, including the LSUN-Bedrooms, LSUN-Churches and the FFHQ dataset, all with a resolution of 256 × 256. To evaluate the generation quality of the diffusion model, we utilize several evaluation metrics, including Inception Score (IS), Fréchet Inception Distance (FID) [17] , Sliding Fréchet Inception Distance (sFID) [52] , and Precision-and-Recall. After 200,000 iterations of training, we randomly sample and generate 50,000 images from the model and compute the metrics based on reference batches. The reference batches used to evaluate FID and sFID contain all the corresponding datasets. We recorded FID, sFID, and Precision for all tasks and additional IS for CIFAR-10.We utilize OPs as metrics for evaluating theoretical inference efficiency. Taking the convolutional unit as an example, the BOPs for a single computation operation of a single convolution are defined as follows nmk 2 b a b w [69, 71] . It is composed of b w bits for weights, b a bits for activation, n input channels, m output channels, and a k × k convolutional kernel. For the output feature with width w and height h, BOP s ≈ whnmk 2 b a b w . As there might also be full-precision modules in the model, the total OPs of the model are summed up as 1 64 BOP s + F LOP s [3] . All our experiments are conducted on a server with NVIDIA A100 40GB GPU.We conducted more detailed ablation experiments to comprehensively validate our results.Effects of learnable k in TBS. We apply the proposed learnable k to the XNOR baseline. The experimental results shown in Table 5 indicate that this modification can lead to a significant improvement in performance. The model achieved a doubling of improvement in FID, sFID. Their original values were 106.62 and 56.81, respectively, and they decreased to 57.62 and 30.46. The negligible degradation in Recall can be overlooked. Effects of cross-timestep connection in TBS. We investigated the impact of varying the number of TBS connections. Table 6 illustrates that the introduction of TBS cross-timestep connections consistently outperforms models without such connections(n = 0). This validates the efficacy of our cross-timestep linkage strategy based on the high-dimensional feature similarity of LDM. Among the experiments incorporating cross-timestep connections, the models with 1 and 8 connections both achieved equally optimal results. The model with 1 connection demonstrated slightly superior performance in FID and Precision, whereas the model with 8 nodes exhibited marginally better outcomes in sFID and Recall. Effects of SPD. As a general quantization method, real-to-binary [39] suggests that using attention map-based loss during the distillation of a binary model from a full-precision model achieves better results. In contrast, BinaryDM, as the work most closely related to BiDM, directly points out that using L2 loss makes it difficult to align and optimize binary features with full-precision features.These studies indicate that the general L2 loss is inadequate for meeting the optimization needs of binary scenarios. So we also compare our SPD with the commonly used L2 loss function. As shown in Table 7 , by replacing the L2 loss function with patch distillation, the model can achieve better performance. Further Inference Efficiency Analysis. We expand upon the inference process described in Eq.9 and provide a detailed explanation and testing. Since the divisor involved in calculating the mean of A 1,h,w from I c,h,w (i.e., the channel dimension c) can be integrated into k 1,1,3,3 in advance, resulting in k ′1,1,3,3 = k 1,1,. Additionally, α n,1,1,1 derived from W n,c,h,w can also be computed ahead of inference. Therefore, the actual operations involved during inference are as follows:[FP] Original full-precision convolution:• (0) Perform convolution between full-precision I c=448,h=32,w=32 f and full-precision W n=448,c=448,h=32,w=32 f to obtain the full-precision output O 448,32,32 f .[XNOR-Net/BiDM] The inference process for XNOR-Net/BiDM involves the following 6 steps:• Sign operation:(1) Sign operation:• Binary operation:(2) Perform convolution between the binary I 448,32,32 b and the binary W 448,448,3,3 b to obtain the full-precision output O 448,32,32 f . • Full-precision operations:(3) Sum the full-precision I 448,32,32 f across channels to obtain A 1,32,32 .(4) Perform convolution between full-precision A 1,32,32 and k ′1,1,3,3 to obtain O 1,32,32 We utilized the general deployment library Larq [8] on a Qualcomm Snapdragon 855 Plus to test the actual runtime efficiency of the aforementioned single convolution. The runtime results for a single inference are summarized in the Table 8 . Due to limitations of the deployment library and hardware, Baseline achieved a 9.97x speedup, while XNOR-Net / BiDM achieved an 8.07x speedup. Besides, the improvement in generation performance brought by BiDM is even more significant, and we believe that it could achieve better acceleration results in a more optimized environment. (1) TBS includes the learnable convolution of scaling factors (Eq.10) and the cross-time step connection (Eq.12). The increase in training time due to the convolution of trainable scaling factors is minimal, as the depth of the convolution for scaling factors is only 1, and the size of the trainable convolution kernel is only 3 × 3. The cross-time step connection is the primary factor for the increase in training time. Since it requires training α, we introduce this structure during training, so each training sample requires not only noise estimation for T t-1 but also for T t , directly doubling the sampling steps. (2) SPD may lead to a slight increase in training time (an additional 0.18 times), but since we only apply supervision to the larger upsampling/middle/downsampling blocks, the increase is limited.The results in Figure 6 align well with the theoretical analysis mentioned above. BiDM achieved significantly better generative results than baseline methods under the same training iterations, demonstrating that it not only has a higher upper limit of generative capability but is also relatively efficient when considering generative performance.We also tested the FID after uniformly training for 0. We emphasize that the biggest challenge in fully binarizing DM lies in the drop in accuracy. Although BiDM requires a longer training time for the same number of iters, it significantly enhances the quality of generated images, as no other method has been able to produce effective images. Training Loss 9 . Justification:The paper provides open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in the supplemental material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This paper presents BiDM, a method for fully binarizing both the weights and activations of diffusion models. Techincally, BiDM consists of a timestep-friendly binary structure (TBS) to capture the correlation of the activation features over the timesteps and a space patched distillation (SPD) to match the spatial locality of binary features with the full-precision ones during training. Results on common image generation benchmarks were reported. The initial concerns were addressed during rebuttal. I recommend to accept, and encourage the authors to incorporate the discussions into the final version.

Reviews: review: paper_summary: paper_summary: This paper aims to fully binarize weights and activations of diffusion models (DMs) to achieve storage saving and inference acceleration. To this end, the paper proposes timestep-friendly binary structure (TBS), which employs learnable activation binarizers and cross-timestep feature connections to capture the correlation of the activation features over the timesteps. In addition, the paper introduces space patched distillation (SPD) to match the spatial locality of binary features with the full-precision ones during training. The method is tested on some common image generation benchmarks such as LSUN and FFHQ 256x256, CIFAR-10.
strength_weakness: strength_weakness: - The main weakness of the paper is that the proposed method, BiDM, increases the training time of DMS compared to the original process. This issue should be more precise in the paper. The authors should compare the convergence rates of the proposed method and its competitors in terms of wall-clock time and identify which components contribute to the increased training time by breaking down the complexity of all main components of the proposed model. 
- The chosen baselines are quite weak as all of them were originally designed for discriminative tasks such as image classification. The authors should consider stronger baselines dedicated to generative tasks such as [1] and the baselines therein.

Ref:

[1] Xia et al. Basic Binary Convolution Unit for Binarized Image Restoration Network. ICLR 2023
questions: questions: In Table 3, it is not clear why adding SPD alone makes the results worse, whereas combining it with TBS leads to improved performance. Do the authors have any explanation for this phenomenon? This point should be addressed to clarify the underlying reasons for the observed behavior and to provide a deeper understanding of the interaction between SPD and TBS in the proposed method.
limitations: limitations: Yes
paper_summary: paper_summary: The manuscript proposes a method for fully binarizing both the weights and activations of diffusion models, named BiDM. Structurally, it introduces an improved XNOR method for scaling factors of activations and high-level feature connections across time steps, based on observations of existing temporal phenomena. For optimization, the manuscript presents a patch-based attention mechanism distillation method, approached from the perspective of spatial features. Quantitative analysis and generated examples demonstrate that BiDM, as the first work to fully binarize diffusion models, surpasses existing binarization methods, showcasing the potential for deploying diffusion models in resource-constrained scenarios.
strength_weakness: strength_weakness: 1. The description of the cross-time step connection during the training phase is somewhat vague. During the inference phase, the impact of this connection is iteratively attenuated. For example, when α is 0.3, the influence of step T on step T-2 is 0.3×0.3=0.09, rather than 0. However, upon reviewing the source code in the supplementary materials, I found that the authors only consider steps T-1 and T-2 during training. I would like to know if this affects the final accuracy.
2. The scenario assumed in Figure 3 is overly idealized. When only L^t changes, for instance, when L^t is at the lower right of L^(t-1), the weighted average might result in T^(t-1) being further from F^(t-1). How is this situation explained?
3. Although BiDM shows significant improvements over other binarization methods, there remains a substantial performance gap compared to full-precision models. This could still hinder its practical application.
questions: questions: See Weaknesses
limitations: limitations: The manuscript only addresses diffusion models using a U-Net backbone primarily based on convolution. It appears to be inapplicable to diffusion models like DiT, which are primarily based on transformers and not related to U-Net.
paper_summary: paper_summary: This paper proposes BiDM, which focuses on quantizing both weights and activations of diffusion models (DMs). Specifically, the authors introduce:
- Cross-timestep feature connection to enhance the accuracy of noise prediction in binarized DMs.
- Space-patched distillation, a novel variant of distillation loss that emphasizes spatial locality.
strength_weakness: strength_weakness: - The experimental results lack sufficient conviction. Tables 1 and 2 compare BiDM primarily with works focused on quantizing discriminative models, which may not be the most appropriate choice of baseline. It would be more informative to include comparisons with quantization methods designed specifically for diffusion models, such as EfficientDM [cite], given its open-sourced nature and compatibility under W1A1 settings. This comparison is crucial for a comprehensive evaluation.

- Figure.4 may not effectively demonstrate the advantages of paying attention to local information. To better illustrate the effectiveness of Space-Patched Distillation (SPD), the authors should compare experimental results with those obtained using vanilla distillation loss (Mean Squared Error between outputs of a full precision model and a binarized model).

- The proposed binarization method introduces additional convolution computations between two matrices in floating-point, which may not be efficient for hardware deployment. Moreover, the dynamic calculation of matrix $A$ during inference introduces memory access overhead that could be significant in practical deployment scenarios.

- During inference, matrix multiplications (MMs) in the attention mechanism typically consume considerable time. It appears that this work retains these MMs in floating-point arithmetic, which could impact efficiency.

**minor issues**
- Line 93: The citation for BinaryDM is incorrectly linked.
- In Figure 4, the caption states "Ours denotes the self-attention," but in the figure, it's labeled as "SPD."
- The term "self-attention" in Section 3.3 is confusing; perhaps using an alternative term would be clearer.
questions: questions: - Are the Q, K, V in the attention modules binarized during inference?

- During inference, is $A$ in Equation 9 calculated from the inputs?

- In Line 224, the authors claim that "conventional $l_2$  loss makes it difficult for binary DMs to achieve direct matching." Could you provide evidence or reference existing works that have verified this claim?

- Figure 3 is somewhat confusing. Why is $L^{t-1}$ a linear scaling up of $B^{t-1}$? Is this relationship depicted by Equation 12?
limitations: limitations: This paper includes a discussion of limitations.

Comments: title: Thanks for the Response
Re: Further response (2/2)
Re: Further response (1/2)
Thanks for the responses.
Official comment from Reviewer 4HDt
Rebuttal Discussion
comment: My concerns are well addressed and I would like to raise my score by 1 point. Thanks for the authors' detailed responses.
> Q11: Regarding **Q3**, could you elaborate on the computations in Eq.(9)? Specifically, how is the convolution between A and kα executed efficiently?

Yes, in our response to `Q10`, we provided a detailed breakdown of the computation sequence in Eq.(9), including the pre-computed $k'$ and $\alpha$, and the calculations (1)~(6) that need to be performed during inference.

For the convolution between $ A $ and $ k\alpha $, the following calculations are involved:

- (a) Pre-computation before inference, which does not need to be repeated during inference. This includes:
  - The divisor $ c $ (the channel dimension) involved in calculating the mean of $ A^{1,h,w} $ from $ I^{c,h,w} $ in a single convolution
  - $ \alpha^{n,1,1,1} $ obtained from $ W^{n,c,h,w} $
  - $ k'^{1,1,3,3} = \frac{k^{1,1,3,3}}{c} $
- (b) As mentioned in `Q10` (4):
  - Perform the convolution between full-precision $ A^{1,32,32} $ and $ k'^{1,1,3,3} $ to obtain $ O_1^{1,32,32} $
- (c) $ \alpha $ needs to be used in the operation in `Q10` (6): $ O_2^{448,32,32} \odot \alpha^{448,1,1,1} $

The efficiency of our operators (as in XNOR-Net) is reflected in the following:

- The operations in (a) can be pre-computed before inference, which means:
  - Step (3) in `Q10` does not involve high-cost multiplication or division operations.
  - Step (6) in `Q10` does not require computing $ \alpha $ during inference.
- The convolution performed in part (b) has a channel count of only 1.
- Step (5) and (6) in `Q10` are point-wise multiplications rather than matrix multiplications, resulting in a lower computational burden.

In terms of actual inference time, the inference efficiency of our operators is only 0.19x slower than that of a fully binarized convolution in a Baseline without any full-precision scaling factors, even though we use full-precision inference components, consistent with XNOR-Net, to achieve viewable generative performance.
Thank you for your prompt response. We are glad to clarify the inference efficiency of BiDM.

> Q10: Thank you for the authors' response. My concerns regarding **Q1, Q2, Q4, Q5, Q6, Q8** have been satisfactorily addressed. Nonetheless, I remain uncertain about the deployment efficiency, particularly given that practical inference speeds often diverge from theoretical operations per second (OPs). The dynamic *min-max* operations, despite their minimal theoretical OPs, may introduce substantial latency in practical deployment, especially with smaller model sizes where matrix multiplications are less significant. I am keen to know if there are effective tools available to verify this.

In BiDM, there are no dynamic *min-max* operations; instead, our design involves only basic arithmetic operations such as convolution, matrix multiplication, addition and summation, and matrix dot multiplication. Additionally, as stated in Eq.(9) of our manuscript and in our response to `Q3`, the operators in BiDM behave the same as those in XNOR-Net[1] during inference, which corresponds to Eq.(11) in the XNOR-Net's original paper.

Following your suggestion, we expand upon the inference process described in `Eq.(9)` and provide a detailed explanation and testing. Since the divisor involved in calculating the mean of $ A^{1,h,w} $ from $ I^{c,h,w} $ (i.e., the channel dimension $ c $) can be integrated into $ k^{1,1,3,3} $ in advance, resulting in $ k'^{1,1,3,3} = \frac{k^{1,1,3,3}}{c} $. Additionally, $ \alpha^{n,1,1,1} $ derived from $ W^{n,c,h,w} $ can also be computed ahead of inference. Therefore, the actual operations involved during inference are as follows:

[FP] Original full-precision convolution:

- (0) Perform convolution between full-precision $I_f^{c=448,w=32,h=32}$ and full-precision $W_f^{n=448,c=448,w=32,h=32}$ to obtain the full-precision output $O_f^{448,32,32}$.

[XNOR-Net/BiDM] The inference process for XNOR-Net/BiDM involves the following 6 steps:

- Sign operation:
  - (1) Compute $I_b^{448,32,32} = \text{sign}(I_f^{448,32,32})$.
- Binary operation:
  - (2) Perform convolution between the binary $I_b^{448,32,32}$ and the binary $W_b^{448,448,3,3}$ to obtain the full-precision output $O_f^{448,32,32}$.
- Full-precision operations:
  - (3) Sum the full-precision $I_f^{448,32,32}$ across channels to obtain $A^{1,32,32}$.
  - (4) Perform convolution between full-precision $A^{1,32,32}$ and $k'^{1,1,3,3}$ to obtain $O_1^{1,32,32}$.
  - (5) Pointwise multiply $O_f^{448,32,32}$ by $O_1^{1,32,32}$ to obtain the full-precision output $O_2^{448,32,32}$
  - (6) Pointwise multiply $O_2^{448,32,32}$ by $\alpha^{448,1,1}$ to obtain the final full-precision output $O^{448,32,32}$

We utilized the general deployment library Larq[2] on a Qualcomm Snapdragon 855 Plus to test the actual runtime efficiency of the aforementioned single convolution. The runtime results for a single inference are summarized in the table below. Due to limitations of the deployment library and hardware, Baseline achieved a 9.97x speedup, while XNOR-Net/BiDM achieved an 8.07x speedup. Besides, the improvement in generation performance brought by BiDM is even more significant, and we believe that it could achieve better acceleration results in a more optimized environment.

|                  |   (0)    | (1)+(2) |  (3)   |  (4)   | (5)  | (6)  | Runtime($\mu s$/convolution) | FID$\downarrow$ |
| :--------------: | :------: | :-----: | :----: | :----: | :--: | :--: | ---------------------------: | --------------: |
|        FP        | 176371.0 |         |        |        |      |      |                     176371.0 |            2.99 |
| Baseline(DoReFa) |          | 17695.2 |        |        |      | 4.3  |                      17699.5 |          188.30 |
| XNOR-Net / BiDM  |          | 17695.2 | 2948.8 | 1133.3 | 83.2 | 4.3  |                      21864.8 |           22.74 |

[1] Rastegari et al. Xnor-net: Imagenet classification using binary convolutional neural networks. ECCV 2016

[2] "LarQ". https://github.com/larq/larq
Thanks for the authors' response. My concerns regarding **Q1, Q2, Q4, Q5, Q6, Q8** have been satisfactorily addressed. Nonetheless, I remain uncertain about the deployment efficiency, particularly given that practical inference speeds often diverge from theoretical operations per second (OPs). The dynamic *min-max* operations, despite their minimal theoretical OPs, may introduce substantial latency in practical deployment, especially with smaller model sizes where matrix multiplications are less significant. I am keen to know if there are effective tools available to verify this.

Regarding **Q3**, could you elaborate on the computations in Eq.(9)? Specifically, how is the convolution between $A$ and $ k\alpha $ executed efficiently?
Thank you for the detailed rebuttal and the additional results provided. I appreciate the effort in addressing the issues raised. Due to the novelty and potential broad applications, I am willing to vote for acceptance for this paper.
Thanks the author(s) for the rebuttal.

As most of my concerns have been addressed, I will increase score by 1 point.
Dear Reviewer AtKg,

Thank you for your thorough review of our work, BiDM, during the review stage. We have carefully considered your concerns during the rebuttal stage and made revisions to the relevant sections of the manuscript.

We are looking forward you to reviewing our response and we are also willing to answer any further questions.

Best regards,

Authors
Dear Reviewer cNoX,

Thank you for your thorough review of our work, BiDM, during the review stage. We have carefully considered your concerns during the rebuttal stage and made revisions to the relevant sections of the manuscript.

We are looking forward you to reviewing our response and we are also willing to answer any further questions.

Best regards,

Authors
Dear Reviewer 4HDt,

Thank you for your thorough review of our work, BiDM, during the review stage. We have carefully considered your concerns during the rebuttal stage and made revisions to the relevant sections of the manuscript.

We are looking forward you to reviewing our response and we are also willing to answer any further questions.

Best regards,

Authors
Dear Reviewers,

The authors have provided a response to the comments. Please respond to the rebuttal actively.

Best,
AC