Title: Structural Pruning for Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Model

Network Pruning

Model Compression

Efficient Deep Learning
Abstract: Generative modeling has recently undergone remarkable advancements, primarily propelled by the transformative implications of Diffusion Probabilistic Models (DPMs). The impressive capability of these models, however, often entails significant computational overhead during both training and inference. To tackle this challenge, we present Diff-Pruning, an efficient compression method tailored for learning lightweight diffusion models from pre-existing ones, without the need for extensive re-training. The essence of Diff-Pruning is encapsulated in a Taylor expansion over pruned timesteps, a process that disregards non-contributory diffusion steps and ensembles informative gradients to identify important weights. Our empirical assessment, undertaken across several datasets highlights two primary benefits of our proposed method: 1) Efficiency: it enables approximately a 50\% reduction in FLOPs at a mere 10% to 20% of the original training expenditure; 2) Consistency: the pruned diffusion models inherently preserve generative behavior congruent with their pre-trained models.

Structural Pruning for Diffusion Models

Abstract

Generative modeling has recently undergone remarkable advancements, primarily propelled by the transformative implications of Diffusion Probabilistic Models (DPMs). The impressive capability of these models, however, often entails significant computational overhead during both training and inference. To tackle this challenge, we present Diff-Pruning , an efficient compression method tailored for learning lightweight diffusion models from pre-existing ones, without the need for extensive re-training. The essence of Diff-Pruning is encapsulated in a Taylor expansion over pruned timesteps , a process that disregards non-contributory diffusion steps and ensembles informative gradients to identify important weights. Our empirical assessment, undertaken across several datasets highlights two primary benefits of our proposed method: 1) Efficiency: it enables approximately a 50% reduction in FLOPs at a mere 10% to 20% of the original training expenditure; 2) Consistency : the pruned diffusion models inherently preserve generative behavior congruent with their pre-trained models. Code is available at https://github.com/VainF/Diff-Pruning .

1 Introduction

Generative modeling has undergone significant advancements in the past few years, largely propelled by the advent of Diffusion Probabilistic Models (DPMs) [18, 41, 37]. These models have derived numerous applications ranging from text-to-image generation [40], image editing [58], image translation[45], and even discriminative tasks [2, 1]. The incredible power of DPMs, however, often comes at the expense of considerable computational overhead during both training [49] and inference [43]. This trade-off between performance and efficiency presents a critical challenge in the broader application of these models, particularly in resource-constrained environments.

In the literature, huge efforts have been made to improve diffusion models, which primarily revolved around three broad themes: improving model architectures [41, 39, 52], optimizing training methods [49, 11] and accelerating sampling [46, 43, 12]. As a result, a multitude of well-trained diffusion models has been created in these valuable works, showcasing their potential for various applications [48]. However, the notable challenge still remains: the absence of a general compression method that enables the efficient reuse and customization of these pre-existing models without heavy re-training. Overcoming this gap is of paramount importance to fully harness the power of pre-trained diffusion models and facilitate their widespread application across different domains and tasks.

In this work, we demonstrate the remarkable effectiveness of structural pruning [23, 8, 26, 4] as a method for compressing diffusion models, which offers a flexible trade-off between efficiency and quality. Structural pruning is a classic technique that effectively reduces model sizes by eliminating redundant parameters and sub-structures from networks. While it has been extensively studied in discriminative tasks such as classification [16], detection [54], and segmentation [13], applying structural pruning techniques to Diffusion Probabilistic Models poses unique challenges that necessitate

a rethinking of traditional pruning strategies. For example, the iterative nature of the generative process in DPMs, the models' sensitivity to small perturbations in different timesteps, and the intricate interplay in the diffusion process collectively create a landscape where conventional pruning strategies often fall short.

To this end, we introduce a novel approach called Diff-Pruning , explicitly tailored for the compression of diffusion models. Our method is motivated by the observation in previous works [41, 52] that different stages in the diffusion process contribute variably to the generated samples. At the heart of our method lies a Taylor expansion over pruned timesteps, which deftly balances the image content, details, and the negative impact of noisy diffusion steps during pruning. Initially, we show that the objective of diffusion models at late timesteps ( t → T ) prioritize the high-level content of the generated images during pruning, while the early ones ( t → 0 ) refine the images with finer details. However, it is also observed that, when using Taylor expansion for pruning, the noisy stages with large t can not provide informative gradients for importance estimation and can even harm the compressed performance. Therefore, we propose to model the trade-off between contents, details, and noises as a pruning problem of the diffusion timesteps, which leads to an efficient and flexible pruning algorithm for diffusion models.

Through extensive empirical evaluations across diverse datasets, we demonstrate that our method achieves substantial compression rates while preserving and in some cases even improving the generative quality of the models. Our experiments also highlight two significant features of DiffPruning: efficiency and consistency. For example, when applying our method to an off-the-shelf diffusion model pre-trained on LSUN Church [57], we achieve an impressive compression rate of 50% FLOPs, with only 10% of the training cost required by the original models, equating to 0.5 million steps compared to the 4.4 million steps of the pre-existing models. Furthermore, we have thoroughly assessed the generative behavior of the compressed models both qualitatively and quantitatively. Our evaluations demonstrate that the compressed model can effectively preserve a similar generation behavior as the pre-trained model, meaning that when provided with the same inputs, both models yield consistent outputs. Such consistency further reveals the practicality and reliability of Diff-Pruning as a compression method for diffusion models.

In summary, this paper introduces Diff-Pruning as an efficient method for compressing Diffusion Probabilistic Models, which is able to achieve compression with only 10% to 20% of the training costs compared to pre-training. This work may serve as an initial baseline and provide a foundation for future research aiming to enhance the quality and consistency of compressed diffusion models.

2 Ralted Works

Efficient Diffusion Models The existing methodologies principally address the efficiency issues associated with diffusion models via three primary strategies: the refinement of network architectures [41, 52, 37], the enhancement of training procedures [11, 49], and the acceleration of sampling [18, 27, 12]. Diffusion models typically employ U-Net models as denoisers, of which the efficiency can be improved via the introduction of hierarchical designs [40] or by executing the training within a novel latent space [41, 19, 25]. Recent studies also suggest integrating more efficient layers or structures into the denoiser to bolster the performance of the U-Net model [52, 39], thereby facilitating superior image quality learning during the training phase. Moreover, a considerable number of studies concentrate on amplifying the training efficiency of diffusion models, with some demonstrating that the diffusion training can be expedited by modulating the weights allocated to distinct timesteps [43, 11]. The training efficiency can also be advanced by learning diffusion models at the patch level [49]. In addition, some approaches underscore the efficiency of sampling, which typically does not necessitate the retraining of diffusion models [27]. In this area, numerous studies aim to diminish the required steps through methods such as early stopping [34] or distillation [43].

Network Pruning In recent years, the field of network acceleration [59, 3, 20, 53, 51, 29, 30] has seen notable progress through the deployment of network pruning techniques [31, 16, 33, 23, 14, 5, 15]. The taxonomy of pruning methodologies typically bifurcates into two main categories: structural pruning [23, 6, 56, 26, 56] and unstructured pruning [38, 7, 44, 22]. The distinguishing trait of structural pruning is its ability to physically eliminate parameters and substructures from networks, while unstructured pruning essentially masks parameters by zeroing them out [8, 4]. However, the preponderance of network pruning research is primarily focused on discriminative

tasks, particularly classification tasks [16]. A limited number of studies have ventured into examining the effectiveness of pruning in generative tasks, such as GAN compression [24, 47]. Moreover, the application of structural pruning techniques to Diffusion Probabilistic Models introduces unique challenges that demand a reevaluation of conventional pruning strategies. In this work, we introduce the first dedicated method explicitly designed for pruning diffusion models, which may serve as a useful baseline for future works.

3 Diffusion Model Objectives

Given a data distribution q ( x ) , diffusion models aim to model a generative distribution p θ ( x ) to approximate q ( x ) , taking the form

And x 1 , ..., x T refer to the latent variables, which contribute to the joint distribution p θ ( x 0: T ) with learned Gaussian transitions p θ ( x t -1 | x t ) = N ( x t -1 ; µ θ ( x t , t ) , Σ θ ( x t , t )) . Diffusion Models involve two opposite processes: a forward (diffusion) process q ( x t | x t -1 ) = N ( x t ; √ 1 -β t x t -1 , β t I ) that adds noises to the x t -1 , based on a pre-defined variance schedule β 1: T ; and a reverse process q ( x t -1 | x t ) which "denoises" the observation x t to get x t -1 . Using the notation α t = 1 -β t and ¯ α t = ∏ t s =1 α s , DDPMs [18] trains a noise predictor with the objective:

where ϵ is a random noise drawn from a fixed Gaussian distribution and ϵ θ refers to a learned noise predictor, which is usually an U-Net autoencoder [42] in practice. After training, synthetic images x 0 can be sampled through an iterative process from a noise x T ∼ N ( 0 , 1 ) with the formular:

where z ∼ N ( 0 , I ) for steps t > 1 and z = 0 for t = 1 . In this work, we aim to craft a lightweight ϵ θ ′ by removing redundant parameters of ϵ θ , which are expected to produce similar x 0 while the same x T are presented.

4 Structrual Pruning for Diffusion Models

Given the parameter θ of a pre-trained diffusion model, our goal is to craft a lightweight θ ′ by removing sub-structures from the network following existing paradigms [35, 8]. Without loss of generality, we assume that the parameter θ is a simple 2-D matrix, where each sub-structure θ i = [ θ i 0 , θ i 1 , ..., θ iK ] is a row vector that contains K scalar parameters. Structural pruning aims to find a sparse parameter matrix θ ′ that maximally preserves the original performance. Thus, a natural choice is to optimize the loss disruption caused by pruning:

The term | θ ′ | 0 denotes the L-0 norm of the parameters, which counts the number of non-zero row vectors, and s represents the sparsity of the pruned model. Nevertheless, due to the iterative nature intrinsic to diffusion models, the training objective, denoted by L , can be perceived as a composition of T interconnected tasks: {L 1 , L 2 , ..., L T } . Each task affects and depends on the others, thereby posing a new challenge distinct from traditional pruning problems, which primarily concentrate on optimizing a single objective. In light of the pruning objective as defined in Equation 4, we initially delve into the individual contributions of each loss component, L t in pruning, and subsequently propose a tailored method, Diff-Pruning, designed for diffusion models pruning.

Taylor Expansion at L t Initially, we need to model the contribution of L t for structural pruning. This work leverages Taylor expansion [35] on L t to linearly approximate the loss disruption:

X

!

X

"#

q 𝑥

!

𝑥

!"#

)

X

"#$#

X

%

𝑝

$

𝑥

!"#

𝑥

!

)

X

"&

…

X

"%

…

…

Preserving Contents

Preserving Details

Contents + Details

𝐼

$,!&

= |	𝜃 ⋅ ∇

$

ℒ

!&

|

Importance

𝐼

$

= |𝜃 ⋅ ∑

!

𝛼

!

⋅ ∇

$

ℒ

!

|

𝐼

$,!'

= |	𝜃 ⋅ ∇

$

ℒ

!'

|

Pruned Noisy Steps

𝛼

!

= 0

→

->

Taylor expansion offers a robust framework for network pruning, as it can estimate the loss disruption using first-order gradients. To evaluate the importance of an individual weight θ ik , we can simply set θ ′ ik = 0 in Equation 5, which results in the following importance criterion:

where ∇ θ ik refer to ∇ θ ik L t ( θ , x ) . In structural pruning, we aim to remove the entire vector θ ′ i concurrently. The standard Taylor expansion for multiple variables, as described in the literature [9], advocates using | ∑ k θ ik · ∇ θ ik L t ( θ , x ) | for importance estimation. This method exclusively takes into account the loss difference between the initial state θ and the final states θ ′ . However, considering the iterative nature of diffusion models, even minor fluctuations in loss can influence the final generation results. To this end, we propose to aggregate the influence of removing each parameter as the final importance. This modification models cumulative loss disturbance induced by each θ ik 's removal and leads to a slightly different score function for structural pruning:

In the following sections, we utilize Equation 7 as the importance function to identify non-critical parameters in diffusion models.

The Contribution of L t . With the Taylor expansion framework, we further explore the contribution of different loss terms {L 1 , ..., L T } during pruning. We consider the functional error δ t = ϵ θ ′ ( x , t ) -ϵ θ ( x , t ) which represents the prediction error for the same inputs at time step t . The reverse process allows us to exam the effects δ t → 0 on the generated images x 0 by iteratively applying the Equation 3 starting from ϵ θ ′ ( x , t ) = ϵ θ ( x , t ) + δ t . At the t -1 step, it leads to the error δ t -1 derived as:

′

This error has a direct impact on the subsequent input, given by x t -1 = x t -1 + δ t -1 . By checking Equation 3, we can observe that these perturbed inputs can further trigger a chained effect through both 1 √ α t -1 x ′ t -1 and -1 √ α t -1 β t -1 √ 1 -¯ α t -1 ϵ θ ′ ( x ′ t -1 , t -1) . In the first term, the distortion progressively amplifies by a factor 1 √ α t -1 > 1 , which means that this error will be enhanced throughout the generation process. Regarding the second term, pruning affects both the functionality parameterized by θ ′ and the inputs x ′ t -1 , which contributes to the final results in a nonlinear and more complicated manner, resulting in a more substantial disturbance on the generated images.

Algorithm 1 Diff-Pruning

As a result, prediction errors occurring at larger t tend to have a larger impact on the images due to the chain effect, which might change the global content of generated images. Conversely, smaller t values focus on refining the images with relatively small modifications. These findings align with our empirical examination using Taylor expansion as illustrated in Figure 1, as well as the observation in previous works [18, 52], which shows that diffusion models tend to generate object-level information at larger t values and fine-tune the features at smaller ones. To this end, we model the pruning problem as a weighted trade-off between contents and details by introducing α t , which acts as a weighting variable for different timesteps t . Nevertheless, unconstrained reweighting can be highly inefficient, as it entails exploring a large parameter space for α t and requires at least T forward-backward passes for Taylor expansion. This results in a vast sampling space and can lead to inaccuracies in the linear approximation. To address this issue, we simplify the re-weighting strategy by treating it as a 'pruning problem', where α t takes the value of either 0 or 1 for all steps, allowing us to only leverage partial steps for pruning. The general importance metric is modeled as the following.

Taylor Score over Pruned Timesteps. In Equation 9, we try to remove some 'unimportant' timesteps in the diffusion process so as to enable an efficient and stable approximation for partial steps. Our empirical results, as will be discussed in the experiments, indicate two key findings. Firstly, we note that the timesteps responsible for generating content are not exclusively found towards the end of the diffusion process ( t → T ). Instead, there are numerous noisy and redundant timesteps that contribute minorly to the overall generation, which is similar to the observations in the related work [34]. Secondly, we discovered that employing the full-step objective can sometimes yield suboptimal results compared to using a partial objective. We attribute this negative impact to the presence of converged gradients in the noisy steps ( t → T ). Taylor approximation in Equation 5 comprises both first-order gradients and higher-order terms. When the loss L t converges, the loss curve is predominantly influenced by the higher-order terms rather than the first-order gradients we utilize. Our experiments on several datasets and diffusion models show that the loss term L t rapidly approaches 0 as t → T . For example in Figure 5, the relative loss L t L max of a pre-trained diffusion model for CIFAR-10 decreases to 0.05 when t = 250 . Consequently, a full Taylor expansion can accumulate a considerable amount of noisy gradients from these converged or unimportant steps, resulting in an inaccurate estimation of weight importance.

Considering the significant impact of larger timesteps, it is necessary to incorporate them for importance estimation. To address this problem, Equation 9 naturally provides a simple and practical thresholding strategy for pruning. To achieve this, we introduce a threshold parameter T based on the relative loss L t L max . Those timesteps with a relative loss below this threshold, i.e., L t L max < T , are considered uninformative and are disregarded by setting α t = 0 , which yields the finalized importance score:

In practice, we need to select an appropriately large value for T to strike a well-balanced preservation of details and content, while also avoiding uninformative gradients from noisy loss terms. The full algorithm is summarized in Alg. 1.

5 Experiments

5.1 Settings

Datasets and Models The efficacy of Diff-Pruning is empirically validated across six diverse datasets, including CIFAR-10 (32 × 32)[21], CelebA-HQ (64 × 64)[32], LSUN Church (256 × 256), LSUN Bedroom (256 × 256) [57] and ImageNet-1K (256 × 256). We focus on two popular DPMs in our experiments, i.e., Denoising Diffusion Probability Models (DDPMs) [18] and Latent Diffusion Models (LDMs) [41]. For the sake of reproducibility, we utilize off-the-shelf DPMs from [18] and [41] as pre-trained models and prune these models in a one-shot fashion[23].

Evaluation Metrics In this paper, we concentrate primarily on three types of metrics: 1) Efficiency metrics, which include the number of parameters (#Params) and Multiply-Add Accumulation (MACs); 2) Quality metric, namely the Frechet Inception Distance (FID) [17]; and 3) Consistency metric, represented by Structural Similarity (SSIM) [50]. Unlike previous generative tasks that lacked reference images, we employ the SSIM index to evaluate the similarity between images generated by pre-trained models and pruned models, given identical noise inputs. We deplpy a 250-step DDIM sampler [46] for ImageNet and a 100-step DDIM sampler for other experiments.

5.2 An Simple Benchmark for Diffusion Pruning

Scratch Training v.s. Pruning. Table 1 shows our results on CIFAR-10 and CelebA-HQ. The first baseline method that piques our interest is scratch training. Numerous studies on network pruning [10] suggest that training a compact network from scratch can be a formidable competitor. To ensure a fair comparison, we create randomly initialized networks with the same architecture as the pruned ones for scratch training. Our results reveal that scratch training demands relatively more steps to reach convergence. This suggests that training lightweight models from scratch may not be an efficient and economical approach, given its training cost is comparable to that of pre-trained models. Conversely, we observe that all pruning methods are able to converge within approximately 100K steps and outperform scratch training in terms of FID and SSIM scores. Thus, pruning emerges as a potent technique for compressing pre-trained Diffusion Models.

Pruning Criteria. A significant aspect of network pruning is the formulation of pruning criteria, which serve to identify superfluous parameters within networks. Due to the absence of dedicated work on Diffusion model pruning, we adapted three basic pruning methods from discriminative tasks: random pruning, magnitude-based pruning [16], and Taylor-based pruning [36], which we refer to as Random, Magnitude, and Taylor respectively in subsequent sections. For a given parameter θ , Random assigns importance scores derived from a uniform distribution to each θ i randomly, denoted as I ( θ ) ∼ U (0 , 1) . This results in a straightforward baseline devoid of any prior or bias, and has been shown to be a competitive baseline for pruning [28]. Magnitude subscribes to the 'smaller-normless-informative' hypothesis [23, 55], modelling the weight importance as I ( θ ) = | θ | . In contrast, Taylor is a data-driven criterion that measures importance as I ( θ , x ) = | θ · ∇ θ L ( x, θ ) | , which aims to minimize loss change as discussed in our method. As shown in 1, an intriguing phenomenon is that these three baseline methods do not maintain a consistent ranking on these two datasets. For

instance, while Magnitude achieves the best FID performance among the three on CIFAR-10, it performs poorly on CelebA datasets. In contrast, our method delivers stable improvements over baseline methods, demonstrating superior performance on both datasets. Remarkably, our method even surpasses the pre-trained model on CelebA-HQ, with only 100K optimizations. Nonetheless, performance degradation is observed on CIFAR-10, which can be attributed to its more complex scene and a larger number of categories.

5.3 Pruning at Higher Resolutions

DDPMs on LSUN To further validate the efficiency and effectiveness of our proposed DiffPruning, we perform pruning experiments on two 256 × 256 scene datasets, LSUN Church, and LSUN Bedroom [57]. The pre-trained models from [18] require around 2.4M and 4.4M training steps, which can be quite time-consuming in practice. We demonstrate that Diff-Pruning can compress these pre-existing models using only 10% of the standard training resources. We report the number of parameters, MACs, and FID scores in Table 2, and compare the pruned methods to the pre-trained ones as well as those trained from scratch. Results show that the pruned model converges with a passable FID score in 10% of the standard steps, while a model trained from scratch is still severely under-fitted. Nevertheless, we also discover that compressing a model trained on large-scale datasets, such as LSUN Bedroom, which contains 300K images, proves to be quite challenging with a very limited number of training steps. We show that, in the supplementary materials, the FID scores can be further improved with more training steps. Moreover, we also visualize the generated images in Figure 2 and report the single-image SSIM score to measure the similarity of generated images. By nature, the pruned models can preserve similar generation capabilities since they inherit most parameters from the pre-trained models.

Conditional LDMs on ImageNet Table 3 and Figure 3 illustrate the pruning results of LDM pre-trained on ImageNet-1K. An LDM consists of an encoder, a decoder, and a U-Net model. Around

SSIM = 0.814

SSIM = 0.861

SSIM = 0.817

SSIM = 0.792

SSIM = 0.821

SSIM = 0.952

SSIM = 0.878

SSIM = 0.876

SSIM = 0.907

SSIM = 0.912

SSIM = 0.907

SSIM = 0.881

400M parameters come from the U-Net architecture and only 55M from the autoencoder. Therefore, we mainly focus on the pruning of the U-Net model. We used the threshold T = 0 . 1 to ignore those converged layers and make the pruning process more efficient. With T = 0 . 1 , only 534 steps participate in the pruning process. After importance estimation, we apply a pre-defined channel sparsity of 30% to all layers, leading to a lightweight U-Net with 189.43M parameters. Finally, we finetune the pruned model for only 4 epochs with the official training script, with a scaled learning rate of 0 . 1 × lr base .

5.4 Ablation Study

Pruned Timesteps. First, we conduct an empirical study evaluating the partial Taylor expansion over pruned timesteps. This approach prioritizes steps with larger gradients and strives to preserve as much content and detail as possible, thereby enabling more accurate and efficient pruning. The impacts of timestep pruning are demonstrated in Figure 5. We seek to prune a pre-trained diffusion model over a range of steps, spanning from 50 to 1000, after which we utilize the SSIM metric to gauge the distortion induced by pruning. In diffusion models, earlier steps ( t → 0 ) usually present larger gradients compared to the later ones ( t → T ) [41]. This inherently leads to gradients that have reached a convergence when t is large. In the CIFAR-10 dataset, we find that the optimal SSIM score

Threshold

T

= 0

.

00

T

T

T

T

= 0

= 0

= 0

= 0

.

.

.

.

01

02

05

10

Thresholding

Steps

FID

1000

707

433

244

↓

5.49

5.41

5.44

5.29

SSIM

↑

0.932

0.932

0.931

0.932

127

5.31

0.931

can be attained at around 250 steps, and adding more steps can slightly deteriorate the quality of the synthetic images. This primarily stems from the inaccuracy of the first-order Taylor expansion at converged points, where the gradient no longer provides useful information and can even distort informative gradients through accumulation. However, we observe that the situation differs slightly with the CelebA dataset, where more steps can be beneficial for importance estimation.

Pruning Ratios. Table 4 presents the #Params, MACs, FID, and SSIM scores of models subjected to various pruning ratios based on MACs. Notably, our findings reveal that, unlike CNNs employed in discriminative models, diffusion models exhibit a significant sensitivity to changes in model size. Even a modest pruning ratio of 16% leads to a noticeable degradation in FID score (4.19 → 4.62). In classification tasks, a perturbation in loss does not necessarily impact the final accuracy; it may only undermine prediction confidence while leaving classification accuracy unaffected. However, in generative models, the FID score is very sensitive, making it more susceptible to domain shift.

Thresholding. In addition, we conducted experiments to investigate the impact of the thresholding parameter T . Setting T = 0 corresponds to a full Taylor expansion at all steps, while T > 0 denotes pruning of certain timesteps during importance estimation. The quantitative findings presented in Table 5 align with the SSIM results depicted in Figure 5. Notably, Diff-Pruning attains optimal performance when the quality of generated images reaches its peak. For datasets such as CIFAR-10, we observed that a 200-step Taylor expansion is sufficient to achieve satisfactory results. Besides, using a full Taylor expansion, in this case, can be detrimental, as it accumulates noisy gradients over approximately 700 steps, which obscures the correct gradient information from earlier steps.

w/o Pruning

SSIM = 1.000

Random

SSIM = 0.744

Magnitude

SSIM = 0.391

SSIM = 0.758

Ours

SSIM = 0.905

Ours (

𝓣

=0)

SSIM = 0.857

0

200

400

600

800

1000

Steps

0.80

0.82

0.84

SSIM

0.0

0.5

1.0

Relative Loss

Threshold

CIFAR-10

SSIM

Relative Loss

0

200

400

600

800

1000

Steps

0.6

0.7

0.8

SSIM

0.0

0.5

1.0

Relative Loss

Threshold

CeleA-HQ

SSIM

Relative Loss

Visualization of Different Importance Criteria. Figure 4 visualizes the images generated by pruned models using different pruning criteria, including the proposed method with T = 0 (w/o timestep pruning) and T > 0 . The SSIM scores of the generated samples are reported for a quantitative comparison. The Diff-Pruning method with T > 0 achieves superior visual quality, with an SSIM score of 0.905 after pruning. It is observed that employing more timesteps in our method could have a negative impact, leading to greater distortion in both textures and contents.

6 Conclusion

This work introduces Diff-Pruning, a dedicated method for compressing diffusion models. It utilizes Taylor expansion over pruned timesteps to identify and remove non-critical parameters. The proposed approach is capable of crafting lightweight yet consistent models from pre-trained ones, incurring only about 10% to 20% of the cost compared to pre-training. This work may set an initial baseline for future research that aims at improving both the generation quality and the consistency of pruned diffusion models.

Acknowledgment

This research is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006).

References

Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390 , 2021.

Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788 , 2022.

Tianyi Chen, Luming Liang, Tianyu Ding, and Ilya Zharkov. Towards automatic neural architecture search within general super-networks. arXiv preprint arXiv:2305.18030 , 2023.

Tianyi Chen, Luming Liang, Tianyu Ding, Zhihui Zhu, and Ilya Zharkov. Otov2: Automatic, generic, user-friendly. arXiv preprint arXiv:2303.06862 , 2023.

Ting-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1518-1528, 2020.

Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4943-4953, 2019.

Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems , 30, 2017.

Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. The IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Gerald B Folland. Higher-order derivatives and taylor's formula in several variables. Preprint , pages 1-4, 2005.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. arXiv preprint arXiv:2303.09556 , 2023.

En hao Liu, Xuefei Ning, Zi-Han Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. International Conference on Machine Learning , 2023.

Wei He, Meiqing Wu, Mingfu Liang, and Siew-Kei Lam. Cap: Context-aware pruning for semantic segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 960-969, 2021.

Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4340-4349, 2019.

Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV) , pages 784-800, 2018.

Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 1389-1397, 2017.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIII , pages 274-289. Springer, 2022.

Yongcheng Jing. Efficient Representation Learning With Graph Neural Networks . PhD thesis, 2023.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv:1906.06307 , 2019.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 , 2016.

Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5284-5294, 2020.

Daochang Liu, Qiyue Li, Anh-Dung Dinh, Tingting Jiang, Mubarak Shah, and Chang Xu. Diffusion action segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10139-10149, 2023.

Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning , pages 7021-7032. PMLR, 2021.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal C Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In International Conference on Learning Representations, ICLR 2022 , 2022.

Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. In Advances in Neural Information Processing Systems , 2022.

Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision , pages 2736-2744, 2017.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision , pages 5058-5066, 2017.

Zhaoyang Lyu, Xudong Xu, Ceyuan Yang, Dahua Lin, and Bo Dai. Accelerating diffusion models via early stop of the diffusion process. arXiv preprint arXiv:2205.12524 , 2022.

Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11264-11272, 2019.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 , 2016.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: a far-sighted alternative of magnitudebased pruning. arXiv preprint arXiv:2002.04809 , 2020.

Hao Phung, Quan Dao, and Anh Tran. Wavelet diffusion models are fast and scalable image generators. arXiv preprint arXiv:2211.16152 , 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems , 33:20378-20389, 2020.

Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358 , 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Duc Minh Vo, Akihiro Sugimoto, and Hideki Nakayama. Ppcd-gan: Progressive pruning and classaware distillation for large-scale conditional gans compression. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2436-2444, 2022.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/ huggingface/diffusers , 2022.

Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-efficient training of diffusion models. arXiv preprint arXiv:2304.12526 , 2023.

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004.

Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In European Conference on Computer Vision , 2022.

Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. In Advances in Neural Information Processing Systems , 2022.

Lewei Yao, Renjie Pi, Hang Xu, Wei Zhang, Zhenguo Li, and Tong Zhang. Joint-detnas: upgrade your detector with nas, pruning and dynamic distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10175-10184, 2021.

Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.00124 , 2018.

Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems , 32, 2019.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633 , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Diff-Pruning, an efficient compression method for learning lightweight diffusion models from pre-existing ones. DPMs have shown impressive capabilities in generative modeling, but they often come with significant computational overhead during training  and inference. Diff-Pruning addresses this challenge by introducing a  Taylor expansion over pruned timesteps, which eliminates  non-contributory diffusion steps and combines informative gradients to  identify important weights. The authors empirically evaluate the proposed method on four diverse datasets, highlighting two primary  benefits: 1) Efficiency, with a 50% reduction in FLOPs at a fraction of  the original training expenditure, and 2) Consistency, as the pruned  diffusion models retain generative behavior congruent with their pre-trained counterparts.\n\nSTRENGTHS:\n- The paper introduces a novel method, Diff-Pruning,  **specifically designed** for compressing diffusion models. \n- The authors conduct empirical assessments on  four diverse datasets, providing a comprehensive analysis of the  proposed method's performance. The evaluation demonstrates the  effectiveness of Diff-Pruning in terms of efficiency and consistency.\n- The paper appears to be well-written overall.\n\nWEAKNESSES:\n- Provide additional details about the pruning process: Although this paper describes the essence of diffo-pruning as Taylor expansions at pruning time steps, providing more specific details about the pruning process itself would enhance the clarity of the proposed method. Providing step-by-step explanations or pseudocode algorithms would help readers understand and replicate this approach.\n\n- In Eq. (4), (5), (6), (10), (11), the symbols $|\\cdot|$, $||\\cdot||$, and $||\\cdot||_0$ appear. Are they representing the same thing? The author needs to carefully check if the symbols in the formulas are correct. Additionally, the notation $\\nabla L_t(\\mathbf{\\theta})(\\mathbf{\\theta}^\\prime - \\theta)$ should be reviewed for its validity since $\\theta$ is a vector. Furthermore, what does the \"$\\cdot$\" symbol represent in Equation (6)? Is it the dot product? If so, why is \"$|\\cdot|$\" included?\n\n- I am unsure how Eq. (6) is derived from Eq. (5). Why can it serve as an importance criterion? Does it have a direct relationship with the model's performance? If so, the author should demonstrate the relationship between this criterion and the performance of the diffusion model, such as through correlation analysis, and so on.\n\n- Why does the pruned model exhibit better performance than the pretrained model? Can the author explain this phenomenon?\n\n- I would like to know if the FID and SSIM metrics are sensitive to the pruned model.\n\n- Can the MACs metric demonstrate the superiority of the proposed method in terms of performance acceleration? I am interested in understanding the effectiveness of the proposed method on training or inference speed under different GPU, e.g. Frame Per Second (FPS).\n\nOverall, this article is well-written and easy to follow, but there are some questions that need to be addressed. If the author can address these questions, I would consider increasing the score.\n\nQUESTIONS:\nsee Weaknesses.\n\nLIMITATIONS:\nThe paper briefly mentions limitations but does not provide detailed explanations. Expanding on the limitations section will help readers understand the potential constraints or assumptions of the proposed method and its applicability in different scenarios. Additionally, it is necessary to discuss specific technical shortcomings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a structural pruning method for diffusion models, called Diff-pruning. The authors leverage Taylor expansion on each term of the ELBO loss as the creteria to decide the pruned weights. By calculating the error of the final image induced by the pruning at timestep t and discussing about the effect of converged loss on the higher order terms of the Taylor expansion, this work manually drop the pruning creteria of timesteps near noise by setting a threshold $\\mathcal{T}$ related to the relative loss. Experiments compare the performance of the proposed pruning method with other common pruning methods, as well as training from scratch.\n\nSTRENGTHS:\n1. This paper proposes a novel pruning method that utilizes the multi-step property of the diffusion model.\n2. The experiment results demonstrate the effectiveness of the proposed method compared with baseline methods.\n\nWEAKNESSES:\n1. My major concern is the motivation of the method is not clearly explained: \n  - The authors conclude an important conclusion from eq.9. However, the derivation process of eq.9 has mistakes and typos. There is a neural inference term with $x_t$ as input in eq.3, the final error $\\delta_0$ can not be derived in such a simple form. Moreover, the meaning of the symbol \"$\\delta$\" is ambiguous. It seems to represent noise prediction error in line 146, but $\\delta_{t-1}$ and $\\delta_0$ seems to represent the error of $x_t$ and $x_0$ caused by pruning. And the subscript $s$ of $\\alpha_s$ is wrongly written in eq.8. \n  - How could Equ. 9 derive the conclusion \"prediction errors occurring at larger t primarily impact the high-level content of generated images, while smaller t values concentrate on refining the images with relatively small modifications\"? Relevant paper presents that the main difference between high-level contents and details is that they come from different frequency components of the whole image [1] , while Equ.9 focus on the \"amplitude of the error $\\delta_0$\". It requires further clarification. \n  - The authors claim that the final distortion is progressively magnified according to eq.9, which indicates that the error at timesteps near the noise has a greater impact on the final image. However, the proposed method truncates the pruning criteria at large t, which does not match such observation.\n\n2. Lack of detailed description of the baseline method ToMe.\n\n[1] Diffusion probabilistic model made slim. arXiv preprint arXiv:2211.17106, 2022.\n\nQUESTIONS:\n1. How is eq.9 derived and why the choice of $\\alpha_t$ according to this equation is opposite to the proposed method (as described in Weaknesses)?\n\nLIMITATIONS:\nThe authors do not include the limitations and potential negative societal impact in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a pruning-based method aimed at addressing the efficiency challenges of diffusion models. Unlike existing approaches that focus on accelerating sampling or enhancing architectures, it specifically focus on the time cost of compression and the consistency in generated images. The authors propose a novel variant of Taylor approximation as the importance function for pruning, which effectively preserves both high-level content and low-level details of generated images, while minimizing the impact of noisy steps. Experimental results demonstrate that the compressed diffusion model successfully generates similar images to the pre-trained model.\n\nSTRENGTHS:\nI agree that efficiency and consistency are indeed crucial considerations in compressing diffusion models, which have received limited attention in prior works. This work contributes to this field by building several initial experiments on datasets such as Cifar, CelebA, and LSUN, effectively showcasing the advantages of the method. This work establishes a solid baseline for future explorations into efficient diffusion models. Moreover, the impressive ability of the pruned model to generate consistent results is particularly noteworthy, which is user-friendly in deployment.\n\nWEAKNESSES:\nIt is observed that in certain cases, the generated images may still exhibit some distortions or changes, such as the watermarks shown in Figure 2. It is worth noting that some of these elements may contain important information that should ideally be preserved. To address this, it would be beneficial to explore methods that allow for controllable preservation of such information while pruning the models. This ability would provide the flexibility to focus on specific parts of the generated images based on different scenarios.\n\nQUESTIONS:\nIt would be valuable for the author to include discussions regarding the controllability of the observed distortions or changes, as mentioned in the weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Diff-Pruning, a novel structural compression technique for learning efficient diffusion models from pre-trained ones. The fundamental idea behind Diff-Pruning is the utilization of Taylor expansion on pruned timesteps, which effectively combines informative and clean gradients to estimate the importance of weights. The authors demonstrate that Diff-Pruning not only achieves compression of pre-trained models within a few training iterations but also maintains the original generation capabilities intact.\n\nSTRENGTHS:\n- The experimental results demonstrate that structural pruning can serve as a powerful and efficient compressor for diffusion models. An important advantage is that the pruned model inherently retains the generative behavior of the pre-trained model. Considering the time-consuming nature of training new diffusion models, this work proves valuable for various downstream applications.\n- The proposed method, which employs the Taylor expansion over pruned timesteps, is both well-motivated and practical. The idea of balancing the contribution of different stages with binary weighting is interesting and easy to practice.\n- The authors provide extensive experimental evidence that substantiates their claims regarding the efficiency and consistency of Diff-Pruning.\n\nWEAKNESSES:\n- Extra comparison of the memory requirement for different pruning methods might provide a more complete assessment of the proposed methods.\n- The experiments reveal that structural pruning can potentially have a negative impact on the performance of pre-trained models. This observation deviates slightly from the results seen in discriminative tasks such as classification, where lossless compression can be achieved for certain networks like ResNets. It would be beneficial for the authors to provide further clarification on this phenomenon\n\nQUESTIONS:\nPlease refer to the weaknesses above.\n\nLIMITATIONS:\nThis work has no negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a method to prune diffusion models to achieve 50% FLOPS reduction at 10% to 20% of original training budget. The authors show that prune and fine-tune strategy based on the random selection, magnitude based selection, Taylor expansion based selection all lead to non-ideal / suboptimal performance. They develops a method based on the modification of Taylor expansion based selection and show it achieves good performance-quality tradeoff. They quantitatively evaluated on standard datasets used for diffusion models at small and medium resolution (from 32^2 to 256^2).\n\nSTRENGTHS:\nThe paper works on a popular task in recent literature of generative models and presented a practice to prune diffusion models while preserves its generative capability. They demonstrated empirical results on 256^2 resolution unconditional image generation, which seems promising.\n\nWEAKNESSES:\nThe lack of objective quantitative metrics in image generation literature makes the evaluation of pruned model very difficult. In addition to FID score, the authors also use SSIM score to compare the image generated from the full model and the pruned model under the same random seed is used. However, this is still hard for me to evaluate the quality of empirical results given limited examples presented in the paper. It would be otherwise more informative, if the authors can demonstrate their method being valid for conditional diffusion models, or even latent diffusion models. \n\n\nThe technical contribution is also limited, as the method authors used already exist in the literature but has not been adopted for diffusion models.\n\nQUESTIONS:\nI don't have other questions\n\nLIMITATIONS:\nI have a feeling that authors tried very hard to prove their technique working to some extent. It would be helpful as a research publication if the authors can present more results telling when and where their method fails or explore more on the boundary of quality and performance trade-offs.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}