Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models
Authors: Muyang Li
Ji Lin
Chenlin Meng
Stefano Ermon
Song Han
Jun-Yan Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sparse
Generative Models
Diffusion Models
GANs
Efficiency

Abstract: During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a significant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general-purpose technique that selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models. Our key observation is that users tend to make gradual changes to the input image. This motivates us to cache and reuse the feature maps of the original image. Given an edited image, we sparsely apply the convolutional filters to the edited regions while reusing the cached features for the unedited regions. Based on our algorithm, we further propose Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to latency reduction on off-the-shelf hardware. With 1.2%-area edited regions, our method reduces the computation of DDIM by $7.5\times$ and GauGAN by $18\times$  while preserving the visual fidelity. With SIGE, we accelerate the inference time of DDIM by $3.0\times$ on RTX 3090 and $6.6\times$ on Apple M1 Pro CPU, and GauGAN by $4.2\times$ on RTX 3090 and $14\times$ on Apple M1 Pro CPU.

Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models
Muyang Li Ji Lin² Chenlin Meng
Stefano Ermon
Song Han Jun-Yan Zhu'
CCarnegie Mellon University "Masachusettss Institute of Technology Stannordd University
Abstract
During image editing, existing deep generative models tend to re-synthesize the entire output from scratch, including the unedited regions. This leads to a sig- nificant waste of computation, especially for minor editing operations. In this work, we present Spatially Sparse Inference (SSI), a general- that selectively performs computation for edited regions and accelerates various generative models, including both conditional GANs and diffusion models. Our key observation is that users tend to make gradual changes to the input image. This motivates us to cache and reuse the feature maps of the original image. Given edited image, sparsely apply the convolutional filters the edited regions while reusing the cached features for the unedited regions. Based on our algorithm, we further propose Sparse Incremental Generative Engine (SIGE) convert the computation reduction to latency reduction on off-the-shelf hardware. With 1.2%- area edited regions, our method reduces the computation of DDIM by 7.5x and GauGAN by 18x while preserving the visual fidelity. With SIGE. we accelerate the inference time of DDIM by 3.0x on RTX 3090 and 6.6x on Apple MI Pro CPU, and GauGAN by 4.2x on RTX 3090 and 14x on Apple MI Pro CPU.
technique
Introduction
Deep generative models, such GANs [1, and diffusion models [3, 4, 5], excel at synthesizing photo-realistic images, enabling many image synthesis and editing applications. For example, users can edit image by drawing sketches [6. 7J, semantic maps [6, 81, strokes [9]. All of these applications require users to interact with generative models frequently and therefore demand fast inference time.
In practice, content creators often edit images gradually and only update a small image region each time. However, even for a minor edit, recent generative models often synthesize the entire image, including the unchanged regions, which leads to a significant waste of computation. As a concrete example shown in Figure 2(a), the result of the previous editing has already been computed, and the user further edits 9.4% area. However, the vanilla DDIM [5] needs to generate the entire image obtain the newly edited regions, wasting 80% computation on the unchanged regions. A naive approach to address this issue would be to first segment the newly edited regions, synthesize the corresponding output regions, and blend the outputs back into the previous output. Unfortunately, this method often creates visible seams between the newly edited and unedited regions. How could we save the computation by only updating the edited regions without losing global coherence?
In this work, we propose Spatially Sparse Inference (SSI), a general method to accelerate deep generative models, including conditional GANs and diffusion models, by utilizing the spatial sparsity of edited regions. Our method is motivated by the observation that feature maps at the unedited regions remain mostly the same during user editing. As shown in Figure 2(b), key idea is reuse the cached feature maps of the previous editing and sparsely update the newly edited regions. Specifically, given user input, we first compute the difference mask to locate the newly edited regions. For each
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Original
Ediding
Original
11.5% Editing
DDIM: 249G MAC's
Ours: 61.3G (4.1-)
GAEGAN: 281G MAC'S
Ours: 19.36
Original
Editing
DDIM:
(6.1*) IGAN Compression: (9.+)GAN Comp-rOurs: 682G (47%)
Figure 1: We introduce Spatially Sparse Inference, general purpose method to selectively perform computations the edited regions image editing applications. Our method reduces the computation DDIM [5] by 4 ~v 6x and GauGAN [8] by 15x for the examples shown in the figures while preserving image quality. When combined with existing model compression methods such as GAN Compression [10], our method further reduces the computation of GauGAN by 47x.
convolution layer in the model, we only apply the filters to the masked regions sparsely while reusing the previous activations for the unchanged regions. The sparse update can significantly reduce the computation without hurting the image quality. However, the sparse update involves gather-scatter process, which often incurs sigmificant latency overheads for existing deep learning frameworks. To address the issue, we propose Sparse Incremental Generative Engine (SIGE) to translate the theoretical computation reduction of our algorithm measured latency reduction on various hardware.
To evaluate our method, we automatically create new image editing benchmark datasets on LSUN Church [11] and Cityscapes [12]. Without loss of visual fidelity, reduce the computation of DDIM [5] by 7.5x, Progressive Distillation [13] by 2.7 and GauGAN by 18x measured by MACs'. Compared existing generative model acceleration methods [10, 14, 15, 16, 17, 18, 19], our method directly uses the off-the-shelf pre-trained weights and could be applied to these methods as plugin. When applied GAN Compression [10], our method reduces the computation of GauGAN by x0x. See Figure for some examples of our method. With SIGE, we accelerate DDIM 3.0x on RTX 3090 GPU and 6.6x on Apple MI Pro CPU, and GauGAN 4.2x on RTX 3090 GPU and 38x on Apple M1 Pro CPU. Our code and benchmarks are available at https: //////ttuubbu com/ hwomm//myy/////.ieea /sie..
2 Related Work
Generative models. Generative models such GANs [1, 2, 20, 21], diffusion models [4, 3. 22]. and ruto-regressive models [23, 24] have demonstrated impressive photorcalistic synthesis capability. They have also been extended to conditional image synthesis tasks such image-to-image translation [25, 6, 26, 27], controllable image generation [9, 28, 8], and real image editing [29, 28, 30, 31, 32, 33, 34, 27]. Unfortunately, recent generative models have become increasingly computationally intensive, compared to their recognition counterparts. For example, GauGAN [8] consumes 281G MACs, 500x
500x more than MobileNet [35, 36, 37J. Similarly, one key limitation of diffusion models [4] is their long inference time and substantial computation cost. To generate one image, DDPM requires hundreds or thousands of forwarding steps [4, 22], which is often infeasible in real-world interactive settings. To improve the sampling efficiency of DDPMs, recent works [5, 38, 39] propose to interpret the sampling process of DDPMs from the perspective of ordinary differential equations. However, these approaches still require hundreds of steps to generate high-quality samples. To further reduce the sampling cost, DDGAN [40] uses multimodal conditional GAN to model each denoising step. Salimans et al. [13] propose to progressively distill a pre-trained DDPM model into a new model that requires fewer steps. Although this approach drastically reduces the sampling steps, the distilled model itself remains computationally prohibitive. Unlike prior work, our work focuses on reducing the computation cost a pre-traines model. Our work complementary to recent efforts on model compression, distillation, and the sampling step reduction of the diffusion models.
Model acceleration. People apply model compression techniques, including pruning [41, 42, 43, 44, 45, 46] and quantization [41, 47, 18.484444 49, 50, 51], to reduce the computation and model size of off- the-shelf deep learning models. Recent works apply Neural Architecture Search (NAS) [52, 53, 3, 54, 55,
WWe measure the computational cost with the number of Multiply-Accumulat operations (MACs). MAC=2
FLOPs.

---

Computed Activations for Previous Editing
Computed Activations for New Editing
Cached with DDIM
DDIM: 2490 MACS
Previous Edining
Previous Editing
Reuse Feature Maps
DDIM with $0.1G
Editing
Editing
Generated
Vanilla DDIM
DDIM with Sponially Sparse Inference (Our)
Figure 2: In the intcractive editing scenario, a user edits 9.4% area. (a) Vanilla DDIM has to regenerate the entire image for every single editing, though only 9.4% editing is made. (b) Our method instead reuses the feature maps of the previous editing and only applies convolutions the newly edited regions sparsely, which has 5.0x MACs reduction this case.
56, 57, 58] to automatically design efficient neural architectures. The above ideas can be successfully applied to accelerate the inference of GANs [10, 59, 60, 61, 14, 62, 15, 16, 17, 18, 19, 63). Although these methods have achieved prominent compression and speedup ratios, they all reduce the computa- tion from the model dimension but fail to exploit the redundancy in the spatial dimension during image editing. Besides, these methods require re-training the compressed model maintain performance, while our method can be directly applied to existing pre-trained models. We show that our method can be combined with model compression [10] to achieve 50x MACs reduction in Section 4. I.
Sparse computation. Sparse computation has been widely explored in the weight domain [64, 65, 66, 67], input domain [68, 69], and activation domain [70, 71, 72, 73]. For activation sparsity, RRN [74] utilizes the sparsity in the consecutive video frame difference to accelerate video models. However, their sparsity is unstructured, which requires special hardware to reach its full speedup potential. Several works instead use structured sparsity. Li et al. [75] use a deep layer cascade to apply more convolutions the hard regions than the easy regions to improve the accuracy and speed of semantic segmentation. To accelerate 3D object detection, SBNet [70] uses a spatial mask, either from priori problem knowledge or auxiliary network, sparsify the activations. It adopts a tiling-based sparse convolution algorithm to handle spatial sparsity. Recent works further integrate the spatial mask generation network into the sparse inference network in end-to-end manner [76] and extend the idea to different tasks [77, 78, 79, 80). Compared to SBNet [70], our mask is directly derived from the difference between the original image and the edited image. Additionally, our method does not require any auxiliary network or extra model training. We also introduce other optimizations, such normalization removal and kernel fusions, to better adapt our engine for image editing.
3 Method
We build our method based on the following observation: during interactive image editing, user often only changes the image content gradually. As result, only small subset of pixels in local region is being updated at any moment. Therefore, we can reuse the activations of the original image for the unedited regions. As shown in Figure 3, we first pre-compute all activations of the original input image. During the editing process, we locate the editing regions by computing a difference mask between the original and edited image. We then reuse the pre-computed activations for the unedited regions and only update the edited regions by applying convolutional filters them. In Section 3.1, show the sparsity in the intermediate activations and present main algorithm. In Section 3.2, we discuss the technical details how our Sparse Incremental Generative Engine (SIGE) supports the sparse inference and converts the theoretical computation reduction to measured speedup on hardware.
3.1 Activation Sparsity
Preliminary. First, we closely study the computation within single layer. We denote A Aoriginal and Aqdited the input tensor of the original image and edited image to the I-th convolution layer Fi, respectively. Wi and by are the weight and bias of F,. The output of Fi with input Apdited could be
3

---

Original Image
Ariiiinal
Conv
F/A urnginnly
Generated
Cony;
SIGE
Cony
Gather
Seatter
Edited Image
Active Dlocks
Generated
A
pdittdd
(Stack Batch Dimension)
SIGE Conv;
Keduce
Difference Mask Reduced Indices
Figure 3: Tiling-based sparse convolution overview. For cach convolution Fi in the network, we wrap it into SIGE Conv; The activations of the original image already pre-computed, When getting the edited image, first compute difference mask between the original and edited image and reduce the mask the active block indices to locate the edited regions. In each SIGE Conv,, we directly gather the active blocks from the edited activation Af according to the reduced indices, stack the blocks along the batch dimension, and feed them into Fj. The gathered blocks have overlap of width 2 if F; is 3 x convolution [70]. After getting the output blocks from Fi, we seatter them back into (AA (etiginal) to get the edited output, which approximates Fi(gllie))
Original
Edited
256 256
128 128
64 x 64
32 32
16
8
Detailed editing example. Right: Channel-wise average of A Aul at the I-th layer DDIM with different feature map resolutions. Ail is sparse and non-zero values are aggregated at the edited regions.
Figure 4: Left:
computed in the following way due to the linearity of convolution:
Fr(A F(
Wi Aptlted + by
W, (Agditedd
A Original
+ (Wr A
Apriginal
+ bi)
Wi AA +
original
sparse
pre-computed
where * is the convolution operator and AA, Apdited
If we already pre-computed all the (Arrign)) ), we only need compute Wi AA₁. Naively, computing Wi AAt has the same complexity W3 Apdited However, since the edited image shares similar features with the original image given a small edit, AA, should be sparse. Below, we discuss different strategies to leverage the activation sparsity accelerate model inference.
A, original
Our first attempt was to prune A, by zeroing out elements smaller than a certain threshold to achieve the target sparsity. Unfortunately, this pruning method fails achieve measured speedup due to the overheads of the on-the-fly fly pruning and irregular sparsity pattern.
Structured sparsity. Fortunately, user edits are often highly structured and localized. As a result, A A, should also share the structured spatial sparsity, where non-zero values mostly aggregated within the edited regions, as shown in Figure 4. We then directly use the original image and edited image compute a difference mask and sparsify A with this mask.
3.2 Sparse Engine SIGE
But how could we leverage the structured sparsity to accelerate W; * AA2? A naïve approach is to crop rectangular edited region out of A for each convolution and only compute features for the cropped regions. Unfortunately. this naïve cropping method works poorly for the irregular edited regions (e.g., the example shown in Figure 4).
Tiling-based sparse convolution. Instead, as shown Figure 5(a), we use tiling-based sparse convolution algorithm. We first downsample the difference mask to different scales and dilate the downsampled masks with extra pixels (1 for diffusion models and 2 for GauGAN). Then we divide A into multiple small blocks of the same size spatially and index the difference mask

---

original
Indices
Indices
F (figina)))
Indices
Indices
Apcitted
Gather
Cony F;
Scatter
+
apdited
Gather
Conv
Scatter
wio Bias
(a) Residual pipline
(b) Improved pipeline
FRA, FAAprigginal
Figure 5: The titling-basee based sparse convolution pipelines. (a) We first compute the activation difference AA; and gather the active blocks along the batch dimension from according to the indices reduced from the difference mask. We then feed blocks into the convolution Fù without bias, scatter the output into a zero tensor, and add the residual Fi (A (Aqqiiiaa) back. (b) We directly gather the blocks from Apflted without computing At, computed with bias. We scatter the output into ((( instead of zero tensor.
at the corresponding resolution. Each block index refers a single block with non-zero elements. We then gather the non-zero blocks (we also call them active blocks) accordingly along the batch dimension and feed them into the convolution Fi. Finally, we scatter the output blocks into zero tensor according to the indices to recover the original spatial size and add the pre-computed residual F((origg))))) back. The gathered active blocks have an overlap with width for 3 convolution to ensure the output blocks of the adjacent input blocks are seamlessly stitched together [70].
This pipeline in Figure 5(a) is equivalent to simpler pipeline in Figure 5(b). Instead of gathering AA, we could directly gather Apalited The convolution needs to be computed with bias b. Besides, we need to scatter the output blocks into (A4 instead of zero tensor. Thus, we do not need to store A prriginal anymore, which further saves memory and removes the overheads of addition and subtraction. Figure visualizes the pipeline.
However, the aforementioned pipeline still fails produce noticeable speedup due to extra kernel calls and memory movement overheads in Gather and Scatter. For example, the original dense 3 3 convolution with 128 channels and input resolution 256 256 would take 0.78ms on RTX 3090. The sparse convolution using pipeline Figure 5(b) on the example shown in Figure 4 (15.5% edited regions) needs 0.42ms total, while Gather and Scatter cost a significant overhead (0. 7m.s about 41 %). To reduce it, we further optimize SIGE by pre-computing normalization parameters and applying kernel fusion.
Pre-computing normalization parameters. For batch normalization [81], it is easy to remove the normalization layer during inference time since from model training. However, recent deep generative models often use instance normalization [82, 83] or group normalization [84, 85), which compute the statistics the fly during inference. These normalization layers create overheads as we need to estimate the statistics from the full-size tensors. However, the original and edited images are quite similar given small user edit, we assume A original Apdited This allows us to reuse the statistics of A Apriginal for the normalization instead of recomputing them for Apdited Thus, normalization layers could be replaced by simple Scale+Shift operations with pre-computed A original statistics.
pre-computed mean and variance statisties
Kernel fusion. As mentioned before, both the Gather and Scatter operations introduce signifi- cant data movement overheads. To reduce it, we fuse several element-wise operations "Scale+Shife and Nonl mallineerity into Gather and Scatter [70, 86, 87] and only apply these element operations to the active blocks (i.e., edited regions). Furthermore, we perform in-place computation to reduce the number of kernel calls and memory allocation overheads.
In Scatter, we need to copy the pre-computed activation (A( This copying operation is highly redundant, most elements from (A priginall do not involve any computation given small edit and will be discarded in the next Gather. To reduce the tensor copying overheads, we fuse the Scatter and the following Gather by directly gathering the active blocks from (AAriigi)))) and the input blocks to be scattered. Sometimes, the residual connection in the ResBlock [88] contains shortcut 1 x convolution to match the channel number of the residual and the ResBlock output. We also fuse the Scatter the shorteut branch, main branch, and the residual addition together avoid the tensor copying overheads in the shortcut Scatter. Please refer to Appendix A for more details.
5

---

Experiments
Below we first describe our models, baselines, datasets, and evaluation protocols. We then discuss main qualitative and quantitative results. Finally, we include detailed ablation study regarding the importance of each algorithmic design.
Models. We conduct experiments three models, including diffusion models and GAN-based models, to explore the generality of our method.
DDIM [5] is a fast sampling approach for diffusion models. proposes to interpret the sampling process of diffusion models through the lens of ordinary differential equations.
Progressive Distillation (PD) [13] adopts network distillation [89] to progressively reduce the number of steps for diffusion models.
GauGAN [8] is paired image-to-image translation model which learns to generate a fideliyiy image given semantic label map.
Baselines. We compare methods against the following baselines:
Putch. We crop the smallest patch coverring all the edited regions, feed it into the model, and blend the output patch into the original image.
Crop. For each convolution Fi, we crop the smallest rectangular region that covers all masked elements of the activation Aplited feed it into Fi, and scatter the output patch into (A (rriinnal), )
40% Pruning. We uniformly prune 40% weights of the models without further fine-tuming as our method directly uses the pre-trained weights. Since the fine-grained pruming is unstructured, requires special hardware to achieve measured speedup,
do not report MACs for this baseline.
0.19 GauGAN. We reduce each convolution layer of GauGAN to 19% channels (21x MACs reduction) and train it from scratch.
GAN Compression [10]. A
compression method for conditional GANs. GAN Comp. (S) means GAN Compression with a larger compression ratio.
0.5 Original means linearly scaling each layer of the original model to 50% channels, and we only use this benchmark our efficiency results.
Datasets. We use the following two datasets in experiments:
LSUN Church. We use the LSUN Church Outdoor dataset [11] and follow the same preprocessing steps as prior works [4, 38]. To automatically generate stroke editing benchmark, we first use Detic [90] to segment the images in the validation set. For each segmented object, we use its segmentation mask to inpaint the image by CoModGAN [91] and treat the inpainted image as the original image. We generate the corresponding user strokes by first blurring the masked regions with the median filter and quantizing it into colors following SDEdit [9]. We collect 454 editing pairs in total (431 synthetic + 23 manual). We evaluate DDIM [5] and PD [13] this dataset.
Cityscapes. The dataset [12] contains images of German street scenes. The training and validation sets consist of 2975 and 500 images, respectively. Our editing dataset has 1505 editing pairs in total. We evaluate GauGAN [8] on this dataset.
Please refer Appendix B for more details about the benchmark datasets.
Metries. Following previous works [9, 10, 8]. we use the standard metrics Peak Signal Noise Ratio (PSNR, higher is better), LPIPS (lower is better) [92], and Fréchet Inception Distance (FID, lower better) [93, 94] to evaluate the image quality on both LSUN Church [11] and Cityscapes [12]. For Cityscapes, we adopt semantic segmentation metric to evaluate the generated images. Specif- ically, we run DRN-D. 105 [95] on the generated images and compute the mean Intersection over Union (mloU) of the segmentation results. Generally, higher mIOU indicates that the generated images look more realistic and better align to the input.
We use clean fid for FID calculation.

---

MACs
PSNR (f)
LPIPS (J)
Model
Method
HID (4)
mloU (t)
Value
Ratio
with G.T.
with Orig.
with G.T.
with Orig.
Original
249G
26.8
0.069
65.4
40% Pruning
24.9
31.0
0.991
0.101
72.2
DDIM
Patch
72.06
3.5x
26.8
0.076
66.4
Ours
65.3G
3.8x
26,8
53.4
0.070
0.009
65.8
Original
66 9G
21.9
0.182
90.0
PD
40% Pruning
21.6
37.6
0.189
0.051
101
Ours
32.5G
2
21.9
60.7
0.183
0.003
90.1
Original
281G
15.8
0.451
GAN Comp. [10]
31.2G
9.0x
15.7
19.5
0.453
0.282
55.3
61.5
Ours
30.7G
9.2x
15.8
26.5
0.454
0.113
54.2
62.1
GouGAN
0.19 GaIGAN
13.3G
21 x
15.5
18.6
0.462
0.319
57.8
53.5
GAN Comp. (S)
9.64G
15.7
0.461
57.4
GAN Comp.+Ours
7.06G
40x
15.7
19.1
0.457
0.294
54.4
60.1
Table 1: Quantitative quality evaluation. PSNR/LPIPS with G.T. means computing the metrics with the ground- truth images, and with Orig. means computing with the generated samples from the original model. 40% Pruming: Uniformly pruning 40% weights of the model without fine-tuning. Patch: Cropping the smallest image patch that covers all the edited region and blending the output patch into the original image. 0.19 GauGAN: Uniformly reducing each layer of GauGAN to 19% channels and training from scratch. GAN Comp. (S): GAN Compression with a larger compression ratio. For all models, our method outperforms other baselines with less computation. When applying method to GAN Compression, we reduce the MACs of GauGAN by x0x with minor performance degradation.
Implementation details. The number of denoising steps for DDIM and PD are 100 and 8, respec- tively, and we use 50 and 5 steps for SDEdit. We dilate the difference mask by 5, 5, and I pixels for DDIM, PD with resolution 128, PD with resolution 256 and GauGAN, respectively. Besides, we apply our sparse kernel to all convolution layers whose input feature map resolutions are larger than 32 x 32, 16 x 16, 8 x 16 and 16 x 32 for DDIM, PD, original GauGAN and GAN Compression, respectively. For DDIM [5] and PD [13], we pre-compute and reuse the statistics of the original image for all group normalization layers [84]. For GAN Compression [10], we pre-compute and reuse the statistics of the original image for all instance normalization layers [82] whose resolution is higher than 16 x 32, For all models, the sparse block size for 3 x 3 convolution is 6 and I X I convolution is 4.
4.1 Main Results
Image quality. We report the quantitative results of applying method on DDIM [5]. Progressive Distillation (PD) [13], and GauGAN [8] in Table and show the qualitative results in Figure 6. For PSNR and LPIPS, with G.T. means computing the metric with the ground-truth images. With Orig. means computing the metric with the samples generated by the original model. On LSUN Church, we only use 431 synthetic images for the PSNR/LPIPS with G.T. metrics, as manual editing does not have ground truths. For the other metrics, we use the entire LSUN Church dataset (431 synthetic + 23 manual). On Cityscapes, we view the synthetic semantic maps the original input and the ground-truth semantic maps as the edited input for the PSNR/LPIPS with G.T. metrics, which has 1505 samples. For the other metrics, we include symmetric editing (view the ground-truth semantic maps as the original inputs and synthetic semantic maps as the edited inputs), which has 3010 samples in total. For the models with method Patch and Ours, whose computation is editing dependent, we measure the average MACs over the whole dataset.
For DDIM and Progressive Distillation, our method outperforms all baselines consistently and achieves results on par with the original model. The Patch inference fails when the edited region is small as the global context insufficient. Although our method only applies convolutional filters to the local edited regions, we could reuse the global context stored in the original activations. Therefore, our method could perform the same as the original model. For GauGAN, our method also performs better than GAN Compression [ [10] with an even larger MACs reduction. When applying our method to GAN Compression, we further achieve a 40x MACs reduction with minor performance degradation, beating both 0.19 GauGAN and GAN Comp. (S).
7

---

Original
120% Editina
DOIM: MACS FID: 65.4
Punch:
FID: 06.4
FID: 66.6
FID: S0L0
663G
FTD: 96.1
34.5
(4.8g)
FID:654
FID: 66.4
68.
Uriginal
keiting
DOIM:
Parch (LG
1246 (1.9)
Original
1.8% Editlig
mloth 62.4
anoli 42.1
mldii
Original
9.4%% Ediiig
GRoCAN: 2IGGAAC miat: $24
Ours: 46.6 (LD))) mbel:: 63.1
GAN Comp.: 31.2G (UUP)) misli
GAN Comg--Ours: -Gars- 1.54 (30
mieli:
GAN Comp. 31.20 (10g) GAN Cong-Dars: JDSG (6-) miel: 61.5 Figure 6: Qualitative results of our method under different editing sizes. Our method well preserves the visual fidelity of the original model without losing global context. On the contrast, Putch (cropping the smallest image patch that covers all the edited regions and scatter the output patch back into the original image) performs poorly because the lack global context when the editing small.
Origina
11.5% Editing
GEUGAN: MEGMACS
mbot: #24
Ouru 63.G (4D))
miol:: 92.1
misi: 601
Model efficiency. For real-world interactive image editing applications, inference acceleration on hardware is more critical than the computation reduction. To verify the effectiveness of our proposed engine, we measure the speedup of the editing examples shown in Figure 6 on four devices, including NVIDIA RTX 3090, NVIDIA RTX 2080Ti, Intel Core 19- 10920X CPU, and Apple M1 Pro CPU, with different computational powers. We use batch size to simulate real-world use. For GPU devices, we first perform 200 warm-up runs and measure the average latency of the next 200 runs. For CPU devices, we perform 10 warm-up runs and 10 test runs, repeat this process 5 times and report the average latency. The latency is measured in PyTorch 1.7 The results are shown in Table 2.
The original Progressive Distillation [13] can only generate 128 x 128 images, which too small for real We add some extra layers adapt the model to resolution 256 x 256. For fair comparisons, we also pre-compute the normalization parameters for the Crop baseline. When the editing pattern is like rectangle, this baseline reduce similar computation with in Figure 6). However, the speedup still worse than ours on RTX 3090, 2080Ti and Intel Core 19-10920X due to the large memory index overheads in native PyTorch. When the edited region is far from rectangle (e.g., the third example of DDIM), the cropped patches have much redundancy. Therefore, even though only 15.5% region edited, the MACs reduction only 1.6x. With 1.2% editing size, our method achieves 7.5x, 4.6x, and 1 8x MACs reduction for DDIM, Progressive Distillation, and GauGAN, respectively. With SIGE, we achieve at most 4. 2.9x, 6.0x and 14x speedup on RTX 3090, 2080Ti, Intel Core i9- 10-009 and Apple MI Pro CPU, respectively. When applied to GAN Compression. SIGE achieves 9.5 and 38x latency reduction on Intel Core 19-10920X and Apple MI Pro respectiely.....
(e.g.. the first example of DDIM

---

MACs
3090
2080Ti
Intel Core i9- 10-1090XX
Apple
Pro
Model
Editing Size
Method
Value
Ratio
Value
Ratio
Value
Ratio
Value
Ratio
Value
Ratio
Original
248G
37.5ms
54.6ms
609ms
12.9%
0.5 Original
62.5G
4.0%
20.0ms
1.9x
31.2ms
1.8x
215ms
2.8x
3.22s
4.0%
Crop
32.6G
7.6x
15.5ms
2.4%
29.3ms
185ms
1.85s
6.9%
DDIM
1.20%
Ours
33.4G
x.5x
12.6ms
3.0%
19.1ms
x
147ms
4.1x
1.96s
6.6%
Cmp
I55G
1.6x
30.5ms
1.2x
44,5ms
1.2x
44
1.4x
8.09s
1.6%
15.5%
Ours
78.9G
3.2x
19.4ms
1.9x
29.8ms
1.8%
304ms
2.0x
5.04s
2.6x
Original
1196
35.1ms
51.2ms
388ms
6.18s
0.5 Original
31.0G
3
29.4ms
1.2x
43.2ms
1.2x
186ms
2
1.72s
3.6x
PD256
1.20%
Ours
25.9G
4.6%
18.6ms
19x
26.4ms
1,9x
152ms
x,5x
1.55s
4.0%
15.5%
Ours
48.5G
2,5x
21 Ams
1.6x
30.7ms
1.7x
250ms
1.6x
3.22s
1.9x
Original
281G
45.4ms
49.5ms
682ms
14.1s
GAN Compression 31.2G
9.0X
17.0ms
2.7x
25.0ms
200%
333ms
2.1x
2.11s
6.7%
Ours
153G
18x
4.1x
19.3ms
114ms
0.990s
14x
GauGAN 1.18%
GAN Comp-+Ours
5.59G
50x
10.8ms
4.2%
16.2ms
3.1x
53.1ms
13x
0.370s
38x
Ours
69.8G
4.0%
17.8ms
2.5%
27.1ms
188
278ms
2
4.06s
3.5x
13.5%
GAN Comp.+OOrs
10.8G
x6x
11.8ms
388
17.4ms
288
94.4ms
7.2x
0.741s
19x
Table 2: Measured latency speedup on differrnnt devices. The detailed editing examples shown in Figure 6. 0.5 Original: Linearly scaling each layer of the model to 50% channels. Crop: For each convolution, we find small- est patch covering the masked elements, crop it out, feed it into the convolution and scatter the output patch into the original image activation. Our method could reduce up to 18x MACs, and achieve to 4. x, 2.9x, 6.0x and 14x lateney reduction on NVIDIA RTX 3090, 2080Ti, Intel Core 19- 19-100 and MI Pro CPU. With GAN Compression, we could further speed up GauGAN by 9.5x on Intel Core-i9 and 38x on Apple MI Pro CPU.
MACs
Optimizations
Lateney
MACs
PyTorch
TensoRRT
Sparse Norm. Elem. Set.
Value Ratio
Method Editing Size
Value Ratio
Ratio
Value
Ratio
249G
54.6ms
Original
249G
54.6ms
47.7ms
34.0ms 1.6x
1.20%
33.40 7,5x 19.1ms 2.9x 14.4ms 3.3
32.6G
V
V
29. 19.6ms 1.8
Ours
7.19%
51.80 4.8
1m. 2,5x
18.6ms 2.6x
(7.6x)
V
d.
/
20.7ms x
1.55%
78.9G 3.2x 29.8ms 1,8x 26 26.9ms L8X
V
19.1ms 2
(a)
Table 3: (a) Ablation study of each kernel optimization. Sparse: Using tiling-hased sparse convolution. Norm.: Pre-computing normalization parameters. Elem.: Fusing element wise operations. Set.: Fusing Scatter reduce the tensor copying overheads. With all optimizations, we could reduce the latency of DDIM by 2,9x on NVIDIA RTX 2080Ti. (b) Latency comparisons of DDIM on RTX 2080Ti between PyTorch and TensorRT. The speedup ratio is larger in TensorRT than Py Torch, especially when the editing size is small.
4.2 Ablation Study
Below we perform several ablation studies to show the effectiveness of each design choice.
Memory usage. The pre-computed activations of the original image require additional memory storage. We profile the peak memory usage of the original model and our method in PyTorch. Our method only increases the peak memory usage of a single forward for DDIM, PD, GauGAN, and GAN Compression by 0.1G, 0.1G, 0.8G, and 0.3G, respectively. Specifically, our method needs to store additional 169M, 56M, 275M, and 120M parameters for DDIM [5], PD [13], original GauGAN [8] and GAN Compression [10], respectively, for a single forward. For the diffusion models, we need to store activations for all iteration steps (e.g., 50 for DDIM and 5 for PD). However, data movement and kernel computation are asynchronized on GPU, so we could store the activations in CPU memory and load the on-demand ones GPU to reduce peak memory usage.
Speedup of each design. Table 3a shows the effectiveness of each kernel optimization we add to SIGE for DDIM [5] on RTX 2080Ti. Naively applying the tiling-based sparse convolution could reduce the computation by 7.6x. Still, the latency reduction is only 1.6x due to the large memory overheads in Gather and Scatter. Pre-computing the normalization parameters could remove the latency of normalization statistics calculation and reduce the overall latency to 29, 29.6ms. Fusing

---

element-wise operations into the Gather and Scatter could remove some redundant operations that are upplied to the unedited regions and also reduce the memory allocation overheads (about 9ms). Finally, fusing the Scatter and Cather to Scatter- and Scatter in the shorteut branch and main branch could further reduce about 1. 1.6ms tensor copying overheads, achieving 2.9x speedup.
Experiments with TensorRT. Real-world model deployment also depends on deep learning backends with optimized libraries and runtimes. To demonstrate the effectiveness and extensibility of SIGE, we also implement our kernels in a widely-used backend TensorRT and benchmark the DDIM latency results RTX 2080Ti in Table 3b. Specifically, our speedup ratio becomes more prominent with TensorRT compared to PyTorch, especially for small edits, as TensorRT better supports small convolutional kernels with higher GPU utilization than PyTorch.
5 Conclusion & Discussion
For image editing, existing deep generative models often waste computation by re-synthesizing the image regions that do not require modifications. To solve this issue, we have presented general- purpose method, Spatially Sparse Inference (SSD, to selectively perform computation edited regions, and Sparse Incremental Generative Engine (SIGE) to convert the computation reduction to latency reduction on commonly-used hardware. We have demonstrated the effectiveness of our approach in various hardware settings.
Limitations. As discussed in Section 4.2, our method requires extra memory to store the original activations, which slightly increases the peak GPU memory usage. It may not work on certain memory-constrained devices, especially for the diffusion models (e.g.. DDIM (5]), since our method requires storing activations of all denoising steps.
Our engine has limited speedup on convolution with low resolution. When the input resolution is low, the active block size needs to be even smaller to get a decent sparsity, such as I or 2. However, such extremely small block sizes have bad memory locality and will result in low hardware efficiency.
Besides, we sometimes observe noticeable boundary between the edited regions and unedited regions in our generated samples of GauGAN [8]. This is because, for GauGAN model, the unedited region will also change slightly when we perform normal inference. However, since our method does not update the unedited region, there may be some visible seams between the edited and unedited regions, even though the semantic is coherent. Dilating the difference mask would help reduce the gap.
In most cases, the edit will only update the edited regions. However, sometimes the edit will also introduce global illumination changes such as shadow and reflection. For this case, as we only update the edited regions, we cannot update the global changes outside the edited regions accordingly.
Societal impact. In this paper, we investigate how update user editing locally without losing global coherence to enable smoother interaction with the generative models. In real-world scenarios, people could use an interactive interface to edit an image, and our method could provide quick and high-quality preview for their editing, which eases the process of visual content creation and reduces energy consumption, leading to a greener AI application. The reduced cost also provides good user experience for lower-end devices, which further democratize the applications of generative models.
However, our method can be utilized by malicious users to generate fake contents, deceive people, and spread misinformation, which may lead to potential negative social impacts. Following previous works [9]. we explicitly specify the usage permission of our engine with proper licenses. Additionally, we run a forensics detector [96] to detect the generated results of our method. On GauGAN, our generated images can be detected with 97. .7% average precision (AP). However, on DDIM [5] and Progressive Distillation [13], the APs are only 56.6% and 52.4%. Such low APs are caused by the model differences between GANs and diffusion models, as observed in SDEdit [9]. We believe developing forensic methods for diffusion models is a critical future research direction.
Acknowledgment. We thank Yaoyao Ding, Zihao Ye, Lianmin Zheng, Haotian Tang, and Ligeng Zhu for the helpful comments on the engine design. We also thank George Cazenavette, Kangle Deng, Ruihan Gao, Daohan Lu, Sheng-Yt Yu Wang and Bingliang Zhang for their valuable feedback. The project is partly supported by NSF, MIT-IBM Watson AI Lab, Kwai Inc, and Sony Corporation.
We benchmark the results with TensorRT 8.4.
10

---

References
[1] lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014. 1,2
[2] Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 1,2
[3] Jascha DicksDicctt,in, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, In ICML, 2015. 1.2
[4] Jomathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrIPS, 2020. 1, 2,
[5] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020. 1. 2. 7, 10, 17, 19. 20, 23
[6] Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 1,2
[7] Patsorn Sangkloy, Jingwan Lu, Chen Fang. Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In CVPR, 2017. 1
[8] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In CVPR, 2019. 1, 2. 6, 7, 9, 10, 19, 20, 24
[9] Chenlin Meng. Yutong He, Yang Song. Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022. 1,2, 10, 19
[10] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun- Yan Zhu, and Song Han. Gan compression: Efficient architectures for interactive conditional gans. In CVPR, 2020. 2, 3, 6, 7, 9, 19, 20
[11] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1rXiv:1506.033365 2015. 2, 6, 20, 21
[12] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2, 6, 20, 21
[13] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2021. 2, 6. 7, 8, 9, 10. 19, 20
[14] Liang Hou, Zehuan Yuan, Lei Huang, Huawei Shen, Xueqi Cheng, and Changhu Wang. Slimmable generative adversarial networks. In AAAI, 2021. 2,3
[15] Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan- distiller: Searching to compress generative adversarial networks. In JCML, 2020. 2.
[16] Shaojie Li. Mingbao Lin, Wang. Chao Fei, Ling Shao, and Rongrong Ji. Learning efficient gans for image translation via differentiable masks and co-attention distillation. IEEE Transactions on Multimedia, 2022. 2,3 3
[17] Qing Jin, Jian Ren, Oliver J Woodford. Jiazhuo Wang. Geng Yuan, Yanzhi Wang, and Sergey Tulyakov. Teachers do more than teach: Compressing image-to- timemaee models. In CVPR, 2021. 2,
[18] Tamar Rott Shaham, Michaël Gharbi, Richard Zhang, Eli Shechtman, and Tomer Michaeli. Spatially- adaptive pixelwise networks for fast image translation. CVPR, 2021. 2,3
[19] Haotao Wang. Shupeng Gui, Haichuan Yang. Ji Liu, and Zhangyang Wang. Gan slimming: All-in-one gan compression by unified optimization framework. In ECCV, 2020. 2,
[20] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020. 2
[21] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019. 2

---

[22] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NewrIPS, 2021.
[23] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 2
[24] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vac-2. In NeurIPS, volume 32, 2019.
[25] Chitwan Saharia, William Chan, Huiwen Chang. Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palete: Image-to-image diffusion models. In SIGGRAPH, 2022.
[26] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[27] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaptive normalization CVPR, 2020. 2
[28] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with extt-guided diffusion models. arXiv preprint arXiv:2112.
2021. 2
[29] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong. Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021. 2
[30] Gwanghyun Kim and Jong Chul Ye. Diffusionclip: Text-guided image manipulation using diffusion models. arXiv preprint arXiv:2/ 10.02711, 2021. 2
[31] Jun Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexci A Efros. Generative visual manipulation on the natural image manifold. In ECCV, 2016. 2
[32] Or Patashnik. Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021. 2
[33] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019.
[34] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In CVPR, 2020. 2
[35] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019.
Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang,
[36] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 170400170..4866,, 2017. 2
[37] Mark Sandler, Andrew Howard, Menglong 7hu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[38] Yang Song, Jascha Sabk-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. In ICLR, 2020. 2,
[39] Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. 2
[40] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In ICLR, 2022. 2
[41] Song Han, Huizi Mao, and William Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In ICLR, 2016. 2
[42] Yihui He. Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. AMC: AutoML for Model Compression and Acceleration Mobile Devices. In ECCV, 2018. 2
[43] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurlPS, 2017. 2
[44] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017. 2
12

---

[45] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang, Learning efficient convolutional networks through network slimming. In ICCV, 2017. 2
[46] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang. Kwang- Ting Cheng. and Jian Sun. MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning. ICCV, 2019. 2
[47] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1606, 06/60, 2016. 2
[48] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifica- tion using binary convolutional neural networks. In ECCV, 2016. 2
[49] Kuan Wang, Zhijian Liu, Yujun Lin. Ji Lin, and Song Han. HAQ: Hardware-Aware Automated Quantization with Mixed Precision. In CVPR. 2
[50] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang. Vijayalakshii Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint N.X.::8885000000 2018. 2
[51] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In CVPR, 2018. 2
[52] Barret Zoph and Quoe V Le.. Neural Architecture Search with Reinforcement Learning. In ICLR, 2017. 2
[53] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Lc. Learning Transferable Architectures for Scalable Image Recognition. In CVPR, 2018.
[54] Haoxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable Architecture Search. In ICLR, 2019. 2
[55] Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In ICLR, 2019. 2
[56] Mingxing Tan, Bo Chen, Ruoming Pang. Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. MnasNet: Platform-Ave Aware Neural Architecture Search for Mobile. In CVPR, 2019. 3
[57] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardwaro-Aware Efficient ConwNet Design via Differentiable Neural Architecture Search. In CVPR, 2019.
[58) Ji Lin, Wei-Ming Chen, Yujun Lin, John Cohn, Chuang Gan, and Song Han. Meunet: Tiny deep learning on iot devices. In NeurlPS, 2020.
[59] Ji Lin, Richard Zhang, Frieder Ganz, Song Han, and Jun- Zhu. Anycost gans for interactive image synthesis and editing. In CVPR, 2021. 3
[60] Han Shu, Yunhe Wang, Xu Jia. Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu. Co- evolutionary compression for unpaired image translation. In ICCV, 2019. 3
[61] Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Federico Perazzi, and Lun-Yuan Kung. Content-aware gan compression. In CVPR, 2021. 3
[62] Ruixin Ma and Junying Lou. Cpgan: An efficient architecture designing for text-to-image generative adversarial networks based canonical polyadic decomposition. Scientific Programminng 2021. 3
[63] Angeline Aguinaldo, Ping-Yeh Chiang, Alex Gain, Ameya Patil, Kolten Pearson, and Soheil Feizi. Compressing gans using knowledge distillation. arXiv prepeint 1xXiv0190,.0019, 2019.
[64] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. NewrlPS, 2015. 3
[65] Hao Li, Asim Kaday, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ICLR, 2016. 3
[66] Baoyuan Liu, Min Wang. Hassan Foroosh, Marshall Tappen, and Marianna Pensky. Sparse convolutional neural networks. In CVPR, 2015. 3
13

---

[67] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014. 3
[68] Haotian Tang, Zhijian Liu, Xiuyu Li, Yujun Lin, and Song Han. Torchsparse: Efficient point cloud
inference engine. In Sys, 2022.
[69] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations high resolutions. CVPR, 2017.
[70] Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. Sbnet: Sparse blocks network for fast inference. CVPR, 2018. 3,4,5
[71] Patrick Judd, Alberto Delmas, Saych Sharify, and Andreas Moshovos. Cnvlutin2: Ineffectuul-acciivttin-- and-weigtt-free deep neural network computing. arXiv preprint arXiv:/705.00/2: 2017.
[72] Shaohuai Shi and Xiaowen Chu. Speeding up convolutional neural networks by exploiting the sparsity of rectifier units, arXiv preprint arXiv: 174.00744 2017. 3
[73] Xuanyi Dong, Junshi Huang. Yang. and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In CVPR, 2017. 3
[74] Bowen Pan, Wuwei Lin, Xiaolin Fang, Chaoqin Huang, Bolei Zhou, and Cewu Lu. Recurrent residual module for fast inference in videos. In CVPR, 2018. 3
[75] Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty- aware semantic segmentation via deep layer cascade. In CVPR, 2017. 3
[76] Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. In CVPR, 2020. 3
[77] Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo. Exploring sparsity in image super-resolution for efficient inference. In CVPR, 2021. 3
[78] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Yitian Zhang, and Haojun Jiang. Spatially adaptive feature refinement for efficient inference. TIP, 2021.
[79] Yulin Wang, Yang Yue, Yuanze Lin, Haojun Jiang. Zihang Lai, Victor Kulikov, Nikita Orlov, Humphrey Shi, and Gao Huang. Adafocus v2: End-to-end training of spatial dynamic networks for video recognition. In CVPR, 2022. 3
[80] Mathias Parger, Chengcheng Tang, Christopher D Twigg, Cem Keskin, Robert Wang, and Markus Stein- berger. Deltacnn: End-to-end enn inference of sparse frame differences in videos. In CVPR, 2022.
[81] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015. 5
[82] Dmitryy Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: (60.:6600.0022 2016. 5,7
[83] Xun Huang and Serge Belongic. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017. 5
[84] Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018. 5,
[85] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[86] Yaoyao Ding, Ligeng Zhu, Zhihao Jia, Gennady Pekhimenko, and Song Han. los: Inter-operator scheduler for enn acceleration. MLSys, 2021. 5
[87] Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken, Taso: optimizing deep learning computation with automatic generation of graph substitutions. In SOSP, 2019. 5
[88] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In CVPR, 2016. 5, 17
[89] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint 503.030 2(7), 2015. 6
14

---

[90] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In arXiv preprint arXiv:2201.0266 02600, 2021.
[91] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong. Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In ICLR, 2021. 6
[92] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metrie. In CVPR, 2018. 6
[93] Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Berhhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NewrlPS, 2017. 6
[94] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022. 6
[95] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, 2017. 6
[96] Sheng- Wang, Oliver Wang. Richard Zhang, Andrew Owens, and Alexei Efros. Can-generated images are surprisingly casy to spot... for now. In CVPR, 2020, 10
197] Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. JCML., 2019. 19
[98] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D.
4. If you are using existing assets (e.g., code, data, models) or curating/relcusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix D.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Our code and benchmark datasets are available at https: /gi thub. com com/lmxyy/ sige.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Appendix D.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 5.
15

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "fefQ6Z09uJE", "reviewer": "Reviewer_ckKG", "summary": "This paper proposes spatially sparse inference (SSI) to accelerate interactive generative image editing. The key idea is to keep a backup of the original features and only perform computation for edited regions, which is usually sparse in the interactive setting. The paper also introduces some implementation improvements to further reduce the computational overhead, including pre-computing normalization parameters and kernel fusion. Experiments on several pipelines, datasets, and devices show that the proposed method can drastically reduce the computation and running speed, and outperforms other pruning or patch-based baselines. Ablation studies are conducted to verify the effectiveness of each proposed module.", "questions": "1. Considering that the key idea of spatially sparse inference is not new and that the key spatially sparse computing module is based on SBNet [67], the novelty of the proposed method is not significant to me. Authors may highlight their novelty.\n\n2. Please explain why $\\Delta A_l$ is divided into blocks instead of pixels (line 130).\n\n3. Do the \"patch\" and \"crop\" baselines use pre-computing normalization and kernel fusion? Please clarify this. ", "limitations": "Limitations and potential negative societal impact have been properly discussed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper takes the first attempt to apply region-specific computation to generative image editing. It makes use of the sparse nature of interactive image editing and avoids redundant computation in the unedited regions, which is very reasonable. As interactive image editing is a broadly studied application scenario and the running speed is important, I believe this work is of high interest to the community.\n\n2. To improve practical computing efficiency, the paper proposes pre-computing normalization parameters and kernel fusion. These implementation improvements effectively reduce computational overhead and accelerate inference in practice.\n\n3. Experimental studies are thorough. The proposed method is evaluated on several different pipelines, datasets, and devices, and its advantages over baselines are clearly demonstrated.\n\n4. The writing is generally clear and easy to follow.\n\nWeaknesses:\n\n1. The main weakness is the novelty, as the key idea of spatially sparse inference is not new and has been widely used before, e.g., [67, 72, 73, 74]. There is also a missing related work that performs region-specific computation: \"Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade\". In particular, the proposed method is mainly built based on SBNet [67]'s sparse kernel implementation. The differences in terms of how mask is derived and other implementation improvements are not very significant. \n\nThe following weaknesses are minor:\n\n2. At line 130, it is not explained why $\\Delta A_l$ is divided into blocks instead of pixels.\n\n3. Do the \"patch\" and \"crop\" baselines use pre-computing normalization and kernel fusion? Please clarify this. \n\nTypos:   \nLine 251: could saves -> could save   \nLine 262: the activations -> activations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4SWB61qGiU_", "reviewer": "Reviewer_2N5G", "summary": "This paper produces a speedup technique, spatially sparse inference, for the image manipulation method. The SIGE significantly reduces inference time and does not harm the performance of the original network.", "questions": "1. How about performance when editing regions are large(over 30%)?\n2. How to get the correct index of editing regions. In my opinion, since the convolution process, the difference mask  in Fig.3 can not represent the editing regions in the deep level of network.", "limitations": "See the questions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The motivation is reasonable. Finding the edit regions and making the inference process focus on the edit regions, this strategy can reduce the time cost intuitively.\n2. The experiments are sufficient. Table 2 can prove the ability of SSI well.\n\nWeaknesses:\nSee the questions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3k3NZqlNbo", "reviewer": "Reviewer_oeJz", "summary": "This paper proposed the Sparse Incremental Generative Engine (SIGE), which consists of several tricks (tiling-based sparse convolution, the use of pre-computing normalization parameters, and kernel fusion) to accelerate local image edits of generative models with little sacrifice in image quality.", "questions": "Please address the comments in the Weakness section.", "limitations": "Please see the Weakness section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- (Clarification) This paper is well-written and easy to follow.\n- (Quality) The evaluation of latency reduction on various hardware is interesting and useful. \n\nWeakness:\n- (Originality) This paper is a bit incremental and engineering: i) the key insight is straightforward and not surprising as the spatial correspondences between features of convolutional layers and generated images are well-known. ii) the proposed three tricks in SIGE are mostly engineering ones and a bit thin. Although they are given good names, they are a bit trivial without significant technical contributions, e.g. the core idea of tiling-based sparse convolution is mostly borrowed from SBNet with an incremental improvement trick, the use of pre-computing normalization parameters is a simple approximation that can be explained in a single sentence, the kernel fusion is mostly engineering tricks that only perform computation in the edited region and the copying overhead.\n- (Significance) The significance of the proposed method could be less than expected as it has an important limitation: its benefits may not hold for overlapping editing regions in a sequence of edits, which is a common case in the real world. In this case, the pre-computation stage is no longer one-time (i.e. the generated image may need to be pre-computed again before the next edit) and can make the proposed method less useful.\n- (Quality) The dilation hyper-parameters could be important to the final results and should be discussed in more depth.\n\nMissing References:\n\nFor \"Generative models\" in section 2, there are missing references for image-to-image translation and real image editing:\n\n[1] Abdal, R., Qin, Y. and Wonka, P., 2019. Image2stylegan: How to embed images into the stylegan latent space?. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4432-4441).\n\n[2] Abdal, R., Qin, Y. and Wonka, P., 2020. Image2stylegan++: How to edit the embedded images?. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8296-8305).\n\n[3] Zhu, P., Abdal, R., Qin, Y. and Wonka, P., 2020. Sean: Image synthesis with semantic region-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5104-5113).\n\nMinor/Typo:\n- Line 131: missing \".\"\n- Table 1 caption: \"LIPS\" -> LPIPS\n- In Table 1, the bold fonts are not always the best results, e. g. GAN Comp. LPIPS with G.T. is better than the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept