Wavelet Feature Maps Compression for Image-to-Image CNNs
Authors: Shahaf E. Finder
Yair Zohav
Maor Ashkenazi
Eran Treister
Venue: NeurIPS 2022 Accept (2022)
Keywords: Convolutional Neural Networks
Quantization
Wavelet Transform

Abstract: Convolutional Neural Networks (CNNs) are known for requiring extensive computational resources, and quantization is among the best and most common methods for compressing them. While aggressive quantization (i.e., less than 4-bits) performs well for classification, it may cause severe performance degradation in image-to-image tasks such as semantic segmentation and depth estimation. In this paper, we propose Wavelet Compressed Convolution (WCC)---a novel approach for high-resolution activation maps compression integrated with point-wise convolutions, which are the main computational cost of modern architectures. To this end, we use an efficient and hardware-friendly Haar-wavelet transform, known for its effectiveness in image compression, and define the convolution on the compressed activation map. We experiment with various tasks that benefit from high-resolution input. By combining WCC with light quantization, we achieve compression rates equivalent to 1-4bit activation quantization with relatively small and much more graceful degradation in performance. Our code is available at https://github.com/BGUCompSci/WaveletCompressedConvolution.

Wavelet Feature Maps Compression for Image-to-Image CNNs
Shahaf F. Finder , Yair Zohav * Maor Ashkenazi ", Eran Treister
The Department of Computer Science, Ben-Gurion University
[finders maorash] @post. bgu erantOcs. bgu.ac.il ac.
Abstract
Convolutional Neural Networks (CNNs) are known for requiring extensive compu- tational resources, and quantization among the best and most common methods for compressing them. While aggressive quantization (i.e., less than 4-bits) per- forms well for classification, it may cause severe performance degradation image-to-imas timage tasks such as semantic segmentation and depth estimation. In this paper, we propose Wavelet Compressed Convolution (WCC)-a novel ap- proach for high-resolution activation maps compression integrated with point-wise convolutions, which are the main computational cost of modern architectures. To this end, use efficient and hardware-friendly Haar-wavelet transform, known for its effectiveness in image compression, and define the convolution on the compressed activation map. We experiment with various tasks that benefit from high- righ-resolution input. By combining WCC with light quantization, we achieve compression rates equivalent 1-4bit activation quantization with relatively small and much more graceful degradation in performance. Our code is available at https:///ithuu. www.coom//ooo/e/ http://////ooo///aaaaaaaaaaaaaaa
Introduction
Over the past years, Convolutional Neural Networks (CNNs) have brought significant improvement processing images, video, and audio [35] 34). However, CNNs require significant computational and memory costs, which makes the usage of CNNs difficult in applications where computing power limited, e.g., on edge devices. address this limitation, several approaches have been proposed to reduce the computational costs of neural networks. Among the popular ones are weight pruning 255 223, architecture search [27 33), and quantization [36 4. principle, all these approaches can be applied simultancously top of each other to reduce the computational costs of CNNs.
Specifically, the quantization approach relieves the computational cost of CNNs by quantizing their weights and activation (feature) maps using low numerical precision so that they can be stored and applied as fixed point integers (29 4 In particular, is common apply aggressive quantization
(less than b-t precision) to compress the activation maps [18] However, it is known that compressing natural images using uniform quantization is sub-optimal. Indeed, applying aggressive quantization in certain CNNs can lead to significant degradation in the network's 's performance. The impact is especially evident for image-to- mage-tt--image tasks such as semantic segmentation (55) and depth prediction
(22). where each pixel has to be assigned value. E.g.. in recent work that targets quantized U-Nets [55]. activations bit rates are kept relatively high (~~ bits) while the weight bit rates are lower (down
to bits). Beyond that, note that the majority of the quantization works applied and tested on image classification 18 37 and references therein] and rarely on other tasks.
"Contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Standard quantization
10-1
Wavelet compression
MSE
10
10
2
4
5
8
Effective bit rate
Figure I: Comparison of standard quantization and our proposed wavelet compression. The ordinate represents MSE between the quantized and original activation maps based on 10 activation maps from the second hidden layer of MobileNetV3 (small) using the ImageNet data set.
This work aims to improve the compression of the activation maps by introducing Wavelet Com- pressed Convolution (WCC) layers, which use wavelet transforms to compress activation maps before applying convolutions. To this extent, we utilize to apply it (and its inverse) efficiently in linear complexity for each channel, using additions and subtractions only, thanks to the simplicity the Haar basis. The core idea of our approach is to keep the same top k entries in magnitude of the transformed activation maps with respect to all channels (dubbed as joint shrinkage) and perform the convolution in the wavelet domain on the compressed signals, saving significant computations. We show that the transform and shrinkage operations commute with the 1 x 1 (point-wiss) convolution, the heart of modern CNNs. This procedure is applied along with modest quantization to reduce computational costs further.
transform
[33 due to our ability
To summarize, our key contributions are: 1) We propose Wavelet Compressed Convolution (WCC), a novel approach to compress x convolutions using modified wavelet compression technique. 2) We demonstrate that applying low-bit quantization on popular image-to-image CNNs may yield degradation in performance. Specifically, we show that for object detection, semantic segmentation, depth prediction, and super-resolution. 3) We show that using WCC dramatically improves the results for the same compression rates, using it as drop-in replacement for the 1 x I convolutions in the baseline network architectures.
2 Related Work
Quantized neural networks. Quantized neural networks have been quite popular recently and are cxhibiting impressive progress in the goal of true network compression and efficient CNN deployment. Quantization methods include [70] 69 5 and in particular [8 [I] which show that the clipping parameters- an essential parameter in quantization schemes- can be learned through optimization. Beyond that, there are more sophisticated methods to improve the mentioned schemes. For example, dynamic quantization schemes utilize different bit allocations at every layer [4] (,5 Non-uniform methods can improve the quantization accuracy [66] but require a look-up table, which reduces hardware efficiency. Quantization methods can also be enhanced by combination with pruning (56) and knowledge distillation for better training
The works above focus on image classification. When considering image-to-image tasks (e.g. semantic segmentation) networks tend to be more sensitive to quantization of the activations. In the work of [21. targeting segmentation of medical images, the lowest bit rate for the activations is 4 bits, and significant degradation the performance is evident compared
bits. These results are consistent with the work of [55) that was mentioned earlier, which uses a higher bit rate for the activations than for the weights.
(65) use weight (only) quantization for medical image segmentation as an attempt to remove noise and not for computational efficiency. The recent work of [1]] shows both a sophisticated post-training quantization scheme and includes fine-tuned semantic segmentation results. Again significant degradation is observed when going from 6 to 4 bits. One exception is the work of (26 that uses ternary networks (equivalent to bits here) and segments one medical data set relatively well compared to its full-precision baseline.

---

In this work, we focus on the simplest possible quantization scheme: uniform (across all weights and activations). quantization-awan aware training, and per-layer clipping parameters. That to ensure hard- ware compatibility and efficient use of available training data. In principle, one can run our platform regardless of the quantization type and scenario (e.g., non-uniform/mixed precision quantization). Also, since we target activations' compression, any method focused on the weights (e.g., pruning 56)) can also be combined with our proposed method.
Wavelet transforms in neural networks. Wavelet transforms are widely used in image processing
48 For example, the JPEG2000 format 49 uses the wavelet domain to represent images as highly sparse feature maps. Recently, wavelet transforms have been used to define convolutions and architectures CNNs for various imaging tasks: [20] incorporate wavelets in Generative Adversarial
Networks to enhance the visual quality of generated images as well as improve computational
performance;
(28) present a network architecture for super-resolution, where the wavelet coefficients are predicted and used to reconstruct the high-resolution image; [15] and [6] use wavelet transforms
in place of pooling operators to improve CNNs performance. The former uses dual-tree complex wavelet transform (32), and the latter learns the wavelet basis as part of the network optimization. In all of these cases, the wavelet transform is not used for compression but rather to preserve information with its low-pass filters. 40] suggest using modified U-Net architecture for image-to-inen translation. There, wavelet transforms are used for down- nown-sampling, and the inverse is used for up-sampling. This work is rrchitecture-speciie architectures. In contrast, our proposed WCC layer can easily replace 1 x 1 convolutions regardless of the CNN architecture. Hence our framework is, in principle, also suitable for post-training quantization 6 where the data is unavailable, and the original network is not retrained.
and the method can not be easily integrated into other
In the context of compression, 633 propose wavelet-based approach to learn basis functions for the wavelet transform to compress the weights of linear layers, as opposed to compression of the activation as we apply here. We use the Haar transform (as opposed to learned one) for its hardware efficiency. Using a different transform (known or learned) is also possible at the corresponding computational cost. [51] introduce quantization in the wavelet domain, as we do in this work. However, the authors suggest improving the quantization scheme by learning a different clipping parameter per wavelet component, but without the feature shrinkage stage, which is the heart of our approach (we use the same clipping parameter for the whole layer using 8 bits for hardware efficiency). As mentioned before, one schemes, including the one proposed by [51]. taking into account the additional hardware complexity of using different clipping parameters per component.
our WCC together with different types of quantization
3 BBckkround
Quantization-aware training. Quantization schemes can be divided into post- and quantization- aware training schemes. Post-training schemes perform model training and model quantization separately, which is most suitable when the training data is unavailable during the quantization phase [6 45 g. On the other hand, quantization-awaa model's 's weights as an additional training phase. Such schemes do require training data but generally provide better performance. Quantization schemes can also be divided into uniform vs. non-uniform methods, where the latter is more accurate, but the former is more hardware- hardware-friendly [37 30) Lastly, quantization schemes can utilize quantization parameters per-channel within each layer or utilize
training schemes are used to adapt the
these parameters only per layer (where all the channels share the same parameters). Similar to before, per-channel methods are difficult to exploit in hardware, while per-layer methods are less accurate but more feasible for deployment on edge devices. This paper focuses on per-layer and uniform quantization-avee -aware training for both weights and activations and aims to improve on top of it. Other quantization schemes can be applied within our wavelet compression framework well.
In quantization-awan training, we set the values of the weights to belong to a small set so that after training, the application of the network can be carried out in integer arithmetic, i.e., activation maps are quantized as well. Even though we use discontinuous rounding functions throughout the network, quantization-awve aware training schemes utilize muddient-based methods to optimize the network's weights (25 68) a nutshell, when training, we iterate on the floating-point values the weights. During the forward pass, both the weights and activation maps are quantized, while during the backward pass, the Straight Through Estimator (STE) approach used 7 where we ignore the
rounding function, whose exact derivative is zero
3

---

The specific quantization scheme that we use is based on the work of [3]. First, the pointwise quantization operator is defined by:
(t) (2"-))
(I)
where E [-1, or 1 [0,
1] for signed or unsigned quantization, respectively The parameter b the number of bits that are used to represent t. Each entry undergoes three steps during the forward pass: scale, clip, and round. That is, to get the quantized version of a number, we apply:
requ-1(cling = 11, if signed aqpb (clip( & 0))
if unsigned
Here, æ. 2h are the real-valued and quantized tensors, and is the clipping parameter. The parameters in (2) play an important role in the eπror generated at each quantization and should be chosen carefully. Recently, [37[8] introduced an effective gradient-based optimization find the clipping values a for each layer, again using the STE approximation. This enables the quantized network to be trained in an end-to-end manner with backpropagation. To further improve the optimization, weight normalization is also used before each quantization.
Haar wavelet transform and compression. In this section, we describe the Haar-wavelet transform in deep learning language and its usage for compression. See [59] for more details on the use of wavelets for image compression. Given an image channel x, the one-level Haar transform can be achieved by separable 2D convolution with a stride of using the following weight tensor:
[ [ W 2
1
1 =|
These kernels can also be expressed as a composition of separable ID kernels [L, 1//v and [1, v2. The result of the convolution Y2. Yal = Conv(W: x) has four channels, each of which has half the resolution of x. The leftmost kernel in (3) is averaging kernel, and the three right kernels are deteeddetecoors. This, together with the fact that images are piece-wiss wise smooth, leads relatively sparse images y2, Ya, Y4. Hence, if retain only the few top-magnitude entries in these vectors, we keep most of the information, as most of the entries we drop are zeros. That is the main idea of wavelet compression. We denote the Haar-waveles transform by y HWT((),, where y is defined as the concatenation of the vectors y1, .... Y4 into one. Since the kernels in (3) form an orthonormal basis, applying the inverse transform obtained by the transposed convolution of (3)
iHWT(y) Conv-transposed W, ( Y23,
However, unlike y2, V3, Y4, the image yı is not sparse, and is just the down-sampled Hence, in the multilevel wavelet transform, we recursively apply the convolution with W on low-pass filtered sub-bands yı generate further down-sampled sparse channels. For example, 2-level Haar transform can be summarized as
Conv(W x) = y2 vi]:
Conv(W,y) yl) = 12. (
(5)
and the resulting transformed image with 2-levels can be written as the concatenated vector HWT(x) = y = vv..2..0 y2.
(6)
In this work we use levels all the experiments. To apply compression we define the operator T that extracts the top k entries in magnitude from the vector y:
compressed
T HWT(x).
To de-compress this vector we first zero- fill compessee (i.e., multiply with T) and apply the inverse Haar transform, which involves with the transposed operations in the opposite order.
4 Wavelet Compressed Convolution
In this work, we aim reduce the memory bandwidth and computational cost associated with convolutions performed on intermediate activation maps. To this end, we apply the Haar-wavelet
In most standard CNNs, the ReLU activation is used: hence the activation feature maps are non-neet non-negative and can be quantized using an unsigned scheme. If diffeernnt activation function that is not non negatvee is used, or, in our case, the wavelet coefficients quantized, signed quantization should be used instead.

---

Figure 2: The workflow of WCC on a 3-channel input from the Cityscapes dataset. A: Haar-wavelet transform; B: Joint shrinkage, transforming the input into equally sized ID vectors and single bit-map to represent the spatial location of the top entries; C: I convolution over the ID vectors; D: Inverse shrinkage (zero filling); E: Inverse Haar transform.
transform to compress the activation maps, in addition to light quantization of bits. Our method is most efficient for scenarios with high-resolution feature maps (i.e., large images, whether in 2D or 3D), where the wavelet compression is most effective. Such cases are mostly encountered in image-to-image tasks like semantic segmentation, depth prediction, image denoising, in-painting, super-resolution, and more. Typically, such scenarios, the size and memory bandwidth used for the weights are relatively small compared to those used for the features.
Convolution in the wavelet domain. We focus on the compression of fully-coupled lutions, as these are the workhorse of lightweight and efficient architectures like MobileNet [50], ShuffleNet 44 EfficientNet [52]. ResNeXt [64], and ConvNext 42). All these modern architectures rely on 1 x 1 point-wise convolutions in addition to grouped or depthwise spatial convolutions (i.e., with network- the x operations dominate the inference cost (see Appendix A and Appendix C). Since we focus on computational efficiency, we use the Haar transform, as it is the simplest and most computationally efficient wavelet variant. Indeed, the Haar transform involves binary pooling- like operations, which include only additions, subtractions, and a bit-wise shift. Other types of wavelets also suitable in our framework while considering the corresponding computational costs-we demonstrate this point in Appendix
convo-
larger kemels), which comprise a small part of the computational effort in the
The main idea of our method is to transform and compress the input using the wavelet transform prior to the 1 convolution, then apply it in the wavelet domain on a fraction of the input size. Since the wavelet compression is applied separately on each channel, it commutes with the 1 x 1 convolution. Hence, applying the convolution in the wavelet domain equivalent to applying it in the spatial domain.
More precisely, the advantage of the wavelet transforms is their ability to compress images. Denote the Haar transform operator as H, i.e., Hx HWT(x). Then, for most natural images x we have
x H
(8)
where T is the shrinkage operator in Eq. 7 (top & extractor). Because of its orthogonality. H is the inverse transform iHWT. Our WCC layer is defined by:
WCC(K1x1.x) = H T THxT
(9)
where here x is a feature tensor and K xi a learned x convolution operator. The workflow illustrated in [Fiuure 2] and an explicit algorithm appears in Appendix B] The convolution operates on can significantly reduce the dimensions of the channels without
the compressed domain, hence, loosing significant information, this leads to major savings in computations. Note that H and T operate on all channels in the same way, i.e., extracts the same entries from all channels. We motivate on this next.
Joint hard shrinkage. Prior the 1 x convolution, the spatial wavelet transform applied, and we get sparse feature maps. Since the different channels result from the same input image propagated through the network, they typically include patterns and edges at similar locations. Hence the sparsity pattern of their wavelet-domain representation is relatively similar. This idea of redundancy in the channel space exploited in the works of [2 16 [) where parts of the channels are used to represent the others. Therefore, in T, perform joint shrinkage operation between all the channels, in
5

---

which we zero and remove the entries with the smallest feature norms across channels, resulting in compressed representation of the activation maps] The locations of the non-zeros in the original image are kept in single index list or a bit-map for all the channels in the layer, as they are needed for the inverse transform back to the spatial domain. We also apply light 8-bit quantization to the transformed images to improve the compression rate further. The weights and wavelet-domain activations are quantized using the symmetric scheme described in insecciionn 3]
Equivalence of the convolutions. Our method aims at compression only. Hence, we show that a 1 x convolution kernel can be applied both in the spatial and in the compressed wavelet domain. By its definition, we can write a 1 x 1 convolution as summation over channels. That is:
x = y; y; Σ3 Kiy
(10)
where ki E are the weights of the convolution tensor. Now, suppose we wish to compress the result y, through (8). Because T and H are separable and spatial, we get:
y: y (IxIx)))); = T ky,TTH
(11)
where the exact equality holds assuming that T extracts the same entries in both cases. Hence,
y = H KIxTTHX
(12)
That is, the wavelet compression operator, which known to be highly efficient for natural images, commutes with the 1 x convolution, so the latter can be applied on the compressed signal and get the same result as compressing the result directly. This introduces an opportunity to save computations on the one hand and use more accurate compression on the other.
The description above is suited for 1 convolutions only, while many CNN architectures involve spatial convolutions with larger kernels, strides, etc. The main concept is that fully coupled convo- lutions that mix all channels are expensive, inefficient, and redundant when used with large spatial kernels [7]. The spatial mixing can be obtained using separable convolutions at less cost without losing efficiency (see Appendix Ap. Since we aim at saving computations, separating the kernels in the architecture is recommended even before discussing any form of lossy compression [10] (50] Furthermore, separable convolutions (and the Haar transform) can be applied separately in chunks of channels, and the memory bandwidth can be reduced. The 1 x 1 convolution, on the other hand, involves all channels at once, and it is harder to reduce the bandwidth in this case. With our approach, the input for the convolution is compressed, and the bandwidth is reduced (see Appendix H.
Limitations. Our approach best suited for applications where feature maps are of high resolution.
This includes various computer vision tasks, as we demonstrate in section 5] The majority of other works show only classification results, which resolution. For example, for ImageNet, images are typically resized to 224 x 224 and down- downledmpled 7 using few layers. For such a low resolution, the wavelet shrinkage is not as effective as it is for high-resolution feature maps. On the other hand, quantization by itself is already very effective for classification, hence tackling other tasks in this context may be highly beneficial.
less sensitive task than others for the input's
aggressively
5 Results
This section evaluates our proposed WCC layer's performance on four tasks: object detection, semantic segmentation, monocular depth estimation, and super-resolution. The majority of the feature maps in these tasks are relatively large and very well-suited for wavelet compression. We use ouantization-awall training based the work of 33 for the baseline with and without WCC.
As mentioned infsection 2 most quantization works evaluate their performance on image classification. An exception is the work of [46), which shows results for object detection and semantic segmentation, although we were unable to reproduce their results. Therefore, when comparing with their work, we measure the effectiveness of WCC in the relative degradation of the score when compressing the activation further than 8-bit; this is due to the absolute accuracy being strongly dependent on the baseline guantization mechanism and training process.
'Regardless of the joint sparsity of the channels, some approaches suggest taking the left-upmost part of the wavelet transform for any image, in principle, the joint sparsity may suffice for general set of images as well.

---

Original input image from ImageNet
(b) Feature maps from ResNet50 layers.
(c) 2-bit quantized layers (unsigned).
(d) 2-bit equivalent wavelet compressed.
Figure 3: Feature maps from layers and (top and bottom triplets, respectively) of pre-trained ResNet50 ImageNet. The maps are compressed with uniform quantization (2-bit) and wavelet compression (25% shrinkage + 8-bit quantization, equivalent to 2-bit quantization). Clearly, the wavelet compression loses much less information than aggressive quantization.
The general training scheme used in the subsequent experiments is similar to 37 We either start with full-precision pre-trained network weights or train a full-precision network to convergence. Afterward, gradually reduced the bit rates and the WCC compression factor. We detail the rest of our experimental setup in each section separately. In each network presented in the results, the 1 x convolution layers comprise 85% 90% of the total operations (e-g., see see aeelAppeddix Cj. We implemented our code using ToTch [47), based on Torchvision and public implementations of the chosen networks. We cost of each model is measured in Bit-Operations (BOPs, see Appendix D)
experiments NVIDIA 24GB RTX 3090 GPU. The computational
5.1 Qualitative Compression Assessment
First, qualitatively demonstrate the advantage of approach. We compare standard quanti- zation's 's mean square error (MSE) to our proposed method on feature map from a pre-trained MobileNetV3 applied on a batch of 10 images from the ImageNet dataset [34]. We consider an effective bit rate range of [2, 8). The bit rate of the shrunk wavelet coefficients is 8; multiplying it with the compression ratio yields the effective bit rate (e.g., bit and 25% shrinkage is equivalent to bit). Figure I shows the MSE comparison per effective bit rate, and Figure 3Jpresents visualization comparison of typical activation map compressed by a standard quantization and our method. Both show that the information loss is more significant in standard quantization.
5.2 Object Detection
We apply our method EfficientDet 54 using the EfficientDet-D1 variant. We train and evaluate the
networks the MS COCO 2017 39 object detection dataset. The dataset contains 1 11K training images and 5K validation images of complex everyday scenes, including detection annotations for 80 classes of common objects in their natural context. Images are of size 640 x 480 pixels, and as defined for the DI variant, are resized to 640 640 pixels. We use a popular publicly available PyTorch EfficientDet implementation" and adopt the training scheme suggested by its authors. We use the AdamW optimizer, with a learning rate of 10 when initially applying WCC layers and -4 for finetuning. In addition, apply learning rate warm-up in the first epoch of training, followed by a cosine learning rate decay. Each compression step is finetuned for 20 to 40 epochs. The results

---

Table 1: Validation results on MS COCO using EfficientDet-D1. Degradation from the baseline parentheses.
Table Validation results for Pascal VOC us- ing
Degrada-
tion from the baseline is in parentheses.
Precision
Wavelet
BOPs (B)
t mAP
Precision (W/A)
Wavelet shrinkage
BOPs (B)
t mloU
(W/A) shrinkage
FP32
None
6,144
40.08
46
446
FP32
None
4,534
0.729
8bit 4bit / 4bit
None
280.4
35.34
4bit 8bit
None
141
0.709
None
185.8
24.70 (-1061)
4bit / 4bit
None
70
0.668 (0004)
Our baseline + WCC
Our baseline
WCC
4bit 8bit
None
280.4
31.44
FP32
None
4,534
0.715
4bit/ / 8bit
50%
198.5
31.15 (0.29)
4bit 8bit
None
0,675
4bit / 8bit
25%
155.4
27.49 (3.95)
4bit 8bit
50%
76
0.661 (4014)
4bit / 8bit
25%
42
0.583 (2002)
4bit / 8bit
12.5%
24
0.515 (0.100)
presented in Table I We train quantized network using 4-bit weights and 8-bit activations, used as baseline, and then further compress the activations using WCC with 50% compression factor, which results in comparable BOPs of 4bit/4bit quantized network. The result significantly surpasses the accuracy achieved by [46], both in the absolute score and in relative degradation from the baseline. We continue to show that even when applying a 25% compression factor, method outperforms the 4bit/4bit network of [46] while reducing the BOPs significantly.
5.3 Semantic Segmentation
For this experiment, use the popular DeeplabV3plus [0] with MobileNetV2 backbone [50]. We evaluated our proposed method on the Cityscapes and Pascal VOC datasets. The Cityscapes dataset contains images of urban street scenes; the images are of size 1024 2048 with pixel-level annotation of 19 classes. During training, we used random crop of size 768 x 768 and no crop for
12
the validation set. The Pascal VOC [19] dataset contains images of size 513 513 with pixel-level annotation of 20 object classes and background class. We augmented the dataset similarly to used two configurations for the weights- and bits-and for each, we used various compression rates for the activations, both for the standard quantization and WCC.
10
For optimization, we used SGD with mo- mentum 0.9, weight decay 10-4, Iearning rate decay 0.9, and batch size 16. For Cityscapes, we trained the full-precision model for 160 epochs with a base learning rate of 10-1 Each compression step was finetuned for 50 epochs with learning rate of 10-2 For Pascal VOC, we trained the full-precision model for 50 epochs with a learning rate of 10-2 Each compression step was finetuned for 25 epochs with a base learning rate of 2 10
0.6
mloU
0.4
WOC-AS 12.55
0.2
Quaviied SEE8.
Quanteed 458A
0
1,000 1,500
2,000
Bil-Operattios
Table 2 presents the results for Pascal VOC.
In relative degradation. [46] experienced drop of ~0.041 between 4bit/8bit and 4bit/4bit. In contrast, our method experi- ences ~0.014 drop when applying 50% compression, achieving an equivalent score while using worse baseline. FFigure 4add Figure coomaar the results for Cityscapes with different compression configurations. Note that we show compression rates that are computationally equal to
Figure 4: Validation results for Cityscapes using (ilaabbbpllusMooolele compared to BOPs for different bit-rate quantizations and WCC configurations.

---

Input image
Ground truth
Baseline 8/8
(d) Baseline 8/4
(c) WCC 8/8 100%
WCC 8/8 50%
(g) WCC 8/8 25%
(h) WCC 8/8 12.5%
Figure 5: Cityscapes segmentation results. All networks use weight quantization of 8- bits. aa) input
image. (b) ground truth. [c) (d) normal quantization with 8- and 4-bits activations respectively.
e
WCC with 8-bits activations and shrinkage rate of 100%, 50%, 25% and 12.5% respectively.
2- and 1-bit, which degrade gracefully. See|Appendix E for the detailed results, and Appendix F for additional results with different types of wavelets.
5.4 Monocular Depth Estimation
We apply our method Monodepth2 [22] a network architecture for monocular depth estimation task of estimating scene depth using single image. We evaluated the results on the KITTI dataset [1]] containing images of autonomous driving scenarios, each of size ~ 1241 x376 pixels. The train/validation split is the default selected by Monodepth2 (based on and we evaluate it on the ground truths provided by the KITTI depth benchmark.
We extended the code base to support Mo- bileNetV2 a backbone, and adapted the con- depth decoder to use compression factor, volutions for an extra without harming the accuracy. All the lay- ers, except the first and final convolutions are compressed. We use the A optimizer, learning rate of 1.4 10-1, a weight decay of 10-3 and run each experiment for 20 epochs.
Table 3: Monndeptt2/MobbieNeet2)) results mea- sured by AbsRel and RMSE.
Precision
Wavelet
BOPs (B)
4 AbsRel RMSE
(W/A)
shrinkage
FP32
None
1.163.6
0.093
4.022
Sbit / 8bit 8bit 4bit
None
133.6
0.092
4,018
Table presents a comparison between stan- dard quantization and WCC for Monodepth2. Using WCC, we compress the network to very low bit-rates while having relatively mini- mal and more graceful performance degra- dation. Using WCC layers with 50% com- pression resulted in comparable scores to alternative 8bit/4bit quantization (with similar BOPs). When applying compression factor of 25%,
None
99.26
0.097
4.166
Sbit / 2bit
None
82.1
0.268
8.223
50%
103.9
0.098
4.217
8bit 8hit
25%
88.5
0.112
4.663
8bit / 8bit
12.5%
80.8
0.131
5,046
and even 12.5%, we achieve superior results to the quantized alternative. qualitative comparison is presented in Appendix G
5.5 Super-resolution
Another experiment we perform for the task of super-resolution, which aims to reconstruct high-resolution image from a single low-resolution image. For this task, we chose the popular EDSR network [38], trained on the DIV2K dataset [] and used its basic configuration for 2x super-resolution. DIV2K contains images in 2K resolution, and the model is trained on 48 x 48 random crops of the down-scaled train images as input (which are not down-scaled further throughout the network). We used the official implementation of EDSR and did not change the training process.

---

The model consists of three parts: head, body, and tail. In our experience, quantizing the head and tail showed severe degradation with even the lightest quantization. Therefore, for this experiment, we only compress the body. It is important to mention that this method uses skip-connection between the head and the tail, so is to be expected that even aggressive quantization can achieve reasonable results, differently than the other models we show in the paper.
In addition, EDSR uses full 3 3 convolutions. Hence, we use the technique described in Appendix A] replacing each full-convolution with a set of two convolutions, a depthwise- Empir-
Table 4: EDSR compression results for the task of 2x super-resolution. Note that the first row is the reported result from reproduction of it using separable convolutions.
[8), while the second is our
3 x followed by pointwise-1 ically, we see no change in results in our ex- periment following this change, and we could still to replicate the results reported by [38
Precision (WIA)
Wavelet shrinkage
BOPs (B)
PSNR
Table shows the results for two configura- tions, WCC without quantization and WCC with light quantization (8/16-bit). The BOPs are calculated for the body's convolutions. The first two rows show our reproduction of 38] results using separable convolutions. The next three rows show the results using our WCC with no quantization, for which the drop in accuracy minimal. Furthermore, in the quantized results, we can see that adding 50% compression to the 8bit/16bit and Sbit/Sbit setups did not influence the accuracy while reducing the costs. Besides that, we again see that WCC achieves better accuracy than quantization alone with aggressive compres- sion (e.g.. 8bit/4bit yields PSNR of 34.49 while 8bit/Sbit+50% yields 34.53).
FP32/FP32 38
None
975,838
35.03
FP32/FP32
None
123,674
35.02
FP32/FP32
50%
70,017
34.98
FP32/FP32
25%
42,910
34.93
FP32/FP32
12.5%
29,357
34.76
8bit / 16bit
None
15,459
34.55
8hit 8hit 8hit 4hit
None
7,730
34.53
None
3,865
34.49
8bit / 16bit
50%
8,961
34.55
Shit 8hit 8hit / 8hit
50%
4,480
34.53
25%
2,786
34.50
Conclusion
In this work, we presented a new approach for feature map compression, aiming reduce the memory bandwidth and computational cost of CNNs working at a high resolution. Our approach is based on the classical Haar-wavelet image compression, which has been used for years in standard devices and simple hardware, and in real-time. We save computational costs by applying the shrunk wavelet domain together with light quantization. We show that this approach surpasses aggressive quantization using equivalent compression rates.
convolutions on
Acknowledgments and Disclosure of Funding
The research reported in this paper was supported by the Israel Innovation Authority through the Avatar consortium. The authors also thank the Israeli Council for Higher Education (CHE) via the Data Science Research Center and the Lynn and William Frankel Center for Computer Science at BGU. SF is supported by the Kreitman High-tech scholarship.
References
[1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[2] MohammadHossein AskariHemmat, Sina Honari, Lucas Rouhier, Christian S. Perone, Julien Cohen-Adaf Yvon Savaria, and Jean-Pierre David. U-net fixed point quantization for medical image segmentation. Medical Imaging and Computer Assisted Intervention (MICCAI), Hardware Aware Learning Workshop (HAL-MICCAI), Springer, 2019.
10

---

[3] Seung- Seung-Hwan Bae, Hyuk-Jae Lee, and Hyun Kim. De-ac: Deep correlation praseation-bbsed adaptive compression of feature map planes in convolutional neural networks. In 2021 IEEE International Symposium Circuits and Systems (ISCAS), pages 1-5. IEFE, 2021.
[4] Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. Advances in neural information processing systems, 31, 2018.
[5] Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Aciq: Analytical clipping for integer quantization of neural networks, ArXiv, abs/ 0bs//18100555 110.05723, 2018.
[6] Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems, 32, 2019.
[7] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 318.34322.2 2013.
[8] Yaohui Cai, Zhewei Yao. Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178, 2020.
[9] Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed-precision neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2349-2358, 2020.
[10] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
[11] Jungwook Choi, Swagath Venkataramani, Vijayalasshmi Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2-bit quantized neural networks. In MLSys, 2019.
[12] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[13] Ingrid Daubechies. Ten lectures on wavelets. SIAM, 1992.
[14] Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. The Conference on Neural Information Processing Systems (NeurIPS), 2020.
[15] Yiping Duan, Fang Liu, Licheng Jiao, Peng Zhao, and Lu Zhang. SAR image segmentation based on nnavolutionalwwavell neural network and markov random field. Pattern Recognition, 64:255-267, 2017.
[16] Moshe Eliasof, Jonathan Ephrath, Lars Ruthotto, and Eran Treister. Multigrid-in--cannels neural network architectures. arXiv preprint 1xX0:20011009222 2020.
[17] Jonathan Ephrath, Moshe Eliasof, Lars Ruthotto, Eldad Haber, and Eran Treister. Leanconvnets: Low-cost yet effective convolutional neural networks. IEEE Journal of Selected Topics in Signal Processing, 14(4):894 904, 2020.
[18] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. The International Conference on Learning Representations (ICLR), 2020.
[19] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-136, 2015.
[20] Rinon Gal, Dana Cohen Hochberg, Amit Bermano, and Daniel Cohen-Or. Swagan: A style- based wavelet-driven generative model. ACM Transactions on Graphics (TOG), 111, 2021.
11

---

[21] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
[22] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth prediction. In The International Conference on Computer Vision (ICCV), October 2019.
[23] Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
[24] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 1580-1589 2020.
[25] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[26] Mattias P Heinrich, Max Blendowski, and Ozan Oktay. Ternaryntt: faster deep model inference without gpus for medical 3d segmentation using sparse and binary convolutions. International journal of computer assisted radiology and surgery, 13(9):3111-120 2018.
[27] Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for MobileNetV3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[28] Huaibo Huang, Ran Hc, Zhenan Sun, and Tieniu Tan. Wavelet-smnet: A wavelet-based cnn for multi-scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision, pages 1689-16999 2017.
[29] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18:187:1-187:30, 2017.
[30] Sangil Jung, Changyong Son, Scohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[31] Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization- aware knowledge distillation. arXiv preprint 1911.29111./24// 2019.
[32] Nick Kingsbury. The dual-tree complex wavelet transform: new efficient tool for image restoration and enhancement, In 9th European Signal Processing Conference (EUSIPCO 1998), pages 1-4. IEEE, 1998.
[33] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9799-9808, 2020.
[34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[35] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 52167553):436-444 2015.
[36] Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 5813-5823 2017.
[37] Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. The International Conference on Learning Representations (ICLR), 2020.
12

---

[38] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nab, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 136-144, 2017.
[39] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[40] Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet- enn for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 773-782, 2018.
[41] Yuang Liu, Wei Zhang, and Jun Wang. Zero-shot adversarial quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1512-1521, 2021.
[42] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 11976-11986, 2022.
[43] Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling, Relaxed quantization for discretized neural networks. arXiv preprint arXiv:1810.01875. 2015 8.
[44] Ningning Ma, Xiangyu Zhang, Hai-Th Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018
[45] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight cqualization and bias correction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1325-1334, 2019.
[46] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart van Baalen, and Tijmen Blankevoort. white paper on neural network quantization. arXiv preprini arXiv:2106.08295, 2021.
[47] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vetioo Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems, 2017.
[48] Piotr Porwik and Agnieszka Lisowska. The haar-wavelet transform in digital image processing: its status and achievements. Machine graphics and vision, 13(1/2):79 2004.
[49] Majid Rabbani. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging. 11(2):286, 2002.
[50] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear boutenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510 4520,,, 2018
[51] Qigong Sun, Yan Ren, Licheng Jiao, Xiufang Li, Fanhua Shang, and Fang Liu. MWQ: Multiscale wavelet quantized neural networks. arXiv preprint 1XXv:2203300363. 2021.
[52] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-61 14. PMIR, 2019.
[53] Mingxing Tan and Quoc Le. EfficientnetV2: Smaller models and faster training. In International Conference Machine Learning, pages 10096-10106. PMLR, 2021.
[54] Mingxing Tan, Ruoming Pang. and Quoc V Le. Efficientdet: Scalable and efficient object detec- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781-10790. 2020.
[55] Zhiqiang Tang, Xi Peng, Kang Li, and Dimitris N Metaxas. Towards efficient u-nets: A coupled and quantized approach. IEEE transactions on pattern analysis and machine intelligence, 42(8))2038-2083 2050, 2019.
13

---

[56] Frederick Tung and Greg Mori. Deep neural network compression by in-parallel pruning- quantization. IEEE transactions on pattern analysis and machine intelligence, 42(3):568- 2018.
[57] S. Uhlich, L. Mauch, F. Cardinaux, K. Yoshiyama, J.A. Garcia, S. Tiedemann, T. Kemp, and
A. Nakamura. Mixed precision DNNs: All you need is a good parametrization. The International Conference Learning Representations (ICLR), 2020.
[58] Vincent Vanhoucke. Learning visual representations at scale. ICLR invited talk, 1:2, 2014.
[59] Aparna Vyas, Soohwan Yu, and Joonki Paik. Multiscale transforms with application image processing. Springer, 2018.
[60] Min Wang, Baoyuan Liu, and Hassan Foroosh. Factorized convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 545-553, 2017.
[61] Ying Wang, Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference on Computer Vision, pages 259-277. Springer, 2020.
[62] Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In Interna- lional Conference on Learning Representations, 2018.
[63] Moritz Wolter, Shaohui Lin, and Angela Yao. Neural network compression via learnable wavelet transforms. International Conference on Artificial Neural Networks, pages 31-51. Springer, 2020.
[64] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[65] Xiaowei Xu, Qing Lu, Lin Yang, Sharon Hu, Danny Chen, Yu Hu, and Yiyu Shi. Quantization of fully convolutional networks for accurate biomedical image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1000-8308, 2018.
[66] Kohei Yamamoto. Learnable companding quantization for accurate low-bit neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5029-5038, 2021.
[67] Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami. Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael Mahoney, et al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pages 11875-11886. PMLR, 2021.
[68] Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-through estimator in training activation quantized neural nets. International Conference Learning Representations, (ICLR), 2019.
[69] Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantiza- tion for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 365-382, 2018.
[70] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa- net: Training low bitwidth convolutional neural networks with low bitwidth gradients. The International Conference on Machine Learning (ICML), 2018
[71] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In section 4, the last paragraph is
dedicated to the limitations of our proposed method.
(c) Did you discuss any potential negative societal impacts your work? [No] Our work aims at reducing the computational resources needed for using CNNs, we suggest no new use-cases for such networks. While the positive impact quite clear (less resources needed leads to less carbon dioxide emission) we could not find a negative socictal impact to report.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We reference publicly available data in (section 5) include pseudo-code in Appenddx B) and include python code for training and testing in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Each experiment in section 5 includes all the training details.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [No] We performed a large set of experiments over different tasks and datasets, each of these runs require extensive computational resources. Our results show a consistent trend of improvement among the different tasks and the various configurations within those tasks.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] section 3) includes that information.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes) section Soonntinss the relevant references.
(b) Did you mention the license of the assets? [No] We include only popular and publicly available data and cite the sources.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include our code and several saved model states.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] We include only popular and publicly available data and cite the sources.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] We include only popular and publicly available data and cite the sources.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "mCVbegDf8g", "reviewer": "Reviewer_QigK", "summary": "This paper newly suggests feature map compression methods for Image-to-Image CNNs, while most of the compression methods focus on classification models. The authors utilize wavelet transform to adaptively extract spatially meaningful pixels and perform pointwise convolution(=conv1x1) on the sparse feature map. Combining with light 8-bit quantization, it shows outstanding results compared to previous methods on various image-to-image vision tasks.", "questions": "- Although pointwise convolution is performed in the sparse feature map, because selected pixels are not contiguous, the computational boost might not be noticeable. Also, it has to zero-fill the output after pointwise conv1x1 to perform inverse wavelet transform which requires memory for the full feature map anyway in the end. How was the inference time compared to the quantization-only method?\n\n- ~~If only BOP matters (and memory doesn't matter for the evaluation), have you tried residual filling (or the same filling = just reuse non-top k entries for inverse transform) rather than zero-filling?~~\n\n- Depending on the bias of conv1x1 kernel, the output from zero input can be far from zero. In this case, zero-filing can be harmful. Though in most cases, pixelwise operation use bias=False option, some might use bias for conv1x1 operations. How can your algorithm deal with such case?\n\n- Notation for the concatenation in Eq.(6) is confusing, since the dimension of $y_1^2$ and $y_2^1$ are different ($C\\times(H/4)\\times(W/4)$ and $C\\times(H/2)\\times(W/2)$). Since 'concatenate' means channel-wise concatenation in many literatures. \nspecifying the orientation of concatenation and the dimension of concatenated vector $y$ would be helpful. \n\n- How does the distribution of feature norm look like before the joint shrinkage? Depending on the number of features with near zero norm, this quantization technique could be either very harmful or effective.", "limitations": "- Dense prediction models that do not use conv1x1 operation can not adopt this quantization technique.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- Problem setting: This paper clearly states the limitation of previous methods (mainly focused on the classification model and not effective enough for image-to-image CNNs) and suggests a proper algorithm to solve the given problem.\n- Robustness: Since this method is simple yet powerful, it can be easily adopted to most types of Image-to-Image CNN architectures. \n- Novelty: Utilizing the equivalence of conv1x1 operation, selecting top-k pixels on spatially-concatenated feature map in wavelet domain is impressive.\n\nWeakness\n- Lack of comparison with previous methods: Since it is more of a pixel-wise compression method, rather than a quantization method. It would be better to add a comparison with other efficient (or light) models in Image-to-Image domains. For example, PointRend is a segmentation model that also adaptively selects the point features using a tiny prediction model for efficient spatially-adaptive operation. Also, a comparison with other quantization methods seems missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gDsM_NpWUvd", "reviewer": "Reviewer_m652", "summary": "This paper proposes to incorporate wavelet decomposition and compression for quantization in image-to-image tasks.", "questions": "See the question in Weakness", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. It proposes Wavelet Compressed Convolution (WCC) for high-resolution activation map compression, which is interesting.\n2. The proposed method dramatically improves the results over aggressive quantization for the same compression rates while retaining the baseline network architecture.\n\nWeaknesses\n1. The appendix should be put in the supplementary files instead of the main paper\n2. What are the top-k/discarded components of wavelet compositions? Are the discarded ones the high-frequency components?\n3. If the discarded ones are high-frequency components, the use of wavelet decomposition and choice of top-k entries will lead to detail loss. It may be effective for image-to-image tasks, such as depth estimation and segmentation, where the detailed textures are not important (but the sharp edges are important). What if for the super-resolution task with high-resolution inputs?\n4. The proposed method can only be applied in 1x1 convolutions, largely limiting its application.\n5. For each feature, the proposed method performs the wavelet composition, top-k choose, and gather operation, which will interrupt the calculation flow of GPU. I doubt such a process will slow down the actual inference time, as the original GPU calculation flow is much smoother. The authors are supposed to provide the actual running time.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fr0LxbAMQFJ", "reviewer": "Reviewer_385X", "summary": "Wavelet Compressed Convolution (WCC) is introduced which addresses the problem of compressing CNNs to reduce memory and runtime requirements. WCC is motivated by the fact that \"aggressive\" quantization (fp32 -> int4 or smaller) often works well for classification problems but not for dense prediction tasks like image-to-image translation models. WCC solves this problem by applying a wavelet transform to the activation maps within a model. This has the effect of increasing sparsity which is utilized by mapping the large, sparse tensor to a compact representation holding the top-k largest values along with their location. A 1x1 conv is then applied to this smaller representation (the authors show that the wavelet transform and 1x1 conv commute), which is much cheaper. The wavelet transform is then inverted to recover the (approximate) activations.\n\nWCC is empirically evaluated on several tasks (object detection, semantic segmentation, and monocular depth estimation) where the experiments show that WCC outperforms naive quantization at equivalent bit depths (e.g. 2-bit quantization = 8-bit quantization + WCC that compresses to 25% of the original size).", "questions": "As mentioned in the strengths and weakness section:\n\nWhy are the results for \"our baseline + WCC\" with \"wavelet shrinkage = None\" worse than the published results? I think this is ok since any improvement in the cases with WCC and lower quantization levels is even more impressive, but a close baseline strengthens the results as well as confidence in the implemented model.\n", "limitations": "Yes. The authors explain that WCC does not work well for small feature maps at the end of Section 4.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors make an important observation that quantization methods that work for classification often don't work for dense prediction tasks. The proposed method (wavelet transformation and top-k selection followed by 1x1 conv) is conceptually simple, well-motivated, and appears to work well in practice when more extreme quantization levels are needed (less than 4-bits per value).\n\nWeaknesses:\n1. WCC often doesn't work as well as naive quantization at moderate quantization levels, though this may be due to baseline models that don't reach the performance of published results (e.g. \"4bit / 8 bit None\" in Tables 1 and 2).\n\n2. The work immediately raises the question of why the transform (Haar was used in the paper) isn't learned. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TN-urr9d_pR", "reviewer": "Reviewer_Jstv", "summary": "The paper studies the network quantization and proposes a wavelet compressed convolution for the high-resolution activation maps. Experiments show that the proposed approach achieves compression rates equivalent to 1-4bit activation quantization with relatively small.", "questions": "see weaknesses", "limitations": "I do not see too many limitations. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper introduces a wavelet compressed convolution to reduce the required computational cost and memory usage for CNNs under high-resolution-based feature maps.\n2. The paper tries to solve a practical problem. The memory usage and the computational cost are quite high when the activation map is under high resolution. \n3. The idea is interesting. The authors introduce a Haar-wavelet transform into the convolution process to reduce the computational cost, instead of directly quantizing the feature map. This direction is interesting to explore further.\n\n**Weaknesses**\n\n1. The experimental results are not sufficient to support the author's claim. The authors claim their method can reduce memory usage. However, the reviewer does not see such a report on memory usage after compression.\n2. It is better to further polish the introduction section. In the current version, it is hard to identify the technical contribution of this paper.\n3. It is better to choose another baseline method or make the current baseline method stronger for comparison as the reproduction of the baseline method in this paper is much worse than the reported results in [43]. For example, in Table 1, [43] has 35.34 mAP around 280 BOPs. However, the author's reproduction is only 31.44. If the baseline method is worse, it is easier to achieve a smaller performance drop after compression.\n4. Can the Haar-wavelet transform be used to the CNN without quantization. It seems that this method can also be used for full-precision CNNs (correct me if I am wrong). The reviewer does not see the reason why it is necessary to further quantize the model after the Haar-wavelet transform. Why not directly compare the BOPs after the Haar-wavelet transform.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept