Wavelet Feature Maps Compression for Low Bandwidth Convolutional Neural Networks
Authors: Yair Zohav
Shahaf E Finder
Maor Ashkenazi
Eran Treister
Venue: ICLR 2022 Submitted (2022)
Keywords: Convolutional Neural Networks
Quantization
Wavelet Transform

Abstract: Quantization is one of the most effective techniques for compressing Convolutional Neural Networks (CNNs), which are known for requiring extensive computational resources. However, aggressive quantization may cause severe degradation in the prediction accuracy of such networks, especially in image-to-image tasks such as semantic segmentation and depth prediction. In this paper, we propose Wavelet Compressed Convolution (WCC)---a novel approach for activation maps compression for $1\times1$ convolutions (the workhorse of modern CNNs). WCC achieves compression ratios and computational savings that are equivalent to low bit quantization rates at a relatively minimal loss of accuracy. To this end, we use a hardware-friendly Haar-wavelet transform, known for its effectiveness in image compression, and define the convolution on the compressed activation map. WCC can be utilized with any $1\times1$ convolution in an existing network architecture. By combining WCC with light quantization, we show that we achieve compression rates equal to 2-bit and 1-bit with minimal degradation in image-to-image tasks.

Under review as a conference paper at ICLR 2022
WAVELET FEATURE MAPS COMPRESSION FOR Low BANDWIDTH CONVOLUTIONAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Quantization is one of the most effective techniques for compressing Convolutional Neural Networks (CNNs), which are known for requiring extensive computational resources. However, aggressive quantization may cause severe degradation in the prediction accuracy of such networks, especially in image-to-image tasks such as semantic segmentation and depth prediction. In this paper, we propose Wavelet Compressed Convolution (WCCya novel approach for activation maps compression for achieves compression ratios and computational savings that are equivalent to low bit quantization rates relatively minimal loss of accuracy. To this end, we hardware-friend friendly wavelet transform, known for its effectiveness in image compression, and define the convolution on the compressed activation map. WCC can be utilized with any 1 x convolution in an existing network architecture. By combining WCC with light quantization. we show that we achieve compression rates equal to 2-bit and 1-bit with minimal degradation in image-to-image tasks.
convolutions (the workhorse of modern CNNs). WCC
INTRODUCTION
Over the past years, Convolutional Neural Networks (CNNs) have brought significant improvement in processing images, video, and audio (LeCun al., 2015; Krizhevsky et al., 2017). However, CNNs require significant computational and memory costs, which makes the usage of CNNs difficult in applications where computing power is limited, e.g., on edge devices. To address this limitation, several approaches have been proposed to reduce the computational costs of neural networks. Among the popular ones are weight pruning (Han et al., 2015; Guo et al., 2016), architecture search (Howard et al., 2019), and quantization (Li et al., 2017; Banner et al., 2a). In principle, all these approaches can be applied simultancously on top of each other to reduce the computational costs of CNNs.
In this work, we focus on quantization. This approach relieves the computational cost of CNNs by quantizing their weights and activation (feature) maps using low numerical precision so that they can be stored and applied as fixed point integers (Hubara al., 2017: Banner et al., 2018a). In particular, it is common to apply aggressive quantization (less than 4-bit precision) to compress the activation maps (Esser al., 2020). Yet, it is known that compressing natural images using uniform quantization is sub-optimal. Indeed, applying aggressive quantization in certain CNNs can lead to significant degradation in the accuracy of the network. The impact is especially evident for the image-to-image tasks such as semantic segmentation (Tang et al., 2019) and depth prediction (Lee et al., 2020), where each pixel has be assigned value. In recent work that targets quantized U-Nets (Tang et al., 2019), activations bit rates are kept relatively high (about 8 bits) while the weight bit rates are lower (down to bits). Beyond that, we note that the majority of the quantization works are applied and tested on image classification (Esser et al., 2020; Li et al., 2020, and references therein).
This work aims to improve the compression of the activation maps by introducing Wavelet Com- pressed Convolution (WCC) layers. These layers utilize the Have-wavelet transform (Daubechies, 1992) to compress the activation maps before applying convolutions. This wavelet transform and its inverse can be applied efficiently in linear complexity for each channel, using additions and subtractions, thanks to the simplicity of the Haar basis. The main idea of our approach is to keep the same top k entries (in magnitude) of the transformed activations maps with respect all channels (dubbed as joint shrinkage) and perform the convolution in the wavelet domain on the compressed signals, saving significant computations. We show that the transform and shrinkage operations

---

Under review as a conference paper at ICLR 2022
Standard quantization
10
Wavelet compression
MSE
10
10-3
3
7
8
Effective hit rate
Figure/ I: Comparison between standard quantization and our (joint-channel) wavelet compression. The plot shows MSE between the quantized and original activation maps based on a batch of 1000 activation maps from the second hidden layer of MobileNetV3 (small) using the ImageNet data set.
commute with the 1 convolution, the heart of modern CNNs. This procedure is applied along with modest quantization to reduce the computational costs further.
We demonstrate the effectiveness and flexibility of WCC by applying it in popular network architectures: DeeplabV3plut (Chen et al., 2018) with MobileNetV2 encoder (Sandler et al., 2018) for semantic segmentation, and BTS (Lee et al., 2020) with ResNeXt50 encoder (Xie et al., 2017) for depth prediction. We show that using WCC dramatically improves the results over aggressive quantization for the same compression rates while retaining the baseline network architecture.
RELATED WORK
Quantized neural networks have been quite popular recently and are exhibiting impressive progress in the goal for true network compression and efficient CNN deployment. Quantization methods include Zhou et al. (2018); Zhang et al. (2018); Banner al. (2018b), and in particular Li et al. (2020); Esser et al. (2020); Choi et al. (2019), which show that the clipping parameters- highly important parameter quantization schemes can be learned through optimization. Beyond that, there are more sophisticated methods to improve the mentioned schemes. For example, dynamic quantization schemes utilize different bit allocations at every layer (Dong et al., 2020; Cai & Vasconcelos, 2020; Uhlich et al., 2020). Non-uniform methods can improve the quantization accuracy (Yamamoto, 2021) but require a look-up table, which reduces hardware efficiency. Quantization methods can also be enhanced by combination with pruning (Tung & Mori, 2018) and knowledge distillation for better training (Kim et al., 2019).
Quantized Neural Networks:
The works above focus image classification. When considering image-to-ing tomiiioiiigeeee tasks like semantic segmentation, networks tend to be more sensitive to quantization of the activations. In the work of AskariHemmat al. (2019), targeting segmentation of medical images, the lowest bit rate for the activations bits, and significant degradation in the performance is evident compared to bits. These results are consistent with the work of Tang et al. (2019) that was mentioned earlier, which uses higher bit rate for the activations than for the weights. The work of Xu et al. (2018) uses weight (only) quantization for medical image segmentation as attempt remove noise and not for computational efficiency. The recent work of Liu et al. (2021) shows both sophisticated post-training quantization scheme and includes fine-tuned semantic segmentation results. Again significant degradation is observed when going from 6 to 4 bits. One exception is the work of Heinrich al. (2018) that uses ternary networks (equivalent to 2 bits here) and segments one medical data set relatively well compared to its full precision baseline.
In this work, we focus on the simplest possible quantization scheme: uniform (across all weights and activations), quantization-awall training, with per-layer clipping parameters. That is to ensure hardware compatibility and efficient use of available training data. In principle, one can run our platform regardless of the quantization type and scenario (c.g., non-uniform/dyname quantization). Also, since we target the compression of the activations, then any method that is focused on the weights (e.g. pruning (Tung & Mori, 2018)) can be combined here as well.
2

---

Under review as a conference paper at ICLR 2022
Wavelet transforms and compression: Wavelet transforms are widely used in image processing (Porwik & Lisowska, 2004). For example, the JPEG2000 format (Rabbani. 2002) uses the wavelet domain to represent images highly sparse feature maps. Recently, wavelet transforms have been used to define convolutions and architectures in CNNs for various of imaging tasks. For example, the work of Huang et al. (2017) presents network architecture for super-resolution, where the wavelet coefficients are predicted and used to reconstruct the high-resolution image. The works of Duan et al. (2017); Williams & Li (2018) use wavelet transforms in place of pooling operators to improve CNNs performance. The former uses dual-tree complex wavelet transform (Kingsbury, 1998), and the latter learns the wavelet basis as part of the network optimization. In these cases, the wavelet transform is not used for compression, but to better preserve information with its low-pass filters.
The work of Wolter et al. (2020) proposes a wavelet-based approach to learn basis functions for the wavelet transform to compresses the weights of linear layers, as opposed to compression of the activation as we apply here. We use the Haar transform (as opposed to learned one) for its hardware efficiency. Using a different transform (known or learned) is also possible, at the corresponding computational cost. The work of Sun et al. (2021) introduces quantization in the wavelet domain, somewhat similarly to this work. However, the authors suggest to improve the quantization scheme by learning a different clipping parameter per wavelet component, but without the feature shrinkage stage, which is the heart of our approach (we use the same clipping parameter for the whole layer using 8 bits for hardware efficiency). As mentioned before, one can use our WCC together with different types of quantization schemes, in particular including Sun et al. (2021), taking the additional hardware complexity of using different clipping parameter per component into account.
Lastly, Liu et al. (2018) suggest using modified U-Net architecture for image-to-image translation. There, wavelet transforms are used for down-sampling, and the inverse is used for up-sampling. This work is architecture- -etsitecctuu--sciiiice.ee and the method can not be easily integrated into other architectures. In contrast, here we propose the WCC layer to easily replace 1 x convolutions regardless of the network architecture. Hence our framework is, in principle, also suitable for post-training quantization (Banner et al., 2019), where the data is not available, and the original network is not retrained.
BACKGROUND
Quantization-aware training Quantization schemes can be divided into post- and quantization- aware training schemes. Post training schemes perform model training and model quantization separately, which is most suitable when the training data is not available during the quantization phase (Banner al., 2019; Nagel et al., 2019; Cai et al., 2020). On the other hand, quantization-awaan training schemes are used to adapt the model's weights as an additional training phase. Such schemes do require training data but generally provide better performance. Quantization schemes can also be divided into uniform vs. non-uniform methods, where the latter is more accurate but the former more hardware friendly (Li al., 2020; Jung et al., 2019). Lastly, quantization schemes can utilize quantization parameters per channel within each layer or utilize these parameters only per layer (where all the channels share the same parameters). Similar to before, per-channel methods are difficult to exploit in hardware, while per-layer methods are less accurate but are more feasible for deployment on edge devices. This paper focuses on per-layer and uniform quantization-awaar training for both weights and activations and aims to improve this with wavelet transforms. Other quantization schemes can be applicd within our wavelet compression framework as well.
In quantization-aware training, we set the values of the weights to belong to a small set so that after training, the application of the network can be carried out in integer arithmetic, i.e., activation maps are quantized well. Even though that we use discontinuous rounding functions throughout the network, quantization-awee aware training schemes utilize gradient- based methods to optimize the network's weights (Han et al., 2015; Yin al., 2019). In nutshell, when training, we iterate on the floating-pon point values of the weights. During the forward pass, both the weights and activation maps are quantized, while during the backward pass, the Straight Through Estimator (STE) approach is used (Bengio et al., 2013), where we ignore the rounding function, whose exact derivative is zero
The specific quantization scheme that we use is based on Li et al. (2020). First, the pointwise quantization operator is defined by:
T6 (() round -1)-0)
2-
3

---

Under review as a conference paper at ICLR 2022
where E [-1, or [0, [] for signed unsigned quantization, respectively'. The parameter b is the number of bits that are used to represent t. During the forward pass, each number undergoes three steps: scale, clip, and round. That is, to get the quantized version of any number, we apply:
If signed: 26 = 106-- _h - 11,11,) 1)); If unsigned: 2s = Qx(x) (cg((clip(_ 0,))-
(2)
Here, 2b are the real-valued and quantized tensors, and a is the clipping parameter. The parameters in (2) play an important role in the error generated at each quantization and should be chosen carefully. The works of Li et al. (2020); Esser et al. (2020) introduced an effective madient--bssed optimization to find the clipping values a for each layer, again using the STE approximation. This enables the quantized network to be trained in an end-to-end manner with backpropagation. To further improve the optimization, weight normalization also used before each quantization.
Haar wavelet transform and compression. In this section, we describe the Haar- wavelet transform in deep learning language and its usage for compression. See Vyas et al. (2018) for more details on the use of wavelets for image compression. Given image channel the one-level Haar transform can be achieved by separable 2D stride two convolution with the following kernel:
-
1 -1: 1 w =
- - +
These kernels can also be expressed as composition of separable ID kernels [1, 121l and [1, [1.-1//v2. The result of the convolution y2, Y2,y3, y4] Conv (W, x) has four channels, each of which has half the resolution of x. The leftmost kernel in (3) is an averaging kernel, and the three right kemnels are edge- detectors. This, together with the fact that images are piece-wise smooth, leads to relatively sparse images y2. Ya. Y4. Hence, if we retain only the few cop-magnitude entries in these vectors, keep most of the information, most of the entries we drop zeros. That is the main idea of wavelet compression. We denote the wavelet transform by y HWT(x), where defined the concatenation of the vectors y1. .... Y4 into one. Since the kernels in (3) form an orthonormal basis, applying the inverse transform is obtained by the transposed convolution of (3);
x = iy)T(y)) = transpooa W,
N3.
However, unlike y2, Y3. Y4, the averaged image y1 is not sparse, and is just the down-sampled original x. Hence, in the multilevel wavelet transform, we apply the kernel in (3) on yı to generate further down-sampled sparse channels. For example, a 2- Haar transform can be summarized as
y2. y
Conv x))
12
Conv(W.y) yt),
(4)
and the resulting transformed image with 2-1evels can be written as the concatenated vector
y = [y v2. y. y2 y!l = HWT(x).
(5)
In this work we use 3 levels in all the experiments. To apply compression we define the operator T that extracts the top & entries in magnitude from the vector y: compressed = T HWT(x). To de-compress this vector we first zero-fill y compressed (multiply with T) and apply the inverse Haar transform, which is the transposed convolutions in the opposite order.
4 WAVELET COMPRESSED CONVOLUTION
In this work, we aim to reduce the memory bandwidth and computational cost associated with convolutions performed on intermediate activation maps. To this end, we apply the Havrlea transform to compress the activation maps, in addition to light quantization of 8 bits. Our method most efficient for scenarios with high-resolution feature maps (i.e., large images, whether in 2D or 3D), where the wavelet compression is most effective. Such cases are mostly encountered in image-to-image tasks like semantic segmentation, depth prediction, image denoising, in in-painting, super-resolution, and more. Typically, in such scenarios, the size and memory bandwidth used for the weights are relatively small compared to those used for the features (e.g., point cloud segmentation).
In most standard CNNs. the ReLU activation is used: hence the activation feature maps are non- negaaveee and can be quantized using an unsigned scheme. If different activation function that is not non-negative is used, or, as in our case, the wavelet coefficients are quantized, signed quantization should be used instead.
4

---

Under review as a conference paper at ICLR 2022
Figure 2: The workflow of WCC. From Left right: the input channels, their Haar transform, the joint shrinkage of the 2D Haar representation into equal sized 1D vectors and single bit-map (or list of indices), the application of the x convolution on the 1D vectors, and lastly on the right: the inverse transform back to the spatial domain. Here, the shrinkage ratio is set to 0.1.
In addition, since we have to predict each pixel, typical architectures do not aggressively down-sample the images, as opposed to multi-class image classification, where activation maps become small, and the number of channels grows towards the end of the network.
Since we focus on computational efficiency, we the Haar transform, as it is the simplest and most computationally efficient wavelet variant. Indeed, the Haar transform involves binary pooling-like operations. which include only additions, subtractions, and bit-wise shift. We focus on the compression of fully-coupled x convolutions, as these are the workhorse of lightweight and efficient architectures like MobileNet (Sandler et al., 2018), ShuffleNet (Ma et al., 2018) 8), EfficientNer (Tan & Le, 2019), and ResNeXt (Xie et al., 2017). All these modern architectures rely on 1 X convolutions in addition to grouped or depthwise spatial convolutions (i.e.. with 3 x 3 larger kernels), which comprise small part of the computational effort in the network- the X 1 operations dominate the inference cost (see Appendix B). The main idea of method is to transform and compress the input using the wavelet transform prior to the 1 x 1 convolution, then apply it in the wavelet domain on a fraction of the input size. Since the wavelet compression is applied separately on each channel, it commutes with the 1 1 convolution. Hence, applying the convolution in the wavelet domain equivalent to applying it in the spatial domain.
Convolution in the wavelet domain:
Prior to the x convolution, the spatial wavelet transform is applied, and we get sparse feature maps. Since the different channels result from the same input image propagated through the network, they typically include patterns at similar locations, and hence the sparsity pattern of their -ereett-dominn representation is relatively similar. This idea of redundancy in the channel space is exploited in the works of Han (2020); Eliasof et al. (2020); Bae al. (2021), where part of the channels are used to represent the others. Therefore, we perform joint shrinkage operation between all the channels, in which we zero and remove the entries with the smallest feature norms across channels, resulting in compressed representation of the activation maps?. The locations of the non-zero in the original image are kept in a single index list or a bit-map for all the channels in the layer, as they needed for the inverse transform back to the spatial domain. Lastly, also apply light 8-bit quantization to the transformed images to further improve the compression rate. The weights and wavelet-domale activations are quantized using the symmetric scheme described in section 3.
Joint hard shrinkage:
More precisely, the advantage of the wavelet transforms is their ability compress images. Denote the Haar transform matrix as H, i.e., Hx = HWT(x). Then, for most natural images we have that
x x THTTTTS
(6)
where T is the shrinkage operator described above, and because of its orthogonality, H is the inverse transform IHWT. Our WCC layer is defined by:
WCC(KI 1xI, x) = H H Kixi THx,
(7)
where 1 is the learned convolution matrix. The workflow is illustrated in Figure 2. Note that the convolution operates on the compressed domain, hence, if T can significantly reduce the dimensions of the channels, this leads major savings in computations.
Regardless of the joint sparsity of the channels, some approaches suggest taking the left-upmost part of the wavelet transform for any image, in principle, the joint sparsity may suffice for general set of images as well.
5

---

Under review as a conference paper at ICLR 2022
Our method aims at compression only. Hence, show that x convolution kernel can be applied both in the spatial and in the compressed wavelet domain. By its definition, we can write a 1 x 1 convolution as summation over channels. That is:
+ E, Kiyxy,
(8)
where kig IR are the weights of the convolution tensor. Now, suppose we wish to compress the result through (6). Because T and H are separable and spatial, we get by simple linear arithmetie:
~ H T (H(K
= H Σ, 2, Kiy THX; y H H 1 H THx.
(9)
Hence, the wavelet compression operator, which is known to be highly efficient for natural images, commutes with the 1 x convolution, so the latter can be applied on the compressed signal and get the same result as compressing the result directly. This introduces an opportunity save computations on the one hand and use more accurate compression on the other.
The description above is suited for spatial convolutions with larger kernels, strides, etc. The main concept is that fully coupled convo- lutions that mix all channels are expensive, inefficient, and redundant when used with large spatial kernels (Ephrath et al., 2020). The spatial mixing can be obtained using separable convolutions at less cost without losing efficiency (Chen et al., 2018). Since we aim at saving computations, separating the kernels in the architecture is recommended even before discussing any type of lossy compression. Furthermore, separable convolutions (and the Haar transform) can be applied separately in chunks of channels or together as part of the 1 x 1 convolution using specialized implementation. Hence the memory bandwidth can be reduced. The 1 x convolution, on the other hand, involves all channels at once; hence it is harder reduce the bandwidth in this case.
convolutions only, while many CNN architectures involve
4.1 COMPUTATIONAL COSTS IN BIT OPERATIONS (BOPS)
To evaluate the computational cost involved in WCC we use the measure of Bit-Operations (BOPs) (Wang et al., 2020; Louizos et al., 2018). First, the number of Autiply-And-AAccumllaee (MAC) operations in a convolutional layer is given by
MAC(conv) = Ci Cou - NNw . Nu Kw KH Sw
(10)
where Cin and Cout are the number of input and output channels, (Nw .H) is the size of the input, (Kw, KH) is the size of the kernel, and (Sw, SH) is the stride value. The BOPs count is then
(OPs()onv) MAC(conv) ba ba: where by and by denote the number of bits used for weight and activations.
(11)
As described in section 3, the Haar transform is separable between the input channels, and can convolutions with stride (2, 2) and binary weights. Hence, the one-level transform requires 4n W. bb, BOPs. The transform can be used with more levels of compression explained in section 3, on down-scaled inputs, resulting in a total of
be viewed as four
Σ 4 Cin Nw NH / bb.
(12)
BOPs, where L is the level of compression. Similarly, the inverse-trand transform result in the same calculation, only with Coun in place of Cin. To demonstrate the relatively small cost of the compression, consider a x 1 convolution with Cin = 160, Cou = 960, input size of 34), and quantization by = be 8 (which is part of a network used later in section 5). This layer costs 11, 364M BOPs. Using a 3 levels wavelet transform and its inverse for this layer results in 54M BOPs, a negligible cost which allows for better compression, as we demonstrate next.
RESULTS
In this section, we evaluate our WCC layer's performance in image-to-image segmentation and depth estimation-when the feature maps are relatively large and suitable for wavelet compression. We compare our method to quantization-awaal et al. (2020), which we also use in our method for 8-bit quantization of the shrunk signals. Our code implemented in PyTorch (Paszke et al., 2017) based on Torchvision implementations, including the same data augmentation. We ran our experiments on NVIDIA 24GB RTX 3090 GPU. We detail our experimental setup for each section separately.
training based on the work of Li

---

Under review as a conference paper at ICLR 2022
Original input image from ImageNet
(b) Feature maps from ResNet50 layers.
Ill
(c) 2-bit quantized layers (unsigned).
(d) 2-bit equivalent wavelet compressed.
Figure 3: Feature maps from layers 2 and 3 (top and bottom triplets, respectively) of pretrained ResNe150 on ImageNet. The maps are compressed with uniform quantization (2-bit) and wavelet compression (25% shrinkage 8- bit quantization, equivalent to 2-bit quantization in terms of size). It clear that wavelet compression loses much less information than aggressive quantization.
5.1 QUALITATIVE COMPRESSION ASSESSMENT
Before describing experiments with common datasets and tasks, we first qualitatively demonstrate the advantage of our approach. First, compare the MSE of standard quantization a feature map from a pretrained MobileNetV3 applied on a batch of 1000 images from the ImageNet dataset (Krizhevsky et al., 2017). We consider effective bit rate range of [2.8]. The quantization of the shrunk wavelet coefficients is 8 bit, and multiplying it with the compression ratio yields the effective bit rate. E.g., 8 bit and 25% shrinkage is effectively 2 bits. This also agrees with the BOPs measure in Eq. (LI). Figure shows the MSE comparison per effective bit rate. One can see that the information loss based on the MSE is more significant in standard quantization. We note that the quantization parameters in this experiment were chosen by ex exhaustive search for minimizing the MSE for every dot the plot. Furthermore, in Figure we can see visualization comparison of a typical activation map compressed by standard quantization and our method. It is obvious that the perceptive degradation in the image approximation more significant the standard quantization. This is not surprising since wavelet image compression is more advanced than uniform quantization.
approach
5.2 SEMANTIC SEGMENTATION
Semantic segmentation is the task of assigning a label to every pixel in the input image. CNNs have shown significant improvements over traditional methods but at a high computational cost. Our model of choice here is the popular Deeplab V3plus (Chen et al., 2018) with MobileNetV2 backbone (Sandler et al., 2018). We evaluated the proposed method on the Cityscapes and Pascal VOC. The Cityscapes dataset (Cordts et al., 2016) contains images of urban street scenes. The images are of size 1024x2048 with lixel-level annotation of 19 classes. During training, we used a random crop of size 768 and no crop for the validation set. The Pascal VOC (Everingham et al., 2015) dataset contains images of size 513x513 with pixel-level annotation of 20 foreground object classes and a background class. We augmented the dataset similarly Chen et al. (2018).
7

---

Under review as a conference paper at ICLR 2022
(a) Input image
(b) Ground truth
(c) Quantization 8/8
(d) Quantization 8/4
(c) WCC 8/8 100%
(f) WCC 8/8 50%
(g) WCC 8/8 25%
(h) WCC 12.5%
Figure 4: Cityscapes segmentation results. All the networks weight quantization of 8-bits. (a) input image. (b) ground truth. (c), (d) normal quantization with 8- and 4-bits activations respectively. (e)-(h) WCC with 8-bits activations and shrinkage rate of 100%, 50%, 25% and 12.5% respectively.
Precision
Wavelet shrinkage
Cityscapes BOPs (B)
Pascal VOC
(Weight/ / Activation)
mloU
BOPs (B)
mloU
Full precision
None
36,377
0.717
4,534
0.715
8bit / 8bit
None
2,273
0.701
283
0.712
8bit / 6bit
None
1,705
0.683
212
0.678
8bit / 4bit
None
1,136
0.173
141
0.095
8bit / 8bit
100%
2,292
0.697
285
0.695
8bit / Sbit
50%
1,213
0.681
150
0.675
8bit / 8bit
25%
673
0,620
82
0,611
8bit / Sbit
12.5%
403
0.552
48
0.519
4bit / Sbit
None
1,136
0.682
141
0.675
4bit / 6bit
None
852
0.669
106
0.657
4bit / 4bit
None
568
0.190
70
0.099
4bit / Sbit
100%
1,156
0.672
144
0.678
4bit / Sbit
50%
616
0.667
76
0.661
4bit / Sbit
25%
346
0.621
42
0.583
4bit / 8bit
12.5%
211
0.549
24
0.515
Table 1: Validation results for semantic segmentation task using V3plus with MobileNetV2 the backbone. Segmentation performance measured by mean intersection over union (mloU),
We used configurations for the weights-44 and bits-and and for each them we used different compression rates for the activations, both for the standard quantization and our WCC layer. Like before, our WCC layer with shrinkage rate of 50% has nearly equal BOPs reducing the activation bits from 8 to 4. Likewise, a shrinkage rate of 25% nearly equals to 2 bits.
The training scheme used similar to Li et al. (2020). We first trained network in full precision and then gradually reduced the bit rates (this was used for the WCC as well). For all datasets and model optimization, we used SGD with momentum 0.9, weight decay 10-1 learning rate decay 0.9, and set the batch size to 16. For Cityscapes, learning rate of 0, and each of the retrains was trained for 50 epochs with a base learning rate of 0.01. For Pascal VOC, we trained the full precision model for 50 epochs with base learning rate of 0.01, and each of the retrains was trained for 25 epochs with a base learning rate of 0.002.
trained the full precision model for 160 epochs with a base
Table shows the performance and BOPs of each model, from which achieves superior performances at high compression configurations, outperforming the 4-bit activation with compression rates equivalent to 2- and 1-bits. Figure 4 shows visual example.
that model
8

---

Under review as a conference paper at ICLR 2022
Precision (Weight Activation)
Wavelet shrinkage
BOPs (B)
AbsRel
RMSE
Full precision
None
156,882
0.061
2.774
8hit 8hit
None
9,805
0.068
2.889
8bit / 8bit
50%
4,910
0.071
3.063
8bit 4bit
None
4,902
0.070
3.065
Sbit 8bit
25%-50%
4,549
0.075
3.228
8bit / 2bit-4bit
None
4,541
0.106
4.082
8bit / bit
25%
4,297
0.08
3.352
Table 2: Validation results BTS, using ResNeXt50 backbone encoder, measured by absolute relative difference (AbsRel) and root mean squared error (RMSE), in both lower is better.
Comparison Liu et (2021): This work deals with zero-shot quantization and shows results on quantization-awar training (fine-tuned) for Cityscapes, albeit using 256 x 256 input images. In their paper, the 8bit/Sbit fine-tuned result has mloU of 0.613 compared to about 0.7 here, both the standard and wavelet cases. Our standard 4bit/8bit setting achieves 0.67 vs. 0.6 for 6bit/6bit by Liu et al. (2021). Furthermore, when considering 4bit weights, all our wavelet compression results (up to 12.5%) outperform or par with the 4bit/4bit result of 0.56 by Liu et al. (2021).
5.3 MONOCULAR DEPTH ESTIMATION
In this section, we apply our compression technique on Big-to-Small (BTS, Lee et al., 2020), network architecture for monocular depth estimation a task of estimating scene depth using single image. We evaluated the results on the KITTI dataset (Geiger et al., 2013), containing images of autonomous driving scenarios, each of size 1124 x376 pixels. The dataset splits are as in BTS, using the strategy of Eigen et al. (2014). As in BTS, we use maximum value of 80 meters for prediction when testing, and the performance evaluation based on the cropping scheme of Garg et al. (2016). We did not modify the initial convolution, the final convolutions, and the depth prediction layer when applying the compression. Using any compression technique on these layers resulted in significant degradation in the results.
In our experiments, we focus on a ResNeXt50 backbone. The compressed models are fine-tuned from 8bit/8bit quantized network, trained from scratch. In local planar guidance layers, use 8bit/4bit quantization instead of our wavelet scheme. When training, we use the same configuration presented in the BTS repository. We the AdamW optimizer (Loshchilov & Hutter, 2019), learning rate of 0,00014, weight decay of 0.001, a batch size of 4 images, and train for 50 epochs.
Table 2 shows a comparison between the standard quantization and WCC for BTS. Using our WCC layer with 50% compression resulted in comparable scores to the alternative method of 8bit/4bit quantization (with similar BOPs). Moreover, when applying a compression factor of 25% (50% on low spatial-sized tensors), we achieve superior results to the quantized alternative. This comparison presented qualitatively in Appendix Finally, in the full 25% compression experiment, we see that even when aggressively compressing the low spatial-sized tensors, we achieve better results than using low- precision standard guantization.
CONCLUSION
In this work, we presented a new approach for feature map compression, aiming to reduce the memory bandwidth and computational cost of image-to-image CNNs, where typically the resolution is high. Our approach is based on the classical Haar-wavelet image compression, which has been used for years in standard devices and simple hardware, and in real-time. We save the computational cost by applying x 1 convolutions on the shrunk wavelet domain. We show that this approach surpasses aggressive quantization using the same bit rate minimal additional costs of the transform.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The code for running and reproducing the experiments reported in this paper is attached as supple- mentary material.
REFERENCES
MosemmadHossein AskariHemmat,, Sina Honari, Lucas Rouhier, Christian S. Perone, Julien Cohen- Adad, Yvon Savaria, and Jean-Pierre David. U-net fixed point quantization for medical image segmentation. In Medical Imaging and Computer Assisted Intervention (MICCAI), Hardware Aware Learning Workshop (HAL-MICCAI), Springer, 2019.
Seung- Seung-Hwan Bae, Hyuk-Jae Lee, and Hyun Kim. De-ac: Deep correlation-base adaptive com- pression of feature map planes in convolutional neural networks. In 2021 IEEE International Symposium Circuits and Systems (ISCAS), pp. 1-5. IEEE, 2021.
R. Banner, Nahshan, and D. Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. The Conference on Neural Information Processing Systems (NeurIPS), pp- 7948-7956, 2019.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. Advances in Neural Information Processing Systems, pp. 5145-5153, 2018a.
Ron Banner, Yury Nahshan, Elad Hoffer, and Danicl Soudry. Aciq: Analytical clipping for integer quantization of neural networks. ArXiv, abs/18 abs/1810.05723, 10.05723, 20188.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint /308. 2013.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Zhaowei Cai and Nuno Vasconcelos. Rethinking differentiable search for mixed-precision neural net- works. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2349-2358, 2020.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818 2018.
Jungwook Choi, Swagath Venkataramani, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan, Zhuo Wang, and Pierce Chuang. Accurate and efficient 2- quantized neural networks. MLSys, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 3213-3223, 2016.
Ingrid Daubechies. Ten lectures wavelets. SIAM, 1992.
Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trightee quantization of neural networks. The Conference on Neural Information Processing Systems (NeurIPS), 2020.
Yiping Duan, Fang Liu, Licheng Jiao, Peng Zhao, and Lu Zhang. SAR image segmentation based on convolutional-waaeer neural network and markov random field. Pattern Recognition, 64:255-267, 2017.
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network, 2014.
10

---

Under review as a conference paper at ICLR 2022
Moshe Eliasof, Jonathan Ephrath, Lars Ruthotto, and Eran Treister. cultigrid-in-channde neural network architectures. arXiv preprint arXiv:2011.09128, 2020.
Jonathan Ephrath, Moshe Eliasof, Lars Ruthotto, Eldad Haber, and Eran Treister. Leanconvnets: Low-cost yet effective convolutional neural networks. IEEE Journal of Selected Topics in Signal Processing, 14(4):894-904 2020.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. The International Conference Learning Representations (ICI.R,, 2020.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1 139,-136, 2015.
Ravi Garg, Vijay Kumar BG, Gustavo Carniir,, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue, 2016.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (I/RR), 2013.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. arXiv preprint arXiv:1608. (4f1::6608.04493,, 2016.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1580-1589, 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv: 000.5006622222 02626, 2015.
1rxivv:5006.220
Mattias P Heinrich. Max Blendowski, and Ozan Oktay. Ternarynet: faster deep model inference without gpus for medical 3d segmentation using sparse and binary convolutions. International journal of computer assisted radiology and surgery, 13(9):1311-1320, 2018.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for MobileNetV3, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314-1321 2019.
Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Wavelet-smet: A wavelet-based cnn for multi- scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1689-1697, 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18:187:1-187:30, 2017.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-awarc knowledge distillation. arXiv preprint wrXiv:1911.12499 2019.
Nick Kingsbury. The dual-tree complex wavelet transform: efficient tool for image restoration and enhancement. In 9th European Signal Processing Conference (EUSIPCO 1998), pp. 1 1-4. IEEE, 1998.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
11

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Jin Han Lee, Myung-Ky -yu Han, Dong Wook Ko, and II Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation, 2020.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. Proceedings of the s International Conference on Neural Information Processing Systems, pp. 5813-5823, 2017.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non-uniform discretization for neural networks. The International Conference on Learning Representations (ICLR), 2020.
Pengju Liu, Hongzhi Zhang. Kai Zhang, Liang Lin, and Wangmeng Zuo. Multi-level wavelet-emn for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 773-782, 2018
Yuang Liu, Wei Zhang. and Jun Wang. Zero-shot adversarial quantization. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1512-1521, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. arXiv preprint arXiv: 2018.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference computer vision (ECCV), pp. 116-131, 2018.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In ICCV, pp. 1325- 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Veto, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems, 2017.
Piotr Porwik and Agnieszka Lisowska. The haar-wavelet transform in digital image processing: its status and achievements. Machine graphics and vision, 13(1/2):79-98, 2004.
Majid Rabbani. Jpeg2000: Image compression fundamentals, standards and practice. Journal of Electronic Imaging, 11(2):286, 2002.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 4520, 2018.
Qigong Sun, Yan Ren, Licheng Jiao, Xiufang Li, Fanhua Shang, and Fang Liu. MWQ: Multiscale wavelet quantized neural networks. arXiv preprint arXiv:2103.05363 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-6114 PMIR, 2019.
Zhiqiang Tang, Xi Peng, Kang Li, and Dimitris Metaxas. Towards efficient u-nets: A coupled and quantized approach. IEEE transactions pattern analysis and machine intelligence, 42(8): 2038-2050, 2019.
Frederick Tung and Greg Mori. Deep neural network compression by in-parallel pruning-quantizatinnn IEEE transactions pattern analysis and machine intelligence, 42(3):568-55 50:-999999999906 2018
S. Uhlich, L. Mauch, F. Cardinaux, K. Yoshiyama, J.A. Garcia, S. Tiedemann, T. Kemp, and
A. Nakamura. Mixed precision DNNs: All you need is a good parametrization. The International Conference Learning Representations (ICLR), 2020.
Vincent Vanhoucke. Learning visual representations at scale. ICLR invited talk, 1:2, 2014.
12

---

Under review as a conference paper at ICLR 2022
Apama Vyas, Soohwan Yu, and Joonki Paik. Multiscale transforms with application to image processing. Springer, 2018.
Min Wang, Baoyuan Liu, and Hassan Foroosh. Factorized convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 545-553,
2017.
Ying Wang, Yadong Lu, and Tijmen Blankevoort. Differentiable joint pruning and quantization for hardware efficiency. In European Conference on Computer Vision, pp. 259-277 Springer, 2020.
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. International Conference Learning Representations, 2018.
Moritz Wolter, Shaohui Lin, and Angela Yao. Neural network compression via learnable wavelet transforms. In International Conference on Artificial Neural Networks, pp. 39-51. Springer, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500,, 2017.
Xiaowei Xu, Qing Lu, Lin Yang, Sharon Hu, Danny Chen, Yu Hu, and Yiyu Shi. Quantization of fully convolutional networks for accurate biomedical image segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8300-8300 2018.
Kohei Yamamoto. Learnable companding quantization for accurate low-bit neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
5029-5038, 2021.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-through estimator in training activation quantized neural nets. International Conference on Learning Representations, (ICLR), 2019.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks, In Proceedings of the European conference computer vision (ECCV), pp. 365-382, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. The International Conference on Machine Learning (ICML), 2018.
13

---

Under review as a conference paper at ICLR 2022
A BTS COMPRESSION RESULTS
(a) Input Image
(b) Wavelet Compression 8/8 25% (50% in small tensors)
(c) Quantization 8/2 (8/4 in small tensors)
Figure 5: Kitti depth estimation prediction examples. We compare compressed networks with the same BOP magnitude, the first uses our wavelet compression method and the second uses low-bit quantization.
B NOTE ON X 1-CONVOLUTIONS
In the case when a certain CNN use 3 convolution only, one can split to two convolution, depthwise-3 x 3 and a x 1 (Wang et al., 2017; Vanhoucke, 2014). The depthwise conv involves with Cin 3 3 Nw Nи MAC operations (assuming no strides), while the 1 x conv includes Cin Cout Nw NH MAC operations (Cout/9 /9 times more expensive than depthwise). Meaning, for a large enough Couts the x 3 convolution has about 8-9 times more MAC operations than the depthwise-3 x 3 convolution and 1 1 convolution.
Some models, such as the ones referenced in section 4, are defined based on that concept. For example, MobilenetV2 consists of residual blocks that perform 1 x 1, depthwise-3 x 3, and an additional 1 x 1, and for an image input of size 1024 x 2048 (e.g. cityscapes), the x 1-conv has MAC count of 18,022M, while the 3 3-conv has a MAC count of 1,056M (a full breakdown is provided in Table 3).
14

---

Under review as a conference paper at ICLR 2022
Module id
Cin
Cout
K
Groups
S
Dilation
H
W
MAC
InvResl convl
32
32
3
32
513
1025
150,552,864
InvResl conv2
32
511
1023
267,649,536
InvRes2 convl
16
96
1
1
513
1025
807.667,200
InvRes2 conv2
96
96
96
513
1025
112,914,648
InvRes2 conv3
96
24
1
256
512
InvRes3 convl
24
144
1
258
514
458,307,072
InvRes3 conv2
144
144
144
258
514
169,869,318
InvRes conv3
144
24
256
512
452,984,832
InvRes4 convl
24
144
258
514
458,307,072
InvRes4 conv2
144
144
3
144
256
514
42,467,328
InvRes4 conv3
144
32
128
256
150,994,944
InvRes5 convl
32
192
258
514
206,069,766
InvRes5 conv2
192
192
256
514
56,623, 104
InvRes5 conv3
192
32
I
128
256
201,326,592
InvResó convl
32
192
258
514
206,069,700
InvRes6 conv2
192
192
192
256
514
56,623,104
InvRes6 conv3
192
32
128
256
201,326,592
InvRes7 convl
32
192
258
514
206,069,760
InvRes7 conv2
192
192
192
256
514
14,155,776
InvRes7 conv3
192
64
64
128
100,663,296
InvRes8 convi I
64
384
66
130
210,862,080
InvRes8 conv2
384
384
384
66
130
28,311,552
InvRes8 conv3
384
64
64
128
201,326,592
InvRes9 convl
64
384
66
130
210,862,080
InvRes9 conv2
384
384
3
384
66
130
28,311,552
InvRes9 conv3
384
128
201,326,592
InvRes10 conv
64
384
66
130
210,862,080
InvRes10 conv2
384
384
3
384
66
130
28,311,552
InvRes10 conv
384
64
64
128
201,326,592
InvResl cony
64
384
66
130
210,862,080
InvRes11 conv2
384
384
384
66
130
28,311,552
InvRes conv3
384
96
1
64
128
301,989,889
InvRes convl
96
576
66
130
474,439, 680
InvRes12 conv2
576
576
3
576
66
130
42,467,328
InvRes12 conv3
576
96
64
128
452,984,832
InvRes13 conv
96
576
1
66
130
474,439,680
InvRes13 conv2
576
576
576
66
130
42,467,328
InvRes13 conv3
576
96
64
128
452,984,832
InvRes14 conv1
96
576
1
66
130
474,439,681
InvRes14 conv2
576
576
3
576
66
130
42,467,328
InvRes14 conv3
576
160
1
64
128
754,974,720
InvRes15 conv1
160
960
66
130
1,378,713,
InvRes15 conv2
960
960
3
960
66
130
70,778,888
1nvRes15 conv3
960
160
64
128
1,258,291,200
In conv
160
960
66
130
1,378,713,600
InvRes16 conv2
960
960
960
66
130
70,778,
InvRes16 conv3
960
160
1
1
64
128
1,258,291,22 2,8,,99,200
InvRes17 conv1
160
960
66
130
1,378,713,600
InvRes17 conv2
960
960
3
960
2
66
130
70,778,880
1 conv3
960
320
1
64
128
2,516,582,400
Total x 1
18,022,413,312
Total of x
1,056,190,968
Table 3: In depth breakdown of MobilenetV2 (as a backbone for deeplabv3+) for single Cityscapes' image input. K and refer the size of the symmetric kemnels and strides respectively. The first convolution of the network is omitted, since it is common practice to avoid quantizing it.
15


Reviews:
[{"review_id": "I-MsYhafPn9", "reviewer": "Reviewer_QwGk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n- The paper presents a neat and clever way to apply traditional wavelet-image compression in CNN.\n- The qualitative compression assessment appears to be visually convincing and supported by past work on image compression.\n- In the method performs decently on a segmentation network on cityscapes as well as a depth estimation network on kitty.\n- Supplementary source code: Decompressing the zip file worked. The source code is not documented but readable.\n- Working with Haar-wavelets is usually a good start and not unheard of in the machine learning literature:\n   - Rinon Gal, Dana Cohen, Amit Bermano, and Daniel Cohen-Or. Swagan: A style-based wavelet-driven\n     generative model. arXiv preprint arXiv:2102.06108, 2021.\n   - Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International\n   Conference on Learning Representations, 2018\n\n   Do so as well for image generation and pooling tasks. Having established the utility of Haar wavelets, more complex wavelets could be explored in future work.\n\n\n## Open Questions:\n### Related work:\n- page 3: Williams and Li (2018) write, \" For our proposed method, the wavelet basis is the Haar wavelet, mainly for\nits even, square subbands. \" the pooling filters are, presumably, static in this case?\n\n### Background:\n- Filter construction:\nIt appears the two-dimensional filters in eq. (3) are constructed from 1d-pairs using outer products:\n$$ f_{ll} = f_0 \\cdot f_0^T , f_{hl} =  f_1 \\cdot f_0^T , f_{hl} =  f_0 \\cdot f_1^T , f_{hh} =  f_1 \\cdot f_1^T ,$$\nas described in i. e.\n    - [Vyas et al., 2018] Vyas, A., Yu, S., and Paik, J. (2018). Multiscale transforms with application to image processing. Springer.\n    \n    The process is currently not obvious. Would authors consider adding a reference or explanation,\n    to help readers understand the genesis of the 2d-Haar filters in equation 3?\n\n- Convolution definition:\nWhy not be more specific just above equation three and speak of a stride two convolution? Doing so would explain why the resolution is \nhalved for the resulting coefficients $\\mathbf{y}_1 , \\mathbf{y}_2, \\mathbf{y}_3, \\mathbf{y}_4 $.\n\n### Wavelet Compressed Convolution\n- What are the matrix dimensions in equations six and seven? Are the wavelet and the convolution matrices square or $\\mathbb{R}^{n,n}$ in this case?\nWhat about $\\mathbf{T}$ and what would the impact of a nonsquare $\\mathbf{T}$ on $\\mathbf{K}_{1x1}$ be?\nAssuming most of the computational work happens there. \nWriting this down could help make this section's main points and equation nine easier to understand.\n\n- How are the (tensor ?) dimensions of $\\mathbf{x}$ defined? https://www.deeplearningbook.org/contents/notation.html suggests lowercase boldface\nshould be a vector. A vector makes sense because fast wavelet matrices can be computed by matrix multiplication with an image channel\nflattened into a vector. But it seems in equation eight $\\sum_j k_{ij} x_j$ sums along a channel dimension?\nIf that is the case, does $\\mathbf{H}\\mathbf{x}$ transform the channel dimension in equation seven? \n\n### Computational cost\n- How do the input and channel sizes from equations ten, eleven, and twelve relate to the matrices and vectors (?) in equations six to nine? \n\n### Experiments\n- For all experiments: of all convolution operations how many are 1x1?\n- Segmentation:\n    - Does (W / A) mean, weights / activations in Table 1? Perhaps this is something for the caption.\n    - The authors may want to consider adding the results of Liu et al. (2021) to table 1.\n\n- Depth estimation:\n    - What does BTS stand for, what does it mean? \n\n### Minor remarks:\n- page 3: ..., or for utilize ... -> ..., or utilize ...\n- page 4: ..., to be trained end to end manner, ... -> .., to be trained in an end to end manner ...\n- page 5: wights -> weights\n- page 9: Conclusions -> Conclusion", "summary_of_the_paper": "The paper proposes to use sparsity inducing Haar-Wavelet transforms within 1x1 convolution layers\nto reduce memory and compute consumption.\nAfter theoretically motivating their approach, the new method is evaluated.\nThe evaluation consists of proof of concept as well as two experiments on deep models.\nFirst, a DeeplabV3plus model is compressed and tested on the Cityscapes dataset,\nsecond, a BTS (not explained?) model is compressed and tested on a depth estimation task using\nthe Kitty dataset. Good performance is observed in both cases.", "summary_of_the_review": "Overall the paper's main idea is novel and interesting. Many minor issues make the description of the method harder to understand than necessary. I am recommending weak acceptance, for now, assuming that the most important questions will be answered properly during the rebuttal phase. Depending on the answers I am willing to reconsider my recommendation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7hk1j43XjMM", "reviewer": "Reviewer_pDi1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this paper:\n\n1， The authors take the advantage of wavelet to compress the intermediate activation maps\n\n2， The experiments are  sufficient.\n\n\n\nThe weaknesses of this paper:\n\n1， It seems that only harr wavelet is used. I think Harr Wavelet Feature Maps Compression for Low Bandwidth Convolutional Neural Networks is more appropriate.\n\n2， WCC is designed for 1x1 convolution. However, the kernels of CNNs is usually larger than 1x1.  I think the application spectrum is limited.\n\n3， The method used to compress actitation maps is borrowed from the traditional image compression technique. People is able to take other transform such as Cosine transformation, Fourier transformation to replace the Wavelets transformation. Why authors choose wavelets  here. \n", "summary_of_the_paper": "This paper proposes the wavelet compressed convolution for the activation maps of 1x1 convolutions.  The convolution achieves high compression rations and low computational cost at the cost of minimal accuracy loss. The Haar wavelet transform is adopt to conduct hardware friendly  computation.  Extensive experiments proves the performance of this method. ", "summary_of_the_review": "This paper applies the traditional image compression method to compress the activation map and conduct convolution on the transformation domain. This idea is trivial. I do not find the novelty of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3vz-wgTWKId", "reviewer": "Reviewer_mk6D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The motivation is clear. Feature map compression is heavy in image-to-image modes, and more sensitive than weight quantization in terms of precision (bit). So why not treat feature maps as images and apply image compression techniques there (with wavelet).\n- This approach is quite general; it can be applied to any vision models with Conv1x1 layers.\n- On semantic segmentation, wavelet compression with shrinkage 25% (effective 2 bit) only only has a small drop in mIOU while direct quantization with 4-bit gives very bad results. From Table 1, wavelet compressed activation leads to a graceful degradation of accuracy while there is a sudden drop of accuracy with direct quantization from 6 bit to 4 bit.\n\nWeakness\n- The paper overclaims the effectiveness of the proposed method in the abstract or introduction. It gives the reader the impression that the gain is for general image-to-image tasks. However the improvement is clear for segmentation, but  I don’t see much improvement on depth prediction, even with sophisticated (not-clean) quantization setup. (will be detailed below).  Another misleading message is Figure 1. It basically shows transform coding based image compression is better than quantizing an image in pixel space. However, the MSE for reconstruction of activation maps is not the same thing for the performance of specific tasks. E.g. in Table 1, W8A8 with 0.5 shrinkage (effective W8A4) has similar segmentation mIoU as vanilla W8A6, while in Figure 1,  vanilla A6 achieves similar mIoU to A8 with 0.5 shrinkage (effective A4), instead of A8 with 0.25 shrinkage (effective A2).\n- The evaluation of activation compression is through Bits-Operations (BoPs), but no inference time is provided. While BoPs may be one standard metric in model compression literature, it may not directly translate to inference time. E.g. transformers may have lower GMACs than ConvNets, but it does not mean they run faster on GPUs.  The reader would be more convinced if actual runtime on standard hardware (e.g. GPU or NSP) is provided. E.g. the shrinkage operator and zero filling before and after the Conv1x1 may be heterogeneous, and not as efficient as directly applying conv1x1.\n- I am not much aligned with the author’s claim on Haar wavelet in image compression (e..g the related sentence in conclusion). About Haar wavelet: 1) it  is not efficient for compression, alternative wavelet with nearly the same complexity achieves significantly better PSNR, 2) it creates very annoying visual artifacts even at moderately low bit rates, similar to blocking, but at many scales, 3) even for very demanding applications, better transforms are used (https://ieeexplore.ieee.org/abstract/document/9190899). I guess the author did not choose a better wavelet as used in JPEG2000, because otherwise the wavelet transforms becomes another 2 layers of separable conv3x3 (or conv5x5). It may go against the original motivation to compress activations after conv1x1.\n- The experiment for depth prediction is not clean, with too many models specific quantization setup, e.g. not all conv layers are quantized, it is not clear which layers are actually quantized, and compressed with proposed methods; even when some layers are compressed, certain layers (local planar guidance layers) do not apply wavelet compression; for more aggressive compression, low res feature maps use 50% shrinkage while higher res uses 25%, (25%-50%, 2-4 bit---) etc. \n- The result on depth prediction is not good even with those efforts above. From Table 2, the proposed method only achieves on-par accuracy with similar effective bit rate, e.g. vanilla W8A4 is on par with W8A8 plus 50% wavelet shrinkage. It did outperform vanilla A4 on the segmentation task, but again there is a sudden drop from A6 to A4. The last sentence in the abstract is overstatement, because this is the observation only on segmentation tasks, not for general image-to-image tasks.\n- One limitation of this paper is that the method only applies to conv 1x1, which is commutative to Haar wavelet transform. The authors claim that conv 1x1 dominates the inference time of major lightweight models, e.g. mobilenet, efficientnet. I would like to see more detailed evidence in the appendix, e.g. some runtime breakdown of conv1x1, conv3x3, etc.  in representative efficient models. Since fully connected layers take lots of compute in big transformer models, I am wondering if it is also suited there?\n- Misc\n  * It is not clear from the paper how the conv1x1 in the wavelet domain is implemented. Is it  implemented as a fully-connected layer? (in Figure 2) Again what’s the implication for runtime, it would be nice to have a break down of runtime for each stage in the Figure 2, and comparison to the baseline.\n  * One thing I am not sure about is whether the baseline quantization method is the sota in model compression literature. I would also suggest the author provide more details on quantization parameters for the experiment in the appendix.\n  * I also found the writing in Section 4 is a bit redundant. It would be easier to follow if the author directly provided the equations followed by an explanation.\n", "summary_of_the_paper": "The paper proposes to use the Haar wavelet to compress the input  feature maps (activations) of Conv1x1 layers in image-to-image tasks instead of directly quantizing them. The main observation is that wavelet transform is applied channel-wise (separable) and shrinkage of wavelet coefficients is spatial-wise, so they are commutative with Conv1x1. Thus the Conv1x1 layer can be applied after wavelet transform and shrinkage instead of on the original (larger) feature map, leading to savings in the num of bit-operations. The paper claims that the proposed scheme achieves a compression rate comparable to 2-bit on image-to-image translation tasks with only a minor drop in accuracy from 8-bit quantization. However, from the reported results, it is only the case for semantic segmentation, but there is not an obvious gain for depth prediction.", "summary_of_the_review": "The paper is well motivated and the technical part is straightforward to follow. But the actual gain with this wavelet-based compression of feature maps is not convincing, mainly the added complexity is not reflected in the Bits-Operation metric, the results on experiments are not consistently performing well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2p_RjNbL_lt", "reviewer": "Reviewer_C23G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Incomplete experimental results\n1-1) Experimental results on more and more various datasets (e.g., COCO), networks (e.g., instance segmentation, super resolution), and configurations (e.g., a variety of precision) should be included. \n1-2) Although there are not many low-precision studies targeting segmentation or depth prediction, comparison with SOTA studies must be included. If necessary, it is possible to apply the proposed method to the network for classification and compare this result with various SOTA studies.\n1-3) In addition to Haar-Wavelet, there are various wavelet transform techniques such as DWT, so it is necessary to propose a change in performance according to the wavelet technique. Also, it would be good if the performance comparison with a simple down sampling technique is added.\n\n2. Is it possible to extend the proposed method not only to the inference phase but also to the training phase?\n\n3. It is difficult to agree with the argument of this paper because the motivation of this paper is not sufficiently presented. The motivation of this study needs to be clearly presented at the beginning to secure differentiation from many recently published feature-map compression studies.\n\n4. The size of the feature maps and the size of the weight parameters vary depending on whether the layer is located at the front or the deep position, but this paper does not provide a clear consideration for these. Also, weights are often quantized much more aggressively (lower than 8-bits), but the explanation for the relationship between various weight precision and various activation precision is not included.\n\n5. In the process of searching for related research, I found a paper with a similar concept. \n\"DC-AC: Deep Correlation-based Adaptive Compression of Feature Map Planes in Convolutional Neural Networks,\" IEEE International Symposium on Circuits and Systems (ISCAS 2021)\nThis paper also seems to use the concept of image compression for feature-map compression, and therefore, it is necessary to explain the differentiation of the proposed method from this paper.\n\n6. I hope that the description of joint shrinkage can be clearly presented through the figure. In addition, although this paper claims that the goal is to reduce the amount of computation, there is a concern that using shrinkage will eventually lead to an increase in the amount of computation.\n\n7. It seems good to add a summary of the contribution to the last paragraph of the introduction.\n\n8. The composition of Section 4 seems to need some modifications. In other words, since many paragraphs already exist before Section 4-1, it is necessary to present these paragraphs as one new subsection (i.e., Section 4-1) and change the current Section 4-1 to Section 4-2.\n\n9. It is necessary to supplement the explanation of each step in Fig. 2.\n\n10. Why are only 1000 images used as presented in “batch of 1000 images” of Section 5.1? Since there are few cases, it would be good to experiment with all datasets.\n\n11. There are some typos. (e.g., 5page. “wights”  “weights” Table 1. “Wavelet shrikage”  “Wavelet shrinkage”)", "summary_of_the_paper": "This paper proposed Wavelet Compressed Convolution (WCC), a new approach for activation maps compression for 1x1 convolutions in CNN networks. The proposed method utilizes a hardware-friendly Haar-Wavelet transform, which is widely used in image compression, and performs the convolution on the compressed activation map. This method has high compatibility that can be applied to any 1x1 convolution in the existing network architecture, and shows the outstanding compression efficiency.", "summary_of_the_review": "The motivation and contribution need to be made clearer. In addition, the experiment part that supports the contribution needs to be thoroughly supplemented. Please address my concerns in \"Main Review\" through the rebuttal process.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper receives mixed ratings. All reviewers agree that the paper is well-motivated and the updated draft is clear. However, the experiment results are not fully convincing especially given the added complexity. In addition, it is hard to fully understand where the gain comes from. We hope the reviews can help improve the draft for a strong publication in the future. 
Decision: Reject