Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use the Haar wavelet to compress the input  feature maps (activations) of Conv1x1 layers in image-to-image tasks instead of directly quantizing them. The main observation is that wavelet transform is applied channel-wise (separable) and shrinkage of wavelet coefficients is spatial-wise, so they are commutative with Conv1x1. Thus the Conv1x1 layer can be applied after wavelet transform and shrinkage instead of on the original (larger) feature map, leading to savings in the num of bit-operations. The paper claims that the proposed scheme achieves a compression rate comparable to 2-bit on image-to-image translation tasks with only a minor drop in accuracy from 8-bit quantization. However, from the reported results, it is only the case for semantic segmentation, but there is not an obvious gain for depth prediction.
main_review: Strengths
- The motivation is clear. Feature map compression is heavy in image-to-image modes, and more sensitive than weight quantization in terms of precision (bit). So why not treat feature maps as images and apply image compression techniques there (with wavelet).
- This approach is quite general; it can be applied to any vision models with Conv1x1 layers.
- On semantic segmentation, wavelet compression with shrinkage 25% (effective 2 bit) only only has a small drop in mIOU while direct quantization with 4-bit gives very bad results. From Table 1, wavelet compressed activation leads to a graceful degradation of accuracy while there is a sudden drop of accuracy with direct quantization from 6 bit to 4 bit.

Weakness
- The paper overclaims the effectiveness of the proposed method in the abstract or introduction. It gives the reader the impression that the gain is for general image-to-image tasks. However the improvement is clear for segmentation, but  I don’t see much improvement on depth prediction, even with sophisticated (not-clean) quantization setup. (will be detailed below).  Another misleading message is Figure 1. It basically shows transform coding based image compression is better than quantizing an image in pixel space. However, the MSE for reconstruction of activation maps is not the same thing for the performance of specific tasks. E.g. in Table 1, W8A8 with 0.5 shrinkage (effective W8A4) has similar segmentation mIoU as vanilla W8A6, while in Figure 1,  vanilla A6 achieves similar mIoU to A8 with 0.5 shrinkage (effective A4), instead of A8 with 0.25 shrinkage (effective A2).
- The evaluation of activation compression is through Bits-Operations (BoPs), but no inference time is provided. While BoPs may be one standard metric in model compression literature, it may not directly translate to inference time. E.g. transformers may have lower GMACs than ConvNets, but it does not mean they run faster on GPUs.  The reader would be more convinced if actual runtime on standard hardware (e.g. GPU or NSP) is provided. E.g. the shrinkage operator and zero filling before and after the Conv1x1 may be heterogeneous, and not as efficient as directly applying conv1x1.
- I am not much aligned with the author’s claim on Haar wavelet in image compression (e..g the related sentence in conclusion). About Haar wavelet: 1) it  is not efficient for compression, alternative wavelet with nearly the same complexity achieves significantly better PSNR, 2) it creates very annoying visual artifacts even at moderately low bit rates, similar to blocking, but at many scales, 3) even for very demanding applications, better transforms are used (https://ieeexplore.ieee.org/abstract/document/9190899). I guess the author did not choose a better wavelet as used in JPEG2000, because otherwise the wavelet transforms becomes another 2 layers of separable conv3x3 (or conv5x5). It may go against the original motivation to compress activations after conv1x1.
- The experiment for depth prediction is not clean, with too many models specific quantization setup, e.g. not all conv layers are quantized, it is not clear which layers are actually quantized, and compressed with proposed methods; even when some layers are compressed, certain layers (local planar guidance layers) do not apply wavelet compression; for more aggressive compression, low res feature maps use 50% shrinkage while higher res uses 25%, (25%-50%, 2-4 bit---) etc. 
- The result on depth prediction is not good even with those efforts above. From Table 2, the proposed method only achieves on-par accuracy with similar effective bit rate, e.g. vanilla W8A4 is on par with W8A8 plus 50% wavelet shrinkage. It did outperform vanilla A4 on the segmentation task, but again there is a sudden drop from A6 to A4. The last sentence in the abstract is overstatement, because this is the observation only on segmentation tasks, not for general image-to-image tasks.
- One limitation of this paper is that the method only applies to conv 1x1, which is commutative to Haar wavelet transform. The authors claim that conv 1x1 dominates the inference time of major lightweight models, e.g. mobilenet, efficientnet. I would like to see more detailed evidence in the appendix, e.g. some runtime breakdown of conv1x1, conv3x3, etc.  in representative efficient models. Since fully connected layers take lots of compute in big transformer models, I am wondering if it is also suited there?
- Misc
  * It is not clear from the paper how the conv1x1 in the wavelet domain is implemented. Is it  implemented as a fully-connected layer? (in Figure 2) Again what’s the implication for runtime, it would be nice to have a break down of runtime for each stage in the Figure 2, and comparison to the baseline.
  * One thing I am not sure about is whether the baseline quantization method is the sota in model compression literature. I would also suggest the author provide more details on quantization parameters for the experiment in the appendix.
  * I also found the writing in Section 4 is a bit redundant. It would be easier to follow if the author directly provided the equations followed by an explanation.

summary_of_the_review: The paper is well motivated and the technical part is straightforward to follow. But the actual gain with this wavelet-based compression of feature maps is not convincing, mainly the added complexity is not reflected in the Bits-Operation metric, the results on experiments are not consistently performing well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to use sparsity inducing Haar-Wavelet transforms within 1x1 convolution layers
to reduce memory and compute consumption.
After theoretically motivating their approach, the new method is evaluated.
The evaluation consists of proof of concept as well as two experiments on deep models.
First, a DeeplabV3plus model is compressed and tested on the Cityscapes dataset,
second, a BTS (not explained?) model is compressed and tested on a depth estimation task using
the Kitty dataset. Good performance is observed in both cases.
main_review: ## Strengths:
- The paper presents a neat and clever way to apply traditional wavelet-image compression in CNN.
- The qualitative compression assessment appears to be visually convincing and supported by past work on image compression.
- In the method performs decently on a segmentation network on cityscapes as well as a depth estimation network on kitty.
- Supplementary source code: Decompressing the zip file worked. The source code is not documented but readable.
- Working with Haar-wavelets is usually a good start and not unheard of in the machine learning literature:
   - Rinon Gal, Dana Cohen, Amit Bermano, and Daniel Cohen-Or. Swagan: A style-based wavelet-driven
     generative model. arXiv preprint arXiv:2102.06108, 2021.
   - Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International
   Conference on Learning Representations, 2018

   Do so as well for image generation and pooling tasks. Having established the utility of Haar wavelets, more complex wavelets could be explored in future work.


## Open Questions:
### Related work:
- page 3: Williams and Li (2018) write, " For our proposed method, the wavelet basis is the Haar wavelet, mainly for
its even, square subbands. " the pooling filters are, presumably, static in this case?

### Background:
- Filter construction:
It appears the two-dimensional filters in eq. (3) are constructed from 1d-pairs using outer products:
$$ f_{ll} = f_0 \cdot f_0^T , f_{hl} =  f_1 \cdot f_0^T , f_{hl} =  f_0 \cdot f_1^T , f_{hh} =  f_1 \cdot f_1^T ,$$
as described in i. e.
    - [Vyas et al., 2018] Vyas, A., Yu, S., and Paik, J. (2018). Multiscale transforms with application to image processing. Springer.
    
    The process is currently not obvious. Would authors consider adding a reference or explanation,
    to help readers understand the genesis of the 2d-Haar filters in equation 3?

- Convolution definition:
Why not be more specific just above equation three and speak of a stride two convolution? Doing so would explain why the resolution is 
halved for the resulting coefficients $\mathbf{y}_1 , \mathbf{y}_2, \mathbf{y}_3, \mathbf{y}_4 $.

### Wavelet Compressed Convolution
- What are the matrix dimensions in equations six and seven? Are the wavelet and the convolution matrices square or $\mathbb{R}^{n,n}$ in this case?
What about $\mathbf{T}$ and what would the impact of a nonsquare $\mathbf{T}$ on $\mathbf{K}_{1x1}$ be?
Assuming most of the computational work happens there. 
Writing this down could help make this section's main points and equation nine easier to understand.

- How are the (tensor ?) dimensions of $\mathbf{x}$ defined? https://www.deeplearningbook.org/contents/notation.html suggests lowercase boldface
should be a vector. A vector makes sense because fast wavelet matrices can be computed by matrix multiplication with an image channel
flattened into a vector. But it seems in equation eight $\sum_j k_{ij} x_j$ sums along a channel dimension?
If that is the case, does $\mathbf{H}\mathbf{x}$ transform the channel dimension in equation seven? 

### Computational cost
- How do the input and channel sizes from equations ten, eleven, and twelve relate to the matrices and vectors (?) in equations six to nine? 

### Experiments
- For all experiments: of all convolution operations how many are 1x1?
- Segmentation:
    - Does (W / A) mean, weights / activations in Table 1? Perhaps this is something for the caption.
    - The authors may want to consider adding the results of Liu et al. (2021) to table 1.

- Depth estimation:
    - What does BTS stand for, what does it mean? 

### Minor remarks:
- page 3: ..., or for utilize ... -> ..., or utilize ...
- page 4: ..., to be trained end to end manner, ... -> .., to be trained in an end to end manner ...
- page 5: wights -> weights
- page 9: Conclusions -> Conclusion
summary_of_the_review: Overall the paper's main idea is novel and interesting. Many minor issues make the description of the method harder to understand than necessary. I am recommending weak acceptance, for now, assuming that the most important questions will be answered properly during the rebuttal phase. Depending on the answers I am willing to reconsider my recommendation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed Wavelet Compressed Convolution (WCC), a new approach for activation maps compression for 1x1 convolutions in CNN networks. The proposed method utilizes a hardware-friendly Haar-Wavelet transform, which is widely used in image compression, and performs the convolution on the compressed activation map. This method has high compatibility that can be applied to any 1x1 convolution in the existing network architecture, and shows the outstanding compression efficiency.
main_review: 1. Incomplete experimental results
1-1) Experimental results on more and more various datasets (e.g., COCO), networks (e.g., instance segmentation, super resolution), and configurations (e.g., a variety of precision) should be included. 
1-2) Although there are not many low-precision studies targeting segmentation or depth prediction, comparison with SOTA studies must be included. If necessary, it is possible to apply the proposed method to the network for classification and compare this result with various SOTA studies.
1-3) In addition to Haar-Wavelet, there are various wavelet transform techniques such as DWT, so it is necessary to propose a change in performance according to the wavelet technique. Also, it would be good if the performance comparison with a simple down sampling technique is added.

2. Is it possible to extend the proposed method not only to the inference phase but also to the training phase?

3. It is difficult to agree with the argument of this paper because the motivation of this paper is not sufficiently presented. The motivation of this study needs to be clearly presented at the beginning to secure differentiation from many recently published feature-map compression studies.

4. The size of the feature maps and the size of the weight parameters vary depending on whether the layer is located at the front or the deep position, but this paper does not provide a clear consideration for these. Also, weights are often quantized much more aggressively (lower than 8-bits), but the explanation for the relationship between various weight precision and various activation precision is not included.

5. In the process of searching for related research, I found a paper with a similar concept. 
"DC-AC: Deep Correlation-based Adaptive Compression of Feature Map Planes in Convolutional Neural Networks," IEEE International Symposium on Circuits and Systems (ISCAS 2021)
This paper also seems to use the concept of image compression for feature-map compression, and therefore, it is necessary to explain the differentiation of the proposed method from this paper.

6. I hope that the description of joint shrinkage can be clearly presented through the figure. In addition, although this paper claims that the goal is to reduce the amount of computation, there is a concern that using shrinkage will eventually lead to an increase in the amount of computation.

7. It seems good to add a summary of the contribution to the last paragraph of the introduction.

8. The composition of Section 4 seems to need some modifications. In other words, since many paragraphs already exist before Section 4-1, it is necessary to present these paragraphs as one new subsection (i.e., Section 4-1) and change the current Section 4-1 to Section 4-2.

9. It is necessary to supplement the explanation of each step in Fig. 2.

10. Why are only 1000 images used as presented in “batch of 1000 images” of Section 5.1? Since there are few cases, it would be good to experiment with all datasets.

11. There are some typos. (e.g., 5page. “wights”  “weights” Table 1. “Wavelet shrikage”  “Wavelet shrinkage”)
summary_of_the_review: The motivation and contribution need to be made clearer. In addition, the experiment part that supports the contribution needs to be thoroughly supplemented. Please address my concerns in "Main Review" through the rebuttal process.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the wavelet compressed convolution for the activation maps of 1x1 convolutions.  The convolution achieves high compression rations and low computational cost at the cost of minimal accuracy loss. The Haar wavelet transform is adopt to conduct hardware friendly  computation.  Extensive experiments proves the performance of this method. 
main_review: The strengths of this paper:

1， The authors take the advantage of wavelet to compress the intermediate activation maps

2， The experiments are  sufficient.



The weaknesses of this paper:

1， It seems that only harr wavelet is used. I think Harr Wavelet Feature Maps Compression for Low Bandwidth Convolutional Neural Networks is more appropriate.

2， WCC is designed for 1x1 convolution. However, the kernels of CNNs is usually larger than 1x1.  I think the application spectrum is limited.

3， The method used to compress actitation maps is borrowed from the traditional image compression technique. People is able to take other transform such as Cosine transformation, Fourier transformation to replace the Wavelets transformation. Why authors choose wavelets  here. 

summary_of_the_review: This paper applies the traditional image compression method to compress the activation map and conduct convolution on the transformation domain. This idea is trivial. I do not find the novelty of this paper.
review_title: null

Meta-review: The paper receives mixed ratings. All reviewers agree that the paper is well-motivated and the updated draft is clear. However, the experiment results are not fully convincing especially given the added complexity. In addition, it is hard to fully understand where the gain comes from. We hope the reviews can help improve the draft for a strong publication in the future. 
Decision: Reject