Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes a variation of the popular AlexNet architecture, where the first convolutional layer is replaced with a wavelet packet decomposition. This is motivated by the fact that the first-layer convolutional kernels look very similar to wavelet filters in that they mostly extract oriented edges and smooth gradients. The wavelet packet coefficients are then weighted using a single mixing layer implemented as a 1×1 convolution. The resulting module (wavelet packet decomposition plus 1×1 convolution) has a smaller number of learnable parameters, but achieves close to the same performance as the standard AlexNet on the ImageNet ILSVRC2012 dataset.

The paper is well laid-out and the important technical concepts are explained in a clear and concise manner. The various wavelet packet modules used to replace the first layer of AlexNet are well described and their differences clear. The same goes for the experimental section and the evaluation of the results. On the theoretical side, I am not sure that the result on page 6 qualifies as “theorem” – the fact that two successive convolutions can be written as another convolution with a wider kernel is well-known in the signal processing literature. While the experimental results are impressive, more could also be made out of the analysis. I also do not see any evidence that the authors intend to publish the code for this experiment. Despite these issues, I think the results are interesting enough that the paper be accepted for publication in the proceedings.

As discussed in the paper, there is a great need for theory explaining the performance of deep neural networks. This work is a step in that direction, reducing the number of learnable components and replacing them with fixed representations (here: wavelet packet decompositions) and achieving the same performance.

Regarding the theorem, again, this is not a new result, fundamentally, and I think this should be made clear. That being said, a potential subtlety here that needs to be addressed, is the behavior at the edge: for large enough kernels, unless the boundary extension is periodic, the composition of two convolution kernels is not necessarily another convolution. This potential issue is not discussed in the main text nor in the proof of the theorem.

Currently, the main push of the analysis is that the first-layer kernels can be replaced by similar-looking wavelet packet kernels. While important, similar results have been established before (by Oyallon et al., 2018, for example), and it is not clear what the wavelet packet approach brings to the table. Do these filters have certain properties that make them easier to analyze? Is the resulting reduction in number of learnable parameters greater?

Section 4.3 is also a bit odd. It is mostly speculation about how the number of parameters could be reduced further. Since the outlined scheme is rather simple, why have the authors not tried it? Otherwise, I do not quite understand why this section is included. Similarly, the authors speculate that the dual-tree real wavelet packets perform worse than the complex variant due to higher sensitivity to image shifts. Whether this is the case can be easily tested experimentally by shifting the images in the test set. Have the authors explored validating the hypothesis in this manner?

There are also a few small problems in the main text:
– p. 3, in Definition 1, it is not clear how the operator is defined. A reference to eq. (3) should suffice.
– p. 4, “every input channels” should be “every input channel”.
– p. 7, “predicting power” should probably be “predictive power” here and throughout.
– p. 8, “extract lesser information” should be “extract less information”.
– p. 8, “does not restricts” should be “does not restrict”.
review_title: Interesting results but more analysis is needed

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The authors propose a scheme for combining the trusted mathematical properties of Dual Tree Wavelet Packets with CNN-style feature extraction. Specifically, they learn simple functions of wavelet kernel outputs, rather than learning kernels themselves from scratch. The fundamental gains are a significant drop in the number of parameters, while retaining feature expressiveness and intuition.

Clarity:
The paper clarity is significantly above average.

Quality:
The paper is very clear and concise, but it might be better if it compared across other nets besides AlexNet. For example, how does it compare to nets with feature extractors in the same order of magnitude (3k parameters), vs much bigger (alexnet: 25k).

Originality/Significance:
I am not extremely familiar with literature related to this idea. However, I think constraining CNN filters in this way is an important area of research.
review_title: Succinct Paper with a Good Idea

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper proposes a modification to the prevalent CNN architecture that leverages filter banks based on wavelets with the motivation to reduce trainable parameters and improve interpretability. More specifically, the first CNN layer of the AlexNet architecture is replaced with a wavelet packet transform (WPT). The authors show the wavelet packet transform can be written as a series of convolution operation and visually compare the filters of the trained AlexNet to the WPT filters. Experiments on ImageNet show the proposed method can match the performance of AlexNet on ImageNet with fewer parameters.

Pros:
This work tackles in a focused way the interpretability and formalism of deep neural networks, specifically CNN, by grounding the first layers of the network in the frame work of the wavelet transform.

The similarities shown in Figure 1 between trained AlexNet kernels and kernels from the proposed method are interesting

The reduction in learned parameters for the network under the proposed approach is a nice feature, and shows that stronger priors can guide learning.

Cons:
A crucial limitation of the proposed approach seems to be its application only to the first layer of the CNN. It would be interesting to apply the method to multiple layers of the network or in the extreme case, a network based solely on the WPT.

Another limitation of this work is that the proposed method is not applied to other popular and performant CNN architectures, e.g. ResNet (He et al. 2015), and is only applied to the relatively older AlexNet model.

To be more convincing, the method must be demonstrated on more datasets or tasks. For example, other image recognition datasets or other tasks (such as ASR). Results on one CNN architecture on one dataset does not fully demonstrate the generalizability of the method.

The authors show the proposed method reduces the number of learned parameters, however does the method reduce the total number of parameters? Is the inference time of the network improved/degraded with respect to the CNN baseline?


review_title: The authors propose a nice approach incorporating the wavelet packet transform into CNN, but the limited experiments fail to demonstrate the generalizability of the method.

final_score: rating: 4: Ok but not good enough - rejection
review_content: To improve the understanding of CNNs, this paper proposes a method to constrain the behavior of AlexNet by replacing its first layer with a module based on wavelet packet decompositions. Three variants of wavelet decompositions are evaluated, including the separable wavelet packet transform and the 2D dual-tree real and complex wavelet packet transforms. A visualization algorithm is implemented to compare the extracted features with that of AlexNet. Results on an image classification task show that the accuracy of standard AlexNet can be achieved with less trainable parameters. 

Pros:
+ It is interesting to see the combined convolutional kernels, as shown in figure 3, resembles that of AlexNet. 
+ The paper presents some intuitive explanations on some aspects of the results, including (1) why and how the proposed DT-CWPT module can reduce the number of trainable parameters, with the accuracy rate maintained compared with standard AlexNet, and (2) why the three variants differ in accuracy. 

Cons:
- The insight provided by this paper is vague, and it is not so clear how this can improve our understanding of CNNs (which was claimed as one of the contributions). Furthermore, what is the benefit of replacing the first layer of AlexNet with pre-designed non-trainable convolutions? Do we have better adversarial robustness, out-of-distribution transfer, etc.? 
- The experimental results of the paper do not seem very convincing. 
(1) Despite the reasons stated in the first paragraph of page 2, AlexNet is not SOTA nor near SOTA, and the adaption of such a benchmark does not seem to provide persuasive results. 
(2) In the proposed networks, only the first layer of AlexNet is replaced, but most of the parameters are in the deeper layers, especially the FC layers. It is hard to see the advantage of such a reduction of trainable parameters. 
- Lack of experiments. 
(1) Only one dataset is used, and we do not know how the filters would be affected when applied to a different dataset. 
(2) Do other choices of CMFs generate different results? 
(3) In the introduction, the paper refers to some previous work on replacing freely-trained CNN layers with more constrained structures. What is the advantage of the proposed work over the existing ones? 

Further Questions:
* In figure 3, only filters of the red channel (i.e., $W[0,\cdot]$) are shown. But readers could also be interested in $W[i,\cdot]$ for $i=1,2$. Maybe a colored visualization of the filters is better. 
* Still figure 3, how exactly is the filter cropping done, and how can we be convinced that the cropped parts of the matrices are negligible, e.g., with small Lp norms? 
* Appendix, algorithm 1: Does $\langle W_r\rangle$ mean the shape of the tensor $W_r$? 
* Section 3.1: The number $N'$ in the dimension of $D$ becomes 56 in  $Y$. Does the full-convolution (as defined in section 2) guarantee $N'=56$, or is there a cropping procedure? 
review_title: Replacing the first layer of AlexNet with wavelet packet transforms to reduce model parameters. Interesting but lack of explanations and comparisons.

Meta-review: The paper is motivated by the observed similarity between learned filters at the low layers of a convolutional neural network and oriented Gabor filters. It proposes to replace the lower layers with dual tree wavelet packet transforms, which yield fixed oriented frequency-selective features. Instead of learning filters from scratch, it proposes to learn only a scalar importance for each of these features, reducing the number of learned parameters. Experiments with the AlexNet architecture on ImageNet indicate that this modification does not reduce performance, but does significantly reduce the number of parameters. The paper argues that this modification also improves the interpretability (and in the case of complex dual tree wavelets, potentially the invariance properties) of the low-level features. 

Pros and cons:

[+/-] As the paper clearly argues, replacing learned filters with wavelet packet transforms improves the interpretability of the low layers of a convolutional network. While other works have pursued similar ideas, limiting the conceptual novelty, at a technical level this is the first work to use the dual tree complex wavelet transform for this purpose. The DT-CWT may have mathematical advantages. The paper and rebuttal argue that it it is conceptually cleaner (“sparser”, since the transform is generated by a single filter) although there may not be a greater reduction in the number of trainable parameters.

[+] The additional per-channel weights are redundant in terms of the representation capacity of the network, but may effectively introduce sparse regularization (see work on the “Hadamard parameterization” in implicit sparse regularization), allowing the network to select relevant wavelet features.

[+] The paper is well-organized and cleanly written. The authors revision has done a good job of addressing all clarity concerns of reviewers. 

[-] Several reviewers raised concerns about the limited scope of the experiments: the paper only replaces a single layer of a particular architecture (AlexNet) and evaluates on one particular dataset (ImageNet). 

[-] The main proposed benefit of this modification is in the interpretability of the network and its potential amenability to mathematical analysis. This claim would be stronger if the paper either 1. showed the benefit by exhibiting some rudimentary mathematical theory for this network or 2. used this idea to demonstrate networks that are significantly more interpretable, say by replacing all learned convolutional layers with DT-CWPT. 

The paper’s reviews were split. All reviewers appreciate the paper’s clean exposition of a reasonable idea, and note the novelty of using dual tree wavelets in this context. However, reviewers express concerns about the paper’s significance: it could do more to show how replacing the lowest layer with DT-CWPT yields new insights, and do more to demonstrate (both rhetorically and experimentally) the generality of its ideas. Based on the bulk of the reviews, as written the paper falls slightly below the threshold for acceptance. 
Decision: Reject