Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a new pointwise convolution (PC) method which applies conventional transforms such as DWHT and DCT. The proposed method aims to reduce the computational complexity of CNNs without degrading the performance. Compared with the original PC layer, the DWHT/DCT-based methods do not require any learnable parameters and reduce the floating-point operations. The paper also empirically optimizes the networks by removing ReLU after the proposed PC layers and using conventional transforms for high-level features extraction. Experiments on CIFAR100 show that the DWHT-based model improves the accuracy and reduces parameters and FLOPs compared with MobileNet-V1.

Although this paper is well organized and easy to follow, the novelty of the proposal seems limited and the performance improvement claimed by the author(s) is not very convincing due to the insufficiency of experiments. Firstly, the proposed method is just a manually designed and fixed 1*1 convolutional kernel, and its superiority over random initialization seems very limited as shown in Table 1. Also, the proposed method makes accuracy degrade when applied to low- and middle-level features. I wonder whether there is a more theoretical explanation for that. Moreover, the experiments are performed only on a small dataset CIFAR100. According to my own experience, the artificial convolutional kernels with some prior knowledge may work well on small datasets but tend to fail on larger ones. More experiments on larger-scale datasets like ImageNet are recommended to make results more convincing.

Therefore, my decision leans to a rejection.

Some questions:
1. How is the performance when applying RCPC only to low-/middle-/high-level features? I suppose it should be proved that the proposed method is definitely better than random initialization.
2. Why is only applying the proposed block to high-level layers working? How is the trade-off between parameters and accuracy different for each level of features?

Spelling mistake:
Page 6: in the second last paragraph, 'non-parameteric' should be 'non-parametric'.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: Summary: 

This paper proposes using non-parametric filters like Discrete Cosine Transform (DCT) and Discrete Walsh-Hadamard Transform (DWHT) which have been widely used as feature extractors in vision and image processing before deep learning became prevalent as layers especially to replace pointwise convolution (PC) layers in deep network architectures like ShuffleNet-v2 and MobileNet-V1. 

The motivation is that using such layers can capture cross-channel correlations without addition of extra parameters that need to be learnt and by replacing PC layers which tend to make up the bulk of the parameters in such settings large reduction in number of weights and flops can be achieved with little drop (or even increase) in accuracy.

They show experiments on cifar100 datasets.

Comments:

- The paper is overall easy to read although the writing and presentation can use some work. 

- I really enjoyed reading the paper though because it seems in the deep learning era we have a tendency to re-learn what we already know. This paper shows that combining our knowledge of image processing and compressed sensing with good function approximation leads to better and more compact representation learning. I think these aspects are being generally overlooked currently but wont be surprised to see more of these papers. 

- While reading section 3.2 it occurred to me that it might be interesting to consider throwing these layers in to a neural architecture search (NAS) algorithm and let it figure out the right architecture. (Just a suggestion for future work not asking for this in the rebuttal.)

- Overall I am positive about the paper and have no major asks but if time and resources permit perhaps trying out the same experiments on ImageNet to see if the trend holds.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new pointwise convolution layer, which is non-parametric and can be efficient thanks to the fast conventional transforms. Specifically, it could use either DCT or DHWT to do the transforming job and explores the optimal block structure to use this new kind of PC layer. Extensive experimental studies are provided to verify the new PC layer and experimental results show that the new layer could reduce the parameters and FLOPs while not loosing accuracy.

Overall, this paper provides a promising PC option for the CV community and the experiments seems solid. Although the novelty is limited, which is just to combines the DCT/DHWT with NN, the experiments are sufficient and I am glad to see that this simple idea works in practice. 

Concerns: This paper is based on the practical validation. Is there any theory to support that using DCT/DHWT could achieve better performance. In Image Processing, DCT/DHWT could be used to compress images or videos, but what is the benefit to use them in computer vision?

---post comments after rebuttal---
Thanks for the response. 

However, the rebuttal does not address my concerns. The logic behind the authors response is that since there is much work that uses DCT / DHWT-like features in computer vision tasks, we can also use them to replace the conventional PC layers. Unfortunately, I do not think that is the theory to support the idea described in this paper. Actually, the success of CNN architecture has already confirmed the advantage of conventional PC layer, it seems it is a backward step for the community. Moreover, I agree with Reviewer#4 that ImageNet should be used as benchmark to so the advantage of the paper. 

Hence, after reading the response and the reviews from other reviewers, I support to reject the work.
review_title: Official Blind Review #2

Meta-review: This paper presents an approach to utilize conventional frequency domain basis such as DWHT and DCT to replace the standard point-wise convolution, which can significantly reduce the computational complexity. The paper is generally well-written and easy to follow. However, the technical novelty seems limited as it is basically a simple combination of CNNs and traditional filters. Moreover, as reviewers suggested, it is our history and current consensus in the community that learned representations have significantly outperformed traditional pre-defined features or filters as the training data expands. I do understand the scientific value of revisiting and challenging that belief as commented by R1, but in order to provoke meaningful discussion, experiments on large-scale dataset like ImageNet are definitely necessary. For these reasons, I think the paper is not ready for publication at ICLR and would like to recommend rejection.
Decision: Reject