Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper researches the pooling operation, which is an important component in convolutional neural networks (CNN) for image classification. Taking the perspective from signal processing, this paper proposes a pooling operation called frequency pooling (F-pooling). The key motivation is to make the pooling operation shift-equivalent and anti-aliasing. This paper gives an improved definition on shift-equivalent functions and shows that the proposed F-pooling is optimal in the sense of reconstructing the orignal signal. The F-pooling is then implemented with matrix multiplications and tested with recent convolutional neural networks for image classifiation on CIFAR-100 and a subset of ImageNet dataset. 

It is interesting to take the perspective from signal processing to give pooling operation in CNN a formal treatment. As indicated in the recent literature, enforcing shift-invariance does help to improve the performance of a CNN on classification accuracy and the robustness with respect to image shift. At the same time, this work can be further enhanced at the following aspects:
1. This work can make it clearer in principle how anti-aliasing contributes to improving the classification performance and robustness. This will help to make this paper more self-contained. 
2. When showing the optimality of F-pooling in Section 2.3, the criterion is to reconstruct the original signal x. Considering that the ultimate goal is classification, the information to be maximally preserved through each operation through the layers shall be the information that relates to the class label y. In light of this, some justification and explanation shall be provided for using this criterion for optimality.
3. The experimental study is weak. Experiments could be conducted on more benchmark datasets with more CNN architectures to convincingly show the effectiveness of the proposed F-pooling. Also, from the three Tables in the experimental part, the improvement of F-pooling over AA-pooling (developed by the main reference of this work) does not seem to be significant or consistent. For example, in Table 2, the F-pooling only wins at either accuracy (marginally) or consistency, but not both. In Table 3, the F-pooling consistently shows inferior classification performance, although obtaining slightly higher consistency. This makes the advantage of F-pooling over the existing AA-pooling unclear.      
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposed a new pooling method (Frequency pooling) which is strict shift equivalent and anti-aliasing in theory. The authors first derived the theory of F-pooling to be optimal anti-aliasing down sampling and is shift-equivalent in sec 2, and then demonstrated the experimental results of 1D signals and image classification tasks. 

The experimental results are actually less impressive than what are claimed in contribution and conclusion. The authors stated that "F-pooling remarkably increases accuracy and robustness w.r.t. shifts of moderns CNNs"; however, in Table 1-3, the winning margin of accuracy is actually quite small (<2%), and the consistency (<3.5% compared to the second best baseline except resnet-18 on CIFAR 100 has large improvement ~7-8%). 

Questions:
1. For the experiment of 1D signal on sine wave, the AA-pooling and F-pooling give the same result? 
2. Compared to AA-pooling, it seems that F-pooling has a better theoretical guarantee (i.e. the optimal anti-aliasing down sampling operation given U). But other than this, the empirical performance seem not showing particular advantage over AA-pooling. Are there any other advantages for F-pooling s.t. people might want to use it as opposed to AA-pooling?
3. What are the limitations of the F-pooling? It is good to me that the authors discuss one limitation on the imaginary part of output and I would like to hear more on other potential limitations for this method. 
- also, if the authors can explain more on sec 2.5 it will be helpful. If we simply ignore the imaginary part, although the theory is not applicable, but what would the empirical performance be? 


review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposed "F pooling" for Frequency Pooling, which is a pooling operation satisfying shift equivalence and anti-aliasing properties. The method is very simple: first, transform the input 1D/2D signal into the spectrum domain based on discrete Fourier transform (DFT), then cut the high-frequencies, then transform back to the time domain using the inverse DFT. The method can be implemented using FFT and auto differentiation frameworks. The method is tested on Resnet/Desnet on CIFAR-100 and subsets of ImageNet, showing better performance than the original models.

The reviewer votes for rejection as the method has limited novelty. Spectrum pooling has been used in the community of computer vision and machine learning. Taking a random example (there are others by simple searching), in the ECCV paper "DFT-based Transformation Invariant Pooling Layer for Visual Classification, Ryu et al., 2018" The DFT magnitude pooling is almost the same as the authors' propositions, where the "Fourier coefficients are cropped by cutting off high-frequency components". 

The reviewer encourages the authors to make further new developments and have a more comprehensive literature review. But in the current form, the paper has less value to be published in ICLR.
review_title: Official Blind Review #4

Meta-review: This submission has been assessed by three reviewers and scored 3/6/1. The reviewers also have not increased their scores after the rebuttal. Two reviewers pointed to poor experimental results that do not fully support what is claimed in contributions and conclusions. Theoretical support for the reconstruction criterion was considered weak. Finally, the paer is pointed to be a special case of (Zhang 2019). While the paper has some merits, all reviewers had a large number of unresolved criticism. Thus, this paper cannot be accepted by ICLR2020.

Decision: Reject