Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on designing efficient deep networks with a proposed novel parameter-free layers. Drawing the spirit of depth-wise convolution, shift operation, the paper studies the ways of inserting parameter-free operations into the building block using NAS technique. The resulted models perform both effectively and efficiently on ImageNet dataset.
main_review: Strengths:
1. Simple but novel idea. The first attempt to investigate built-in parameter-free layer as a building block for network efficiency.
2. Good and convincing empirical studies. The proposed method achieves compelling results on ImageNet dataset (Table 3, best trade-off for GPU), with detailed analysis on different network variants (CNN and ViT, + deformable operations etc).
3. Clear and sound technical presentation. The paper clearly elaborates the technical design and insight, and illustrates the difference with existing efficient building block. 
4. Interesting and inspiring findings. For example, similar performances have been achieved using max-pool and avg-pool operation (Appedix B).

Questions:
1. Is there any other parameter-free operations that could possibly achieve good trade-off except max and avg operation? 

summary_of_the_review: A good paper. I definitely suggest acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper authors propose to use simple parameter-free operations to replace several kinds of trainable layers to achieve speedup on hardware with less performance drop. Extensive experiments have been done to demonstrate the effectiveness of the proposed method, including experiments on replacement in both single and multiple bottlenecks and neural architecture search. In addition, authors also employ such parameter-free operations to redesign CNNs and vision transformers for a good balance between speed and performance.
main_review: Pros:
+ A simple yet effective finding of using parameter-free operations to achieve speedup with reasonable performance drop.  
+ Very extensive experiments to support authors' claim on the effectiveness of the parameter-free operations.  

Cons:
- In spite of extensive experiments to demonstrate the effectiveness, it lacks some insightful explanation on the reason why it works, why it could achieve a good balance between speed and performance.
- In the experiment of hybrid architecture with efficient bottlenecks, what about putting the designed efficient block at different position of the architecture? What about alternatively stack baseline and efficient blocks?
- The performance drops more 1% in AP while reducing less than 5 ms running time in the object detection task. It seems not to be a good tradeoff between performance and speed. Any explanation on such performance in the object detection task? Any solution to address this issue?
- Why does ViT-S with maxpooling shows slight improvement in performance while drop in speed?
summary_of_the_review: This paper shows a simple yet effective finding and validate this with extensive experiments. However, it lacks some insightful explanation and novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper argues for more extensive use of max-pooling layers in image classification networks, as an inexpensive substitute for convolutions.  Recent works have removed parameters (and often-expensive operations) using bottlenecks, depthwise/channel-wise operations, etc., and this paper pushes further in this direction by studying max-pooling as a cheap and effective spatial combination mechanism in the context of recent resnet-derived and image transformer architectures.  Empirical experiments are performed on CIFAR-10 and three Imagenet variants, demonstrating the effectiveness of incorporating max-pooling layers beyond applications in strided layers and secondary branches.

main_review: The framing of the method as more general parameter-free operation is over-stated for what has been demonstrated, which is limited to max-pooling (and average pooling in a small number of cases).  In particular, the abstract mentions "parameter-free" several times, and "max-pooling" not once.  (A second operation, deformable max-pool, is mentioned, but only as a direction for future work, and is not studied beyond one small experiment in the appendix).  Likewise, calling the new layers "efficient layers" seems a little inflated to me, if they are just max-pooling substitutions.  It seems more accurate to call them "max-pooling bottlenecks" or something specific to max-pooling.

Still, the empirical results are intriguing, showing that simple max-pooling ops can be used in place of convolutions much more than I might have expected.

Although I feel I was able to follow most of the paper well enough to understand the technique, I did feel like I was guessing a bit at several key points, and would have liked to have seen more precise definitions.  I have listed these in below detailed comments.

Overall, the empirical results are quite interesting, showing that max-pooling can be used in place of convs or depth-wise convs more extensively than I might have expected, and offers a meaningful datapoint in the wider context parameter-free operations.  While the most specific claims are supported by the experiments, I think the overall framing as a demonstration of parameter-free layers is over-generalized for what has been demonstrated around max-pooling.  There are also many details that I was unclear on, and feel could be more clearly described (below).


Additional comments/questions:


* A couple older works might also be interesting to discuss
  - Maxout Networks, Goodfellow et al 2013 https://arxiv.org/pdf/1302.4389.pdf --- uses max as a nonlinearity in place of relu
  - Inception v1, v2, v3, ... etc networks also use max pooling as a branch, and some of the cells found in the NAS in this paper look similar to Inception branches

* sec 3.2:  The use of W = s(g) doesn't quite correspond to what I think is the intended operation.  "W_v,:,: = s(g)" looks like there is one W matrix that is constructed globally based on the entire feature map g, and used used as a convolution kernel.  Instead, I think the intent is to describe a per-window operation, in which case there should be a dependency on the window location i,j.

* sec 3.2:  i,j are used twice in W, argmax description:  once in the definition (i,j)=argmax_hw, and once again in the expression in the argmax g_v,r*i+h,r*j+w.  most likely the i,j inside and outside the argmax are different (new names would help here).


* sec 3.3 multiple bottlenecks experiment:  I had trouble following the description of this experiment.  What was the exact architecture, and how were the replacements of convolutions performed?  How many layers were replaced at a time, and with which operations?  My best guess here, is that starting from a resnet with convolution residuals, new models were created by a procedure where (1) a set of one or more layers were selected for operation replacement (how?), and then (2) in these layers, 3x3 convolutions were replaced with either (a) max pooling or (b) depthwise convolution, creating pairs of models for cases (a) and (b).  The best 20% of models in each (a) and (b) conditions are then plotted in red and blue in Fig 2.  Is this correct?

* Fig 2:  It would also be interesting to see not just the best 20%, but also the other 80%, what the performance looks like as it degrades.


* sec 4.1:  This section is the first use of the phrase "efficient bottleneck" which is a little confusing, as it would seem the same module was investigated in the cifar10 experiments as well, and described in sec 3.  I think it would be clearer to define the "efficient bottleneck" in sec 3 using an equation that closely parallels eqns (1)..(4).  I believe that the "efficient bottleneck" operation is the eq (2) but replacing the innermost double-sum (over space window and \roh*c channels) with per-channel max pooling over k sized windows.


* transformer and Fig A.1:  I'm not sure exactly where the maxpooling operation is applied.  In fig A.1(b), there are three paths into the attention layer, which seem to indicate these correspond to q,k,v.  But the same three paths go into the eff-layer in A.1(d) --- does this layer actually combine these three sets of features somehow, or is it applied to just one set?


* "deformable max-pool operation":  how are the predicted locations are found for this operation, are there parameters involved in this step?  what parts do or do not involve parameters?


* Tables 3,4:  It would be informative to plot the first three columns of tables 3 and 4 vs top1 in a set of scatterplots, to better see the speed/performance tradeoff and whether this method moves along the frontier or pushes it.

* Fig 1:  would be nice to have the condition names in left/top of the plot grid, not just the caption


summary_of_the_review: Overall, the empirical results are quite interesting, showing that max-pooling can be used in place of convs or depth-wise convs more extensively than I might have expected, and offers a meaningful datapoint in the wider context parameter-free operations.  While the most specific claims are supported by the experiments, I think the overall framing as a demonstration of parameter-free layers is over-generalized for what has been demonstrated around max-pooling.  There are also many details that I was unclear on, and feel could be more clearly described.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to use some parameter-free operations to replace the spatial operations (e.g., 3x3 conv, 3x3 depth conv). The author shows that using the parameter-free operations (e.g., max pooling) can also achieve a good performance. The author conducted thorough ablations to show the effectiveness of parameter-free operations, and the method is evaluated on different architectures and datasets.
main_review: Strength: 
1. I think the idea of replacing the spatial operation (e.g., 3x3 conv) with some parameter-free/cheap operations is good. And the results showed that even simple max pooling could work well, which is inspiring.
2. The author conducted thorough ablation studies and evaluations on different architectures and datasets. These experiments clearly demonstrated the advantages and disadvantages of the method.
3. The paper is well written.

Weakness:
1. I think replacing the spatial operations with some cheap operations is a good idea, but the author could dive deeper into this. Currently, the author mainly conducted this with max pooling, and it achieves reasonable results but not surprisingly good. For example, in most cases, the inference speed is improved by 10%-20% but the accuracy is also decreased by 1-2 points. This seems to be a pretty normal accuracy-efficiency trade-off. 
2. The author tried some improvements, i.e., deform_max, which I think is a good direction. But there are few experiments on it (only in Table 3). I am curious about how the deform_max works without improved training tricks and on other tasks.
3. The author claimed that depthwise conv is light in params/flops but heavy in inference speed (which I agree with), while the built-in max pooling is highly optimized and efficient. But in Table 4, the flops seems to be reduced by 30%+ while the inference time is only reduced by 10%-20%. Could the author explain this?
4. The author claimed the deform_max could be optimized further to speed up. I think the author could work on that and show how much speedup it has.

summary_of_the_review: Overall, I think this paper proposed an inspiring idea but the author could explore further on this. The current results are not surprisingly good but promising. It would be good If the author could further improve the method.

=============== Post rebuttal ================

Thank the authors for the detailed response. I am satisfied with most of it. I still have some concerns about the proposed deform_max and its performance. The idea of replacing conv with pooling (or general parameter-free operations) seems promising. But the results in the paper are not suprising yet, even with the deform_max. The accuracy is about 1% lower than baseline, with 1.2 GFLOPs less computation but even more inference time. It would be good if the author could further improve and optimize their method. I would like to keep my score as weak accept.
review_title: null

Meta-review: This paper analyzes the extent to which parameterized layers within a CNN can be replaced by parameter-free layers, with specific focus on utilizing max-pooling as a building block.  After the author response and discussion, all reviewers favor accepting the paper.  The AC agrees that its empirical results open a potentially interesting discussion on network design.
Decision: Accept (Poster)