Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new type of separable convolution to improve ConvNet efficiency. Based on a few assumptions (receptive field condition, channel condition, group conv condition), it mathematically calculate the “optimal” configurations for separable convolutions. Experiments are mostly done on CIFAR and ImageNet.

======== Strengths

1). Comprehensive study on important separable conv building blocks (e.g. Table 1 is quite informative)
2). The idea of split the groups in both stages of separable convs is somewhat new and interesting.


======== Weaknesses

1). My first concern is about the unrealistic assumptions. For example, Eq (5) “channel condition” requires g1 * g2 = C2, which doesn’t make sense to me: there is no intuition, and most existing convs doesn’t satisfy this assumption: (1) regular conv g1=g2=1 != C2 doesn’t satisfy this; (2) spatial separable conv g1=g2=1 != C2. This assumption is critical to arrive equation (7) and (8), but is unclear where this assumption comes from.  Due to these unrealistic assumptions, the term “optimal” is also questionable.

2). Second, the CIFAR results show the new layers are not much better than others. As shown in Figure 3, the largest gain is <1%, and sometimes the o-ResNet (~88%) is slightly worse than d-ResNet (which indicates the propose layers might be not "optimal"?) The improvements on ImageNet in Table 4 seem to be promising, but as discussed in DARTS+ and other recent works, the search process of DARTS is often unstable and could potentially have high variance.

3). My another main concern is about the weak baseline. As this paper is study separable convs, it should compare to separable conv based models like MobileNet/FBNet/EfficientNet, rather than the full conv based ResNet. For example, by leveraging depthwise and seprable convs, MobileNetV3 achieves 75.2% ImageNet top-1 accuracy with 219 FLOPs, which is a much stronger baseline than the on in Table4. I highly recommend the authors to conduct their experiments on these baselines.


======== Suggestions

1). Instead of formulating it as a mathematically optimal solution based on unrealistic assumptions, I recommend the authors to conduct more empirical studies on these design choices. For example, the paper only shows the performance results of “optimal” (g1, g2) computed by equation (7), but it would be helpful to show the performance for different (g1, g2) values, and compare them with the “optimal” (g1, g2).

2). I recommend the authors to use the latest MobileNet or EfficientNet (or other separable conv based models) as baselines, and replace their separable convs with the proposed “optimal separable convs”, and compare the performance gains. 

review_title: Interesting ideas, strong assumptions,  relatively weak results

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This paper proposes a novel analysis for optimal separable convolution considering the number of parameters and FLOPs. The idea is to constraint the input information consumed stationery throughout the optimization of the parameters for separable convolutions. More specifically, this paper proposes the notion of volumetric receptive field and by holding it constant throughout optimization, one can arrive at a constrained optimization problem for solving the parameters for the optimal separable convolution. Empirical results of replacing common convolution with optimal convolution in various modern CNNs have demonstrated the effectiveness of the proposed optimal separable convolution.

### Reasons for score

I like the idea a lot. This paper provides a principled way of designing convolution to minimize FLOPs or parameter counts without resorting to black-box optimization or algorithms of sorts. The key insight brought by the paper is to keep the volumetric receptive field constant, which seems reasonable for me. With such an observation, solving for optimal separable convolution now becomes an optimization problem that can be solved efficiently. Empirical results on modern CNNs have shown the effectiveness of this approach.

### Strengths

- A novel and principled approach to design separable convolution, which is of critical importance. In contrast to existing efforts in designing efficient CNNs (e.g., NAS, pruning, quantization), this paper explores a complimentary yet relative under-explored direction, i.e., optimally design the convolutional operator. I can imagine how NAS and the proposed work be combined to lead to even better results in future work.

- Good empirical results by simply replacing old convolution operators with the proposed one.

- The analysis is easy to follow and generally agreeable to read.


### Weaknesses

- It would be interesting to see results on wall-clock time in addition to FLOPs and # Params. With that said, this is not a deal-breaker. While it might be the case that the proposed convolution operator falls short compared to the existing ones given the hardware and software implementation we have now, I still think this work would be a great motivation for hardware and software research to look into.

- It would be interesting to see results for commonly adopted CNNs on ImageNet, e.g., ResNet-50 and/or MobileNetV2. Again, this is good to have, but not a deal-breaker from my perspective.


### Post rebuttal

I appreciate the authors' efforts in conducting experiments to show the latency results. After reading through the rebuttal and the reviews from other reviewers, I would like to down-grade my score by 1.
Specifically, I agree with R3 and R4 that it would be better if experimental results are done for MobileNets/EfficientNets to empirically demonstrate the effectiveness of the optimal convolution. Those networks present strong baselines and it would be more convincing if optimal convolution indeed outperforms those. With that said, I agree with the authors that the DARTS experiments have shown that the optimal convolution can be better than depth-wise separable convolutions. As a result, I still recommend acceptance for this paper.
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
The paper presents a new convolution structure which tries to achieve a better balance between the efficiency and accuracy. The proposed approach is well motivated and theoretically proved. Reasonable experiments have been provided to validate the proposed algorithm. 

## Pros
1. The paper is well presented and the motivation of the paper is clear.
2. The proposed convolution structure has theoretical small FLOPs and well justified based on the proof. 
3. Reasonable experiments have been reported to valiate the the performance gain over the baselines.

## Cons
1. Besides from the FLOPs, is it possible to provide the computational cost for the proposed algorithm, e.g., including the inference speed in the experiments like Table 3. From the engineering implementation, the proposed structure may not be hardware-friendly. 
2. For the experiments on Full Imagenet, what about the experiments for the comparison with the resnet baseline. Also, I would suggest to include the comparison with the baseline with depthwise convolution. 

## Reasons for the rating
The exploration of the structure of the convolution is challenging but important to the community. The discussion of the convolution based on balance of the efficiency and effectiveness is meaningful. Although the experiments do not cover all of my concerns, I would rate it as marginally above the acceptance threshold.

## Suggestions
Please provde the the comparison of inference speed for the proposed structure. Also, it would be better to report the baseline with depthwise convolution. 
review_title: RETHINKING CONVOLUTION: TOWARDS AN OPTIMAL EFFICIENCY

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**

This paper proposes a novel type of convolution called optimal separable convolution. Compared with existing separable convolutions like depth separable and spatial separable convolutions, the authors design a scheme to achieve an optimal separation. To prevent the proposed convolution from being degenerated, the authors define the volumetric receptive field to be the volume in the input space that affects CNN’s output. The volumetric RF condition requires that a properly decomposed separable convolution maintains the same volumetric RF as the original convolution before decomposition.

**Strengths**

- Contributions clearly stated and validated.
- Comprehensive mathematical proof seems reasonable.
- Ablation experimental results to show the effectiveness of their method.

**Weaknesses**

- The idea seems a little bit incremental in that it is a straightforward combination of group convolution and depthwise separable convolution.
- Some crucial ablation study/experimental results are missing.

**Clarity**

- The paper is well organized and easy to read.

**Comments**

- To show the efficiency of the proposed convolution, the authors are suggested to present a runtime comparison with existing separable convolutions. I understand the FLOPS results have shown the efficiency of the proposed optimal separable convolution, but it's still necessary to show the actual running times.
- The volumetric receptive field (RF) condition seems reasonable. However, the authors don't provide any ablation study on the volumetric RF condition. Say, what if removing this condition?
- The author obtains the optimal value of three sets of parameters like the number of groups, the internal channel size, and the internal kernel size.  It’s better to present some ablation studies on these parameters. These comparisons will be more powerful to demonstrate that the obtained optimal values can lead to the best result.
- This work seems an incremental version of group convolution and depthwise separable convolution. Could the authors give more discussion on this concern?
- The authors only provide experimental results on the classification task. It's interesting to see if this proposed convolution can be applied to other tasks, like segmentation.

**After Rebuttal**
I appreciate that the authors partly answered my questions and conducted experiments to show the runtime. After reading through their rebuttal and the other reviews, I will keep my original rating. 
review_title: A novel efficient separable convolution operator

Meta-review: This paper introduces a novel convolution-like operator called "optimal separable convolution" which is based on minimizing number of operations given a fixed receptive field.  Authors provide further empirical results to show the effectiveness of their proposed operator.

Overall, this is a very interesting work. There is a consensus among reviewers that this work is well-motivated, novel and principled. However, reviewers have pointed to several issues that makes this a borderline paper and consequently none of the reviewers were willing to argue for the acceptance. After reading the paper, reviewers' comments and authors' response, I would summarize the main areas of improvements as follows:

1- The "optimal separable convolution" is derived theoretically using "volumetric receptive field condition". However, this condition is not discussed and motivated enough in the paper. For example, different parametrization with the same volumetric receptive field could impose very different expressive power or implicit bias. Why is this not important? Adding discussions/experiments to motivate this condition would improve the paper.

2- The derivations in Sections 2.3 and 2.4 are not well-presented and are hard to follow. I suggest authors to use the convention of having a formal Theorem statement followed by the proof. This is important since one of the main contributions of the paper is a principled derivation.

3- All reviewers were concerned with the wall-clock time. Authors responded that theoretical #FLOPs is more important because wall-clock time is hardware dependent. However, authors reported the wall-clock time using CPUs. I understand that wall-clock time is hardware dependent but that only means algorithms that can have better wall-clock time on the current hardware are more likely to be useful because there is no guarantee that the hardware would be adjusted based on one algorithm especially if the promised improvement is not large enough. Therefore, I think reporting Wall-clock time on GPUs is important which was not done here.

4- Even though authors mention several operators in Table 1, they only compare against depth separable conv in the experiments. Even based on FLOPs, the current empirical results are not very promising. For example: 

a) The gap between o-ResNet (the proposed method) and d-ResNet is not significant in Fig 3. In particular when #FLOPs is low, d-ResNet and o-ResNet have similar performance. 

b) In Tables 2 and 4, o-ResNet shows small improvements but uses more FLOPs. Even if authors can't exactly match #FLOPs, they should make sure that the proposed method uses less FLOPs than others not the other way around.

c) In Table 3, authors only compare to ResNet and d-ResNet is removed.

Considering the above issues, I think the paper is marginally below acceptance threshold. Given the novelty of the work, I want to encourage the authors to improve the paper by taking Reviewers' comments into account and resubmit their work.


Decision: Reject