Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a novel variant of convolutional layer (log-polar space convolution LPSC) in neural networks, wherein the convolution kernel is not rectangular, but rather elliptical with different sized regions, which are pooled. It also describes a log-polar space pooling for convenient implementation. Finally, it applies the proposed method to various neural network architectures and tasks thereof.
strengths_and_weaknesses: Strengths:
--------------

Strength 1: The idea of log-polar convolution makes intuitively sense since the rectangular pixel quantization is more an artifact of the capture and storage systems rather than a feature of nature.

Strength 2: A practical pooling method for easily incorporating the method into existing SW, HW and networks.

Weaknesses:
------------------

Weakness 1: There are other non-typical kernel shapes that have been proposed in the literature (some are cited in the paper). The authors should compare their method to the best of the previously published methods.

Weakness 2: The method does not seem to bring a huge gain, but result into slower training and inference and additional hyperparameters. While this should still be ok for data-limited setup because of the potential regularization effect of the proposed system, it is not clear whether it is better to use the proposed method or just a somewhat larger CNN to achieve similar gains. Please study the regularization / generalization effect of the proposed method further.

Weakness 3: The motivation of the log-polar space convolution would be better if natural image statistics were used to justify it. Now it stands out as rather ad-hoc method.

After the rebuttal
---------------------
I have reviewed the author feedback and the they have done a good job in clarifying their work. I have reflected this in the score.

questions: Question 1: Although only a bit earlier work, the authors should discuss the relation to this paper : He, Kun, et al. "Integrating Large Circular Kernels into CNNs through Neural Architecture Search." arXiv preprint arXiv:2107.02451 (2021). 
That work also cites other papers that should be looked at (some of these are already cited by the current paper): “... the deformable convolution (Dai et al., 2017; Zhu et al., 2019) … Similarly, the deformable kernel (Gao et al., 2020) …including quasi-hexagonal convolution (Sun et al., 2016), blind-spot convolution (Krull et al., 2019), asymmetric convolution (Ding et al., 2019), etc.”

Question 2: Have the authors studied how much performance comes from the polar coordinates and how much from the growing size of areas that are pooled (further areas are pooled more). Could the pooling be done in the normal rectangular grid and how much would be gained by that.

limitations: The authors properly note the drawbacks of additional hyperparameters and memory overhead. For other limitations, see above for weaknesses and questions in this review.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a new convolution operator. Instead of using a rectangle convolution kernel following the grid structure of the input data, the authors propose to define convolution kernels in a log-polar space. Specifically, the kernel weight depends on the distance and direction wrt to the center, and the density of weights is inversely proportional to the distance to the center. Because the weights are not defined on the same regular grid as the input data, a pooling operation is performed before applying the convolution kernel. The main benefit of the proposed approach is that less parameters are needed when the size of the receptive field increases, based on the insight that pixels adjacent to the center should have more contributions to the output. Empirical results on multiple tasks and models show that the proposed method outperforms models using only standard convolution kernels. 
strengths_and_weaknesses: * Strengths
  * The key idea is intuitive and well motivated
  * The proposed approach is generic and is applicable to most existing CNNs, and it is easy to implement
  * The results suggest that the proposed convolution operator consistently perform better than standard convolution
* Weakness
  * The method requires additional computation and parameter during inference time, and the experiment does not really perform apple to apple comparisons
  * The accuracy improvement is not significant consider that it introduces additional computation and memory overhead
  * Using the same setup does not imply fair comparison. One should optimize each model independently for a fair comparison.
  * The proposed method introduces additional meta-parameters, which are determined by the accuracy on the test set according to L281 and leads to unfair comparison

* After rebuttal
  * The concerns regarding the accuracy are properly addressed, i.e. training setup and meta-parameters
  * The claim for lower computational cost is still not clearly explained. L222 assumes standard convolution and LPSC has comparable M and N. But in practice, most standard convolution has M=N=1, while LPSC has M and N > 1. Therefore, it's not clear why the flops is lower when 3x3 kernels are used.
questions: 1. Given that the proposed method introduces additional computation and parameters, the overhead should be considered in the evaluation, e.g. compare overhead vs accuracy instead of single point accuracy
2. Validation should be use to determine the meta-parameters for a fair comparison

limitations: 1. There doesn't seem to be any obvious negative social impact for this work.
2. The authors describe some of the limitations of the proposed method, although some additional information may help understanding the limitations, e.g. the exact memory overhead.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: # Summary
In this paper, an elliptic convolution kernel is proposed, and its local receptive field is adaptively divided into different regions according to the relative direction and logarithmic distance

strengths_and_weaknesses: # Strengths
1.The proposed method is very novel
# Weaknesses
1.The comparison objects selected in the comparison experiment are very old, and the performance has not been greatly improved.

2.Poor effect in deep network model.

3.Insufficient experiments in image segmentation.
questions: 1.In Table 1, the accuracy of using alexnet in cifar10 and cifar100 is only about 77 and 45 respectively. Although it is improved compared with other convolution cores, it is also difficult to be convincing. At the same time, the improvement on vgg19 and resnet20 is too subtle to verify the effectiveness of the convolution kernel.More experiments are needed to verify the validity of the statements in the paper.

2.Why choose resnet18 instead of resnet50 for experiments on Imagenet.

3.In Table 2 and table 3,Why not use dilated convolution with your method.

4.As mentioned in Section 4.1 of the paper, the advantages of LPSC are weakened for deeper models. Please explain why.

5.More experiments are needed to prove that LPSC can obtain more effect of improving receptive field than dilated convolution. At the same time, improving receptive field is mainly used in semantic segmentation, so more experiments on semantic segmentation are needed.
limitations: The idea is very good, but I need better experimental results to support it.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
In conventional CNNs, large effective receptive fields (ERF) are generally achieved through a cascading of multiple layers of convolutions and pooling operations. The authors note that this requirement may have an impact on the model performance; it may cause vanishing gradients and moreover may not actually have the desired effect of an increased ERF. The effective receptive field theory states that a more effective way of increasing ERF is increasing the kernel size. To this end, the authors explore a parameter efficient way of increasing through defining their kernels in log-polar space. The authors carefully describe and motivate their method and its relation to conventional convolutions, and experimentally show the validity of their method on a range of experiments. Furthermore, the authors show a compute-efficient implementation using conventional convolution operations.
strengths_and_weaknesses: The authors give a very clear overview of the relevance and motivation behind their method, by first highlighting limitations of current approaches to larger ERFs and from there introducing their approach to address these issues. As far as I know, their work is highly original.

Section 3.1 contains a comprehensive overview of the method, although I get the sense that an illustration visually explaining your method (i.e. show how R relates to the kernel size, how $L_{\theta}, L_r$ change the kernel), as you did for initial angle and eccentricity in fig 2, could be very helpful in getting the specifics across. In general, the paper is written in a clear manner, but the number of inline equations and parameters make it somewhat hard to read at points.

The choice of experiments is motivated well; the goal of the authors is to make a direct comparison between conventional convolutional layers and the introduced log-polar space convolutions. The experiments themselves seem to be chosen fairly for this goal, and decidedly show the improvement of LPSCs over conventional convolutions.

I appreciate that the authors chose to investigate how their framework could be implemented through highly optimised conventional convolutions. This allows for a much more direct comparison with conventional convolutions.

Misc:  Under fig 2a and in line 171, 187 “is slide” -> “is slid”.

questions: 1. Do you have an intuition for when you would want to use ellipsoid LPSCs? Are there any specific use-cases that come to mind?
2. Your method actually gives a definition for kernel values over the entire continuous input space. Does this mean your method could be readily applied to irregularly sampled data? Are there any specifics of your method that would inhibit you from doing this? Have you attempted this?
3. The authors motivate their log-polar space kernel definition by saying that in natural images often correlation between local pixels is higher than correlation between more distant pixels and therefore we can assign the same parameters to larger patches of pixels further away. I don’t think this argument necessarily makes sense; indeed natural images are locally correlated, but I believe this only says that local pixels should be assigned larger weights values, not that larger patches of pixels further away should be assigned identical weight values. Indeed, your method allows for larger receptive field sizes, but because of the increased weight sharing further from the center point, this comes at the trade-off of lower resolution information at higher distances correct? Could you touch upon this?
4. In this same line, I would like to see the authors more explicitly expand on what they think ultimately results in the performance increase of their method over conventional CNNs? Do you think it is a result of the increased receptive field, the way in which you treat local and nonlocal information at different resolutions, both?
5. Also, can you touch upon some use-cases in which this assumption of locality necessarily does not hold, but for which conventional CNNs are still used in practice? For example, how would your method perform on speech speech signals, which are generally sampled at very high rates and may exhibit very non-local patterns?

**Post-rebuttal response**
I would like to thank the authors for their extensive responses to my questions, and the questions of other reviewers. I appreciate the fact that you included a discussion for multiple of the concerns and questions raised by me and other reviewers in your manuscript, I feel this improves the quality of the work.

I noticed that my concern regarding the motivation for your specific approach to weight sharing in convolution kernels was shared by other reviewers as well. I think you greatly improved your motivation by including an analysis on pixel statistics.

The additional experiments and comparisons made by the authors in their revision strengthen this submission further, and show the relevance of this approach by showing it performs even when compared to more recent approaches.

The authors additionally addressed a number of limitations of their work in their revision, which make the work more transparent.

In light of these improvements, I am slightly raising my recommendation. I thank the authors for an interesting submission!

limitations: Authors briefly discuss the limitations of their work; it adds hyperparameters and isn’t very memory efficient. What I would like the authors to touch upon are limitations which may be present in their method inherently, as a result of using the log-polar space to define convolution kernels (see also my questions above).
review_title: null

Meta-review: metareview: Thank you for the submission. After reading the paper (twice) and all reviews, my summary could be found below.

Overall, the reviewers agree that the paper has some positive aspects, namely:
+ The idea of log-polar convolution makes intuitive sense since the rectangular pixel quantization is more an artifact of the capture and storage systems rather than a feature of nature.
+ A practical pooling method for easily incorporating the method into existing SW, HW, and networks.
+ The proposed approach is generic and applicable to most existing CNNs, and it is easy to implement.
+ The choice of experiments is motivated well.

However, there is mainly one deficiency reported:
- The method seems to bring some gain in performance, but it also results in slower training and inference, and it requires additional hyperparameters.

I would like to thank the authors for their rebuttal, it helped a lot to improve the paper. I want to also thank the reviewers for their discussions. After the rebuttal, it seems that most concerns have been solved and even though the average score is borderline accept/weak accept, I believe that the paper could be accepted.

confidence: Less certain
Decision: Accept