Log-Polar Space Convolution
Authors: Bing Su
Ji-Rong Wen
Venue: ICLR 2022 Submitted (2022)

Abstract: Convolutional neural networks use regular quadrilateral convolution kernels to extract features. Since the number of parameters increases quadratically with the size of the convolution kernel, many popular models use small convolution kernels, resulting in small local receptive fields in lower layers. This paper proposes a novel log-polar space convolution (LPSC) method, where the convolution kernel is elliptical and adaptively divides its local receptive field into different regions according to the relative directions and logarithmic distances. The local receptive field grows exponentially with the number of distance levels. Therefore, the proposed LPSC not only naturally encodes local spatial structures, but also greatly increases the single-layer receptive field while maintaining the number of parameters. We show that LPSC can be implemented with conventional convolution via log-polar space pooling and can be applied in any network architecture to replace conventional convolutions. Experiments on different tasks and datasets demonstrate the effectiveness of the proposed LPSC.

Under review as a conference paper at ICLR 2022
LOG-POLAR SPACE CONVOLUTION LAYERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Convolutional neural networks use regular quadrilateral convolution kernels to extract features. Since the number of parameters increases quadratically with the size the convolution kernel, many popular models use small convolution ker- nels, resulting in small local receptive fields in lower layers. This paper proposes a novel log-polar space convolution (LPSC) layer, where the convolution kernel is elliptical and adaptively divides its local receptive field into different regions according to the relative directions and logarithmic distances. The local receptive field grows exponentially with the number of distance levels. Therefore, the pro- posed LPSC not only naturally encodes local spatial structures, but also grearly increases the single-layer receptive field while maintaining the number of param- eters. We show that LPSC can be implemented with conventional convolution via log-polar space pooling and can be applied in any network architecture to replace conventional convolutions. Experiments different tasks and datasets demon- strate the effectiveness of the proposed LPSC.
INTRODUCTION
Convolutional neural networks (LeCun et al. 1998 Krizhevsky al.] 2012) have achieved great success in the field of computer vision. The size of the convolution kernel determines the locally weighted range of the image or feature map, which called the local receptive field (LRF). In many computer vision tasks such as image classification and intensive prediction, larger LRF is generally desired to capture the dependencies between long- long-distance spatial positions and a wide range of context information. Simply increasing the size of the convolution kernel is not plausible because the number of parameters increases quadratically with the size.
In practice, commonly used techniques to obtain larger receptive fields include replacing a single- layer large convolution kernel with multi-layer small convolution kernels, adding pooling layers, and using dilated convolutions (Holschneider et al. 1990 Yu & Koltun 2015). Increasing the number of convolutional layers may cause vanishing gradients and make training more difficult. The pooling process often causes information loss. Dilated convolution kernels pixels in the LRF are involved in convolution calculation. The skipped pixels are regularly selected. With the same number of parameters, the larger the LRF, the more pixels are skipped, which may miss some details and cause discontinuity of information.
not continuous since not all
Moreover, conventional and dilated convolutions use regular square kernels. Each position as- signed different weight within the LRF. All positions are equally treated regardless of the size of the kernel. However, intuitively, the correlation between neighboring pixels and the center pixel is usually higher, while the farther the pixel, the smaller the impact on the center pixel. The effects of two adjacent pixels that are far away from the center are usually similar, thus they can share the same parameter rather than be assigned different weights separately. As shown in red in Fig. I(a] according to the configuration of surrounding regions, it can be inferred that the center position located the upper edge of the nose. Pixels in the same upper-left outer half-fan sah--ee region show that the far upper left of the center point is white fur, but there little difference in the effects of two specific fur points.
In this paper, we propose a novel log-pott pog-polar space convolution (LPSC) method. The shape of the LPSC kernel is not regular square, but ellipse. Parameters of the kernel are not evenly distributed in the LRF, but are assigned in the log polar coordinate space. As shown in Fig. [[b] the LPSC kernel divides the LRF into different regions, where regions become larger with the increase of the

---

Under review as a conference paper at ICLR 2022
We
Wn
Wis
W
Wir
W
Win
Wip
Wa
WN
W
(A)
(h)
Figure 1: (a) At different locations of an image, local contextual pixels can be divided into different regions according to their relative distances and directions in the log-polar space. For each location, pixels falling in the same region are generally similar and can share the same weight. (b) The LPSCC kernel. For this example, I 3, Lo = 6, g 2, thus there are only 18+1 parameters. The LRF can be arbitrarily large.
distance to the center. Pixels that fall into the same region share the same weight. In this way, LPSC can increase the LRF exponentially without increasing the number of parameters. Besides, LPSC naturally imposes contextual structure on the local neighboring distribution.
The main contributions of this paper include: We propose a new convolution method where the kernel lies in the log-polar space capture the structured context information and greatly cxpand the LRF without increasing the number of parameters. 2. We propose log-polar space pooling to up-sample the feature map, by which conventional convolution can be conveniently used to achieve LPSC. 3. We apply LPSC to replace the conventional and dilated convolution in different network architectures including AlexNet, VGGNet, ResNet, DeepLabv3+, and CE-Net. We demonstrate the effectiveness of LPSC through empirical evaluations on different tasks and datasets.
RELATED WORK
Context pooling. Our method highly motivated by shape context (Belongie et al., 2001 2002).
Centered at reference point, all other points are divided into bins that are uniformly distributed in the log-polar space. The histogram among these bins is used as the descriptor. Geometric blur (Berg & Malik, 2001) sparsely samples and aggregates a blurred signal in the log-polar space. Pyramid context (Barron et al., 2013) pools log-spaced context points at multiple scales. Different from these methods, we design kernel in the log-polar space for convolution, each region is assigned a weight to aggregate information from the bins. We incorporate the kernel into deep neural networks.
Methods increase LRFs. In Simonyan & Zisserman (014) and He al. (206)), it is found that imposing a regularization on large convolution kernels is equivalent to the superposition of multiple convolution layers with smaller kernels. Based on this observation, many state-of-the-ard network architectures use multi-layer small kernels. However, deeper layers may cause vanishing gradients, making the network more difficult train. Moreover, according Luo et al. (016), the effective receptive field (ERF) is proportional to the square root of the depth and proportional to the kernel size. Thus is easier achieve large ERF by increasing the kernel size than by adding layers. We provide a way to increase the LRF without increasing either the number of layers or the number of parameters. In cases where large input or LRF required but very deep networks are not allowed restricted by resources, method may be applied to construct lightweight model.
In Holschneider et al. (1990). atrous (or dilated) convolution increases the LRF by inserting holes (zeros) between parameters in the kernel, where the interval is determined by dilation rate. Dilated convolution has been applied in different tasks (Dai et al. 2016 Yu & Koltun 2015 Chen et al. 2017a Sevilla- Lara et al. 2016 Yang al. 2018 Chen al. (01 8a). In Zhang et al. (2017a) and Zhang et al. (2019), scale-adaptive convolution learns adaptive dilation rate with a scale regression layer. Due to the insertion of holes, not all pixels in the LRF used for calculating the output.
2

---

Under review as a conference paper at ICLR 2022
In Wang al. (2018) and Wu et al. (2019), this problem is alleviated by hybrid dilated convolution and Kronecker convolution that uses the Kronecker product to share parameters.
Other convolution methods. Fractionally strided convolution Zeiler et al. 2010; 2011) up-samples the input by padding. In Jaderberg et al. (005)), a spatial transformer transtorms the regular spatial grid into sampling grid. Active convolution (Jeon & Kim (017) learns the shape of convolution by introducing the convolution unit with position parameters. Deformable convolution (Dai et al.) 2017) learns additional offsets to augment the sampling locations, thereby adaptively changing the LRF into a polygon. For active and deformable convolutions, the adapted LRF contains holes, the positions and offsets are learned through additional convolutions, which increases the parameters. Deformable kernels (Gao et al.] 2019)) resample the original kernel space and adapt it to the defor- mation of objects. The offsets for kernel positions also need to be learned.
Group convolution (Krizhevsky et al., 2012 Zhang et al. 2017b 2018) and separable convolu- tion (Chollet, 2017) do not increase the LRF of kernels. Octave convolution (Chen et al.) 2019) de- composes the feature map into high-frequene is performed in Peng et al. (2019) and [i et al. (2020). In RRaachandran et al. (2019) and Cordon- nier et al. (019)) stand-alone self-attention is used to replace convolution. The filter in the attention module also lies in a regular and square grid. Esteves et al. (2018), the polar transformer network (PTN) generates a log-polar representation of the input by differentiable sampling and interpolation techniques. The polar transform is only applied to a single predicted origin location. In contrast, LPSC performs log-polar pooling via binning and can be applied at any location.
and low-frequency features. Multi-scale convolution
Differences. For dilated and other advanced convolutions, the kernel is still performed in a regular grid and all parameters are treated equally. Regardless of the distance from the center, the interval or the sharing range of a parameter is the same among different positions. In contrast, the proposed LPSC expands the LRF in the log-polar space, where near and far regions are distinguished in parameter sharing. The farther away from the center, the larger the range of parameter sharing.
3 LOG-POLAR SPACE CONVOLUTION
Let X E R'XXXXCC be the imput image or feature map, where H. W, and C are the height, width, and number of channels of X, respectively. W R(M++Dx(2-II)C is conventional convolu-
tion kernel with a size of (2.M- + 1) (2.V + 1). The central parameter of W is indexed by (0, O), parameters of W lie in regular grid {( [(-M, -N), (-M, + 1),............... ... (M I, N), (M,NN); The convolutionn operation is performed in the 2D spatial domain across the channels. For spatial location (i,j), the output of the conventional convolution is calculated as
M
N
(X W)
Σ
Σ
(X m,j + nn) W n)) + +b,
(1)
where b is the bias. Strictly, Eq. ①1 actually performs cross-correlation. For convolution, W needs to be rotated 180 degrees. However, since we can view the learned W as the rotated kernel, we follow the common practice of CNN formulate convolution into Eq. (I). Parameters of the kernel are uniformly distributed in the regular grid, thus each pixel of X falling into the field is weighted by separate parameter, i.e., all positions are equally treated. However, pixels that have different distances and directions from the center may have different impacts, e.g., pixels adjacent to the center should have larger contributions to the output. Pixels in the input image usually change gently, adjacent pixels far away from the center often have similar impacts on the center. Based on these intuitions, we design a convolution kernel with a special structure, namely Log-Polar Space Convolution (LPSC) kernel, to express wide range of contextual configurations.
3.1 LPSC KERNEL
As shown in Fig. I(b), the proposed LPSC kernel lies in the log-polar space and is shaped by the size 2R + 1, the number of distance levels Ly, the number of direction levels Le. and the growth rate g. The LRF of the kernel is the area of the outermost circle whose radius is R. It is uniformly divided into I,, x Lo regions in the log-polar space. Specifically, the log radius is uniformly divided into L, levels, i.e.,
(Rg+)) log( R2) log (Ri) log Mog(R(1)) R log(g);
3

---

Under review as a conference paper at ICLR 2022
(a)
(b)
Figure 3: (a) At each position, log-polar space pooling generates a 2L, Le/2 matrix using the pre-computed mask. (b) To perform LPSC on the original input (left), log-polar pooling gener- the feature map. (b) The shape and inclination of ates a matrix for each pixel, resulting in an up- the LPSC kernel can be changed.
(a)
(b)
Figure 2: (a) The LPSC kernel is slide through
sampled map (middle), conventional convolution with kernel size and stride equaling (2hr, La is applied to this map (right).
where Ra, 1,.... Ly is the radius of the 1-th level and the growth rate y hyperparameter controlling the expansion speed. When the center of the kernel is located at position (Ch, Cw.), all pixels of X in the range of A [cs- R, ch R] x [cwu R2 Cw + R] are divided into L, levels according to their relative squared distances to the center position. The position (i. E A belongs to the (-th distance level if R S di, Rd, where dr,
) ij cw) ) From Eq. ( we have R 1 R1. When the innermost radius R is fixed, the LRF grows exponentially with the increase of L,. The LRF is determined by R which can be set arbitrarily. Given RL, = R² and g. we calculate R1 ma R2 /g/.-1). We use R = / Rt. as a hyperparameter instead of R1, which is more flexible. Since we use the squared distance, we impose minimum value of to ensure that all n-neiigborrhoo pixels fall into the 1-st level.
All positions in the range of A are also uniformly divided into Le levels according to their relative directions from the center. The position (i.j) belongs to the m-th level if 2 - 1)) S Or < 2mm Lgg where 0z,5 is the counterclockwise angle from the vector (0, 1) to the vector (i-ch.3-Ce Combining the distance levels and the direction levels, the LRF is divided into L, Lg regions.
The LPSC kernel assigns a parameter to each region. All pixels of X falling into the same region share the same parameter. For the region with the (-th distance level and m-th direction level, the assigned parameter is denoted by Wt,m- The areas of regions increase with 7, the farther away from the center, the larger the area, the more pixels sharing parameters. Because the center position of the kernel is important and forms the basis of regions, we assign an additional separate parameter wo,0 for the center pixel. A conventional kernel with size of (2R + 1) x (2R + 1) has (2R + 1)a parameters, while a LPSC kernel only has L, x Lg + parameters no matter how large R is. When R ranges from to 9, single conventional kernel has 25 361 parameters. In this range, it is sufficient to set L, to 2 or 3 and set Lo to 6 or 8, so an LPSC kernel only has 13 to 25 parameters.
Let Nr,,. denote the number of pixels falling into the region bin(l, m) with the (-th distance level and the m-th direction level. In faraway regions with large 1, Nt., the impacts of pixels in them should be weakened. Therefore, we regularize the weight W).o of each region by Nr. Wt.,/ / As result, the LPSC kernel aggregates finer information from pixels nearing the center and is less sensitive to those of pixels farther away. Similar to conventional convolution, the LPSC kernel slide along the input feature map X with pre-defined stride to perform convolution, shown in Fig. ((a) When the kernel is located at a spatial location (i, j). the output response is calculated as
L La
(X ()(t.) (0,00) X ((i.))
Σ W(, =1
Σ
(N
Σ X(u,v)) (3)
u.rEbin
For the LPSC kernel, the shape of its LRF is not necessarily a standard circle, but can be an oblique ellipse. As shown in Fig. 2(b) two additional typer-parameters are introduced: the initial angle and the eccentricity of the ellipse e. When dividing the regions, the distances are calculated accord- ing to the squared ellipse distance and the initial angle is added to the calculated directions. In this way, the LPSC kernel can better fit objects with different rotations and scales. In our experiments, only evaluate the standard circular LRF by setting a O and = h/w
4

---

Under review as a conference paper at ICLR 2022
3.2 LPSC VIA LOG-POLAR SPACE POOLING
Due to the special structure and parameter sharing, LPSC cannot be directly performed by popular deep learning frameworks. In this subsection, we show that LPSC can be readily implemented by conventional convolutions via log-polar space pooling to utilize efficient convolution modules.
Given the hyper-paramenn matrix I to indicate the region indexes of positions. The size of the mask I is (2R + 1) x (2R + 1).
R. Lr, Lo, and g of the proposed LPSC, we can pre-compute a mask
1, .Lg x L, in I indicates the region index of the corresponding position. indicates that the corresponding position does not fall into the LRF, since the region of the mask is the cireumscribed rectangle of the LRF. The mask is slide through the input feature map X with the same stride of the LPSC convolution. As shown in Fig. 3(b) when the mask is located at spatial location (i. pixels of X in the range are divided into regions indicated by the mask. All pixels in the same region are encoded into single pixel by mean pooling. We re-arrange the pooled pixels of different regions into a matrix of 2L, X L 22 to preserve their relative spatial positions, as shown in Fig. 3(a) In this way, given H' x W" convolution locations (H' = H and W = W if the stride is I with padding), the spatial size of the output map Xp p after log-polar space pooling equals 2H' L, X Lo/
We perform conventional convolution with C" output channels on the output map X, without padding. The size of the conventional convolution kernel is set to (2L,, Lg /2) and the stride is also Lo/ /2). The output feature map Y, has a size of H' x W x C This equivalent to performing the second term in Eq. (3) To model the first term, we use separate 1 x conventional convolution with the same C' channels on the original X. The stride the same as the log log-polar space pooling. The output feature map Y. contains the convolution responses of the center pixels. We add this separate center pixel convolution output Y, to the contextual convolution output Y,, Y. + Yp serves the output feature map of the proposed LPSC.
3.3 INCORPORATING LPSC INTO DIFFERENT CNNS
LPSC can be integrated into different CNN architectures. A straightforward way is to replace all con- ventional convolution kernels with LPSC kernels in part of convolution layers. For plain CNN ar- chitectures such as AlexNet (Krizhevsky et al. 2012) and VGGNet (Simonyan & Zisserman, 2014). simply perform this strategy in lower layers increase the LRFs. However, some network archi- tectures such as ResNet (He et al. 2016) are constituted of specifically designed blocks. In ResNet, either the bottleneck or the basicblock structure only contains 3 x 3 and x 1 convolutions. Due to the difference in the local receptive field, the information captured by these small convolutions and LPSC may be different. In order better incorporate these two types of information, we propose cross convolution strategy as an alternative to replacing all convolutions in each layer of the block. Specifically, we a ratio p. For each of several consecutive layers, we replace p% of all convolu- tion kernels to LPSC kernels, while the remaining (100- - p)% of conventional kernels remain the same. In this way, each convolution kernel in the next layer, whether it is a conventional or an LPSC kernel, perceives the outputs generated by both the conventional and LPSC kernels of the previous layer. We denote this cross-convolttion strategy by LPSC-CC. Details on how to incorporate L.PSCs depend on the CNN architecture and will be presented in Section
3.4 DISCUSSIONS
Complexity. For a (2M+1) x +1) C kernel, conventional convolution involves (2. + 1) x (2N+ +1 x C' multiplications and +1M+1)x x (2N +1) x C' additions. LPSC with L, distance levels and Lg direction levels only involves * L, x Lg C multiplications. (2M + 1) (2N + 1) x C additions, and (2M + 1) x (2N + 1) lookups. The complexity of pre-computing the mask for lookup O(R),, which only needs to be calculated once when initialing the layer. Typically, L, 2, Lg 6, LPSC only executes 24C' multiplications for any size. However, even for small (2M + 1) x (2N + 1) = 5 x 5 kernel, conventional convolution executes 25C' multiplications; for a 19* x 9 kernel, multiplications increase to 81C.
With the special log-polar structure, the LPSC kernel naturally encodes the local spatial distribution of pixels w.r.t. the center and puts more attention to those adjacent pix- els. Pixels with similar relative distances and directions share the same parameter, which not only reduces the number of parameters, but also makes the filter more robust and compact. Due to the
Structural benefits.
5

---

Under review as a conference paper at ICLR 2022
logarithm effect, when located at different objects, small objects are relatively enlarged, while large objects are relatively reduced. Therefore, LPSC is less sensitive to the size objects.
Advantages of log-polar space pooling. In log-polar space pooling, use mean pooling to achieve the regularization for regions. However, other pooling methods are also allowed. For ex- ample, if sum pooling is used, it is equivalent remove the regularization. If max pooling is used, LPSC can be viewed as a special dilated convolution with irregular and data-driven holes.
Relation with effective receptive field (Luo et al., 2016). In Luo al. (2016), it is found that the effective receptive field only occupies a fraction of the full theoretical receptive field. Therefore, convolutions with large LRF required. It is also found that not all pixels in the LRF contribute equally, where the impacts of pixels near the center are much larger. The LPSC kernel follows this spirit to treat pixels near the center finely and increase the LRF exponentially.
Relation with advanced convolution methods. For the proposed LPSC, parameters distribute in the log-polar space. Most advanced convolution methods discussed in Section P not change the regular sampling grid and focus on different aspects with LPSC. Therefore, these advances may be propagated to LPSC as well. For example, group and separable convolution (Choller) 2017: Zhang et al. 2018) separates the channel dimension into groups and hence can be combined with the LPSC kernel in spatial dimensions. Graph convolution can also take advantage of LPSC as long as the relative distance and direction between nodes are defined.
Drawbacks. LPSC has two main drawbacks. (1) It introduces three additional hyper-parameters: h,, Lo. and g. However, in practice, their selectable ranges are quite limited. Generally, to make the 8-neighborhoods of the center pixel have finer and non-redundant regional resolution, L, is set to 2 or 3. Le is set to 6 8, and 9 is set to 2 or 3. (2) Its implementation via log-polar space pooling incurs large memory overhead. The space complexity of the upsam- pled feature map X, is O(H'W' LT, For single layer, the space complexity of LPSC is O'H'W'L, Lg - LCC + H'W'
4 EXPERIMENTS
We empirically study the effects and potential of the proposed LPSC on two tasks: image classifica- tion and semantic segmentation.
4.1 IMAGE CLASSIFICATION EXPERIMENTS
For image classification, we evaluate the behaviors of LPSC integrated with different CNN archi- tectures three datasets: CIFAR- 10, CIFAR-100 (Krizhevsky 2009), and ImageNet (Russakovsky ct al. 2015). We plug LPSC into three typical CNN architectures, including AlexNet (Krizhevsky al. 2012), VG (Simonyan & Zisserman| 2014), and ResNer20 (He et al. (016), by placing a part of the conventional convolution layers. We use the Pytorch (Paszke et al., 2019) implementation of these architectures as our baseline. For the AlexNet. there are 5 convolution layers each followed by a ReLU activation layer. The sizes of the convolution kernels are 11 x 11, x 5, x 3, x 3. and 3 3, respectively. For the VGG19 Net, there are sixteen convolution layers. The kernel size for all convolution layers is 3 x 3. For the ResNet-20, there are 9 basic blocks. Each block contains two 3 X 3 convolution layers. A blocks. When the conventional convolutions in layer or block are replaced by LPSCs, the number of kernels and the size of the output feature map remain the same as the original convolution layer.
convolution layer applied before all
To make a fair comparison, all experimental setup and details including the learning rate, batch size, number of filters per layer, hyper-parametert for the optimizer (c.g.. 7, momentum, weight decay) remain exactly the same as in the baseline. We did not tune any of these setups for LPSC. Therefore, the differences in performances only come from the changes in convolution layers. The numbers of parameters are computed on the CIFAR-10 dataset. Top-1 accuracy is used as the performance measure.
Results on the CIFAR10 and CIFAR100 dataset. We train the AlexNet, VGGNet-19, and ResNet- 20 with conventional convolution, dilation convolution, and LPSC five times by using different ran-

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of different convolution methods.
(a) AlexNet
Convolution
Ori
Dilation
LPSC
Params (M)
2.47
2.34
2.31
Acc. CIFAR-I0 (%)
77.43 (0.25)
75.42 (0.06)
78.44 (0.12)
Acc. CIFAR- (%)
43.98 (0.43) 44.43 (0.10)
47.43 (0.20)
(b) VGGNet-
Convolution
Oni
Dilation
LPSC
Params (M)
20.04
20.08
20.08
Acc. CIFAR-I0 10 (%)
93.54 (0.06)
93,46 (0.14)
93.92 (0.06)
Acc. CIFAR-100 (%)
72.41 (0.17) 73.03 (0.34)
73.13 (0.12)
(c) ResNet-20
Convolution
Ori
Dilation
LPSC
LPSC-CC
# Params (M)
0.27
0.27
0.27
0.27
Acc. CIFAR-10 1AARR-00CA (%)
91.66 (0.13)
91.44 (0.10)
91.81 (0.21)
92.01 (0.08)
Acc. CIFAR- 100 (%)
67.56 (0.27) 66.90 (0.25)
67.63 (0.27)
68.09 (0.27)
dom seeds for initialization, respectively, and compare the average accuracies and standard devia- tions. "Mean accuracy (standard deviation)" results are reported in Table We use LPSC in the first two convolution layers for AlexNet, the added first convolution before all blocks for VGGNett and in the first convolution layer before all residual blocks for ResNet-20. plyper-parameters of the LPSC kernels in different layers and networks are the same as the first three columns in Table d)) respectively. These choices are based on the ablation study as described in the appendix A. [and]]] For dilation convolution, we replace the conventional convolutions with dilation convolution in the same layers in the three architectures, respectively, where the kernel size and dilation rates set so that the LRF and number of parameters are comparable with LPSC. Specifically, for AlexNet, the kernel size and dilation rate are set to 5 and 2 in the first convolution layer, respectively, and 4 and 2 in the second convolution layer, respectively. For VGGNet-19, the kernel size and dilation rate are set to and in the added first convolution layer before all blocks, respectively. For ResNet-20, the kernel size and dilation rate are set to 4 and 3 in the first convolution layer before all residual blocks, respectively. These choices are based on the evaluations in Table 7 of the appendix A.. From Table [ we observe that LPSC outperforms dilation convolutions with comparable LRF and parameters. The standard deviations for LPSC are limited, which shows that LPSC is not particu- larly sensitive to initializations. In some cases, the worst results also exceed those of the original networks with conventional convolutions and dilation convolutions by a margin.
and A.2
We also evaluate the cross convolution strategy for ResNet-20. We apply LPSC-CC to the layer before all blocks and all 3 x 3 layers of the first block with a fixed p of 50. We observe that the cross convolution strategy further improves the performances.
Comparison of FLOPs. On the CIFARIO 10 dataset with AlexNet, the FLOPs (recorded by the fvcore toolbox ) of conventional convolution, dilated convolution, and LPSC are 14.95M, 24.71M, and 11.42M, respectively. LPSC has much lower FLOPs than other convolution methods.
Results on the ImageNet dataset. ImageNet (Russakovsky et al. 2015) contains 1.28 million train- ing images and 50k validation images from 1000 classes. We again use the Pytorch implementation"" of 18 as the baseline. We replace conventional convolution with LPSC in the first convolu- tion layer before all blocks of ResNet-18. Due to the limitation of computing resources, we reduced the batch size and learning rate by 4 times. Other hyper-parameters remain the same and we com- pare with the reported results in Tab. 2 LPSC slightly improves both top- 1 and top-5 accuracies of ResNet- 18 on this large-scale dataset.
7

---

Under review as a conference paper at ICLR 2022
Table 2: Results the ImageNet dataset.
Model
ResNet-18
ResNet- MeeNNct-18--RR 18-PPCC
#params(M)
11.69
11.69
Top-1 Acc (%)
69.91
70.09
Top-5 Acc (%)
89.22
89.26
Table 3: Results the VOC 2012 dataset. our reproduced results. Method
means that the results reported.
indicates
oAcc
mAcc
fAce
mloU
Deepl.abyy
0.701
Dev+Labv33+
0.711
DecpLabx3+
0,9230
0.8332 0.8388
0.8652
0.7144
DePPL
0.9273
0,8714
0.7260
4.2 SEGMENTATION ON PASCAL AND MEDICAL IMAGES
LPSC can also be applied to other vision tasks. On the PASCAL VOC 2012 dataset (Everingham
et al. 2010 2015) for general image semantic segmentation, we adopt the Pytorch implementatior of DeepLabv3+ (Chen et al., 2018b) with the MobileNet (Howard et al., (017)) backbone as the baseline. The training set is augmented by extra annotations provided in Hariharan et al. (201 Overall accuracy (oAcc), mean accuracy (mAcc), freqw accuracy (fAcc), and mean loU (mloU) the validation set are evaluated. In abb33, the atrous spatial pyramid pooling (ASPP) module probes multi-scale features by applying atrous/dilated convolutions with three different rates. For DeepLabb3+LPSC we replace the dilated convolution with the largest rate by LPSC in ASPP. The kernel size, Ly, Le, and g of LPSC are set 9, 2, 8, respectively. Comparisons with the reported and reproduced results are shown in Tab. 3 LPSC improves DeepLabv3+ by margin of 1.1% on mloU. All hyper- param-psrameterss and setups such as the learning rate, batch size, etc, remain the same, so the performance gains are only attributed to the proposed LPSC.
On the DRIVE dataset (Staal al., 2004) for retinal vessel detection, we adopt CE-Net (Gu al. 2019) the baseline. Sensitivity (Sen), accuracy (Acc), and AUC are evaluated on the test set. The dense atrous convolution (DAC) block of CE-Net uses four cascade branches with increasing numbers of dilated convolutions. For CE-Net-LPSC-II, we replace the dilated convolutions with rates of and by LPSCs with sizes and 11 DAC, respectively, so that LPSCs have the same LRFs with dilated convolutions. L,, Lg. and g of LPSCs are set to 2, 6, respectively. For CE-Net- LPSC-2, we increase the kernel sizes of LPSCs to 9 and 15, respectively, to further increase LRFs. We accordingly more parameters by setting L., Le, and g to 3, 8, 1.5, respectively. Other hyper- parameters remain the same We run our models ten times and report the average performances. Comparisons with the reported results are shown in Tab. 4 Our LPSC achieves good generalization performances on medical image segmentation with limited training samples.
Table 4: Results on the DRIVE dataset (Staal et al. 2004).
indicates our reproduced results.
Method
Sen
Acc
AUC
HED (Xie &
2015)
0.7364
0,9434
0.9723
Azzopardi et al. Azzopardi el al.
2015
0,7655
0,9442
0.9614
Zhao et al. iZhao el al.
2013
0.7420
0,9540
0.8620
U-Net (Ronneberger
2015
0.7537
0,9531
0.9601
Deep Vessel (Fu et 2016)
0.7603
0,9523
0,9752
CB-Net (Gu al. (201)
0.8309
0,9545
0,9779
CE-Net [Gu al 2019
0.8266 (0.0106)
0.9550 (0.0009)
0.9782 (0.0008)
CE-Net-I.PSC-I
0.8300 (0.0079)
0.9552 (0.0011)
0.9782 (0.0008)
CE-Net-L.PSC-2
0.8312 (0.0075)
0.9548 (0.001
0.9784 (0.0007)
8

---

Under review as a conference paper at ICLR 2022
Figure 4: Visualization of the learned circular LPSC kernels without center convolution in the first
convolution layer of Alexnet the CIFAR-10 10 dataset.
namsaled
Horall
(a)
(b)
(c)
(d)
Figure 5: Estimated RFs with (a) conventional convolution and (b) LPSC. The normalized gradient
map with (c) conventional convolution and (d) LPSC sampled location.
4.3 VISUALIZATION
Visualization of the learned LPSC kernels. In Fig. 4 visualize the learned LPSC kernels in the first convolution layer of AlexNet on the CIFAR-TO dataset. The 11 x 11 LPSC kernels have 3 distance levels and 8 direction levels. In LPSC kernels, the closer to the center, the higher the regional resolution; the more outward, the larger the range for parameter sharing. We observe that the Iearned LPSC kernels capture some special local structures and contextual configuration. In some kernels, the weights for adjacent regions are continuous; some kernels are sensitive to specific directions, edges, colors, or local changes; in some other kernels, specific combinations of regions are highlighted. More visualizations are shown in the appendix A.3
Comparison of effective receptive field (ERF): Fig. 5 and (b) show the estimated RFs of Sim- pleVGGNet on the default example using conventional convolutions and LPSCCs in the first two layers by the gradient- brasdd RF estimation respectively. LPSC enlarges the estimated RFs from 14 x 14 to 22 x 22. The normalized gradient maps w.r.t. position of the output for estimating the RF using conventional convolutions and LPSCs are shown in Fig. 5(c) and (d). With LPSC, gradients can be poc-pprop more pixels of the input image.
CONCLUSION
In this paper, we have presented LPSC that naturally encodes the local contextual structures. LPSC distinguishes regions with different distance levels and direction levels, reduces the resolution of remote regions, and reduces the number of parameters by weight sharing for pixels in the same region. The LRF of LPSC increases exponentially with the number of distance levels. We impose regularization on the parameters and implement LPSC with conventional convolutions by log-polar space pooling and separable center pixel convolution. We analyze the interests and drawbacks of LPSC from different aspects. We empirically show the effectiveness of the proposed LPSC on five datasets for classification and segmentation tasks.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper proposes a new log-polar space convolution kernel and a method to achieve log-polar space convolution with conventional convolution. CNN has been applied wide range of appli- cations. The proposed kernel can be applied to any CNN backbone to increase the local receptive field and reduce the number of parameters. Because this work presents such a general convolution module, we did not see any particular foreseeable ethics consequence.
REPRODUCIBILITY STATEMENT
We specify all the details on architectures. hyper-parae parameters. experimental settings, etc in Section Appendix A. and Appendix A.2 Code attached in the supplementary file.
REFERENCES
George Azzopardi, Nicola Strisciuglio, Mario Vento, and Nicolai Petkov. Trainable cosfire filters for vessel delineation with application to retinal images. Medical image analysis, 19(1):46-57,
2015.
Jonathan T Barron, Mark D Biggin, Pablo Arbelaez, David W Knowles, Soile VE Keranen, and Jitendra Malik. Volumetric semantic segmentation using pyramid context features. In Proceedings of the IEEE international conference on computer vision, pp. 1444833455 2013.
Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape context: A new descriptor for shape match- ing and object recogmition. In Advances in neural information processing systems, pp. 831-837, 2001.
Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape matching and object recognition using shape contexts. IEEE transactions on pattern analysis and machine intelligence, 24(4):509-522, 2002.
Alexander C Berg and Jitendra Malik. Geometric blur for template matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2001.
Liang- Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4):
834-848, 2017a.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05887 2017b.
Liang-Chich Chen, Maxwell D. Collins, Yukun Zhu, George Papandreou, Barret Zoph. Florian Schroff, Hartwig Adam, and Jonathon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8724,, 2018a.
Liang-Chich Chich Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference Computer Vision, pp. 801-818. 2018b.
Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Mareus Rohrbach, Shuicheng Yan, and Jiashi Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In Proceedings of the IEEE international conference on computer vision, pp. 1435-3444, 2019.
François Chollet. Xception: Deep learning with depthwise separable convolutions. P Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Jean- Caan-Baptitte Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint urXiv:191 19t:11351....44 2019.
10

---

Under review as a conference paper at ICLR 2022
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region- based fully convo- lutional networks. In Advances in neural information processing systems, pp. 379-387, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Carlos Esteves, Christine Allen-Blanchette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. International Conference on Learning Representations, 2018.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-13)) -36,, 2015.
Huazhu Fu, Yanwu Xu, Stephen Lin, Damon Wing Kee Wong, and Jiang Liu. Deepvessel: Retinal vessel segmentation via deep learning and conditional random field. In International conference on medical image compuling and compuler-assister intervention, pp. 132-139. Springer, 2016.
Hang Gao, Xizhou Zhu, Steve Lin, and Jifeng Dai. Deformable kernels: Adapting effective receptive fields for object deformation. arXiv preprint (rXXiv:1910.2000 2019.
Zaiwang Gu, Jun Cheng, Huazhu Fu, Kang Zhou, Huaying Hao, Yitian Zhao, Tianyang Zhang, Shenghua Gao, and Jiang Liu. Ce-net: Context encoder network for 2d medical image segmenta- tion. IEEE transactions medical imaging, (0(10))2281-29999 2019.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In Proceedings of the IEEE international conference computer vision, pp. 991-998. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Matthias Holschneider, Richard algorithm for signal analysis with the help of the wavelet transform. In Wavelets, pp. 286-297. Springer, 1990.
Jean Morlet, and Ph Tchamitchian. A real-time
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Mareo Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 170v041704.4861, 2017.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances neural information processing systems, pp. 2017-2025, 2015.
Yunho Jeon and Junmo Kim. Active convolution: Learning the shape of convolution for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4201 4209, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227--2224, 1998.
Duo Li, Anbang Yao, and Qifeng Chen. Psconv: Squeezing feature pyramid into one compact poly-scale convolutional layer. arXiv preprint arXiv:2007.06111, 2020.

---

Under review as a conference paper at ICLR 2022
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in neural information processing sys- tems, pp. 11898-4906 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019.
Junran Peng, Ming Sun, Zhaoxiang Zhang. Tieniu Tan, and Junjie Yan. Pod: Practical object detec- tion with scale-sensitive network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9607-9616, 2019.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: 05909, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11533)211-252, 2015.
Laura Sevilla-Lara, Deqing Sun, Varun Jampani, and Michael J Black. Optical flow with semantic segmentation and localized layers. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3889-3898, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:
2014.
Joes Staal, Michael D Abråmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken. Ridge-based vessel segmentation in color images of the retina. IEEE transactions on medical imaging, 23(4):501-509, 2004.
Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cot- trell. Understanding convolution for semantic segmentation. In 2018 IEEE winter conference on applications of computer vision (WACV), pp- 1461-1460. IEEE, 2018.
Tianyi Wu, Sheng Tang, Rui Zhang, Juan Cao, and Jintao Li. Tree-structured kronecker convolu- tional network for semantic segmentation. In IEEE International Conference on Multimedia and Expo, pp. 940 945. IEEE, 2019.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1395-1403, 2015.
M. Yang, K. Yu, Z. Chi, Z. Li, and K. Yang. Denseaspp for semantic segmentation in street scenes.
In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv
preprint arXiv: 151.00712,. 2015.
Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2528- 2535. IEEE, 2010.
Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In Proceedings of the IEEE international conference on computer vision, pp. 2018-2025. IEEE, 2011.
Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. Scale-adaptive convolu- tions for scene parsing. In Proceedings of the IEEE international conference on computer vision, pp. 2031-2039, 2017a.
12

---

Under review as a conference paper at ICLR 2022
Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. Perspective Perspetive--adapiive convolutions for scene parsing. IEEE transactions on pattern analysis and machine intelligence, 42(():909-92) 924, 2019.
Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. Interleaved group convolutions. In Pro- ceedings of the IEEE international conference on computer vision, pp. 4373-4382, 2017b.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018.
Yitian Zhao, Lavdie Rada, Ke Chen, Simon P Harding, and Yalin Zheng. Automated vessel seg- mentation using infinite perimeter active contour model with hybrid region information with ap- plication to retinal images. IEEE transactions on medical imaging, 349)) 2015.
A APPENDIX
A.1 ABLATION STUDY
Influence of hyper-parameters. The proposed LPSC kernel has four hyper-parameters: the kernel size 2R+ + 1, the number of distance levels Ly, the number of direction levels Lg. and the growth rate g. For small kernels, since we set minimum value 2 for the smallest distance level, the largest L, can be determined by R² and g. If Tur is too large, no pixels will fall into regions of large distance levels. Lo can be set to 6 or 8, since most cases there are only about pixels the smallest circle in Fig. [(b) We evaluate the influences of L,,, Lg. and g by replacing the large 11 x 11 convolution kernels with the LPSC kernels in the first layer of AlexNet Tab. ())) and by applying LPSC kernels with different sizes in the first convolution layer before all blocks of VGGNet and ResNet in Tab ((c) and (d), respectively. We run all models for only one time. Increasing the values of L, and Lg results in finer regions and improves the performances, but the number of parameters also increases.
VGGNet and ResNet use small 3 kernels. To make the number of parameters comparable, fix (I,, Lo) to 62 and evaluate the influence of the kernel size 2R + 1 in Tab. (lc) When 2R+11 1 is too small, the LRF is limited. When 2R + 1 is too large, the regions with large distance levels may be coarse, i.e., a large amount of positions share the same weight, which may decrease the resolutions of parameters. Overall, for large kernels (11 x 11), we can set (L,, Lo.g) (3,8,2). For small kernels (5 x 5), we may fix (L,, Le. 9) to (2, 3
Influence of the plugged layer. Tab. 5]b) and (d) show the results of using LPSC in different layers or blocks for AlexNet and ResNet-20. respectively. It seems that performing LPSC in low layers more beneficial. This may be because pixels in high layers have merged information from large LRFs so that even adjacent pixels may have different influences on the center pixel and the weights for different positions are not suitable for sharing. Applying LPSC in low layers conducive to increase the LRFs, filter redundant details, and back-propagate the gradients to more bottom pixels. In the last column of Tab. d), the two successive convolution layers in each block are replaced with a single LPSC layer, the number of convolution layers is reduced by half and the number of parameters is reduced by one third, but the performances are only reduced by 2% using half the layers with I.PSC in ResNet.
Effects of weight regularization. In Tab. 6 we evaluate the effects of the weight regularization in Eq. (3) based on AlexNet. "Sum" shows the results by using sum pooling instead of mean pooling in log-polar space pooling in the first two LPSC layers. This is equivalent to remove the regularization and the performances are severely degraded. This is because far regions are exponentially larger than nearer regions. If positions in all regions are treated equally, even the weight for a far region not too large, the accumulation of less relevant distant pixels will still produce an overwhelming response. We also try max pooling. It also performs worse than mean pooling. Due to the large LRF, regions with large distance levels for many adjacent center locations will have large overlaps.
II this case, since we set a lower bound of the first level distance, g 2 and 3 will result in the same LPSC kernel.
13

---

Under review as a conference paper at ICLR 2022
Table 5: Ablation study based on (a)(b) AlexNet (c) VGGNet-19 and (d) ResNet-20.
(a) 11 x 11 LPSC kernels with different hyper-parameters
used in the first convolution layer of AlexNet.
(Iv, Lo.g)
(3.8.2)
(3,6,2)
(2,8,3)
(2,8.2)
# Params (M)
2.45
2.45
2.45
2.45
Acc. CIFAR-10 (%)
77.27
76.83
76.23
75,95
Acc. CIFAR-100 (%)
46.35
45.86
45.07
44.61
(b) Effects of using LPSCs in different con- volution layers of AlexNet. Kernel size re- mains the same in the corresponding layer. (I,, Lg, g) fixed 3, 8,2 and 2, 6, when applied to the 1st and 2nd layer, respectively.
LPSC layer
2
+
# Params (M)
2,45
2.33
2.31
Acc. CIFAR- 10 (%)
77.27
78.31
78.28
Acc. CIFAR-ION 100 (%)
46.35
44.81
47.31
(c) LPSCs with different sizes and hyper-paraan parameters are used in additionally added convolution layer before all blocks VGGNet-19.
2R+1
5
13
17
Lo.g)
(2,6.3)
(2,6,3)
(2,6.3)
(3,8,4)
Params (M)
20.08
20.08
20.08
20.08
Acc. CIFAR-10(%)
93.66
94.01
93.86
93.73
Acc. CIFAR-IO0 (
72.95
73.13
73.08
73.37
(d) Effects of using LPSCs in different layers and blocks of ResNet-20. The first two rows denote the sizes and hyper- parameters of IPSCCs in the first convolution layer before all blocks. The third denotes the sizes of LPSC's with fixed hyper- parameters (2,6.) in all blocks. "." means that no LPSCs are used in blocks. "B" means that two successive 3 x 3 convolution layers are replaced with a single LPSC layer in the BasicBlock.
2R
9
13
13
(Lr, Lo.g) g)
(2,6,2)
(2,6,3)
(2,6,3)
(2,6,2
(2,6,2)
2R + 1 for Blocks
9(B)
Params (M)
0.27
0.27
0.27
0.39
0.18
Acc. CIFAR- (%)
91.55
91.67
92.11
91.82
89.78
Acc. CIFAR- 100 (%)
67.23
67.19
66.97
67.98
65.10
Table 6: Effects of the weight regularization and center pixel convolution based on AlexNet.
Method
Sum
Max
No CenterConv Mcan
Acc. CIFAR-IOK%
21.61
76.65
78.51
78.28
Ace. CIFAR- 100(%)
5.53
44.63
47.13
47.31
Some large responses may dominant repeatedly in many regions for different center locations, which suppress other useful local information.
Effects of center pixel convolution. In the fourth column of Tab. we remove the center pixel con- volution, i.e., the first term in Eq. (3). Center pixel convolution enlarges the importance of the center pixel. Contextual information itself may be sufficient for classification when there are few classes. For more complex tasks with more classes, center pixel convolution may provide complementary information.
Running times. On the CIFARIO dataset, the training time for one epoch and the testing time for AlexNet are 3.4808 and 0.9083, respectively: after replacing conventional convolutions with LPSCs in the first two layers, the training time for one epoch and the testing time are 23.3981 and 3.7476, respectively. In our implementation, IPSSC runs much slower than conventional convolution, but this
14

---

Under review as a conference paper at ICLR 2022
Table 7: Comparison with different convolution methods based different architectures.
(a) AlexNet
Dilation (size, dilation ratc)
Square (size, pool size)
Conv Type
Ori
Deformable
hy--para L-1
5,3
7,2
7,2
11,5
11,5
hy-para. oth.
3,3
3,3
9,3
9,3
Layer
1
2
1,2
1,2
1.2
#params(M)
2.47
2,45
2.46
2.28
2.28
2.26
2.47
2.55
3.21
7.04
2,45
2.28
2.26
CIFAR10
77.22
73.43
75.7
74.94
78.11
75.95
75.98
55.34
32.96
55.22
76.10
75.17
73.26
CIFAR100
43.87
44.86
45.98
41.08
44.21
44.26
41.96
30.30
30.36
30.82
44.50
42.43
41.89
(b) VGGNet-19
Dilation (size, dilation rate)
Square
Conv Type
Deformable
(size, pool size)
hy--para. L-1+
13,4
9,3
9,3
hy-para. oth.
3,2
3,2
3,2
3,2
5,3
Block
2
1,2
1,2,3
1+
14,1
14,1,2
1+,all
1+
1+
1+,1
(params(M)
20.04
20.04
20.04
20.04
20.04
20.04
20.11
20.48
58.53
20.04
20.04
20.04
CIFARI0
93.34
91.53
91.97
89.61
89.56
92.53
70.37
90.64
90.02
87.42
90.01
89.4
CIFAR100
71.95
68.46
69.30
63.75
63.83
69.32
37.50
64.37
61.26
62.51
66.64
65.79
(c) Re
Dilation (size, dilation rate)
Square (size, pool size)
Conv Type
Ori
Deformable
hy--para. L-1+
3,2
3,3
5,2
3,2
13,4
9,3
9,3
9,3
hy--para. oth.
3,2
5,3
5,3
Block
Ori
1+
1+
1+
1,,1,2.3
1+
1,2,3
1+
1+
14,1
14,1,2
#params(M)
0.27
0.27
0.27
0.27
0.27
0.27
0.78
0.27
0.27
0.27
0.28
CIFAR-10
91.96
91.39
91.77
91.33
86.09
90.27
46.57
88.09
89.84
88.43
87.19
CIFAR
67.83
66.95
67.58
67.08
59.26
65.44
13.45
61.43
63.54
62.05
60.37
(d) Results of LPSC with different architectures
ResNet
Architecture
AlexNet
VGGNet
(size, Tr,, Too, 9) 13,2,6,3
hy--para. L-1+
11,3,8,2
9,2,6,3
5,2,6,2
hy--para. oth.
9,2,6,3
5,2,6,2
Layer/Block
1,2
1+
1+
1+.all
#params(M)
2.31
20.08
0.27
0.39
CIFAR-10
78.28
94.01
92.11
91.82
CIFAR-100
47.31
73.13
66.97
67.98
because of-the-shell conventional convolution modules to implement LPSC. To this end, we must first apply log-polar space pooling with the fold and unfold operations Pytorch, which consume much time and space complexity. LPSC can be greatly accelerated if it is directly imple- mented with CUDA or by directly adapting the underlying code of convolutions in the integrated framework.
A.2 COMPARISON WITH OTHER CONVOLUTION METHODS
Dilated convolution (Chen et al. 2017b) can also exponentially increase the LRF without increasing the number of parameters. Deformable convolution (Dai et al.) (017)) adaptively adjusts the LRF by using additional parameters to infer the offsets for each position. We use the Pytorch implementation of deformable convolution in our experiments. Different from LPSC where regions are divided in the log-polar space, can also averagely divide the kernel into different square regions, e.g., 9 x kernel can be divided into 3 x 3 regions with a size of 3 x All positions in the same square region share the same parameter. We denote this alternative convolution method by square
15

---

Under review as a conference paper at ICLR 2022
(a)
(b)
Figure 6: Visualization of the learned circular LPSC kernels without center convolution in the first convolution layer of Alexnet on (a) the CIFAR-10 dataset and (b) the CIFAR-100 dataset.
convolution, which also increases LRF with fewer parameters. Similarly, they can also be used to replace conventional convolution at different layers in different architectures.
In Tab. we compare LPSC with conventional convolution, dilation convolution, deformable con- volution, and square convolution in the three architectures, respectively. For dilation convolution, square convolution, and LPSC, the second and third rows show the byper-parametee of the corre- sponding kernels in the first convolution layer and in other layers or blocks, respectively. We run all models for only one time. "Size" indicates the kernel size. For VGGNet and ResNet, "1+" indicates the first convolution layer before all blocks. The fourth row indicates the indexes of layers or blocks in which conventional convolution is replaced by the corresponding convolution.
The second columns show the results of the three original architectures. With fewer or compara- ble parameters, LPSC outperforms conventional convolution in AlexNet and VGGNet, and obtains comparable results in ResNet.
The third column blocks show the results of dilated convolution with different ayper-parametter in different layers. The hyper-parameters, including the kernel size and the dilation rate, are set keep the total number of parameters and the LRF comparable to conventional convolution and LPSSC. In AlexNet and VGGNet, LPSC outperforms dilated convolution with different hyper-parameters significantly. This shows the effectiveness of spatial structure and parameter sharing in LSPC. Ap- plying dilated convolution in higher layers also leads to worse performances, which further verifies our analysis in Sec. A. but LPSC outperforms dilated convolution. When only used in the first layer of ResNet, sometimes dilated convolution achieves slightly better results than LPSC. The rea- son may be that ResNet can stack deeper layers with residual connections and hence achieve large LRF using multi-layer small regular kernels, which eases the need for large LRF in lower layers. Moreover, the non-uniform distribution of parameters in LPSC may cause information dispersion and over-smoothing. therefore it is more difficult to model residuals.
Deformable convolution introduces additional parameters. When applied to all blocks of VGGNet or ResNet, the parameters are more than doubled. Deformable convolution causes performance degradation of different architectures. LPSC also outperforms the average square convolution with This indicates that the spatial structure designed in log-polar space can better capture the contextual information.
different
16

---

Under review as a conference paper at ICLR 2022
H
(a)
Figure 7: Visualization of the filled 11 11 LPSC kernels without center convolution in the first
convolution layer of Alexnet on (a) the CIFAR- 10 dataset and (b) the CIFAR- dataset.
(a)
(b)
(c)
(d)
Merral glud
Nerall
c
g
(h)
Figure 8: Estimated RFs with (a) conventional convolution and (c) LPSC. The normalized gradient
map with (b-d) conventional convolution and (f-h) LPSC at different locations.
A.3 VISUALIZATION
Visualization of the learned LPSC kernels. Fig. 6 we visualize the learned LPSC kernels in the first convolution layer of AlexNet the CIFAR-10 and CIFAR-100 datasets. The LPSC kernlls have a size of 11 11, distance levels, direction levels, and a growth factor 2. Since LPSC kernels in the first layer have three channels, normalize the values of kerness into the range of [0. 255] and view each position the kemee as an RGB pixel. Different from conventional convolution kernels, in LPSC kernels, the closer the center, the higher the regional resolution;
17

---

Under review as a conference paper at ICLR 2022
the more outward, the larger the range for parameter sharing. We observe that the learned LPSC kernels capture some special local structures and contextual configuration. In some kernels, the weights for adjacent regions are continuous; some kernels are sensitive to simple directions and edges, some others are sensitive to complex boundaries and color combinations, and in some other kernels, specific combinations of regions are highlighted. To fully utilize the space circumscribed by the LRF of the kernel, we fill four corners (positions that do not fall into the LRF) with the weights of corresponding nearest regions all experiments, respectively, as shown Fig.
Comparison of effective receptive field (ERF): Fig. a(a) and (e) show the estimated RFs of Sim- pleVGGNet on the default example using conventional convolutions and LPSCs in the first two layers by the gradient-basee based RF estimation" 1) respectively. LPSC enlarges the estimated RFs from 14 to 22 x 22. The normalized gradient maps w.r.t. different positions of the output for estimat- ing the RFs using conventional convolutions and LPSCs are shown in Fig. ((b-d) and Fig, [if-h), respectively. With LPSC, gradients can be back-pree propagated to more pixels the input image.
18


Reviews:
[{"review_id": "V-L1agUezCl", "reviewer": "Reviewer_TRTh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of this paper is sound and intuitive, and the empirical results on multiple models and datasets demonstrate the superior performance of the proposed paper. However, some important aspects are not carefully considered or explained in the paper:\n\n1. The authors claim that they use the same meta-parameters for all methods for a fair comparison. However, this claim is problematic. It is well known that the meta-parameters can significantly affect the performance of machine learning models, and proper tuning of meta-parameters is a necessary step in practice. Using the very same meta-parameter may lead to non-informative results, because no practical model will be trained this way and different models will require different meta-parameters. Unless the authors can show that all the methods are affected equally by the meta-parameters, which is rarely the case, the evaluations provide very limited information. Also, the authors seem to report the best result in ablation study as the final performance, which optimizes the model performance on the test set and may lead to overly optimistic estimation for the performance.\n\n2. Comparing models with the same number of parameters may not be fair. Note that the proposed method can be considered as a special case of separable convolution with a fixed, carefully initialized weight for the depth-wise convolution. Therefore, the effective capacity of the model is larger than a model with plain 2D convolution. The authors should discuss how these fixed convolution layers (i.e. log-polar space pooling) affect the performance, e.g. is this the best weight, how does learned weights compare with fixed weights. Also, the authors should also consider other aspects of the model such as the computational cost, memory usage, etc. to provide a more complete picture for the pros and cons of the proposed method.\n\n3. The authors only evaluate the performance on relatively shallow models, i.e. ResNet-20. Because deeper models may learn the effective receptive field better, the proposed method may have diminished gain as the model becomes deeper or even harm the performance. Also, more recent architectures such as transformer and active convolution may be able to learn the appropriate receptive field directly. Given that the trend of architecture design is to use deeper and / or more flexible architecture and learn the regularization from data, the authors should discuss the implication of the proposed method on more recent models.\n\n4. Related to 1 and 3, the performance of the baseline methods seem to be sub-optimal. Therefore, the performance gain might not be very meaningful as the gap may reduce as the baseline models are properly trained.\n", "summary_of_the_paper": "This paper introduces a new formulation for the convolution operator. Instead of defining convolution kernels on a regular 2D grid, this paper proposes to define convolution kernels on a circle divided by angle and radius. Note that the radius is divided equally in log-scale to account for the observation that content near the center should have more impact on the outputs. The new convolution operator is generic and can replace existing 2D convolution operators. Also, it can be implemented in existing frameworks easily using a customized pooling operator plus standard convolution. Empirical results on 1) multiple architectures, 2) multiple tasks, and 3) multiple datasets show that the proposed method outperforms standard convolution under a similar number of parameters.", "summary_of_the_review": "The method introduced in this paper is intuitive and seems promising empirically. However, some important aspects are not carefully considered in the evaluation. Therefore, it is hard to justify the benefit and contribution of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ned0LfAIuwB", "reviewer": "Reviewer_pwSX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1. The idea of LPSC is interesting and seems reasonable to me. It does provide a different perspective of performing convolutional operators, while the ability to exponentially expand local receptive field (LRF) without significantly increasing the parameters can be potential in image recognition tasks, especially for dense prediction ones like image segmentation.\n2. The performance of LPSC-based networks has surpassed that of original ones on a variety of architectures and on different datasets of classification/segmentation tasks, which can be a good sign of generality.\n3. The idea is straightforward to understand and the paper writing is generally easy to follow.\n\nWeakness\n1. The ability to expand LRF is a major claim, however, the current experimental results may be insufficient to support the claim. The authors only evaluate their methods on light-weight network structures like ResNet-18/20, MobileNet, etc., while the receptive fields in those models might not be dense or broad enough and therefore can be compensated by the proposed LPSC. However, what about those more complicated models like ResNet-101 that may already have a good LRF coverage? Will LPSC still work on them?\n2. Expanding LRF is a popular topic in semantic segmentation, and there are some missing related works like Dense ASPP [a] or [b].\n3. The computational inefficiency can also be a concern here. The authors have already discussed that the spatial complexity can be a limitation (which I appreciate), but I think this also indicates an increased time complexity since the mask has to be pre-computed. Besides, as the radius grows, the area of outer rings can grow exponentially, which can lead to an exponentially-increased complexity. \n4. The log-polar form of LPSC kernel assumes that the features near the centre can be more important than those more distant ones, since the learnable parameters are much dense in central regions. However, I am not sure why this kind of data distribution assumption can work better than the normal convolutional operators or other operators like deformable or dilated convs. I think a discussion on the data distribution can be necessary.\n5. The achieved improvement on some datasets is somehow marginal. The lack of more advanced models like ResNet-101 or Xception-65 can make the results unconvincing. The authors only compared their method with dilated convs in segmentation experiments, but I think it can be important to also compare with other related methods like DenseASPP [a], Deformable Conv or Self-Attention module. \n \n[a] Yang, Maoke, et al. \"Denseaspp for semantic segmentation in street scenes.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.\n[b] Chen, Liang-Chieh, et al. \"Searching for efficient multi-scale architectures for dense image prediction.\" arXiv preprint arXiv:1809.04184 (2018).", "summary_of_the_paper": "This paper proposes a Log-Polar Space Convolution (LPSC) for general image recognition tasks. The proposed convolutional kernel distributes in the form of log-polar spaces, and therefore the receptive field can expand exponentially as the radius grows without significantly increasing the learning parameters. To implement LPSC, the authors first utilize a log-polar space pooling to pre-compute an up-sampled feature map and then LPSC is performed just like normal convolutional filters.  The proposed LPSC are integrated into several popular CNN backbones like ResNet, VGGNet, MobileNet, etc., and the performance of modified networks has exceeded the original structure on several image recognition and segmentation datasets. ", "summary_of_the_review": "Overall, the idea of this paper is interesting and reasonable, while the proposed LPSC can be potential for dense prediction tasks. However, there are several limitations of this method, while the experimental results are not very convincing at this stage. For now, I will vote for a marginally below.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LGESf7JU69", "reviewer": "Reviewer_MHsy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written, and the main idea seems like a logical thing to try. There are a variety of precedents for log-polar features such as the organization of the human retina and the success of classical methods like shape contexts. The authors are honest about the shortcomings (such as large memory requirements) of their method. \n\nWhile there is nothing terribly objectionable about this paper, I'm afraid the result is not terribly compelling. While the exact architectures proposed by the authors are not directly implementable with standard techniques in the literature, one can come awfully close.\n\nFor example, if one makes an architecture with 3 types of dilation (say, dilations of 1, 2, and 4) in a particular layer, and makes all of the filters relatively small, then the next layer could assemble the outputs of such filters to learn something extremely similar to the filters prescribed in this paper. Now, a counterargument might be that in such a dilation-based architecture, the network might not learn to put together filters in a way that mimics the log-polar structure produced here. On the other hand, if such an architecture does not learn to assemble filters in this way, the counter-counterargument would be that these filters are not the most effective filters to reduce the loss function.\n\nIrrespective of this argument, I would expect the authors to provide stronger baselines. For example, a strong baseline would be, \"how could I build an architecture, using traditional methods, that would allow filters to be learned that incorporated dense sample around a point combined with sparse sampling far from the point?\" Such a strong baseline would be more interesting that simple dilation architectures.\n\nThe main point of all of this is that it seems highly unlikely that such a specific structural change to an architecture would allow a network to learn functions that another standard network could not learn. While the experiments support the idea that the presented architecture has a minor advantage over other simple baselines, it by no means makes a compelling argument that such architectures are interesting from a real practical point of view on the tough problems people are currently addressing. In short, the argument that this is a compelling new architectural element has not been made.\n\nAnd if the network is not more expressive, then the question becomes whether it is significantly faster to train or does it require less memory? Apparently, this network offer none of these advantages and even some disadvantages. As such, it seems unlikely that anyone would adapt it for a specific task over current state of the art models.\n\n", "summary_of_the_paper": "This paper revisits the old of idea encodings that have higher density around a particular point and lower density as one moves away from a particular analysis point. In the literature, these have often been called log-polar filters, and including classical computer vision techniques such as shape context features. The authors bring these types of filters to CNNs, by modifying architectures (using pooling techniques) to allow for filters that have this type of structure.  They do a variety of experiments showing that these methods can outperform certain simple baselines, such as local filters and dilated filters.\n", "summary_of_the_review": "The paper is well-written with a somewhat interesting idea, but ultimately, it seems unlikely that such an architecture provides advantages over strong baselines, which are not really explored. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "461bpAQj29c", "reviewer": "Reviewer_2ZGD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n---------\n\nOverall this is a nice idea, and the experimental results are fairly convincing in showing that the proposed method does not hurt in existing model architectures. I think that this provides a nice new potential \"tool in the toolbox\" for future investigations.\n\n\nConcerns\n--------\n\nBroadly speaking my concerns fall into three areas:\n\n- __Writing/Presentation__: The mathematical parts of the paper need revisiting/correcting; in the current version equations 1 and 3 need to either incorporate a dot symbol, a transpose on the X, or an extra summation over channels to make sense. Additionally, from a mathematical perspective the equations are those of cross-correlation rather than convolution, so it would probably be better to either use the cross-correlation operator symbol or reverse the indexing of the kernel (or input). Some mention of this in the text would be wise, and it may be better to for example change the title of the paper to something like \"log-polar space convolutional layers\" to make it absolutely clear that the paper is about CNN layers rather than more general convolutions. Further, the authors need to clarify the use of the bias term in their implementation - it is not clear from the math/text if there is one or not.\n\n- __Results__: Overall the results come across as weakly positive; The results on imagenet and voc2012 feel rather superficial without quantification of variance given the gains are so marginal. I realise that this might exceed the computational resources available to the authors to acheive, however the marginal gains are deserving of a much more detailed discussion. In particular, I'd like significantly more discussion and analysis of what the numerical accuracy/auc/etc metrics mean in light of the what is shown in figure 4, where it appears that a significant number of the kernels do not benefit from the increased receptive field size. Also regarding fig 4, it is not clear quite how this was created - does a mid-grey value represent kernel values of zero, or are the values somehow normalised? This is important as it would completly change one's interpretation of the results.\n\n- __Complexity Analysis__: Firstly, I'm not sure I understand the reported FLOPs result - why is the dilated convolution much higher? The number of FLOPs to apply a standard convolution and a dilated convolution with the same number of weights should be identical. Further, I suspect that this analysis would only tell part of the story about complexity. More specifically one would want to understand the actual latency of the different approaches because the memory access patterns are going to be significantly different. In terms of wall time, how do the different methods compare during training and inference for example?\n", "summary_of_the_paper": "The paper proposes a log-polar space convolution in which elliptical kernels are convolved over the input. The kernels are broken into radial segments and each value of each segment is applied to all the pixels in the underlying portion of the image, which are then summed and potentially normalised. Actual implementation is achieved through use of log-polar space pooling which can be used to create a set of pooled featuremaps to which a regular convolution can be applied. The advantage of such an approach is that it allows for much larger receptive fields without the need forr more weights. \n", "summary_of_the_review": "Overall, I think there are some interesting ideas emerging from this work, but in its present state it's a bit borderline. As outlined above if the problems with presentation could be addressed, and further information given on the computational complexity with more analysis/discussion of the results then my concerns would likely be assuaged. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an alternative for constructing convolution kernels: instead of uniform spatial resolution, it proposes a spatially varying resolution with higher precision at the center of the kernel. The resolution decreases logarithmically as a function of the distance to the center. All reviewers agree that the idea is interesting, but in its current form, the submission is not mature enough to be published.

In particular, reviewers raised some concerns about computational efficiency of the method. The authors explain that their method runs slower than conventional convolution because the implementation uses of-the-shell conventional convolution modules, and they speculate that the speed can be accelerated if the method is directly implemented with CUDA or by directly adapting the underlying code of convolutions in the integrated framework. While this is a reasonable argument, it is not actually verified. This it is not clear if there would be other road blockers to achieve the promised performance. It would be great if authors could present actual performance of the method using either of their suggested solutions (CUDA or modifying code of convolutions).

In addition, reviewers raised concerns about some aspects of the evaluation setup, where test data is used to report the best performance. Authors respond that baselines are trained in the same fashion, hence the comparison is still fair. However, the reviewers were not convinced by this response. In concordance, I also think the use of test data during training is misleading, even if all methods use the same strategy, because this may tell us more about which approach can better (over)fit to the data as opposed to how well the methods are able to generalize to unseen samples.

Another concern relates to the diminishing return in the performance as networks get larger. The authors respond that this might be because only the first layer uses the proposed log-polar convolution, speculating the problem will go away if the proposed approach is used in all layers. However, this is not empirically verified again and remains unclear if this is indeed the reason.

I suggest authors resubmit after accommodating the provided feedback.
Decision: Reject