Log-Polar Space Convolution Layers
Authors: Bing Su
Ji-Rong Wen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Log-Polar Space
Convolution Kernel
Local Receptive Field
Spatial Context

Abstract: Convolutional neural networks use regular quadrilateral convolution kernels to extract features. Since the number of parameters increases quadratically with the size of the convolution kernel, many popular models use small convolution kernels, resulting in small local receptive fields in lower layers. This paper proposes a novel log-polar space convolution (LPSC) layer, where the convolution kernel is elliptical and adaptively divides its local receptive field into different regions according to the relative directions and logarithmic distances. The local receptive field grows exponentially with the number of distance levels. Therefore, the proposed LPSC not only naturally encodes local spatial structures, but also greatly increases the single-layer receptive field while maintaining the number of parameters. We show that LPSC can be implemented with conventional convolution via log-polar space pooling and can be applied in any network architecture to replace conventional convolutions. Experiments on different tasks and datasets demonstrate the effectiveness of the proposed LPSC.

Log-Polar Space Convolution Layers
Bing Su, Ji-Rong Wen
Beijing Key Laboratory of Big Data Management and Analysis Methods Gaoling School of Artificial Intelligence, Renmin University of China Beijing 100872, China
subingais@gmal com; jrwenBruc, edwiccedduccoom
Abstract
Convolutional neural networks use regular quadrilateral convolution kernels to extract features. Since the number parameters increases quadratically with the size of the convolution kernel, many popular models use small convolution kernels, resulting in small local receptive fields lower layers. This paper proposes novel log-polar space convolution (LPSC) layer, where the convolution kernel is elliptical and adaptively divides its local receptive field into different regions according to the relative directions and logarithmic distances. The local receptive field grows exponentially with the number of distance levels. Therefore, the proposed LPSC not only naturally encodes local spatial structures, but also greatly increases the single-layer receptive field while maintaining the number of parameters. We show that LPSC can be implemented with conventional convolution via log-polar space pooling and can be applied in any network architecture to replace conventional con- volutions. Experiments different tasks and datasets demonstrate the effectiveness of the proposed LPSC.
Introduction
Convolutional neural networks 1 2 have achieved great success in the field of computer vision. The size of the convolution kernel determines the locally weighted range the image or feature map, which is called the local receptive field (LRF). In many computer vision tasks such as image classification 2
and intensive prediction [5] 6 77, larger LRF is generally desired to capture the dependencies between long-distance spatial positions and a wide range of context information. Simply increasing the size of the convolution kernel is not plausible because the number of parameters increases quadratically with the size.
In practice, commonly used techniques to obtain larger receptive fields include adding pooling layers, replacing single-layer large convolution kernel with multi-layer small convolution kernels, and using dilated convolutions g 99. The pooling process often causes information loss. Increasing
the number of convolutional layers may cause vanishing gradients and make training more difficult.
Moreover, going deeper with small kernels may not indicate a larger receptive field. A plain CNN with all 3 x 3 convolution kernels cannot be too deep without residual connections. Some studies [TO] have found that ResNets behave like ensembles of shallow networks. Regardless of the actual depth, the effective number of layers for ResNets maybe limited. That is, even if ResNet with hundreds of layers is stacked, its actual receptive field may be equivalent to that of a shallow network.
According to the effective receptive field (ERF) theory [1I. the ERF is proportional to the square root of the depth and directly proportional to the kernel size. Therefore, it easier to achieve a large ERF by increasing the kernel size than by adding layers. The success of Vision Transformers [2 13 may also reveal the effectiveness large local windows, while various sparse attention mechanisms 14
"Corresponiigg author: Ji-Rong Wen.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

15 [6 for Transformers are proposed to allow larger LRFs with limited increases of calculations. In this paper, we reconsider lightweight CNNs with large convolution kernels. Dilated convolution kernels are able to increase the LRFs greatly, but they are not continuous since not all pixels in the LRF are involved in convolution calculation. The skipped pixels are regularly selected. With the same number of parameters, the larger the LRF, the more pixels are skipped, which may miss some details and cause discontinuity of information.
In addition, conventional and dilated convolu- tions use regular square kernels. Each position is assigned a different weight within the LRF. All positions are equally treated regardless of the size of the kernel. However, intuitively, the correlation between neighboring pixels and the center pixel is usually higher, while the farther the pixel, the smaller the impact on the center pixel, which is evidenced by statistics from nat- ural images presented in Appendix A. The effects of two adjacent pixels that are far away from the center are usually similar, thus they can share the same parameter rather than be assigned different weights separately. As shown in red in Fig. I[a] according to the configuration of surrounding regions, it can be inferred that the center position is located on the upper edge of the nose. Pixels in the same upper-left outer half- fan-shaped region show that the far upper left of the center point is white fur, but there is little difference in the effects of two specific fur points.
1
W
W2
Wa
Wo
(b)
Figure 1: (a) At different locations of an image, lo- cal contextual pixels can be divided into different regions according to their relative distances and di- rections in the log-polar space. For each location, pixels falling in the same region are generally sim- ilar and can share the same weight. (b) The LPSC kernel. For this example, L, = Lg = 6, g 2, thus there are only 18+1 parameters.
In this paper, we propose a novel tog-polar space convolution (LPSC) method. The shape of the LPSC kernel is not regular square, but an ellipse. Purameters of the kernel not evenly distributed in the LRF, but are assigned in the log-polat coordinate space. As shown in Fig. [bb]] the LPSC kernel divides the LRF into different regions, where regions become larger with the increase of the
distance to the center. Pixels that fall into the same region share the same weight. In this way, LPSC increase the LRF exponentially without increasing the number of parameters. Besides, LPSC naturally imposes a contextual structure on the local neighboring distribution.
The main contributions of this paper include: 1. We propose a new convolution method where the kernel lies in the log-polar space to capture the structured context information and greatly expand the LRF without increasing the number of parameters. 2. We propose log-polat space pooling to up-sample the feature map, by which conventional convolution can be conveniently used to achieve LPSC. 3. We apply LPSC replace the conventional and dilated convolution in different network architectures including AlexNet, VGGNet, ResNet, Deepl.abv3+ and CE-Net. We demonstrate the effectiveness of LPSC through empirical evaluations different tasks and datasets.
2
Related work
Context pooling. Our method is highly motivated by shape context [1 18 Centered at a reference point, all other points are divided into bins that are uniformly distributed in the log-polar space. The histogram among these bins is used as the descriptor. The statistics in the log-polar space have also been shown to be effective for word recognition in [19]. Geometric blur (00 sparsely samples and aggregates a blurred signal in the log-polar space. Pyramid context [2]] pools log-spaced context points at multiple scales. Different from these methods, we design kernel in the log-polar space for convolution, each region is assigned a weight to aggregate information from the bins. We incorporate the kernel into deep neural networks.
Methods to increase LRFs. In (22 and 23 is found that imposing a regularization on large convolution kernels is equivalent to the superposition of multiple convolution layers with smaller kernels. Based on this observation, many state-of-the-arr network architectures use multi-layer small kernels. However, deeper layers may cause vanishing gradients, making the network more difficult to

---

train. Moreover, according to M. the effective receptive field (ERF) is proportional to the square root of the depth and proportional to the kernel size. Thus it is easier to achieve a large ERF by increasing the kernel size than by adding layers. We provide a way to increase the LRF without increasing either the number of layers or the number of parameters. In cases where large input or LRF is required but very deep networks are not allowed restricted by resources, our method may be applied to construct a lightweight model.
In 8 91. atrous (or dilated) convolution increases the LRF by inserting holes (zeros) between parameters in the kernel, where the interval is determined by dilation rate. Dilated convolution has been applied in different tasks [24] [25 7 26 27 (8)). In [29] and [30], acale-adaptive convolution learns adaptive dilation rate with a scale regression layer. Due the insertion of holes, not all pixels in the LRF are used for calculating the output. In [1 and [2], this problem is alleviated by hybrid dilated convolution and Kronecker convolution that uses the Kronecker product to share parameters.
Other convolution methods. Fractionally strided convolution E3 34 up-samples the input by padding. In (35), a spatial transformer transforms the regular spatial grid into a sampling grid. Active convolution (36] learns the shape of convolution by introducing the convolution unit with position parameters. Deformable convolution and kernels 66 37 learn additional offsets or perform resampling to augment the sampling locations, thereby adaptively changing the LRF into a polygon. For active and deformable convolutions, the adapted LRF contains holes, the positions and offsets are learned through additional convolutions, which increases the parameters. Deformable kernels [38] resample the original kernel space and adapt to the deformation of objects. The offsets for kernel positions also need to be learned. Quasi-hexagonal kernels (39], blind-spot kernels 44), asymmetric blocks [4], and circle kernels [42] also have non-regular shapes, but generally they cannot enlarge LRFs without increasing parameters.
Group convolution [2. 43 and separable convolution 445 do not increase the LRF of kernels. Octave convolution [46 decomposes the feature map into high-frequency and low-frequency features. Multi-scale convolution is performed in 47 and [48). In [49 and [50), stand-alone self-attention is used to replace convolution. The filter in the attention module lies in regular and square grid. In [5]], the polar transformer network generates log-polar representation of the input by differentiable sampling and interpolation techniques. The polar transform is applied to single predicted origin location. In contrast, LPSC performs log-polar pooling via binning and can be applied at any location.
For dilated and other advanced convolutions, the kernel is still performed in a regular grid and all parameters are treated equally. Regardless of the distance from the center, the interval or the sharing range of a parameter is the same among different positions. In contrast, the proposed LPSC expands the LRF in the log-polar space, where near and far regions are distinguished in parameter sharing. The farther away from the center, the larger the range of parameter sharing.
Differences.
3 Log-polar space convolution
Let X E "XXCC be the input image or feature map. where H, W, and C are the height, width, and number of channels of X, respectively. W E B(2M+1)x(N++) is conventional convolution
kernel with a size of (2M + 1) x (2N + 1). The central parameter of W is indexed by (0, parameters of W lie in regular grid {(-M, M-,, + 1), (M - 1, N), (M, N)).
The convolution operation performed in the 2D spatial domain across the channels. For a spatial location (i. the output of the conventional convolution is calculated as
M. N (x * WXi,i)= Σ E(x(i+m,++)) n)- W (m,,n) + b, ME-M
(1)
where is the bias. Strictly, Eq. I actually performs cross-correlation. For convolution, W needs to be rotated 180 degrees. However, since we can view the learned W as the rotated kernel, we follow the common practice of CNN to formulate convolution into Eq. (T). Parameters of the kernel are uniformly distributed in the regular grid, thus each pixel of X falling into the field is weighted by separate parameter, i.e., all positions are equally treated. However, pixels that have different distances and directions from the center may have different impacts, e.g., pixels adjacent to the center should have larger contributions to the output. Pixels in the input image usually change gently, adjacent pixels far away from the center often have similar impacts the center. Based on these intuitions,
3

---

(a)
(b)
Figure 3: (a) Log-polar pooling generates a 2T, x T matrix per position using the pre- computed mask. (b) To perform LPSC on the Figure 2: (a) The LPSC kernel is slid through the original input (left), log-polar pooling generates feature map. (b) The shape and inclination of the matrix for each pixel, resulting in upsam- LPSC kernel can be changed.
(a)
(b)
pled map (middle), conventional convolution applied this map (right).
we design convolution kernel with special structure, namely Log-Polar Space Convolution (LPSC) kernel, to express wide range of contextual configurations.
3.1 LPSC kernel
As shown in Fig. I(b) the proposed LPSC kernel lies in the log-polar space and shaped by the size
2R + 1, the number of distance levels L,,, the number of direction levels Lg. and the growth rate The LRF of the kernel is the area of the outermost circle whose radius is R. It is uniformly divided
into L, x Lg regions in the log-polar space. Specifically, the log radius is uniformly divided into L, levels, i.e.,
() - log( RR) = (og((R) (og(((-)) log(g).
(2)
where R₁, 1 = I,... L, is the radius of the 1-th level and the growth rate g is hyperparameter controlling the expansion speed. When the center the kernel is located at position (ch, cw), all pixels of X in the range of A [c₁, R, c₁ R) x [cw R, Cu R] are divided into L, levels according to their relative squared distances to the center position. The position (i. () E A belongs to the I-th distance level if R I d.j < R2, where d (i C₁2 )2 + (j c.. )² From Eq. (2). we have R₁ o² R1. When the innermost radius RS is fixed, the LRF grows exponentially with the increase of L,, The LRF determined by R which can be set arbitrarily. Given RI, = R² and we calculate R = max R² /2/g" " -m) We use R = RR,,, as a hyperparameter instead of R, which more flexible. Since we use the squared distance, impose minimum value of ensure that all 8-neighborhood pixels fall into the -st level.
All positions in the range of A are also uniformly divided into Lg levels according to their relative directions from the center. The position (i, belongs to the m-th level if 2T - L) S 0G. < 2Tm/Lp. where 02, is the counterclockwing angle from the vector (0, 1) to the vector (i Ch. j Cw). Combining the distance levels and the direction levels, the LRF is divided into L, x Lo regions.
The LPSC kernel assigns a parameter to each region. All pixels of X falling into the same region share the same parameter. For the region with the I-th distance level and m-th direction level, the assigned parameter is denoted by Wt,m- The areas of regions increase with the farther away from the center, the larger the area, the more pixels sharing parameters. Because the center position of the kernel is important and forms the basis of regions, we assign additional separate parameter Wo,0 for the center pixel. A conventional kernel with a size of (2R + 1) X (2R + 1) has (2R + 1)² matter how large R. is. When R ranges from 2 to 9, a single conventional kernel has 25 to 361 parameters. In this range, it is sufficient to set L, 2 or 3 and set Lo to 6 or 8, so an LPSC kernel only has 13 to 25 parameters.
parameters, while LPSC kernel only has L, Le + 1 parameters
Let Nr,, denote the number of pixels falling into the region bin(l, m) with the 1-th distance level and the m-th direction level. In faraway regions with large I, .N... the impacts of pixels in them should weakened. Therefore, we regularize the weight Wr.m each region by Nu.. Wt,:WW,,. NN..- As result, the LPSC kernel aggregates finer information from pixels nearing the center and is less sensitive those of pixels farther away. Similar conventional convolution, the LPSC kernel is slid along the input feature map X with pre-defined stride to perform convolution, as shown in Fig 2(a)

---

When the kernel is located at spatial location (i.5), the output response is calculated as
(X ()(i,) (0000) - (000)xX((i))) ((,)))
Σ w()
E v)) - (3)
N1.mn
M.00bin(1.m)
For the LPSC kemel, the shape of its LRF is not necessarily a standard circle, but can be an oblique ellipse. As shown in Fig. 2 two additional
are introduced: the initial angle a and the eccentricity of the ellipse e. When dividing the regions, the distances are calculated according to the squared ellipse distance and the initial angle is added to the calculated directions. In this way, the LPSC kernel can better fit objects with different rotations and scales. In our experiments, we only evaluate the standard circular LRF by setting a 0 and e. = h//
3.2 LPSC via log-polar space pooling
Due to the special structure and parameter sharing, LPSC cannot be directly performed by popular deep learning frameworks. In this subsection, we show that LPSC can be readily implemented by conventional convolutions via log-polar space pooling to utilize efficient convolution modules.
Given the matrix I to indicate the region indexes of positions. The size of the mask I is (2R + 1) x (2R + 1). 1, 1. Lgg x L.r in I indicates the region index of the corresponding position. indicates that the corresponding position does not fall into the LRF, since the region of the mask is the cireumscribed rectangle the LRF. The mask is slid through the input feature map X with the same stride of the LPSC convolution. As shown in Fig. 3(b when the mask is located at spatial location (i, pixels of in the range are divided into regions indicated by the mask. All pixels in the same region are encoded into a single pixel by mean pooling. We re- arrange the pooled pixels of different regions into a matrix of 2L, x Le/2 to preserve their relative spatial positions, as shown in Fig. ((a) In this way, given H' x W convolution locations (H' = H and W = W if the stride is 1 with padding), the spatial size of the output map X, after log-polar space pooling equals 2H' L, x L2/
R, L,, Lg. and g of the proposed LPSC, we can pre-compute a mask
We perform conventional convolution with C' output channels on the output map X, without padding. The size of the conventional convolution kernel set to (2I,r, Lo/2) and the stride is also (2L,, L2) The output feature map Y, has size of H' x W' x C". This is equivalent to performing the second term in Eq. B.. To model the first term, we use a separate x 1 conventional convolution with the same C' channels on the original X. The stride is the same as the log-polar space pooling. The output feature map Y. contains the convolution responses of the center pixels. We add this separate center pixel convolution output Y. to the contextual convolution output Yp Y. + Y, serves as the output feature map of the proposed LPSC.
3.3 Incorporating LPSC into different CNNs
LPSC can be integrated into different CNN architectures. A straightforward way is replace all conventional convolution kernels with LPSC kernels in a part of convolution layers. For plain CNN architectures such AlexNet (2) and VGGNet 22]] we simply perform this strategy in lower layers to increase the LRFs. However, some network architectures such as ResNet [3] are constituted of specifically designed blocks. In ResNet, either the bottleneck or the basicblock structure only contains 3 x 3 and x convolutions. Due to the difference in the local receptive field, the information captured by these small convolutions and LPSC may be different. In order to better incorporate these two types of information, we propose cross convolution strategy as an alternative to replacing all convolutions in each layer of the block. Specifically, we set ratio p. For each of several consecutive layers, we replace p% of all convolution kernels to LPSC kernels, while the remaining (100- - p)% % of conventional kernels remain the same. In this way, each convolution kernel in the next layer, whether it is conventional or an LPSC kernel, perceives the outputs generated by both the conventional and LPSC kernels of the previous layer. We denote this cross-convolution strategy by LPSC-CC. Details on how to incorporate LPSCs depend on the CNN architecture and will be presented in Section 4 Our code available hhtpp:: / //gi
//tpthbb.tt/t///// thub. com/ B 1gSu12
Log Log-Polar- Space- Convolution
5

---

3.4 Discussions
Complexity, For (2M +1 x (2N + 1) x C'kemel,, conventional convolution involves (2M + +1) x (2NV + +1) x C multiplicatiins and (2. + 1) x (2NN + 1) x C additions. LPSC with L, distance levels and Lg direction levels only involves 2 L, x Lg x C' multiplications, (2. + 1) x (2N + 1) x C additions, and (2.M- + 1) x (2NV- + 1) lookups. The complexity of pre-computing the mask for lookup is O which only needs to be calculated once when initialing the layer. Typically, if L, 2. Lg 6, LPSC only executes 24C' multiplications for any size. However, even for a small (2M 1) x (2N + 1) = 5 x 5 kernel, conventional convolution executes 25C multiplications; for 9 x 9 kernel, multiplications increase to 81C.
Structural benefits. With the special log-polar structure, the LPSC kernel naturally encodes the local spatial distribution of pixels w.r.t. the center and puts more attention to those adjacent pixels. Pixels with similar relative distances and directions share the same parameter, which not only reduces the number of parameters, but also makes the filter more robust and compact. Due to the logarithm effect, when located at different objects, small objects are relatively enlarged, while large objects are relatively reduced. Therefore, LPSC is less sensitive to the size of objects. Advantages of log-polan space pooling and extensions of LPSC to 1-D and 3-D data are discussed in the appendix.
Relation with effective receptive field [1.. In [II] it is found that the ERF only occupies a fraction of the full theoretical receptive field. Specifically, the ERF size is O(kvn), where k = 2/2 + 1 is the kernel size and is the number of layers. Therefore, increasing the kernel size has greater effect on expanding the ERF. It is also found that not all pixels in the LRF contribute equally, where the impacts of pixels near the center are much larger. The LPSC kernel follows this spirit to treat pixels near the center finely and increase the LRF exponentially.
Drawbacks. LPSC has two main drawbacks. (1) It introduces three additional hyper-parametent Iy. Lo, and g. However, in practice, their selectable ranges are quite limited. Generally, to make the 8- neighborhoods of the center pixel have finer and non-redundant regional resolution, L,, is set to 3, La is set to 6 or 8, and is set to 2 or 3. (2) Its implementation via log-polar space pooling incurs large memory overhead. The space complexity of the upsampled feature map X, is O L,, LoC). For a single layer, the space complexity of LPSC is O(H'W" L, LoC + L, CC + H'W"C".
Limitations. Parameter sharing in LPSC aims to expand the local receptive field without increasing the number of parameters, but the cost is the loss of some fine- ane-grained information. LPSC is more suitable for semantically sparse visual data that contains redundant information. As long as the data distribution conforms to the local correlation assumption, our LPSC can also be applied to irregularly sampled data, provided that the relative distances and angles between data points defined. However, the mask matrix to indicate the region indexes of positions canmot be pre- computed, the speed of LPSC will be very slow, because the region that each sampled data falls in should be calculated on-the-fly. LPSC may not be suitable for semantically dense data such as speech signals, text sequences, and amino acid sequences.
4 Experiments
4.1 Image classification experiments
For image classification, we evaluate the behaviors of LPSC integrated with different CNN archi- tectures three datasets: CIFAR-10, CIFAR- 100
(52). and ImageNet (53). We plug LPSC into three typical CNN architectures, including AlexNet 2 VGGNet-19 (2). and ResNet20 23 by replacing a part of the conventional convolution layers. We use the Pytorch (54] implementatior of these architectures as our baseline. For the AlexNet, there are 5 convolution layers each followed by ReLU activation layer. The sizes of the convolution kernels are 11 x 11, 5 5, 3 3, 3, and 3 x 3, respectively. For the VGG19 Net, there are sixteen convolution layers. The kernel size for all convolution layers is 3 x 3. For the ResNet-20, there are basic blocks. Each block contains two 3 x 3 convolution layers. A 3 x 3 convolution layer is applied before all blocks. When the conventional convolutions in a layer or block are replaced by LPSCs, the number of kernels and the size of the output feature map remain the same as the original convolution layer.

---

Table 1: Comparison of different convolution methods.
(a) Accuracy (%) with AlexNet and VGGNet-19
AlexNet Dilation
VGGNet-19
Convolution
Ori
LPSC
Ori
Dilation
LPSC
Params (M)
2.47
2.34
2.31
20.04
20.08
20.08
CIFAR-10
77.43 (0.25)
75.42 (0.06)
78.44 (0.12)
93.54 (0.06)
93.46 (0.14)
93.92 (0.06)
CIFAR- 100
43.98 (0.43)
44.43 (0.10)
47.43 (0.20)
72.41 (0.17) 73.03 (0.34)
73.13 (0.12)
(b) Accuracy (%)
with ResNet-20
Convolution
Ori
Dilation
LPSC
LPSC-CC
# Params (M)
0.27
0.27
0.27
0,27
Acc. CIFAR-10 (%)
91.66 (0.13) 91.44 (0.10)
91.81 (0.21)
92.01 (0.08)
Acc. CIFAR- 100 (%)
67.56 (0.27) 66.90 (0.25)
67.63 (0.27)
68.09 (0.27)
Table 2: Results with ResNet- 10 on CIFAR-100.
Table 3: Results with ResNet- 18 on ImageNet.
Model
Inference Time
Acc
Model
#params(M)
Top-I Acc (%)
Conv
0.025 (0.0053)
73.50 (0.24)
Conv
11.69
69.86 (0.082)
Dilation
0.025 (0.0041)
73.75 (0.48)
LPSC
11.69
69.96 (0.081)
LPSC
0,036 (0,0047)
73.54 (0.38)
LPSC-CC
0.135 (0.0035)
74.13 (0.14)
LPSC-CC
11.70
69.95 (0.079)
To make a fair comparison, all experimental setup and details including the learning rate, batch size, number of filters per layer, decay) remain exactly the same as in the baseline. We did not tune any of these setups for our Therefore, the differences in performances only come from the changes in convolution layers. The numbers of parameters are computed on the CIFAR-10 dataset. Top-1 accuracy is used as the performance measure.
for the optimizer (e.g., Y, momentum, weight
Results on the CIFAR1O and CIFAR100 dataset. We train the AlexNet, VGGNet-19, and ResNet- 20 with conventional convolution, dilation convolution, and LPSC five times by using different random seeds for initialization, respectively, and compare the average accuracies and standard deviations. "Mean accuracy (standard deviation)" results are reported in Table We use LPSC in the first two convolution layers for AlexNet, in the added first convolution before all blocks for VGGNet- 19, and in the first convolution layer before all residual blocks for ResNet-20. the LPSC kernels different layers and networks are the same as the first three columns in Table A4(d) in the appendix, respectively. These choices are based on the ablation study as described in Appendix A.2 and A.3. For dilation convolution, we replace the conventional convolutions with dilation convolution in the same layers in the three architectures, respectively, where the kernel size and dilation rates are set so that the LRF and number of parameters are comparable with LPSC. Specifically, for AlexNet, the kernel size and dilation rate are set to 5 and 2 in the first convolution layer, respectively, and 4 and 2 in the second convolution layer, respectively. For VGGNet- 19, the kernel size and dilation rate are set to 4 and in the added first convolution layer before all blocks, respectively. For ResNet-20, the kernel size and dilation rate are set to 4 and in the first convolution layer before all residual blocks, respectively. These choices are based on the evaluations in Table A4 of Appendix A.3. From Table [ comparable LRF and parameters. The standard deviations for LPSC are limited, which shows that IPSS is not particularly sensitive to initializations. In some cases, the worst results also exceed those of the original networks with conventional convolutions and dilation convolutions by a margin.
of
we observe that LPSC outperforms dilation convolutions with
We also evaluate the cross convolution strategy for ResNet-20. We apply LPSC-CC to the layer before all blocks and all 3 x 3 layers of the first block with fixed p of 50. From Table [bb,, we observe that the cross convolution strategy further improves the performances.
Results with ResNet-110. We train ResNet- 1 10 with different convolutions on CIFAR-100 in Tab. 2 We follow the same setting for evaluating ResNer20, where x 5 LPSC kernels (Ir, Le, 9 = 2, 6, 3) are used to replace 3 x 3 convolutions in the first layer before all blocks in LPSC and in the first three layers with a fixed p of 50 in LPSC-CC. For the deeper model, the advantage of LPSC weakened, but LPSC-CC still improves ResNet 10 significantly.
7

---

Table 4: Results on the VOC 2012 dataset. "-" means that the results are not reported. ".* indicates our reproduced results.
Method
oAcc
mAce
fAcc
mloU
Deepl.abyy
0.701
Lav33
0.711
DeepLabv3+
0.9230
0.8332
0.8652
0.7144
DeepLabv3+LPSC
0.9273
0.8388
0.8714
0.7260
Table 5: Results the DRIVE dataset [55
indicates reproduced results.
Method
Sen
Acc
AUC
HED [56]
0.7364
0.9434
0,9723
Azzopardi al.
57
0.7655
0,9442
0,9614
Zhao 58
0.7420
0,9540
0.8620
U-Net 59
0.7537
0,9531
0,9601
Deep Vessel
60
0.7603
0,9523
0,9752
CE-Net [6]
0.8309
0.9545
0,9779
CE-Net" 61
0.8266 (0.0106)
0.9550 (0.0009)
0,9782 (0.0008)
CF-Net-LPSC-1
0.8300 (0.0079)
0.9552 (0.0011)
0.9782 (0.0008)
CF-Net-LPPC--2
0.8312 (0.0075)
0.9548 (0.0011)
0.9784 (0.0007)
Comparison of FLOPs. Comparisons of the average runtime per batch for using different convolu- tions in ResNetl 10 are shown in Tab LPSC runs slower than conventional convolution, but this is because which are highly optimized and very efficient for conventional convolution. LPSC can be greatly accelerated if it can be directly implemented with CUDA or by directly adapting the underlying code of convolutions in the integrated framework. On CIFAR10 10 with AlexNet, the FLOPS (recorded by the fvcore toolbox of conventional convolution, dilated convolution, and LPSC are 14.95M, 24.71M, and 11.42M respectively. LPSC has much lower FLOPs than other convolution methods.
of-the-shell conventional convolution modules in Pytorch to implement LPSC,
Results on the ImageNet dataset. ImageNet [53] contains 1.28 million training images and 50k validation images from 1000 classes. We again use the Pytorch implementatior of ResNet- 18 as the baseline. For LPSC, replace conventional convolution with LPSC in the first convolution layer before all blocks of ResNet- where the size 2R + 1, L,, Le. and g for LPSC kernels are 9, 3, 8, and 2, respectively. For LPSC-CC, in addition reduce p from 100 25 in the first layer, also replace a quarter of 3 x 3 kernels with LPSC kernels the first residual block (i.e., 25), where the size 2R 1, T.. Lo, and g for LPSC kernels in the block are 5, 2, 6, and respectively. The setting of these hyper-parameters for LPSC follows the suggestions the ablation study in Appendix A.2. Due the limitation of computing resources, we reduced the batch size and learning rate by 4 times. Other hyper-pauee parameters remain the same. We compare the mean top-1 accuracy and the standard deviation of the last epoches in Tab, 3 Both LPSC and LPSC-CC slightly improve the top-1 accuracy and the standard deviation of R8sNet-18.
4.2 Semantic segmentation experiments
LPSC can also be applied to other vision tasks. On the PASCAL VOC 2012 dataset 62
63 for
general image semantic segmentation, we adopt the Pytorch implementationS of Deeplabv33 3+ 64 with the MobileNet (65) backbone as the baseline. The training set is augmented by extra annotations provided in (66). Overall accuracy (oAcc), mean accuracy (mAcc), freqw accuracy (fAcc), and mean IoU (mloU) on the validation set are evaluated. In DeepLabv3+, the atrous spatial pyramid pooling (ASPP) module probes multi-scale features by applying atrous/dilated convolutions with three different rates. For DeepLabv3+LPSC, we replace the dilated convolution with the largest rate by LPSC in ASPP. The kernel size, L,, Lg, and g LPSC are set to 9, 2, 8. 2, respectively. Comparisons with the reported and reproduced results are shown in Tab. 4 LPSC improves DeepLabv3+ by margin of 1.1%% % on mloU. All hyper- parameerss and setups such as the learning rate, batch size, etc, remain the same, so the performance gains are only attributed to the proposed LPSSC.
"hurps//gittuubb bbanss//gituuh.co//haappapp http://hhhhh.commmmmmmm///yy/yes bttps://gttuub..co//aaaiiiiiooi
8

---

Figure 4: Visualization of the learned circular LPSC kemnels without center convolution in the first
convolution layer of Alexnet on the CIFAR-10 10 dataset.
(a)
(b)
(c)
(d)
Figure 5: Estimated RFs with (a) conventional convolution and (b) LPSC. The normalized gradient
map with (c) conventional convolution and (d) LPSC at sampled location.
On the DRIVE dataset 55 for retinal vessel detection, adopt CE-Net 61 the baseline. Sensitivity (Sen), accuracy (Acc), and AUC are evaluated on the test set. The dense atrous convolution (DAC) block of CE-Net uses four cascade branches with increasing numbers of dilated convolutions. For CE-Net-L we replace the dilated convolutions with rates and 5 by LPSCCs with sizes that LPSCs have the same LRFs with dilated convolutions. Ly- Le. and of LPSCs are set to 2, 6, 3, respectively. For CE-Net-LPSC-2 we increase the kernel sizes of LPSCs to 9 and 15, respectively, to further increase LRFs. We accordingly use more parameters by setting L,, Lg. and to 3, 8, 1.5, respectively. Other hyper-parameters remain the same 16) We run our models ten times and report the average performances. Comparisons with the reported results are shown in Tab. 5 Our LPSC achieves good generalization performances on medical image segmentation with limited training samples.
5f and 11 in DAC, respectively,
4.3 Visualization
Visualization of the learned LPSC kernels. Fig we visualize the learned LPSC kernels in the first convolution layer of AlexNet the CIFAR-I0 dataset. The 11 11 LPSC kernels have distance levels and direction levels. LPSC kernels, the closer to the center, the higher the regional resolution; the more outward, the larger the range for parameter sharing. We observe that the learned LPSC kernels capture some special local structures and contextual configuration. kernels sensitive specific directions, edges,
some kernels, the
weights for adjacent regions continuous; colors, or local changes; in some other kernels, specific combinations of regions are highlighted. More visualizations shown in Appendix A.4.
Comparison effective receptive field (ERF): Fig. 5(a) and (b) show the estimated RFs of SimpleVGGNei the default example using conventional convolutions and LPSCs in the first two layers by the gradient-based RF estimation" respectively. LPSC enlarges the estimated RFs from 14 14 to 22 x 22. The normalized gradient maps w.r.t. position of the output for estimating the RF using conventional convolutions and LPSCs are shown in Fig. c) and (d). With LPSC, gradients can be procggarpe to pixels of the input image.
bhtps://gituub.com/Guuuzzi http:///gttibbbccmmmmommaaccccct

---

Conclusion
In this paper, we have presented LPSC that naturally encodes the local contextual structures. LPSC distinguishes regions with different distance levels and direction levels, reduces the resolution of remote regions, and reduces the number of parameters by weight sharing for pixels in the same region. The LRF of LPSC increases exponentially with the number of distance levels. We impose regularization on the parameters and implement LPSC with conventional convolutions by log-polar space pooling and separable center pixel convolution. We analyze the interests and drawbacks of LPSC from different aspects. We empirically show the effectiveness of the proposed LPSC on five datasets for classification and segmentation tasks.
Acknowledgments
The authors would like thank the anonymous reviewers for their valuable comments. This work was supported in part by the National Natural Science Foundation of China No. 61976206 and No. 61832017, Beijing Outstanding Young Scientist Program NO. BJJWZYJHO1 10002009,, Beijing Academy of Artificial Intelligence (BAAI), the Fundamental Research Funds for the Central Universities, the Research Funds of Renmin University of China 21XNLG05, and Public Computing Cloud, Renmin University of China. This work was also supported in part by Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, and Public Policy and Decision-making Research Lab of Renmin University of China.
References
[1] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
2324, 1998.
[2] Alex Krizhevsky, Ila Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[3] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou Global filter networks for image classification. Advances in Neural Information Processing Systems, 34, 2021.
[4] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13733-13743 2021.
[5] Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters- improve semantic segmentation by global convolutional network. Proceedings of the IEEE conference computer vision and pattern recognition, pages 4353 2017.
[6] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
[7] Liang-Chiet Chen, George Papandrcou. Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-848, 848, 2017.
[8] Matthias Holschneider, Richard real-time algorithm for signal analysis with the help of the wavelet transform. In Wavelets, pages 286-297. Springer, 1990.
Martinet, Jean Morlet, and Ph Tchamitchian. A
[9] Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 2015.
[10] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks, Advances in neural information processing systems, 29:550-558, 2016.
10

---

[11] Wenjie Luo. Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances in neural information processing 2016.
systems, pages
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
[13] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[14] Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Scungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invarial neural networks. In International Conference Machine Learning, pages 3744-3753 PMLR, 2019.
[15] Hongyu Ren, Hanjun Dai, Zihang Dai, Mengjiao Yang, Jure Leskovec, Dale Schuurmans, and Bo Dai. Combiner: Full attention transformer with sparse computation cost. Advances in Neural Information Processing Systems, 34, 2021.
[16] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. arXiv preprint arXiv:2106.04200 2021.
[17] Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape context: A new descriptor for shape matching and object recognition. In Advances in neural information processing systems, pages 831-837, 2001.
[18] Serge Belongie, Jitendra Malik, and Jan Puzicha. Shape matching and object recognition using shape contexts. IEEE transactions on pattern analysis and machine intelligence, 24(4):509-522 2002.
[19] Bing Su, Xiaoqing Ding, Liangrui Peng, and Changsong Liu. A novel baseline-independent feature set for arabic handwriting recognition. In International Conference on Document Analysis and Recognition, pages 1250-1254. IEEE, 2013.
[20] Alexander C Berg and Jitendra Malik. Geometric blur for template matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2001.
[21] Jonathan T Barron, Mark D Biggin, Pablo Arbelaez, David W Knowles, Soile VE Keranen, and Jitendra Malik. Volumetric semantic segmentation using pyramid context features. In Proceedings of the IEEE international conference on computer vision, pages 3448-3455, 2013.
[22] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 2014.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[24] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fen: Object detection via region-based based fully convolutional networks. In Advances neural information processing systems, pages 379-387, 2016.
[25] Laura Sevilla-Lara, Deqing Sun, Varun Jampani, and Michael Black. Optical flow with semantic segmentation and localized layers. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 3889-3898, 2016.
[26] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint a7Xiv:1706.05587, 2017.
[27] M. Yang, K. Yu, Z. Chi, Z. Li, and K. Yang. Denseaspp for semantic segmentation in street scenes. Proceedings of the IEFE conference on computer vision and pattern recognition, 2018.
11

---

[28] Liang-Chied Chen, Maxwell D. Collins, Yukun Zhu, George Papandreou. Barret Zoph, Florian Schroff. Hartwig Adam, and Jonathon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, page 8713-8724, 2018.
[29] Rui Zhang. Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. Scale-adaptive convolutions for scene parsing, In Proceedings of the IEEE international conference computer vision, pages 2031-2039, 2017.
[30] Rui Zhang, Sheng Tang, Yongdong Zhang, Jintao Li, and Shuicheng Yan. -easpective-ddaptive convolutions for scene parsing. IEEE transactions on pattern analysis and machine intelligence, 42(4):909-9244 2019.
[31] Panqu Wang. Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell. Understanding convolution for semantic segmentation. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 1451-1460. IEEE, 2018.
[32] Tianyi Wu, Sheng Tang, Rui Zhang, Juan Cao, and Jintao Li. Trec-structured kronecker convo- lutional network for semantic segmentation. In IEEE International Conference on Multimedia and Expo, pages 940-945 IEEE, 2019.
[33] Matthew D Zeiler, Dilip Krishnan, Graham W Taylor, and Rob Fergus. Deconvolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 228-2535. IEEE, 2010.
[34] Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature lcarning. In Proceedings of the IEEE international conference on computer vision, pages 2018-2022 IEEE, 2011.
[35] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pages 2017-2025, 2015.
[36] Yunho Jeon and Junmo Kim. Active convolution: Learning the shape of convolution for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pages 4201-4209, 2017.
[37] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable. better results, In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[38] Hang Gao, Xizhou Zhu, Steve Lin, and Jifeng Dai. Deformable kernels: Adapting effective receptive fields for object deformation, arXiv preprint urXiv:1990.00000 2019.
[39] Zhun Sun, Mete Ozay, and Takayuki Okatani. Design of kernels in convolutional neural networks for image classification. In Proceedings of the European Conference on Computer Vision, pages 51-66, 2016.
[40] Alexander Krull, Fim-Oliver Buchholz, and Florian Jug. Noise2void learning denoising from single noisy images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, page 2229-2137,21 2019.
[41] Guiguang Ding Xiaohan Ding, Yuchen Guo and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In Proceedings of the IEEE/CVF International Conference on Computer Vision page 1911 1-120,, 2019.
[42] Kun He, Chao Li, Yixiao Yang, Gao Huang, and John E. Hopcroft. Integrating large circular kernels into cnns through neural architecture search. arXiv preprint arXiv:2107.02451, 2021.
[43] Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. Interleaved group convolutions. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[44] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6888550 2018.
12

---

[45] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258. 2017.
[46] Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Mareus Rohrbach, Shuicheng Yan, and Jiashi Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In Proceedings of the IEEE international conference on computer vision, pages 3435-3444, 2019.
[47] Junran Peng, Ming Sun, Zhaoxiang Zhang, Tieniu Tan, and Junjie Yan. Pod: Practical object de- tection with scale-sensitive network. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9607-9616, 2019.
[48] Duo Li, Anbang Yao, and Qifeng Chen. Psconv: Squeezing feature pyramid into one compact poly-scale convolutional layer. arXiv preprini arXiv:2007.06191, 2020.
[49] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention vision models. arXiv preprint arKiv:1906.0553 2019.
[50] Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. arXiv preprint arXiv: 1eee:99111.5000 2019.
[51] Carlos Esteves, Christine Blenc-Blanchette, Xiaowei Zhou, and Kostas Danilidiss Polar transformer networks. International Conference on Learning Representations, 2018.
[52] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical Report, 2009.
[53] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):111-252, 2015.
journaa
[54]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024 8024-8035, 2019.
[55] Joes Staal, Michael D Abràmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Gin- neken. Ridge-based vessel segmentation in color images of the retina. IEEE transactions medical imaging, 23(4):501-505 2004.
[56] Saining Xie and Zhuowen Tu. Holistically-nestee edge detection. In Proceedings of the IEEE international conference on computer vision, pages 1493-1103, 2015.
[57] George Azzopardi, Nicola Strisciuglio, Mario Vento, and Nicolai Petkov. Trainable cosfire filters for vessel delineation with application to retinal images. Medical image analysis, 19(1):4655, 2015.
[58] Yitian Zhao, Lavdie Rada, Ke Chen, Simon P Harding, and Yalin Zheng. Automated vessel segmentation using infinite perimeter active contour model with hybrid region information with application to retinal images. IEEE transactions on medical imaging, 34(9): 1 1897- 2015.
[59] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 334-241.2 Springer, 2015.
[60] Huazhu Fu, Yanwu Xu, Stephen Lin, Damon Wing Kee Wong, and Jiang Liu. Deepvessel: Retinal vessel segmentation via deep learning and conditional random field. In International conference on medical image computing and computer-assisted intervention, pages 132-139. Springer, 2016.
13

---

[61] Zaiwang Gu, Jun Cheng, Huazhu Fu, Kang Zhou, Huaying Hao, Yitian Zhao. Tianyang Zhang. Shenghua Gao, and Jiang Liu. Ce-net: Context encoder network for 2d medical image segmentation. IEEE transactions medical imaging, 38( 28(10))2281-2222, 2019.
[62] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-336 2010.
[63] Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-136, 2015.
[64] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference Computer Vision, pages 801-818, 2018.
[65] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: w 04446, 2017.
[66] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In Proceedings of the IEEE international conference on computer vision, pages 991-998. IEEE, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.4
(c) Did you discuss any potential negative societal impacts of your work? [No] Because this
work presents a general convolution module, we did not see any particular foreseeable ethics consequence.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the supplementary files. Our code is available at atthtps:: /tap:::///ithbss com wtt.ccoom//oiuuee Bing Poog-Polar-ppal -onooee---oooolooo utioo
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] See Section [
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section
(b) Did you mention the license of the assets? [No] The assets we adopted are totally public for research purposes.
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
See the supplementary files.
14

---

(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [No] All datasets used in this work are public for research purposes. (c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] All used datasets do not contain personally identifiable information or offensive content
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "zDcDLKAYkn", "reviewer": "Reviewer_aXhX", "summary": "This paper presents a novel variant of convolutional layer (log-polar space convolution LPSC) in neural networks, wherein the convolution kernel is not rectangular, but rather elliptical with different sized regions, which are pooled. It also describes a log-polar space pooling for convenient implementation. Finally, it applies the proposed method to various neural network architectures and tasks thereof.", "questions": "Question 1: Although only a bit earlier work, the authors should discuss the relation to this paper : He, Kun, et al. \"Integrating Large Circular Kernels into CNNs through Neural Architecture Search.\" arXiv preprint arXiv:2107.02451 (2021). \nThat work also cites other papers that should be looked at (some of these are already cited by the current paper): “... the deformable convolution (Dai et al., 2017; Zhu et al., 2019) … Similarly, the deformable kernel (Gao et al., 2020) …including quasi-hexagonal convolution (Sun et al., 2016), blind-spot convolution (Krull et al., 2019), asymmetric convolution (Ding et al., 2019), etc.”\n\nQuestion 2: Have the authors studied how much performance comes from the polar coordinates and how much from the growing size of areas that are pooled (further areas are pooled more). Could the pooling be done in the normal rectangular grid and how much would be gained by that.\n", "limitations": "The authors properly note the drawbacks of additional hyperparameters and memory overhead. For other limitations, see above for weaknesses and questions in this review.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n--------------\n\nStrength 1: The idea of log-polar convolution makes intuitively sense since the rectangular pixel quantization is more an artifact of the capture and storage systems rather than a feature of nature.\n\nStrength 2: A practical pooling method for easily incorporating the method into existing SW, HW and networks.\n\nWeaknesses:\n------------------\n\nWeakness 1: There are other non-typical kernel shapes that have been proposed in the literature (some are cited in the paper). The authors should compare their method to the best of the previously published methods.\n\nWeakness 2: The method does not seem to bring a huge gain, but result into slower training and inference and additional hyperparameters. While this should still be ok for data-limited setup because of the potential regularization effect of the proposed system, it is not clear whether it is better to use the proposed method or just a somewhat larger CNN to achieve similar gains. Please study the regularization / generalization effect of the proposed method further.\n\nWeakness 3: The motivation of the log-polar space convolution would be better if natural image statistics were used to justify it. Now it stands out as rather ad-hoc method.\n\nAfter the rebuttal\n---------------------\nI have reviewed the author feedback and the they have done a good job in clarifying their work. I have reflected this in the score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eJNBzl9TPzg", "reviewer": "Reviewer_4RP8", "summary": "# Summary\nIn this paper, an elliptic convolution kernel is proposed, and its local receptive field is adaptively divided into different regions according to the relative direction and logarithmic distance\n", "questions": "1.In Table 1, the accuracy of using alexnet in cifar10 and cifar100 is only about 77 and 45 respectively. Although it is improved compared with other convolution cores, it is also difficult to be convincing. At the same time, the improvement on vgg19 and resnet20 is too subtle to verify the effectiveness of the convolution kernel.More experiments are needed to verify the validity of the statements in the paper.\n\n2.Why choose resnet18 instead of resnet50 for experiments on Imagenet.\n\n3.In Table 2 and table 3,Why not use dilated convolution with your method.\n\n4.As mentioned in Section 4.1 of the paper, the advantages of LPSC are weakened for deeper models. Please explain why.\n\n5.More experiments are needed to prove that LPSC can obtain more effect of improving receptive field than dilated convolution. At the same time, improving receptive field is mainly used in semantic segmentation, so more experiments on semantic segmentation are needed.", "limitations": "The idea is very good, but I need better experimental results to support it.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1.The proposed method is very novel\n# Weaknesses\n1.The comparison objects selected in the comparison experiment are very old, and the performance has not been greatly improved.\n\n2.Poor effect in deep network model.\n\n3.Insufficient experiments in image segmentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ZswnsMb0eRM", "reviewer": "Reviewer_aBNv", "summary": "This paper introduces a new convolution operator. Instead of using a rectangle convolution kernel following the grid structure of the input data, the authors propose to define convolution kernels in a log-polar space. Specifically, the kernel weight depends on the distance and direction wrt to the center, and the density of weights is inversely proportional to the distance to the center. Because the weights are not defined on the same regular grid as the input data, a pooling operation is performed before applying the convolution kernel. The main benefit of the proposed approach is that less parameters are needed when the size of the receptive field increases, based on the insight that pixels adjacent to the center should have more contributions to the output. Empirical results on multiple tasks and models show that the proposed method outperforms models using only standard convolution kernels. ", "questions": "1. Given that the proposed method introduces additional computation and parameters, the overhead should be considered in the evaluation, e.g. compare overhead vs accuracy instead of single point accuracy\n2. Validation should be use to determine the meta-parameters for a fair comparison\n", "limitations": "1. There doesn't seem to be any obvious negative social impact for this work.\n2. The authors describe some of the limitations of the proposed method, although some additional information may help understanding the limitations, e.g. the exact memory overhead. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths\n  * The key idea is intuitive and well motivated\n  * The proposed approach is generic and is applicable to most existing CNNs, and it is easy to implement\n  * The results suggest that the proposed convolution operator consistently perform better than standard convolution\n* Weakness\n  * The method requires additional computation and parameter during inference time, and the experiment does not really perform apple to apple comparisons\n  * The accuracy improvement is not significant consider that it introduces additional computation and memory overhead\n  * Using the same setup does not imply fair comparison. One should optimize each model independently for a fair comparison.\n  * The proposed method introduces additional meta-parameters, which are determined by the accuracy on the test set according to L281 and leads to unfair comparison\n\n* After rebuttal\n  * The concerns regarding the accuracy are properly addressed, i.e. training setup and meta-parameters\n  * The claim for lower computational cost is still not clearly explained. L222 assumes standard convolution and LPSC has comparable M and N. But in practice, most standard convolution has M=N=1, while LPSC has M and N > 1. Therefore, it's not clear why the flops is lower when 3x3 kernels are used.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "R9VtCgRC11Y", "reviewer": "Reviewer_AZ8Z", "summary": "\nIn conventional CNNs, large effective receptive fields (ERF) are generally achieved through a cascading of multiple layers of convolutions and pooling operations. The authors note that this requirement may have an impact on the model performance; it may cause vanishing gradients and moreover may not actually have the desired effect of an increased ERF. The effective receptive field theory states that a more effective way of increasing ERF is increasing the kernel size. To this end, the authors explore a parameter efficient way of increasing through defining their kernels in log-polar space. The authors carefully describe and motivate their method and its relation to conventional convolutions, and experimentally show the validity of their method on a range of experiments. Furthermore, the authors show a compute-efficient implementation using conventional convolution operations.", "questions": "1. Do you have an intuition for when you would want to use ellipsoid LPSCs? Are there any specific use-cases that come to mind?\n2. Your method actually gives a definition for kernel values over the entire continuous input space. Does this mean your method could be readily applied to irregularly sampled data? Are there any specifics of your method that would inhibit you from doing this? Have you attempted this?\n3. The authors motivate their log-polar space kernel definition by saying that in natural images often correlation between local pixels is higher than correlation between more distant pixels and therefore we can assign the same parameters to larger patches of pixels further away. I don’t think this argument necessarily makes sense; indeed natural images are locally correlated, but I believe this only says that local pixels should be assigned larger weights values, not that larger patches of pixels further away should be assigned identical weight values. Indeed, your method allows for larger receptive field sizes, but because of the increased weight sharing further from the center point, this comes at the trade-off of lower resolution information at higher distances correct? Could you touch upon this?\n4. In this same line, I would like to see the authors more explicitly expand on what they think ultimately results in the performance increase of their method over conventional CNNs? Do you think it is a result of the increased receptive field, the way in which you treat local and nonlocal information at different resolutions, both?\n5. Also, can you touch upon some use-cases in which this assumption of locality necessarily does not hold, but for which conventional CNNs are still used in practice? For example, how would your method perform on speech speech signals, which are generally sampled at very high rates and may exhibit very non-local patterns?\n\n**Post-rebuttal response**\nI would like to thank the authors for their extensive responses to my questions, and the questions of other reviewers. I appreciate the fact that you included a discussion for multiple of the concerns and questions raised by me and other reviewers in your manuscript, I feel this improves the quality of the work.\n\nI noticed that my concern regarding the motivation for your specific approach to weight sharing in convolution kernels was shared by other reviewers as well. I think you greatly improved your motivation by including an analysis on pixel statistics.\n\nThe additional experiments and comparisons made by the authors in their revision strengthen this submission further, and show the relevance of this approach by showing it performs even when compared to more recent approaches.\n\nThe authors additionally addressed a number of limitations of their work in their revision, which make the work more transparent.\n\nIn light of these improvements, I am slightly raising my recommendation. I thank the authors for an interesting submission!\n", "limitations": "Authors briefly discuss the limitations of their work; it adds hyperparameters and isn’t very memory efficient. What I would like the authors to touch upon are limitations which may be present in their method inherently, as a result of using the log-polar space to define convolution kernels (see also my questions above).\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors give a very clear overview of the relevance and motivation behind their method, by first highlighting limitations of current approaches to larger ERFs and from there introducing their approach to address these issues. As far as I know, their work is highly original.\n\nSection 3.1 contains a comprehensive overview of the method, although I get the sense that an illustration visually explaining your method (i.e. show how R relates to the kernel size, how $L_{\\theta}, L_r$ change the kernel), as you did for initial angle and eccentricity in fig 2, could be very helpful in getting the specifics across. In general, the paper is written in a clear manner, but the number of inline equations and parameters make it somewhat hard to read at points.\n\nThe choice of experiments is motivated well; the goal of the authors is to make a direct comparison between conventional convolutional layers and the introduced log-polar space convolutions. The experiments themselves seem to be chosen fairly for this goal, and decidedly show the improvement of LPSCs over conventional convolutions.\n\nI appreciate that the authors chose to investigate how their framework could be implemented through highly optimised conventional convolutions. This allows for a much more direct comparison with conventional convolutions.\n\nMisc:  Under fig 2a and in line 171, 187 “is slide” -> “is slid”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept