Dilated convolution with learnable spacings
Authors: Ismail Khalfaoui Hassani
Thomas Pellegrini
Timothée Masquelier
Venue: ICLR 2023 poster (2023)
Keywords: deep learning
convolution
dilated convolution
receptive field

Abstract: Recent works indicate that convolutional neural networks (CNN) need large receptive fields (RF) to compete with visual transformers and their attention mechanism. In CNNs, RFs can simply be enlarged by increasing the convolution kernel sizes. Yet the number of trainable parameters, which scales quadratically with the kernel's size in the 2D case, rapidly becomes prohibitive, and the training is notoriously difficult. This paper presents a new method to increase the RF size without increasing the number of parameters. The dilated convolution (DC) has already been proposed for the same purpose. DC can be seen as a convolution with a kernel that contains only a few non-zero elements placed on a regular grid. Here we present a new version of the DC in which the spacings between the non-zero elements, or equivalently their positions, are no longer fixed but learnable via backpropagation thanks to an interpolation technique. We call this method “Dilated Convolution with Learnable Spacings” (DCLS) and generalize it to the n-dimensional convolution case. However, our main focus here will be on the 2D case. We first tried our approach on ResNet50: we drop-in replaced the standard convolutions with DCLS ones, which increased the accuracy of ImageNet1k classification at iso-parameters, but at the expense of the throughput. Next, we used the recent ConvNeXt state-of-the-art convolutional architecture and drop-in replaced the depthwise convolutions with DCLS ones. This not only increased the accuracy of ImageNet1k classification but also of typical downstream and robustness tasks, again at iso-parameters but this time with negligible cost on throughput, as ConvNeXt uses separable convolutions. Conversely, classic DC led to poor performance with both ResNet50 and ConvNeXt. The code of the method is based on PyTorch and available at: https://github.com/K-H-Ismail/Dilated-Convolution-with-Learnable-Spacings-PyTorch.

Published as a conference paper at ICLR 2023
DILATED CONVOLUTION WITH LEARNABLE SPACINGS
Ismail Hatalfaoui-Hassaie Artificial and Natural Intelligence Toulouse Institute (ANITI)
Université de Toulouse, France
ismail .haal
Thomas Pellegrini
Timothée Masquelier
IRIT, ANITI, Université de Toulouse, CNRS, Toulouse INP, UT3, France thomas, melllegrinnee iiriii.iiriirfr...
CerCo UMR 5549
CNRS Université de Toulouse, France t. timothee masque wesssueleerree i fr
ABSTRACT
Recent works indicate that convolutional neural networks (CNN) need large re- ceptive fields (RF) to compete with visual transformers and their attention mecha- nism. In CNNs, RFs simply be enlarged by increasing the convolution kernel sizes. Yet the number of trainable parameters, which scales quadratically with the kernel's size in the 2D case, rapidly becomes prohibitive, and the training is notoriously difficult. This paper presents a new method to increase the RF size without increasing the number of parameters. The dilated convolution (DC) has already been proposed for the same purpose. DC can be seen as a convolution with kernel that contains only a few non-zero elements placed regular grid. Here we present new version of the DC in which the spacings between the non- zero elements, equivalently their positions, are no longer fixed but learnable via backpropagation thanks to an interpolation technique. We call this method the
"Dilated Convolution with Learnable Spacings" (DCL.S) and generalize n- n-dimensional convolution case. However, our main focus here will be on the 2D case for computer vision only. We first tried our approach on ResNet50: we drop-in replaced the standard convolutions with DCLS ones, which increased the accuracy ImageNetlk classification at iso iso- parameters, but at the expense of the throughput. Next, used the recent ConvNeXt oftte-of-the-ane art convolutional ar- chitecture and drop-in replaced the depthwise convolutions with DCLS ones. This not only increased the accuracy of ImageNetlk classification but also of typical downstream and robustness tasks, again at iso- parameters but this time with negli- gible cost on throughput, as ConvNeXt uses separable convolutions. Conversely, classic DC led to poor performance with both ResNet50 and ConvNeXt. The code of the method is based on PyTorch and available.
INTRODUCTION
The receptive field of a deep convolutional network is a crucial element to consider when dealing with recognition and downstream tasks in computer vision. For instance, a logarithmic relationship between classification accuracy and receptive field size was observed in Araujo et al. (2019). This tells us that large receptive fields are necessary for high-level vision tasks, but with logarithmically decreasing rewards and thus higher computational cost to reach them.
Recent advances in vision transformers (Dosovitskiy et al. 2020) and CNNs (Liu al. 2022b Ding et al.] 2022 Trockman & Kolter) 2022; Liu et al. (022a)) highlight the beneficial effect that a large convolution kernel can have, compared to the 3 x 3 kernels traditionally used in previous state-of-the-art CNN models (He et al. 2016). However, when naively increasing the kernel size, the accuracy rapidly plateaus or even decreases. For example, in ConvNeXt, the best accuracy was achieved by 7 7 kernel (Liu et al. 2022b a). Using structural re- parameterizztion trick, Ding et al.] (2022) demonstrated the benefit of increasing the kernel size up to 31 by 31. Thereafter, Liu et al. (2022a) showed that there still room for improvement by moving to 51 by 51,

---

Published as a conference paper at ICLR 2023
ing the depthwise implicit matrix multiplication (gemm) method developed by Ding al. (2022) and for which the implementation has been integrated into the open-sourced framework MegEngine (Megvii 2020). in addition to spatial separation of the depthwise kernel followed by accumu- lation of the resulting activations. Yet, all these improvements have cost in terms of memory and computation, and does not seem possible to increase the size of the kernels indefinitely.
One of the first approaches that allow inflating the receptive field of a convolutional layer without increasing the number of learnable parameters nor the computational cost is called dilated convolu- tion (DC). DC or "atrous convolution" was first described in Holschneider et al. (1990) and [Shensa (1992), , under the name "convolution with dilated filter" before being referred as "dilated con- volution" in Yu & Koltun (2015). The purpose of this approach is to inflate the convolutional kernel by regularly inserting spaces (i.e. zeros) between the kernel elements, as depicted in Figure 2h. The spacing between elements is thus constant, it is hyper-parameter usually referred to "dilation" or "dilation rate". Despite early successes in classification since Yu et al. (2017). and its even
ComiieXi Del
SLaK
StalK with sparsity
RepLKNes-31B
Comfeeet
ImageNetlk
Dianttr
89 122
200M paramenes
3.5
4.5
Latency img)
Figure Classification accuracy on ImageNet-IK as a function latency (i.e. inverse of the throughput). Dot diameter corresponds to the number of parameters.
more convincing results in semantic segmentation Sandler et al. (018); Chen et al. 2017 2018 and object detection Lu et al. (2019). DC has gradually fallen out of favor and has been confined downstream tasks such those described above. Without much success, Ding et al. (2222) tried to implement DC in their ReplKNet architecture. Our own investigation on ResNet and ConvNeXt with standard dilated convolution (Section 4.2) will lead a similar conclusion. The failure of this method for classification tasks could be attributed to the great rigidity imposed by its regular grid as discussed in Wang 2018).
In this context, we propose DCLS (Dilated Convolution with Learnable Spacings), new convolu- tion method. DCLS, the positions of the non-zero elements within the convolutional kernels are leamned in gradient- based manner. The inherent problem of non-differentiabiiiiy due to the integer nature of the positions in the kernel is circumvented by interpolation (Fig ( DCLS is a differen- tiable method that only constructs the convolutional kernel. To actually apply the method, either use the native convolution provided by Py PoTch or a more advanced one such as the depthwise implicit gemm convolution method (Ding et al. 2022), using the constructed kernel. DCLS comes in six sub-versions: ID, 2D, 3D and what we call N-MD methods, namely: "2-1D, 3-1D and 3-2D" where N-dimension kerne is used but positions learned only along M dimension(s). The main
could
2

---

Published as a conference paper at ICLR 2023
focus of this paper will be the 2D version for which detail mathematical proofs, implementation specificities and results on image classification, downstream and robustness tasks.
The principal motivation of DCLS is to explore the possibilityy of improving the fixed grid imposed by the standard DC via learning the spacings in an input-independent way. Instead of having a grid of kernel elements like in standard and dilated convolutions, DCLS allows arbitrary number of kernel elements (Fig. 2 We refer to this free tunable paper, we in order to be iso fewer parameters than the baselines will compare ourselves to. Conversely, we refer to the size of the kernel, or rather the maximum size in which the kernel elements are allowed move inside the dilated kernel, the "dilated kernel size". It is also tunable
as "kernel count". this
The positions of kernel elements in DCLS are randomly initialized. and are allowed move throughout the learning process within the dilated kernel size limit. We will then show how sharing positions across multiple blocks of a same convolution stage could further increase aceuracy while reducing the number of learnable parameters. This, together with other learning techniques, em- pirically and consistently improve the overall performance the method. They are summarized in Section 3
12
36
8
2-
12
24
36
10
6
6-
48
96
9G
4
4-
-5-1
0-
-36
-24
-12
84
0
1
2
0
2 4
6
0 2
(8)
(b)
(C)
(d)
Figure 2: (a): a standard 3 x kernel. (b): a dilated 3 x kernel with dilation rate 4. (c): a 2D-DCLS kernel with 9 kernel elements and dilated kernel size of 9. Each weight is spread over up to four adjacent pixels. (d): a 2D-DCLS kernel with kernel elements and still a dilated kernel size of 9.
2 KERNEL CONSTRUCTION IN DCLS
Our method entails that we learn the float coordinates (p² and p² the 2D case for example) for every weight of the dilated kernel (in addition to the actual weights themselves). The positions or coordinates of the weights within the convolution kernel are conceptually integer parameters, yet in order to compute their derivatives, should consider them as float parameters. This problem of integer positions learning is smoothed by making use of bilinear interpolation that will be described in equations and
2.1
NOTATION AND PRELIMINARIES
We denote by [] the floor function and we define its derivative by the zero function.
def
Yo R,
(1)
We denote by m N the number of kernel elements inside the constructed kernel and refer to it as the "kemnel count". Moreover, we denote respectively by 81, 82 E N* x N the sizes of the constructed kernel along the x-axis and the y-axis. The latter could be seen the limits of the dilated kernel, and we refer to them as the "dilated kernel size".
The 51 $2 matrix space over is defined as the set of all s x s2 matrices over I. and is denoted Ma,.s2 (R).
The characters w, p² and p² respectively stand for the weight, the position of that weight along the x-axis (width) and its position along the y-axis (height) in the scalar case while the bold
3

---

Published as a conference paper at ICLR 2023
(wi)isism p' (p) )isism and p² (si)i respectively stand for the weight, the width- position of that weight and its height-posee sight-position in the vector case.
2.2 MATHEMATICAL FORMULATION
The mathematical construction of the forward pass as well as the proofs for the derivations used in the backward pass in Appendix 7 described by the following function:
This construction relies bilinear interpolation and could be
F:
R
x
R
X
R
Ms (RR)
w.p'p p²
K
flw,, Pk: Pt:)
With defined as follows:
f: J: R M
(3)
w. p.p ppppp= + K
where Vi E [1. 81]. Vj E [p]
w (1
72) if = |p
j [F²
if i p
+ 1,
p2
Kij
w (1
if i = Ip
j = 1p
+1
(4)
w
i = Ip +1, j = 1 11
0
otherwise
and where the fractional parts are:
1p'1 and ,2 1
(5)
We invite the reader look Appendix 7 for more details the mathematical proofs and deriva- tives that permits to calculate the gradients of a differentiable loss function with respect to the weights and their positions. Those derivations lead to the DCLS kernel construction algorithm described for the 2D case in pseudo-code in Appendix Furthermore, the real code for the kernel construction in ID, 2D, and 3D cases is included Appendix | This code written in native Py PyTorch language, with classical modules, and does not require any compilation or adaptation.
LEARNING TECHNIQUES
So far, have seen how implement the DCLS method. We now turn the techniques that allow us to get the most out of the method. In what follows, we list the training techniques that we have retained and for which we have noticed a consistent and empirical advantage on validation accuracy.
Weight decay: weight decay is regularization method widely used in variety of deep learning models. Though its beneficial effect on generalization, we noticed that when applied to the kernel positions in DCLS method, weight decay tends to "artificially" over-concentrate the positions around the center of the kernel, resulting in poorer accuracy. Therefore, we set this hyperparameter for the kernel positions and kept it unchanged for all the other parameters.
Positions initialization: the DCLS positions tend cluster around the center of the RF through- out the learning process (see Appendix TO). In an attempt to facilitate learning, we chose an initial distribution close the one obtained at the end of training, that centered normal law of standard deviation 0.5. Yet in practice, the uniform law gives similar performance.
Positions clamping / overlapping: kernel elements that reach the dilated kernel size limit are clamped. This is done at the end of every batch step to force kernel positions to stay within limits. Agglutination around those limits
sometimes be observed and this indicates that the dilated kernel size too small and should be enlarged. Positions of different kernel weights (or their interpolations) could also overlap. They added together in such case.
4

---

Published as a conference paper at ICLR 2023
Dilated kernel size tuning: empirically tuned it using the remark above. For simplicity, we used the same dilated kernel size in all the model layers (7 for ResNet-50-dels and 17 for ConvNcXt-dels; larger values did not bring any gain in accuracy). Note that increasing the dilated kernel size has no cost on the number of trainable parameters, but has a cost on throughput, especially when using non-separable convolutions. Besides, the convolution algorithm (which is the most time-consuming part of DCLS) that is used after the kernel construction (whether it is the native one or the depthwise implicit gemmn one) does not leverage the kernel sparsity. Thus, the kernel count does not impact the throughput; only the dilated kernel size does. The development of a convolution method that takes into account sparse kernels, such as [Kundu et al. (2019), could further help to reduce the throughput gap between DCLS convolution and standard dilated convolution.
Kernel count tuning: said before, we have set this hyper-paramen to the maximal integer value that allows us to be below the baselines to which we compare ourselves in terms of the number of trainable parameters. Note that adding one element to the 2D-DCLS kernel leads to having three additional learnable parameters: the weight, its vertical and its horizontal position. For simplicity, we used the same kernel count in all the model layers.
Positions learning rate scaling: we found that kernel positions could benefit from special scaling when it comes to the learning rate. As their magnitude is different from regular weights, we scaled the learning rate of all the kernel positions by factor of 5. This is the best factor we found empirically. Custom- made schedulers for positions have been tested, but scaling the learning rate while using the same scheduler as for the kernel weights remained the best choice. Interestingly, we found that throughout learning, the average speed of the positions follows precisely the shape of the learning rate scheduler curve (see Appendix IT].
Synchronizing positions: we shared the kernel positions across convolution layers with the same number of parameters (lypically, those belonging to a same stage of the ConvNeXt/ResNet model), without sharing the weights. Positions in this kind of stages were centralized in common param- eters that accumulate the gradients. This constraint has surprisingly enhanced the accuracy while reducing the number of extra parameters dedicated to positions (an ablation of this technique led to a 0.13% accuracy drop on ImageNet1k ks with ConvNeXt-T-dels)
Repulsive loss: following the work of Thomas al. (2019) on 3D cloud points, we implemented the repulsive loss for the DCLS kernel positions to discourage multiple elements from overlapping. Despite a slight advantage with the ResNet-50-dcls model, this technique did not significantly improve the results with ConvNeXt-deks:
Depthwise implicit gemm: This method has been developed by Ding et (2022) and integrated in Megvii (2020), it has for goal to modify the im2col algorithm as well as the matrix multiplica- tion that follows, both used in the native 2D convolution method of PyTorch, by a very efficient algorithm which does not build explicitly the im2col tensor. The advantage of this method is that it is much faster than the native one for large kernel sizes, without affecting the convolution speed for small kernel ones. In our experiments, the largest dilated kernel size is 17, therefore this method is not absolutely necessary. However, the DCLS user can choose to use it instead of the native method, which will improve the throughput.
RESULTS AND DISCUSSION
4.1 SETTINGS
We started with an exploratory study on ResNer-50, where we drop-in replaced all the 3 x 3 con- volutions of the model by 2D-DCLS ones. For that, we used lightweight procedure named the "A3 configuration", described in Wightman et al. (202T). We then moved to the ConvNeXt mod- els, where we limited our studies to its three first variants namely: the tiny, the small and the base models with input crops of size 224 x 224 (Liu et al. 2022b). Here, we drop-in replaced all the depthwise convolutions of the model by DCLS ones. We reconducted all the experiments, and eval- uated the seed sensitivity for the Co model by calculating the standard deviation of the t accuracy on three different seeds, for the tiny variant. We found the standard deviation to be +0.04, which is compatible with what was found in Liu et al. (2022b). Given this reasonably low variability, the remaining experiments were done on one seed only. Code and scripts to reproduce the training are available (see reproducibility statement
5

---

Published as a conference paper at ICLR 2023
4.2 EMPIRICAL EVALUATIONS ON IMAGENETIK
In the following, we report the top-1 accuracies found on the ImageNetlk validation dataset (Deng al. 2009), using ImageNetlk training dataset only.
Using ResNet-50. Table presents the results obtained for the ResNet-50 experiment using the A3 configuration. The aim of this study is not to exceed the state-of-the--at for this particular configuration, but rather to give a first experimental evidence of the relevance of the DCLS method with non-separable convolutions and with one of the most popular CNN architectures in computer vision. We can observe that when using the standard dilated convolution, the results only get worse as we increase dilation rate. Moreover, increasing convolution increases the accuracy but the expense of not only the throughput, which decreases, but also of the number parameters, which triples.
kernel size (3 + 7) in the case standard
With fewer parameters, the ResNet-50-dcls model surpasses the baseline but the expense of the throughput. This last disadvantage is due to the fact that ResNet-50 uses non-separable convolutions. We will see in Table for the ConvNeXt model, this extra cost in throughput is minimized thanks to the depthwise separable convolution.
kernel size / count
throughput (image
Top-1 acc.
model
# param.
FLOPs
(crop 160)
ResNet-50
3/9
25.6M
4.1G
1021.9
75.8
ResNet-50
7/49
75.9M
12.3G
642.6
77.0
ResNet-50
3/9
2
25.6M
4.1G
931.8
71.7
ResNet-50
3/9
25.6M
4.1G
943.4
70.1
ResNet50-dels
7/5
24.0M
12.3G
627.2
76.5
ResNet50-dels
7/6
26.0M
12.3G
627.1
76.5
Table 1: Classification accuracy on ImageNet-1K using ResNet-50. The throughput was calcu- lated inference time, on image crops of size 224 224 using a single V100-32gb gpu. When the model contains DCLS convolutions, we reported the kernel count and dilated kernel size. Otherwise, the kernel size is reported and thus the kernel count fact the square that parameter.
throughput (image s)
model
img size
param.
FLOPs
Top-1 acc.
Swin-T
224
28M
4.5G
757.9
81.3
ConyNeXt-Te
224
29M
4.5G
774.7
82.1
ConvNeXt-T-dil2
224
29M
4.5G
773.6
80.8
ConvNeXt-T-ker17
224
30M
5G
560.0
82.0
SLaK-T a
2242
30 / 38 M
5.0
9
583.5
82.5
ConvNeXt-T-dels
2242
29M
5.0G
725.3
82.5
Swin-S
224
50M
8.7G
436.7
83.0
ConvNeXt-S
2242
50M
8.7G
447.1
83.1
SLaK-S
224
55
/ 75
9.8
16.6"G
367.9
83.8
ConvNeXt-S-dels
2242
50M
9.5G
433.4
83.7
Swin-B 8
224"
8&M
15.4G
286.6
83.5
ConvNeXI-B
224
89M
15.4G
292.1
83.8
RepLKNet-31B
2242
79M
15.4G
295.5
83.5
SLaK-B
2242
95
122 M 17.1
25.9"G
245.4
84.0
ConvNeXt-B-dcls-
224
89M
16.5G
285.4
84.1
Table 2: Classification accuracy ImageNet-IK. The inference throughput was calculated at inference using a single V100-32gb gpu and scaled to take into account the optimizations used in Liu al, (2022b). For the SLaK model. report both the effective number of parameters and FLOPss returned by PyTorch e and the one reported in Liu et al. (2022a) e, that takes sparsity into account.

---

Published as a conference paper at ICLR 2023
Using ConvNeXt. We present numerically in Table graphically in Fig. I the results obtained for ConvNeXt using the settings for ImageNetlk training with input crops of size 224 x 224, as described in Liu et al. (2022b): Table 5. Exponential Moving Average (EMA) (Polyak & Juditsky we report the accuracy found with this technique. We replaced Con'NeXt's depthwise separable convolutions (of kernel size 7 x 7), by 2D-DCLS ones of dilated kernel size 17 x 17 and of kernel count equal to 34 for the tiny variant, and 40 for the small and base variants. We used all the techniques previously described in Section 3 during training. From Table 2 we highlight the fact that ConvNeXt with DCLS convolutions always surpasses the ConvNeXt baseline accuracy (gains ranging from 0.3 to 0.6) with the same number of parameters and only a little cost on throughput. We believe that this gain in accuracy is remarkable, given that we only replaced the depthwise convolutional layers, which represent just about 1% of the total number of parameters and 2% of the total number of FL.PSs in CenvNeXt models. ConvNeXt model with a standard dilation of rate 2 performed poorly (see ConvNcXi-T-ddi22 SLaK model performs about well DCLS but with higher cost on throughput and parameters count.
1992] was used, and for all models in Table 2
DCLS can be seen as a kernel reparametrization technique that reduces the number of trainable parameters, and thus regularizes large kernels. For example, in the case of ConyNeXt-T-dde 17 x 17 kernel (289 parameters) is parameterized by 34 triplets (x-position, y-position, weight), i.e. 102 parameters. The kernels that could be represented by the DCLS reparametrization constitute subset of all possible dense kernels. In fact, by learning the suitable weights during training, a dense kernel could implement any DCLS one. may therefore be counter-intuitive that DCLS leads to higher accuracy than dense 17 x 17 convolution layer (see ConvNcXt-TTkeer dense convolutional layers having large kernel sizes is that the number of trainable parameters is huge, which makes learning impractical. Finally, we observe that after training, the DCLS position density is higher around the center of the RF (see Appendix [T)), suggesting that the central region is the most important one (yet experienced that reducing the dilated kernel size to values < 17 degrades the accuracy, so the positions far from the center also matter). Conversely, DC samples the whole RF uniformly. which is most likely sub-optimal, which could explain its poor performance (see Con-NcXt-T-ddl2),
The problem with
4.3 EMPIRICAL EVALUATION ON DOWNSTREAM AND ROBUSTNESS TASKS
We now report the results found for semantic segmentation on the ADE20K dataset (Zhou et al., 2019) and for object detection on the COCO dataset (I.in et al. 2014) using ConyNeXt-dels back- bones. Note that the depthwise implcit gemm algorithm was not used for those tasks as it led to throughput issues. In addition, present the results found for robustness tasks consisting of di- rectly testing (without further tuning) the previously obtained ConyNeXt-dels backbones on the fol- (Hendrycks & Dietterich, 2019 Mintun et al. 2021 Hendrycks et al. 2021b Wang et al. 2019).
lowing robustness benchmarks: BangeNet-CCC/A/RRAA
Semantic segmentation ADE20K. The results obtained in semantic segmentation show im- provement in performance by the ConvNeXt-dels tiny and hase backbones with equal number of parameters and FLOPs (Table B). As in Liu al.] (202I) and Bao al. (202I), we evaluated the mloU with single scale testing and used the exact same configurations as in Liu et al. (2022b).
throughput (image /
backbone
input crop.
mIoU (ss)
param.
FLOPs
ConvNeXt-T
512
46.0
60M
939G
23.9
SLaK-T
5122
47.1
65M
945G
ConvNeXt---delss
5122
47.1
60M
950G
21.1
ConvNeXt-S
512"
48.7
82M
1027G
22.1
ConvNeXt-S-dcls
5122
48.4
82M
1045G
19.5
ConvNeXt-B
512"
49.1
122M
1170G
21.7
ConvNeXt-B-dels 5122
49.3
122M
1193G
18.6
Table 3: ADE20K validation results using UperNet (Xiao al. 2018). We report mloU results with single-scen scale testing. FLOPs are based on input sizes of (2048, 312). The inference throughput was calculated at inference using single A A100-80gb gpu and for input sizes (3, 512, 512).
7

---

Published as a conference paper at ICLR 2023
Object detection and segmentation COCO. All ConvNeXt-dcks backbones have shown ticeable improvement in average accuracy on both the object detection and segmentation tasks on the COCO dataset, again iso-parameters and iso-FLOPS (Table I We only tested with Cascade Mask-RCNN (Cai & Vasconcelos 2018) and used the exact same configurations as in Liu et al.
2022b).
backbone
FLOPs
TPUT
AP box
AP
box 50
box AP
AP mask
APmask
AP mask 75
75
Cascade Mask Bosk-RCNN
schedule
ResNet-50
739G
46.3
64.3
50.5
40.1
61.7
43.4
X101-32
819G
48.1
66.5
52.4
41.6
63.9
45.2
X101-64
972G
48.3
66.4
52.3
41.7
64.0
45.1
Swin-T
745G
50.4
69.2
54.7
43.7
66.6
47.3
ConvNeXt-T
741G
11.6
50.4
69.1
54.8
43.7
66.5
47.3
CNeXt-dels-T
751G
11.2
51.2
69.9
55.7
44.5
67.5
48.3
Swin-S
838G
51.9
70.7
56.3
45.0
68.2
48.8
ConvNeXt-S
827G
11.2
51.9
70.8
56.5
45.0
68.4
49.1
CNeXt-dels-S
844G
10.5
52.8
71.6
57.6
45.6
69.0
49.3
Swin-B
982G
51.9
70.5
56.4
45.0
68.1
48.9
ConvNeXi-B
964G
11.1
52.7
71.3
57.2
45.6
68.9
49.5
CNeXt-dels-B
987G
10.3
53.0
71.5
57.7
46.0
69.3
50.0
Table COCO object detection and segmentation results using Cascade -Cas. Average Precision of the ResNet-50 and X101 models are from (Liu al. 2021). FLOPs are calculated with image size (3, 1280, 800). The inference throughput ("PPU""") was calculated at inference using single A A100-80gb gpu and for input sizes of (3, 512, 512).
Robustness Evaluation on MhageNet-CCCC/A///kkeeee good performances when it comes to robustness. This is illustrated by the results obtained for the different benchmarks have tried and for which have reconducted the experiments. All of
ConvNeXt-dels backbones show very
them show a gain in classification accuracy with DCLS, except SK with the model (Table 5).
Model
FLOPs Params
Clean
C(L)
C(4)
A
R
SK
ResNet-50
4.1/25.6
76.1
76.7
57.7
0.0
36.1
24.1
ConvNeXt-T
4.5/28.6
82.1
41.6
41.2
23.5
47.6
33.8
ConvNeXt-dels-T
5.0/28.6
82.5
41.5
39.7
23.9
47.8
34.7
ConvNeXi-S
8.7/50.2
83.1
38.9
37.8
30.1
50.1
37.1
ConvNeXt-dels-S
9.5/50.2
83.7
37.8
35.2
33.7
50.4
36.7
ConvNeXt-B
15.4/88.6
83.8
37.0
35.7
35.5
51.7
38.2
ConvNext-dels-B
16.5/88.6
84.1
36.3
34.3
36.8
52.6
38.4
Table 5: Robustness evaluation of ConyNeXt-dels. We reconducted this study for ConvNeXt. For ImageNet-C and ImageNet-Char the error reported rather than the accuracy. was calculated for both datasets by taking the average error over 5 levels of noise severity and over all the noise categories available in the datasets.
5 RELATED WORK
One of the studies that motivated the DCLS method is that of the effective receptive field (ERF) (Luo et al.) 2016). which characterizes how much each input pixel in a receptive field can impact output a unit in the downstream convolutional layers of a neural network, leading to the notion of an effective receptive field. Among the findings of this study, the most relevant ones for ours are that not all pixels in a receptive field contribute equally to an output response, the kernel center has much larger impact, and that the effective receptive field size increases linearly with the square root of convolutional layers. Given these findings, introducing a new degree freedom by learning the positions of non-zero weights in dilated kernels might increase the expressive power of convolutional neural networks, A visual comparison between DCLS and con-DCLS ERFs is available in Appendix 12
8

---

Published as a conference paper at ICLR 2023
The work that relates the most to DCLS is that of deformable convolutions Dai et al. (2017) where offsets from the regular dilated grid are optimized. Deformable convolutions are used for sev- eral computer vision tasks such as object detection. Even if both approaches share the use of rather similar bilinear interpolation, DCLS remains very different from deformable convolution in several aspects: firstly, deformable convolutions require the application of a regular convolution to get the offsets (which are thus input-dependent) that are then passed to the actual deformable method. Conversely, in DCLS method, a dilated kernel with learnable positions is constructed and then passed to the convolution operation, making the positions input-indepent inpppiindeeenent.. Secondly, DCLS unlike deformable convolution ones. Thirdly, deformable convo-
positions are Shannel-depender lutions have been developed for plain convolutions, not for depthwise separable ones. Finally, the number of extra learnable parameters in 2D deformable convolutions is the number of kernel el- ements in the preliminary convolution, precisely (channels_in // groups, 2 * KH Kw: KH. Kw) with KH and Kw being the kernel height and width, which establishes a strong dependence between the offsets and the input feature map. Contrarily, in 2D-DCLS, the number of ex- tra parameters dedicated to learning positions is simply twice the number of kernel weights work
(2, channels out, channels in // groups, KH, Kw)- We have not been able to compare to the deformable convolution v2 in ConvNeXt as the training becomes very slow (the throughput is divided by for the Co-vNeXt-tiny model). In addition, we noticed that the training loss grew the first epochs. which means that the method is not adapted as drop-in replacement of the depthwise separable convolution in the ConyNeXt architecture. Another input-dependent convolution method which seeks to learn the dilation rates rather than the kernel positions is the one of ADCNN (Yao) et al. 2022). But in contrast to DCL.S, this method uses regular grids with learnable rates.
DCLS is also similar to other inputinndependent kernel re- parameterizationn techniques, yet different from them. For example, in CKConv (Romero et al.] 202 and FlexConv (Romero et al.] 2021a), the kernel weights are not learned directly; what is learned is the continuous function that maps the positions to the weights. In Jacobsen et al. (2016), the kernel is modeled as a weighted sum of basis functions, which consist of centered Gaussian filters and their derivatives. PPintea et al. (2021) extended the approach by also learning the width of the Gaussians, which is equivalent to learning the optimal resolution. [Shelhamer et al. (2019) proposed to factorize the kernel as the composition of a standard kernel with a structured Gaussian one. Finally, other methods such as Worrall & Welling 2019 Sosnovik et al., 2019 2021a by [eekkers| 2019 Zhu al.) 2019), where the goal is build a scale equivariant neural network, could be considered as similar to the approach. One limitation of all these studies is that only small datasets were used (e.g.. CIFAR), and whether they scale well to larger and more challenging datasets like ImageNetlk is unknown. In addition, they cannot be used as a drop-in replacement for the depthwise separable convolution in ConvNeXt, at least in their current forms, so we could not benchmark them with DCLSS
6 CONCLUSION
In this work, we proposed DCLS, a new dilated convolution method where the positions of non-zero kernel elements are made learnable via backpropagation. The non-differentiabiliy issue of learning discrete positions was circumvented by interpolation. We demonstrated that DCLS often outper- forms the standard and the dilated convolution. We listed number of techniques that improve the learning process of DCLS, in particular sharing the positions within stages was key. We provided evidence that searching for optimal positions of weights within a dilated kernel can improve not only the accuracy in image classification, but also in downstream and robustness tasks, using exist- ing CNN architectures, without increasing their number of parameters. We reported throughput overhead introduced by DCLS, but it remains marginal, provided that we use CNN architectures involving separable convolutions, which is the case for most modern CNNs, such as ConvNeXts. Future work: So far we have shown that DCLS can be used to drop-in replace standard convolu- tion layers in existing architectures. We now would like to search for an architecture dedicated to DCLS, that would get the maximum benefit out of the method. In addition, we will explore alter- natives to bilinear interpolation, e.g. bicubic. Finally, we wish to show that the method also of interest in ID and 3D use cases. For ID cases, one can think of audio waveform generation, where dilated ID convolution filters with large receptive fields are key, as, for example, in the autoregres- sive WaveNet model (Oord et al. 2016), and in generative adversarial networks (Yamamoto et al.) 2020 Greshler et al. 2021). For 3D applications, video classification with architectures using 3D convolutions, e.g., X3D Feichtenhofer (2020), would be a good fit.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was performed using HPC resources from GENCI-IDRIS (Grant 2021-IAD01 1013219]) and from CALMIP (Grant 2021-IP21052J.. Support from the ANR-3IA Artificial and Natural In- telligence Toulouse Institute is gratefully acknowledged. We would also like to thank the region of Toulouse Occitanie.
REPRODUCIBILITY STATEMENT
The code of the method based on PyTorch and available at /ttttps://githuu..omm// smail 1 /Di lated- Convo ut i Lo-ww t. Lea rab lepllee-ee ee Spacpacinni www.Spppccnnnnoooooeeeee Code and scripts to reproduce the training and the experiments are available at/ttts:///ithhhb com/K -H Iamail/ConvWeettt dols
REFERENCES
André Araujo, Wade Norris, and Jack Sim. Computing receptive fields of convolutional neural net- works. Distill, 2019. doi: 01223915/distill.00022 https://distillppub//) httbh//0///commuugggeeeeee fields.
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint vrXiv:2/06.082.5 08254, 2021.
Erik J Bekkers. B-spline cmns lie groups. arXiv preprint UrXiv:1909.12055 2019.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 1154-6165 2018.
Liang- Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell., 40(4):834 848. 2017.
Liang-Chiee Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proc. Eur. Conf. Comput. Vis. (ECCV), pp. 801-815 2018.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Int. Conf. Comput. Vis., pp. 764-773, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 248 255. IEEE, 2009.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design in CNNs. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 11963-11955. 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 203-213, 2020.
Gal Greshler, Tamar Shaham, and Tomer Michaeli. Catch-a-waveform:: Learning to generate audio from single short example. Advances in Neural Information Processing Systems, 34:20916- 6- 20928, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proc. IEEF/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv:1903.12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Int. Conf. Comput. Vis., pp. 8340-8349 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proc. JEEE/CVE Conf. Compul. Vis. Pattern Recog. (CVPR), pp. 15262-15271, 2021b.
Matthias Holschneider, Richard Kronland-Martine Jean Morlet, and Ph Tchamitchian. A real-time algorithm for signal analysis with the help of the wavelet transform. In Wavelets, pp. 286-297. Springer, 1990.
Jorn-Henrik Jacobsen, Jan Van Gemert, Zhongyu Lou, and Arnold WM Smeulders. Structured receptive fields in cnns. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2610-2619, 2016.
Souvik Kundu, Saurav Prakash, Halch Akrami, Peter A Beerel, and Keith M Chugg. pre-defined sparse kernel based convolution for deep cnns. arXiv preprint arXiv:1910.00724 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. Eur. Conf. Comput. Vis. (ECCV), pp. 740-755. Springer, 2014.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pech- enizkiy, Decebal Mocanu, and Zhangyang Wang. More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity. arXiv preprint arXiv:2207.0366 2022a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Int. Conf. Comput. Vis., pp. 10012-10022. 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp- 11976-1 1986, 2022b.
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid R-CNN. In Proc. IEEE/CVF Conf. Comput, Vis. Patlern Recog, (CVPR), pp. 7363-7372, 2019.
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Proc. Adv. Neural Inform. Process. Syst. (NIPS), pp. 4905-4913, 2016.
Megvii. Megengine:a fast, scalable and easy-to-use deep learning framework. https://githtb 2020.
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and cor- ruptions natural corruption robustness. Proc. Ady. Neural Inform. Process. Syst. (NIPS), 34: 3571-3583, 2021.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016.
Silvia L Pintea, Nergis Tömen, Stanley F Goes. Marco Loog, and Jan C van Gemert. Resolution learning in deep convolutional networks using scale-space theory. IEEE Transactions on Image Processing, 30:8342-888 8353, 2021.
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4):838 855, 1992.

---

Published as a conference paper at ICLR 2023
David W Romero, Robert-Jan Bruintjes, Jakub M Tomezak, Erik J Bekkers, Mark Hoogendoom, and Jan C van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059, 2021a.
David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoom. Ckconv: Continuous kemel convolution for sequential data. arXiv preprint arXiv:2102.02611 2021b.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 4510-4520, 2018.
Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line between structure and learning to optimize and adapt receptive fields. arXiv preprint arXiv:1904.1 1487, 2019.
Mark J Shensa. The discrete wavelet transform: wedding the a trous and mallat algorithms. IEEE Trans. Signal Process., 40(10):2464 244-22422, 1992.
Ivan Sosnovik, Michal Szmaja, and Amoold Smeulders. Scale-equivariant steerable networks. arXiv preprint arXiv:1910. 1093, 2019.
Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. Disco: accurate discrete scale convolu- tions. arXiv preprint 1rXi::2106.02733, 2021a.
Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. How to transform kernels for scale- convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1092-1097, 2021b.
Hugues Thomas, Charles R. Qi, -ammauell Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. Int. Conf. Compul. Vis., 2019.
Asher Trockman and Zico Kolter. Patches are all you need? arXiv preprint 2rXiv:2201.09792, 2022.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Proc. Adv. Neural Inform. Process. Syst. (NIPS), 32, 2019.
Zhengyang Wang and Shuiwang Ji. Smoothed dilated convolutions for improved dense prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2486-2499 2018
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. arXiv preprint arXiv:2110.00476, 2021.
Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
Tete Xiao, Yingeheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proc. Eur. Conf. Comput. Vis. (ECCV), pp. 418 434, 2018.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel WaveGAN: A fast waveform genera- tion model based on generative adversarial networks with meltitresolltionn spectrogram. In Proc. ICASSP, pp. 6199-6203 IEEE, 2020.
Jie Yao, Dongdong Wang, Hao Hu, Weiwei Xing, and Liqiang Wang. Adenn: Towards learning adaptive dilation for convolutional neural networks. Pattern Recognition, 103:19 2022.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.071 2015.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proc. IEEE/CVF
Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 472-480, 2017.
12

---

The Frobenius inner product x of two matrices and B of Ma.(( is defined by: Published as a conference paper at ICLR 2023
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. Int. J. Comput. Vis., 127(3):302- 321, 2019.
Sanja
Wei Zhu, Qiang Qiu, Robert Calderbank, Guillermo Sapiro, and Xiuyuan Cheng. Scaling- translation-squivariann networks with decomposed convolutional filters. arXiv preprint arXiv:1909.11193, 2019.
APPENDIX: PROOFS AND DERIVATION FOR THE DCLS METHOD
In the following, we show how mathematically describe the DCLS kernel construction and how to explicitly calculate the gradients of the loss function with respect to weights and positions that are used the backpropagation algorithm. These gradients are useful to implement the DCLS method in way that is compatible with the automatic differentiation of PyTorch.
7.1 NOTATION AND PRELIMINARIES
We denote by [] the floor function and we define its derivative by the zero function.
Yx R. |x]
O
(6)
We denote by m E N the number of kernel elements inside the constructed kernel and refer to it as the "kernel count". Moreover, we denote respectively by $1, 82 E No x N", the sizes of the constructed kernel along the x-axis and the y-axis. The latter could be seen the limits of the dilated kernel, and we refer to them as the "dilated kernel size".
The p matrix space over is defined as the set of all x p matrices over and is denoted (R).
F
ur(A B) A B
Where "tr" stands for the trace of the square matrix A ATB.
The characters p² and Dd respectively stand for the weight, the position of that weight along the x-axis (width) and its position along the y-axis (height) in the scalar case while the bold w = (wa)isism p' (p) sisism and p² (p Isigm respectively stand for the weight, the width- position of that weight and its height-position in the vector case.
The proofs and algorithms that will be shown in the next subsections made for the of tensors with one input channel and one output channel. Without loss of generality, those proofs and algorithms hold for the general case of 4D tensors and higher by considering and applying them channel-wise.
7.2 2D-DCLS, SCALAR WEIGHT CASE
We begin by the case of kernel containing only element.
The function f that defines the kernel construction the scalar weight case is as follows:
f:RxR R M...o.
w.p w.p'pp² + K
where E [11. s1]. Vj [1.. sp] I]:
r!) 22) if pp
1p²
wr 2
if i = p
+ = 1p²
Kij
w
if p
j == 1
+
w 0
if pp'
+1, j +1
otherwise
13

---

Published as a conference paper at ICLR 2023
and where the fractional parts are:
[x²} - p²
Lo'J
and y²= [1²] - p² |هما
The constructed kernel K is zero except for at most the 4 adjacent positions that represent the 2D
"2
$1
interpolation of the single weight Note that Σ Σ
Kij
w.
We then define the scalar loss function as:
loss = g(f(w, ,²))
(10)
with g: Ma (R) E a differentiable function that models the action of all the layers that will follow f in the model.
By applying the chain rule we obtain:
Ōloss
g''((u.pp'pp 0f(w.p'.p' p²))
=
x
(11)
dw
dw
Ōloss
af(w,p',p ,2)
g (f(ww.p' ('..p'pp'' .p²)) x
(12)
Op
Oy2
Ōloss
af(w.p', (w.p'pp') p²)
= o ( ,1²)) x
(13)
0p2
8p²
with
Ōloss
Ōloss
9 df(wwpp' .p)))
(14)
0K
0f(w, p2)
Let us put
911
912
9102
d'('(w,p' "'tt,,',,70 ,²)) -G
921
9252
(15)
9s12
85182
Jeal 1
and let us consider two column vectors
[
32
of
RR
and y
[y y/2 Usa] of R We have:
x f(w.
K Kijz,ll
(16)
1=
Since K zero except for the 4 aforementioned positions, we have:
TT f(w p¹, ,p)
K [p*) Tlpll M|r
K K|pl T1p" J+1 M1'21
(17)
K [[n J+1 "[ MLPJ+1
K 1+11+11111 2 [+1 11
14

---

Published as a conference paper at ICLR 2023
By deriving this expression with respect to w, p' and p² we obtain:
0(x² [w,pp' "p))))))
(1 - -- - ولااl D1rA1
Ow
+,² (1 - x2) 311 1+1 M1Ad
(18)
+ (1 r²) ,2 ",," M1
+ p² p² x|n0| 81
(x:" ((w.p',1²)) Opl
= - (1 r²) T y|
+ (1 r²) -1p' 21d21
(19)
- r2 "1p" M1p21+11
2 +lp' W|++++1]
0xx f(w.f((ppp "(w.p'pp)) ,p')) Öp²
w
(1 rt) T1p" 3|
rd - |+1 V|
(20)
+ (1 r²) T'Dn! D|+
+ + ++ 0|+
Because of the linearity of the differentiation in the three previous equations, we could write:
0(x" f(w,p' ppppppr'p pp²)g)
x T 0f(w, p' p²) y x= G.!y
(21)
Ow
Fw
0 2 [w, p² (²)
x² -s(w p² p²))
x² Gp. Op!
(22)
Op
an T [w, p² p2))) -- -5 p² p²)
TT G
(23)
Op?
0p2
where Gw,, Gp', Gp², respectively stand for the by matrices described below and which have zeros everywhere except at the four positions of interpolation.
G
Ip21
1
+1
0
0
0
(1-r2)
(1-12)
1
0
Ipt
0
rd
(1-12)
1,,
0
1x_
+1
(24)
0
0
Gpt
+1
11 72)
2
p
(25)
wi 1 2
0
p
+1
0
15

---

Published as a conference paper at ICLR 2023
Gp
Lp²]
+1
0
w (1-r!)
w
r
Ip'
(26)
wr
Wr
0
p
[++1
0
From the three equations
21
22 and
23
we can identify
0f(w.p', ,p²) G
(27)
dw
0 0s,, p², w²¹)
Gp-
(28)
Op1
0 Of(w, pppp²) Gp Op2
(29)
Finally, we have:
0loss
G G (1 (1-r)) (1 (1)) 910')1>3)
dw
+r² (1 - -2) gL
(30)
+ (1 - r²)) x² 91
+ ,² |
Ōloss
G x Gp-
w |- (1 ,²) g1p ||
0p
F
+ (1 - و²) [+1
(31)
r2 g ||
2 ++
Oloss
G
G22 w |- (1 r²) g1 [|ا
Öp2
F
r² g1z J+1 |جا
(32)
+ (1 r²) 91 ||p
+ + g |+11
In the next subsection, we will see how this result can be generalized to the vector case.
7.3 2D-DCLS, GENERAL CASE
The general case is the where the weights
[w] W2 wmn' and the positions
p² =
1 ph ... p T p² = Ir p
ph are stored in vectors, with the frac-
tional parts y = [p-} = p² 1pp1 [r] r
r and p {p²) = p" -p'
T
[r 2 2 extended as well.
The function f defined in equation is then extended the function F defined as follows:
F: m
T
x
R
Mss. (R)
(33)
w.p pu pp + K
fw,, P..P.)
16

---

Published as a conference paper at ICLR 2023
The constructed kernel K here is the result of a summation of the function f defined in 7 over the elements of weight and position vectors. We then define the scalar loss function as in (0
loss (( .²))
(34)
with Ma (R) + R a differentiable function that models the action of all the layers that will follow F in the model.
Let us put
911
912
91.2
g' ('([w.p.pp)) G
921
g2
(35)
1
9x12
85182-
As in 11
12 and
13
by applying the chain rule we obtain:
Wi m]
Öloss
OF(w, p'. .p²²)
(F p'y²)) x
(36)
Om,
Ow:,
Ōloss
OF (w, p' .p²)
of (Fw.. p' p²))
x
(37)
0p!
Op;
Ōloss
OF w, p .p²)
(F ,²))) x
(38)
Op
with
Ôloss
Ōloss
g'(F(w,p' (0,
(39)
aK
OF un p p2)
Let us put this time
911
912
11
g' ('() G
921
11.2
(40)
L9,11
912
95182-
Using the definition of F, and by substituting the last equation in 36
and
38 we have:
Õloss
0Σ m i=
f(w,,
p!)
G
(41)
Ow;
dw;
Oloss
أ f(w,, pp.)
G
x
(42)
Op!
Op!
Øloss
8 Σ m
[WW, p!
G
i=
(43)
Opp
Op}
And we know that:
V(i.j) E [[..m]] m2
0f (W,,P),PP
0f(w,,P)),
Ow;
Op:
which simplifies equations
44
and
43
17

---

Published as a conference paper at ICLR 2023
vi []..m]
Ōloss
af(ws, P.. p?)
x
(44)
Ow,
Ōloss
2 (sw pł pz)
G
(45)
Öp:
F
Op:
Óloss
0f (wi, p!.
x
(46)
0p: F api
We can notice that we are brought back to the scalar case, and we deduce from (30), the gradients of the loss function with respect to weights and positions in the general case:
31 and
32
vi E [1 mm:
Ōloss Ow
= (1- (1 rs) Millle
+ (1 - -r²)) - 9
(47)
+ (1 - (1-r:) rī 9p!lle
+ + r g
Öloss
0p! ) =
- (1 r 9
+ (1 - - 9
(48)
r 9lttlle] +1
+ r3 glr ++1pp//+1
Öloss 0p²
- (1 r 9
- r Flrt |+1|+?
(49)
+ (1 r 9
+ r 91
The results found for the general case are nothing but a component-wise application of the result obtained in the scalar case. In addition, we show in Appendix 774 the extension to the 1D and 3D convolution cases.
7.4 ID-DCLS, 3D-DCLS
We denote respectively by $1, 83 E No x N x N*, the sizes of the constructed kernel along the x-axis, y-axis and the z-axis. Moreover, the x q tensor space of third dimension is denoted Raxpxq
The function f defined in 7 could be adapted in order construct suitable kernel for the 1D convolution in the scalar weight case as follows:
FID: RR x R + R
(50)
w.p k
where Vi E [1 [1..]]]
w (1 r) if [p]
k:
if [p]
(51)
else
18

---

Published as a conference paper at ICLR 2023
and where the fractional part is:
r= {p) [p]
(52)
Following the same construction in Appendix 7.3 we can show that the gradients of the loss function with respect to weights and positions in the general ID case are:
Vi E [1.. m.
Oloss
-1-r:: Ilri] 9lre] + r. I1
(53)
dw
Øloss
wy (g1ml+1 - 91p.1)
(54)
0p
Furthermore, define the function faD, the suitable kernel construction function in the 3D convo- lution case, as such:
f: + R4 w. w.p'.p' .
(55)
where Vi E [1 [1 s1]. vj
[1 s2 [1..6 ss]
if = p
w
if p
[
p
p3
if p
if = PP'
++.
1 +1
Kija=
if |P'
(56)
if pp'
+1,
p3
if = p
1,
+
+1
w
if P
else
and where the fractional parts are:
r (p' } = p¹ 1p']
2 {p²) p² 1²
(57)
3 (p3) p2 |
We can show that the gradients of the loss function with respect to weights and positions in the general 3D case are:
19

---

Published as a conference paper at ICLR 2023
vi []..m]:
[loss Ow
= -11-)) (1 , 9 I
(1-r)) 91:
(1- 012
+r/ -
(58)
+(1-)))1--)) 90: +1
+ 0000
dloss
= w/
[ - (1 -
Opp
+ (1- - ( 911
- r7 (1 + |
+ r (1 -r²) 9
(59)
- (1 ? r² 01p! 11
+ (1 ??) 11
r² r +1000
of r +11111111
(llss
wi (1-
(1-r)) r:) (1 +3) I
Ōp² )
-r. (1 r3) 9
+ (1 - +ス) |
+ (1 (1
(60)
(1 ( 904b00
r rē 01
+ (1 (1-r)) r3 16111121111
+ r. 110++
(loss 0p3
= wi
[
(1 - (1 ( 9
r| (1 (1- |
(1 -1-++) 7 -
r re -
(61)
+ (1 (1-rt) - (1 (1- 91s 11
+ r -1-18
r3) -
910
+ (1 (1- r 10111121
+ rr r -11111111111111111110001
20

---

Published as a conference paper at ICLR 2023
8 APPENDIX: THE 2D-DCLS KERNEL CONSTRUCTION ALGORITHM
In the following, describe with pseudocode the forward and backward passes for kernel con- struction used in 2D-DCLS. In practice, W, Pl and Pa are 3-D tensors of size (channel s_out, channels.ir // groups, K_count), but the algorithms presented here are easily extended to this case by applying them channel-wise.
Algorithm 2D-DCLS kernel construction forward pass
Input: W. Pl p!, vectors of dimension
Output: K the constructed kernel, of size (81 82)
1: K 5-0 0
2: p² <- P
P P2 4: ure_for_backwart (p',p² RR'
p
3: R -
-P'pp;
R² P<p2
p²
5: for
m do
6:
K(p:,P,] += W; * (1 ) (1 R3)
7: K(p: +1,p; += W, (R) (R)) + R
8: K(p!. - += W; *1 R (R))
9: K(p] +1, + +1]+= W, * (R) (R.)
10: end for
Algorithm 2 2D-DCLS kernel construction backward pass
Input: GradK LLoss matrix of dimension (31 X $2) aK
Output: Loos ôLoss 0Loss vectors of dimension aw
2PT 0P2
1: Dosse
0,
Lose
Bpl +- O,
âLoss
8P +
BW
2: p² pp. R R <- load_saved
3: for = 0 m do
4:
LLoss OW
[1]
ÔLoss
pp: pł] [0]]((11RM)) (1-R)) (1-R))
Losss
[p) + 1,, R RC)
0K
0K
aLoss
iLos8
p P. 1] * (1-R)) R +
0K (p] 1. + 1] R R
0K
5:
aLoss
1 += W.
aLoss
p..P P.l
R +
aLoss
[p]
1,P (1 R
0P1
0K
dk
Tooss
0osss
Ip.
+ 1] R
[p]
1, p +
0K
OK
6:
Tooss
[1] += W. aToss
aLoss
0K p.pp. +(1 R
[p + +1,p * R
0p2
OK
ÔLoss
ôooss
(p. pł + 1] (1 R
[p. + 1, p. + R
0K
OK
7: end for
The 'for' loops in the algorithms are fully parallelized using GPU threads. The 2D-DCLS convo- lution with kernel construction is then obtained by applying the classical cocovollution provided natively by PyTorch or any other method such as the depthwise implicit gemm convolution method Ding et al. (2022) using the constructed kernel.
21

---

Published as a conference paper at ICLR 2023
When considering concurrent execution of this pseudocode, the additions may result, in case of overlapping. in a replacement of the overlapped values instead of the desired accumulation. This problem can be addressed by using atomic addition operations.
APPENDIX: THE DCLS KERNEL CONSTRUCTION ALGORITHM IN NATIVE PYTORCH
In the following, describe with PyTorch code the DCLS construction module for ID, 2D and 3D versions. The DCLS convolution method is obtained by using the constructed kernel as weight for the native torch.nn.Com (1,2,3) or another convolution method such as depthwise implicit gemm Megvii (2020).
class ConstructKernelld (Module
def init _onit__self,, out out_channels,
in in_channels, groups, kerrel_count, di dilaate__eernelssi (upar()._iit__]
self.out_Channels out_channels
self.in_channels in_channels
self.groups groups
seff:dilaatd__eree__dll = dilated_kernel_size
self- kernel_count
I = torch. (0, dilatte_kerreel_size([]]]
e.xpand (out_channels,
in www.cchannncccccce /gropps kernel_counts -1) permute (3, 0,,,,2
self. Parameter requires_graddPaie False)
self 1im torch. zeros (1)
self.lim [0] = dilated sellated_kernelssizer
self.lim self.lim. ierea.nddexpanddieee (out_channe in_chaaneels//roopps.
kernel_coure -1) permute (3, 1,2)
self. lim Parameter (self. lim,
def forward W, P):
self.lim //
Pr = P
P.floor()
R = (Pr P) expaand (self.dilated Beaatted_keeee_ssieeee [0], -0,,,,,,,,,,,,1------0-,-,,,,-,- -1,,1-1)
R.selec (2, P1 P. P.select (0,
condl (self.I == P1)
cond2 (self.I
P1+1)
W1 torch
torch.where where (condl, 1.0, 0.0) W2 torch. where (cond2, 1.0, 0.0)
K W1 R1 * (M2 W1)
K
K K. (3)
K.permute (1, 21,2,0)
return K
class ConstructKernel2d (MM.rreeerre11ddMdMMM
def
out_channels, in_channels, groups,
kernel_count, dilated_kernel_sizee)
self.out_channels out out_channels
Self.in_channele = ir_channels
self.groups groups
boli.dilated_eeree___.
kelated_
self.kernel_com ternel_coul
22

---

Published as a conference paper at ICLR 2023
J torch. aranhe (0,
dilated_kernel size [ -1)
I torch.
bblaatee_kernel_sizeelll
I I. exxand (out_chanee__
in mt__channeis/ggroup, k permutte (3, 4, 1,2)
J J. [_e
in ht__chaneeia//ee /ropppps - permute (4, 0,
self.I Parameter (I, requires__rad-False)
self.J Parameter (J, requires_grad=Fals)
self.lim torch.zeros: zeros (2)
self.1im [0] - dilated dilated kezeee_ieeelll
self.lim [1] - dilated [Baated_kemee_siieei]]
self.lim = self.lim. expand (outt_chnnnns,, in pt_channeess//groups
kernel_count -1) permute (3, 0, 0,1,
self.lim Parameter (self. lim, requires__raa
def forward W, P):
self.lim //
Pr =
...floo()
R = (Pr P) expand (self.dilated Bet_lated_keree_steeee [0],
self. self.dilated dil.tttteeereee_ssseee -1,,,,
R1 R. s.seltt (2, 0); P1 P.select (0,
R2 = R. select (2, P2 P.select (0,1)
R1R2 R1*R2
condl (self.1 I == P1)
cond2 (self.J P2)
cond3 (self.I == P1+1)
cond4 (self.J P2+1)
W1 torch. torch.where where (condl #cond2,
1.0, 0.0)
W2 torch.whers (cond, 1.0, 0.0)
W3 = torch.whers (cond3+cond2, 1.0, 0.0)
W4 forch.whers (cond,+ 1.0, 0.0)
K W1 R1R2* (W1 W2 w3 + W4) R1x (W3 - W1)
R2x
K
K K. (4)
K K. K.permute (,,3,0,11
return
class ConstructKerne (Morrut
def init ini__iintt____________ (self, out out_channels, in_channels, groups,
kernel_count, dilated_kernel_size
()
self.out_channels out_channels
sel.f.n__chnnells = ir_channels
self.groups groups
selffddilates_termeessizzz = dilaate__erree_ssiee
self.kemeelcoont kernel_count
L = torcchaaaanee(0,,
dilatte__eernel_siee[1111 expp) expand (ddp::eeeeeeeddddaaadedeeeeellll (dilatee__ermelssizee)))
[_late__kernel_sizee]]]]1111
J torchh
dilatee_kernel__sieeel [1]) exppad [b
dilated kaa_a_keeeeel_eeee2000
I torch
Sellated_krnel_see exppx (esilated_kenne__iieeee
dtllated_kernel_siiie
23

---

Published as a conference paper at ICLR 2023
L L. exp in ctt.aees//ggoppppeeeeeeooottttta permute
I I. cxppauddoutttchameess,
in hin_ahammas///rpppppeeeeseoooooo
permute (3,4, 1,2)
J. J.expand
in htt.fhaaass///oppppp/eeoooooooor pepmutteeeeeeee.eeee (3,5, 4, 4, 2,2)
self. Parameter requires_grad-Palse
self.I Parameter tequiires_grad-Fals)
self.J Parameter requiires_grad=aalss)
self.lim torch.zerosu.
self.lim(0) cilatted_kernee_siee)0
self.lim[] = cilatee_kernee_sizz
self.lim [2] dilated Sbllate__kemne_sszeel(
self.lim self.lim. (ot_ineepaaaddttttaaaaaaatt in pt__channels//groouss
kernel_count, -1) permute (3, 0, 2)
self.lim Parameter (self.
def fowward(sel: W, P): P self.lim // 2
Pr
P = P.floor()
R (Pr P)
(self [egemmdlell..dd.eeeeeeeeesssssll
self. dilatee_kernel_sst _asellssieee____ [1], self =1 -1,,,,,,,,
R1 R. select 0); P1 P.select (0,
R2 R. select (3,1); P2 P.select (0,1)
R3 = R. select ;, P3 P.select (0,2)
condl (self.1 == P1)
cond2 (self.I P2)
cond3 (self.J = P3)
cond4 (self.L P1+1]
cond5 (self.I -- P2+1]
cond6 (self.J P3+1]
W1 torch. where (condl=cond2=condd) 1.0, 0.0)
W2 torch. (cond=<ccccccoo2220oo3 1.0, 0.0)
W3 = torch. ( (cond:oocoo 1.0, 0.0)
W4 (crcch..hereeccndCcccco/coo
1.0, 0.0)
W5 (orchh.hhereecoodlooooooo-----
1.0, 0.0)
W6
1.0, 0.0)
W7 (erch..wereccond)cccooo
1.0, 0.0)
WB = torch. where (cond4 1.0, 0.0)
# needs better computing
K W1 (1
R1)
(1
R2)
* (1 R3)
R1
R2)
(1 R3)
K + W3 (1
Rl) R2
(1 R3)
R1
(1 R3)
K + W5 (1
R1)
(1
R2)
R3
K W6 R1
(1 R2)
K + W7 (1
R1)
R2
R3
K W8 R1
R2
R3
K
K
K = K. S uum (5)
K K. K.pernute 0,
return K
24

---

Published as a conference paper at ICLR 2023
10 APPENDIX: HISTOGRAMS OF POSITIONS
the following, we show as histograms over training epochs, the distribution the four kernel positions for the 2D-DCLS convolutions of the ConvNeXt-T-dels model. Note that there is no agglutination or edge effect around the kernel limits, and that the distributions are relatively stable, with higher concentration around the center of the kernel. Individual positions, however, constantly moving; see animation at:
Dels heatmap hists/[=0,b0)
Dcis heatmap hists/(s1,b0)
spach
30
250
opoch
100
190
200
250
The distribution over epochs of kernel positions for the stage of the ConvNeXt---dclk model.
Figure 3:
Figure 4: Idem for stage
Dels heatmap hists/ls2,b0)
Dets heatmap hists/(s3,b0)
spach
epech
0
30
100
130
200
250
100
150
200
250
Figure Idem for stage
Figure 6: Idem for stage
11 APPENDIX: SPEED CURVES AND LEARNING RATE SCHEDULE
Here, we plot the average speed curves the four position tensors for the 2D-DCLS convolutions Vp of the ConvNeXt--Tdels model functions of the training epochs. The general formula for the
25

---

Published as a conference paper at ICLR 2023
average speed of DCLS position tensor of size (cout, cin, m))
N epoch t, is follows:
coul cin m
Vt [1 .. mmax
Vp(t) coul cin mn
ΣΣΣ
pt pst 15k
Dcls pos avg speed
Dcls pos avg (s0, bo) Dcls pos avg (s1, b0)
Dcls pos avg (s2, bo) - Dcls pos avg (s3, bo)
0.2
0.15
0.1
0.05
epoch
100
150
200
250
Figure 7: The average speed of the four position tensors for the Con-NeXt-T-dcls model function of epochs.
Global Train/train, sitrain_lr__oo Ir_pos
0.015
0.01
0.005
spoch
50
100
150
200
250
Figure The learning schedule used for training ConyNeXt-T-deh. model.
At epoch O, we can notice that the average speed is abnormally high, this is due to the fact that the beginning, the positions are initialized randomly and we arbitrarily considered that V/((0) 0, thus the large speed gap at initialization. Another peak can be seen at epoch 20, this one is due to the this precise epoch during training. This
introduction of the repulsive loss (Thomas et al. 2019) last causes a momentary increase in the average speed of the DCLS positions. general, we can say that over epochs, the average DCLSS positions follow the shape of the scheduler used in training.
26

---

Published as a conference paper at ICLR 2023
12 APPENDIX: EFFECTIVE RECEPTIVE FIELDS COMPARISON
In the following, we show the effective receptive fields (ERF) calculated respectively for ConvNeXt- T-dels, -onvNeXI-T with standard dilated kernel of rate and ConvNeXI-T models. The imput crops used here are of size 1024 x 1024 and the heatmaps are normalized (between 0 and 1). These ERFs are obtained by finding (via backpropagation) the input pixels that most impact the activations of pretrained model using samples of the dataset. We observe that the ERF of ConvNeXt-T-dccl has particular shape that resembles a square with more prominent diagonals and medians. The ERF of ConvNeXt-T with standard dilated kernel is larger but with gridding artifacts. In all plots, seems that the center has more importance.
200
400
600
800
800
1000
1000
600
800
1000
Figure 10: The effective receptive field the ConvNeXt model with dilated kernels (dil rate 2) instead of the dense 7 ones.
Figure 9: The effective receptive field (ERF) of the ConvNeXt---ddla model.
200
400
600
800
1000
200
400
600
800
1000
Figure 11: The effective receptive field of the ConvNeXt-T model.
27

---

Published as a conference paper at ICLR 2023
13 APPENDIX: TIME MEASUREMENTS
We notice that when using the depthwise implicit gemm algorithm, DCLS forward slightly slower than the PyTorch native Conv2d forward pass, but the DCLS backward pass is faster, in particular when increasing the model size.
500
Forward
300
200
100
mn.oonvd
DCLS2d
nn.Comv2d
DCLS2d
nn Convzd
DCLS2d
tin
small
base
ConvNeXt
Figure 12: For the 3 ConvNeXt variants (tiny, small and base), we measure the elapsed time in ms for forward backward pass with fixed batch size 128 and inputs of size (3,224,224) using DCLS2d convolution accelerated by the depthwise implicit gemm algorithm. Measures were carried using single A100-80gb gpu. We also compare those timings the 3 ConvNeXt baselines.
When not using the depthwise implicit gemin algorithm but the PyTorch native Conv2d in DCLS, the forward and backward are about twice slower.
Backward
Forword
an.Conv2d DCL52d
nn.Comi2d
DCL52d
nn. mn.oon2dd
DCLS26 base
thy
small
ConvNeXt model
Figure 13: For the ConvNeXt variants (tiny, small and base), we measure the elapsed time in ms for forward backward pass with fixed batch size 128 and inputs of size (3,224,224) using DCL.2dd convolution with the PyTorch native 2D convolution algorithm. Measures were carried using a single A A100-80gb gpu. We compare those timings to the 3 ConvNeXt baselines. We also compare those timings to the 3 ConvNeXt baselines.
28

---

Published as a conference paper at ICLR 2023
We notice that when using the depthwise implicit genin algorithm, the DCLS construction algorithm negligible time compared to the convolution method for large input map sizes. For small map sizes (example 7x7), the DCLS construction algorithm takes much time the convolution.
Komei Construction-on/
Komei Construction
50-
tine (In ms)
20
10
13
19
25
31
18
25
13
19
25
18
19
25
map size channels
Figure 14: For different dilated kernel sizes (ranging from 31), and different map sizes, we measure the elapsed time in ms for forward backward pass with a fixed batch size 128 and fixed kernel count 34 using single DCLS2d construct module with 2D convolution accelerated by the depthwise implicit gemm algorithm. Measures were carried using single Quadro RTX 8000 gpu.
When not using the depthwise implicit gemmn algorithm but the PyTorch native Conv2d in DCLS, the construction algorithm time significantly lower than the convolution time even for small input map sizes. This is due to the fact the convolution time with the PyTorch native Conv2d is significantly higher than the one with the depthwise implicil gemm algorithm.
Kernel Constructon+Comv
300
Kernel Construction
Une (n)))
50
59156- 96
28x28-192
1Ax14-394
767-768
map size channels
For different dilated kernel sizes (ranging from to 31), and 4 different map sizes / channels, we measure the elapsed time for forward + backward pass with fixed batch size 128 and a fixed kernel count 34 using single DCLS2d construct module with the Py Poroh native 2D convolution algorithm. Measures were carried using single Quadro RTX 8000 gpu.
Figure 15:
29


Reviews:
[{"review_id": "kgUi1-AVyi", "reviewer": "Reviewer_8tHP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a technique to learn the shape of a dilated convolution kernel to allow via backprop the learning of the positions of the non-zero elements of the kernel. They do this by reparameterizing it as a series of weights (the “kernel count”), and a kernel window size (the “dilated kernel size”), and use bilinear interpolation to perform the convolution. They show modest gains on ImageNet image classification, Mscoco Object Detection, and ADE20k Semantic segmentation, at the cost of modestly decreased throughput. \n", "summary_of_the_review": "While the paper itself as an artifact feels high quality, the experiments done are (mostly) appropriate and well executed, and the project requiring significant impressive engineering work, I just don't see this paper as impactful or meaningful. The results simply don't justify the extra complexity (in method, in code, in platform lock in) or run time hit. As a result, I don't see it as being something that gets built upon, used by others, etc. \n\nBeyond the empirical and practical tradeoffs, the paper/project lacked a convincing underlying hypothesis of why we would want to do this at all. As a result, from reading the paper, the main takeaway of a reader will be \"this specific idea, when implemented this specific way, works reasonably\". There aren't generalizable insights that other researchers can build upon, learn from, or discuss. \n\nMy score is a result of a combination of these things. Because the paper feels pretty polished and finished, but I don't find the motivation or results themselves convincing, I see myself unlikely to raise my score.\n\n**Update**: After conversations with other reviewers, I've tried to re-calibrate what I think the bar for acceptance should be. As a result, I've updated my score to a 6. After reading the rebuttal, I stand by my assessment that the actual paper quality (in writing, experiment design, code availability, etc) is high, but still have concerns about how well motivated or impactful the work will be. As a result, I am OK to see this accepted.", "strength_and_weaknesses": "Strengths:\n\n- The paper itself is well structured, well written, and very easy to follow.\n- The experiments are thorough and well done, and include most of the basic data I would request as a reviewer (e.g. they have throughput, though they probably should also have it for object detection and semantic segmentation, and have appropriate baselines with modified receptive fields). \n- The method is relatively simple to understand, and the paper correctly focuses on one single idea throughout, holding everything else constant. I didn't have to guess where their gains came from.\n\nMain Weaknesses:\n\n- The paper, and associated project, feel poorly motivated to me. I don't get the why of this project from several angles: Why do we want or need to learn the kernel locations? When do we expect this to help? And why would people use this? \n  - The intro states `Our own investigation on ResNet and ConvNeXt with standard dilated convolution (Section 4.2) will lead to a similar conclusion. The failure of this method for classification tasks could be attributed to the great rigidity imposed by its regular grid as discussed in Wang & Ji (2018).`\n    - I don't buy this motivation. The main use case of dilated convolutions was to grow the receptive field of models when applied to large images, like in semantic segmentation or other dense prediction tasks. For smaller images, like those used in classification like the authors do the dilated convolution study on, it makes no sense, as the receptive field is already the whole image. \n- While the authors spend a lot of their related work contrasting the methods, it seems the situations (if any) you may want to use DCLS you may instead use deformable convolutions (https://arxiv.org/abs/1703.06211) which are popular in the object detection community. The authors should have compared directly to deformable convolutions.\n- The results just aren't that strong. e.g. the authors state: `From Table 2, we highlight the fact that ConvNeXt with DCLS convolutions always surpasses the ConvNeXt baseline in accuracy (gains ranging from 0.3 to 0.6) with the same number of parameters and only a little cost on throughput`\n  - I unfortunately don’t see these gains as meaningful at all. 0.3 - 0.6 accuracy improvement (e.g. 82.1 -> 82.5) on ImageNet is really small at the cost of ~7% lower throughput, but even more importantly, this would require researchers/users to use non-standard convolution operations (including compiling and packing custom CUDA operations with their models). \n  - Furthermore, using non-standard convolution operations limits the platforms you can run this model on efficiently, and prohibits the advantages of different optimizations people may want to use (e.g. more efficient conv kernels for inference). \n\n\nOther feedback:\n- `“The problem with dense convolutional layers having large kernel sizes is that the number of trainable parameters is huge, which makes learning impractical.Finally, we observe that after training, the DCLS position density is higher around the center of the RF (see Appendix 8), suggesting that the central region is the most important one.”`\n  - I don’t think the first claim here is really supported. The ConvNeXt-T-ker17 seems to do reasonably well. Perhaps it just needs a bit more regularization (which you can think of DCLS as a version of). \n  - To the second point, the DCLS position density being higher around the center of the RF to me points to the fact that you probably don’t need that large of kernels. \n- Re: Semantic Segmentation / Object Detection Results\n  - Thank you for including these! These two tasks are a much better application of dilated convolutions than ImageNet classification. That said, in my opinion, we end up at the same place that the results, though a small improvement, are not really meaningful enough to justify using this method.\n  - Throughput is also dropped here from the tables, and I suspect the small gains in accuracy (e.g. AP increases of 0.3-0.9 or mIoU increases/decreases of -0.3-1.1) are overshadowed by the increase in actual runtime.\n- Re: Robustness: Is there a hypothesis driving this set of experiments? Why would these kernels lead to more robust networks? The gains here are also relatively minor.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZScK7az7OJw", "reviewer": "Reviewer_2fMY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Dilated or \"à trous\" convolutions allow increasing the receptive field of convolutional networks by orders of magnitude, at a constant time and space complexity. While a standard approach is to define the dilation rate as an hyperparameter, to define a regularly spaced grid, authors propose and adaptive alternative which, given a budget of kernel positions, learns where to place these positions on the grid.\n\nThey show improved or identical performance on computer vision tasks (classification, segmentation, detection), improved at constant parameter number and identical with fewer parameters.", "summary_of_the_review": "Anyone who ever worked with dilated convolutions probably thought at some point \"what if I could learn this pattern instead of cross-validating it?\", and dilated convolutions are a fundamental building block of so many architectures that the potential impact of such a method is very significant. The main weakness of the paper is that they completely ignore the audio domain which is probably the field where dilated convolutions had the biggest impact. While this would be acceptable in computer vision conference, it's unfortunate in a machine learning conference to not consider such a field that would have been the perfect testbed for the author's method. In particular, the experimental setting chosen by the authors remains limited to very small kernels, which limits the understanding of the method's potential in more general settings.", "strength_and_weaknesses": "Strengths:\n* the dilation rate of dilated convolution layers requires a careful cross-validation, with sometimes very little intuition to guide the exploration, which becomes prohibitively expensive in networks that involve many dilated layers as they interact globally. Having 1) a way to replace cross-validation by backpropagation 2) handle irregular grids are very desirable features for such layers, to the potential impact is large.\n* the method of authors is a drop-in replacement to standard convolutions and comes at a light increased computational cost, which makes it useful in practice and not just in an exploratory setting.\n* the method is conceptually simple (reparametrization + bilinear interpolation) which is a clear plus\n* section 3 is a great idea, it's rare to see a paper detail as much the various explorations of authors, including unsuccessful ones.\n\nWeaknesses:\n* the main weakness of the paper is that it completely ignores the groundbreaking advances that dilated convolutions brought to audio, where the scale of dependencies to handle is much higher than in images and where dilation rates are typically increase in an exponential fashion along the network to go up to receptive fields of 10^3 to 10^6 values. The most famous success of this approach is [Wavenet](https://arxiv.org/abs/1609.03499) which inspired many works in tasks such as [speech separation](https://arxiv.org/abs/1809.07454) or [audio coding](https://arxiv.org/abs/2107.03312). While I understand that extensive experiments in computer vision are already costly in time and resources, it is unfortunate that authors did not consider (or even mention) the huge opportunity that their method could represent for audio understanding and generation.\n* the second big weakness is the model section is very austere and does not guide the reader at all. Not only notations are not detailed (one needs to go to appendix to get the definition of p1 and p2) but the reader is left with a matrix definition and two algorithms without further explanation or high-level description of the framework. I strongly recommend refactoring this section to make it more readable, as a high-level of the concepts, and pseudo-code if necessary belong more in the main text (in my opinion) than the exact backpropagation algorithm which takes half a page.\n* as far as I understand, the authors use a standard convolution algorithm regardless of the level of sparsity of their kernel, which explains why they maintain a high computational cost that does not correlated with the reduced number of parameters. This is unlike standard dilated convolutions that exploit their regular grid to reshape inputs before applying the standard convolution with a kernel of size `k` rather than a kernel of size `s1*s2` (cf. this [tensorflow function](https://www.tensorflow.org/api_docs/python/tf/space_to_batch_nd)). It would be nice if authors discussed this issue.\n* There is a lack of discussion. Do the authors consider the problem of learning the spacings now solved? If not, what would they like to improve in their method and how?\n\nQuestions:\n* I assume the FLOPs reported for DCLS are at inference. It would be interesting to know how training time is affected by the forward and backward passes through the DCLS layers.\n* As explained by authors, dilated convolutions allow increasing the receptive field with a limited parameter budget and computational cost. Recent approaches such as [CKConv](https://arxiv.org/abs/2102.02611) parametrize kernels of the size of the input with a small neural network, and compute this convolution in the Fourier domain. Such an approach offers both the limited parameter budget and computational cost without requiring dilation, and I would be interested to read (in the related work or as an answer) the author's view of such approaches.\n\nTypos:\n* \"FlOPs\" in Table 5 should be \"FLOPs\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QKHwsWcO0F", "reviewer": "Reviewer_N17u", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an advanced version of dilated convolution. In particular, this would mean learning the exact (float) spacing of the convolution locations in addition to the weights, instead of treating the former as a hyper-parameter. Learning of this spacing was enabled through bilinear interpolation as well as deriving the backpropagation rules. Empirically, the paper found that this additional degree of freedom can improve image recognition performance as well as downstream performance, even on very competitive CNN architectures such as ConvNeXt. The authors also provided important techniques that are needed to train this model successfully and analyses such as the empirical receptive field size.", "summary_of_the_review": "Overall I feel positive about the paper at this stage. The problem is an interesting one; not addressed in an overly-complicated manner; and the experimental advantage is solid. But I also feel very strongly about my questions. I feel having concrete answers to those would make the paper even stronger. \n\n=======================================\n\nPOST-REBUTTAL UPDATE\n\nI have read the authors' rebuttal, which I think addresses my questions reasonably well. I also appreciate the authors' efforts in actively improving the paper throughout the process. I have increased my score which shows up as an 8, but that is because there is no 7 to choose between 6 and 8. I would really give this paper a \"7 - Good paper, accept\" rating, and personally vote for acceptance.", "strength_and_weaknesses": "Strength:\n- The paper shows that the convolution coordinates in dilated convolution can be more flexible than a fixed, grid structure. In this sense, this work showcases the middle point between dilated convolution and deformable convolution (the latter has the added flexibility being input-dependent). \n- Though this paper is framed as an extension to dilated convolution, in my opinion it really answers the broader question, which is whether or not the coordinates in convolution can be learned instead of being pre-determined. In this sense, it brings the concept of neural architecture search to a very elementary neuron level (i.e. convolution coordinates).\n- The performance improvement is solid, considering that DCLS is building on competitive architectures (e.g. ConvNeXt) on competitive datasets / tasks (e.g. ImageNet classification). \n- The majority of the paper is well-written and easy to follow, minus a few places which I will mention in the weaknesses below.\n\nWeaknesses:\nQ1 - The clarity of Section 2 can be greatly improved. $p^1$ and $p^2$ are not clearly defined as the x, y float location among $m$ convolution coordinates, and it is up to the reader to connect the dots. Also it wasn't until Section 3 when \"three additional learnable parameters\" is explicitly mentioned. Explicitly mentioning towards the beginning of Section 2 that \"our extension entails that we learn the float coordinates ($p^1$, $p^2$ in the 2D case) for convolution input in addition to the true weight\" would make things a lot easier to understand. Finally, the explanation here can be done in conjunction with Figure 2. Maybe add a big (e) panel which zooms in on four adjacent pixels and draw what $p^1$, $p^2$, $r^1$, $r^2$ mean.\nQ2 - I understand that the length of Section 2 is limited. But I think one important question that needs to be answered in the main text is a clear intuition and explanation why the bilinear interpolation formulation unlocks the learning of the coordinates. How does the derivation change / where does the derivation break,  when $p$ must be integers or if $K$ only considers the closest element instead of the four neighbors? Empirically, will 'nearest' perform much worse than 'bilinear'? The conclusion section writes \"circumvented by interpolation\" but I did not find its justification in the main text.\nQ3 - Why set a limit to the dilated kernel size? Ignoring all model capacity related concerns, does removing this limit always result in better performance?\nQ4 - Deformable convolution was discussed in Section 5, which is very good because it is indeed very related. However, I think one angle the authors did not explicitly mention is the fact that deformable convolutions are *input-dependent*, and dilated convolution / DCLS are not. As such, the deformable convolution paper was able to show visualization where the kernel looks at different (but key) areas when presented with different inputs. Though DCLS also tries to be flexible in its convolution coordinates, I feel the intuition here may be less clear. I wonder if the authors have any comment on this, or if can show visualization of 10 randomly selected learned convolution input patterns?\nQ5 - I appreciate the authors describing the important techniques needed to train DCLS. It is also mentioned in Section 6 that \"in particular sharing the positions within stages was key\". However, I do not see ablation study results / tables that support these claims. Having some of them, or at least the key ones, will strengthen the paper. \nQ6 - Appendix 10, the text writes \"the ERF of ConvNext-T-dcls is larger\", but the ERF in Figure 9 seems smaller than that in Figure 11. Were the captions flipped?\n\nNit:\n- Add space after \"impractical.\" on page 7\n- Cai & Vasconcelos (2018) should be \\citep towards the end of page 7", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "41O2fg8Lqf", "reviewer": "Reviewer_zKnb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of how to allow for larger receptive fields for better accuracy with affordable compuitational complexity. The authors propose an algorithm based on learnable spacing for dilated convolutions.", "summary_of_the_review": "The current version of the manuscript misses several very important papers which have a similar spirit while the motivation may slightly vary. Understanding the significance of paper without comparing it to other approaches is not possible.\n\n===================\n\nWhile the intiial rating was 6, I adjusted it to 8. The authors improved the paper according to my recommendations.", "strength_and_weaknesses": "The motivation of the paper is clear. It is well-understood both from the previous experience, from the related work and from the results presented in the paper, that larger receptive fields allow for more accurate CNNs in some scenatrious. \n\n**Strengths**\n1. The paper is well-written. The story is coherent and it is easy to follow. The main results are presented in an understandable form. \n2. The authors share the implementation of the algorithm which is a huge plus as it allows one to understand it better.\n\n**Weaknesses**\nThe main weakness of the current manuscript is that it does not consider a huge part of the related work. I am sure that the following papers should be taken into consideration. At least, they should be mentioned. It will be a huge plus to my current rating if the proposed method is somehow compared to these papers\n1. In [1] the authors propose to decompose filters as gaussian derivatives and vary their scales. Thus the model is able to utilize larger filters with the same number of trainable parameters. Their results demonstrate that such a method significantly improves the models' performance.\n2. In [2, 3] the authors propose to reparametrizer convolutional filters and learn their parameters. Which is very close to the proposed method in its spirit\n3. In [4, 5, 6, 7, 8, 9] the authors buid scale-scale equivariant neural networks which focus aroung the scale problem and propose everal interesting methods for a problem similar to the considered\n4. In [4, 6, 7] the authors use dilation and an alternative of dilation for fractional spacing, which is close to the proposed idea, although the initial motivation may vary.\n\n- [1] Jacobsen J. H. et al. Structured receptive fields in cnns. \n- [2] Romero D. W. et al. Flexconv: Continuous kernel convolutions with differentiable kernel sizes\n- [3] Romero D. W. et al. Ckconv: Continuous kernel convolution for sequential data\n- [4] Worrall D., Welling M. Deep scale-spaces: Equivariance over scale\n- [5] Sosnovik I., Szmaja M., Smeulders A. Scale-equivariant steerable networks\n- [6] Sosnovik I., Moskalev A., Smeulders A. Disco: accurate discrete scale convolutions\n- [7] Sosnovik I., Moskalev A., Smeulders A. How to Transform Kernels for Scale-Convolutions\n- [8] Bekkers E. J. B-spline cnns on lie groups\n- [9] Zhu W. et al. Scaling-Translation-Equivariant Networks with Decomposed Convolutional Filters\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0-DWPuZLC8", "reviewer": "Reviewer_2NcJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a method for learning sparse convolution kernels (like dilated convolutions) that consists of a weighted sum of shifted pixels. I.e. the kernels can be understood to be expanded of a basis of shifted dirac-deltas, and the learnable parameters are the shifts for each \"pixel\" and its corresponding weight. The layers/kernels are differentiable w.r.t. the shift parameters via bilinear interpolation. The paper then explores what the effect is of changing the standard dense convolution kernels, with the sparse kernels that are able to grow to bigger receptive field sizes while training. Results show that learning the kernel sizes during training is beneficial, whilst typically being more parameter efficient (due to the sparse parametrization), at the cost of only a moderate computational overhead.", "summary_of_the_review": "Not particularly novel, but the value of the paper is in its experimental work and code release, which seems to be properly documented. I cannot recommend accept as the first part of the paper is sloppy and it isn't very clear what the paper adds relative to existing works on kernels in adaptive bases.\n", "strength_and_weaknesses": "**Strengths**\n1. The paper proposes a relatively simple, but effect idea\n2. The method is extensively tested on 2D image tasks and results are clearly presented.\n3. The method has practical value in that it seems to consistently improve performance at only a moderate reduction of througput. Moreover, the authors published their code with the paper.\n4. The paper is transparent in also reporting negative findings; it reports things that worked and things tried, but didn't work \n\n**Weaknesses**\n1. The presentation of the bilinear interpolation is really confusing and possibly incorrect.\n    * Equation 1 defines the kernel in terms of a function f but the function f is nowhere explicitly defined. The definition is somehow implicit in equation 3.\n    * p^1 and p^2 are not defined. I suppose these are the \"x\" and \"y\" coordinates (centers) of the pixels. In the start of section 2, $n$ is not defined, nor is $p$. This is in some sense OK because the labels are only used to indicated the # rows and columns in a matrix, but it confuses with the notation of p for the positions.\n    * In equation 1 however, index i is used to iterate over the weights/centers of the kernel, wherease in (3) it is used to iterate over the rows of K. This is somewhat confusing.\n    * Even more so as in (3) the p^1 and p^j no longer have indices, which p are are talking about?\n    * After a while I figured out that indeed this is just bilinear interpolation where each of the rows in (3) correspond to each of the 4 nearest corner points, but I feel like the presentation could have been much more effective.\n2. Recent works on learnable receptive field sizes seem to be missing. In particlar the works of Romero et al. which show the benefit of (extremely) large receptive field sizes:\n    * Romero, D. W., Kuzina, A., Bekkers, E. J., Tomczak, J. M., & Hoogendoorn, M. (2021). Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611.\nand learnable receptive field sizes:\n    * Romero, D. W., Bruintjes, R. J., Tomczak, J. M., Bekkers, E. J., Hoogendoorn, M., & van Gemert, J. C. (2021). Flexconv: Continuous kernel convolutions with differentiable kernel sizes. arXiv preprint arXiv:2110.08059.\n\n    The idea of using kernels parametrized by a set of shifted weights has been done before by Dai et al. (already discussed in the paper) by shifted dirac deltas, as in this paper, or shifted Gaussians as in:\n    * Jacobsen, J. H., Van Gemert, J., Lou, Z., & Smeulders, A. W. (2016). Structured receptive fields in cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 2610-2619).\n    * Shelhamer, E., Wang, D., & Darrell, T. (2019). Blurring the line between structure and learning to optimize and adapt receptive fields. arXiv preprint arXiv:1904.11487.\n    * Pintea, S. L., Tömen, N., Goes, S. F., Loog, M., & van Gemert, J. C. (2021). Resolution learning in deep convolutional networks using scale-space theory. IEEE Transactions on Image Processing, 30, 8342-8353.\n\n    The idea of sparse localized basis functions with learnable centers has also already been extended to group convolutional neural networks in\n    * Bekkers, E. J. (2019, September). B-Spline CNNs on Lie groups. In International Conference on Learning Representations.\n3. It should be discussed what the current work has to add to the above works. I believe that the emperical study presented in this paper is valuable in itself, but I believe some of the obtained insights have also already been discussed in the above works.\n\nMinor details:\n* It would be nice to give a bit more detail on what depthwise implicit gemm convolution entails. It is mentioned several times and a high level description could help the reader without them having to look up the original paper.\n* Regarding the position initialization (section 3). \"In an attempt to facilitate learning, we chose an initial distribution close to the one obtained at the end of training\", is this reasonable? What if the initial distribution was chosen poorly, then also the one at the end of training would be right? \"Yet in practice, ...\" so would you recommend using uniform initialization then instead?\n* On \"Dilated kernel size tuning\" I can imagine that the overhead could be independent from the constrained kernel size. Since convolution and kernel construction are linear, one could do the interpolation on the input feature maps instead (for the m different shifts) and then simply combine the results with a linear layer afterwards.\n* In related work the effective RF size is discussed and the reader is referred to appendix 10. The idea of looking at the ERFs is interesting, however, app 10 does not give much detail on how the \"heatmaps\" are obtained.\n* When discussin the method in comparison to deformable convolutions, it mentions \"Firstly, in deformable convolutions, the offsets ...\" I do not consider this to be a difference. Parametrizing the locations as offsets rather then absolute points (which are offsets to an origin) boils down to the same thing. Also, the \"secondly\", it indeed describes a difference, but I consider dilated convs to be strictly more general than the proposed work (i.e. one could decide with deformable convs not to make the kernels dependent on the input)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster