Dilated Convolution with Learnable Spacings: beyond bilinear interpolation

Ismail Khalfaoui-Hassani 1 2 Thomas Pellegrini 1 3 Timothée Masquelier 2

Abstract

PyTorch or a more efficient implementation such as the“depthwise implicit gemm” convolution method proposed
by Ding et al. (2022), which is suitable for large kernels.

Dilated Convolution with Learnable Spacings(DCLS) is a recently proposed variation of the dilated convolution in which the spacings between
the non-zero elements in the kernel, or equivalently their positions, are learnable. Non-integer
positions are handled via interpolation. Thanks tothis trick, positions have well-defined gradients.
The original DCLS used bilinear interpolation,and thus only considered the four nearest pixels.
Yet here we show that longer range interpolations,and in particular a Gaussian interpolation, allow
improving performance on ImageNet1k classification on two state-of-the-art convolutional architectures (ConvNeXt and ConvFormer), without increasing the number of parameters. The
method code is based on PyTorch and is available
at github.com/K-H-Ismail/Dilated-Convolutionwith-Learnable-Spacings-PyTorch.

The primary motivation behind the development of DCLS
was to investigate the potential for enhancing the fixed grid
structure imposed by standard dilated convolution in an
input-independent way. By allowing an arbitrary numberof kernel elements, DCLS introduces a free tunable hyperparameter called the “kernel count”. Additionally, the “dilated kernel size” refers to the maximum extent to which
the kernel elements are permitted to move within the dilated
kernel (Fig. 1c). Both of these parameters can be adjusted
to optimize the performance of DCLS. The positions of
the kernel elements in DCLS are initially randomized andsubsequently allowed to evolve within the limits of the dilated kernel size during the learning process. The main
focus of this paper will be to question the choice of bilinear interpolation used by default in DCLS. We tested
several interpolations and found in particular that a Gaussian interpolation with learnable standard deviations madethe approach more effective.

1. Introduction

To evaluate the effectiveness of DCLS with Gaussian interpolation, we integrate it as a drop-in replacement for the
standard depthwise separable convolution in two state-ofthe-art convolutional models: the ConvNext-T model (Liuet al., 2022) and the ConvFormer-S18 model (Yu et al.,2022). In Section 5, we evaluate the training loss and theclassification accuracy of these models on the ImageNet1k
dataset (Deng et al., 2009). The remainder of this paper
will present a detailed analysis of the methods, equations,algorithms and techniques regarding the application of theGaussian interpolation in DCLS.

Dilated Convolution with Learnable Spacings (DCLS) isan innovative convolutional method whose effectivenessin computer vision was recently demonstrated (KhalfaouiHassani et al., 2023). In DCLS, the positions of the
non-zero elements within the convolutional kernels arelearned in a gradient-based manner. The challenge of nondifferentiability caused by the integer nature of the positions
is addressed through the application of bilinear interpolation. By doing so, DCLS enables the construction of adifferentiable convolutional kernel.
DCLS is a differentiable method that only constructs the
convolutional kernel. To implement the whole convolution,
one can utilize either the native convolution provided by

2. Related work
In the field of convolutional neural networks (CNNs), various approaches have been explored to improve the performance and efficiency of convolutional operations. Gaussian
mixture convolutional networks have investigated the fit of
input channels with Gaussian mixtures (Celarek et al., 2022),
while Chen et al. (2023) utilized Gaussian masks in theirwork. Additionally, continuous kernel convolution was studied in the context of image processing by Kim & Park (2023).
Their approach is similar to the linear correlation introduced

1

Artificial and Natural Intelligence Toulouse Institute (ANITI)
CerCo UMR 5549, CNRS, Université Toulouse III, Toulouse,
France 3 IRIT, CNRS, Toulouse INP, Université Toulouse III,
Toulouse, France. Correspondence to: Ismail Khalfaoui-Hassani<ismail.khalfaoui-hassani@univ-tlse3.fr>.
2

Published at the Differentiable Almost Everything Workshop of the
40 th International Conference on Machine Learning, Honolulu,
Hawaii, USA. July 2023. Copyright 2023 by the author(s).

1Dilated Convolution with Learnable Spacings: beyond bilinear interpolation.

2 12 24 36
1 48 96 -48
0 -36 -24 -12
0 1 2
(a)

36
-48
8 -12
6
96
24
4 -24
2
0 -36
48
12
0 2 4 6 8

8
6
4
2
0

8
6
4
2
0

-2 -5-3
42-10 -27-13 2010-5-36432 4
-11-5
20
115
-12 211131-24
53
0 2 4 6 8

(b)

0

2

(c)

4

6

8

(d)

Figure 1. (a) a standard 3 × 3 kernel. (b) a standard dilated 3 × 3 kernel. (c) a 2D-DCLS kernel using bilinear interpolation with 9 kernel
elements and a kernel size of 9. (d) the same kernel as (c) with Gaussian interpolation. The numbers have been rounded in all figures andomitted in (d) for readability.

in Thomas et al. (2019). The interpolation function usedin the last two works corresponds to the DCLS-Triangle
method described in 3.1. Romero et al. have also madenotable contributions in learning continuous functions that
map the positions to the weights (Romero et al., 2022a;b).

limits of the dilated kernel, and we refer to them as the“dilated kernel size”.
The sx × sy matrix space over R is defined as the set of all
sx × sy matrices over R, and is denoted Msx ,sy (R). The
real numbers w, px , σ x , py and σ y respectively stand forthe weight, the mean position and standard deviation of thatweight along the x-axis (width) and its mean position andstandard deviation along the y-axis (height).

In the work by Jacobsen et al. (2016), the kernel is represented as a weighted sum of basis functions, including
centered Gaussian filters and their derivatives. Pintea et al.
(2021) extended this approach by incorporating the learningof Gaussian width, effectively optimizing the resolution.
Shelhamer et al. (2019) introduced a kernel factorization
method where the kernel is expressed as a composition of astandard kernel and a structured Gaussian one. In these last
three works the Gaussians are centered on the kernel.

The mathematical construction of the 2D-DCLS kernel in
Khalfaoui-Hassani et al. (2023) relies on bilinear interpolation and is described as follows :f : R × R × R → Msx ,sy (R)w, px , py 7→

K(1)

Furthermore, the utilization of bilinear interpolation within
deformable convolution modules has already shown its
effectiveness. Dai et al. (2017), Qi et al. (2017) and recently Wang et al. (2022) leveraged bilinear interpolation
to smoothen the non-differentiable regular-grid offsets inthe deformable convolution method. Even more recently, in
Kim et al. (2023), a Gaussian attention bias with learnable
standard deviations has been successfully used in the positional embedding of the attention module of the ViT model(Dosovitskiy et al., 2021) and leads to reasonable gains on
ImageNet1k.

where ∀i ∈ J1 .. sx K, ∀j ∈ J1 .. sy K :w (1 − rx ) (1 − ry ) if i = ⌊px ⌋, j = ⌊py ⌋if i = ⌊px ⌋ + 1, j = ⌊py ⌋ w rx (1 − ry )xyw (1 − r ) r
if i = ⌊px ⌋, j = ⌊py ⌋ + 1
Kij =

x
y

wr r
if i = ⌊px ⌋+1, j = ⌊py ⌋+10otherwise(2)
and where the fractional parts are:

3. Methods

An equivalent way of describing the constructed kernel K
in Equation 2 is:rx = {px } = px − ⌊px ⌋3.1. From bilinear to Gaussian interpolation
We denote by m ∈ N∗ the number of kernel elements inside the dilated constructed kernel and we refer to it as
the “kernel count”. Moreover, we denote respectively bysx , sy ∈ N∗ × N∗ , the sizes of the constructed kernel along
the x-axis and the y-axis. The latter could be seen as theand ry = {py } = py − ⌊py ⌋(3)

Kij = w · g(px − i) · g(py − j)(4)

g : x 7→ max(0, 1 − |x|)

(5)

with
This expression corresponds to the bilinear interpolation as
described in Dai et al. (2017, eq. 4).
2

Dilated Convolution with Learnable Spacings: beyond bilinear interpolation.

0.2

In fact, this last g function is known as the triangle function(refer to Fig. 2 for a graphic representation), and is widelyused in kernel density estimation. From now on, we willnote it as∀x ∈ R

0.15
0.1

def

Λ(x) = max(0, 1 − |x|)

(6)
0.05

First, we consider a scaling by a parameter σ ∈ R+ for the
triangle function (the bilinear interpolation corresponds to
σ = 1),∀x ∈ R,∀σ ∈ R+defΛσ (x) = max(0, σ − |x|)

0
-10

5

10

been tested with no great success. In addition, learning acorrelation parameter ρ ∈ [−1, 1] or equivalently a rotation
parameter θ ∈ [0, 2π] as in the bivariate normal distribution density did not improve performance (maybe because
cardinal orientations predominate in natural images).
3.2. The 2D-DCLS-Gauss kernel construction algorithm

Second, we tried a smoother function rather than the piecewise affine triangle function, namely the Gaussian function:


x2
def
(8)
∀x ∈ R, ∀σ ∈ R∗ , Gσ (x) = exp − 2
2σ

In the following, we describe with pseudocode the kernel construction used in 2D-DCLS-Gauss and 2D-DCLSTriangle. I is the interpolation function (Λ or G in our case)and ϵ = 1e − 7. In practice, w, px , py , σ x and σ y are 3D tensors of size (channels out, channels in //groups, K count), but the algorithm presented here iseasily extended to this case by applying it channel-wise.

We refer to the DCLS method with Gaussian interpolation as
the DCLS-Gauss method. In practice, instead of Equations
7 and 8, we respectively use:∀x ∈ R, ∀σ ∈ R, Λσ0 +σ (x) = max(0, σ0 + |σ| − |x|)
(9)


1
x2
∀x ∈ R, ∀σ ∈ R, Gσ0 +σ (x) = exp −2 (σ0 + |σ|)2(10)
with σ0 ∈ R∗+ a constant that determines the minimum
standard deviation that the interpolation could reach. For
the triangle interpolation, we take σ0 = 1 in order to haveat least 4 adjacent interpolation values (see Figure 1c). And
for the Gaussian interpolation, we set σ0 = 0.27.

Algorithm 1 2D-DCLS-interpolation kernel construction
Require: w, px , py , σ x , σ y : vectors of dimension m
Ensure: K : the constructed kernel, of size (sx × sy )1: K ← 0sx ,sy {zero tensor of size sx , sy }2: for k = 0 to m − 1 do
3:
H ← 0sx ,sy4:
pxk ← pxk + sx //2; pyk ← pyk + sy //25:
σkx ← |σkx | + σ0I ; σky ← |σky | + σ0I
6:for i = 0 to sx − 1 do
7:
for j = 0 to sy − 1 do
8:
H[i, j] ← Iσkx (pxk − i) ∗ Iσky (pyk − j)9:end for10:
end for
−1
sx
−1 sy
P
P
H[i, j])11:
H[:, :] ← H[:, :] /(ϵ +Last, to make the sum of the interpolation over the dilated
kernel size equal to 1, we divide the interpolations by thefollowing normalization term :A=ϵ+0

Figure 2. 1D view of Gaussian and Λ functions with σ = 5.

(7)

We found that this scaling parameter σ could be learned
by backpropagation and that doing so increases the performance of the DCLS method. As we have different σ
parameters for the x and y-axes in 2D-DCLS, learning thestandard deviations costs two additional learnable parameters and two additional FLOPs (multiplied by the number ofthe channels of the kernel and the kernel count). We referto the DCLS method with triangle function interpolation as
the DCLS-Triangle method.

sy
sx X
X

-5

Iσ0 +σx (px − i) · Iσ0 +σy (py − j) (11)i=1 j=1i=0 j=0

K ← K + H ∗ wk
13: end for12:with I an interpolation function (Λ or G in our case) and
ϵ = 1e − 7 for example, to avoid division by zero.
Other interpolations Based on our tests, other functions
such as Lorentz, hyper-Gaussians and sinc functions have3

Dilated Convolution with Learnable Spacings: beyond bilinear interpolation.

Table 1. Classification accuracy on the validation set and training loss on ImageNet-1K. For the 17/34 bilinear, the 23/26 Triangle
and Gaussian cases, the results have been averaged over 3 distinct seeds (the corresponding lines are highlighted in yellow).

MODEL @ 224

KER . SIZE
/ COUNTC ONV N E X T-T ▨C ONV N E X T-T ▩C ONV N E X T-T ⊙C ONV N E X T-T ⋆C ONV N E X T-T
C ONV N E X T-T
C ONV F ORMER -S18 ▨C ONV F ORMER -S18 ▩C ONV F ORMER -S18 ⊙C ONV F ORMER -S18 ⋆INTERPOLATION# PARAM .

TRAIN LOSS

T OP -5 ACC .

T OP -1 ACC .

72 / 49
172 / 34
232 / 26
232 / 26
172 / 26
232 / 34

B ILINEAR
T RIANGLE
G AUSSIAN
G AUSSIAN
G AUSSIAN

28.59M
28.59M
28.59M
28.59M
28.59M
28.69M

2.828
2, 775
2.787
2.762
2.773
2.758

96.05
96.11
96.09
96.18
96.17
96.22

82.08
82.44
82.34
82.44
82.40
82.60

72 / 49
172 / 40232 / 26
232 / 26

B ILINEAR
T RIANGLE
G AUSSIAN

26.77M
26.76M
26.76M
26.76M

2.807
2.764
2.761
2.747

96.17
96.42
96.38
96.31

82.84
83.14
83.09
82.99

the depthwise convolutions by DCLS ones, using the threedifferent interpolations (bilinear, triangle or Gauss). Table 1
reports the results in terms of training loss and validation
accuracy.
A first observation is that all the DCLS models perform
much better than the baselines, whereas they have the same
number of parameters. There are also subtle differences
between interpolation functions. As Figure 3 shows, triangle and bilinear interpolations perform similarly, but theGaussian interpolation performs significantly better.
Furthermore, the advantage of the Gaussian interpolation
w.r.t. bilinear is not only due to the use of a larger kernel, as a
17x17 Gaussian kernel (5th line in Table 1) still outperforms
the bilinear case (2nd line). Finally, the 6th line in Table 1
shows that there is still room for improvement by increasing
the kernel count, although this slightly increases the numberof trainable parameters w.r.t. the baseline.

Figure 3. Training loss for ConvNeXt-T and ConvFormer-S18
models with DCLS according to interpolation type (lower is better).
The pairwise p-values have been calculated using an independent
two-sample Student t-test assuming equal variances. The verticalline segments stand for the standard errors.

6. Conclusion
4. Learning techniques

In conclusion, this study introduces Gaussian and Λ interpolation methods as alternatives to bilinear interpolationin Dilated Convolution with Learnable Spacings (DCLS).
Evaluations on state-of-the-art convolutional architectures
demonstrate that Gaussian interpolation improves performance of image classification task on ImageNet1k without
increasing parameters. Future work could implement theWhittaker-Shannon interpolation instead of the Gaussianinterpolation and search for a dedicated architecture, thatwill make the most of DCLS.

Having discussed the implementation of the interpolation in
the DCLS method, we now shift our focus to the techniques
employed to maximize its potential. We retained most of
the techniques used in Khalfaoui-Hassani et al. (2023), andsuggest new ones for learning standard deviations parameters. In Appendix C, we present the training techniques thathave been selected based on consistent empirical evidence,yielding improved training loss and validation accuracy.

5. Results
We took two recent state-of-the-art convolutional architectures, ConvNeXt and ConvFormer, and drop-in replaced all4

Dilated Convolution with Learnable Spacings: beyond bilinear interpolation.

Acknowledgments

Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T.,
and Xie, S. A convnet for the 2020s. In Proc. IEEE/CVF
Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 11976–
11986, 2022.

This work was performed using HPC resources fromGENCI–IDRIS (Grant 2021-[AD011013219]). Support
from the ANR-3IA Artificial and Natural Intelligence
Toulouse Institute is gratefully acknowledged. We would
also like to thank the region of Toulouse Occitanie.

Pintea, S. L., Tömen, N., Goes, S. F., Loog, M., and vanGemert, J. C. Resolution learning in deep convolutional
networks using scale-space theory. IEEE Transactions
on Image Processing, 30:8342–8353, 2021.

References

Qi, H., Zhang, Z., Xiao, B., Hu, H., Cheng, B., Wei, Y.,
and Dai, J. Deformable convolutional networks–cocodetection and segmentation challenge 2017 entry. In
Proc. ICCV COCO Challenge Workshop, volume 15, pp.
1, 2017.

Celarek, A., Hermosilla, P., Kerbl, B., Ropinski, T., andWimmer, M. Gaussian mixture convolution networks. In
International Conference on Learning Representations,2022.
Chen, Q., Li, C., Ning, J., and He, K. Gaussian mask
convolution for convolutional neural networks. arXiv
preprint arXiv:2302.04544, 2023.

Romero, D. W., Bruintjes, R., Bekkers, E. J., Tomczak,
J. M., Hoogendoorn, M., and van Gemert, J. Flexconv:
Continuous kernel convolutions with differentiable kernel
sizes. In 10th International Conference on Learning
Representations, 2022a.

Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., andWei, Y. Deformable convolutional networks. In Int. Conf.
Comput. Vis., pp. 764–773, 2017.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei,
L. Imagenet: A large-scale hierarchical image database.
In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog.
(CVPR), pp. 248–255. IEEE, 2009.

Romero, D. W., Kuzina, A., Bekkers, E. J., Tomczak, J. M.,
and Hoogendoorn, M. CKConv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=8FhxBtXSl0.

Ding, X., Zhang, X., Han, J., and Ding, G. Scaling up yourkernels to 31x31: Revisiting large kernel design in CNNs.
In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recog.
(CVPR), pp. 11963–11975, 2022.

Shelhamer, E., Wang, D., and Darrell, T. Blurring the line
between structure and learning to optimize and adapt
receptive fields. arXiv preprint arXiv:1904.11487, 2019.
Thomas, H., Qi, C. R., Deschaud, J.-E., Marcotegui, B.,
Goulette, F., and Guibas, L. J. Kpconv: Flexible and deformable convolution for point clouds. Int. Conf. Comput.
Vis., 2019.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,
Heigold, G., Gelly, S., et al. An image is worth 16x16
words: Transformers for image recognition at scale. In
International Conference on Learning Representations,
2021.

Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu,
X., Lu, T., Lu, L., Li, H., et al. Internimage: Exploringlarge-scale vision foundation models with deformableconvolutions. arXiv preprint arXiv:2211.05778, 2022.

Jacobsen, J.-H., Van Gemert, J., Lou, Z., and Smeulders,A. W. Structured receptive fields in cnns. In Proceedingsof the IEEE Conference on Computer Vision and Pattern
Recognition, pp. 2610–2619, 2016.

Yu, W., Si, C., Zhou, P., Luo, M., Zhou, Y., Feng, J., Yan,S., and Wang, X. Metaformer baselines for vision. arXiv
preprint arXiv:2210.13452, 2022.

Khalfaoui-Hassani, I., Pellegrini, T., and Masquelier, T.
Dilated convolution with learnable spacings. In The
Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Q3-1vRh3HOA.
Kim, B. J., Choi, H., Jang, H., and Kim, S. W. Understanding gaussian attention bias of vision transformers using effective receptive fields. arXiv preprint arXiv:2305.04722,
2023.
Kim, S. and Park, E. Smpconv: Self-moving point representations for continuous convolution. arXiv preprint
arXiv:2304.02330, 2023.
5

Dilated Convolution with Learnable Spacings: beyond bilinear interpolation.

A. Code and reproducibility
The code of the method is based on PyTorch and available at https://github.com/K-H-Ismail/Dilated-Convolution-withLearnable-Spacings-PyTorch.

B. Pytorch implementation of the 2D-DCLS-Gauss and 2D-DCLS-Triangle forward algorithm
class ConstructKernel2d(Module):
def __init__(self, out_channels, in_channels, groups, kernel_count,dilated_kernel_size, version):3
super().__init__()4
self.version = version
5
self.out_channels, self.in_channels = out_channels, in_channels
6
self.groups = groups
7
self.dilated_kernel_size = dilated_kernel_size
8
self.kernel_count = kernel_count9
self.IDX, self.lim = None, None
1
2

10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

def __init_tmp_variables__(self, device):
if self.IDX is None or self.lim is None:
J = Parameter(torch.arange(0, self.dilated_kernel_size[0]),requires_grad=False).to(device)
I = Parameter(torch.arange(0, self.dilated_kernel_size[1]),requires_grad=False).to(device)
I = I.expand(self.dilated_kernel_size[0],-1)
J = J.expand(self.dilated_kernel_size[1],-1).t()
IDX = torch.cat((I.unsqueeze(0),J.unsqueeze(0)), 0)
IDX = IDX.expand(self.out_channels, self.in_channels//self.groups,
self.kernel_count,-1,-1,-1).permute(4,5,3,0,1,2)self.IDX = IDX
lim = torch.tensor(self.dilated_kernel_size).to(device)self.lim = lim.expand(self.out_channels,
self.in_channels//self.groups, self.kernel_count, -1).permute(3,0,1,2)
else:
pass

28
29
30
31
32
33
34
35
36
37

def forward_vtriangle(self, W, P, SIG):
P = P + self.lim // 2
SIG = SIG.abs() + 1.0
X = (self.IDX - P)
X = ((SIG - X.abs()).relu()).prod(2)X = X / (X.sum((0,1)) + 1e-7) # normalization
K = (X * W).sum(-1)
K = K.permute(2,3,0,1)
return K

38
39
40
41
42
43
44
45
46
47

def forward_vgauss(self, W, P, SIG):
P = P + self.lim // 2
SIG = SIG.abs() + 0.27
X = ((self.IDX - P) / SIG).norm(2, dim=2)
X = (-0.5 * X**2).exp()
X = X / (X.sum((0,1)) + 1e-7) # normalization
K = (X * W).sum(-1)
K = K.permute(2,3,0,1)
return K

48
49
50
51
52
53
54
55
56

def forward(self, W, P, SIG):
self.__init_tmp_variables__(W.device)elif self.version == 'triangle':
return self.forward_vtriangle(W, P, SIG)
elif self.version == 'gauss':
return self.forward_vgauss(W, P, SIG)
else:
raise

6

Dilated Convolution with Learnable Spacings: beyond bilinear interpolation.

C. Learning techniques• Weight decay: No weight decay was used for positions. We apply the same for standard deviation parameters.
• Positions and standard deviations initialization: position parameters were initialized following a centered normal lawof standard deviation 0.5. Standard deviation parameters were initialized to a constant 0.23 in DCLS-Gauss and to 0 in
DCLS-Triangle in order to have a similar initialisation to DCLS with bilinear interpolation at the beginning.
• Positions clamping : Previously in DCLS, kernel elements that reach the dilated kernel size limit were clamped. It turns
out that this operation is no longer necessary with the Gauss and Λ interpolations.
• Dilated kernel size tuning: When utilizing bilinear interpolation in ConvNeXt-dcls, a dilated kernel size of 17 was
found to be optimal, as larger sizes did not yield improved accuracy. However, with Gaussian and Λ interpolations, there
appears to be no strict limit to the dilated kernel size. Accuracy tends to increase logarithmically as the size grows, withimprovements observed up to kernel sizes of 51. It is important to note that increasing the dilated kernel size does not
impact the number of trainable parameters, but it does affect throughput. Therefore, a compromise between accuracy andthroughput was achieved by setting the dilated kernel size to 23.
• Kernel count tuning: This hyper-parameter has been configured to the maximum integer value while still remaining
below the baselines to which we compare ourselves in terms of trainable parameters. It is worth noting that each additional
element in the 2D-DCLS-Gauss or 2D-DCLS-Triangle methods introduces five more learnable parameters: weight,
vertical and horizontal position, and their respective standard deviations. To maintain simplicity, the same kernel count
was applied across all model layers.
• Learning rate scaling: To maintain consistency between positions and standard deviations, we applied the same learning
rate scaling ratio of 5 to both. In contrast, the learning rate for weights remained unchanged.
• Synchronizing positions: we shared the kernel positions and standard deviations across convolution layers with the same
number of parameters, without sharing the weights. Parameters in these stages were centralized in common parametersthat accumulate the gradients.

D. 1D and 3D convolution cases
For the 3D case, Equation 4 can be generalized as a product along spatial dimensions. We denote respectively bysx , sy , sz ∈ N∗ × N∗ × N∗ , the sizes of the constructed kernel along the x-axis, the y-axis and the z-axis. The constructed
kernel tensor K 3D ∈ Msx ,sy ,sz (R) is therefore:∀i ∈ J1 .. sx K, ∀j ∈ J1 .. sy K, ∀k ∈ J1 .. sz K :3D= w · Iσ0 +σx (px − i) · Iσ0 +σy (py − j) · Iσ0 +σz (pz − k)
Kijk

(12)

with I an interpolation function (Λ or G), σ0 = 1 for the Λ interpolation and σ0 = 0.27 for the Gaussian one. w, px , σ x , py ,σ y , pz and σ z respectively representing the weight, the mean position and standard deviation of that weight along the x-axis(width), the mean position and standard deviation along the y-axis (height) and its mean position and standard deviation
along the z-axis (depth).
The constructed kernel vector K 1D ∈ Rsx for the 1D case is simply:∀i ∈ J1 .. sx K :Ki1D = w · Iσ0 +σx (px − i)
The Algorithm 1 as well as the Pytorch code B are readily adapted to these cases by following the above note.

7(13)