Scale-Equivariant Steerable Networks
Authors: Ivan Sosnovik
Michał Szmaja
Arnold Smeulders
Venue:  (2020)
Keywords: Scale Equivariance
Steerable Filters

Abstract: The effectiveness of Convolutional Neural Networks (CNNs) has been substantially attributed to their built-in property of translation equivariance. However, CNNs do not have embedded mechanisms to handle other types of transformations. In this work, we pay attention to scale changes, which regularly appear in various tasks due to the changing distances between the objects and the camera. First, we introduce the general theory for building scale-equivariant convolutional networks with steerable filters. We develop scale-convolution and generalize other common blocks to be scale-equivariant. We demonstrate the computational efficiency and numerical stability of the proposed method. We compare the proposed models to the previously developed methods for scale equivariance and local scale invariance. We demonstrate state-of-the-art results on the MNIST-scale dataset and on the STL-10 dataset in the supervised learning setting.

Published as a conference paper at ICLR 2020
SCALE-EQUIVARIANT STEERABLE NETWORKS
Ivan Sosnovik', Michal Szmaja!, Arnold Smeulders Delta Lab University of Amsterdam
ABSTRACT
The effectiveness of Convolutional Neural Networks (CNNs) has been substan- tially attributed their built-in property of translation equivariance. However, CNNs do not have embedded mechanisms to handle other types of transforma- tions. In this work, we pay attention to scale changes, which regularly appear various tasks due to the changing distances between the objects and the camera. First, we introduce the general theory for building scale-equivariant convolutional networks with steerable filters. We develop scale-cone convoltonooltion and generalize other common blocks to be scale-equivariant. We demonstrate the computational effi- ciency and numerical stability of the proposed method. We compare the proposed models to the previously developed methods for scale equivariance and local scale invariance. We demonstrate state-of-the-an results the MNIST-scale dataset and on the STL-10 dataset in the supervised learning setting.
INTRODUCTION
Scale transformations occur in many image and video analysis tasks. They are a natural consequence of the variable distances among objects, or between objects and the camera. Such transformations result in significant changes in the input space which are often difficult for models to handle ap- propriately without careful consideration. At a high level, there are two modeling paradigms which allow model to deal with scale changes: models can be endowed with an internal notion of scale and transform their predictions accordingly, or instead, models can be designed to be specifically invariant to scale changes. In image classification, when scale changes commonly factor of 2, it is often sufficient to make class prediction independent of scale. However, in tasks such as image segmentation, visual tracking, or object detection, scale changes can reach factors of 10 more. In these cases, it is intuitive that the ideal prediction should scale proportionally to the input. For example, the segmentation map of a nearby pedestrian should be easily converted that of distant person simply by downscaling.
Convolutional Neural Networks (CNNs) demonstrate state-of-the-art performance in a wide range of tasks. Yet, despite their built-in translation equivariance, they do not have particular mechanism for dealing with scale changes. One way to make CNNs account for scale is to train them with data augmentation Barnard & Casasent 1991). This is, however, suitable only for global transformations. As an alternative, [enriqqes & Vedaldi (207)) and Tai et al. (219)) use the canonical coordinates of scale transformations to reduce scaling to well-studied translations. While these approaches do allow for scale equivariance, they consequently break translation equivariance.
Several attempts have thus been made extend CNNs to both scale and translation symmetry si- multaneously. Some works use input or filter resizing to account for scaling in deep layers Xu et al. (204)); Kanazawa et al. (2014). Such methods suboptimal due the time complexity of tensor resizing and the need for interpolation. In Ghosh & Gupta (2019) the authors pre-calculate filters defined on several scales to build scale-invariant networks, while ignoring the important case of scale equivariance. In contrast, Worrall & Welling (019)) employ the theory of semigroup equiv- ariant networks with scale-space as an example; however, this method is only suitable for integer downscale factors and therefore limited.
Coorespondence to i. www.svikkuvv......
Work was done when the author had an internship the UvA-Bosch Delta Lab Source code available at at/ttp: thub. cww/i sosnovi k/ k/sean

---

Published as a conference paper at ICLR 2020
In this paper develop theory of scale-equivariant networks. We demonstrate the concept of steerable filter parametrization which allows for scaling without the need for tensor resizing. Then derive scale-equivariant convolution and demonstrate a fast algorithm for its implementation. Furthermore, we experiment to determine to what degree the mathematical properties actually hold true. Finally, conduct set of experiments comparing model with other methods for scale cquivariance and local scale invariance.
The proposed model has the following advantages compared to other scale-equivariant models:
It is equivariant to scale transformations with arbitrary discrete scale factors and is not limited to either integer scales or scales tailored by the image pixel grid.
2. It does not rely on any image resampling techniques during training, and therefore, pro- duces deep scale-equivariant representations free of any interpolation artifacts.
3. The algorithm based on the combination of tensor expansion and 2-dimensional convo- lution, and demonstrates the same computation time as the general CNN with a comparable filter bank.
PRELIMINARIES
Before we move into eqale-equivariant mappings, we discuss some aspects of equivariance, scaling transformations, symmetry groups, and the functions defined on them. For simplicity, in this sec- tion, we consider only 1-dimensional functions. The generalization higher-dimensional cases straightforward.
Equivariance Let us consider some mapping g. It is equivariant under La and only if there exists LA such that g Lg Lg g. In case Lg is the identity mapping, the function is invariant.
In this paper we consider scaling transformations. In order to guarantee the equivariance of predictions to such transformations, and to improve the performance of the model, we seek to incor- porate this property directly inside CNNs.
Scaling Given function R + I., scale transformation is defined as follows:
L.ff((x) f(s='r), vs > 0
(1)
We refer to cases with s > as upscale and to cases with s as downscale. we convolve the downscaled function with arbitrary filter to and perform simple change of variables inside the integral, we get the following property:
/ L.If(('))(l -1) xjdd'' / f(s-'2a) 2'''''''''' x))dx
[L.f]] x)x)
(2)
1
-s
= sLL I. [V](x))
In other words, convolution of the downscaled function with a filter can be expressed through convolution of the function with the correspondingly upscaled filter where downscaling is performed
gfterwards. Equation 2 shows us that the standard convolution not scale-equivariant.
Steerable Filters In order to make computations simpler, reparametrize (x) o- -'abbo (b)
which has the following property:
L [Va](x) = (srr = - 1 to (x)
(3)
It gives shorter version of Equation
I. tho Lolf 1//--1a
(4)
We will refer to such a parameterization filters as Steerable Filters because the scaling these filters is the transformation of its parameters. Note that may construct steerable filters from any
2

---

Published as a conference paper at ICLR 2020
function. This has the important consequence that it does not restrict our approach. Rather it will make the analysis easier for discrete data. Moreover, note that any linear combination of steerable filters is still steerable.
Scale-Tree Translation Group All possible scales form the scaling group S. Here we consider the parameter of
discrete scale group, i.e. scales of the form ... ......a!__.. , a, ... with base our method. Analysis of this group by itself breaks the translation equivariance of CNNs. Thus we seek incorporate scale and translation symmetries into CNNs, and, therefore consider the Scale- Translation Group H. It is semidirect product of the scaling group S and the group of translations T = R. In other words: H {(s.t)]s E S,t E T). For multiplication of group elements, we have (2,tt) t1) ($251, s2t1 + (2) and for the inverse (s2, 12) -1 (s1, (1) (s. 51, $22 (t1 ((I-)2)). Additionally, for the corresponding scaling and translation transformations, we have Lat = Lااا # Lg which means that the order of the operations matters.
From now on, we will work with functions defined on groups, i.e. mappings H + R. Note, that simple function f RR + R may considered as function on H with constant value along the S axis. Therefore, Equation A holds true for functions on H as well. One thing we should keep in mind is that when we apply I., to functions on H and R. we use different notations. For example (x') = as= and T. [=',)
[((s, (s', = f(s=' 8---8")--11)
Group-Equivarinn Convolution Given group and two functions f and &b defined on it, G- equivariant convolution is given by
[ [o'))
[ " gg))dp(g))
V ke ()(g)=
Here j((') is the Haar measure also known as invariant measure Folland (016). For 7 we have du(g') dg'- For discrete groups, the Haar measure is the counting measure, and integration becomes a discrete sum. This formula tells us that the output of the convolution evaluated at point the inner product between the function f and the transformed filter Lg [+)].
SCALF-EQUIVARIAN MAPPINGS
Now we define the main building blocks of scale-equivariant models.
Scale Convolution In order to derive scale convolution, we start from group equivariant convolution with G H. We first use the property of semidirect product of groups which splits the integral, then choose the appropriate Haar measures, and finally use the properties of steerable filters. Given the function f(s, t) and steerable filter di (s, defined on H, scale convolution is given by:
if * (s, I ((),t))alll ())
(6)
Σ S ((d (1), '', -)dd''
And for the case of Cin input and Cost output channels we have:
If * the (s,
Σ EIS((''))) Un,m (s (()) 1... Cou
n=1
The proof of the equivariance of this convolution to transformations from H is given in Appendix
Kondor & Trivedi 2018) prove that feed-forward neural network is equivariant to transformations
from G if and only if it is constructed from G-equivariant convolutional layers. Thus Equation 7 shows the most general form of -quivvrriiaai layers which allows for building scale-equivariant convolutional networks with such choice of S. We will refer to models using scale-equivariant layers with steerable filters Scale-Equivvriant Steerable Networks, shortly SESNI
Nonlinearities In order to guarantee the equivariance of the network to scale transformations, we use scale equivariant nonlinearities. We are free use simple point-wise nonlinearities. Indeed,
pronounced 'scason"
3

---

Published as a conference paper at ICLR 2020
point-wise nonlinearities v, like ReLU, commute with scaling transformations:
[ o L.IfI((ss, z') = v(L. = "(( [L, v(f)((s',=')
(8)
= =u(f((5>'s
Pooling Until now we did not discuss how to convert an equivariant mapping to invariant onc. One way to do this is calculate the invariant measure of the signal. In case of translation, such a measure could be the maximum value for example.
First, we propose the maximum scale projection defined as f(s, + max, f(s, This trans- formation projects the function f from H to T. Therefore, the representation stays equivariant to scaling, but loses all information about the scale itself.
Second, we are free to use spatial max max-pooling with moving window or global max pooling. Trans- formation fs, z) + max, f(s, x) projects the function f from to S. The obtained representation is invariant to scaling in spatial domain, however, it stores the information about scale.
Finally, we can combine both of these pooling mechanisms in any order. The obtained transforma- tion produces scale invariant function. It is useful to utilize this transformation closer the end of the network, when the deep representation must be invariant to nuisance input variations, but already has very rich semantic meaning.
IMPLEMENTATION
In this paragraph discuss efficient implementation of Scale-Equivariant Steerable Networks. We illustrate all algorithms in Figure [ For simplicity we assume that zero padding is applied when it is needed for both the spatial axes and the scale axis.
Filter Basis A direct implementation of Equation i impossible due to several limitations. First, the infinite number of scales in calls for a discrete approximation. We truncate the scale group and limit ourselves to Ns scales and use discrete translations instead of continuous ones. Train- ing of SESN involves searching for the optimal filter in functional space which is problem by itself. Rather than solving it directly, we choose a complete basis of N₁ steerable functions I = [v.. ta and represent convolutional filter as linear combination of basis functions with trainable parameters
( In other words, do the following substitution in Equation 7 V/ - K Σ, w.VV;
In experiments
basis of 2D Hermite polynomials with 2D Gaussian envelope, it demonstrates good results. The basis is pre-calculated for all scales and fixed. For filters of size V x V. the basis is stored as an array of shape [Nb, S, V, VJ. See Appendix C Cfor more details.
Cony T + H If the input signal is just function on T with spatial size U x U, stored as an array of shape [Cin: U, U}, then Equation can be simplified. The summation over degenerates, and the final result can be written in the following form:
convTH(f, w, v = squeeze(convzááú, expand x v)))
Here is array of shape (Cous, Cin, No). We compute filter V of shape [Couts Cin, S, VJ and expand it to shape [Cout, Sn V, V]. Then we use standard 2D convolution to produce the output shape [Cout, S, U.. Note that the output can be viewed as
with SourS channels and squeeze a stack of feature maps, where all the features in each spatial position are vectors of S components instead of being scalars as in standard CNNs.
Conv H + H The function on H has a scale axis and therefore there are two options for choosing weights of the convolutional filter. The filter may have just one scale and, therefore. does not capture the correlations between different scales of the input function; or, it may have a non-unitary extent Kss in the scale axis and capture the correlation between Ks neighhoring scales. We refer to the second case as interscale interaction.
the first case w has shape [Cout; Cin; N₁] and Equation 7 degenerates in the same way as before
convHH(J, w, V) squeeze(conv26((xxar(((( expand(w V)))
(10)

---

Published as a conference paper at ICLR 2020
We expand to array of shape S,,,,U U. and expand x v have shape [Cour,, Sn V, V]. The result the convolution then squeezed the same way as before.
In the case of interscale interaction, has shape [Cont Cin- ,, interaction, shift f for each scale, choose a corresponding part of w, and apply conviH to them. We sum the obtained Ks results afterwards.
Nol-- We iterate over all scales in
*
X
expand
squeeze
expand
expand
squeeze
Filter Basis
Conv T' - H
Conv H - H
Figure 1: Left: the way steerable filters are computed using steerable filter basis. Middle and right: representation of scale-convolution using Equation P and Equation TO As an example we use input signal f with 3 channels. It has scale on 7 and 4 scales on H. It is convolved with filter K x W without scale interaction, which produces the output with 2 channels and scales as well. Here we represent only channels of the signals and the filter. Spatial components are hidden for simplicity.
RELATED WORK
Various works on group-equivariant convolutional networks have been published recently. These
works have considered anto-translation groups 2D Cohen & Welling (2016a); [Hoogeboom et al.
2018) Worrall et al. (017); Weiler & Cesa (2099) and 3D Worrall & Brostow (2018): Kondor
2018 Thomas et al. 2018 and rotation equivariant networks 3D Cohen et (017): Esteves et al. (2018): Cohen et al. (2019). In |Fremmnn & Adelson (1991) authors describe the algorithm
for designing steerable filters for rotations. Rotation steerable filters are used in Cohen & Welling
(2016b); Weiler al. (2018a by for building equivariant networks. In Jacobsen et al. (2017) the authors build convolutional blocks locally equivariant to arbitrary k-parameter Lie group by using a steerable basis. And in Murugan al. the authors discuss the approach for learning steerable filters from data. To date, the majority of papers on group equivariant networks have considered rotations in 2D and 3D, but have not payed attention to scale symmetry. As have argued above, it is fundamentally different case.
Many papers and even conferences have been dedicated to image scale-space concept where the image is analyzed together with all its downscaled versions. Initially introduced in [ijima (1959) and later developed by WWikki (1987); Perona & Malik (1990); Lindeberg (2013) scale space relies on the scale symmetry of images. The differential structure of the image Koenderink (1984) allows one to make connection between image formation mechanisms and the space solutions of the 2- dimensional heat equation, which significantly improved the image analysis models in the pre-deep learning era.
One of the first works on scale equivariance and local scale invariance in the framework of CNNs was proposed by Xu et al. (2014) named SiCNN. The authors describe the model with siamese CNNs, where the filters of each instance are rescaled using interpolation techniques. This is the simplest case of equivariance where no interaction between different scales is done in intermediate layers. In SI-ConyNet by Kanazawa et al. (2014) the original network is modified such that, in each layer, the input is first rescaled, then convolved and rescaled back to the original size. Finally, the response with the maximum values is chosen between the scales. Thus, the model is locally scale-invariant. In
5

---

Published as a conference paper at ICLR 2020
Method
Equivariance
Admissible Scales
Approach
Interscale
SICNN
Grid
Filter Rescaling
SI-ConvNet
X
Grid
Input Rescaling
X
SEVE
Grid
Input Rescaling
DSS
Integer
Filter Dilation
SS-CNN
X
Any
Steerable Filters
X
SESN, Ours
Any
Steerable Filters
Table 1: Comparing SESN SiCNN Xu et al. (2014), SI-ConvNet Kanazawa et al. (2014), SEVF
Marcos et al. (2018), DSS Worrall & Welling 2019) and SS-CNN Ghosh & Gupta (2019). "In-
terscale" refers the ability of capturing interscale interactions with kernels of non-unitary scale extent. "Grid" stands for the scales which generate images which lie exactly on the initial pixel grid.
Marcos et (2018), the SEVF model, the input the layers is rescaled and convolved multiple times form vector features instead of scalar ones. The length of the vector in each position is the maximum magnitude the convolution, while the direction of the angle encodes the scale of the image which gave this response. These scale-equivariant networks rely image rescaling which is quite slow. |Worrlll & Welling (019) (DSS) generalize the concept scale-space to deep networks. They use filter dilation analyze the images on different scales. While this approach the standard CNN, it is restricted only to integer downscale factors 2, 4, ..... In Ghosh & Gupta (2019), while discussing SS- SS-CNN the authors use scale-steerable filters to deal with scale changes. The paper does not discuss equivariance, which is an important aspect for scale.
fast as
We summarize the information about these models in Table In contrast to other scale-equivariant models, SESN uses steerable filters which allows for fast scale-convolution with no limitation of flexibility. With the framework of Scale-Equee Equivvraui Convolutional Networks we free build both equivariant and invariant models of different kinds.
EXPERIMENTS
100
10
50-
2-
0.1
0
30
50
# layers
downscaled, x
# scales in interaction
Figure 2:
Equivariance error as function the number of layers (left), downscaling applied to the input image (middle), and function of number of scales in interscale interactions (right). The bars indicate standard deviation.
In this section we conduct the experiments and compare various methods for working with scale variations in input data. Alongside SESN, we test local scale invariant SI-ConvNet and SS-CNN, scale equivariant SiCNN, SEVF and DSS. For SEVF, DSS and SS-CNN by authors, while for others we reimplement the main buildings blocks.
the code provided
We provide additional experimental results time performance of all these methods in Appendix B Due to the algorithm proposed in Section 4 SESN allows for training several times faster than other methods which rely image rescaling.

---

Published as a conference paper at ICLR 2020
Method
(28 x 28)
(28 x 28) +
(56 x 56)
(56 56) +
Params
CNN
2.56 = 0.04
1.96 0.07
2.02
1.60 = 0.09
495
SiCNN
2.40 + 0.03
1.86+
2.02
1.59 + 0.03
497 K
SI-ConvNet
2.40 + 0.12
1.00
1.82
1.59 = 0.10
495
SEVF Scalar
2.30 + 0.06
0.07
1.87
1.62 = 0.07
494
SEVF Vector
2.63 + 0.09
2.009
2.13
1.81 = 0.09
475 K
DSS Scalar
2.53 0.10
2.04
1.02
1.57 = 0.08
494
DSS Vector
2.58 = 0.11
1.95
1.57 = 0.09
494 K
SS-CNN
2.32 + 0.15
2.10
0.10
1.76 - 0.07
494
SESN Scalar
2.10 = 0.10
1.79 0.09
1.74 0.09
1.50 = 0.07
495
SESN Vector
2.08 + 0.09
1.76 + 0.08
1.68 0.06
1.42 0.07
495 K
Table 2: Classification error of different methods on MNIST-scale dataset, lower is better. In ex- periment we use image resolution of 28 x 28 and 56 x 56. We test both the regime without data augmentation, and the regime with scaling data augmentation, denoted with "+" All results are reported as mean + std over 6 different fixed realizations of the dataset. The best results are bold.
6.1 EQUIVARIANCE ERROR
We have presented scale-convolution which is equivariant to scale transformation and translation for continuous signals. While translation equivariance holds true even for discretized signals and filters, scale equivariance may not be exact. Therefore, before starting any experiments, we check to which degree the predicted properties of scale-convolution hold true. We do so by measuring the difference (() where & is scale-convolution with randomly initialized
A [| ( L. ( weights.
In case of perfect equivariance the difference is equal to zero. We calculate the error randomly sampled images from the STL-10 dataset Coates et al. (001I). The results are represented in Fig- the left and the middle plots do not have interscale interactions. The
ure The networks networks on the middle and on the right plots consist of just one layer. We use No- 5, 13, 5 scales for the networks on the left, the middle, and the right plots respectively. While discretization intro- duces some error, it stays very low, and not much higher than 6% for the networks with 50 layers. The difference, however, increases if the input image is downscaled more than 16 times. There- fore, we are free to use deep networks. However, we should pay extra attention to extreme cases where scale changes are of very big magnitude. These are quite rare but still appear in practice. Finally, we see that using SESN with interscale interaction introduces extra equivariance eгror due the truncation of S. We will build the networks with either no scale interaction interaction of scales.
6.2 MNIST-SCALE
Following Kanazawa al. 2014); Marcos (2018) Ghosh & Gupta 2019) we conduct experi- ments on the MNIST-scale dataset. We rescale the images of the MNIST dataset LeCun et al. (T998) to 0.3 1.0 of the original size and pad them with zeros to retain the initial resolution. The scal- ing factors are sampled uniformly and independently for each image. The obtained dataset is then split into 10,000 for training, 2,000 for evaluation and 50,000 for testing. We generate different realizations and fix them for all experiments.
As a baseline model we use the model described in Ghosh & Gupta (019),, which currently holds the flateoffthe-aa result this dataset. It consists of 3 convolutional and 2 fully-connected layers. Each layer has filters of size x We keep the number of trainable parameters almost the same for all tested methods. This is achieved by varying the number of channels. For scale equivariant models we add scale projection at the end of the convolutional block.
For SiCNN, DSS, SEVF and model, additionally train counterparts where after each volution, an extra projection layer is inserted. Projection layers transform vector features in each spatial position of each channel into scalar ones. All of the layers have now scalar inputs instead of

---

Published as a conference paper at ICLR 2020
vector inputs. Therefore, we denote these models with "Scalar". The original models are denoted as "Vector". The exact type of projection depends on the way the vector features are constructed. For maximum pooling along the scale dimension, while for SEVF, it
SiCNN, DSS, and SESN, a calculation of the L2-norm of the vector.
All models are trained with the Adam optimizer Kingma & Ba (2014) for 60 epochs with batch size 128. Initial learning rate is set to 0.01 and divided by 10 after 20 and 40 epochs. We conduct the experiments with different settings. Following the idea discussed in Ghosh & Gupta (2019), in addition to the standard setting we train the networks with input images upscaled to 56 x 56 using bilinear interpolation. This results all image transformations performed by the network becoming more stable, which produces less interpolation artifacts. For both input sizes we conduct the experiments without data augmentation and with scaling augmentation, which results in setups in total. We run the experiments on 6 different realizations of MNIST-scale and report mean + std calculated over these runs.
The obtained results summarized in Table 2 The reported errors may differ a bit from the ones the original paper because of the variations in generated datasets and slightly different training procedure. Nevertheless, we try to keep our configuration as close as possible to Ghosh & Gupta 2019) which currently demonstrated the best classification accuracy on MNIST-scale. For example, SS -SN reports error of 1.91 + 0.04 in Ghosh & Gupta (2019) while it has 1.84 + 0.10 in experiments.
SESN significantly outperforms other methods in all 4 regimes. "Scalar" versions of it already outperform all previous methods, and "Vector" versions make the gain even more significant. The global architectures of all models the same for all rows, which indicates that the way scale convolution is done plays an important role.
6.3
STL
In order to evaluate the role of scale equivariance nat- ural image classification, we conduct the experiments on
dataset[Coates STL-10 dataset Coates et al. (00TT). This dataset consists
Method
Error, %
Params
of 8,000 training and 5,000 testing labeled images. Ad-
WRN
11.48
11.0
ditionally, it includes 100,000 unlabeled images. The im-
SiCNN
11.62
11.0 M
ages have resolution of 96 x 96 pixels and RGB channels.
SI-ConvNet
12.48
11.0
Labeled images belong to 10 classes such as bird, horse or We use only the labeled subset to demonstrate the performance of the models in the low data regime.
DSS
11.28
11.0 M
SS-CNN
25.47
10.8 M
SESN-A
10.83
11.0M
The dataset is normalized by subtracting the per-channel mean and dividing by the per-channel standard deviation. During training,
SESN-B
8.51
11.0
SESN-C
14.08
11.0 M
augment the dataset by applying 12 pixel zero padding and randomly cropping the images to size 96 96. Additionally, random horizontal flips with probability 50% and Cutout De Vries & Taylor 2017 with hole of 32 pixels are used.
Harm WRN
9.55
11.0M
Table 3: Classification error on
STL-10. The best results are bold. We additionally report the current best result achieved by Harm WRN from Ulieny al. (2019).
As a baseline we choose WideResNet Zagoruyko & Ko- mooaaii (2016) with 16 layers and widening factor We set dropout probability to 0.3 in all blocks. We train SESN-A with just vector features. For SESN-B we use maximum scalar projection several times in the interme- diate layers, and for SESN-C we use interscale interaction.
All models trained for 1000 epochs with batch size of 128. We use SGD optimizer with Nesterov momentum of 0.9 and weight decay of 5 10-4. The initial learning rate is set to 0.1 and divided by 5 after 300, 400, 600 and 800 epochs.
The results are summarized Table B. We found SEVF training unstable and therefore do not include it in the table. Pure scale-invariant SI-ConvNet and SS-CNN demonstrate significantly worse results than the bascline. We note the importance of equivariance for deep networks. We also find that SESN-C performs significantly worse than SESN-A and SESN-B due high equivariance
8

---

Published as a conference paper at ICLR 2020
caused by interscale interaction. SESN-B significantly improves the results of both WRN and DSS due to the projection between scales. The maximum scale projection makes the weights of the next layer to have maximum receptive field in the space of scales. This is easy yet effective method for capturing the correlations between different scales. This experiment shows that scale-equivariance is very useful inductive bias for natural image classification with deep neural networks.
To the best of our knowledge, the proposed method achieves new state-of-the-art result on the STL-10 dataset in the supervised learning setting. The previous lowest error is demonstrated in Ulieny et al. (2019). The authors propose Harm WRN a network where the convolutional kernels are represented as a linear combination of Discrete Cosine Transform filters.
7 DISCUSSION
In this paper, we have presented the theory of Scale-Fquivariant Steerable Networks. We started from the scaling transformation and its application to continuous functions. We have obtained the exact formula for scale-equivariant mappings and demonstrated how it can be implemented for discretized signals. We have demonstrated that this approach outperforms other methods for scale-equivariant and local scale-invariant CNNs. It demonstrated new state-of-the-ard results on MNIST-scale and on the STL-10 dataset the supervised learning setting.
We suppose that the most exciting possible application of SESN is in computer vision for au- tonomous vehicles. Rapidly changing distances between the objects cause significant scale vari- ations which makes this well suited for our work. We especially highlight the direction of siamese visual tracking where the equivariance to principle transformations plays an important role.
ACKNOWLEDGMENTS
We thank Daniel Worrall for insightful discussion, Thomas Andy Keller, Victor Garcia, Artem Moskalev and Konrad Groh for valuable comments and feedback.
REFERENCES
Etienne Barnard and David Casasent. Invariance and neural nets. IEEE Transactions on neural networks, 2(5):498 508, 1991.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statisties, pp. 215 215-223, 223, 2011.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning. pp. 2990-2999, 2016a.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling, Convolutional networks for spherical signals. arXiv preprint (700:0099999 2017.
Taco Cohen and Max Welling, Steerable enns. arXiv preprint arXiv: 2016b.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolu- tional networks and the icosahedral enn. arXiv preprint 1rXiv:1902.04615,, 2019.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 68- 2018.
Gerald B Folland. course in abstract harmonic analysis. Chapman and Hall/CRC, 2016.
William T. Freeman and Edward H Adelson. The design and use of steerable filters. IEEE Transac- fions Pattern Analysis & Machine Intelligence, (9):891-906, 1991.

---

Published as a conference paper at ICLR 2020
Rohan Ghosh and Anupam K Gupta. Scale steerable filters for locally neural networks. arXiv preprint arXiv:1906.03380 03861, 2019.
convolutional
Joao F Henriques and Andrea Vedaldi. Warped convolutions: Efficient invariance to spatial trans-
formations. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1461-1469. JMLR. org, 2017.
Emiel Hoogeboom, Jorn WT Peters, Taco S Cohen, and Max Welling, Hexaconv. arXiv preprint arXiv: 020.0 2018.
Taizo lijima. Basic theory of pattern observation. Technical Group on Automata and Automatic Control, pp. 3-32, 1959.
Jörn-Henrik Jacobsen, Bert De Brabandere, and Arnold WM Smeulders. Dynamic steerable blocks in deep residual networks. arXiv preprint arXiv: 00598, 2017.
Angjoo Kanazawa, Abhishek Sharma, and David Jacobs. Locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1412.5104, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jan J Koenderink. The structure of images. Biological cybernetics, 50(5):363-370, 1984.
Risi Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint arXiv:1803.01583 2018.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. arXiv preprint arXiv: (rXXv:8802.0369, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278- 1998.
Tony Lindeberg. Scale-space theory in computer vision, volume 256. Springer Science & Business Media, 2013.
Diego Marcos, Benjamin Kellenberger, Sylvain Lobry, and Devis Tuia. Scale equivariance in enns with vector fields. arXiv preprint arXiv: 2018.
Muthuvel Murugan, KV Subrahmanyam, and Siruseri IT Park. So (2)-equivariance in neural net- works using tensor nonlinearity.
Pietro Perona and Jitendra Malik, Scale-space and edge detection using anisotropic diffusion. IEEE Transactions pattern analysis and machine intelligence, 12(7):629-6)) 6399, 1990.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant transformer networks. arXiv preprint arXiv:1901.11999 2019.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an and translation-equivariant neural networks for 3d point clouds. arXiv preprint /02.v21882.082/9, 2018.
Matej Ulicny, Vladimir A Krylov, and Rozenn Dahyot. Harmonic networks with limited training samples. arXiv preprint wrXiv:7905.0013: 2019.
Maurice Weiler and Gabriele Cesa. General e (2)vvvuiaa steerable cnns. In Advances Neural Information Processing Systems, pp. 14334-14345, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, pp. 10381-10392) 2018a.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equiv- ariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849-858, 2018b.
10

---

Published as a conference paper at ICLR 2020
Andrew P Witkin. Scale-space filtering. In Readings in Computer Vision, pp. 329-332. Elsevier, 1987.
Daniel Worrall and Gabriel Brostow. Cubenet:
Equivariance to 3d rotation and translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 567-584, 2018.
Daniel E Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. arXiv preprint
a 1697, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 5028-5037 2017.
Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang. Scale-invariant volutional neural networks. arXiv preprint arXiv:1411.6369, 2014,
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint

---

Published as a conference paper at ICLR 2020
A PROOF OF EQUIVARIANCE
Let us first show that scale-convolution defined in Equation 6 is equivariant to translations.
[TA]]]]] *n (s)
> (Tlll((s's)) * (5-),'
Σ
(()))
(II)
LR Eit(s' x (s 1555s//1 s .2 )1 (t)
= L,U * (s,(s,
Now we show that scale convolution is equivariant to scale transformations:
[Ta[f] *H 4o (=)(s,
> [Tglf((s')) ( 8 (1)
Lals(3 (., ths 1s0 1s ((1))
(12)
Eif(s" U (Ss -1 1s" (](
[f * vol(s ss (t) Lslf * vo (s, t)
Finally, we can use the property of semidirect product of groups
ssil *H Vo Labils] *N Vo = La(4,l/] * va) = La L *M vol = Lsels ** wal
(13)
12

---

Published as a conference paper at ICLR 2020
B TIME PERFORMANCE
We report the average time per epoch of different methods for scale equivariance and local scale invariance in Table 4 Experimental setups from Section 6.2 are used. We used 1 Nvidia GeForce GTX 1080Ti GPU for training the models.
The methods relying on image rescaling techniques during training (SiCNN, SI-ConvNet, SEVF) demonstrate significantly worse time performance that the ones, using either steerable filters or filter dilation. Additionally, we see that our method outperforms SS-CNN by wide margin. Despite the similar filter sizes and comparable number of parameters between SS-CNN and SESN Scalar, the second one demonstrates significantly better results due to the algorithm proposed Section 4 Finally, DSS performs slightly faster in some cases than our method as each convolution involves less FLOPs. Dilated filters are sparse, while steerable filters are dense.
Method
28 x 28, $
56 x 56, S
CNN
3.8
3.8
SiCNN Scalar
13.5
18.9
SiCNN Vector
15.3
22.8
SI-ConvNet
18.4
33.1
SEVF Scalar
21.0
38.4
SEVF Vector
25.4
46.0
DSS Scalar
3.9
5.0
DSS Vector
3.9
4.8
SS-CNN
14.8
16.6
SESN Scalar
3.8
5.1
SESN Vector
3.8
6.8
Table 4: Average time per epoch during training input data with resolution 28 x 28 and 56 56.
C
BASIS
Assuming that the center of the filter is point ().0) in coordinates (x, we use the filters of the following form:
x² + y²
Hn (.. 1 0²
1
Hm
exp
(14)
20
Here A a constant independent on a, Hn Hermite polynomial of the n-th order. We iterate over increasing pairs of n, generate the required number of functions.
13

---

Published as a conference paper at ICLR 2020
D MODEL CONFIGURATION
D.1 MNIST-SCALE
Method
Conv
Conv
Conv
FC
# Scales
CNN
32
SiCNN
32
63
95
SI-ConvNet
32
95
SEVE Scalar
32
63
95
8
256
SEVF Vector
23
45
68
DSS
32
63
95
SS-CNN
30
60
90
6
SESN
Table Number channels in convolutional layers, number of units in fully-connected layers and number of scales used by different models in Section 6.2
D.2 STL-10
Method
Block 1
Block 2
Block 3
Scales
CNN
16
32
64
1
SiCNN
16
32
64
3
SI-ConvNet
16
32
64
3
SEVF
11
23
45
3
DSS
16
32
64
4
SS-CNN
11
22
44
3
SESN
16
32
64
3
Table Number of channels convolutional blocks and number of scales used by different models in Section 6.2 We report the number of channels up to the widening factor.
14


Reviews:
[{"review_id": "SkgxHUDycS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework (SESN) for learning deep networks that possess scale equivariance in addition to translation invariance. The formulation is based on group convolution on the scale-translation group. Filters are represented as the coefficients of a set of continuous basis functions, which are sampled (once) at a discrete set of scales. The theoretical formulatioin is clear and interesting. The approach is evaluated in terms of image classification accuracy. The set of baselines is quite exhaustive, including recent papers and papers that are not widely-known.\n\nThe most significant improvement for the STL-10 dataset was obtained by the SESN-B variant. This is interesting, because it applies the same operation independently at multiple scales and periodically performs global pooling over scale.\n\nThe effectiveness of the approach was demonstrated in the low-data regime, where the inductive bias of scale equivariance is more likely to help.\n\nOverall I found the paper to be thought-provoking and well-executed. There are a number of questions that I would still like to see investigated, but nevertheless I feel that this paper already represents a worthwhile contribution.\n\nMost important issues and questions:\n\n(1.1) The SESN-B architecture resembles quite closely the SI-ConvNet architecture of Kanazawa et al. (except that that paper resized the images instead of the filters). While your approach may be more computationally efficient, it's not clear what leads to the improvement in accuracy here? Can you explain the difference?\n\n(1.2) I would have preferred to see the approach demonstrated on a task which possesses scale equivariance, such as semantic segmentation.\n\n(1.3) To argue in favour of the continuous basis, it would have been more convincing to compare against directly learning the filters at the highest resolution and obtaining the other filters by downsampling. This would not represent a runtime cost during inference.\n\n(1.4a) It seems that SESN-C should contain SESN-A as a special case. However, SESN-C is worse than SESN-A. Do you have any idea whether this is due to optimization difficulty or over-fitting? Could you compare the training objectives?\n(1.4b) It is stated that the scale equivariance of SESN-C is worse than SESN-A and -B. However, it should still be scale equivariant, except for boundary effects in scale? What is the parameter N_S in this experiment compared to the number of scales S? And the same question for the plot on the right in Figure 2.\n(1.4c) How does SESN-C have the same number of parameters as SESN-A and SESN-B? I thought that more parameters would be required to compute interscale interaction. Was the number of channels reduced?\n\nIssues with clarity:\n\n(2.1) The explanation of equation 10 is not clear. In particular, the diagonal structure in Figure 1 is not stated anywhere in the text, it is simply explained as an expansion from [C_out, C_in, S, V, V] to [C_out S, C_in S, V, V].\n\n(2.2) It's not immediately apparent how multiple applications of convHH are used to provide interscale interaction. I assume it is achieved by shifting f or psi in the scale dimension for each application of convHH, or equivalently by modifying the base-scale in the basis?\n\n(2.3) The explanation of \"scalar\" and \"vector\" variants in the experimental section was not perfectly clear. It is stated that \"all the layers have now scalar input instead of vector input.\" However, I understood that the max-reduction was only over the scale dimension, not the channel dimension, so that the inputs are still vectors? This is confusing as a reader.\n\n(2.4) The expansion of the filters to a diagonal structure is described in the \"implementation\" section. However, it seems that this would entail wasteful multiplications by zero. Nevertheless, SESN is shown to be highly efficient in the appendix. Do you avoid these pointless operations in the actual implementation?\n\n(2.5) It was not immediately obvious how $\\psi_{\\sigma}(s, t)$ was related to $\\psi_{\\sigma}(x)$.\n\nOther details:\n\n(3.1) In Figure 2, how many layers does the network have which was used to construct the middle plot?\n\n(3.2) It would have been useful to include a study of the effect of the range and resolution of the scale space.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g-eN-3KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method for integrating scale equivariance into convolutional networks using steerable filters.  After developing the theory using continuous scale and translation space, a discretized implementation using a fixed set of steerable basis elements is described.  Experiments are performed measuring the error from true equivariance, varying number of layers, image scale and scales in scale interactions.  The method is evaluated using MNIST-scale and STL-10, with convincing results on MNIST-scale and bit less convincing but still good results on STL-10.\n\nOverall, I think this is a nice paper with generally good explanations and experiments probing the behavior.  I would have liked to see more probing into the effects of number and distance between scales.  Table 1 and corresponding text say that a significant advantage of the approach is that it can handle arbitrary scale values, but there was no explicit exploration of the effects of using this beyond one set of scales per experiment/dataset.  What scale values can be sampled, which work best, and why?\n\nAlso, while the MNIST-scale experiment seems convincing, I think the STL-10 is a bit less (but still OK):  Although the method outperforms other methods and appropriate baseline models, it's a little disappointing that pooling over scales (which I would would convert the equivariance to invariance) is best, and inter-scale interactions increase error.  (Perhaps this is not too surprising in retrospect, as images may have limited scale variation from camera position in this dataset, but significant within-class viewpoint variation.)\n\nEven so, I still find the method concise and of interest, with the basics evaluated, even if some of its unique advantages may have been better explored.\n\n\nAdditional Questions:\n\n* Inter-scale interaction could be elaborated a bit more.  End of sec 4 says, \"use convHH for each scale sequentially and .. sum\".  I believe this is sequencing over scales in the kernel; explaining a bit better how this is implemented, including the shape of w in this case, would be helpful.\n\n* Which scales were chosen for the fixed basis?  How large in spatial extent are the kernels in the basis elements, at each scale?\n\n* In the implementation, what is the value of V (sampled 2d conv kernel size)?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJgeyM2iYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed scale-equivariant steerable convolutional neural networks that is able to preserve both the translation and scaling symmetry of the data in the representation. To achieve this, the authors developed the scale-convolution blocks in the network, and generalized other common blocks, such as pooling and nonlinearity, to remain scale-equivariant. Extensive experiments have been conducted to show that the proposed scale-equivariant network\n(a) is indeed scale-equivariant even with numerical discretization\n(b) achieves better classification performance when compared to non-scale-equivariant networks as well as previously proposed locally-scale invariant networks.\n\nOverall, this is a very good paper. The paper is well-written and well-organized. The newly proposed scale-convolution is the most general way of achieving scale-equivariant representations. Experiments are convincing and justifies the usage of the proposed architecture in dealing with multi-scale inputs.\n\nOne question to ask that does not effect my rating:\nThere is a very similar paper submitted to this conference:\nhttps://openreview.net/forum?id=rkgCJ64tDB\nWould you care to make a comparison between these two manuscript?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a theory for building scale-equivariant CNNs with steerable filters. The proposed method is compared with some of the related techniques . SOTA is achieved on MNIST-scale dataset and gains on STL-10 is demonstrated. The reviewers had some concern related to the method, clarity, and comparison with related works. The authors have successfully addressed most of these concerns. Overall, the reviewers are positive about this work and appreciate the generality of the presented theory and its good empirical performance. All the reviewers recommend accept.
Decision: Accept (Poster)