Deformation Robust Roto-Scale-Translation Equivariant CNNs
Authors: Liyao Gao
Guang Lin
Wei Zhu
Venue: Accepted by TMLR ()

Abstract: Incorporating group symmetry directly into the learning process has proved to be an effective guideline for model design. By producing features that are guaranteed to transform covariantly to the group actions on the inputs, group-equivariant convolutional neural networks (G-CNNs) achieve significantly improved generalization performance in learning tasks with intrinsic symmetry. General theory and practical implementation of G-CNNs have been studied for planar images under either rotation or scaling transformation, but only individually. We present, in this paper, a roto-scale-translation equivariant CNN ($\mathcal{RST}$-CNN), that is guaranteed to achieve equivariance jointly over these three groups via coupled group convolutions. Moreover, as symmetry transformations in reality are rarely perfect and typically subject to input deformation, we provide a stability analysis of the equivariance of representation to input distortion, which motivates the truncated expansion of the convolutional filters under (pre-fixed) low-frequency spatial modes. The resulting model provably achieves deformation-robust $\mathcal{RST}$ equivariance, i.e., the $\mathcal{RST}$ symmetry is still "approximately” preserved  when  the  transformation is "contaminated” by a nuisance data deformation, a property that is especially  important  for  out-of-distribution generalization. Numerical experiments on MNIST, Fashion-MNIST, and STL-10 demonstrate that the proposed model yields remarkable gains over prior arts, especially in the small data regime where both rotation and scaling variations are present within the data.

Published in Transactions on Machine Learning Research (07/2022)
Deformation Robust CNNs
Roto-Scale- Translaionnn
Equivariant
L. Mars Gao Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195, USA
marsgno@uu.eddv
Guang Lin
guanglin@@purden edu
Department of Mathematics and School of Mechanical Engineering Purdue University West Lafayelle, IN 47907, USA
Wei Zhu
weizhushumasss.o edu
Department of Mathematics and Statistics University of Massachusetts Amherst Amherst, MA 01008, USA
Reviewed on OpenReview: https: openreview. net/ forum? id- d=yVkkpas77cD
Abstract
Incorporating group symmetry directly into the learning process has proved to be an effec- tive guideline for model design. By producing features that are guaranteed to transform covariantly the group actions on the inputs, group-equivvriant convolutional neural net- works (G-CNNs) achieve significantly improved generalization performance in learning tasks with intrinsic symmetry. General theory and practical implementation of G-CNNs have been studied for planar images under either rotation scaling transformation, but only individu- ally. We present, in this paper, aito-scale-translatio equivariant CNN (RST-CNN), that is guaranteed to achieve equivariance jointly over these three groups via coupled group con- volutions. Moreover, as symmetry transformations in reality are rarely perfect and typically subject to input deformation, we provide a stability analysis the equivariance of repre- sentation input distortion, which motivates the truncated expansion of the convolutional filters under (pre-fixed) low-frequency spatial modes. The resulting model provably achieves deformationn-rousst RST equivariance, i.e., the RST symmetry is still "approximately" preserved when the transformation is "contaminated" by nuisance data deformation, property that is especially important for distribbtttiinno generalization. Numerical ex- periments on MNIST, Fashion-MNIST, and STL-10 demonstrate that the proposed model yields remarkable gains over prior arts, especially in the small data regime where both rotation and scaling variations are present within the data.
1 Introduction
Symmetry is ubiquitous in machine learning. For instance, in image classification, the class label of an image remains the same when the image is spatially translated. Convolutional neural networks (CNNs) through spatial weight sharing achieve built-in hunslation-eqiiurrannee i.e., a shift of the input leads to a correspond- ing shift of the output, a property that improves the generalization performance and sample complexity of the model for computer vision tasks with translation symmetry, such as image classification (Krizhevsky et al., 2012), object detection (Ren al., 2015), and segmentation (Long et al., 2015; Ronneberger et al., 2015).

---

Published in Transactions on Machine Learning Research (07/2022)
Inspired by the standard CNNs, researchers in recent years have developed both theoretical foundations and practical implementations of group equivarianl CNNs (G-CNNs), i.e, generalized CNN models that guarantee a desired transformation on layer-wise features under a given input group transformation, for signals defined on Euclidean spaces (Cohen & Welling, 2016; 2017; Worrall et al., 2017; Weiler ct al., 2018b;a; Worrall & Welling, 2019; Sosnovik et al., 2020; Cheng et al., 2019; Zhu et al., 2019; Weiler & Cesa, 2019; Hoogeboom et al., 2018; Zhou et al., 2017; Worrall & Brostow, 2018; Kanazawa ct al., 2014; Xu et al., 2014; Marcos et al., 2018), manifolds (Cohen et al., 2018; 2019; Kondor et al., 2018; Defferrard et al., 2020), point clouds (Thomas et al., 2018; Chen et al., 2021; Zhao et al., 2020), and graphs (Kondor, 2018; Anderson et al., 2019; Keriven & Poyré, 2019). In particular, equivariant CNNs for either rotation (Weiler & Cosa, 2019; Cheng et al., 2019; Hoogeboom et al., 2018; Worrall et al., 2017; Zhou et al., 2017; Marcos et al., 2017; Weiler et al., 2018b) or scaling (Kanazawa et al., 2014; Marcos et al., 2018; Xu el al., 2014; Worrall & Welling, 2019; Sosnovik et al., 2020; Zhu et al., 2019; Sosnovik et al., 2021) transforms on 2D inputs have been well studied separately, and their advantage has been empirically verified in scttings where the data have rich variance in either rotation or scale individually.
However, for many vision tasks, it is beneficial for model to simultaneously incorporate translation, rotation, and scaling symmetry directly into its representation. For example, a self-driving vehicle is required to recognize and locate pedestrians, objects, and road signs under random translation (e.g., moving pedestrians). rotation (e.g., tilted road signs), and scaling (e.g., close and distant, objects) (Bojarski et al., 2016)- Moreover, in realistic settings, symmetry transformations are rarely perfect; for instance, a tilted stop sign located faraway can be modeled in reality as if it were transformed through a sequence of translation, rotation and scaling following local data deformation, which results from (unavoidable) changing view angle and/or digitization. It is thus erucial to design roto-scalc-translatio equívariant CNNs (RST-CNNs) with provably robust equivariant represenlation such that the RST symmetry is still "approximately" preserved when the transformation is "contaminated" by a muisance data deformation. Such deformation robustness is especially important for out-of-distribution generalization. However, the design of ReS--quivariant CNNs with theoretically guaranteed deformation robustness is challenging both in theory and in practice due to the intertwined convolution on the non-compact RST group with infinite Haar measure.
The purpose of this paper is to address both the theoretical and practical aspects of constructing deformation robust RST-CNNs, which, to the best of our knowledge, have not been jointly studied in the computer vision community. Specifically, our contribution is three-fold:
We propose ettosccae-trranlatiin equivariant CNNs with joint convolutions over the space R2, the rotation group SO(2), and the scaling group S, which shown to be sufficient and necessary for equivariance with respect to the regular representation of the group RST.
2. We provide a stability analysis of the proposed model, guaranteeing its ability to achieve equivariant representations that are robust to muisance data deformation.
3. Numerical experiments are conducted to demonstrate the superior (both in-distribution and out-of- distribution) generalization performance of our proposed model for vision tasks with intrinsic RST symmetry, especially in the small data regime.
2 Related Works
Group-equivariant CNNs (G-CNNs). Since its introduction by Cohen & Welling (2016), a variety of works on G-CNNs have been conducted that consistently demonstrate the benefits of bringing equivariance prior into network designs. Based on the idea proposed in (Cohen & Welling, 2016) for discrete symmetry groups, G-CNNs with group convolutions which achieve equivariance under regular representations of the group have been studied for the 2D (and 3D) rolo-translation group SE()) (and S SE()) (Weiler & Cesa, 2019; Cheng et al., 2019; Hoogeboom et al., 2018; Worrall et al., 2017; Zhou et al., 2017; Marcos et al., 2017; Weiler et al., 2018b; Worrall & Brostow, 2018), iolling-rranslttion group (Kanazawa et al., 2014; Marcos et al., 2018; Xu et al., 2014; Worrall & Welling, 2019; Sosnovik et al., 2020; Zhu al., 2019; Sosnovik et al., 2021), rotation S0(3) on the sphere (Cohen et al., 2018; Kondor et al., 2018; Defferrard et al., 2020), and permutation on
2

---

Published in Transactions on Machine Learning Research (07/2022)
graphs (Kondor, 2018; Anderson et al., 2019; Keriven & Poyré, 2019). Polar transformer networks (Esteves et al., 2017) generalizes group-equivariance to rotation and dilation. B-spline CNNs (Bekkers, 2020) and LieConv (Finzi et al., 2020) generalize group convolutions to arbitrary Lie groups on generic spatial data, albeit achieving slightly inferior performance compared to G-CNNs specialized for Euclidean inputs (Finzi et al., 2020). Steerable CNNs further generalize the network design to realize equivariance under induced representations of the symmetry group (Cohen Welling, 2017; Weiler et al., 2018a; Weiler & Cesa, 2019; Cohen et al., 2019), and the general theory has been summarized by Cohen et al. (2019) for homogeneous spaces.
Representation robustness to input deformations. Input deformations typically introduce noticcable yet uninformative variability within the data. Models that are robust to data deformation are thus favorable for many vision applications. The scattering transform network (Bruna & Mallat, 2013; Mallat, 2010; 2012), a multilayer feature encoder defined by average pooling of wavelet modulus coefficients, has been proved to be stable to both input noise and muisance deformation. Using group convolutions, scattering transform has also been extended in (Oyallon & Mallat, 2015; Sifre & Mallat, 2013) to produce invariant features. Despite being pioneering mathematical model, the scattering network uses pre-fixed wavelet transforms in the model, and is thus non-adaptive to the data. Stability and invariance have also been studied in (Bietti & Mairal, 2017; 2019) for convolutional kernel networks (Mairal, 2016; Mairal et al., 2014). DCFNet (Qiu et al., 2018) combines the regularity of a pre-fixed filter basis and the trainability of the expansion coefficients, achieving both representation stability and data adaptivity. The idea is later adopted in (Cheng et al., 2019) and (Zhu et al., 2019) in building deformation robust models that are equivariant either rotation or scaling individually.
Despite the growing body of literature in G-CNNs, to the best of our knowledge, no G-CNNs have been specifically designed to simultaneously achieve roto-scalettranslation equivariance. More importantly, no stability analysis has been conducted to quantify and promote robustness of such equivariant model to muisance input deformation.
3
Roto-Scale Translation Equivariant CNNs
We first explain, in Section 3.1, the definition of groups group representations, and group-equivvaiance, which serves as a background for constructing RST-CNNs in Section 3.2.
3.1 Preliminaries
Group. A group is a set G equipped with a binary operator : G x G - G, satisfying associativity and the existence of an identily e as well as an inverse element g-1 for all g E G. In this paper, we consider in particular the roto-scale-translation group RST = (5O(2) x R) k R2 = (11,8, v) : n E 20, R, e E RR?. with the group multiplication
(n. a0, u) = (0 + a + B, 2
(1)
where Ry is a counterclockwise rotation (around the origin) by angle n applied to point u E R²,
Group action and representation. Given a group G and a set X, D, : + X' is called a G-action on X if Dy is invertible for all g E G, and Dg, o Dg- = =Da ga, Vgh 92 E G, where o denotes map composition. A G-action D, is called a G-representation if is further assumed to be a vector space and D, is linear for all g E G. In particular, given an input RGB image z0() (u, X) modeled in the continuous setting (i.e., 200) is the intensity of the RGB color channel X E (1,2,3) at the pixel location R²), (representation) D = 1,3.v
transformation on the image x. (0)) (u, A) can be understood as an acting on the input ());
1. (0) = z0)) (R, (u))))))
(2)
i.e. the transformed image [D, (0))) is obtained through an n) rotation, 2 scaling, and translation.
3

---

Published in Transactions on Machine Learning Research (07/2022)
Group Equivariance. Let f: X - V be map between X' and V. and Dÿ Dÿ be G-actions on d' and y respectively. The map f is said to be G-equivariant if
f(D) (f(xe)) VgEG, ZEX.
(3)
A special case of (3) is G-invariance, when Dý is set to Idy, the identity map on V. For vision tasks where the output y E y is known a priori to transform covariantly through D to D2 transformed input, c.g., the class label y remains identical for rotated/rescian rescaaee input a E X, is beneficial to consider only G-equivariant models f to reduce the statistical error of the learning method for improved generalization.
3.2 Equivariant Architecture
Since the composition of equivariant maps remains equivariant, to construct an L-layer RST-CNN, we only need to specify the RST-action ((1) on each feature space Xr0), S 1 S L, and require the layer-wise mapping be equivariant:
1.3.0
D
7. 1, 44 221
(4)
1,8,0
where we slightly abuse the notation x0 feature (((-1)_ In particular, we define n, on the input as in (2); for the hidden layers S S I,, we let x2 be the feature space consisting of features in the form of 21, a, A), where u E IR2 is the spatial position, 0 E [0,2%] 2x] is the rotation index, R is the scale index, X E [Mi] (I....,M, corresponds to the unstructured channels (similar to the RGB channels of the input), and we define the action ( x²
x00(x'' [x to denote the I-th layer output given the (1 1)-th layer
7,8, on
[D (0 a. = (m v -,
-3,)), 1
(5)
We note that (5) corresponds to the regular representation of RST on x110 (Cohen et al., 2019), which is adopted in this work as its ability to encode any function on the group RST leads typically to better model expressiveness and stronger generalization performance (Weiler Cesa, 2019). The following proposition outlines the general network archilecture to achieve RST-equivariance under the representations D 7.8.r (2) (5).
Proposition 1. An I-laaer feedforward neural network is RST -equivariant under the representations (2) (5) if and only if the layer-wiss wise operations are defined as (6) and (7):
(u, = ] x
x. 20 (u+ 202 v (2- du' + tB(()))
(6)
R²
000 (1)))())) 0 0,
=o Σ
(111 u'.0 . aa 2 20(()) "R ,0, a') do' do' + (0(N) (7)
v.
where o R + R. is pointwise nonlinearity, W (u) is the spatial convolutional filter in the first layer with oulpul channel X and impul channel X, wilh X (u,0, 0, a) is the RST joint convolutional filter for layer >> 1, and Sg f(a)da denoles the normalized 51 integral 2u fea (aa)da.
We note that the joint-convolution (7) is the group convolution over RST (whose subgroup 50(2) x R is non-compact Lie group acting on IR?), which is known to achieve equivariance under regular representations (5) (Cohen et al., 2019). We provide an elementary proof of Proposition the appendix for completeness.
4 Robust Equivariance to Input Deformation
Proposition details the network architecture to achieve RST-equivariance for images modeled on the continuous domain 2 undergoing "perfect" RST-transformation (2). However, in practice, symmetry
4

---

Published in Transactions on Machine Learning Research (07 (07/2022)
2
D ( (0) n,8,0
D
2
D (0) n,8,0 o Dr
Figure 1: RrSTttrausformations in reality, e.g.: the blue arrow, are rarely perfect, and they can be modeled as composition of input deformation D, [cf. (12) and an exact RST transforrmmtnn ( n.S.v'
transformations are rarely perfect, as they typically suffer from numerous source of input deformation coming from, for instance, unstable camera position, change of weather, as well as practical issues such as numerical discretization and truncation (see, for example, Figure 1.) We explain, in this section, how to quantify and improve the representation stability of the model such that stays "approximately" RST R.ST-eqivvrriant even if the input transformation is "contaminated" by minute local distortion (see, for instance, Figure 2.)
4.1 Decomposition Convolutional Filters
In order quantify the deformation stability of representation equivariance, motivated by (Qiu et al., 2018; Cheng al., 2019; Zhu et al., 2019), we leverage the geometry of the group RST and decompose the convolutional filters will) ( a) under the separable product three orthogonal function bases, (th (v)( (pm(0)) and (L,(a)).. In particular, we choose {pm)m the Fourier basis on S" and set (wih and hh to be the eigenfunctions of the Dirichlet Laplacian over, respectively, the unit disk D C R2 and the interval la = [-1,]] Le.,
Adn. Mate in D, 24 0 AD,
c =- in la (m(+)
(8)
where Pa and un are the corresponding eigenvalues. Remark One has fleribilily in choosing the spalial function basis {vbkkk We consider mainly, in this work, the rotation-steerable Pourier-Bessed (FB) basis (Abrumowitz & Stegun, 1965) defined in (8), as the spatial regularity of its low-frequency modes leads to robust equivariance to input deformation in an RST- CNN, which will be shown in Theorem 1. One can also choose {dbkE to be the eigenfunctions of Dirichlet Laplacian over the cell [1,,1] (Zhu et al., 2019), i.e., the separable product of the solutions to the 1D Dirichlet Sturm Liouville problem, which leads to a similar stability analysis. We denote such basis the Sturm-Liouite (SL) basis, and its efficacy will be compared FB basis in Section 6.
Since spatial pooling can be modeled as rescaling the convolutional filters in space, assume the filters wi(() N are compactly supported on a rescaled domain as follows
w A E C.(29) D), N.A E C x Ia).
(9)
where ji ju+1 models sequence of filters with decreasing size. Let v3s((u) = 2-22 (b (2- be the rescaled spatial basis function, and we can decompose will) XA under tusik {pm)m. {En)n into
(x) = Σ (1) (1)),,)) W
(),
Σ 22 m, (u))) k,m.n
(10)
where a are the expansion coefficients of the filters. In practice, the filter expansion (10) are truncated only low-frequency components of the scparable basis, which reduces the computation and memory cost of the model. In addition, we will explain in Section 4.2 the effect of the truncated filter expansion on the stability of the RSST- -eq-evvvvrr representation to input deformation.
5

---

Published in Transactions on Machine Learning Research (07 (07/2022)
4.2 Stability under Input Deformation
First, in order to gauge the distance between different inputs and features, we define the layer-wise feature norm as
Mo (u,
1 M Σ
(0)
1
| sup M
(2. 0, a, dud
(11)
MD
for 1 12 1, i.e., the norm is combination of an Lo-no over the poto-translation group SE(() _ S x R² and an _oomnorm over the scaling group S = R. We point out the importance of the -nom in S, as signals after the coupled convolution (7) generally do not vanish as a + 000
We next define the spatial deformation of an input image. Given C² vector field T IR2 + R., the spatial deformation D, on 201) is defined as
D. (u, = (p)))
(12)
where p(u) = u - Tu)) Thus r(u) is understood as the local image distortion (at pixel location u), and D, is the identity map if (w)) = 0, i.e., not input distortion.
The deformation stability of the equivariant representation can be quantified in terms of (11) after we make the following three mild assumptions the model and the input distortion D,:
(A1): The nonlincarity a : R + R is non-expansive, i.e., (a(x) - (3)) S | y/. Vx, y E R. For instance, the rectified linear unit (ReI.U) satisfies this assumption.
(A2): The convolutional filters are bounded in the following sense: A S 1, 2 1, where
Mo sup Mi
A := ग. max
sup
la
|FB,
Σ
X
A = क max
sup
IIBB
2M
Σ sup
Σ l1 (x V> 1,
(13)
Mr
in which the FB-norm || . [IBB of a sequence {a(k)}kzo and double sequence (b(k, m)k.m20 is the weighted 12-norm defined as
P = [maa()) |BB|EB ΣΣ beb(k, m)?
(14)
Ps being the eigenvalues defined in (8). This technical assumption (A2) on the FB-norms of the expansion cocfficients, ||LE, ensures the boundedness of the convolutional filters (hth x (u, f, a) under varions norms (the exact details can be found in Lemma 2 of the appendix), which in turn quantifies the stability of the equivariant representation to input deformation. We point out that the boundedness of A can be facilitated by truncating the expansion coefficients to low-frequency components (i.e., small μu uı's), which is the key idea of our RST-CNN (cf. Remark 3).
(A3): The local input distortion is small:
VT == sup VT(u)]l < 1/5,
(15)
where || || is the operator norm.
Theorem below quantifies the deformation stability of the equivariant representation in an RST-CNN under the assumptions (A1)-(A3):
Theorem 1.
Let ( p0 0 < 1 S L,, be the RST group actions defined in (2)(5), and let D, be small
imput deformation define in (12). Tf an RST-CNI satisfies the assumptions (A1)-(A3) we have, for any
L,
1) - n.li (2000]] < 2811 (AL +2- ||
n,3,0 10
(16)
6

---

Published in Transactions on Machine Learning Research (07 (07/202) /202)
(0)
x []
[]
2
(L) x (0)
Dr
2
(
D(L) n, 20,
n,8,0
[1
1D n,8,0 Dxx
( 7,8,0
D,
Figure Approximate RST-equivariance the presence of nuisance input deformation Dr-
The proof of Theorem is deferred to the Appendix. An important message from Theorem is that as long (A1) satisfied, the model stays approximately RSTT-qquivriannn i.c., x 4,8.0 D,,x!0]
((L) D. (())(x((0))) even with the presence of non-zero (yet small) input deformation D, (see, e.g., Figure 2.)
Remark 2. The fact that the right hand side of (16) grows exponentially with 3 is inevitable, as it comes naturally from the definition of the norm (11): if an image is spatially rescaled by 2 without correspondingly scaling the color range (i.e., pixel intensity), its L2-norm enlarged by 2
Remark 3. According to the definition of the FB-norm (14), the main assumption (A2) can be facilitated by truncating the filter expansion (10) to include only low-frequency (small p&) components. The implementation detail of such truncated filter expansion will be explained in detail in Section 5.
5 Implementation Details
We next discuss the implementation details of the RST-CNN outlined in Proposition 1.
Discrotization. To implement RST-CNN in practice, we first need to discretize the features x()) modeled originally under the continuous setting. First, the input signal ((")(u, A) discretized on a uniform grid into a 3D array of shape [Mo, Ho, Wo), where Ho, Wo, Mo, respectively, are the height, width, and the number of the unstructured channels of the input (e.g., Mo = for RGB images.) For / 2 1, the rotation group S is uniformly discretized into N, points; the scaling group S = R', unlike S1 is unbounded, and thus features g(()) are computed and stored only on truncated scale interval T= (-T, T] C IR, which is uniformly discretized into N, points. The feature (0()) is therefore stored as 5D array of shape [M, Nr, N., Hi, Wi).
Filter expansion. The analysis in Section 4 suggests that robust RST-equivariance is achieved if the con- volutional filters are expanded with only the first K low- www-freqeeccy spatial modes { More specifically, the first K spatial basis functions as well as their rotated and rescaled versions {2 2a (2- Roou')) a. are sampled on uniform grid of size L L and stored as an array of size [K, N,, N., L, L], which is fixed during training. The expansion coefficients (I) ax(A: on the other hand, are the trainable parameters of the model, which are used together with the fixed basis to linearly expand the filters. The resulting filters (2 (- X. (2- "Roo"))) as tensors of size [Mo MM, Nr, N,, L, L) and [Mi-1, Mt, N,, La, N., Lo. L, LJ, where La is the number of grid points sampling the interval I, in (8) (this is typically smaller than N,, i.., the number of the grid points discretizing the scale interval -= [-T, and Lg is the number of grid points sampling S on which the integral [g d0 in (7) is performed.
and {2- (2-2< x ( (2-aR_ouk , a'))x..000000....
are stored, respectively,

---

Published in Transactions on Machine Learning Research (07 (022)
Remark 4. The namber La measures the support lo (8) of the convolutional filters in scale, which cor- responds to the amount of "inter-scale" information transfer when performing the convolution over scale Sa(..-)da' in (7). It is typically chosen to be a small number (e.g., 1 or 2) to avoid the "boundary leakage effect' (Worrall & Welling, 2019; Sosnovik et al., 2020; Zhu et al., 2019), as one needs to pad unknown values beyond the truncated scale channel [-T,T] during convolution (7) when La >. The number L,, on the other hand, corresponds the "inter-rotation" information transfer when performing the convolution over the rotation group Ss1 (---)dd'' in (7); it does not have to be small since periodic-padding of known volues is adopted when conducting integrals on S with "boundary leakoge effect'. We only require Lg to divide Nr such that fg (-.. )dd' is computed on a (potentially) courser grid (of size Lo) compared to the finer grid (of size N,) on which we discrelize the rolation chamnel of the feature x!1)
Discrete convolution. After generating, in the previous step, the discrete joint convolutional filters to- gether with their rotated and rescaled versions, the continuous convolutions in Proposition 1 can be efficiently implemented using regular 2D discrete convolutions.
More specifically, let xx!")(,, X) be an input image shape [Mo, Ho. Wo]- lotal of N, N,, discrete 2D convo- lutions with the rotated and rescaled filters {2 2" (2 (RR_ i.e., replacing the spatial integrals in (7) by summations, are conducted to obtain the first-layer feature x² (u, 0, a, A) of size [M N,, N₂, H,, Wi]. For the subsequent layers, given feature 2(t-1)(u, a. of shape NN,, N, Hi-1, Wi-1] and the joint filters Fr(1 = {2 2a WX X2 RR="R 0', a'))xxA.888
of size [M Mr, NN,, Le, N.. Lo L, LJ, the next- layer feature x²(1) is computed in the following way: for each la E [0, I., 1] and to [0, To 1], we shift the signal z!(-1) in the scale channel by la and in the rotation channel by Lo.N,/Lo, which is then convolved with the filter Fl.,:, l,, la,.i,aa.i lani, i (after proper reshaping and combining adjacent dimensions) to produce an output array of shape [M, Nr, Ns, H1, Wi). The 1-th layer feature map x. (0 (u, 0, a, A) is then computed as the sum of the Lo x La tensors obtained by iterating over lo E [0, Lg - 1] and la E [0, -1.
Group pooling. For learning tasks where the outputs are supposed to remain unchanged RST- transformed inputs, e.g., image classification, a max-pooling over the entire group RST is performed on the last-layer feature ((,)(u, 0, az A) of shape [ML, N,, No. HL, My] to produce an RST-invariant 1D output of length Mr.. We only perform the RST group-pooling in the last layer without explicit mention.
6 Numerical Experiments
We conduct numerical experiments, in this section, to demonstrate:
The proposed model indeed achieves robust RST-equivariana under realistic settings.
RST-CNN yields remarkable gains over prior arts in vision tasks with intrinsic RST-symmelry, especially in the small data regime.
Software implementation of the experiments is included in the supplementary materials.
6.1 Data Sets and Models
We conduct the experiments on the Rotated-and-Scalle MNIST (RS-MNIST), Rotated-and-Scaal Fashion- MNIST (RS-Fashion), SIM2MNIST (Esteves et al., 2017), as well as the STL-10 data sets (Coates et al., 2011b).
RS-MNIST and RS-Fashion constructed through randomly rotating (by an angle uniformly distributed on [0, 2x]) as well as rescaling (by uniformly random factor from [0.3, 1]) the original MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017a) images. The transformed images are zero-padded back to a size of 28 x 28. We upsize the image to 56 x 56 for better comparison of the models. In addition, we also consider the noise-introduced SIM2MNIST data set, obtained from randomly rotating (uniformly on [0, 2Tr),, rescaling (uniformly on [1, 2.4]), and translating the MNIST images before upsampling a spatial dimension of 96 x 96. The resulting SIM2MNIST data set is split into 10K, 5K, 50K samples for training, validation, and testing respectively.
8

---

Published in Transactions on Machine Learning Research (07/202) 2022)
20
19
K 5. L. 3
5, 1, 2
18
La
La
Equivariance error (%)
16
K 6, La
-1
Equivariance (56)
16-
14
12-
20
8
6
6
0
Depth
Depth
(a) Equivariance error, K = 5.
(b) Equivariance error, K = 10.
Figure 3: Layer-wise equivariance error (17) of the RST-CNN. Convolutional filters with varying number La of "inter-scale" channels are constructed from low-frequency spatial modes { The equivariance error is smaller when K docreases, verifying our thcoretical analysis (Theorem 1) that truncated basis expansion to low-frequency modes improves deformation robustness of the equivariant representation.
The STL-10 data set has 5,000 training and 8,000 testing RGB images of size 96 x 96 belonging to 10 different classes such as cat, deer, and dog. We use this data set to evaluate different models under both in-distribution (ID) and out-of-distribution (OOD) settings. More specifically, the training set remains unchanged, while the testing set is either unaltered for ID) testing, or randomly rotated (by an angle uniformly distributed on 1- 7/2) and rescaled (by a factor uniformly distributed on [0.8, 1J) for OOD testing.
We evaluate the performance of the proposed RSST-CNN against other models that equivariant to either roto-translation (SF(2)) or scale-translation (ST) of the inputs. The SE(())-quivvriian models consider in this section include the Rotation Decomposed Convolutional Filters network (RDCF) (Cheng et al., 2019) and the Rotation Equivariant Steerable Network (RESN) (Weiler et al., 2018b), which is shown to achieve best performance among all (F(()vequivarriat CNNs in (Weiler & Cesa, 2019). The ST-equivariant models include the Scale Equivariant Voctor Field Network (SEVF) (Marcos al., 2018), Scale Equivariant Steerable Network (SESN) (Sosnovik et al., 2020), and Scale Decomposed Convolutional Filters network (SDCF) (Zhu et al., 2019).
6.2 Equivariance Error
We first measure the RST-oquivariance error of proposed model with the presence of discretization and scale channel truncation. More specifically, we construct 5-layer RST-CNN with randomly initialized expansion coefficients (1) axi> truncated to K = or K = 10 low-frequency spatial (FB) modes (*h)) The scale channel truncated to [-1,1], which is uniformly discretized into N, = 9 points; the rotation group S' is sampled on uniform grid of size N, = 8. The RST equivariance error is computed on random RS-MNIST images, and measured in relative 1,2 sense at the scale = 0 and rotation 0 = 0, with the RRST-action corresponding to the group element (17 17.8,u) (- 005, 0), i.e.
(1x00) [0) - D De. (1) (0) 0)1
(17)
(1 [,: (2(0))
(., a,
We fix Lg, ie., the number of the "inter-rotation" channels corresponding to the "coarser" grid of S for discrete 51 integration, to Lag = 4, and examine the equivariance error induced by the "boundary leakage effect" with different numbers L., of the "inter-scale" channels (ef. Remark 4].
Figure 3 displays the equivariance error (17) of the RST-CNN different layers I (1,-.. 5) with varying Ia E 3). It can be observed that the equivariance error is inevitable due numerical ixation and

---

Published in Transactions on Machine Learning Research (07/2022)
RS-MNIST test accuracy (%)
RS-MNIST+ test accuracy (%)
Models
Nur 2, 000
Ntr 5, 000
Nur 2, 000
Ntr 5, 000
CNN
80.63 + 0.11
87.41 + 0.32
89.52 0.21
93.08 0.04
RESN
85.71 + 0.18
89.69 + 0.40
90.86 + 0.20
93.78 I 0.35
RESN+
87.96 + 0.05
92.29 + 0.10
92.20 + 0.09
95.32 + 0.09
RDCF
86.79 + 0.12
90.46 + 0.33
90.40 + 0.31
94.84 0.15
RDCF+
87.68 = 0.52
91.74 = 0.37
92.49 = 0.28
95.81 0.11
SEVF
85.76 + 0.38
90.29 + 0.37
89.97 + 0.42
93.47 0.18
SESN
84.58 + 0.29
90.19 = 0.39
90.33 + 0.30
9 0.29
SDCF
85.62 + 0.51
90.40 + 0.09
90.14 + 0.16
93.47 + 0.05
RST-CNN(FB)
89.16 + 0.32
93.19 + 0.29
92.58 + 0.35
96.33 t 0.26
RST-CNN(SL)
88.97 + 0.17
93.03 + 0.20
92.30 + 0.19
96.04 + 0.11
RST-CNN+
89.53 + 0.27
93.40 + 0.26
93.99 + 0.07
96.53 + 0.25
RST-CNN+(SL)
90.26 + 0.37
93.59 0.06
93.82 0.25
96.76 0.11
Table 1: Classification accuracy on the RS-MNIST data set. Models are trained on Ntr = 2K or 5K images with spatial resolution 56 56. A plus sign "+" on the data, i.c., RS-MNIST+, is used to denote the presence of data augmentation during training. A plus sign "+" on the model, e.g., RDCF+, denotes a larger network with more "inter-rotation correlation Lg |cf. Section 5). The mcan + std of the test accuracy over five independent trials reported.
truncation as the model goes deeper. However, it can be mitigated by choosing a small Lg, i.c., less "inter- scale" information transfer, to avoid the "boundary leakage effect", or expanding the filters with a small number K of low-frequency spatial components (1/x) supporting our theoretical analysis Theorem 1. Due to this finding, will consider in the following experiments RST-CNNs with L,- 1, which has the additional benefit of better model scalability due to the reduced scale channel convolution.
6.3 Image Classification
We next demonstrate the superior performance of the proposed RST-CNN in image classification under settings where large variation of rotation and scale is present in the test and/or the training data.
6.3.1 RS-MNIST, RS-Fashion and SIM2MNIST
We first benchmark the performance of different models on the RS-MNIST, and RS-Fashion data sets. We generate five independent realizations of the rotated and rescaled data [cf. Secion 6.1], which are split into N,, 5,000 or 2,000 images for training, 2,000 images for validation, and 50,000 images for testing.
For fair comparison among different models, use a benchmark CNN with three convolutional and two fully-connected layers as a baseline. Each hidden layer (i.e., the three convolutional and the first fully- connected layer) is followed by batch--orrmalization, and we set the number of output channels of the hidden layers to [32, 63, 95, 256]. The size the convolutional filters is set to 7 x for each layer in the CNN. All comparing models (those Table 1-4 without "+" after the name of the model) are built on the same CNN baseline, and we keep the trainable parameters almost the same (~500K) by modifying the number of the unstructured channels. For models that equivariant rotation (including RESN, RDCF, and RST-CNN), we set the number N, of rotation channels to N, [cf. Section 5]; for scale-equivariant models (including SESN, SDCF, and RST-CNN), the number N, of scale channels is set to N, 4. In addition, for rolation-equivarian CNNs, we also construct larger models (with the number of trainable parameters~11.MM after increasing the "inter-rotation" information transfer [cf. Section 5] by setting Lg = 4; we attach a "+" symbol to the end of the model name (c.g., RDCF+) to denote such larger models with more "inter-rotation.. A group max-pooling is performed only after the final convolutional layer achieve group-invariant representations for classification. Moreover, for RST-CNN, consider two different spatial
10

---

Published in Transactions on Machine Learning Research (07/2022)
RS-Fashion test accuracy (%)
RS--asshio++ test accuracy (%)
Models
Ntr 2, 000
Nir 5, 000
Nir 2, 000
Ntr 5, 000
CNN
62.71 + 0.37
67.91 + 0.28
67.92 + 0.12
72.41 0.46
RESN
70.80 + 0.41
75.80 + 0.11
74.10 I 0.46
77.76 + 0.16
RESN+
71.59 + 0.71
76.32 0.26
76.80 + 0.55
80.89 + 0.41
RDCF
70.72
73.96 0.19
73.46 + 0.10
77.53 + 0.11
RDCF+
71.27 0.34
16.94 0.35
76.87 0.57
80.66 + 0.48
SEVF
66.57 + 0.32
71.03 + 0.31
68.83 + 0.52
73.25 + 0.22
SESN
66.28 + 0.14
72.19 1 0.05
69.43 + 0.07
75.85 + 0.26
SDCF
66.29 + 0.23
72.24 + 0.23
68.40 + 0.05
75.11 + 0.18
RST-CNN(FB)
73.31 + 0.16
78.64 + 0.60
76.43 + 0.59
81.93 + 0.04
RST-CNN(SL)
72.90 + 0.34
78.37 + 0.22
76.06 + 0.13
80.81 + 0.29
RST-CNN+(FB)
74.37 + 0.08
79.19 + 0.36
80.65 + 0.31
84.37 + 0.19
RST-CNN+(SL)
74.68 + 0.29
79.81 + 0.06
80.65 + 0.46
84.09 0.09
Table 2: Classification accuracy on the RS-Fashion data set. Models are trained on Nrr = 2K or 5K images with spatial resolution 56 56. A plus sign "+" on the data, i.e. RS-Fashion+, is used to denote the presence of data augmentation during training. A plus sign "4" on the model, e.g., RDCF+, denotes a larger network with more "inter-rotation correlation Lg (cf. Section 5). The mcan + std of the test accuracy over five independent trials are reported.
Models
CNN
RDCF
RESN
SEVF
SESN
SDCF
RST-CNN
RST-CNN+ 94.81 + 0.12
Accuracy (%) 86.6 + D.14
89.3 + 0.37
89.2 + 0.36
87.31 # 0.05
87.86
0.31
88.1 1 0.62
93.34 = 0.13
Table 3: Classification accuracy on the SIM2MNIST data set. The mean + std of the test accuracy over three independent trials are reported.
function basis for filter expansion, namely the Fourier-Bessel (FB) basis, and Sturn-Liouville (SL) basis jef. Remark 1].
We use the Adam optimizer (Kingma & Ba, 2014) to train all models for 60 epochs with the batch size set to 128. We set the initial learning rate 0.01, which is scheduled decrease tenfold after 30 epochs. We conduct the experiments in 4 different settings, where the number Nrr of training samples either 2,000 or 5,000, and the models are trained with or without RST data augmentation.
We report the mean + std of the test accuracy after five independent trials in Table and Table 2, where, for example, RS-MNIST (or RS-MNIST+) denotes models are trained on the RS-MNIST data set without (or with) data augmentation. It is clear from Table and Table that RST-CNN has superior generalization capability compared to other models with approximately the same trainable parameters, especially in the small data regime. Furthermore, RST-CNN+ with more inter-rotation correlation (i.e., I.p = 4) has further improved performance, achicving the best accuracy among all methods. can also be observed that neither of the two spatial basis functions (i.e., FB and SL) has significant advantage over the other. The preferred choice of the basis for filter expansion could depend experimental settings including sample size, the number of filters, original data characteristies and the presence (or lack thereof) of data augmentation.
Similarly, adopting the same hyper-parameler setting as the RS-MNIST experiment, we report the test accuracy after three independent trails on the SIM2MNIST data set (Esteves et al., 2017) in Table One can observe that RST-CNN once again outperforms other models with approximately the same number of parameters. In addition, RST-CNN- with notariotation correlation (Lg 4) further improves RST-CNN, achieving the best performance among all comparing methods.
11

---

Published in Transactions on Machine Learning Research (07/2022)
Models
ID Accuracy (%)
OOD Accuracy (%)
ResNet-16
82.66 + 0.53
37.63 + 1.95
RESN
83.84 + 0.67
51.28 + 2.29
RDCF
83.66 + 0.57
51.12 + 4.21
SESN
83.79 + 0.24
47.26 + 0.63
SDCF
83.83 + 0.41
43.60 + 0.87
RST-ResNet
84.08 + 0.11
58.31 + 3.62
Table 4: Test accuracy the STL-10 data set for both in-distribution (ID) and out-of-distribution (OOD) settings.
6.3.2 STL-10
Finally, we use the STL-10 data set as an example to showcase both ID and OOD generalization capacity of the proposed RSS-ResNet. As explained in Section 6.1, the training set remains unaltered, while the testing set is either (a) unchanged for ID) testing, or (b) randomly rolated and rescaled for OOD testing. In order for all models to be trained on single GPU, we choose ResNet (He et al., 2016) with 16 layers as the baseline this is different from previous works such as (Sosnovik et al., 2020) where WideResNet baseline (Zagoruyko Komodakis, 2016) adopted- -aapted-adddddd -and we keep the number of trainable parameters almost the same for comparing networks. group max-pooling performed after the final residual block to achieve invariant representations.
Similar to the idea of (Sosnovik et al, 2020; Zhu et al., 2019), the data set is augmented (without rotation and rescaling) during training by randomly cropping a 12 pixel zero-padded image. Furthermore, random horizontal flipping and Cutout (DeVries & Taylor, 2017) with 32 pixels are applied to the cropped image. We train all models for 1000 epochs with a batch size of 64, using an SGD optimizer with Nesterov momentum set to 0.9 and weight decay set to x 10-4, Learning rate starts at 0.1, and is scheduled to decrease tenfold after 300, 400, 600, and 800 epochs.
The mean + std after three independent trials of both ID) and OOD test accuracy displayed in Table 4. It can be observed that the proposed RST-ResNet significantly outperforms all comparing models, especially for OOD gencralization, demonstrating further its advantage in computer vision where both rotation and scale transformations are intrinsic symmetries of the learning task.
7 Conclusion
In this paper, have proposed the roto-scale-translation cquivariant CNN (RST-CNN), which is able to achieve equivariance jointly over these three groups. Through truncated expansion of the joint convolutional filters under pre-fixed low-frequency spatial modes, which is motivated by rigorous stability analysis of the representation, the proposed model provably attains deformation robust equivariance, i.e., the features stay "approximately" equivariant even if the RST transformation is "contaminated" by muisance input distortion, a property that crucial for out-of-distribution model generalization. Experiments on vision tasks with intrinsic RST symmetry are conducted to demonstrate the improved generalization capability of our proposed model under both in-distribution and out-of-distribution setting, especially in the small data regime.
One limitation of the current work is that we have considered deformation robust neural networks that are equivariant to only the regular representation of the group RST. Such models have empirically proved to exhibit stronger generalization performance because of their ability to encode any function on the group. However, regular representation requires high dimensional feature spaces, and the memory consumption (mainly from storing intermediate features) of the proposed RST-CNN N, N, times that of a regular CNN with the same number of unstructured channels. For future work, will extend the idea in this
12

---

Published in Transactions on Machine Learning Research (07/2022)
paper to construct deformation robust steerable RST-CNNs with reduced model size and efficient network implementation.
References
Milton Abramowitz and Irene Stegun. Handbook of mathematical functions: with formulas, graphs, and mathematical tables, volume 55. Courier Corporation, 1965.
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:// pdff. 2, 3.3
proceediins..euuipp.. http:/app...pppppa//////////////
Erik J Bekkers. B-spline cmns on lie groups. In International Conference on Learning Represenlations, 2020.
URL https://openreviee http:/ppenneeeee.eeefforrrrrrr
3
Alberto Bietti and Julien Mairal. Invariance and stability of deep convolutional representations. In NIPS 2017-31st Conference on Advances in Neural Information Processing Systems, pp. 1622 1632, 2017. 3
Alberto Bietti and Julien Mairal. Group invariance, stability to deformations, and complexity of deep convolutional representations. The Journal of Machine Learning Research, 20(1):876 924, 2019.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. Xixiv preprint or 1iw::60440075 2016.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions pattern analysis and machine intelligence, 35(8):1872-1886 2013.
Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVP Conference on Computer Vision and Patlern Recognition, pp. 14514 14523, 2021. 2
Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. RotDCF: Decomposition of convo- lutional filters for rotation-equiiariant deep networks. In International Conference on Learning Represen- tations, 2019. 2, 3, 5, 9, 21
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 201 la. 25
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011b. 8
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pp. 1990-29990 2016. 2
Taco Cohen and Max Welling. Steerable CNNs. In International Conference on Learning Representations, 2017. 2, 3
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In International Conference on Learning Representations, 2018.
Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogencous spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alchh-BB E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 2, 3,
13

---

Published in Transactions on Machine Learning Research (07/2022)
Michaël Defferrard, Martino Milani, Frédérick Gusset, and Nathanaël Perraudin. Deepsphere: graph-
based spherical enn. In International Conference on Learning Represenlations, 2020. URL https://
openreview. http://w..ee.orrmniace.eeeee111.
2
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1 04552, 2017. 12
Carlos Esteves, Christine Allen- Blanchela,,,,,t,, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. arXiv preprint ar 109. 7tw.11999...,8,,, 2017. 3, 8, 11
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance lie groups on arbitrary continuous data. In International Conference on Machine Learning, pp. 3165 3176. PMLR, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 12
Emiel Hoogeboom, Jorn W.T. Peters, Taco S. Cohen, and Max Welling. Hexaconv. In International Con- ference on Learning Representations, 2018.
Angjoo Kanazawa, Abhishek Sharma, and David Jacobs. Locally scale-invariant convolutional neural net- works. Xi preprint
2014. 2
Nicolas Keriven and Gabriel Peyré. Universal invariant and equivariant graph neural networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Ine., 2019. URL https: / //proceedings hewrrpps.ccppppe//211333 /ttp://///////////55555550110000
Paper. pappr.ppf... 2,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arviv preprint
arXiv:1412.6600 6980, 2014. 11
Risi Kondor. N-body networks: a covariant hicrarchical neural network architecture for learning atomic potentials. aiXiv preprint orXiv: 01588, 2018. 2,
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch gordan nets: a fully fourier space spherical con- volutional neural network. Advances in Neural Information Processing Systems, 31:10117-10126, 2018. 2
Alex Krizhevsky, llya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In Advonces in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. hon/exdb/mmiise 1998. 25
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. 8
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
Julien Mairal. End-to-end kernel learning with supervised convolutional kernel networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Process- ing Systems, volume 29. Curran Associates, Inc., 2016. 3
Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia Schmid. Convolutional kernel networks. Tn Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Stéphane Mallat. Recursive interferometric representation. In Proc. of EUSICO conference, Danemark. 2010.
14

---

Published in Transactions on Machine Learning Research (07/2022)
Stéphane Mallat. Group invariant scattering. Communications Pure and Applied Mathematics, 65
1331-1398, 2012. 3
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 5048-5057, 2017. 2
Diego Marcos, Benjamin Kellenberger, Sylvain Lobry, and Devis Tuia. Scale equivariance in CNNs with vector fields. Xiv preprint arXiv:1807.1 11783, 2018. 2,
Edouard Oyallon and Stéphane Mallat. Deep rolo-translation scabtering for object classification. In Pro- ceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 2865 2873, 2015. 3
Qiang Qiu, Xiuyuan Cheng, Robert Calderbank, and Guillermo Sapiro. DCFNet: Deep neural network with docomposed convolutional filters. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4198 4207, Stockholmsmüssan, Stockholm Sweden, 10 15 Jul 2018. PMLR. 3,
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Newral Information Processing Systems 28, pp. 91 99. Curran Associates, Inc., 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer- Intervention - MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319-2457444.
Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2233-12401 2013. 3
Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scalo-cquivariant stecrable networks. In International Conference on Learning Representations, 2020. 2, 8, 9, 12, 25
Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. Disco: accurate discrete scale convolutions. ar preprint arXi::2106.0077 02733,, 2021. 2
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-oquivvriant neural networks for 3d point clouds. arXie preprint arXiv:1802.0825 2018. 2
Maurice Weiler and Gabriele Cesa. General equ)eequiaariant steerable enns. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alech--Bue, E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 2, 3, 4,
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learn- ing rotationally equivariant features in volumetrie data. In Advances in Neural Information Processing Systems, pp. 10381-10392, 2018a. 2,
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8499858, 2018b. 9
Daniel Worrall and Gabriel Brostow, Cubenet: Equivariance 3D rotation and translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 567 584, 2018. 2
Daniel Worrall and Max Welling. Deep scalo-spaces: Equivariance over scale. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. 2,
15

---

Published in Transactions on Machine Learning Research (07 (222)
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetor, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Compuler Vision and Pattern Recognition, pp. 5028 5037, 2017. 2
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist novel image dataset for benchmarking machine learning algorithms, 2017a. 8
Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:100..0774., 2017b. 25
Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang. Scale-invariant convolutional neural networks. arxic preprint 1xXiv1411.6366 2014. 2
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1xXtv:1005.07144 2016.
12
Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Federico Tombari. Quaternion equivariant capsule networks for 3d point clouds. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 1 19, Cham, 2020. Springer International Publishing, ISBN 978-3-030-58852-8. 2
Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. Proceedings of the IEEE Conference on Compuler Vision and Pattern Recognition, pp. 519 528, 2017. 2
Wei Zhu, Qiang Qiu, Robert Calderbank, Guillermo Sapiro, and Xiuyuan Cheng. Scale-equivariant neural networks with decomposed convolutional filters. orXiv preprint Xix:11999 11198, 2019. 2, 3, 5, 8, 9, 12, 20, 22, 24, 25
A Appendix
B Proof of Proposition
We first recall for an input RGB image 2.10) (u, \), roto-ssaleetrranslation transformation on this image can be understood as an RST-action (representation) D.00 = n33.v on r,())
(0))
[D 1 (0)) (u, A) = (0) (v-)),)
(18)
For the hidden layers 1 1S1. S I, the action Th n,8.v x (u, 0, ar, X) is defined as
the space 21) consisting of features in the form of
(D
n.9,0 (u, 0 a. (0) (0- 17, (
-3 3, vi 1.
(19)
We restate below Proposition of the paper that outlines the general network architecture to achieve RST- oquivariance under the representations D 0 <I<L. L.
Proposition 1. An I--aager feedforward neural network is RST -qquivarann under the representations (18) (19) if and only if the layer-wise operations are defined as (20) and (21):
a (, 0, ar A)
Σ
x (0) (u + x' 20
X. (2- R (2-nR_uu)) du' + (0)())
(20)
ga
[W (, = σ
x (1--1) u 0 8 8' ad 51
2 2- 20 (((
N.A Rou 0 a.)d) do' dol du' + (0)
(21)
where I.-- - R. is pointwise nonlincarity, W (u) is the spatial convolutional filter in the first layer with oulput. channel X and input channel X, (( (u, (u,00 a) is the RST joint convolutional filter for lager > 1, and Sgr f(a)dd denoles the normalized St integral 2n2 f²r f(a)da.
16

---

Published in Transactions on Machine Learning Research (07 (07/2022)
B.1 Sufficient Part of the Proof of Proposition 1
Proof. We first prove the sufficient part of Proposition 1. That is, given the layer-wise definition of the L-layer feodforward neural network (20) and (21), RST RST-equivariance under the regular representation (18) (19) can be achieved, i.e.,
11) [1 (1-1)
11 =D
"ii. 20"0 [2(11 14 2 1.
(22)
n.3.0
Indeed, for the first layer, i.e,, 1 1, we wrile the LHS of (22) as
x0i D n. ((4,,,)) =o Σ JR2
2-2 (1))
Mill n (0) (u XIW 22220aa RRou + +6(l(a))
Σ Σ/ (0) ( (u u' - -v) X) W R R2 (2)(N)) ;
(23)
122
The RHS of (22) when is
((1)
1.3,00000 (2.00) 0,, A) 22'(211"11 (R_ $ u), 7,, a 3,
Σ
2 2(a x (R_,2- (u - v) + ū, X) (1)) (2 (a Ru) dd + ((0
B2
0
Σ
2- 222222 x (0) RR 8 (u - u) + 22 R_p X)
x1,1 W (2 R Rou')) 23 du + +Bo)(a))
(24)
Σ
2 2n (0) (R_, (u ( ul - 1 W (2 "Rou')) + 60((N)
(25)
R2
where (24) uses the change of variable u' = 2- Thus (23) and (25) implics that (22) holds valid for 1=1. 1.
For / >> 1, with the definition of the (-th layer operation (21), the LHS of (22) becomes
(1) (1l1 (1 (,, a,
m.B."
Σ
2 2a D (1- (1-1)
"R""Rou', 80,'0'a)) a.) do d8'd + b10) (A)
n.B.v
(26)
Σ E/1l /
20 (1-1) (1-1) (111)11)10000 (R (u + - 0 17, + a' -B,2)
(1(() X.A (2- R Rord', H' a' do' do'dal + B09) (2)) (A))
(27)
17

---

Published in Transactions on Machine Learning Research (07 (07/2022) /022)
On the other hand, the RHS of (22) is
(
n. 00 (20l))((1)]] (u, 0 a, 2)
=x (2l1-11) 111 (R_ 2 (u (u r), 0 17,a a- --- -.A)
(28)
Σ 2 2(a (1-1) nR_ 2- 8 - -00+)000+++00 B x
(a Ry Uū 0 a') da' d0' du dū w 2'2
b
6()
(29)
Σ/ll>
= Σ
2-22222 (t- R (u v) + R 2 u 17 -0 -- a
2 20228
X. 22-aRooo 0 a') da' d8 du dā
+
(2)
(30)
where the last equation again uses the same change of variable. Equation (27) combined with (30) implies
that (22) holds true for all 1
B.2 Necessary Part of the Proof of Proposition 1
Proof. We first note that a general leedforward neural network propagating through the feature spaces x(0
has the following form: when / ==1,
x 100 a, 2)= =0 (E/
(E) N R2
/ 100) (u wd)) a, 0, a, A)du' + +b)(2))
(31)
and, for >>
a, a, = <
V R2 1
(', (11 1 a 0,0++,0++d,,
JR
W00 (u', 0' a' N, u, 0.0, 0, a. A)d d0l dul + (0)
(32)
To prove the necessary part of Proposition 1, we want to verify that RST RST-equivariance (22) implies that the weight matrices W()) in (31) and (32) take the special convolutional form in (20) and (21).
Indeed, when I 1, the RHS of (22) under the operation (31)
D11) 1,3,0 00 [2
a,
(33)
(1) (R_, 2- 0 (u
v),0 - a,,
(34)
X, (0) R n 2 (u - v) (Z, (u) n.a
B, A,A + ((0()))
(35)
2 28 m (0) R 2 8 /ut w x),
((1) RR,22 2- pnl, X, v) 7, a B, dul + (0))))
(36)
18

---

Published in Transactions on Machine Learning Research (07/2022)
with change of variable similar to the sufficient part. For the RHS, we have
z: 1D 1,.8
(u,
(37)
113.0 (u+ (u N,,,,,,,,,,, Ajdn' (
(38)
w) ()) x (0) (R (u (
A)du' (()
(39)
Hence, for (22) hold when / 1, we need
( X, u. a, 2- -
(. 100,0--7--220 B)
(40)
For all u, a, U, that (w X, 0 a, does not depend the third variable u. Hence (w ,,,,0, X, Vu R Further define A (w²) as
u, n.
Keeping u,O, a, X, , n, fixed while varying (40), we deduce
(41)
Therefore, for any fixed u', X,, u, 0, at A, setting = a, 1) 0 in (40) yields
( X, u. a, A) = = (R u X, 0,0,0,,,,)2
R_ v), 0 0, a, A)2
(42)
W 02 2 "2')222 2a
(43)
Hence (31) has the special form (20).
For the subsequent layers / > 1, similar argument yields (w ,'' X, u, a, A) (R
R (. - v). 1000-10000 17 - -3,1222-22
(44)
for all u, a, A, v', 0', a' X, U,n,B B. Similarly, we can keep u, 0,000 a, X, u' 0' a', X, N,,, fixed while varying in (44), which implies that W 0',00.0 a', N,, does not depend on the fifth variable Again, let define
Wh (''''''''))) a²) w (u', N',,0,,,
(45)
For any given d',0',a' a', ''' X, u, 0 az A, setting 3 a. 1) 0 leads to
( 0¹ a', u, a. = (R 0¹ a' X, R_ a u), 0, 0, 0,0, -W
a' X, 0, 2)2-2a -w -W uu' uu' - 0 0,'' aa,
(46)
which implies that (32) can be written in the form of (21). This concludes the proof of Proposition 1.
C Proof of Theorem 1
We prove, in this section, the deformation stability of the RST-CNN (Theorem of the paper) under the following three assumptions:
(A1): The pisntwise noulinearity - is non-expansive, i.e., la(x) - a(y) S l vl. WR
(A2): The convolutional filters are bounded in the following sense: A S 1, 2 1, where
Mo sup
A max
sup
WAllFBB IIBB,
Σ l1 }
(47)
Mi
x
A max
sup
|a
(1)))))FBBBB. 2M
Σ
sup
-l (1)))l)) VI 1,
(48)
Mr
X
19

---

Published in Transactions on Machine Learning Research (07 (07/2022)
in which the FB-norm || I|e of sequence {a(k))<zo and double sequence {b(k, h) is the weighted 12-norm defined as
|alleB Euaa(k)) |00|
ΣΣ psb(k, m)?
(49)
with Ms being the eigenvalues of the Dirichlet Laplacian a unit disk.
(A3): The input distortion is small. More specifically, let
D,I (u, A) 50 (p(u), A); and D. (,, a. (p(u)))) where p(u) T()), and 2 ?Ra is C² local (spatial) distortion. We assume
O,, 12 1
(50)
V == sup VT(ui)] < 1/5,
(51)
with || being the operator norm.
We repeat below Theorem of the paper that quantifies the deformation stability of the equivariant repre- sentation in an RST-CNN under the assumptions (A1)-(A3):
( Theorem 2. Let 9,8,0r 0 < l < L, be the RST group actions defined in (18)(19), and let D, be small imput deformation define in (50). If on RST-CNE satisfies the assumptions (A1) we have, for any L,
2 (0 n1 D,
12000) s 200 (ALIVT) +2- (T))) ||
(52)
where the layer-wise feature norm is defined
Ma
1/=00//==
(x 2)
Mu
A=
1
R2
(53)
M
|2 sup
1
(u, 0, a, 2)d
M
S1X
The proof of Theorem follows the similar steps in (Zhu et al., 2019). More specifically, we aim establish the following two propositions that show the layer-wise non-expansiveness of the model (Proposition 2) and quantify the perturbation of equivariance with the presence of layer-wise spatial deformation D, (Proposi- tion 3).
Proposition 2. Under (A1) and (A2), an RST -NN satisfies:
(a) For any 1 12 1, the 1-th layer mapping z [ defined in (21) is non-expansive, i.e.,
[x)] a: [x2]]] < || - æ2l1,
V21,
(54)
(b) Let (1) (u. (v, a. A) be the 1-th layer output given depends only on X, ie., x0
input (u, 0, then = (u, 0 a.
( (v, 0 a. A) = (1) (A).
(c) Let (1) the centered version of ( after subtracting (1))) ie.,
(u, (1, x (2) (0 (u, A), x!!!! (u,
0, a, a)
(1) (A), 12 1,
(55)
then || 12 1 11 1.2. 1. As result, ||= < || (0) ||z(00]||
Proposition 3. In RST-CNN satisfying (A1) to (A3), the following statements hold true.
20

---

Published in Transactions on Machine Learning Research (07 (07/2022) /022)
(a) Given any 2
(D,""111111)) DD (x000(((--))) < (
(56)
where Te
(1-1) is defined in (55)
(b) Given any 1 2 1, we have
moo x 2011 -2 x.10)
(57)
and
1 1B, o D= (10)
9:3.0 D (,00")(0011111
(1--1
(58)
(c) For any 1 12 1,
(D(( n. o D,x.1-11]
(10)
n.8,0
1=200
<
20
|
(59)
(d) For any 121,
((1) -0(( ( n.D,o P. D,,
S 2821-111
11 b [-(-1))
12:00/1
(60)
C.1 Proof of Proposition
Before proving Proposition 2, we present the following two lemmas that are crucial to bound various norms of the convolutional filters using their Fourier-Bessle (FB) norm
Lemma 1. Let fu be the Fourier-Besse basis on the wnit disk D C R2, and let {smm Jm be the Fourier basis on the unit circle St. Assume that
F(u) Σ a(())bb(u), G(u,
ΣΣ Mk, m) ())m
(61)
are functions in (2' D) and H2 D) x T??(St), respeclively. Then
1/2
I [F(W))du, / |ullVF(u)ddd, VF(u)ldu S IT|a||EB== = ( " _uaa(k))
(62)
1/2
/f G(u, Ojdddd, If VullV
/f V..G())ddud S <T|[b||
Σ
Hubble, m)²
k.m
(63)
The proof of Lemma 1 can be found in Proposition A.1 of (Cheng et al., 2019). direct application of Lemma 1 leads to the following lemma.
Lemma 2. Let ( (k, m, be the coefficients of the filter w x (u, O a) (supported on 2 D x St x (-),1])
under the separable bases (w)))(u))Rr (0)m and (En(a)), defined in the main paper, and define (i(t) Mu, f) as
( (u, (1,00 ΣΣ (k)
(64)
We have
(1) (1) (2)
NA N.A: XA 2 Bl Cll) Ao 2 D v.An S
V 1,
(65)
21

---

Published in Transactions on Machine Learning Research (07 (07/2022)
where
BLA / (xx(())) du,
B = / JR²
W N, X. (u. 0) dud0, 1> 1,
CH / |u]| V. 2/12 (12(1)) du,
c(t)
Midin 1s1 R22
/ /
|u V will (u, 0) dud0, 1> 1,
(66)
NA = / Vu du.
D
D,nn"" = /
V. 2. (u. 0) dud0,
1> 1.
J22
Hence have
Bi, Ci, C1, Di S An,
(67)
where
B1 = max
sup
Mo
(2)
P''
M
MU
sup
Mi
B
X
A=1
C max
sup
(I)
X,A*
Mo
M1 X
sup
(() X1
(68)
x=1
Mo
Mn
Mi
D1 = max
sup
D (1)
X
M
sup
D 21
A=1
and, for 1 > 1,
Mc
Mi
B = max
sup
X Am
2M
Mi
Σ
Bin
BI, = sup
Bi0
Mnns
X 13
A=1
G = max
sup Σ
Mi Cill) Adn
2M1-1
Σ
Cim
Oi = sup
Mi
(t)
(69)
N.A.
X=1
A=1
M
Mi
Dj max
sup
2M "''EDa
Din
Di, sup X
D (1)
Mi
x=1
The bound v, v2 1, i..., (A2), therefore implies that.
B, 2 D.
V221.
(70)
Proof of Lemma 2. Applying Lemma to the filters ( (1) and (0 V.A. (u, 0) defined in (64) after rescaling the spatial variable u casily leads to the desired bounds (67). The rest of the lemma follows from the assumption (A2).
Proof of Proposition 2. To avoid cumbersome notations, we drop the layer index (l) in the filters w(th and 80), and let M = Mr, .M' = Mr-1 when the context is clear. The proof of (a) is similar to that of Proposition 2(a) in (Zhu et al., 2019) after a further integration on S More specifically, when = 1, the definition Bi in (66) implies that
X
sup Σ B < Bi, A
sup
Σ
(L) A <
B
M M
(71)
A
22

---

Published in Transactions on Machine Learning Research (07/ (07/2022)
Therefore, given two inputs 21 and T2, we have
[x1] - [xal)))) (u, a,
(72)
o Σ 11 + A "2 2 dul + b(A)
x2(u+ vl, d(,') (2-R__u') 2 20 dul + b(1) )
Σ
(73)
21(u+ (w + N'WW, (2 R_ 2 dud' -E
u', (2-0R 2 20 de!
(74)
Σ / - (o (2 (o 2- 2-20 du'
(75)
/ [(- - (2n-
Σ /
(2 22 dui
2 2-2adu
(76)
x
Σl / 101- - x2) (2 (2 2-mn) du'
E DDSt
(77)
< B1 Σ/ / (00 - (2, Wx 1 R (ū - =))l 2-1)/222000l 2a di
(78)
x
Hence, given any a E I, we have
Σ/ /
( = (u, a, 0, 0.2) dud0
x R2
Σ E1 8t R22
/
B1 Σ / =2) ( ( u))l 2 20 düdudo
(79)
=B, Σ
/
(00 22)
Σ S1 JR2
/
Wx a RRo (ū --))) 2 dud0 dā
(80)
M2
B1 Σ / (021 (o,
Σ B dā
(81)
B2 M
Σ/ / (0) x22 221 du
(82)
M
BR M 1 - 22212
(83)
<MIaa - =2ll"
(84)
where the last incquality comes from Lemma 2, and (80) makes use of the fact that
| [ "Rou)] 2 2udu [ [W())ldu, Va R. Y0
(85)
and Jgi d0- due to the normalization factor 1/2x in the definition. Therefore, we have
1 1 Σ Σ/ (dW(ai) (mal)))) (u, 0, a, 2)|" dudd 5 |||| - x2ll?. [1=0) - -00) sup M
(86)
This concludes the proof of (a) for the case 1 = 1. For the case 1 1> 1, the same technique applies by considering the joint convolution Jg [gz(... )dudd while making of (85), and omit the detail.
23

---

Published in Transactions on Machine Learning Research (07 (07/2022)
For part (b), we use mathematical induction. More specifically, 20 20 (1)
(0) (u, A) 0 by definition. For 1 1,
(u, (u,0,a, a, A) (A)). Assuming that z0
(A) for some I > 1, we have 1011)) (w, 0, a, A) - = (A)
11) [u, a, A)
20
(1-1)
(u + a0. ,' (2 (2- oR a.)) 2') 2-2uda da' du' du'd'' + +(0(())
(87)
X 51
RR JR.
Σ 20-1
(11) (2)
/
(1 N.A (2 R R 0', a²) 2- da' du' do' + 60(1))
(88)
89 22 R
x
Σ (x)
/ f1 /
W() N (u', 0 a') da' do' + (00
(89)
51 22
x
(1) (A)-
(90)
To prove part (c): for any l >>1 1, we have
| = (|x(9 10 1,-00 =000 [ - S ||a((111) 20 2 |>
(91)
where the inequality comes from the layer-wise non-expansiveness (54) in part (a). An easy induction leads to (c).
C.2 Proof of Proposition 3
Proof. Just like Proposition 2(a), the proofs of part (a) and part (d), respectively, of Proposition 3 are similar
to those of Proposition 3(a) and Proposition 4 of (Zhu et al., 2019). More specifically, making use of (85)
and Sg. d0- when = 1, and further taking the integral Ss (re( )dudf over 5 R22 instead of just TR22 when 1, we arrive the following
1D (,00")00- S ((BI+CI)IVVi | S 4(BB+ CDIVT|> 2000)
2 1, (92)
| - (()))
niblu
MBB.u 20/
S 2 2 |To Di ||! < 2 11 Dr
10
AA 1, (93)
where Bi, Ct, Dj are defined in (68) and (69). Lemma 2 thus implics that
| (D, D 11, [zs s oo
- s 8/ 100
[000
M21, 1,
(94)
D n, Dr - ((0) - 23+1-51 1700 ((((_))
< 2
10
111 1.
(95)
For part (b), given any (n, 1) E S x R x 22, we have
(1D00)
n.O. 20 1012=supp
Σ
D "u 0, a, w11² dud0
(96)
Mi X
J R2
sup Mi 51 x2
Σ
(10 (2 8 v 10.0--.-10 (-BB)) x/21 -1.11 dudo
(97)
sup Mi 5t /
1
Σ
/
(000 (u', - a -.a-B.1) 229dr''d du'd0
(98)
= 1120
(99)
24

---

Published in Transactions on Machine Learning Research (07 (07/2022 2022)
Thus (57) is valid.
The second half of part (b) holds since
(1) 1
1, Di n, D [2 [-
n.8,v
1 (1)
n D 11 D 7,8,0
Dr [
(100)
22
(1 D 11 D, D. [zo (11111))
(101)
< V 11
(102)
where the first equality holds because of the RST-equivariance, i.e., Theorem 1, and the second equality follows from (57).
The proof of part (c) is exactly the same as that Proposition 3(c) of (Zhu et al., 2019). Specifically, we telescope the inequality (58) while making use of the non-expansiveness of the layer-wise features, i.e. Proposition 2(c). The detail is omitted.
C.3 Proof of Theorem 1
Proof. Theorem is direct consequence of Proposition More specifically,
( (0)
n&e (,x(10]- D ((L)
7,8.0 [x: [01]
s
D 00)
D [0)
(( D 100 [
+
T. Dr (x00 - n,
(z.00)
(103)
R.
520+3 20 (0)
+ 28+1-11 | [1x
(104)
-20+1 (4L
21 ||.
(105)
where the second inequality comes from Proposition 3(c) and Proposition 3(d). This concludes the proof of Theorem 1.
D Reproducibility
D.1 License of Datasets
MNIST. Creative Commons Attribution-Shar Alike 3.0 license LeCun (1998).
Fashion-MNIST. MIT license Xiao et al. (2017b).
STL-10. BSD 3-Clause License Coales el al. (2011a).
D.2 Code Implementation
Our code and experiments in our paper are available at https://github. com/ /amp/laoaya///// aato-scale-traassaaiom---- Equiiiin-niivaiiattttnnn and Mashion-MNIST for this version.
We specifically include the experiments for MNIST
The code is built upon the GitHub repository the paper Sosnovik et al. (2020) under MIT license https://github. catei/.ooomooeeeeeeeeeeee comm/ISosnvvik//tte /een. For the implementation of Fourier-Bessel bases and Decomposed Convolutional Filters, build our code Zhu et al. (2019). In the submitted files, keep the names of the authors of Sosnovik et al. (2020), while our names remain anonymous throughout the repository.
25


Reviews:
[{"review_id": "P0a9QqrzxP", "reviewer": "Reviewer_3L9Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The idea is quite clean and straightforward. \n- The paper is written very clearly, with all the appropriate mathematics introduced in an understandable manner. Overall, the paper is a pleasure to read. \n- The claimed contribution of the paper (of treating scale + rotations + translations together) is indeed novel (although some papers, like the Polar transformer networks of Esteves et al. come close).\n\nDespite liking the paper and thinking it has a clear and useful contribution, in my opinion, it also has the following weaknesses:\n- The implementation details indicate that the RST architecture uses a truncated interval and expands filters of the G-CNN in the scale-dimension. This, I am afraid, suggests that it is a straightforward extension of the usual GCNN implementation-wise. \n- The method has a strong reliance on two papers (both cited in the paper): Cheng et al. and Sosnovik et al. With the former the main difference is the introduction of another dimension to handle scale. The second introduced steerable scale-equivariant filters. The paper is essentially a combination of these two works. The use of truncated intervals is present in most steerable CNNs (since one has to choose a finite basis).\n- The above can be overlooked if the experimental evaluation is scaled up and made more convincing. While the experiments are thorough on the datasets that are considered, they do need to be matched with other similar works in the area. Another source of weakness is that some modifications are made to existing baselines -- I think they should be compared to published results. I also think that the most appropriate datasets would be SIM2MNIST (from Esteves et al. \"Polar Transformer Networks\" ICLR'18) and MNIST-RTS (from Jaderberg et al, \"Spatial Transformer Networks\", NeurIPS'15). The one benchmark used is STL-10. However, the results reported are far from the state-of-art (including those for published GCNN baselines). I think this weakens the paper. \n- The deformation stability contribution needs to be better integrated into the wider paper. It is a bit abstract -- it would strengthen the paper if more examples and experiments are presented that validate their motivations and result. \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The primary contribution of the paper is to present a roto-scale-translation equivariant CNN for vision tasks. The paper begins with the motivation that these aspects are usually treated separately in NN design, but in many cases (such as in vision systems of autonomous vehicles), they are best treated jointly. The underlying mathematics is presented in a clear and concise manner, including the necessity and sufficiency of the correct notion of convolution (following now standard results in the literature). A secondary contribution is to present a (deformation) stability analysis (when the underlying \"transformation\" is not perfect  -- as might be seen in many cases of interest). Experiments on three tasks (one being a standard benchmark) are used to verify the efficacy of the approach. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the last two points made in \"weaknesses.\"", "broader_impact_concerns": "No impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ckeFtNzCSm", "reviewer": "Reviewer_iYwU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts: the work leads to new practical architectures for incorporating roto-scale invariances in CNNs while ensuring deformation stability by truncating filter decompositions to low-frequency components in appropriate bases. Empirically, these perform well.\n\nWeaknesses: the paper is quite incremental, as it mostly builds upon the previous works by Qiu et al (2018), Cheng et al (2019), Zhu et al (2019), which studied very similar constructions for translation, rotation and scaling, separately (rather than jointly).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes new architectures satisfying equivariance to the roto-scale-translation group, using (truncated) decompositions of convolutional filters in appropriate bases. The authors then show deformation stability bounds, showing that when appropriate norms of the filter coefficients are controlled, the model is near-equivariant to roto-scale-translations even when the input is slightly deformed.\nThe authors then provide numerical experiments on image datasets to illustrate the benefits of these architectures.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "presentation:\n* it is often hard to distinguish text and citations, e.g. in section 2. Please use \\citep more abundantly.\n* many equation references point to the appendix instead of the main text, please fix this\n\na couple of points that should be discussed more thoroughly:\n* the meaning of norms in assumption (A2): it is stated that these norms control certain smoothness properties of filters which help with stability -- it would be great to expand on this further\n* regarding scale invariance, you mention that using a single value of alpha, i.e. a single scale ($L_\\alpha = 1$ with $\\alpha = 0$ only) is best. I'm not sure I understand how such a model provides any scale invariance, given that you're only looking at a single scale?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r6L7D3P6Qe", "reviewer": "Reviewer_3Kxh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, and the mathematical background material on groups, actions, representations and equivariance is explained in a clear, concise and correct way. \n\nAs mentioned, the paper does something new that is quite hard (RST equivariance).\n\nThe stability analysis is a somewhat unique feature and is fairly convincing. The assumptions seem quite reasonable to me, and the resulting method is not too complicated to implement (requires a filter expansion as most equivariant CNNs).\n\nThe main weakness is in the scalability of the proposed method, and the small scale experiments. The paper uses regular representations and associated to that convolutions on the group itself. Since the group is 4 dimensional, this means feature maps are 4 dimensional. Aside from the rest of the implementation details, which seem not too problematic, the fact that you need to convolve in four dimensions makes the method compute and memory intensive. A better approach, as the authors note, would be to develop a steerable CNN version of the method, so that only 2D convolutions are required, and this is left for future work.\n\nThe other weakness of the paper is that the method is only evaluated on small and simple datasets. It would be nice to see future work (with a more scalable method) evaluated on tasks where scale equivariance is critical, such as perhaps ego-centric vision datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper makes three contributions:\n1. The first CNN to simultaneously achieve equivariance to rotation, scale, and translation\n2. An analysis of the stability to deformations outside of this group, and a method of constraining the filters to achieve stability.\n3. Results on in distribution and out of distribution performance on small image datasets\n\nAchieving simultaneous equivariance to rotation, scale and translation is quite difficult and has not been achieved before as far as I know. And most papers on equivariance only consider the transformations inside the group of interest, and not general deformations (diffeomorphisms) around it. Being exactly equivariant to deformations is not feasible, but as the paper shows, one can make the network stable to them.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Typo in Section 6, bullet 1: 'realist' -> 'realistic'", "broader_impact_concerns": "No concerns beyond those that apply to all work in DL architectures.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a CNN that achieves simultaneous invariance to rotations, scales, and translations. Reviewers noted that the problem is challenging, the approach is clean, and the paper is well-written, but they initially raised concerns regarding the empirical comparison between the proposed approach and previous work. The authors addressed these concerns with additional experiments during the rebuttal period. Reviewers also noted that the paper is essentially a combination of approaches from previous work, but given the difficulty of the problem, the AE believes that the paper is likely to be of interest to some individuals in TMLR's audience. All reviewers now either learn toward or recommend accepting the paper. The AE is thus happy to recommend accepting the paper as is.
Decision: Accept (poster)