B-Spline CNNs on Lie groups
Authors: Erik J Bekkers
Venue:  (2020)
Keywords: equivariance
Lie groups
B-Splines
G-CNNs
deep learning
group convolution
computer vision
medical image analysis

Abstract: Group convolutional neural networks (G-CNNs) can be used to improve classical CNNs by equipping them with the geometric structure of groups. Central in the success of G-CNNs is the lifting of feature maps to higher dimensional disentangled representations, in which data characteristics are effectively learned, geometric data-augmentations are made obsolete, and predictable behavior under geometric transformations (equivariance) is guaranteed via group theory. Currently, however, the practical implementations of G-CNNs are limited to either discrete groups (that leave the grid intact) or continuous compact groups such as rotations (that enable the use of Fourier theory). In this paper we lift these limitations and propose a modular framework for the design and implementation of G-CNNs for arbitrary Lie groups. In our approach the differential structure of Lie groups is used to expand convolution kernels in a generic basis of B-splines that is defined on the Lie algebra. This leads to a flexible framework that enables localized, atrous, and deformable convolutions in G-CNNs by means of respectively localized, sparse and non-uniform B-spline expansions. The impact and potential of our approach is studied on two benchmark datasets: cancer detection in histopathology slides (PCam dataset) in which rotation equivariance plays a key role and facial landmark localization (CelebA dataset) in which scale equivariance is important. In both cases, G-CNN architectures outperform their classical 2D counterparts and the added value of atrous and localized group convolutions is studied in detail.

Published as a conference paper at ICLR 2020
B-SPLINE CNNS ON LIE GROUPS
Erik J. Bekkers" Amsterdam Machine Learning Lab Informaties Institute
Centre for Analysis and Scientific Computing Applied Mathematics and Computer Science Eindhoven University of Technology
University of Amsterdam
J. b.eekers@uua...
ABSTRACT
Group convolutional neural networks (G-CNNs) can be used to improve classi- cal CNNs by equipping them with the geometric structure of groups. Central in the success of G-CNNs is the lifting of feature maps to higher dimensional disentangled representations in which data characteristics are effectively learned, geometric data-augmentattins are made obsolete, and predictable behavior under geometric transformations (equivariance) is guaranteed via group theory. Cur- rently, however, the practical implementations of G-CNNs are limited to either discrete groups (that leave the grid intact) or continuous compact groups such as rotations (that enable the use of Fourier theory). In this paper we lift these lim- itations and propose a modular framework for the design and implementation of G-CNNs for arbitrary Lie groups. In our approach the differential structure of Lie groups is used to expand convolution kernels in generic basis of B-splines that is defined on the Lie algebra. This leads to flexible framework that enables localized, atrous, and deformable convolutions in G-CNNs by means of respec- tively localized, sparse and non-uniform B-spline expansions. The impact and potential of our approach is studied on two benchmark datasets: cancer detection in histopathology slides in which rotation equivariance plays key role and facial landmark localization in which scale equivariance is important. In both cases, G- CNN architectures outperform their classical 2D counterparts and the added value of atrous and localized group convolutions is studied in detail.
INTRODUCTION
Group convolutional neural networks (G-CNNs) are a class of neural networks that are equipped with the geometry of groups. This enables them to profit from the structure and symmetries in signal data such as images (Cohen & Welling 2016). A key feature of G-CNNs is that they are equivariant with respect to transformations described by the group, i.e., they guarantee predictable behavior under such transformations and are insensitive to both local and global transformations on the input data. Classical CNNs special case of G-CNNs that equivariant to translations and, in contrast to unconstrained NNs, they make advantage of (and preserve) the basic structure of signal data throughout the network (LeCun al. 1990). By considering larger groups (i.e. considering not just translation equivariance) additional geometric structure can be utilized in order to improve performance and data efficiency (see G-CNN literature in Sec. 2}.
Part of the success of G-CNNs can be attributed to the lifting of feature maps to higher dimen- sional objects that generated by matching kernels under range of poses (transformations in the group). This leads to disentanglement with respect to the pose and together with the group structure this enables a flexible way of learning high level representations terms of low-level acti- vated neurons observed in specific configurations, which we conceptually illustrate in Fig, T From a psucoopsycoolooical viewpoint, this resembles a hierarchical composition from low- to high-level features akin to the recognition-be segnittonnby-componenttt model by [Biederman (1987), a viewpoint which is also adopted in work on capsule networks (Hinton et al., 2011 1; /abour et al. 2017). In particular in (Lenssen et al. (2018) the group theoretical connection is made explicit with equivariant capsules that provide a sparse index/value representation of feature maps on groups (Gens & Domingos 2014).
Work done Eindhoven University of Techology

---

Published as conference paper at ICLR 2020
Hoto transsatssltnon group SE(2) R22 SO(2)
soup
uap (ectivatirs for ertented
Eurtuze stpp
teap
Liits st traeojomes Geeen sersoh
2D cann bernals
ust.. trouditice mislis the lrpat
P-1
Pryatiin over vakgroup parrstder food incertaser
Lifting layer
Group cony lnger
Projection layer
Scale: Aranslation group R²2 x lt
Irovétice ondissiis ife oppa
Gioms keradd congee iniiatt axtiontions
Actlestion for cirlls caol possiico and scal
Actisation for foors moch passiim and roak
In G-CNNs feature maps are lifted to the high-dimensions domain of the group G which features are disentangled with respect to pose/transformation parameters. G-convolution kernels then learn to recognize high-level features in terms of patterns of relative transformations, described by the group structure. This is conceptually illustrated for the detection of faces, which in the (2) case are considered as pattern of lines in relative positions and orientations, in the IRa g+ case as blobs/circles relative positions and scales.
Figure 1:
Representing low-level features via features maps on groups, is done G-CNNs, is also moti- vated by the findings of Hubel & Wiesel (1959) and Bosking et al.] (1997) on the organization of orientation sensitive simple cells in the primary visual cortex VI. These findings are mathematically
modeled by sub-Riemannian geometry Lie groups (Petitot 2003 Citti & Sarti 20066 Duits et al. 2014) and led effective algorithms in image analysis (Franken Duits) 2009 Bekkers et al,
2015b. Favali al., 2016 Duits et al., [0188] [Baspinar] 2018). In recent work Montobbio et al. (2019) show that such advanced VI modeling geometries emerge in specific CNN architectures and Ecker al. (209)) the relation between group structure and the organization of VI is explicitly employed to effectively recover actual V1 neuronal activities from stimuli by means of G-CNNs.
G-CNNs well motivated from both mathematical point of view (Cohen et al.) 2018: Sa Kondor wwwps//hhhooiiiceeeooomaaaeeeeee
& Trivedi 2018) and classical CNNs is convincingly demonstrated by the growing body of G-CNN literature (see Sec. (2. However, their practical implementations are limited to either discrete groups (that leave the grid intact) or continuous, (locally) compact, unimodular groups such as roto-translations (that enable the use of Fourier theory). In this paper we lift these limitations and propose framework for the design and implementation of G-CNNs for arbitrary Lie groups.
point of view and their improvement
The proposed approach for G-CNNs relies on definition of B-splines on Lie groups which we use to expand and sample group convolution kernels. B-splines are piece-wise polynomials with local support and are classically defined on flat Fuclidean spaces Rdd In this paper we generalize B-splines to Lie groups and formulate a definition using the differential structure Lie groups in which B-splines are essentially defined the (flat) vector space of the Lie algebra obtained by the logarithmic map, see Fig. 2 The result is a flexible framework for B-splines on arbitrary Lie groups and it enables the construction of G-CNNs with properties that cannot be achieved via traditional Fourier-type basis expansion methods. Such properties include localized, atrous, and deformable convolutions in G-CNNs by means of respectively localized, sparse and non-uniform B- B-splines.
Although concepts described in this paper apply to arbitrary Lie groups, we here concentrate on the analysis of input data that lives on Rdd and consider G-CNNs for affine groups G Rd H
2

---

Published as a conference paper at ICLR 2020
B-spline on R
The Log-map allows us to map elements from curved manifolds such as the 2-sphere to a flat Euclidean tangent space. For Lie groups the Log- map is analytic, globally defined, and it provides us with flexible tool to de- fine group convolution kernels via B- splines. In our Lie group context the 2-sphere is treated as a quotient group 50(3) (2))). Technical details are given in Sec. and App.
Figure 2:
Exp
g
Exр
Log
91
Log
B-spline on S
that are the semi-direct product of the translation group with Lie group H. As such, only a few core definitions about the Lie group H (group product, inverse, Log, and action on Re)) need to be implemented in order to build full G-CNNs that are locally equivariant the transformations H.
The impact and potential of our approach is studied on two datasets in which respectively rotation and scale equivariance plays key role: cancer detection in histopathology slides (PCam dataset) and facial landmark localization (CelebA dataset). In both cases G-CNNs out-perform their classical 2D counterparts and the added value of atrous and localized G-convolutions is studied in detail.
2 RELATED WORK
G-CNNs The introduction of G-CNNs to the machine learning community by Cohen & Welling (216)) led to a growing body of G-CNN literature that consistently demonstrates an improvement of G-CNNs over classical CNNs. It can be roughly divided into work on discrete G-CNNs (Co-
hen & Welling 2016 Dieleman et al. 2016 Winkels & Cohen, 2018; Worrall & Brostow 2018
Hoogeboom et al. 20.8). regular continuous G-CNNs (Oyallon & Mallat [015 Bekkers et al.
2015a 2018b Weiler et al. 2018a Zhou et al., 2017 Marcos et al. (017)) and steerable contin-
uous G-CNNs (Cohen et al.
2018b; Worrall et al.J 2017 Kondor & Trivedi 2018: Thomas et al.)
2018 Weiler et al. 2018a| Esteves
20188a) Andrearczyk al. 2019)) Since 3D rotations can only be sampled in very restrictive ways (without destroying the group structure) the construction of 3D roto-translation G-CNNs is limited. In order avoid having to sample all together, steer- able (G-)CNNs can be used. These are specialized G-CNNs in which the kernels are expanded in circlar/spherical harmonics and computations take place using the basis coefficients only (Chirikjian & Kyatkin 2000 Franken 2008 Almsick 2007 Skibbe & Reisert (017). The latter approach is however only possible for unimodular groups such as roto-translations.
Scale equivariance In this paper we experiment with scale-translation G-CNNs, which is the first direct application of G-CNNs to achieve equivariance beyond roto roto-translations. Scale equivariance
is however addressed in several settings (Henriques & Vedaldi 2017; Esteves et al., 2018b Marcos
cct al. 2018 Tai et al.) 2019)) Worrall & Welling [019]] Jaderberg et al.] 2015; Li et al. 2019). of which (Worrall & Welling (019) most related. There, scale-space theory and semi-group theory is used to construct scale equivariant layers that elegantly take care of moving band-limits due to rescaling. Although our work differs in several ways (e.g. non-learned lifting layer, discrete group convolutions via atrous kernels, semi-group theory), the first two layers of deep scale-space networks relate our lifting layer by treating our B-splines der the semi-group action of (Worrall & Welling 2019), as we show in App. C.I Li et al. (2019) achieve scale invariance by sharing weights among kernels with different dilation rates. Their ap- proach can be considered a special case of our proposed B-spline G-CNNs with kernels that do not encode scale interactions. Related work by Tai et al. (2099) and Henriques & Vedaldi (2017) relies on the same Lie group principles as log-polar coordinates Esteves et al. (2018b), to handle equivariance. Such
superposition of dirac deltas transformed un-
do in this paper (the Log map) construct convenient co-
ordinate systems, methods are however generally not translation equivariant and do not deal with local symmetries as they act globally on feature maps, much like spatial transformer networks (Jaderberg et al. 2015).
B-splines and vector fields in deep learning The current work can be seen as generalization of the B-spline based SE(() CNNs of Bekkers et al.] (2015a] 2018b), see Sec. 3.3 Closely related is also the work of Fey et al. (2018) in which B-splines are used generalize CNNs Sun-Fuclidean
3

---

Published as a conference paper at ICLR 2020
data (graphs). There it is proposed to perform convolution via B-spline kernels R that take as inputs vectors u(i.j) E R that relate any two points i. E G in the graph to each other. How u((,5) is constructed is left by embedding the graph in an Euclidean space where points relate via offset vectors. In our work on Lie G-CNNs, two points g.g' E G' in the Lie group G' relate via the logarithmic map u(g.g') Log "'dg', Another related approach in which convolutions take place on manifolds in terms of "offset vectors" is the work by Cohen et al. (2019). There, points relate via the exponential map with respect to gauge frames rather than the left-invariant vector fields as in this paper, see App.
design choice, however, in (Fey et al.) 2018) this is typically done
3 LIE GROUP CNNS
3.1 PRELIMINARIES AND NOTATION
The following describes the essential tools required for deriving generic framework G-CNNs. Although we treat groups mostly in an abstract setting, we here provide examples for the roto- translation group and refer to App. B for more details, explicit formula's, s, and figures for several groups. Implementations are available at poattps://qttee
com/ ebekkers wttpkkkkersssppppessggsseee
Group A group is defined by set G together with binary operator :, the group product, that satisfies the following axioms: Closure: For all h, g E G we have h 9 E G; Identiy: There exists an identity element et Inverse: for each g E G there exists an inverse element g E G such that gg'.g=1 g g = e; and Associativity: For each g, h, i E G we have (g h) g (h-
Lie group and Lie algebra If furthermore the group has the structure of differential manifold and the group product and inverse are smooth, it is called a Lie group. The differentiability of the group induces notion of infinitesimal generators (see also the exponential map below), which are elements of the Lic algebra g. The Lie algebra consists of a vector space (of generators), that is typically identified with the tangent space (G) at the identity e, together with bilinear operator called the Lie bracket. In this work the Lie bracket is not of interest and we simply say T.)
Exponential and logarithmic map We expand vectors in g in basis {A,) and write A R Σ. a' A; g, with components a = (a'..... a²) E R". This allows us to identify the Lie algebra with R". Lie groups come with a logarithmic map that maps elements from the typically non-flat manifolds G to the flat Euclidean vector space g via A Logg (conversely y Exp A), see Fig. 2
Semi-direct product groups In this paper we specifically consider (affine) Lie groups of type G Rd x H that are the semi-direct product of the translation group Rd with a Lie group H that acts on
Rd Let & Oxx x denote the action of h E H on x E IR; it describes how a vector in Rd is transformed by h. Then the group product of R2d H is given by
(I) with 011 = (x1, 1,),)).). 92 = (x2, h22 E G. X1, X2 E Rd and N1, E H. For example the special
91 92 = (x1, h1) (x2, h2) = (x1 + h1 @ x2, h1 f2),
Euclidean motion group (E is constructed by choosing H = SO()),, the group of x 2 rotation matrices with matrix multiplication as the group product. The group product of G is then given by
(x1, Re,)) (x2, Re,) = (x1 + Rs, X2, X2, Ro.):
with X1, X2 E I2 and Re, Re, E SO(2) rotation matrices parameterized by rotation angle 0,, and in which rotations act on vectors in R² simply by matrix vector multiplication.
Group representations We consider linear transformations Cg (G-Ls(X) L2(X) - Lx(X) that transform functions (or feature maps) f E Lx(X) on some space X as representations of a group G if they share the group structure via
(C C. (X) f)(z) (C (C f)))
with o denoting function composition. Thus, a concatenation of two such transformations, param- eterized by Φ1 and J2. can be described by single transformation parameterized by 91 92- For semi-direc product groups G = Rd x H such representation can be split into
CG (x) L -lx())
H (X)(
Equivariance An operator & Lx(X)= + L2(Y) is said to be equivariant to G when it satisfies
Wgeg: : cg (G-La(Y) + t 2 (X))
(3)
4

---

Published as a conference paper at ICLR 2020
3.2 GROUP CONVOLUTIONAL NEURAL NETWORKS
Equivariance of artificial neural networks (NNs) with respect to a group G is a desirable property as it guarantees that no information is lost when applying transformations to the input, the information is merely shifted to different locations in the network. It turns out that if we want NNs to be equivariant, then our only option to use layers whose linear operator is defined by group convolutions. We summarize this in Thm. I To get there, we start with the traditional definition of NN layers via
y (K b)
with 2 E X' the input vector, Kgg : X' + 3/ linear map parameterized by weight vector w, with l E V a bias term, and á a point-wise non-linearity. In classical neural networks X' RN= and = R are Euclidean vector spaces and the linear map Ku = IR-, XNa a weight matrix. this work we instead focus structured data and consider feature maps some domain X as functions f: - RN,, the space of which we denote with (L)(X)) In this case X = (Lg(X)) No and y = (z) are the spaces of multi- caalt-l feature maps, b E RN,, and K X' - y is kernel operator: Now when we constrain the linear operator Kr to be equivariant under transformations in some group G we arrive at group convolutional neural networks. This is formalized in the following theorem on equivariant maps between homogeneous spaces (see (Duits Burgeth 2007 Kondor & Trivedi 2018 Cohen et al., 2018 for related statements).
Theorem 1, Let operator K. L2(X) - LY be linear and bounded, let X, Y be homogeneous spaces which Lie group G act transitively, and dux a Radon measure on X, then
1. K. is a kernel operator, i.e., (xxx) (KI))))
fx kly. (x)dxxx
2. under the equivariance constraint of Eq. 3 the map is defined by a one-argument kernel
k(,, dux (g @ux,,
k(g O
(4)
dax(a)
for any 9y E G such that y 9y @ yo for some fixed origin yo E Y.
3. Y G/H is the quotient of G with H = Stabo(yo) = {g E Glg ① = yot then the kernel is constrained via
YREH, YEEX:
k(x)=
dux Gw) dux(x)
k(h -1 x))
(5)
Proof. See App. A
Corollary 1. !fX = Rd is homogeneous space of an affine Lie group G Rdd dd x H and djux (x)
dr is the Lebesgue measure && then the kernel front-factor simplifies to dpx ((x)
dux(x)
det
with det fu] denoting the determinant of the matrix representation of h, for any g = (x, E If
X G and (x) is Haar measure on G then dax(o-1@x) 1.
dux(*)
In view of Thm. [ we see that standard CNNs to translations. In this case the domain of the feature maps X = I2 coincides with the space of translation vectors in the translation group G (Rd +). It is well known that if we want the networks to be translation and rotation equivariant (G = Rd x SO(d)), but stick to planar feature maps, then the kernels should be rotation invariant, which of course limits representation power. This constraint is due Thm. [ item 3, since the domain of such features maps is quotient group Rd = (0) x (Odd) in which rotations (SO( are factored out of the roto-translation group G = Rd SO(d). Thus, in order to maximize representation power (without constrains on k) the feature maps should be lifted to the higher dimensional domain of the group itself (i.c. Y = G). We therefore propose to build G-CNNs with the following 3 types of layers (illustrated in Fig. M):
special case of G-CNNs that are equivariant
Lifting layer (X = Rd ; Y G): this layer is defined by lifting correlations
(K)())) (k* (g) det
(cg (2(*)
La (=d,xxx)
with -xx h), which by splitting of the representation (Eq. 2 can be written as
(KC)) = (g) = (kL, * (x),
(6)
5

---

Published as a conference paper at ICLR 2020
with (x)x) = det
(f (x) the transformed kernel. Lifting correlations thus
match kernel with the input feature map under all possible transformations in G.
Group correlation layer (X G, Y G): In this case K. is defined by group correlations
(KCF) (K F) CG ) - K (()dda))()dd(()))
with (p(gg a Haar measure on G. We can again split this cross-correlation into a transfor- mation of K followed by a spatial cross-correlation via
(KF) - FK = (Kn *Rय (xx,,
with Ks. (x, h) = (khh+++ xx h- hh) the convolution kernel transformed by E and in which we overload *gg to indicate cross-correlation the Rd part of G = R x H.
Projection layer (X G, Rd): In this case K. is linear projection defined by
(KF)) - F(x, h)dp(h))
where simply integrate over H instead of using a kernel that would otherwise be stant over H and spatially isotropic with respect to H.
App. provides explicit formula's for these layers for several groups E.g., in the SE(2) we get:
(E)))-ifftn::
(kif)) 0) = k(R 1 .(x' -x))f(x''xxx'
JR2
[ K(R5".(X' -x)- 0 mod Fxx) (f j'''''' de'.
(E(())coolllttio:: (K+F)(x,))
JB2
3.3 B-SPLINES ON LIE GROUPS
Central in our formulation of G-CNNs is the transformation of convolution kernels under the action
of H as described above in Eqs. (6) and 7 in the continuous setting. For the implementation of G-CNNs the kemnels and their transformations need to be sampled on discrete grid. We expand on the idea's in (Bekkers et al. 2015a; 2018b) Weiler et al. (018b) to express the kernels in an analytic form which computations. In particular we generalize the approach of Bekkers et al. [2015a] 2018b) to expand group correlation kernels in basis of shifted cardinal B-splines, which are localized polynomial functions on IR with finite support. In (Bekkers et al. 2015a; 2018b). B-splines on I2 could be used construct kernels on SE(2) by identifying the group with the space of positions and orientations and simply using periodic splines the 1D orientation axis S'. However, in order to construct B-splines on arbitrary Lie groups, we need a generalization.
then sample under arbitrary transformations in G perform the actual
the following we propose a new
definition of B-splines on Lie groups H which enables us to construct the kernels on Rdd H that are required in the G-correlations (Eq. 7
Definition (Cardinal B-spline on I. The ID cardinal B-Spline of degree n be is defined as
(x) = (1[-2-1) (1))
(- (x),
where (x) denotes re-fldd convolution of the indicator function
[ The multi-variate cardinal
B-spline on Rd, with coordinates x (zo...., E: Rd. is defined via the tensor product
B (x) =
Bda (x)) (3 (²aaa)
(10)
d times
Cardinal B-splines are piece-wist wise polynomials and are localized on support |- 1=t _ Functions be expanded in basis of shifted cardinal B-splines, which we simply refer B-splines.

---

Published as a conference paper at ICLR 2020
Definition 2 (B- (B-splines on I.). A B-spline is function f R IR expanded in basis that consists of shifted and scaled copies of the cardinal B-spline
f(x)
BB
x x-xi
(11)
8x
and is fully characterized by spline degree scale 8x, set of centers {x:) with X, E Rd and Cx)"
corresponding coefficients с c= (cI C2...., ...)) E R... The B- B-spline is called uniform if the set of centers [x= forms uniform grid on RR, which the distance |xj - x;|| between neighbouring centers X X, E Red is constant along each axis and equal to 8x.
Definition 3 (B (B-splines on Lie group H). A B-spline on H is a function f: H R expanded in basis that consists of shifted (by left multiplication) and scaled copies of the cardinal B-spline
N
in (Log h
f(h) = Σ
C C
(12)
with h E H and Log: H + h the logarithmic map on H. The B-spline is fully characterized by the spline degree n, scale 8h, set of centers {hr) with h, E H and corresponding coefficients
с == (c1, ((1,02, c2...., c2,) E RN The spline called uniform if the distance [| h h2ll between
neighbouring centers ha by E H is constant.
Examples of B-splines on Lie groups H are given in Fig. 3 In this paper choose to expand convolution kernels on G Rd x as the tensor product of B-splines on IRd and respectively
and obtain functions Rd x H via
f(x, = Σ
N
BB BRR
(xxx) BRA,n (Toghh boghh"h1 x X-
(13)
8h
Note that that one could also directly define B-splines on G via (2). however, this splitting ensures we can use a regular Cartesion grid on the Rd part. In our experiments we use B-splines as in (3 and consider the coefficients and/or 5n)) are fixed by design. Some design choices are the following (and illustrated in Fig.
trainable parameters and the centers (x; and/or h1) and scales (3x
Global vs localized uniform B-splines The notion of uniform B B-spline globally covering H exists only for a small set of Lie groups, e.g. for any ID group and abelian groups, and it is not possible to construct uniform B-splines on Lie groups in general due we find that it is possible to construct approximately uniform B-splines on compact groups either by constructing grid of centers (h; J on H that approximately uniformly covers H, e.g. by using repulsion model in which |[Lo h -...h hn]l between any two grid points h;, h5 E H is maximized (as is done in Fig. 3 or by specifying a uniform localized grid on the lie algebra h and obtaining the centers via the exponential map. The latter approach is in fact possible for any Lie group and leads to notion of localized convolution kernels that have a finite support on H, see Fig.
zero commutators. Nevertheless,
Atrous B-splines Atrous convolutions, i.c. convolutions with sparse kernels defined by weights interleaved with zeros (Holschneider et al.) 1990), are commonly used to increase the effective receptive field size and add a notion of scale deep CNNs YYu & Koltun 2016 Chen al. 2018). Atrous convolution kernels can be constructed with B-splines by fixing the scale factors and 8h, e.g. to the grid size, and increasing the distance between the center points x; and ht,
Non-uniform/deformable B-splines In non-uniform B-splines the centers and h; do not neces- sarily need lie on regular grid. Then, deformable CNNs, first proposed by DDi et al. (2017). are obtained by treating the centers trainable parameters. For B-spline CNNs on E of order this in fact leads to the deformable convolution layers as defined in (Dai et al. 2017).
Modular design The design of G-correlation layers (Eqs. (68) using B-spline kernels (Eqs.
I3 results in a generic and modular construction of G-CNNs that are equivariant to Lie groups G and only requires few group specifie definitions (see examples in App.B). The group structure of
H (group product and inverse), the action 3 of H on Rd (together with the group structure of H this automatically defines the structure of G R H), and the logarithmic map Log : H - h.

---

Published as a conference paper at ICLR 2020
Partition of unity
B-Spline
= SO(2), N-8
3.7
27
3.2
2m
(RR+. :.),) N=6
2 (0)3) N=50
N=500
N=5000
Figure 3: Left: The sum of all B-spline basis functions add up to one, illustrating partition of unity on the 2D rotation group SO(2) (row 1), scaling/dilation group (R+,-) (row 2), and the sphere S treated the quotient group (0(3)
with B-spline centers indicated with green dots (row 3-5). Right: A random B-Spline on SO(2) (row 1) and (R+.)) (row 2) and reconstruction of a color texture on the sphere 5 at several scales (row 3-5) to illustrate multi-scale properties.
Figure 4: A B-Spline IR2 (row 1), sampled on a grid (row 2), and B-spline on the sphere (row 3). From left to right: a localized ker- nel, scaled kernel by increas- ing 8x and 85, atrous kernel, deformable kernel. A green circle is drawn around each B- spline center with radius 2 4s4 to indicate the individ- ual basis functions.
8

---

Published as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 ROTO-TRANSLATION CNNS
Data The PatchCamelyon (PCam) dataset (Veeling al. 2018) consists of 327,680 RGB patches taken from histopathologic scans of lymph node sections and is derived from Camelyon16 (Ehte- Shami Bejnordi et al., 2017). The patches are binary labeled for the presence of metastasis. The classification problem is truly rotation invariant as image features appear under arbitrary rotations at all levels of abstraction, e.g. from edges (low-level) to individual cells to the tissue (high-level).
Experiments G-CNNs ensure roto-translation equivariance both locally (low-level) and globally (high-level) and invariance is achieved by means of pooling. In our experiments we test the perfor- mance of roto-translation G-CNNs (with G = IR² x SO(2)) against a 2D baseline and investigate the effect of different choices (local, global, atrous) for defining the kernels on the (O(2)-part of the network, cf. Eq. (13) and Fig. 4 Each network has the same architecture (detailed in App. (D) but the kernels are sampled with varying resolution on = (2),, denoted with Nn. and with varying resolution of the B-splines, which is achieved by varying 8n and the number of basis functions on H, denoted with Ng. Each network has approximately the same number of trainable weights. Each net- work is trained with data-augmentation (see App. D for details), but for reference we also compare to 2D and a (2) model (N₂ = N2, = 12) which are trained without 90 rotation augmentation.
The results are summarized in Fig. Here N₂, = 1 means the kernels are transformed for only one rotation, which coincides with standard 2D convolutions (our baseline). A result labeled "dense" with N₁,- 16 and N& = 8 means the convolution kernels are rotated 16 times and the kernels are expanded in a B-spline hasis with 8 basis functions to fully cover H. The label "local" means the 2= between them, with S/ equal basis is localized with N₁ basis functions with spacing of 8h 16
to the grid resolution. Atrous kernels are spaced equidistantly on H and have 8h No.
Results We generally observe that finer sampling of SO(2) leads to better results up until N₂ = 12 after which results slightly degrade. This is line with findings in (Bekkers et al. 2018a). The degradation after this point could be explained by overfitting; there is limit on the resolution of the signal generated by rotating 5x5 convolution kernels; at some point the splines are described in more detail than the data and thus an unnecessary amount of coefficients are trained. One could still benefit from sampling coarse kemels (low N₂)) on fine grid (high No), e.g. compare the cases N₂, > No for fixed Ng. This is in line with findings in (Weiler et al. 2018b) where a fixed circular harmonic basis is used. Generally, atrous kernels tend to outperform dense kernels as do the localized kernels in the low N& regime. Finally, comparing the models with and without 90" augmentation show that such augmentations are crucial for the 2D model but hardly affect the SE model. Moreover, the SE()) model without outperforms the 2D model with augmentation. This confirms the theory: G-CNNs guarantee both local and global equivariance by construction, whereas with augmentations valuable network capacity is spend on learning (only) global invariance. The very modest drop in the S (2) case may be due to the discretization of the network on grid after which is no longer purely equivariant but rather approximately, which may be compensated for via augmentations.
4.2 SCALE-TRANSLATION CNNS
Data The CelebA dataset LLu et al. 2015) contains 202.599 RGB images of varying size of celebri- ties together with labels for attributes (hair color, glasses, hat, etc) and 5 annotated facial landmarks (2 eyes, 1 nose, 2 corners of the mouth). We reformatted the data as follows. All images are isotrop- ically scaled to a maximum width or height of 128 and if necessary padded in the other dimension with zeros to obtain a size of 128x 1 128. For each image we took the distance between the eyes as reference for the size of the face and categorized each image into above and below average size. For each unique celebrity with at least 1 image per class, we randomly sampled I image per class. The final dataset consists of 17,548 images of 128x 128 of 8,774 celebritics with faces at varying scales. Each image is labeled with 5 heatmaps constructed by sampling Gaussian with standard deviation 1.5 centered around each landmark.
Experiments We train traesltrannlttion G-CNN (with G = R x R) with different choices for kernels. The "dense" networks have kernels defined over the whole discretization of H = R+ and thus consider interactions between features at all scales. The "local" networks consider only

---

Published as a conference paper at ICLR 2020
0.680
0.99
86
0.675
0.870
My=4
0.98
Accuracy
0.085
80
My=12
Ny=8
Accuracy
0.97
2D-conv (Bead scale)
0.860
My=16
20-com (fized sugr)
0.855
Dense
G--onv (dense))
Dense weo augm
(local, MGF11)
0.850
G-conu (loaa,, My=5)
Local
0.35
16
2.53 Scale range of scales sampled)
1.41
4. (9)
Number of basis functions N2
Figure 5: Left: results of roto-translation G-CNNs on tumor classification (PCam dataset), Right: results of scale-translation G-CNNs on landmark localization (CelebA dataset).
interaction between neighbouring scales via localized kernels (N₂ = 3) or no scale interaction at all (N& = 1). Either way, each G-CNN is multi-scale network in which kernels are applied at range of scales. We compared against a 2D baseline with fixed-scale kernels which we tested for several scales separately. In the G-CNNs, H is uniformly sampled (w.rt. to the metric on H) fixed scale range, generating the discrete sets Ha (c(t-1)sh) i- with 8/, 1 In 2. Each G-CNN is sampled with the same resolution in H with Sh, and each B-spline basis function is centered on the discrete grid (i.e. h; E H1... We note that the discretization of // is formally no longer a group as is not closed, however, the group structure still applies locally. The result is that information may leak out of the domain in a similar way as happens spatially in standard zero eero-padded 2D CNNs (translational G- G-CNNs), in which the discretized domain of translations is also no longer (locally) compact. This information leak can be avoided by using localized kernels of size N₂ = along the H axis, as is also done in (Worrall & Welling, 2019 Li et al. (019). The networks are trained without data- augmentation, except for the 2D network, which for reference we train with and without random scale augmentations
time. Augmentations were done with a random scale-factor between 1 and 1.4. We found that scale augmentations beyond 1.4 did not improve results.
Results Fig. 5 summarizes the results. By testing our 2D baseline at several scales we observe that there is an optimal scale (h- = 2) that gives a best trade off between the scale variations in the data. This set of experiments is also used rule out the idea that G-CNNs outperform the 2D bascline simply because they have a larger effective receptive field size. For large scale ranges the G-CNNs start to outperform 2D CNNs these networks consider both small and large scale features simul- tancously (multi-scale behavior). Comparing different G-CNN kernel specifications we observe that enabling interactions between neighbouring scales, via localized kernels on I ("local", Nu. 3), outperforms both all-scale interactions ("dense") and no-scale interaction at all (N₂, = 1). This finding in line with those in (Worrall Welling, 2019). Finally, although 2D CNNs moderately benefit from scale augmentations, it is not possible to achieve the performance of G-CNN.. It seems that a multi-scale approach (G-CNNs) essential.
CONCLUSION
This paper presents a flexible framework for building G-CNNs for arbitrary Lie groups. The pro- posed B-spline basis functions, which are used to represent convolution kernels, have unique prop- erties that cannot be achieved by classical Fourier based basis functions. Such properties include the construction of localized, atrous, and deformable convolution kernels. We experimentally demon- strated the added value of localized and atrous group convolutions on two different applications, considering two different groups. In particular in experiments with scale-translation G-CNNs, ker- nel localization was important. The B-spline basis functions can be considered as smooth pixels on Lie groups and they enable us to design G-CNNs using familiar notions from classical CNN design (localized, atrous, and deformable convolutions). Future work will focus on exploring these options further in new applications that could benefit from equivariance constraints, for which the tools now available for large class of transformation groups via the proposed Lie group B-splines.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
Remco Duits (Eindhoven University Technology (TU/e)) is gratefully acknowledged for his con- tributions to the formulation and proof of Thm. 1amm.aaaa and for helpful discussions on Lie groups. I thank Maxime Lafarge (TU/e) for advice on setting up the PCam experiments. This work is part of the research programme VENI with project number 17290, which is (partly) financed by the Dutch Research Council (NWO).
REFERENCES
M.A. Almsick, van. Context models of lines and contours. PhD Thesis, Department of Biomedical Engineering, 2007.
Vincent Andrearczyk, Julien Fageot, Valentin Oreiller, Xavier Montet, and Adrien Depeursinge. Ex- ploring local rotation invariance in 3d {CNN}s with steerable filters. In International Conference Medical Imaging with Deep Learning Full Paper Track, London, United Kingdom, July 2019. URL https::////...........pp htttlh/tts:://pppt /ttp::::///pee/ee/eeeee
Wolfgang Arendt and Alexander V Bukhvalov. Integral representations of resolvents and semi- groups. In Forum Mathematicum, volume 6, pp. 111-136. Walter de Gruyter, Berlin/New York, 1994.
Emre Baspinar. Minimal Surfaces in Sub-Riemannian Structures and Functional Geometry of the Visual Cortex. PhD) Thesis, alma, 2018. URL httpt:t///// tttp///ttttttttoaeooeooeot /ttlittpp//mmdstttrrtttttt unibo. /668
Erik Bekkers, Remco Duits, and Marco Loog. Training of Templates for Object Recognition In- vertible Orientation Scores: Application to Optic Nerve Head Detection in Retinal Images. In X-C Tai, E Bae, TF Chan, and M Lysaker (eds.), Energy Minimization Methods in Computer Vision and Pattern Recognition, volume 8932 of Lecture Notes in Computer Science, pp. 464 477. Springer International Publishing, 201 5a. ISBN 978-3-319-14611111 doi: 10. 10.1007/978-33319-1461222
URL http://dx doi org 0tt./11.110000
31 14612-1622666
Erik Bekkers, Remco Duits, Alexey P Mashtakov, and Gonzalo Sanguinetti. A PDE Approach to Data-Driven Jub-Riemannian Geodesics in SE(2). SIAM Journal on Imaging Sciences, (4):2740-2770, 2015b. doi: 10.1137/15M1018A1 URL https:// 1ot...o/o/..1131//1/
15M1 0110018460
Erik J Bekkers, Maxime W Lafarge, Mitko Veta, Koen A J Eppenhof, Josien P W Pluim, and Remco Duits. Roaa-Trannnation Covariant Convolutional Networks for Medical Image Analysis. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-L.pez, and Gabor Fichtinger (eds.), Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pp. 440 448, Cham, 2018a. Springer International Publishing. ISBN 978-3-030-00928-1 URL wwtppxww..w..poooooooo 804 T1111/1344.3......
Erik J Bekkers, Marco Loog, Bart M ter Haar Romeny, and Remco Duits. Template matching via densities on the roto-translation group. IEEE transactions on pattern analysis and machine intelligence, 40(2):452-466,, 2018b, doi: (0.1009/PAAM.221..2222222 URLhttps::/doi..
oxg/10.1109/12al wtw.:/A.....1.......
Irving Biederman. cegeiittoon-b-compeeee a theory of human image understanding. Psycho- logical review, 94(2):115, 1987.
William H Bosking, Ying Zhang, Brett Schofield, and David Fitzpatrick. Orientation selectivity and the arrangement of horizontal connections in tree shrew striate cortex. Journal of neuroscience, 17(6):212-21277 1997.
Henri Bouma, Anna Vilanova, Javier Olivn Bescs, Bart M ter Haar Romeny, and Frans A Gerritsen. Fast and accurate Gaussian derivatives based on B-splines. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 406-417. Springer, 2007.
11

---

Published as a conference paper at ICLR 2020
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets. Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-93 848, April 2018. ISSN 0162-8828. doi: 10.1 1090///MM..001.2666655
Gregory S Chirikjian and Alexander B Kyatkin. Engineering applications of noncommutative har- monic analysis: with emphasis on rolation and motion groups. CRC press, 2000.
Giovanna Citti and Alessandro Sarti. A cortical based model of perceptual completion in the roto- translation space. Journal of Mathematical Imaging and Vision, 24(3):307-322 2006.
Taco Cohen, Mario Geiger, and Maurice Weiler. General Theory of Equivariant CNNs on Homo- geneous Spaces. arXiv preprint arXiv:/8/1.02011 2018a.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge Equivariant Convolu- tional Networks and the Icosahedral CNN. In International Conference on Machine Learning, pp. 1321-1330, 2019.
Taco S Cohen and Max Welling. Group equivariant convolutional networks. Int. Conf. on Machine Learning, pp. 2990-29999 2016.
Taco S Cohen, Mario Geiger, Jonas Khler, and Max Welling. Spherical CNNs. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018b.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In International Conference on Computer Vision (ICCV), 2017.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry convo- lutional neural networks. In Proceedings of the International Conference on Machine Learning, 2016.
R. Duits. Perceptual organization in image analysis. PhD thesis, Eindhoven University of Technol- ogy, the Netherlands, 2005.
R. Duits, S. P. L. Meesters, J.-M. Mirebeau, and J. M. Portegies. Optimal Paths for Variants of the 2d and 3d ReedsShepp Car with Applications in Image Analysis. Journal of Mathematical Imaging and Vision, February 2018. ISSN 1683. doi: 1011007//008811-018-779-- URL sttps://ddo doi wt1p10org111... 11t.:111000111--1111111 001-00000000-00000---------
Remco Duits and Bernhard Burgeth. Scale spaces on Lie groups. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 300-312. Springer, 2007.
Remco Duits, Ugo Boscain, Francesco Rossi, and Yuri Sachkov. Association fields via cuspless sub- Riemannian geodesics in SE (2). Journal of mathematical imaging and vision, 49(2):384-411 2014.
Alexander S. Ecker, Fabian Sinz, Emmanouil Froudarakis, Paul G. Fahey, Santiago A. Cadena, Edgar Y. Walker, Erick Cobos, Jacob Reimer, Andreas S. Tolias, and Matthias Bethge. A rotation- equivariant convolutional neural network model of primary visual cortex. In International Confer- ence on Learning Representations, 2019. URL https:://////.... http:::///ppeeeeeeeee net wor//oruummm id=
H1 1081AAgX
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes van Diest, Bram van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen A. W. M. van der Laak, and and the CAMELYONI6 Con- sortium. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast CancerMachine Learning Detection of Breast Cancer Lymph Node Metastases.Machinee Learning Detection of Breast Cancer Lymph Node Metastases, JAMA, 318(22):1199-2210, 2017. ISSN 0098-7484, doi: 10.1000/jama.2117.45555 URL https:
///oi.org/// 001 jama. 201 1 14585
Carlos Esteves, Christine
Ameesh Makadia. and Kostas Daniilidis, Learning SO(3) Equivariant Representations with Spherical CNNs. 201 Sa. URL https: : :tttps://arxvy
org/pdf/1711 00w.1110662111 w67.2122.ppf
12

---

Published as a conference paper at ICLR 2020
Carlos Esteves, Christine Allen-Blanchetee Xiaowei Zhou, and Kostas Daniilidis. Polar Transformer Networks. In Internationaa Conference on Learning Representations, 2018b. URL https:
www. htrpp//wwraaa.eeefoooooooooooood
Marta Favali, Samaneh Abbasi-Sureshjant Bart Haar Romeny, and Alessandro Sarti. Analysis of vessel connectivities in retinal images by cortically inspired spectral clustering. Journal of Mathematicaa Imaging and Vision, 56(1):158 172, 2016.
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Miler. SplineCNN: Fast geomet- ric deep learning with continuous B-spline kernels. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 869-877, 2018.
Erik Franken and Remco Duits. crossing-preservngg cohcrence--nhanccing diffusion on invertible orientation scores. International Journal of Computer Vision, 85(3):253, 2009.
Erik M Franken. Enhancement of crossing elongated strucfures in images. PhD thesis, Eindhoven University of Technology, Eindhoven, The Netherlands, 2008
Robert Gens and Pedro M Domingos. Deep symmetry networks. In Advances in neural information processing systems, pp. 2537-2545, 2014.
Joao F Henriques and Andrea Vedaldi. Warped convolutions: Efficient invariance spatial trans- formations. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1461-1469. JMLR. org 2017.
Geoffrey E Hinton, Alex Krizhevsky, and Sida Wang. Transforming auto-encoders. Interna- tional Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
Matthias Holschneider, Richard Boonland-Martinee Jean Morlet, and Ph Tchamitchian. A real-time algorithm for signal analysis with the help of the wavelet transform. In Wavelets, pp. 286-297. Springer, 1990.
Emiel Hoogeboom, Jorn WT Peters, Taco Cohen, and Max Welling. HexaConv. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
D. H. Hubel and T. N. Wiesel, Receptive fields of single neurones in the cat's striate cortex, The Journal of Physiology, 148(3):574-591, 1959. ISSN 1469-7793. doi: 10.11 10.1113/jphysio.19955 sp006308. URL http: dx doi. wtt.://www.......oooo//..... 1113:1111111///. http://..11...phsi.ooooo 1959. sp006308
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and others. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
L. V. Kantorovich and G. P. Akilov. Functional analysis. English transl.: Pergamon Press, Oxford, Moscow (1977) (Russian). 2nd rev. ed, 'nauka' edition, 1982.
Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceed- ings of Machine Learning Research, pp. 2747-2755 Stockholmsmssan, Stockholm Sweden, July 2018. PMLR. URL http:::/prooeedngggg mir wwwess//80/koneeeee
html
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back-propagation net- work. In Advances in neural information processing systems, pp. 396-404, 1990.
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In Advances in Neural Information Processing Systems, pp. 8844 8853, 2018.
Yanghao Li, Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Scale-Aware Trident Networks for Object Detection. In The IEEE International Conference Computer Vision (ICCV), October 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
13

---

Published as a conference paper at ICLR 2020
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation Equivariant Vector Field Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10:555088577 2017.
Diego Marcos, Benjamin Kellenberger, Sylvain Lobry, and Devis Tuia. Scale equivariance in CNNs with vector fields. arXiv preprint 1rXiv:1807.117555 2018.
Noemi Montobbio, Giovanna Citti, and Alessandro Sarti. From receptive profiles to a metric model of VI. Journal of computational neuroscience, 46(3):257-277, 2019.
Edouard Oyallon and Stphane Mallat. Deep roto-translation scattering for object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2865- 2873, 2015.
Jean Petitot. The neurogeometry of pinwheels as a Rab-Riemanninn contact structure. Journal of Physiology-Paris 97(2-3):265-300 2003.
Jorg Portegies, Gonzalo Sanguinetti, Stephan Meesters, and Remco Duits. New approximation of scale space kernel on SE (3) and applications in neuroimaging. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 40-52. Springer, 2015.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Henrik Skibbe and Marco Reisert. Spherical Tensor Algebra: A Toolkit for 3d Image Pro- cessing. Journal of Mathematical Imaging and Vision 38 2017. doi: 10.1007/ s10851-017-071557. htt.://00........./////////////.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Equivariant Transformer Networks. arXiv preprint 1191999 2019.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor Field Networks: Rotation and Translation-Equive Point Clouds. arXiv preprint arXiv:1802.08219, 2018.
Neural Networks for 3d
Bastiaan Veeling. Jasper Limmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation Equiv- ariant CNNs for Digital Pathology. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-Lpez, and Gabor Fichtinger (eds.), Medical Image Computing and Computer As- sisted Intervention MICCAI 2018, pp. 210-218, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00934-2.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montral, Canada., pp. 10402-10413, 201 8a.
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning Steerable Filters for Rotation Equivariant CNNs. In International Conference on Computer Vision and Pattern Recognition, 2018b.
Marysia Winkels and Taco S Cohen. 3d G-CNNs for Pulmonary Nodule Detection. In Internation Conference on Medical Imaging with Deep Learning (MIDL), 2018.
Daniel Worrall and Gabriel Brostow. CubeNet: Equivariance to 3d Rotation and Translation. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 585-602, Cham, 2018. Springer International Publishing. ISBN 978-3-030- 01228-1.
Daniel E Worrall and Max Welling. Deep Scale-spaces: Equivariance Over Scale. arXiv preprint arXiv: 16697, 2019,
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambeto, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), volume 2, 2017.
14

---

Published as a conference paper at ICLR 2020
Fisher Yu and Vladlen Koltun. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR, 2016.
Yanzhao Zhou, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Oriented response networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4961-4970. IEEE, 2017.
15

---

Published as a conference paper at ICLR 2020
A PROOF OF THEOREM
The following proof' the three sub-items of Thm.
It follows from (Kantorovich & Akilov 1982 Ch 9, Thm 5), or (Duits) 2005 Thm 1), that if K. is linear and bounded it is an integral operator.
Theorem, see e.g. (Arendt Bukhvalov 1994 Thm 1.3),
2. The left-equivariance constraint then imposes bi-left-invariince of the kernel k as follows, where Veec and (EL((X))
(Ko (Kocg (Cg (CG-La()) o ( (f)
J kV,x))(g (W,x))(g-'x)dar /
x klg
M v) (x))d integral 24-9 2x
[ k(y,x)flg -x)ff(
4.g 1.gg-12f(f (=)(((=()((( =x))
[ kly,))flg r)dx
/ i - 20) det dr.
(14)
Since 14 should hold for all f (X) we obtain
YgEG:
k(y. x) = det kg y. (x))
(15)
Furthermore, since G acts transitively on Y have that V V.DEY aoyE such that y = 9y30 and thus
k(y x) = k( yo: x)
15
[yo- By x) det k(o x))
det
for every By G such that y = 9y yo with arbitrary fixed origin yo Y. 3. Every homogeneous space Y of G can be identified with quotient group G/H. Choose an origin yo E Y s.t. VAEH h yo = yo. i.e., H = Stabe yoo then
k(yo, x) ich you x) k(x) det k(h z).
We further remark that When Y === [e) with E G the identify element of G, the symmetry constraint of Eq. (5) vanishes. Thus, in order construct equivariant maps without constraints on the kernel the functions should be lifted to the group G.
B EXAMPLES OF LIE GROUPS
In the following sub-sections some explicit examples of Lie groups H are given, together with their actions on Rdd and the Log operators. The required tools building B-spline based G-CNNs for Lie groups of the form G = Ie H are then automatically derived from these core definitions. E.g., the action @ of H on a space X defines a left-regular representation on functions on X via
(C (CH--a(X) (x) = f(g-t Ox).
When X direct product groups G = Rd x H automatically derived from the action of H on Rdd Eq. (1) and is in turn used to define the representations (see Eq. (2) Some examples are given below.
= G is the group itself, the action equals the group product. The group structure of semi-
B. TRANSLATION GROUP G (R4, +)
The group of translations is given by the space translation vectors Rd with the group product and inverse given by
g g = (x+x')
(-x),
16

---

Published as a conference paper at ICLR 2020
with 9 (x), (x')= G with x' R The identity element is (0). The left-regular representation on d-dimensional functions L produces translations of f via
(cg (CG-tLaRR)) f) - f(g g - f(x -xxx x).
Now, since the space translations can be identified with R2 the lifting (Eq. 6 and group corre- lations (Eq. (7)) coincide with the standard definition of cross-correlation. I.e.,
(x*))(x) (xx / k(x' (x)(xxx)xxx
The logarithmic map simply given by
Logg== = =x,
so the convolution kernels can be defined by regular splines on Rd, cf. Eq.
B.2 THE 2D ROTATION GROUP H = =00(2)
The special orthogonal group SO(2) consists of all orthogonal 2 matrix with determinant i.e. rotation matrices of the form
Cos 0 -sin ):
R
sin 0
cos 0
and the group product and inverse is given by the matrix product and matrix inverse:
h h = (R..Ro) (Ro+0r)
h = (R
with h (Re), h' (Rær) E SO(2) with 0, 0' E St. The identity element is (Ro) (I). The action of H on R² is given by matrix vector multiplication:
h C Rg.x,
with E R
Combining the group structure of 2D translations with rotations in SO(2) as semi-direct product group gives the roto-translation group SE(2) R² SO()), also known the special Euclidean motion group. The group structure of S2(() is automatically derived from that of SO(2), see Sec. for details.
The left-regular representation of SO(2) are given by
(c (c550()))1 f)(x') f(h xx)) f(R,"' 1(,,1'') x'),
(c) (00002))2 F) = F(h-l -h) = F(Rg
(c,, (c,00())
Fx N') = = .h)) = =(R,,' x'' Ro
Note that the latter two representations in terms of the rotation parameters 0, 0' E S represents the periodic shift - mod 2Tt along the rotation axis. See Fig. for an illustration for the trans- formation of S2) convolution kernels via the representation c - ( The determinant
of the Jacobian of the action of H on Rd see corollary [ structure we can write out the explicit forms for the lifting (Eq. (6) and group correlations (Eq. as follows:
det h 1. Using the above group
(xxf)(x,0) = / ((R -x))xxx)(''
(2)))-iif;:::
J22
(K+F)(X,)) -f K(R (R5'.(x'' (xx' x)) mod F d')'x' de'.
SE( correlatio::
The logarithmic map on H is given by the matrix logarithm
0
-0 2T
Log Rg = mod 2n
(SO(2)) = span {A1}
0
17

---

Published as a conference paper at ICLR 2020
A descriptor in terms of collection of group elements
SE(2)
h
5
h-S
A descriptor function (convolution kernel) SE(2) K E L2(E((2)
L
S002) K
2002) (L22)1))(( K
by (200)) K
Planzz cotation
Parisdic diit
R
R
Ran/4
Figure 6: Top row: In group theoretical terms we can describe smiley face via a collection of group elements c SE(2), which e.g. represents the locations and orientations of low-level features such as edges/lines. Such a collection of points transforms via left multiplication, e.g. g S), with h e SO(2). Bottom row: In group convolutional setting
h- S == {(0,h) g can work with convolution kernels which are functions on SE(2) that assign weights to locations in which locally oriented features are expected. Such kernels transform via the group representation.
Which in terms of the Lie algebra basis {A1} with A1 = 0 -1
gives vector with coefficient
a = 0 mod 2m. The B-spline basis, centered around each h; = (Ro,) E H with scale 8/. E R, as depicted in Fig. 3 is thus computed via
Log h 1.. ) = BR.a (0 0, mod 2n
BR."
8h
B.3 SCALING GROUP H (I x)
We call the positive real line R+ together with multiplication, the scaling group. The group product and inverse are given by
h h' (ss')
h (1)
with à = (s), (s') E H with s, s' E RR+. The identity element e = (1). The action of H on _d is given by scalar multiplication
h
with x E Rd. By combining 2D translations with scaling as semi-direct product group we obtain the scale-translation group, which denote with B2 x Rt. The left-regular representation of the scaling group are given by
(c)) (R (12*)) f)(x') = f(h-1 xx) = f(fx'), (c)) -I2(RR) P)()') F(h-1 .1) = F(E),
(c) -lg F)(())))x))) (1.))) F(b-1 o h- R1') F(fx' $)
18

---

Published as a conference paper at ICLR 2020
A descriptor terms of a collection of group elements $ I2 R+
h
h
A descriptor as function (convolution kernel) on IR? I K Ly(R2
R+)
*
x
K
5 -1, x
ckt - x
K
Plancr
Mellippiention bagorichene daatt
lags
h = 1.4
h 2.0
h 2.8
describe smiley face via collection of group elements C I22 x R=, which e.g. represents the locations and scale of low-level features such blobs/circles. Such collection of points transforms via left multiplication, e.g. h S := {(0, h) E S}. with h E R+ Bottom row: In a group convolutional setting we can work with convolution kernels which functions I2² x R+ that assign weights locations in which locally scaled features are expected. Such kernels transform via the group representation.
Figure 7: Top row: In group theoretical terms
The scaling of IR2 x R+ group convolution kernel is thus achieved by planar scaling and logarithmic shift along the scale axis. This is illustrated in Fig. 7 The determinant of the Jacobian of this action is det h| = sd. Using the above group structure we can write out the explicit forms for the lifting (Eq. (6 and group correlations (Eq. (7)) as follows:
(xx / 1 (x' xx))(x)
R² E -ifting:
Ju2
1 ( (x' od)x'd'ss
R² R+ correllati:::
(K+F)(x,s)
The logarithmic map on H is provided by the natural logarithm follows Log h In 8.
The B-spline basis, centered around each h; (st) H with scale s; R, as depicted in Fig. 3 is thus computed via
B.. Log h !
8; -1
B
B.4 THE ROTATION GROUP 50(33
The 3D rotation group is given by space of 3 orthogonal matrices with determinant 1, with the group product and inverse given by matrix product and matrix inverse:
h' = (R.R')
h = (R-1)-
The action of (O(3) 3R3 is given by matrix- multiplication
h R.x.
19

---

Published as a conference paper at ICLR 2020
with E R Combining the group structure of 3D translations with rotations in SO(3) semi- direct product group gives us the roto-translation group SE(3) = R x (O(3), also known as the 3D special Euclidean motion group. The left-regular representation are then
(c (L ()x) = f(h , x') = f(R (x'',
(c,, (00(3)) ((5000)) F)(h') F(h-1-h) -FR--.RR
(c (C5000))11((500)) () (h') =
-1)) = F F xx' R.,,'x,RRR... R').
The determinant of the Jacohian of the action of 50(3) on R23 is det = Using the above group structure can write out the explicit forms for the lifting (Eq. (6) and group correlations (Eq. 7 as follows:
SE(3)-lifti::
(k*f)(x, R)
A:R. x))f(x')xx'
JRa
J/z J JSoSS) K(R K(R-I(x''- x), R),,'. R. RR'F(x'' RR))
SB(3) coorreattt:::::n::
(K+F)(x, R.
with dR' denoting the Haar measure on 5O(3), which depends on the parameterization, see details below.
The logarithmic map from the group 50(3) to the Lie algebra so(3) is given by the matrix logarithm
and the resulting matrix can be expanded in basis (A1, A2, A3) for the Lie algebra
3
Log R Σ
a A;
i=1
with
)
0 0
A1
0
0 -1
A
0
0
0
A₁
which at the origin represent an infinitisimal rotation around the æ, y, and 2 axis respectively. A cardinal B-spline centered at R; SO(3) with scale 8h. can then be computed in terms of
these coefficients via B.... Logg R R
8h.
In practice it is often convenient to rely on a parameterization of the group and define the group structure in terms of these parameters. A common choice is to do this via ZYZ Buler angles via
Ra,3,7 Re.,,.Re R.,.s.RR. Re,.a
with Re, a rotation of around a reference axis e, and a E [0, 2m,, 8 e [], E 2m]. A Haar measure in terms of this parameterization then given by du(R) sin 3dadBdr. We will use this parameterization in construction of the quotient group S0(3)/SO(2) next.
B.5 THE 2-SPHERE H S2 2 80(3)/80(2)
Here we define the 2-sphere as group quotient of 50(3) and remark that the same equations as in the 50(3) case (App. B.4 are used for the lifting and group correlations. This section describes how to construct convolution kernels the sphere using the logarithmic map of SO(3), which can then be used to build G-CNNs on I&d x 52.
The 2-sphere is defined as 52 = [x E TR3 =J. Any point on the sphere can be obtained by R.z.
rotating a reference vector z (0, 0, 1)T with elements of $0(3), Yne. : In other words, the group S0(3) acts transitively on 52. ZYZ Euler angle parameterization of SO(3) all angles leave the reference vector in place, meaning that for each several R E SO(3) that map z to the same n. As such, we can treat S as the quotient group (0(3)) where S0(2) refers to the sub-group of rotations around the z-axis.
5 we have
20

---

Published as a conference paper at ICLR 2020
In order to define B-splines the 2-sphere need logarithmic map from point in S to the (Euclidean) tangent vector space Te()") ) at the origin. We will construct this logarithmic map using the Log define for SO(3). Let parameterize the sphere with
(53,1) 7) = Re,,y B
Any rotation B. with arbitrary a maps to the same (33,n) E S². As such, there are also many vectors A Log Ra, E (S)())) that map to suitable rotation matrix via the exponential map R exp A. We aim to find the vector in Tx(SO(3) for which c² O, which via the exponential map generate torsion free exponential curves. The Log of any Ra,3, with a -y results in such vector (Portegies al. 2015). As such define
Logga (33 7) = Logso(3) R
which maps any point in S to 2-dimensional vector space Te (S?) (e(())3)). A B-spline S can then be defined via
N f(3,7) - Σ BR.a
Loggs R.8.1 3.,3,7.2
(16)
8h
1=1
in which individual splines basis functions are centered around points n (13,, 75).
We remark that the group product R R_ a..,f.. RRu,3,y generates different rotations when varying ag that however still map to the same n. The vectors obtained by taking Logsa of the rotation matrices approximately isotropic neglect this effect
rotate with the choice for 0zg. Since the B-splines and simply set O;; = 0 in Eq. (16). Finally, we remark that the superposition of shifted splines (as in Eq. (16) is not isotropic by construction, which is desirable when using the spline kernel to lift functions to SO(3). When constraining G-CNNs to generate feature maps on Sa, the be isotropie. Alternatively on could stay on S entirely and resort to gauge-equivariant networks (Cohen et al. 2019). for which the proposed splines are highly suited move from the discrete setting (as in (Cohen et al. 2019) the continuous setting, see also For examples of splines on 5 see Figs. and
convolution
kernels are constrained
App. C.2
C RELATED WORK
C.1 DEEP SCALE-SPACS
C.I.1 SCALE SPACE LIFTING AND CORRELATIONS
In WWorrll& & Welling 2019) images (2(RR) are lifted to space of positions and scale param- eters by constructing scale space via
(x) fa(x) = (G. (x)
with (x( (4ms) 4/2 x
The kernels and images are sampled on a discrete grid. Let Ω C nd be the support of the kernel. Then the discrete scale space correlation is given by (Worrall
Welling 2019 Eq. (19))
(K*s (x) = Σ Σ
K(X, 3)fsa x),
REN H.
No. with Ha the discretized set of scales, e.g., Ha {2'-1) 1= where remark that here the convention of scaling of function by 8 instead of using the dilation parameter a in (Worrall & Welling 2019). Next we remark that the scale space corrllatio without any scale interaction (Ha = (1)) is defined by a 2D correlation kernel via
(k * (x)
Σ +*x+ XEZ4
which can be regarded as discrete atrous/dilated correlation on each of the scale slices fx(x with kernels dilated by factor
21

---

Published as a conference paper at ICLR 2020
C.1.2 RELATION TO LIFTING CORRELATIONS (EQ. (6) WITH B-SPLINES
Let our lifting correlation kernel be given in a B-spline basis via Eq. 11 and e Ω C 2 R be the map that assigns the weights to each B B-spline center x, E Ω with 52 the set of spline centers (i.e. c(x,) = c.). Let the Gaussian kernel Gx(x) be approximated by a scaled B-spline (up to a factor) and define B == + B xx). With such an approximation (see also (Bouma et al.
20077) our lifting correlation via Eq. 6 coincides with the lifting of (Worrall & Welling 2019
followed by their non-scale interacting scale- space correlation, i.e.,
(k+ (x, h) = (c*s (x, s).
with c: Ω + the spline coefficients. We show this by rewriting
(kxf)(x,)) h)) = / detn k(h
o (x - x))x(x)dx
/ Σ
c(x:) M ( x) (x)d(
Σ c(xi)
BBtt ((x - (x+ (x)))
XiE
Σ c(x:) (B.tt ff((x + sx,)
X,E
Σ c(x;)
X
C.2 GAUGE EQUIVARIANT NETWORKS
C.2.1 GAUGE EQUIVARIANT CORRELATION
The following highlights commonalities between this paper and the work by Cohen et al. (2019) with respect to of left-invariant vector fields in equivariant neural networks. Consider some Lie group with Lie algebra g = Te (G), the exponential map Exp g + G. and logarithmic map Log G g. Consider the group correlation between kernel and function K, F G - R, given in Eq. 7 which for B-spline kernels K with finite support Ω == supp(K) C reads as
(K F) = / K(g-1 F(F
(17)
with (h) the Haar measure on G, and where write K for the convolution kernel on G, and Kt for the corresponding kernel on g:
K(g) (Log(g)))
(18)
Let 5 == supp(k) C g be the support of K. Finally let Ω be localized such that Exp is a diffeo- morphism (i.e., Exp([)- l and Log(S2) Q).
Now consider the definition of gauge equivariant correlation on manifolds as given by Eq. (3) of
Cohen et al. 2019) for the case of scalar functions (in which case the trivial representation p(g) is to be used). In this case integration takes place over the Lie algebra, and gauge equivariant correlation is defined by
(KTF) = ( ((x)) (x))dxx
(19)
with dx the Lebesgue measure on Rd, and where Exp, denotes the exponential map from T2 - G. In our Lie group setting all tangent spaces can be identified with the tangent space at the origin (via the push-forward of feft-multillication) and we are able to write Exp,, g Expx. In the setting of gauge equivariant CNNs as in (Cohen et al., 2019) the exponential maps are generally dependent on g, using separate reference/gauge frame (basis for the tangent space) at each g.
22

---

Published as a conference paper at ICLR 2020
C.2.2 RELATION TO G-CORRELATIONS (EQ. WITH B-SPLINES
For Lie groups the following identity holds between group correlations with localized B-splines on the one hand, in which integration takes place over the group G and elements are mapped to the algebra via Log, and gauge equivariant correlation on the other hand, in which integration takes place on the tangent spaces and vectors in these tangent spaces are mapped to the manifolds via Exp. other words, given the definition of G cross-correlation in (7). denoted with x, and gauge correlation in 19) or (Cohen et al. 2019 Eq. (1)), denoted with x, the two operators relate via
(K * F) (K)
(20)
We show this by deriving
/ -))
(K x F)())
-
K())F(g .g)dg
R(x)F(g Exp(x))dx,
JQ
((X))
= (g)
In the above dğ is a Haar measure on G. At - the substitution g + 9- 9 is made and invariacee of the Haar measure is used (d(g g) dğ). At switch from integration the region Ω in the Lie group to integration over region ! = (og( in the Lie algebra.
D G-CNN ARCHITECTURES
This section describes the G-CNN architectures used the experiments of Sec. using the layers as defined in Sec. 3.22 and illustrated in Fig. Two slightly different architectures are in the two different tasks (metastasis classifiation and landmark detection), but both are regular sequential G- CNNs that start with lifting layer 6 followed by a several group correlation layers 7 possibly alternated with spatial max-pooling, followed by a projection over via (). and end with I convolution or fully connected layers. The architectures summarized Table. and 2 Note that the output of the PCam architecture is two probabilities (1 for each class), and the output of the CelebA is five heatmaps (1 for each landmark).
D. PCAM
The architecture for metastasis classification in the PCam dataset is given in Tab. [ The input (64 x 64) is first cropped
x 88 and is then used input for the first layer (the lifting layer). None of the layers use spatial padding such that the image is eventually cropped to size 1 x 1. Each layer is followed by batch normalization and ReLU activation function, except for the last layer (layer 7) which is followed by adding bias vector of length 2 and softmax.
Note that the first five layers, including max pooling over rotations, encode the image into a 64- dimensional rotation invariant feature vector. The final two layers (6 and 7) can be regarded classical neural network classifier.
To reduce possible orientation bias aim to approximate the support of the kernels with disk, rather than a rectangle. We do this by only considering splines with basis function centers {x, e zd ||x||| || < r, with radius r. For 5 x 5 kernels we set r = v5 by which we discard the basis functions the corners of the x 5 grid. The grid H is uniformly sampled with N,, samples, giving the discretized grid H. ((i 1)* 1=1
No. The global kernels (both dense and atrous) have
We apply batch normalization over the domain of the feature maps, over X = Rd over X Rd H, in (Cohen & Welling 2016))
23

---

The localized kernels Published as a conference paper at ICLR 2020
their centers also equidistant and globally cover S' i.e., h4 {(i 1)
2n No
Ti=1 with the scales of
the dense and atrous kernels respectively given by 3p No
2x and 8n
2x N₁
[Na/2] have their centers on the grid with h; E {i 2 No [Nc and have scale 8/1
2T No.
Finally, we follow the same data-augmentations These include geometric augmentations such as 90 rotations and horizontal flips, as well color augmentations such as brightness, saturation, hue and contrast variations.
at train time as proposed in (Liu et al. 2015
Table 1: PCam S2( G-CNN settings and the number of free parameters. Here N₂ denotes the number basis functions used on the H = S0(2) part of the group correlation kernels.
Basis size: Layer
N₂ 1
N₂ =3 3
N₂ =4
N =5 5 N₂ =
N₂ = 12 N₂- 16
Nr of output feature maps (# weights)
1: Lifting ts x
40 (2.520)
23 (1,429)
20 (1.200)
18 (1.134)
14 (882)
11 (093)
10 (630)
Spatial max pooling
by Iactor 2
G-corr (5 5 x Nw)
(33,600)
23 (38,227)
(33,600)
18 (34,020)
14 (32,928)
(30,492)
(33,60))
Spatial max pooling (2 2) by factor 2
G-corr (5 x NW
40 (33.600)
23 (33,327) (33,00)
18 (34,020)
14 (32,928)
(30.497)
10 (13,600)
Spatial max pooling (3 x 3) by a factor 3
4: G-corr (s Ni.)
40 (336000
23 (83.227)
20 (33,000
18 (34,020)
14 (32,925)
I1 (30,492)
10 (23,600)
G-corr
64 (2,800
64 (4,416)
64 (5.120)
64 (3,780)
64 (7,178)
64 (8,448)
64 (10,240)
Max pooling over H (Projection layer)
6: 2D-corr
16 (1.24)
16 (1,024)
16 (1,024)
16 (1,024)
16 (1,024)
16 (1,024)
16 (1.024)
7: 2D-corr
oz)
(30)
2 (30)
2 (32)
2 02)
2 (30)
(30)
Softmax
Total # weights: 106,936
106,902
108,236
110,010
107,890
101,673
112,726
D.2 CELEBA
The architecture for landmark detecion in the CelebA dataset is biven in Tab. 2 The input is format- ted according the details in Sec. In each layer zero padding is used in order to map the 128 x 128 input images to 128 x 128 output heatmaps. Fach layer is followed by batch normalization and a ReLU activation function, except for the last layer (layer 10) which is followed by adding bias vector and logistic sigmoid activation function.
Note that the result of the first layers, including average pooling over scale, assign locally scale- invariant feature vectors to each pixel. The final layers convert these feature maps into heatmaps via regular 2D convolutions.
Landmarks are localized via the argmax on each heatmap. The results in Fig. 5 show the success rate for localizing a landmark correctly. The success rate is computed as the average fraction of successful detections for all five landmarks in all images. A detection is considered successful if the distance to the actual landmark is less then 10 pixels.
The H axis is uniformly sampled (w..t. to the metric H) on fixed scale range, generating the discrete sets Ha {e(t- y with & 1 2. The global kernels have their centers on this grid, i.e., h4 E Ha with the scale parameter the same as that of the grid. The local kernels also have their centers equidistant (with scale 8n) eachother, but localized and given by h; fers [2]
1N,
24

---

Published as a conference paper at ICLR 2020
Table 2: CelebA scale-transe translatonnnt G-CNN settings and the number of free parameters. Here No
denotes the number basis functions used on the (R+, x) part of the group correlation kernels.
Basis size: Layer
N₂
N2
No 3
No
N₂
Nr of output feature maps (# weights)
Lifting (5 5)
27 (2025)
21 (1,77)
17 (1,25)
15 (1,12)
(1,050)
2: G-corr to x.x. Ni.
27 (18,225)
21 (22.00))
17 (21,655)
15 (22,500)
14 (24,500)
3: G-corr co N4)
27 (18,225)
21 (22050)
17 (21,75))
15 (22,500)
14 (24.500)
Average pooling
H (Projection layer)
Spatial max pooling (2 2) by factor
4: Lifting (5
27 (18.225)
21 (11.225)
17 (7.225)
15 (5,625)
14 (4,900)
G-corr
Ni)
27 (18.225)
21 (22.050)
17 (21665))
15 (22,500)
14 (24,500)
G-corr (5 x No.
(18.225)
21 (22.000)
(1.675))
15 (22.500)
14 (24,500)
Average pooling over H (Projection layer)
Up-sampling by factor 2
2D-corr 3)
32 (15,522)
32 (12.06)
32 (9,792)
32 (8.640)
32 (8.064)
2D-corr 3)
(9.216)
(9216)
32 (92216)
32 (9226))
32 (9.216)
2D-cor (1 1)
64 (2048)
64 (2018)
64 (2018)
64 (2018)
64 (2018)
10: 2D 20rrorr
5 (320)
(320)
(320)
5 (320)
5 (320)
Logistic sigmoid
Total # weights:
120,286
124,480
116,576
116,974
123,589
25


Reviews:
[{"review_id": "SkgVdEbAFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, a framework for building group CNN with an arbitrary Lie group G is proposed. Generally, such a group CNN consists of 3 types of layers: a lifting layer which lifts a 2D image to a 3D data (G-image) whose domain is G; a group correlation layer which computes a 3D G-image from a 3D G-image; and a projection layer from a 3D G-image to a 2D image. To implement the convolutions in the lifting layer and group correlation layer which are defined in the continuous setting, the B-Spline basis functions are applied to expand the convolution kernels. Experimental results on tumor clarification and landmark localization show the superiority over CNN.\n\nAdvantages:\n1. A flexible framework for group convolutional neural network is proposed with strong theoretical support in Theorem 1.\n2. Familiar properties of convolutions from classical CNN design (like localized, atrous, and deformable convolutions) can also be implemented in G-CNN using specified B-Spline basis functions.\n3. In comparison with standard CNN, the effectiveness of the B-Spline-based G-CNN is validated through experiments on two typical data sets. \n\nWeakness:\n1. [Readability] For readers who are not familiar with Lie groups, this paper is very hard to follow. \n(1)\tFor Theorem 1, the authors are suggested to give some illustrative explanation. Besides, what is “Stab_G”? \n(2)\tThe architecture of G-CNN, i.e., the 3 types of layers, are directly given in Eqs. (5)-(7) without examples, illustrative examinations, or visual illustrations.\n(3)\tFig. 1 can be modified for better readability. \n \n2. [Experiments] The proposed G-CNN has some similarities with data augmentation (like rotation, scaling) based CNN. Then, how better can the G-CNN perform than CNN with data augmentation? More experiments on this point are suggested, and relevant theoretical explanations will be appreciated.  \n\n3. [Implementation] Considering the complicated mathematics in this paper, I am afraid that implementation of the proposed G-CNN is also very hard. It would be better for the authors to discuss the implementation. In my mind, if the implementation is not so hard, then the formulation of G-CNN can also be simplified for better readability. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyeS9jyxqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an (approximately) equivariant neural network architecture for data lying on homogeneous spaces of Lie groups. In contrast to the Gauge equivariant and Fourier approaches that have recently appeared, here the authors simply put a B-spline basis on local patches of the homogeneous space and move the basis elements around explicitly by applying the group action. \n\nThe approach is appealing in its simplicity and generality. No need to worry about irreducible representations and Fourier transforms, the formalism works for virtually any Lie group, no problem with non-compact groups. However, there is a constant need for interpolation. What is more more significant is that both the homogeneous space and the group need to be discretized and in general that cannot be done in a regular manner (no notion of a uniform grid on SO(3) for example). The authors assure us that \"we find that it is possible to find approximately uniform B-splines... e.g. by using a repulsion model\". I am not sure that it is so simple. This is one of those things where the idea is straightforward but the devil is in the details.\n\nTheorem 1 seems important but it is a bit cryptic. What is the statement \"a kernel satisfying such and such properties gives rise to an equivariant CNN\"? Or \"A CNN is equivariant if and only the kernel satisfies such and such properties\"?\n\nConcerningly, the paper is closely related to a few other papers using the spline CNN idea or at least the idea of taking a fixed set of functions and moving it around on the homogeneous space by acting on it with select group elements, most notably \"Roto-translational convolutional neural networks for medical image analysis\" by Bekkers et al.. The main difference of the present paper relative to that one is that the idea is fleshed out in a little more detail and is generalized from SE(2) to arbitrary Lie groups. However, conceptually there is little that is new.\n\nIn such a situation it would be important to present convincing experiments. Unfortunately in the present paper, results are only presented on 2 datasets, and the algorithm is basically only compared to different versions of itself, rather than state of the art competitors.\n\nThe paper is clearly written but the intuitive nature of the core ideas could be better conveyed e.g. by fancy diagrams.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgI5GfF9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural network architecture which that enables the implementation of group convolutional neural networks for arbitrary Lie groups. This lifts a significant limitation of such models which were previously confined to discrete or continuous compact groups due to tractability issues. \nI'm afraid that this paper is over my head. It relies heavily on field-specific terminology and as such is likely to be accessible to a relatively small subset of researchers. This looks to me like a solid contribution, however I'm really not qualified to judge.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes principles for endowing a neural architecture with invariance with respect to a Lie group. The contribution is that these principles can accommodate discrete and continuous groups, through approximation via a base family (B-splines). 

The main criticisms were related to the intelligibility of the paper and the practicality of the approach, implementation-wise. Significant improvements have been done and the paper has been partially rewritten during the rebuttal period.

Other criticisms were related to the efficiency of the approach, regarding how the property of invariance holds under the approximations done. These comments were addressed in the rebuttal and the empirical comparison with data augmentation also supports the merits of the approach.

This leads me to recommend acceptance. I urge the authors to extend the description and discussion about the experimental validation. 

Decision: Accept (Poster)