Gauge Equivariant Spherical CNNs
Authors: Berkay Kicanaoglu
Pim de Haan
Taco Cohen
Venue:  (2020)
Keywords: deep learning
convolutional networks
equivariance
gauge equivariance
symmetry
geometric deep learning
manifold convolution

Abstract: Spherical CNNs are convolutional neural networks that can process signals on the sphere, such as global climate and weather patterns or omnidirectional images. Over the last few years, a number of spherical convolution methods have been proposed, based on generalized spherical FFTs, graph convolutions, and other ideas. However, none of these methods is simultaneously equivariant to 3D rotations, able to detect anisotropic patterns, computationally efficient, agnostic to the type of sample grid used, and able to deal with signals defined on only a part of the sphere. To address these limitations, we introduce the Gauge Equivariant Spherical CNN. Our method is based on the recently proposed theory of Gauge Equivariant CNNs, which is in principle applicable to signals on any manifold, and which can be computed on any set of local charts covering all of the manifold or only part of it. In this paper we show how this method can be implemented efficiently for the sphere, and show that the resulting method is fast, numerically accurate, and achieves good results on the widely used benchmark problems of climate pattern segmentation and omnidirectional semantic segmentation.

Under review as a conference paper at ICLR 2020
GAUGE EQUIVARIANT SPHERICAL CNNS
Anonymous authors
Paper under double- blind review
ABSTRACT
Spherical CNNs are convolutional neural networks that can process signals on the sphere, such as global climate and weather patterns or omnidirectional images. Over the last few years, a number of spherical convolution methods have been proposed, based on generalized spherical FFTs, graph convolutions, and other ideas. However, none of these methods is simultaneously equivariant to 3D ro- tations, able to detect anisotropic patterns, computationally efficient, agnostic the type of sample grid used, and able to deal with signals defined on only a part of the sphere. To address these limitations, Spherical CNN. Our method is based on the recently proposed theory of Gauge Equivariant CNNs, which is in principle applicable to signals on any manifold. and which can be computed on any set of local charts covering all of the manifold or only part of it. In this paper we show how this method can be implemented efficiently for the sphere, and show that the resulting method is fast, numerically accurate, and achieves good results on the widely used benchmark problems of climate pattern segmentation and omnidirectional semantic segmentation.
introduce the Gauge Equivariant
INTRODUCTION
In many disciplines of science and engineering, spherical signals emerge naturally. In the earth and climate sciences, globally distributed sensor arrays collect measurements like temperature, pres- sure, wind directions, and many other variables. Cosmologists are interested in identifying physical model parameters from real and simulated cosmic microwave background measurements sampled on spherical sky maps. In robotics, especially in applications like SL.AM and visual odometry, om- nidirectional and fish-eye cameras are widely used. Thus, it is clear that efficient CNNs that can directly operate on spherical signals are necessary.
When designing Spherical CNNs for these applications, there are number of practical consider- ations that need to taken into account. Firstly, whereas planar images are always sampled on square grid, there is wide variety of spherical grids, and the choice of sampling grid is usu- ally dictated by hardware or task-dependent requirements. Secondly, several applications require the analysis of very high resolution spherical maps. Most existing algorithms cannot handle this scale. Thirdly, in some cases data is only available on a part of the sphere. Fourthly, whereas several graph-based and spectral methods can only learn isotropic blob-like filters, some problems require the detection of anisotropic patterns. Finally, in almost all cases, equivariance to rotations is a sought-after property since results should never depend on how we orient the sphere.
The method we propose in this paper is based on the recently proposed theory of Gauge CNNs (Co- hen et al., 2019). This framework has several appealing properties: (i) it provides a principled way to achieve spatial and rotational weight-sharing on the sphere, (ii) it naturally allows for processing geometric features such as scalar, vector and tensor fields in an anisotropic and equivariant manner, and (iii) it makes it possible in principle to process a signal on overlapping local charts, in case data is not available for the whole sphere or when the whole signal does not fit in memory. To the best of our knowledge, we present the first implementation of a Gauge CNNs curvature. We describe an efficient implementation which is, in principle, applicable to any grid on the sphere. Experiments spherical segmentation of climate patterns and omnidirectional images demonstrate the effectiveness of our method. We show that our method scalable and fast while being numerically accurate as the best existing Spherical CNNs.
manifold with non-trivial

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
Conventional CNNs for Spherical Images. A line of prior work (Su & Grauman 2017 Coors
jet al. (2018) considers using planar CNNs on intrinsically spherical tasks such as omnidirectional image segmentation. These approaches represent the spherical data on R² via equirectangular pro- jection which introduces position-dependent distortion rates due to curvature. In order to counteract the varying distortion rates, [Su & Grauman (2017) uses kernels with increasing support towards the poles thus limiting parameter-sharing to only longitudes. Contrastingly, Coors et al. (2018) use fixed-sizer kernels sampled on the tangent plane spherical signal for better parameter-sharing and distortion invariance. Both methods assume preferred orientation and are not SO(3) equivariant.
Another line of work operates directly on S alleviating the issues related to distortions caused by projection. [Boomsma& & Frellsen (2017) proposes spherical convolutions for molecular modeling both volumetric and mesh representations. Peerraudin et al. (2018) proposes graph-based spherical convolution for cosmological model classification. A subset of this line (Kondor et al. 2018 Cohen et al., 2018 Esteves et al. 2018) extends group equivariance (Cohen & Welling (201)))((0166) over SO(3). Esteves et al. (2018) uses spherical convolutions computed by spherical FFT's. Similarly, Cohen et al. (018) proposes a spherical correlation operation based on the (3)- FFT. As result both methods have inbuilt rotation equivariance property. However, the Generalized FFT algorithms used only work on inhomogeneous grids that over-sample the poles, and in practice the FFT-based methods are slow.
Spherical CNNs.
Recently, a number of spherical CNNs that parametrize the sphere with icosahedral tiling have been proposed (Jiang al. [019 /Zhng al. 2019 Liu al., 2019 Cohen et al.] 2019). These methods are tailored for the icosphere grid since this grid is quite regular and its charts can be easily mapped be used. Out of these methods, Gauge Equivariant Icosahedral CNN of Cohen et al. (019) is the most similar to our method. However, their model operates on the icosahedron, which is only an approximation of the sphere, while ours operates on the sphere. Even though their method is fast and accurate, it is only equívariant to discrete icosahedral symmetries, while our method is fully SQuiieequivariant. In comparison to all methods in this category, our method is agnostic to sampling grid used and better suited for larger scale problems.
on IR? allowing highly optimized convolution routines
Geometric Deep Learning. There have been number of attempts generalize convolution oper- ator to manifolds in the geometric deep learning field (Bronstein et al. 2017). The key issue with manifold convolution is the lack of globally consistent reference frames attached to points over manifold. Therefore, unlike shifting a filter over a flat image grid with a clear sense of up/down and left/rghht, is not clear how place the convolution kernel. To overcome this issue, [Bruna et al. 2014); Boscaini et al. | (2015) have used isotropic filters at the expense of kernel expressivity. Masci (2015) have applied filters in fixed number of orientations and accumulated the responses via max pooling, losing the orientation information. Recently proposed Gauge Equivariant CNNs (Cohen et al., 2019) have shown that is possible have both expressive kernels and orientation formation by allowing the network response change equivariantly with respect to arbitrarily chosen
local reference frames (i.e. gauges).
3 CONTINUOUS THEORY OF GAUGE CNNS
In this section we will review the mathematical theory of gauge CNNs general manifolds, as presented in Cohen et al. (2019). Specifically, we will define mathematical model of feature spaces as fields, and show how one can define convolution-like operation that only makes use of the intrinsic structure of the manifold.
3.1 GEOMETRICAL FEATURES & GAUGE TRANSFORMATIONNS
The feature spaces in Gauge CNNs are modelled fields f over manifold M. For example, the input data could be a vector field of wind directions on earth, or a scalar field of intensity values on the plane (a grayscale image), (scalars, vectors, tensors, and others) as geometrical features and speak of a field of geometrical features geometrical feature field. Even if the input data consists only of scalars, might
field of diffusion tensors on R We will refer to such quantities
2

---

Under review as a conference paper at ICLR 2020
want to other kinds of fields for the internal representation learned by the network, will describe Gauge CNNs for general fields. Because we are primarily interested in the 2-sphere S we will specialize to the case of d dimensional Riemannian manifolds.
In computer science it is common to think of a vector or tensor as a list or array of num- bers, but from physical mathematical perspective these geometrical quantities that ist independent of a coordinatization choice of basis. To represent a geometrical feature nu- merically however,
need to choose a frame for the tangent space T,M at each position E M. A smooth choice of frame is also known as a gauge. Mathematically, a gauge can be defined as a smoothly parameterized set of linear maps up R TyM (see Fig.
Since the choice of gauge is arbitrary, we should consider what happens to the coefficients of geomet- ric features if we change it (i.e. apply a gauge trans- formation). Consider first the coefficients [p) of a tangent vector V in the tangent space T,M at p E M, expressed as pair of numbers = (U1, U2)
T.M
exp, V
Wy
relative to an orthogonal frame (wp(e1). (e2))) at
Figure 1: The exponential map and the gauge
If we rotate the frame at p by r SO(2), i.c.
"p : R2 + T,M. The exponential map
Wpp + Wip r, then the coefficient vector trans- forms as v + r.g. The vector itself an abstract geometrical quantity, invariant to gauge transforma- Wp
takes a tangent vector V E M and follows the geodesic starting at p with speed [ for one unit of time, arrive at qu exp, p V
tions: V = (wpr)r
M. Figure courtesy of Cohen et al. (2019).
Note that we free change the gauge not just at one point, but at all positions simultancously in an arbitrary (smooth) way. However, since we will only want to work with right- right-handed orthogonal frames, we only need to consider rotations of the frames. Thus, for our purposes we can define gauge transformation as a smoothly varying choice of rotation rp E SO(2).
Beyond scalars (which are invariant to gauge transformations) and vectors (which transform like ('v) rj (p)). we will want to consider more general kinds of geometrical features. A (2, 0)- tensor, for instance, is (a linear combination of) tensor products V & W of vectors V, W T,M. Given a frame, such tensor is represented as d x d matrix. Under a change of frame, matrix f() will transform like fp) + rp)(p)rpp We can also flatten the matrix into a de-dimensional coordinate vector fp),, and write the transformation as f(p) + (r, @ fp), where Ty @ rp the Kronecker product.
The tensor product p(r) = r & r an example of group representation. This is map p G - GL(C, I)) taking each element r of G' (the rotation group SO(2) in our case) to an invertible matrix p(r) that acts on an C-dimensional feature vector. To be called representation, it has to satisfy p(rr') ((r),(r') which is easily checked for the tensor Kronecker product.
Thus, we can generalize to geometric feature fields that transform like fp) + p(r under gauge transformations, for any group representation p SO(2). We will refer to such fields as p-field or field of type p. In gauge equivariant CNN, one chooses for each feature space of the network such representation p that determines the kind of features learned by that layer. The network is constructed such that a gauge transformation applied to the input will result in corresponding gauge transformation each feature space. The dimension C' of p is equal to what is normally called the number of channels in the feature space. Typically, one would choose to be Block-diagone containing for instance a number scalar fields (1 x blocks p:(r) = 1), a number of vector fields, etc. The number of copies of each type of feature is called its multiplicity.
3.2 GAUGE EQUIVARIANT CONVOLUTION
For each layer of the network, want to interpret both the input and output as fields of geometrical features. If we apply gauge transformation, the input coefficients change (f(p) + p(r=') fp)),, and we want the same to happen the output (gauge equivariance), as the coefficients of a geometrical quantity relative to a gauge. In this section we will define a convolution-like operation that has this property.
that we may interpret it
3

---

Under review as a conference paper at ICLR 2020
The classical convolution operation involves summing integrating the product of filter and the input signal over a local region. If we want to generalize this to fields on a manifold, we run into difficulty: the geometric feature f(p) and (q) at different points p and q in M live in different vector space, and so we cannot directly compare them or add them up. For instance, we have two tangent vectors V E T.M and W T,M, how can we say that they "the same" or add them up? Having chosen frame for M and TyM we could add the coordinate vectors R and E R2a but because we can change the gauge of both tangent spaces independently, the result is not the coefficient vector of any invariant geometrical quantity.
The solution is to apply parallel transport to the feature vectors before adding them up. Given curve from @ to p. we can transport a vector W E TrM to T,. by applying a rotation T Tp-q E S0( to its coefficient vector w. Since we can interpret pp--ql1 well-defined. In general, for other kinds of geometrical features, parallel transport acts via p. i.e. add + p(T )w. We will use this in the definition of the gauge equivariant convolution.
vector in T,M. the quantity u-
Following Masci et al. (2015). we parameterize a local neighborhood around p E M by the tangent plane T, ~ -² via exponential map (see Fig. []. That is, we index nearby points q by tangent vectors using the exponential map, by defining qu exp, Wpt for E I2 ("Riemannian normal coordinates"). The convolution is then defined by transporting for each nearby point qo the feature vector f(q)) to p by computing p(rptg.) (f()). transforming the resulting features at p using learned kernel K R2 RCa xCr, and integrating the result over the support of K in R2:
* f(p) = [
(1)
As shown by Cohen et al. 20119) this operation is gauge equivariant if and only if ((v) satisfies
K(r-te) P (r (()))(())
In section 4.2 show how we can parameterize such kernel via rotational weight- sharing.
3.3 EQUIVARIANCE TO SO(3)
In addition to gauge equivariance, SO(3) equivariance is desirable property for a Spherical CNN (Cohen et al. 2018). This means that if we apply a 3D rotation to the input of the network, the output also rotated. In this section we show that the gauge equivariant convolution defined is also equivariant to S3))..
Consider a local patch on the sphere (e.g. the support of the kernel), and the signal defined there. When we rotate the sphere, the patch is moved to another place, and it may change its orientation. Moving the patch is not problem: at the new position we apply the same kernel K, so one expects that the convolution result at the new position equals the convolution result of the original signal at the old position. However, since the orientation of the kernel determined by the gauge (which is arbitrary but fixed) and because we can arbitrarily change the orientation of the patch by rotating around its center, the kernel and the patch may be matched in a different relative orientation after applying the rotation. Fortunately, because the kernel satisfies Eq. 2 the result will be equivalent up to gauge transformation acting by Pou., and so we have SO(3) equivariance.
Thus, in the continuous theory, the gauge equivariant convolution is also SO(3) equivariant. How- ever, as we will see, making sure that this holds in discrete implementation is not entirely trivial.
4 DISCRETE IMPLEMENTATION OF SPHERICAL GAUGE CNNS
The theory covered so far tells how, mathematically, we can define convolution-like operation that is gauge cquivariant. However, does not tell us exactly how to implement it on a computer, in order process discretely sampled signals on manifold. How do this exactly can have large effect on the efficiency and numerical accuracy of the method.
list of values f. f(.)) associated with finite number of points P: E V C S (see Sec. (..).. We assume that the kernel K(v) has local support, so K(v) = whenever ||ы||| R for some radius R. Equivalently, we can say that g E S only contributes the
A signal is represented
4

---

Under review as a conference paper at ICLR 2020
convolution result p S if the geodesic distance between p and q is smaller than R. Accordingly, we define the set of neighbors N (p) of p as the set of points q within radius R from p.
A simple way of discretizing the gauge convolution (Eq. is to replace the integral over R2 (iden- tified with T) by a sum over neighbors of p. Each neighbor can be associated with a tangent vector via the logarithmic map: Upg logp 9- This yields the following approximation:
+ * f(p) = Σ
((
(3)
(EN(p)
At this point is worth comparing Eq. to the simplest form of message message-passing based graph convolution (Kipf & Welling. 2017; Gilmer et al. 2017). In both cases, the result of convolution is computed as a sum of messages comming from neighhors. In the case of graph CNNs, these messages are computed Af(q), where A weight matrix that is shared by all neighbors. So graph convolutions use isotropic filters that cannot distinguish where the message comes from.
The gauge convolution sums messages of the form K (vpp)) (tm (f f(g.. Thus, the feature vectors (q) of neighbors q are transformed in way that depends i) on the intrinsic geometry of the man- learnable kernel K ((ppg) We can thus think of gauge convolutions (implemented in this way) as more powerful version of graph convolution that leverages the additional topological and metric structure of a Riemannian manifold to process geometrical data in a more flexible manner.
ifold via rp-g and Ppg. and ii) by non-isotropic (but gguge-qquivarian
The discrete gauge convolution is computed in a few steps, some of which are done during pre- computation and some during the forward pass: i) the logarithmic map Upy logp q, ii) the parallel transporter rp-qq iii) the construction parameterization of the kernel K(v), iv) the linear contrac- tion of the kernel and the signal.
Steps (i) and (ii) can be complicated for general manifold or mesh, but for the sphere are easily and exactly computable in closed form with standard operations, which are detailed in Appendix B We will discuss our method for selecting the finite number of points on the sphere, as well as steps (iii) and (iv) in the following sections.
4.1 THE ICOSPHERE GRID
Even though our method uses the exact geometry of the sphere, a grid of points on the sphere must nevertheless be chosen on which the features will live. The sphere does not admit perfectly symmet- rical and homogeneous high resolution grids. grid that is fairly homogeneous and has been used successfully in Spherical CNNs before is what we call the icosphere grid (Jiang et al., 2019 Liu eet al. 2019. Cohen et al., 2019). The icosphere grid can be puted at different levels of resolution. The lowest resolution & = 0 has as points p; the 12 corners of the icosahedron. Higher resolu- tions are obtained by repeated subdivision of the triangular faces of the icosahedron into 4 sub-triangles, followed by a projection of all points to the sphere. The result is grid H. with 5 x 22++1 + 2 points subdivision level s. We emphasize that our method is not in any way tailored to this grid, and other options such as HEALPix (Görski et al. 2005) could easily be substituted.
Icosphere with The orange points are the 2-ring neighbours of the red point. The green arrow is the logarithmic map corre- sponding to the yellow path.
Figure 2:
8
4.2 PARAMETERIZATION OF THE KERNEL
The kernel ((v) is defined as a continuous matrix-valued function of R2 that satisfies the kernel constraint (Eq. 2 In a classical CNN, where we are dealing with a homogeneous grid of pixels in 2 we can define a small (e.g. 3 x 3) set of neighboring pixels Npp) = {p + p(0), so that we only ever need to evaluate the kernel at a small number (e.g. 9) of points v(0) This results in a parameterization of K an array with C.n. x Cin x 3 3 learnable coefficients.
On the sphere there are no perfectly homogeneous grids, so depending on the point E V where are evaluating the convolution f. the neighhorhood structure N A(p) may look quite different.
5

---

Under review as a conference paper at ICLR 2020
Hence, the points Upg E R where need to evaluate K will differ as well. For this reason, parameterize K as a linear combination of analytically determined continuous basis kernels. The linear coefficients will be learned.
We assume that Pm and Poul are block-diagonal with irreducible representations (irreps) as blocks (any SO(2) representation takes on a block structure as well, with each block corresponding to a particular input/output irrep (Cohen & Welling 2017), with irreps labelled by integer frequency n 2 0 (Worrall al.) (017) (see Appendix D). So we will focus on the case where both input and output representation consist of a single irrep. and construct the full kerne block-wise as described in Appendix
be brought this form by a change of basis). In this case the kernel
As derived in Appendix D the analytical solutions to Eq. 2 can be split in a independent radial part and angular part. The solutions for the angular part K (0) are shown Table [] while the radial part is unconstrained. So if a set of radial functions { (r)) are chosen, and { {K)(0)} is the complete
set of angular solutions, the parameterized kernel is: K ((r 0) = Eab Wab (( ((())) for weights w. Hereafter, we denote one such solution as K,, so that the parameterized kernel is >; W;K, The number basis-kernels is called num fau_basis..
Since the geometry and grid are fixed, we can precompute the basis kernels evalu- ated at all required points. That is, for each P E V and g E (N(p) we evaluate each basis kernel contracted with the input rep- resentation (1ppp) Ptmlpg)pPmnrr (Tp(pp-g)) where Tpt-g and Ung are as computed in Section B. and B.2 The result of this precompula- tion is an array of shape num basis x num v x num mei X c_out X c in, where e in and c_out are the dimen- sionality of Pm and Pout and also the num- ber of channels of the input and output sig-
Pn - Pon
Linearly independent solations for K()))
Ao + Pm
(cossD
sin sof), cos mi sin sin
(sin ull
- -cossn))
slamp mi)
Pu Pm
C+
8+ (-s
Pu + Pm
Table 1: Solutions to the angular kernel constraint for kernels that map from Pn to Pm- We denote
c+ = cos(m + n)0, $= sin(m + n00.
nals.
4.3 COMPUTING THE CONVOLUTION
Having computed the basis kernels at each Upp. we can compute the discretized gauge convolution (Eq. 3 as linear contraction. This is done in two steps. Initially, we expand the signal fp)). which in, to f of shape num_ num neigh x c_in. This is done so that fm is the value of the signal at the g-th neighbor of p.
has shape num
Subsequently, we contract the signal f with basis kernels K2(upa) Pm (Fpe-g and weights wy to obtain the convolution result & f of shape num v c_out. Since a basis-kernel K; only acts on one in/out irrep pair. it is mostly zero. In Appendix A we detail how this block-sparsity can get exploited for computational efficiency. We note that can easily be computed for subset V V.
4.4 NONLINEARITIES
For the network to be gauge equivariant, every layer should be gauge equivariant, including non- linearities. Trrep features do not commute with pointwise nonlinearities (Worrall et al., 2017 Thomas et al. 2018 Weiler et al. 2018 Kondor et al., 2018). However, we can perform a basis transtormation to a basis which pointwise non lineanties are approximately gauge equivariant. Afterwards, we transform the basis back to the irreps.
For simplicity, we assume that the representation is U copies of SedDpr...@@P... One such copy can be treated as the discrete Fourier modes of circular signal with band limit M. An inverse Discrete Fourier Transform (DFT) matrix can map these modes to N spatial samples. Under a gauge trans- formation of multiple of 2T/ /N, the samples are cyclically shifted. The resulting representation can thus be called a regular representation and hence our procedure a ReqularNonlinedde these samples, such as the RELU, commute with such gauge
Non lincarities that act pointwise transformations. The procedure is however only approximately gauge equivariant under gauge trans- formations of angles that not multiples of 2T. /N. Nevertheless, we prove in Appendix E that in

---

(b) For each quadrature point I (green), the signal interpolated from the neighhors Ap)) (red). The convolu- tion is equal to the sum over quadrature Under review as a conference paper at ICLR 2020
interpdatiinn
Rutation of quheee
p
T,M
T
(a) Any spherical grid is irregular. For example, when the the sphere rotated, point p. with nearby neighbors, can be mapped to a grid point p', which has 6 neighhors. As a result, the naive method may fail to be rotation equivariant.
points, but only sum over neighbors during the forward pass.
Figure 3: Quadrature Integration
the limit N + oo, exact cquivariance is restored and we provide a finite error bound, which can help selecting N.
QUADRATURE INTEGRATION
The implementation described in the previous section satisfies gauge equivariance exactly. In ad- dition to gauge symmetry, the sphere has global rotational SO(3) symmetry, and like previous methods would like our method to be equivariant to 3D rotations well. In the continuous case, gauge equivariance implies SO(3) equivariance, but the scheme above not exactly SO(3) cquivariant due to discretization (see FigureBa)) In this section describe numerical method that improves S3(3) cquivariance at no additional computational cost during training.
Our general approach for computing to obtain a continuous function on IR2, and then use quadrature integration to get more precise value for the integral. Quadrature a general numerical technique for approximating integrals with finite sums. For some region A and function g. the integral JA g(r)dr can be approximated by Exer (x) where I C A is some finite set of quadrature points, each with a weight Wg. The goal is selecting and wい such that the approximation is accurate (or even exact), for functions satisfying some regularity assumptions (e.g. being band-limited). In our case, the region A is disk with as radius the support radius of the kernel. The quadrature rule described in Appendix C
f(p) will be interpolate the sample values at (p)))
The signal at E I is inferred from the signals at (p) by interpolation:
[a()) = Z(p,c)
Σ
k(c, (e.g))aa(pp--)))0
(4)
gEN(p)
where k(c, q) = exp (-Ile- log, //2) is Gaussian kernel with scale a, measuring distance between and in the tangent space, and (pp, c) = EmeN( k(c, q) is a normalizing constant.
We compute the integral over R2 by quadrature integration:
f(p) [w.K(e)ib())
(5)
CEI
The convolution Eq. sums homogenized neighborhood and is thus more equivariant to rotations of the sphere. See FigureBp for an illustration. Equivariance improves if large number of quadrature points used, which increases the computational cost. Luckily, since the composition

---

Under review as a conference paper at ICLR 2020
of linear operations is linear, we simplify:
s f(p) = Σ (cc(c)f = E Wekile, 9) (()) E K(p.g)f(g) gEN(p)
(EN(p) ((p,))
CETD
Z(p.c) (pp)g)) The new kernel K can be pre-computed once, that the convolution during run-time involves only sum over the neighbors, just as in the naive convolution (Eq, B). The interpolation thus does not affect computational cost.
for new kernel - - wekle.a)
EXPERIMENTS
In this section, we present experimental results on three benchmark problems. Throughout this sec- tion, we will refer the gauge equivariant convolutions described in 4.3 IrrepConvolution and similarly refer to tailored batch normalization and nonlinearities as IrrepBat chNorm (nor- malizing irreps instead of channels) and Regul rNon inearit ty as we name in implementa- tion. An I rreeConvol ut on and other operations are defined by specifying the number of input and output multiplicities for each irrep order, in analogy number of channels in regular CNNs. In addition, we need to specify the types of irreps by their angular frequency. We do that by max-freq argument which determines the cutoff frequency. We will describe p by parameter max f freq and multiplicity, and include each frequency O,..., w....mma..... freq with the same multiplicity.
6.1 SPHERICAL MNIST
We perform series of experiments on a toy dataset to validate the equivariance and generalization properties of our gauge-equivariant spherical CNN. Specifically, we compare our method with dif- ferent filter sizes (I-ring. 2-ring) with and without interpolation, to three prior methods, as well as an isotropic baseline. The isotropic baseline uses only scalar features and isotropic kernels, but is similar number parameters to our model.
We follow setup similar to that of Cohen et al. (2019) and generate Spherical MNIST in randomly Rotated (R) and Non-rotated (N) conditions. For all experiments, we represent the signals on an icospherical grid at level 4. For network architecture and training procedure, see Appendix F.I
We present our results in Table 2 on three configurations of the Spherical MNIST dataset: N/N, N/R and R/R, where X/Y means train on X and test Y. In addition, to test the effect of quadrature integration on equivariance. Whether interpolation used or not, method performs similarly to relevant spherical CNN baselines in terms of classification score on N/N. We observe that our method automatically generalizes to 3D rotations without data augmentation and thus compares favorably against IcoCNN in the N/R condition. However, in the ablation study, we observe that our method's 's generalization capability against 3D rotations drops in the absence of interpolation, particularly when smaller (1-ring) filters are used. This result shows the usefulness of quadrature integration when using small filters.
ablation study
Our model outperforms FFS2CNN in all categories. Furthermore, performs better than the isotropic baseline, which illustrates the importance of anisotropic filters and non-scalar features for the MNIST dataset. This to expected, as detection of line direction naturally expressed in anisotropic filters.
As a final note, our results confirm the theoretical result that gauge equivariance implies SO(3)- equivariance (Sec. 3.3).
6.2 CLIMATE PATTERN SEGMENTATION
We put our model to the test on a climate pattern segmentation dataset proposed by Mudigonda et al. 2017). The dataset is collection of simulated climate variables over 20 years from the Community Atmosphere Model CAM-5.0 (Neale et al.) 2010). The task to segment out regions of Atmospheric
Rivers (AR) and Tropical Cyclones (TC) from background (BG). We the preprocessed data
Results from Cohen al. 2019
8

---

the more relevant Under review as a conference paper at ICLR 2020
Method
N/N
N/R
R/R
Time epoch
Complexity
S2CNN (Cohen et al. 2018
99.38
99.38
99.12
380s
ON log N)
IcoCNN Cohen et al. 2019
99.43
69.99
99.31
72s
((N)
FFS2CNN (Kondor et al. 2018
96.4
96
96.6
((N)
Isotropic (2-ring, interpolation)
98.27
95.31
97.18
87s
O(N)
Ours (1-ring, interpolation)
99.47
97.65
99.24
94s
O(N)
Ours (2-ring, interpolation)
99.51
99.32
99.43
284s
O(N)
Ours (1-ring, no interpolation)
99.30
91.60
99.17
94s
O(N)
Ours (2-ring, no interpolation)
99.45
98.48
99.28
284s
((N)
Table 2: Spherical MNIST results
released by Jiang al. (2019) is. The data consist of 16 channels sampled icospherical grid at level = 5 (i.c. 10242 vertices).
For this experiment, of [IrrepConvolut inn, tion. We set the multiplicity to 2 the input resolution The rest of the encoder stream progressively reduces the grid resolution down to s = while dou- bling the multiplicities (and thus the number of channels) every time the resolution is decreased, resulting in 196 channels. The decoder stream is mirror-symmetricl to the encoder, replacing downsampling layers with upsampling layers. We use residual blocks at each resolution level. The network is trained for 60 epochs with an initial learning rate of 0.01 which is reduced by factor of 0.4 every 30 epochs. Batch size is 64. We use a weighted cross-entropy loss, due to class imbalance following the setup of Jiang et al. (2019), with Adam optimizer (Kingma & Ba 2013).
residual U-Net architecture. Our residual blocks contain two blocks
cheeBaatchootte RegularNon inearity] and a skip connec-
and fix max.freq
2 throughout.
In Table 3 we present our results. Our model compares favorably against the two spherical CNNs that have reported results on this dataset (Jiang et al. 2019) Zhang et al.] (019). Both methods operate on the same icospherical grid as our method. Our method is on par with the Icosahedral CNN of Cohen et al. (2019) accuracy while significantly outperforming mAP metric. We attribute this gain to the fact that our model reflects the geometry of the signal more faithfully than IcoCNN which operates
approximated sphere, namely, icosahedron.
6.3 OMNIDIRECTIONAL SEMANTIC SEGMENTATION
Next, we run our model omnidirectional semantic segmentation task. For this experiment, we use Stanford 2D3DS dataset which consists of 1413 omnidirectional RGBD images acquired in 6 different locations with labels of 13 semantic categories. We are following the experimental setups in (Jiang al. 2019 Cohen et al. 2019) and report averages over 3 cross-validation splits.
We use a residual U-Net similar to what described 6.2 The only difference is that we use 4 residual blocks per resolution instead of 2. We train the network using weighted cross-entropy loss and Adam optimizer for 100 epochs. We set batch size to 16 and initial learning rate to 0.01. Learning rate is reduced by factor of 0.7 every 20 epochs.
We report our results Table 4 We observe that our rotation equivariant network performs similar to the orientation-aware network Zhang et al. 2019) while outperforming IcoCNN which is only equivariant up to discrete subgroup. A5 (i.e. icosahedral group). of SO(3). We note that 2D3DS acquired with preferred camera orientation rendering additional challenge to equivariant model which assumes no preferred orientation. Yet, the comparison against orientation-aware model of Zhang et al. 2019) reflects the expressive capability of ours despite the disadvantage.
Method
BG
Average (%)
mAP (%)
Method
Mcan Acc. (%)
Mcan IoU (%)
Jiang al. 2019
97.0
94.0
93.0
94.7
N/A
Jiang et al.
2019
54.7
38.3
Zhang at 2019
97.3
96.3
97.5
97.0
55.5
Cohen at
2019
55.9
39.4
97.4
97.9
97.3
97.7
75.9
Zhang et 2019
58.6
43.3
Ours
97.0
96.7
97.6
97.1
80.6
Ours
58.2
39.7
Table 3: Results on Climate Pattern Segmentation
Table 4: Results on Stanford 2D3DS

---

Under review as a conference paper at ICLR 2020
6.4 ATOMIZATION ENERGY PREDICTION
In this experiment, we test our method on molecular energy regression task given the molecular geometry and charges associated with each atom. To this end, we QM7 dataset (Blum & Reymond, [009] Rupp et al. 2012) which consists of molecules with varying number of atoms N ( 23). The molecules considered in this dataset are made up of T 5 different elements (i.e. H. N, O and S). Each molecule is defined by a tuple (pi, 2s) with indexing the atoms. We follow the setup for signal representation outlined in Cohen al. (20T8) and represent geometry of molecules with spheres, S;, of constant radius centered at atomic positions, ps. in a careful ar- rangement such that no two spheres intersect in molecule. Finally, each atom associated with T-channel spherical potential functions describing atomic interactions within the molecule with U. U=(2)xEx;- = = (2:2)/ x
Pil-
We use similar architecture to that of Cohen et al. (0018) in terms of parameters. The details of network and training is available in Appendix F.2 Our method achieves the lowest RMSE error in comparison to the spherical baselines as shown in Table 5
Model
RMS crror
o =
0.0
0 = 0.5
8 == 1.0
d= 1.5
c 2.0
P/Raddm CM (Montavon al. 2012
5.96
SpepSp (FCN)
100.0
99.79
97.70
92.52
86.84
PES2CNN Kondor e al 2018
7.97
Speerr (CNN)
100.0
97.18
91.25
83.61
S2CNN (Cohen al. 2018
Ours
99.27
98.23
96.29
91.80
81.90
Ours
Table 6: Results Cosmology dataset. Split level
Table 5: Results on QM7 Atomization En- ergy Prediction task.
6.5 COSMOLOGICAL MODEL CLASSIHICATION
We test model on cosmological model classification task which is interesting test bed to demonstrate our algorithm's 's flexibility in admitted sampling scheme and scalability aspects. To this end, a dataset released by Perraudin et al. (2018). This dataset consists of cosmological con- vergence maps which are generated by whole-sky N-body simulations for two different parameter settings of ACDM cosmological model. Both cosmological models are simulated 30 times. For testing, 20 simulations are held out. The remaining 40 simulations are further split into training and validation sets of size 32 and 8, respectively. The signals are represented on HEALPix grid. We refer the reader to the original article for more information.
We follow the experimental setup described (Perraudin al. 2018) for evaluation. We split the maps into 12 x o² patches where = 4 indexes the hierarchical level of the patches in HEALPix sampling scheme. We then obtain signals of dimensionality (Neide/o)) where Nsides i.e. number of pixels on one side of HEALPix patch, is set to 200 During training, noise with varying standard deviation, nosse E {0, 0.5, 1, 1.5, 2) and report results separately on each setting. The details of network architecture and training are available in Appendix F.3 The results in To that our model performs similarly to the graph- baspe approach.
apply additive Gaussian
7 CONCLUSION
We have introduced the gauge equivariant spherical CNN, which is simultaneously efficient, numer- ically accurate, flexible with respect to the pixel grid used, and which can in principle be applied to data that defined on local parts of the sphere only. Additionally, it is the first implementation of the theory of gauge CNNs for manifold that is not locally flat. Our experimental results show that the method accurate and achieves strong performance on various benchmark problems. Moreover, our method can be adapted to work on meshes, and we plan to work on this in the future.
Results from running code in repository httpss///gitthb.....ooosssss////
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
L. C. Blum and J.-L. Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. Am. Chem. Soc., 2009.
Wouter Boomsma and Jes Frellsen. Spherical convolutions and their application in molecular mod- clling. In NIPS, 2017.
Davide Boscaini, Jonathan Masci, Simone Melzi, Michael M Bronstein, Umberto Castellani, and Pierre Vandergheynst. Learning class-specific descriptors for deformable shapes using localized spectral convolutional networks. In Computer Graphics Forum, 2015.
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geo-
metric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine, 2017.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In ICLR, 2014.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In ICML, 2016.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical CNNs. In ICLR, 2018.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolu- tional networks and the icosahedral cnn. In ICMI., 2019.
Taco S Cohen and Max Welling. Steerable CNNs. In ICLR, 2017.
Benjamin Coors, Alexandru Condurache, and Andreas Geiger. Spherenet: Learning spherical rep-
resentations for detection and classification in omnidirectional images. In ECCV, 2018.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis, Learning so(3) equivariant representations with spherical cnns. In ECCV, 2018.
Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In ICML, 2017.
K. M. Górski, E. Hivon, A. J. Banday, B. D. Wandelt, F. K. Hansen, M. Reinecke, and M. Bartel- mann. HEALPix: A Framework for High-Resolution Discretization and Fast Analysis of Data Distributed the Sphere. The Astrophysical Journal, 2005.
Francis Begnaud Hildebrand. Introduction numerical analysis. Courier Corporation, 1987.
Chiyu Max Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, and Matthias Nießner. Spherical cnns on unstructured grids. In ICLR, 2019.
Kingma and Ba. Adam: Method for Stochastic Optimization. ICLR, 2015.
Thomas N Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Net- works. In /CLR, 2017.
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordas nets: a fully fourier space spherical convolutional neural network. NIPS, 2018.
Min Liu, Fupin Yao, Chiho Choi, Ayan Sinha, and Karthik Ramani. Deep learning 3d shapes using alt-az anisotropic 2-sphere convolution. In ICLR, 2019.
Jonathan Masei, Davide Boscaini, Michael M. Bronstein, and Pierre Vandergheynst. Geodesic con- volutional neural networks riemannian manifolds. ICCVW. 2015.
Grégoire Montavon, Katja Hansen, Siamac Fazli, Matthias Rupp, Franziska Biegler, Andreas Ziehe, Alexandre Tkatchenko, Anatole V Lilienfeld, and Klaus-Robert Müller. Learning invariant rep- resentations of molecules for atomization energy prediction. In NeurlPS, 2012.
Mayur Mudigonda, Sookyung Kim, Ankur Mahesh, Samira Kahou, Karthik Kashinath, Dean Williams, Vincent Michalski, Travis OBrien, and Mr Prabhat. Segmenting and tracking extreme climate events using neural networks. 2017.
11

---

Under review as a conference paper at ICLR 2020
Richard B Neale et al. Description of the near community atmosphere model (cam 5.0). 2010.
Nathanael Perraudin, Michaël Defferrard, Tomasz Kacprzak, and Raphael Sgier. Deepsphere: Effi- cient spherical convolutional neural network with healpix sampling for cosmological applications. ArXiv, 2018.
M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Physical Review Letters, 2012.
Yu-Chuan Su and Kristen Grauman. Learning spherical convolution for fast features from 360 imagery. In NIPS, 2017.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor Field Networks: Rotation- and Translation-Faqiiarrinn Neural Networks for 31) Point Clouds. 2018.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In NeurIPS, 2018.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov. and Gabriel J Brostow. Harmonic Networks: Deep Translation and Rotation Equivariance. In CVPR, 2017.
Chao Zhang, Stephan Liwicki, William Smith, and Roberto Cipolla. Orientation-awarc semantic segmentation on icosahedron spheres. In ICCV, 2019.
A IMPLEMENTATION OUTLINE
This section outlines how the Gauge equivariant convolution implemented. PyTorch notation is used and some details omitted. During pre-computation, the kernel tensor is constructed, which used during runtime to perform the convolution.
A naive implementation would construct for each basis kernel a large matrix mapping from all input channels to all output channels. The parametrized kernel is then obtained from contracting these with weight vector, which has weight for each basis kernel. However, this stack of basis kernels would be very large for two reasons. Firstly, since the grid is irregular, a separate copy needs to be stored for each point on the sphere. Secondly, is common to irreps of low orders, but many copies of those (so with a multiplicity larger than one), each having the same set basis kernels.
We prevent creation of this large stack of basis kernels by exploiting the fact that each basis kernel is a very sparse linear transformation, as it only has one irrep as input and one as output, making block-sparse. This allows us group the basis kernels mapping from and to the same irrep. So can exploit the block-sparsity and also prevent allocating a kernel basis for each multiplicity.
The kernels between two irreps can be constructed as follows. Arguments the orders of the input and output irreps, set of radial activation functions, and for each point and neighbor q, the radius and angle of the log map from p to q the gauge at p, the parallel transport angle.
def build kernel (order in, order_our ouut andial_functinn log_map_angle, transport, mask)
log tog_map_Ht
*""Last four arguments are [num_v, num_neigh)- Return (num_basis, num_v, num_neigh, dim_out, dim_in], transport)
an order_in
a_out = order_out * log_map_and _nglee
cos cos_min np. cos (a_out-a_in)
cos_plus np cos (a_outaaa_i)
sin_min
sin_plus np. sp.
if order in == 0 or order out == 0:
else:
angular_part np.
12

---

Under review as a conference paper at ICLR 2020
[[cos_min, -sin_min] min], [[ain_min, coa coa_min], [[cos_plup sin [[-sin_plus, cos_plus),
[sin_min, cos cos_min]], [-cos min, sin_min]],
plus, -cos plus]] [cos_plus, sin_plus]],
radial_parrs = np. apparrry ([f (log_map_z) for f in radial_functions kernel np. einsimm
'rpq, pq, bi jpq-bbrqijj', radial_part, mask, angular_part) return kernel. reshape (-1, ekernel.shapp (2:]))
Given such kernels for all irrep orders, we subsequently can define the gauge equivariant convolu- tion.
Basically, we iterate over each irrep the in dimension and each irrep in the dimension. For each block, we have min *
input dimensions and Mous ddim(order output dimensions.
[num_ batch, num_v, v, c_in]
2 The layout of the channels is: cat (flatten (irrep order, irrep multiplicity, irrep dimension) )
4 neighbors =
[num V, num_neigh]
reps_in [(0, 4), (1, 4)1 reps_out = [(0, 4), (1, 3)] # irreps of order 0, 3 of order 1 kernels
irreps of order 0, 4 of order 1
# {(order_ouu, order_ing in)
num num neigh, out, din]]
# One weight for each: in/out irrep number of basis kernels * miltiplicities in/out weights = torch. randn (sum( len (kernels (order_out, order_in) nouut * min
13
for order_out, out in reps_on outt for order_in, m mi in reps_in))
15
17 Neighborhood expansion
x x(:, neighbors] num num_batch, numv, num_neigh, c_in = s_e.shape
# Dimension of zepresentation of order 1 else
dim lambda order: 1 if order
24 out sum (dim (1) * m for 1, m in reps_out) y torch. zeros (num_batchn nem_v, c_out)
26 y v_x 0
widx 0
28 # Iterate over in/out irreps. Concatenate over out irreps
for order_out, m_out in reps_out:
xiixx
for order_in, min in reps_in: Sum over in irreps
k = kernels (order_out, order_in)]
rep x_el;, :, ;, x_idx: idx m in dim (order [order_in)] .view (num houm_batch, num_v, num jum_neigh, miin, dim dorderer__in _i))
in)
w rep
+ len(k) mot m_in] vvew (len (k), m_uut, m_in)
37
y yl:, :, y_idx:y_idx (omderer_oott)
mouut] += torch. w_rep, k, x_rep) x idx += mi * dim dim(ordr__in) in)
39
w idx += (k) m_t m in
__idx += dim(order_out) out
return y
13

---

Under review as a conference paper at ICLR 2020
B SPHERICAL GEOMETRY
To compute the logarithmic map at p whose exponential map lands at and to compute the parallel transporter from y to p along a geodesic, we do the following.
B.I PARALLEL TRANSPORTERS
Since Tp--q planar rotation, is determined by where it sends a single (non-zero) vector. We take the first basis vector by Wg (11, 0)) and express it in 3D Euclidean coordinates, We then rotate this vector by the angle <(p, 9) arccos (p. between p and around the axis p x q which is orthogonal to the pę plane. The resulting vector lies in the tangent plane at p. Then Tp--q is determined as the angle between this vector and the first basis vector b" wp((1,0) 0)) in T,.S..
We precompute the transport angles for every point p in the grid V and every q Np)). This results in an array of angles of size num x num_neigh where num = [V] and num_neigh = maxpev [W(p)] is the maximum neighborhood size. For nodes with non-maximal number of neighbors, we pad with zeros.
B.2 LOGARITHMIC MAPS
For each V and q E (p) need to compute Upg = log, q. which is the vector in T2 that points in the direction of and has length equal to the geodesic distance between and One way compute the log map is project the 3 dimensional vector pointing from plane. This is done by subtracting from q the component parallel to the vector, as the p vector is orthogonal the tangent plane at p. This produces a vector x (g - p) (p.g - p)p q- (p, q)p which has the right direction. Then one can scale the length of ō so that it matches the geodesic distance d(p, q) (the arclength):
g the tangent
log, 9 = d(p.9) 9 (p,
11 (p, g)pll
We express the result Upg logp in polar coordinates relative to the gauge at p. This gives two (the angular part of
arrays 1 log_map_r" (the length radial coordinate of v) and relative to the gauge p). Both shaped num_ num_neighh before. Since the geometry and grid are fixed, these arrays are computed only once before training.
C QUADRATURE
The goal of selecting the quadrature points and weights is approximate integral Jo g(x) dx,, where D C g22 is the unit disk, with finite sum ΣvEZ (x)(x), for points I D and each with weight wz. We use simple method of selecting I and w,. We desire a rotational symmetry, so the points lie on NR radial rings, each with Ne points.
First, we use Gauss-Legent
quadrature (Hildebrand 1987) to obtain Ng quadrature points with weights w. over the interval [-1, 1]. We map this to radial weights and points by r (u + 1)/2, wr = w.//2. For the angular component, we take a uniform grid of No points on (- T]. The integration points on the disk are then (r, 0) with weights w, w,/Ne.
This gives accurate integration, because points (u + 1)/2 and weights w create a quadrature scheme on [0, 1]. The square root creates uniform polar grid, the integration measure for the disk is dd = d2d0/2.
14

---

Under review as a conference paper at ICLR 2020
D SOLVING THE SO( KERNEL CONSTRAINT
m2, 72
d(pm @ Pr))
Linearly independent solutions for K (0)
0, 0
0
0
0,
(cos nd
sin nd), , (sin nó
cos no)
n
m
cos m& sin mf)
(
sin ml cos m8
m,
0
0
m, 71
T
0
0
0
171
c-
-8 8 e_
(-c-
(
8-
c-
8-
8+
c+
-c-)
84
)
(-ot
c+
C+ s+
0
m
n
Table 7: Solutions the kernel constraint for kernels that map from Pn Pm- We denote C1 cos(m + n = sin(m + n)0.
Consider a kernel K a function on the tangent space T,,M, such that for all E T,M. K(v) is linear map from features in representation Poot to features in a representation Pour. This kernel is equivariant if Ve E T and Vg E SO(2) the following kernel constraint is satisfied:
K(g = pounlg-)) (v))mnng) Ve E T2M, g SO(2)
E
(8)
We immediately see that this constraint is independent for each radius, independently for each radial ring, giving rise to the angular constraint on each ring:
solve the constraint
K(0
Paul (-9 K (m) 40 E St,g g SO(2)
where we note that the circle equals S0(2) and the action of SO(2) on itself is simply an addition of angles. Any representation P can be written equivalent to direct sum of the irreducible representations (irreps) of SO(2), which are the following, indexed by their order n. E N:
/oos ng
sin sinngg cos ng
po(g) 1, (n(g)=
) N
sin ng
This means that for any representation p. we can always find an invertible matrix A such: Vy p(g) = block diag Mlag((nn:---PP... -- A, where (M.,,... nxx) indicate the orders of the constituent irreps. Hence, we can solve the angular kernel constraint (Eq. D for two irreps, Pm. and construct the solution for general representations Pin and Pour from the obtained solutions. Noting that all irreps are orthogonal in our chosen basis are orthogonal, so that (g) = p(- we see that, introducing explicit indices:
Ps and Pour =
K(A - g)is = =mm(-)))x((0AAAp
Pm (-))))p((99)))_
(pm 4 (p))(1))())...m K(O)u:
As this equation is ought to be satisfied for any g SO(2) and SO(2) is one-dimensional Lie group, we can equivalently require it to be satisfied by an infinitesimal element of SO(2) close to the identity, creating an ordinary differential equation:
iC(0) =d(pm (.))
(pm @ pu)ls) 1=0.,
where d(pm p.) = 0g
For various m, n, we can construct (pm Pp)) take the derivative with respect to g and set g to to obtain the matrix d(pm @ Pa). Subsequently, any computer algebra system can solve the ODE. The
15

---

Under review as a conference paper at ICLR 2020
solutions can be found in Table 7 Now, for general (in = A block_diag d....( (9) Pou(g) B block..daag(Pm. ....Pass/ (g) B. a general solution has the form:
mm (0)
Kminn mn (0)
((0) = B-1
K Kmmaa mn (0)
K Km (0)
where (0) solution from Table 7 mapping from irrep Pn. to pm- For each in/out irrep pair, get linear solution space of 1, or 4 dimensions. For general Pins Pour. the solution space is the product of the solution spaces of each in/out irrep pair. A parameterized kernel can thus be constructed by assigning weight to each independent solution of each in/out irrep pair.
16

---

Under review as a conference paper at ICLR 2020
E EQUIVARIANCE ERROR BOUNDS ON REGULAR NON-LINEARITY
The regular non-linearity acts each point the sphere in the following way. For simplicity, assume that the representation is copies of po @ P1 @ @ PM- One such copy can be treated as the discrete Fourier modes of a circular signal with band limit M. We map these Fourier modes to N spatial samples with an inverse Discrete Fourier Transform (DFT) matrix. Then apply to those samples point-wise non-linearity, like ReLU, and map back to the Fourier modes with a Discrete Fourier Transform Matrix.
This procedure is exactly equivariant for gauge transformation with angles multiple of 2 I/N, but approximately equivariant for small rotations in between.
In equations, we start with Fourier modes 2o. (Za (m), za (m)) B at some point on the sphere and result Fourier modes 2o. (Za ( We let O, ... N index the spatial samples.
<tt) + Σ (m) cos
2m N
2T Σ (m) sin mt
mt
N
ytt = f(x(t))
E y(t)
20
2
2T
Za (m)
Σ
cos
mt
y(t)
2
2m
2m(m)
sin
y(()
N
Note that Nyquist's sampling theorem requires us to pick N 2 2B + 1, as otherwise information is always lost. The normalization is chosen that 2a (m) xa(m) if is the identity.
Now we are interested in the equivariance error between the following two terms, for small rotation 5 E [0, 1). Any larger rotation can be expressed in rotation by multiple of 2 /N, which is exactly equivariant, followed by a smaller rotation. We let _F (m) be the resulting Fourier mode first the input is trage-transformed and then the regular non-linearity is applied, and let z (m) be the result of first applying the regular non non-lincarity, followed by the gauge transformation.
Fx
FT (m)
N
2
Σ
CO8
2T
N
ml
y(1 u((+8) -0==
N
2
Σ
Cm (m)())(( 8)
Za
TF
(m)
N
2
Σ
cos
N m(1
(((((-)) -6) y(()
N
2
Σ Cmll 0)y(1)
where defined for convenience C. (t) cos(2nmt/NN) We define norms Em (1 (m)l + (x)(m)) and ||xx|] = Σm. m(1 + (m)))
||0|||
|zoo| +
If the input x band limited by B, the output is band limited by B', N samples are used and the non lineariyy has Lipschitz constant L,,, then the evror to the gauge equivariance of the regular non-linedd oin-lineartty bounded by:
Theorem 1.
4T N
FT
TF
(2B's + 1110/11 + B''B' (1)1a]111)
which goes to N- x0.
17

---

Under review as a conference paper at ICLR 2020
Proof. First, note, since the Lipschitz constant of the cosine and sine is 1:
2mmo
2mm
Cm (t - 8) - -cm(t)] -
N
N
2x
2T Σ ((] + (xa(m)))
[o(( + 8) - x(1) -
N |000||1
N
2πT
1u(1 6) - v(1) Ly
|
N
ccm (t) $
[z((1) S |col + Σ
+ (x)(m))) = ||||||
m
lu(()] _ |
Then:
(f)y(t + 6) Cm d)y(t)
= (t) (u(t + -6) - u(t]] u(t) [cm (t 8) - [t)
$lam (t)lly(t +5) -v(t) + +ly(t)llmm(t 5) (t)
2T Lb ||0x|11 + -|||=|||1 N
2mm
N
2xI.
(||00|1 + m||a/11)
N
So that finally:
1:a (m) =/- .FF [m]m
2
(t) (++ - ctm ())(t))
4 (110x111 + ml(x=11)
N
The sinus component 2 5 (m) (t)), which is derived above. So if is band-limited by B':
TF (m) has the same bound, while 1 - =1 lyv(t + 8) 2/
|| FT
TF
120
FT
TF
Z
FT
(m)
TF (m)) + FT (m) - 2/ TF (m) (m))
m=
B¹
4TLs
(2B' + 2
110cc|11 + Σ
2ml 3m||z||1
N
m=
4mLf
(2B's + 2 ||rr|11 + +B'(B'' 1)](=111
N
Since ||a||| = ((B[==]))) we vegee|12= FT=
-2 TF
BB' t = OBBBBBBB
||]|[)) which obviously vanishes
N
as N
cc.
18

---

Under review as a conference paper at ICLR 2020
F EXPERIMENTAL DETAILS
In the following subsections, we provide with more details regarding experiments on MNIST, At- omization Energy Prediction and Cosmological Model Classification.
F.I MNIST
For the MNIST experiments, we disambiguate two variants, both used with interpolation and with- out. The first variant uses only the 1-ring of 7 nearest neighbors and representations po @ P1. This version uses three radial functions: one that covers the self ieffrinterattion and one that covers the other neighbors.
The second variant uses the 2-ring of 19 nearest neighbors and representations Po @ pt + p2- This version uses three radial functions: one for the self-interaction, one for the neighbors on the 1-ring and one for the neighbors on the 2-ring.
For either variant, build network consisting of 8 convolutional layers, with stride after each layers. A stride layer selects the vertices that are present in the IcoSphere with one fewer subdivision, and divides the number of vertices by roughly 4. The input is scalar feature. After convolution, the irrep features with multiplicity after each layer: [16, 16, 32, 32, 64, 64, 64, 64]. The last convolution layer maps to a scalar feature of multiplicity 64. Then the signal averaged over the vertices, creating a gauge invariant network. This is then processed by 2 layer of MLP mapping to multiplicity 50 and then the 10 classification logits.
Nonlinearities are ReLU for scalar features and the tures. Batchnorm used, as well as dropout with drop probability 10%.
for all other fea-
F.2 ATOMIZATION ENERGY PREDICTION
Our network architecture used in this experiment is depicted in Table We use four blocks of gauge equivariant convolutions followed by batchnorm and regular nonlinearities. We also deploy pooling / downsampling layers in blocks (1-3) to reduce the resolution of the spherical maps progressively. Rest the network consists of linear layers followed by batchnorm and ReLU. We sum-pool over the atoms before the loss function (see block-7) to become invariant against permutations of them.
We train the model with batch size 32 and an initial learning rate of 1 for 30 epochs using mean leaun-squared-eree (MSE) loss function. The learning rate is reduced by 0.1 every tenth epoch.
block
Operators
Multiplicity (in/out)
Order (pinl
(Convolution, Downsample, BatchNorm, Regular Nonlinearity)
5/16
0/1
[Convolution. Downsample, BatchNorm. Regular Nonlinearity]
16/32
1/2
[Convolution, Downsample, BatchNorm, Regular Nonlincarity]
32/64
[Convolution, BatchNorm, ReLUU]
64/64
1/0
[Linear, BatchNormid d, ReLU]
64/256
0/0
[Linear, BatchNormid, ReLU]
256/64
0/0
AtomPool
64/64
[Linear, BatchNormld, d, ReI.U]
64/512
0/0
[Linear, BatehNormid, ReLU]
512/1
0/0
Table 8: Network architecture used in Atomization Energy Prediction experiment.
F.3 COSMOLOGICAL MODEL CLASSIFICATION
For this experiment, we use HEALpix sampling on split level = 4 Perrauing which covers 0.5% of the spherical manifold and has signal dimensionality of 65k.
Perraudin et al. (2018)
19

---

Under review as a conference paper at ICLR 2020
For the experiments, 2018) by replacing their graph convolutions with our gauge equivariant convolutions. We elaborate the architecture in Table g
network architecture identical to the one described in (Perraudin al.
For each noise factor, we train an instance of this model with batch size 16 and an initial learning rate of 4 for 120 epochs. Finally, we use Binary Cross Entropy the loss function.
block
Operators
Multiplicity (in/out)
Order (pinlpona)
[Convolution, Downsample, BatchNorm, Regular Noalinearity)
1/10
0/1
[Convolution. Downsample, BatchNorm. Regular Nonlinearity]
10/20
[Convolution, Downsample, BatchNorm, Regular Nonlincarity]
20/40
1/1
[Convolution, Downsample, BatchNorm, Regular Nonlincarity]
40/40
1/1
[Convolution, Downsample, BatchNorm, Regular Nonlinearityl
40/40
[Convolution, Downsample, BatchNorm]
40/2
1/0
Softmax
2/2
0/0
Table 9: Network architecture used in Cosmological Model Classification experiment.
20


Reviews:
[{"review_id": "SJgLAnS0YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes SO(3) equivariant layer, derived from the recently introduced Gauge equivariant CNN framework. The novel contributions are in taking the Gauge equivariance CNN and finding efficient ways to perform logarithmic mapping, parallel transport, and convolution by the equivariant kernel when applied to the sphere. An interpolation scheme for an improved approximation to global SO(3) symmetry is also discussed. Experimental results on spherical MNIST, climate pattern segmentation, and omnidirectional semantic segmentation demonstrate the usefulness of the proposed method for prediction on a sphere.\n\nFor the most part, the paper is very clearly written despite the challenging and technical nature of the topic. In particular, the first four pages provide a nice overview of related work and a clear explanation of the Gauge equivariance framework of Cohen et al’19. Two sections that can benefit from further clarification are the proposed \"regular non-linearities\" (where it would be nice to show in equation why we have equivariance), and equation (4), the logarithmic map (where I had a hard time mapping the discussion in words to the equation). \n\nHowever, the experiments, while satisfactory, are not impressive: one issue is that the results are mostly compared to the results of two relevant papers by Cohen and colleagues. In recent years, there have been other proposals for deep learning on the sphere, and I wonder why experiments do not try to compare with these works?  (see Kondor et al’18, Coors et al’18, and others cited in the paper.) Moreover, although in theory, the proposed framework improves the Icosahedral CNN of Cohen et al’19 (by directly operating on the sphere rather than an Icosahedral approximation), the practical improvements over the Icosahedral CNN seem to be often marginal (with one exception in spherical MNIST). Do you have any explanation for this? Is there any setup where you expect the proposed approach would give a substantial improvement?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeURqxxcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Cohen et al. recently proposed the \"Gauge equivariant CNN\" framework for generalizing convolutions to arbitrary differentiable manifolds. The present paper instantiates this framework for the case of the sphere.\n\nThe sphere is the simplest natural non-trivial manifold to try out gauge invariant networks on, and spherical CNNs have several applications. However, other than the details of the interpolation etc., there is really very little in this paper that is new relative to the original paper by Cohen et al., it reads a bit more like an extended \"experiments\" section. \n\nUnfortunately the experimental results are not all that remarkable either, probably because the tasks are relatively easy, so other SO(3) equivariant architectures do quite well too. Given that there is essentially no new theory in the paper, I would have welcomed a much more thorough experimental section, comparing different architectures, different discretization strategies of the sphere and different interpolations/basis functions.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylO6pcycB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. (p1) ``in almost in all cases\" to \"in almost all cases\" \n2.  (1.1) The authors could explain more about why we would want to consider tensor features. \n3. They conducted experiments on different datasets, including the MNIST dataset. They achieved good results comparing to baseline spherical CNNs. However, the advantage of this method over S2CNN can be further elaborated, as S2CNN already achieved high accuracy; it seems like the one improvement is the complexity (improved from S2CNN's $O(N \\log N)$ to their model's $O(N)$), but the reduction of complexity is not significantly reflected in the training time per epoch (from 380s to 284 s). \n4. Overall, the paper provides clear theoretical backgrounds on gauge CNNs that justifies their definition of convolution operator only uses the intrinsic structure of the manifold (does not reply on higher dimensional embedding).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends Gauge invariant CNNs to Gauge invariant spherical CNNs.  The authors significantly improved both theory and experiments during the rebuttal and the paper is well presented. However, the topic is somewhat niche, and the bar for ICLR this year was very high, so unfortunately this paper did not make it. We encourage the authors to resubmit the work including the new results obtained during the rebuttal period.
Decision: Reject