Scalable and Equivariant Spherical CNNs by Discrete-Continuous (DISCO) Convolutions
Authors: Jeremy Ocampo
Matthew Alexander Price
Jason McEwen
Venue: ICLR 2023 poster (2023)
Keywords: Spherical CNNs
rotational equivariance
efficient algorithms

Abstract: No existing spherical convolutional neural network (CNN) framework is both computationally scalable and rotationally equivariant.  Continuous approaches capture rotational equivariance but are often prohibitively computationally demanding.  Discrete approaches offer more favorable computational performance but at the cost of equivariance. We develop a hybrid discrete-continuous (DISCO) group convolution that is simultaneously equivariant and computationally scalable to high-resolution. While our framework can be applied to any compact group, we specialize to the sphere.  Our DISCO spherical convolutions exhibit $\text{SO}(3)$ rotational equivariance, where $\text{SO}(n)$ is the special orthogonal group representing rotations in $n$-dimensions.  When restricting rotations of the convolution to the quotient space $\text{SO}(3)/\text{SO}(2)$ for further computational enhancements, we recover a form of asymptotic $\text{SO}(3)$ rotational equivariance.  Through a sparse tensor implementation we achieve linear scaling in number of pixels on the sphere for both computational cost and memory usage.  For 4k spherical images we realize a saving of $10^9$ in computational cost and $10^4$ in memory usage when compared to the most efficient alternative equivariant spherical convolution. We apply the DISCO spherical CNN framework to a number of benchmark dense-prediction problems on the sphere, such as semantic segmentation and depth estimation, on all of which we achieve the state-of-the-art performance.

SCALABLE AND EQUIVARIANT SPHERICAL CNNs BY DISCRETE-CONTINUOUS (DISCO) CONVOLUTIONS
Jeremy Ocampo 2.2 Matthew A. Price Jason D. McEwen
Kaaennova Limited, University College London (UCL)
ABSTRACT
No existing spherical convolutional neural network (CNN) framework both computationally scalable and rotationally equivariant. Continuous approaches capture rotational equivariance but are often prohibitively computationally de- manding. Discrete approaches offer more favorable computational performance but at the cost of equivariance. We develop hybrid discrete-comtinuous (DISCO) group convolution that simultaneously equivariant and computationally scalable to high-resolution. While our framework can be applied to any compact group, we specialize to the sphere. Our DISCO spherical convolutions exhibit SO(3) rota- tional equivariance, where (O(n) is the special orthogonal group representing rotations in n-dimensions. When restricting rotations of the convolution the quotient space S0(3)/SO(2) for further computational enhancements, we recover a form of asymptotic SO(3) rotational equivariance. Through a sparse tensor im- plementation we achieve linear scaling in number of pixels on the sphere for both computational cost and memory usage. For 4k spherical images we realize a sav- ing of 10 in computational cost and 10* in memory usage when compared to the most efficient alternative equivariant spherical convolution. We apply the DISCO spherical CNN framework number of benchmark dense- preddctinnnionn problems on the sphere, such as semantic segmentation and depth estimation, on all of which we achieve the state-of-the-art performance.
INTRODUCTION
Spherical data are prevalent across many fields, from the relic radiation of the Big Bang in cosmol- ogy, to 360 imagery in virtual reality and computer vision. High-resolution data on the sphere are increasingly common in these and many other fields. Existing deep learning techniques for spherical data, however, cannot scale to high-resolution while also exhibiting equivariance, powerful induc- tive bias responsible part for the success of Euclidean convolutional neural networks (CNNs). Furthermore, many tasks involve dense predictions (e.g. semantic segmentation, depth estimation), necessitating pixel-wise outputs from deep learning models, exacerbating computational challenges.
Continuous spherical CNNs approaches. Bronstein et al. (021 present the categorization of geometric deep learning approaches. Deep learning on the sphere falls into the group category. since the sphere is homogeneous space with global symmetries
which the group of 3D rotations SO(3) acts. In fact, the sphere is often considered as the prototypical example of the group setting. A number of group-based spherical CNN constructions have been developed (Cohen et al.) 2018
Kondor et a 2018 Esteves et al. 2018 2020 Cobb et al. 2021 McEwen et al. 2022 Mitchel et al., 2022), where Fourier representations of spherical signals (i.e. spherical harmonic representations), combined with sampling theorems on the sphere (Driscoll & Healy] 1994 McEwen & Wiaux (201T), are considered to provide access to the underlying continuous spherical signals and symmetries. While such approaches live natively on the sphere and capture rotational equivariance, they are highly computationally costly due to the need to compute spherical harmonic transforms (while fast spherical harmonic transforms exist they remain computationally demanding). McEwen et al. (022) develop scattering networks on the sphere to alleviate computational considerations. While such approach helps to scale to nighrresolution input data, some high-resolution information is inevitably lost and architectures providing dense predictions are not supported.
"Corresponding author; jason, mcewen@kagennva..com

---

Discrete spherical CNNs approaches. Other approaches spherical CNNs generally fall under the grid, graph or geodesic geometric deep learning categories yielding discrete approaches (Boomsma & Frellsen 2017 Jiang et al. 2019. /Zhan et al. 2019 Perraudin et al. 2019; Cohen al. 2019). These approaches offer more favorable computational per- formance than the group-based frameworks but at the cost of equivariance. Since a completely regular point distribution on the sphere does in general not exist, these discrete approaches lose the connection to the underlying continuous symmetries of the sphere and thus cannot fully capture rota- tional equivariance (although in some cases limited rotational equivariance may be achieved; e.g. Cohen et al. 2019).
(a) Continnous Equivariant O Not Scalable
Our approach. Previous spherical CNN constructions can be categorised into two broad approaches: continuous approaches that capture rotational equivariance but are computationally demanding; and discrete approaches that do not fully capture rotationally equivariance but offer improved com- putational scaling. In this article we develop novel hybrid discrete- continuous (DISCO) approach that is simultaneously computationally scalable high-resolution, while also exhibiting excellent equivariance properties (see Figure D. We define a DISCO group convolution, which we then specialize to the sphere. A transposed DISCO convolution can be considered to support dense-predictes inputs and outputs are supported by our framework. DISCO convolutions afford a computationally scalable implementation through sparse tensor representations. We build DISCO spherical CNNs that achieve state-of- the-art performance on a number of high-resolution dense prediction tasks, including semantic segmentation and depth estimation.
(b) Discrete
+ Not Equivariant Scalable
tasks; hence, both high-resolution
(c) Equivariant Scalable
(Ours)
2
BACKGROUND
Figure 1: Spherical CNN categorization.
2.1 CONTINUOUS GROUP CONVOLUTION
Definition. To generalize CNNs to group geometric deep learning settings the group convolution can be considered (see, e.g.. Cohen & Welling 2016 Esteves 2020) where the convolution (sometimes R on the group G is given by
called correlation) of two functions f.
* (g)) =
J.EG ((u)((o-'uuda uddp(u),
(I)
where g, u E G and du(u)) is the invariant Haar measure. In many cases signals are not defined on a space with a group structure. Often signals are defined on a quotient paac]///I where / is subgroup of G, i.e. is not group but simply a differentiable manifold on which G acts, i.e. a homogeneous space. In either setting the group convolution exhibits equivariance to group actions, that is (Q.f ()) (Q(S * (g), where Q denotes the group action corresponding to E G, i.e. (Qf) f(
G/H and ő G/H + R. For Lie groups, if H is non-normal then G/Il
Fourier representation and sampling theory. The group convolution may be written in Fourier space by the product (f (k) = f(k) (k) (see, e.g.. Fiteves 2020). where & is the Fourier conjugate variable of u, - denotes the Fourier transform, and denotes complex conjugation. For compact manifolds (groups) the Fourier space is discrete (the canonical example being the Fourier series of functions defined on the circle S", ie. periodic functions). Furthermore, for bandlimited signals on compact homogeneous manifolds, exact quadrature formulae exist (Pesenson & Geller 2011), from which the existence of a sampling theorem follows where all information content of a continuous bandlimited signal can be captured in a finite number of spatial samples.
Computation. These results provide a strategy to compute the continuous group convolution ex- actly for bandlimited functions defined on compact homogeneous manifolds: (i) compute the finite
'The quotient space G/ H can he considered (in a non-rigorous way) as formed by collapsing all elements of H to the identity in G. in sense factoring out the elements of H from G.
2

---

set of Fourier coefficients of the signal and filter leveraging the sampling theorem; (ii) compute the group convolution as a product in Fourier space; (iii) compute the convolved signal from its Fourier coefficients. Although the computation is performed in a discrete manner through the finite Fourier representation, the underlying continuous group convolution is computed exactly, without any dis- cretization error. Consequently, perfect group equivariance is achieved. Note, however, that this approach does require computation of the Fourier transform on the manifold, which may be costly.
2.2 CONTINUOUS SPHERICAL CONVOLUTIONS
The strategy outlined above to compute the continuous group convolution exactly for bandlimited signals on compact homogeneous manifolds is the generalization of precisely the approach taken for group-based spherical CNNs (Cohen et al. 2018 Kondor al. 2018 Esteves al. 2018 2020 Cobb et al. 2021 McEwen et al., 2022). where Fourier (i.e. spherical harmonic) representations of signals are computed to provide access to the underlying continuous structure and symmetries of the sphere. The corresponding spherical convolutions exhibit perfect rotational equivariance, with any numerical errors arising simply from finite precision arithemetic (see, e.g., Cobb et al.. 2021). We concisely review the details of this approach to constructing spherical CNNs.
Definition, The spherical convolution of two functions f. b S + R is given by
(J (R) /
w)dn(w),
(2)
where w = (0, denote spherical coordinates with colatitude 0 E [0, π| and longitude yp E [0, 2T). and dy((w) (w) sin 0d0dg is the Haar measure. We consider rotations R E SO(3), where SO(n) is the special orthogonal group of rotations in n-dimensions. The sphere Sa is isomorphic to the quotient space (S))(())); it does not exhibit a group structure but homogeneous manifold.
Fourier representation and sampling theory. Since both S and SO(3) are compact, their Fourier spaces are discrete and sampling theorems can be leveraged compute Fourier representations exactly for sampled bandlimited signals (Driscoll & Healy 1994; Kostelec & Rockmore 2008 McEwen & Wiaux, 2011 McEwen al., 2015). The spherical convolution can then be expressed as product in Fourier space (see, e.g., McFWen et al. 2013 Cohen et al. 2018 Cobb et al. 2021
Computation. For signals on the sphere bandlimited at L (i.e. with Fourier coefficients zero above L). spherical convolutions can be computed exactly as product in Fourier space. Although the computation is performed a discrete manner through a Fourier representation, by appealing to un- derlying sampling theorems the continuous convolution is computed exactly. Consequently, perfect rotational equivariance is achieved. However, the harmonic transforms required are computation- ally demanding (even with fast algorithms; e.g. McEwen & Wiaux 2011 [cEwwen al. 2015)) and present the major computational bottleneck in scaling equivariant spherical CNNs to high-resolution.
2.3 DISCRETE SPHERICAL CONVOLUTIONS
Alternative approaches to constructing spherical CNNs typically consider discrete approximation of the convolution of Equation 2 Boomsma & Frellsen 2017 et al. 2019 Zhang et al.) 2019
Cohen et al. 2019). It well-known that a completely regular point distribution on the sphere does in general not exist. Consequently, while a variety of spherical discretization schemes exists, it is not possible to discretize the sphere in a manner that invariant rotations (Cobb al., 2021). On the other hand, since discrete approaches avoid the need for Fourier transforms on S and SO(3), they are generally more computationally efficient than the continuous approaches discussed above that do exhibit rotational equivariance.
3 DISCRETE-CONTINUON (DISCO) CONVOLUTIONS
To simultaneously realize computationally scalability and rotational equivariance we require a hy- brid convolution that avoids Fourier transforms on manifolds, which otherwise induce consider- able computational cost, while also avoiding a complete discretization. which otherwise sacrifices rotational equivariance. We present novel DISCO adiscrete-continuous) group convolution that achieves precisely this aim. We then specialize to the spherical setting, before discussing equivari- ance, filter parameterization and the corresponding transposed convolution for dense predictions.
3

---

3.1 DISCO GROUP CONVOLUTION
The DISCO group convolution follows by a careful hybrid representation of the group convolution
of Equation | Some components of the representation are left continuous, to facilitate accurate rotational equivariance, while other components are discretized, to yield scalable computation.
Definition. We carefully approximate the group convolution by the DISCO representation
( = JuEG (on))olg'')dplle R
Σ (uu.ww(--u))aau
(3)
where square brackets and index subscripts denote discretized quantities and round brackets denote continuous quantities. Clearly the signal of interest must be discretized at sample positions u4 where i denotes the sample index. Critically, however, the filter vi and the group action remain continuous. This allows the filter v to be transformed continuously by any g. keeping a coherent representation that avoids any discretization errors and, consequently, affords group equivariance, unlike a fully discrete method. The integral with respect to u is also discretized. For bandlimited signals on compact homogeneous manifolds, the existence of a sampling theorem ensures that the integral can be approximated accurately using quadrature weights ((u;)) (see Section 2.1b.
Approximation accuracy and equivariance. The DISCO approximation of Equation 3 is highly accurate for bandlimited signals (real-world signals can be well approximately by bandlimited sig- nals for a sufficient bandlimit). By appealing to a sampling theorem, all information content of the signal can be captured in the finite set of samples [f[u,]]. The filter is represented continuously so does not introduce any error. The only source of approximation error is thus the quadrature used to evaluate the integral. For a sufficiently dense sampling, can appeal to the sampling theorem and corresponding quadrature to evaluate this exactly. Therefore, it is possible in principle to compute the DISCO group convolution exactly, without any approximation error. Since the approximation is highly accurate, which can be made exact for sufficiently dense sampling, and group actions are treated continuously, the DISCO group convolution exhibits excellent equivariance properties.
Computational considerations. The DISCO convolution also affords a scalable implementation. Firstly, it avoids any need for costly Fourier transforms on manifolds and groups. Secondly, as in the Euclidean setting, filters with local spatial support are considered to reduce both memory and computational requirements. The naive scaling of the DISCO group convolution for general kernel is O(N?), where for simplicity N represents both the number of input and output samples. However, for localized filters with K non-zero samples, where K N. the scaling reduces to O(KN) = O(N), just as for Euclidean CNNs.
3.2 DISCO SPHERICAL CONVOLUTION
Definition, We now specialize to the spherical setting, where the spherical convolution of Equation can be accurately approximated by the DISCO representation
f f(w)(((R (R))((R))))((() (l:
(ʃ ())
(4)
where, for now, we consider 3D rotations R SO(3), i.c. f vo SO(3) + R. As in the general group setting the signal must clearly be discretized at sample positions Wi, as is the integral with quadrature weights (w:)). Again, critically, the filter vi and group action, in this case corresponding to rotation R, remain continuous, facilitating rotational equivariance.
Approximation accuracy and equivariance. We appeal the sampling theorem on the sphere of McEwen & Wiaux (201T) since it provides the most efficient sampled signal representation (halv-
ing the Nyquist rate compared to Driscoll & Healy 1994). For spherical signals bandlimited at I, all information content of the underlying continuous signal is captured in ~ 2T,2 samples [f[wa]: The only source of approximation error then the discretization of the integral. By adopting ex- act quadrature (McEwen & Wiaux 2011) this integral could in principle be computed exactly (see Appendix A). However, since the integrand is the product of two signals each bandlimited at L, it is itself bandlimited at 2L. Exact quadrature would thus require the signals to be sampled at ~ 2(22)² = SL2 positions. For our implementation we avoid upsampling signals, trading off a small approximation error avoid four-fold increase in computation. Nevertheless, we do adopt
4

---

the quadrature weights of McEwen & Wiaux (2011) keep approximation errors to minimum. Since the approximation remains highly accurate, and the action of rotations on filters is treated con- tinuously, the DISCO spherical convolution exhibits accurate SO(3) equivariance (see Section 3.3
Computational considerations. For the continuous spherical convolution approach computed via Fourier space, the computation cost is dominated by the Fourier transform on the sphere, which naively is ((N²) but for fast algorithms scales as ((N3/2 (e.g. McEwen & Wiaux 2011). The naive computational scaling of the DISCO spherical convolution is O(N)). When considering lo- calized filters with average K non-zero terms, with K < N, the scaling of the DISCO spherical convolution is reduced to O(N), again just as in the Fuclidean setting.
Restricting rotations to (O(3) (2) While the DISCO spherical convolution is already efficient, we seek further computational savings by reducing the space of rotations considered from SO(3 (3) to
SO(3) (() (we provide visual illustration of SO(3) and SO(3) (2(() rotations in Fig. 3 To
make this explicit, first consider the parameterization of a rotation R E SO(3) by the Euler angles
E 27), E [0, T and 7 [o, 2T),, where the rotation may be written R Z(a)Y ((a)Y(3)2(9) (3) following the 2yz Euler convention for rotations (-) and (-) about the and axes, respectively. Rotations restricted to the quotient space may be written R (() E SO(3)/SO(2), Re- stricting rotations to (S({),,, which is isomorphic to the sphere $2, is analogous to typical Euelidean planar CNNs, where filters are translated across the image but are not rotated in the plane. However, as the space (0(3)/SO(( is not group, when restricting rotations in this manner impor- tant differences to the usual setting arise since we no longer have a group convolution (as elaborated in Section 3.3 We denote the spherical convolution when restricting to R E SO(3) + R. using a different symbol to the SO(3) spherical convolution to avoid is two dimen- reduces the dimension of the output space from ((L²)
by
ff + a : /
any confusion. Since SO(3) is three dimensional space, whereas SO(3)/SO()
sional, restricting rotations to to O(12), yielding an overall computational scaling of O(N)= O(T2), where we recall J. denotes the spherical signal bandlimit.
3.3 ROTATIONAL EQUIVARIANCE
SO(3) rotational equivariance. Consider the DISCO spherical convolution f tb for rotation R SO(3), with Q denoting the action of the rotation Q E SO(3) on signals, which we show satisfies SO(3) rotational equivariance:
((2f) (R) = ((
Σ
(f+ =
(5)
The only approximations involved in Equation 5 (highlighted by the approximate equality symbols)
follow from the quadrature of the integral. Critically, no approximation is involved in applying the rotation Q due to the continuous representation of rotations and the filter th, i.c. the equality labeled (*) is exact. Consequently, the DISCO spherical convolution f * exhibits excellent SO(3) rotational equivariance. An important point note in the SO(3) rotational equivariance proof, which is often ignored, is that it is only possible to relate the expression back to a spherical convolution, i.c. the step labeled (*+), since SO(3) exhibits a group structure and so Q- R E SO(3).
Asymptotic SO(3) rotational equivariance. Consider the DISCO spherical convolution / / d and rotations R SO(3)S0(22) Since SO(3) (2) is not group, Q- & SO(3)/SO(2) and step (**) of the equivariance proof of Equation 5 does not extend to ʃ x th. Consequently, I + & for R E SO(3)SS((2) does in general not satisfy SO(3) or SO(3) ()(()))))))))))))))))))))))))) equivariance (contrast with the Euclidean planar setting that is translationally equivariant for the analogous setting). However, the DISCO spherical convolution f & does satisfy a form of asymptotic SO(3) equivariance. Let = ((a) and R = Z(a')) ((')))))'))))))))))))))) then Q- R 7( (-))(aa Y83 E SO(3). As + 0, we have Q- - ((a' - a (8") E S0(3)/S0(2) and hence ((Of) + ()) + (o( + (R), i.e. we recover asymptotic SO(3) rotational equivariance
3 + O. Note that there is no restriction on a or y, i.e. we acheive full rotational equivariance for rotations about the a axis. This asymptotic SO(3) equivariance is validated numerically in Section 5.1 and shown to be reasonably accurate even for moderately large B. Asymptotic SO(3) equivariance is of significant practical use since content in spherical signals is often orientated and similar content often appears
5

---

similar latitudes, particularly for 360 panoramic photos and video. Hence, asymptotic SO(3) rotational equivariance may even be more desirable than SO(3) equivariance for certain applications.
Axisymmetric filters and (0(3) rotational equivariance. The rotational equivariance consid- crations presented above hold for general filters with directional structure. Axisymmetric filters, L.e. filters that are invariant rotations about their own axis, may also be considered. While ax- isymmetric filters are less expressive than directional filters they have nevertheless been shown to be effective for spherical CNNs (Esteves et al. 2018). For axisymmetric filters the SO(3) spheri- cal convolution f xb and the SO(3)/SO(2) spherical convolution f ② v identify since f * a for (3) independent of y. Consequently, the DISCO spherical convo- lution f + 2b for R E SO(3)/SO(2) with an axisymmetric filter & does exhibit SO(3) rotational equivariance (as validated numerically in Section 5.
R (
3.4 FILTER PARAMETERIZATION
We consider a continuous filter representation a in the DISCO spherical convolution, which allows us to evaluate (w) for any coordinate w E S and thus compute (R (w) for any continuous rotation R exactly. Nevertheless, the filter must be parameterized by finite number of learnable parameters p. We consider filters that are either axisymmetric p0;p), directional and separable (0, o; p) 0(0: p))((; p)) or directional (0, Φ; p) without further constraints. In all cases the filters are parameterized with equally spaced nodes along 0 and ø, with each node value a learnable parameter. To evaluate the filter for any continuous argument simply use linear interpolation. We localize the filter spatially by constraining it to be zero for angles above some cutoff Outoll-
3.5 DISCO TRANSPOSED SPHERICAL. CONVOLUTION
Dense pixel-wise predictions required for numerous deep learning tasks. Existing equivariant spherical CNNs cannot support dense predictions since they are too computationally demanding. A DISCO transposed spherical convolution can be defined to increase the resolution of internal feature representations in an analogous manner to the transposed convolutions considered in Euclidean U- Net architectures (Ronneberger al., 2015), supporting dense predictions. The DISCO transposed V S R. restricting rotations SO(3)/S0(2). is given by
spherical convolution f
(f O+O (w) = 1 f(w
= Σ (l.:))((
(6)
where we introduce the notation Rw (0) E SOG3/SO(2) for w (0, の) in order to make the dependence of the rotation on the integration variable explicit. The output of the transposed convolution the result of summing all filters rotated to each Ru with a weight of [[ws]. analogous to the Euclidean transposed convolution. Identical approximation and equivariance properties as the (non-traw moon-transposed DISCO spherical convolution hold.
COMPUTATIONALLY SCALABLE DISCO SPHERICAL CONVOLUTION
The DISCO convolution affords a computationally scalable implementation through sparse tensor representations. While we focus on the spherical setting of immediate practical interest, the same principles be applied to any compact group. We leverage sparse-dense tensor multiplication operators to compute the DISCO spherical convolution efficiently on hardware accelerators (e.g. GPUs, TPUs). By exploiting symmetries of the sampling scheme also describe how memory usage can be optimized. Automatic differentiation of sparse tensor operations that include learn- able parameters is not supported in either TensorFlow or PyTorch, hence derive custom sparse gradient representations to be used when training models.
4.1 SPARSE TENSOR REPRESENTATION
Representation and sampling. The DISCO spherical convolution for rotations SO(3)/S0(2), here denoted by the variable h f sb for brevity, may be written by the tensor multiplication
hg
Σ V jafio

---

where V is a tensor with elements Vs
wi), wR)) fr is the sampled spherical input sig- nal (collapsed to vector) with quadrature weights absorbed, and hj is the convolved output at = R ,(,))
(.0)),0 (0,)) or more simply, output coordinate (g), We adopt the sampling theorem of McEwen & Wiaux 2011
that all of the information content of bandlimited spherical signals can be captured in a finite set of 2L samples. We also adopt the associated quadrature weights but, discussed in Section 3.2 do not upsample signals, trading off a small approximation error in numerical integration to realize a significant reduction in computational cost. While the numerical integration is then no longer exact, it remains highly accurate.
Sparse representation. To compute the elements of V we first compute the continuously rotated coordinates, which denote tensors (Oji, Pp:)) R2 R and then pass them the continuous filter function to evaluate V,; = ( ; fj without any discretization error. For spatially localized filters, V will be sparse and we need only compute the rotated coordinates if y will be non- non-zero. We exploit knowledge of the support of the filters, given by Bewtoff to limit the number of rotated coordinates that need to be computed. Consequently. θ and & are also sparse tensors since we compute only coordinates where will be non-zero. The sparse representation of v can thus be computed efficiently and Equation [ For transposed spherical convolutions, V/ is simply replaced by v DISCO spherical convolutions and transpose convolutions can thus be computed efficiently by sparse-dense tensor operations that map well onto hardware accelerators, which for practical purposes can result in substantial enhance- ments in computational performance.
can can then be computed by a sparse-dense tensor multiplication.
Scalable computation. Empirically we find that the computational cost of the DISCO spherical convolution scales linearly in the number of spherical pixels, i.e. O(N) O as predicted; furthermore, not only does it scale more favorably than the most efficient alternative equivariant spherical convolution layer but its absolute cost is also considerably lower (see Appendix B for details). For example, for 4k spherical images we achieve a saving in computational cost of 10
Architectures. Efficient spherical implementations of common CNN architectures can then be con- structed by combining the DISCO forward and transpose spherical convolutions with pointwise non-linear activations and other common architectural features, such as skip connections, batch- normalization") multiple channels, etc. Note that in the architectures we consider we typically adopt depthwise separable convolutions (Chollet (017) for computational efficiency.
4.2
MEMORY OPTIMIZATIONS
Explicit sampling. For structured spherical sampling schemes many of the elements of V are re- peated, which opens the possibility of memory compression. While memory may be compressed for various structured samplings, we focus on the sampling scheme that we adopt. Specifically, we sample spherical coordinates by (fz, (p) (Tt/L, Tp/ L) for p E Z, 0 < t < L, 0 p 2L (McEwen & Wiaux 2011 Daducci et al. 2011), where recall L is the spherical harmonic ban- dlimit, which plays the role of resolution, yielding ( 1) pixels on the sphere. Incidentally, this sampling scheme maps onto the typical sampling of equirectangular 360 panoramic images well.
Compressed tensor. For clarity we write the input and output coordinates in terms of their indi- vidual spherical coordinates wi = (0,, Φp) and wj = (0p, (r'), respectively, i.e. i = (4, p) and j p'). It is then clear that the filter tensor is 4-dimensional tensor that may be written explicitly as Lr f Veyttp. By exploiting symmetries of our sampling scheme vi can be reduced to 3 dimensions by noticing that the two tensor elements pept and "roo(ppp')) are equal. This translation in the input coordinates by d:
follows since a rotation (d) is equivalent
(8) For our sampling scheme the set of all translated coordinates falls on the set of original coordinates, i.e (p-1/} = {p). and hence we can store Wrotjp-z' as Wrrotp. For a dense representation V. Veor has )(L²) elements; however, for localized filters this is reduced to O(L)) non-zero elements; hence, memory usage also scales linearly with number of spherical pixels.
V Verg'n - ((Z()) (Y,,)) (0xz)) (0,, (p)) = < (Ox) d))) ) = v (or(p-pp-
In order to ensure batch- maach-oormalization for sampled spherical signals is rotationally equivariant we adapt the standard nutch-normalization compute averages over the sphere, with appropriate sample weighting.
7

---

Efficient compressed sparse-dense tensor multiplication. The compressed version of Y can be used efficiently in the sparse tensor multiplication by re- re-writing Equation
as
hew Vegrtpst
Σ V Vootpp-)) ftp tp
Σ (eonpfippp)
Notice that the last line is a multiplication between fip shifted by p' in the p coordinate and M eoopp- Hence, we can implement the compressed tensor multiplication by looping over p' and simply shift- ing the input in p. which is considerably more efficient than repeatedly shifting the elements of sparse tensor. To support multiple channels we adopt depthwise separable convolutions (Chollet) 2017) not only for computational efficiency, but also for memory efficiency.
Scalable memory usage. Empirically we find that when incorporating the optimizations discussed above memory requirements of the DISCO spherical convolution indeed scale linearly with number of spherical pixels, i.e. as O as predicted, providing considerable saving over the most eff- cient alternative equivariant spherical convolution layer (see Appendix B for details). For example, for 4k spherical images we achieve a saving in memory of 10a.
4.3 SPARSE GRADIENTS
Deep learning models are typically trained by optimization algorithms that gradients, which are computed efficiently by automatic differentiation. However, automatic differentiation of sparse tensor operations that include learnable parameters, in the DISCO convolution, not supported either by TensorFlow or PyTorch. Thus, we implement custom sparse gradients (see Appendix C
5 EXPERIMENTS
We first demonstrate the equivariance properties of our scalable DISCO spherical convolutional layers, before tackling number of seneeeprrdiiiionn benchmark problems with spherical CNNs built using our DISCO framework (implemented in the CopernicAI code). We achieve state-of-the-ar performance on all benchmarks. See Appendix for further details benchmark experiments.
5.1 EQUIVARIANCE TESTS
To test the equivariance of DISCO spherical convolutions we consider random test signals and com- pute the mean relative equivariance error between rotating signals before and after the convolution. For axisymmetric filters achieve excellent rotational equivariance with error 0.04%. For di- rectional filters, for which equivariance is asymptotic since we consider S0(3)/SO(2) rotations, we recover equivariance errors of 0.01% in the limit of latitudinal rotation = 0º As B increases, equivariance errors increase but, nevertheless, we observe only moderate equivariance error just few percent 100 (this compares favorably the equivariance error of 35% for spherical ReLU at 32: Cohen et al. 2018 Cobb et al. 2021). See Appendix D for further details.
5.2 ROTATED MNIST ON THE SPIIRE
We consider the benchmark of classifying randomly rotated MNIST digits projected onto the sphere CChen et al.) 2018) for three experimental modes NR/NR, R/R and NR/R, indicating whether the training/test sets, respectively, have been rotated (R) or not (NR). Results are presented Table f bandlimit I. = 1024. The DISCO franework performs well at high resolution and, moreover, similar performance achieved in all rotational settings, demonstrating excellent rotational invariance.
5.3 SEMANTIC SEGMENTATION
We consider the dense-prediction problem of semantic segmentation of 360" photos, using a com- mon backbone of a DISCO spherical convolutional residual U-Net architecture, and compare to existing benchmark results (note that previous experiments have been performed at different resolu- tions; hence we quote the effective resolution resoutiiv L determined by number of samples on the sphere).
8

---

Table 1: Results for spherical MNIST.
NR/NR
R/R
NR/R
Planur
98.3
53.9
14.3
Table 2: Results for 2D3DS.
DISCO-Axisymmetric
98.6
98.7
98.6
Model
mloU
Planar UNg
35.9
50.8
Table Results for
UGSCNN
ling
54.7
GaugeNett
Leee 0.3
2019
39.1
55.9
Model
mloU
mAce
L
SWSCNNs
HexRUNa
Zhang et si.
Balews
2019
4140
4344
43.3
58.7
58.6
72
Planar UNet
38,8
143
CubeNer
Stagenaai
UGSCNN Jiang et al.
2019
36.9
50.7
143
Mobius ow
MMlictel 04
43.3
60.9
HexUUNet
Zhang elal.
2019
43.6
52.2
143
Tangending eEder 2020
Tangenting [Eder
41.3
52.8
143
HaHoNet Stirera aarmaaaati 2021
43.3
53.9
256
DISCO-AERy AoD--kayymeneeeeoe
39.7
54,1
256
Planar UNt
44.6
52.6
286
DiSCO-Separable (Oars)
43.9
60.9
UGSCNN Jiang
2019
37.6
48.9
286
Directional (Ours)
45.2
61.5
236
HexUNet Zhang cl 9
48.3
57.1
286
DISCO- MiNOO-Dreciioml--ug (Ohrs)
45.7
62.7
256
Tangenting el
286
DISCO- (Ours)
48.3
59.3
256
DISCO--Separable-uug (Ours)
49.2
63.7
256
Table 4: Results for Pano3D,
Model
Parameters
Depth Error Metries
Depth Accuracy Metrics
<RMSE
#RMSLE
w:bbiRel
WSSRee
61.05
81.1
61.35
61.262
Planar UNet
27M
0,4520
0.1300
0.1147
0,0811
36.68%
60.59%
88.31
96.96%
(llbanis al. 2021
DISCO-Direetional (Ours)
658k
0.5063
0.1695
0.1109
0.0852
38.32%
62.12%
88.65%
97.29%
2D3DS dataset of indoor 360 photos. Results for the semantic segmentation of indoor scenes of the 2D3DS dataset (Armeni et al. (017) are presented in Table 2 Our DISCO-Directinal ar- chitecture outperforms the DISCO-Axisymmetric architecture, which we attribute to being due to its asymptotic SO(3) equivariance (that can be considered as "orientation aware") and having more expressive directional filters. We also include augmentation, which helps improve performance. We achieve state-of-the-art performance compared to all previous methods.
Omni-SYNTHIA dataset of outdoor 360 photos. Results for the semantic segmentation of out- door city scenes of the Omni-S Omni-SYNTHIA dataset (Ros et al., 2016) are presented in Table 3 We consider a Depaa-See architecture, with and without augmentation for ablation purposes. In both settings we achieve state-of-the -het--rrt performance.
5.4 DEPTH ESTIMATION
We consider the task of monocular depth estimation from 360 photos, tackling the Pano3D bench- mark (Albanis al., 2021) for the Matterport3D dataset (Chang al.] (017). We adopt the same backbone architecture as for semantic segmentation. Results are presented in Table Pano3D is a relatively new benchmark: the only previous results are those of Albanis et al. (2021). We achieve excellent performance with significantly fewer parameters (40x less). While we do not achieve the best performance across all metrics (which we suspect due the alternative model being trained with virtual normal loss that leverages surface normal information). we perform better on accuracy metries than error metrics and nevertheless achieve the state-of- of-the-art in six of nine metrics.
CONCLUSIONS
We have developed the DISCO (diserete-comiinuuus) group convolution that is simultaneously com- putationally scalable and equivariant. While our framework can be applied to any compact group, we specialize to the sphere. We present sparse tensor implementation with custom gradients that exhibits excellent rotational equivariance properties, is well-suited hardware accelerators (e.g. GPUs, TPUs), and achieves linear scaling in both computational cost and memory usage. We ap- ply our DISCO spherical CNN framework to numerous dense-predietee tasks, such semantic segmentation and depth estimation, achieving state-of- tfae-of-the-aa offtth--n art performance on all benchmarks.

---

ACKNOWLEDGMENTS
We thank Mayeul d' Avezac for general discussions regarding TensorFlow memory usage. We thank Chao Zhang for providing data processing code for the Omni-SYNTHIA benchmark. We thank Georgios Albanis and Nikolaos Zioulis for providing training details on the Pano3D benchmark.
REFERENCES
G. Albanis, N. Zioulis, Drakoulis, V. Gkitsas, V. Sterzentsenko, Alvarez, D. Zarpalas, and
P. Daras. Pano3D: A holistic benchmark and a solid baseline for 360 depth estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021.
I. Armeni, S. Sax, Amir R Zamir, and Silvio Savarese. Joint 2D-3D-Semantic data for indoor scene understanding, ArXiv Preprint, 2017. URL https://aaxii httpg//b///02/0000
W. Boomsma and J. Frellsen. Spherical convolutions and their application in molecular modelling. In Advances Neural Information Processing Systems, 2017.
M. M. Bronstein, Bruna, T. Cohen, and P. Veličković. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. ArXiv Preprint, 2021. URL http:://arxv..org/aas///1 13478
A. Chang, Dai, T. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song. Zeng, and Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In International Conference on 3D Vision (3DV), 2017. URL / 06151
Chollet. Xception: Deep learning with depthwise separable convolutions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 02357
O. Cobb, C. G. R. Wallis, A. N. Mavor-Parker, A. Marignier, M.A. Price, M. d' Avezac, and Jason McEwen. Efficient generalized spherical CNNs. In International Conference on Learning Representations, 2021. /
11661
T.S. Cohen and M. Welling. Group eqpivariiantcoonooltiooaa neworrs. In International Conference on Machine Learning, 2016. htlL:/ttpss///axii.orr/abbbb//b 07576
T. S. Cohen, M. Geiger. J. Köhler, and M. Welling. Spherical CNNs. In International Conference on Learning Representutions, 2018. URL httphtpss///aiii.orr/aaaass//sss
10130
S. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling. Gauge equivariant convolutional networks and the icosahedral CNN. In International Conference on Machine Learning, 2019. URLhttpss 7/arxiv. wwwpobgg////boeee/eo /900. 04615
A. Daducci, J. D. McEwen, D. Van De Ville, J. Ph. Thiran, and Y. Wiaux. Harmonic analysis of spherical sampling in diffusion MRI. In 19th Annual Meeting of the International Society for Magnetic Resonance in Medicine, 2011. URL htttth:ss///iiii wrw.v/orrg//baao abs/g///////////// 0269
J. R. Driscoll and D. M. Jr. Healy. Computing Fourier transforms and convolutions on the sphere. Advances in Applied Mathematics, 15(2), 1994.
M. Eder, M. Shvets, J. Lim, and J. Frahm. Tangent images for mitigating spherical distortion. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URLhtttss
//arxiv.org/abs/1912.. 09390
C. Esteves. Theoretical aseects of group equivariant neural networks. ArXiv Preprint, 2020. URL https://arxivv.rr/aa///000 05154
C. Esteves, C. Allen--lanchette, A. Makadia, and K. Daniilidis. Learning SO(3) equivariant repre- sentations with spherical CNNs. In Proceedings the European Conference Computer Vision (ECCV), 2018.
wttp:/wwwwrg/////////oo 06721
C. Esteves, A. Makadia, and K. Danilidis. Spin- spherical CNNs. In Advances in Neural Information Processing Systems, 2020. UR..hhttp:///rdvvoor///b/s///// 10731
10

---

G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021. URL https://arxivv www.aawwwww.r/b////.22.220
C. M. Jiang, J. Huang, K. Kashinath, P., P. Marcus, and M. Niessner. Spherical CNNs on unstructured grids. In International Conference on Learning Representations, 2019. URL
https://arxiv. org/ wrg.abgg////o/oooooo..ooog. 1901/191..11 02039
D. P. Kingma and J. L. Ba. ADAM: A method for stochastic gradient descent. In International Conference on Learning Representations, 2015. URL
1988
R. Kondor, Z. Lin, and S. Trivedi. Clebsch-Gordan nets: a fully Fourier space spherical convo- lutional neural network. In Advances in Neural Information Processing Systems, 2018. URL
https://arxiv. www.oog/////ooooooo/.ooooo. 1806180660000 09231
Kostelec and D. Rockmore. FFTs on the rotation group. Journal of Fourier Analysis and Appli- cations, 14(2), 2008.
J. D. McEwen and Y. Wiaux. A novel sampling theorem the sphere. IEEE Transactions Signal Processing, 59(12), 2011.
6298
J. D. McEwen, P. Vandergheynst, and Y. Wiaux. On the computation of directional scale-discretized wavelet transforms on the sphere. In Wavelets and Sparsity XV, SPIE international symposium on optics and photonics, volume 8858, 2013. http:/tss//aaxii.orr////////////
5706
D. McEwen, M. Büttner, B. Leistedt, H. V. Peiris, and Y. Wiaux. A novel sampling theorem on the rotation group. IEEE Signal Processing Letters, 22(12), 2015. URL htttps://ariiv.ot org/ abs/1508. 03101
J. D. McEwen, C. Wallis, and A. N. Mavor-Parker. Scattering networks on the sphere for scalable and rotationally equivariant spherical CNNs. In International Conference Learning Represen- tations, 2022. URL wttp://ttttss//aaai..oor/o////// wttp:/wwwwwwggbbb///....ooo
T. W. Mitchel, N. Aigerman, V. G. Kim, and M. Kazhdan. Möbius convolutions for spherical CNNs. ACM SIGGRAPH 2022 Conference Proceedings, 2022. URL http::sss///arxi org/ abs/2201. 12212
N. Perraudin, M. Defferrard, T. Kacprzak, and R. Sgier. DeepSphere: Efficient spherical convo- lutional neural network with HEALPix sampling for cosmological applications. Astronomy and Computing, 27, 2019. URL URL httpp:://aaxi...rr/////////
12186
I. Z. Pesenson and D. Geller. Cubature formulas and discrete Fourier transform on compact mani- folds. Developments in Mathematics, 28, 2011. URL /tsLitsts///aaxioorr///aaa
1.990
O. Ronneberger, Fischer, and T. Brox. U-Net: Convolutional networss for biomedical image seg- mentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. URL http:://rxxv. httt:://axxiv.org/abb///// org/abs/1505. 04597
G. Ros, I.. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez. The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
M. Shakerinava and S. Ravanbakhsh. Equivariant networks for pixelized spheres. In Proceedings of the 38th International Conference on Machine Learning, 2021. URL hnttps://arxiis org/abs/ 2106. .66622
C. Sun, M. Sun, and H. Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. URL httphhtts////ii...tttt http://rrxii.o.r//aa///11111 11498
Y. Wu and K. He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. httpps://rxxiv.orr/dbbae
08494

---

C. Zhang, S. Liwicki, W. Smith, and R. Cipolla. Orientation-awaa semantic segmentation icosa- hedron spheres. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
1907. 12849
A SAMPLING THEOREMS AND QUADRATURE ON THE SPHERE
Signals on the sphere E (S22) may be decomposed into their harmonic representations by
0)
YemYemll (0, 6),
(10)
t= m=
with spherical harmonic coefficients given by
fem -f F(0, () Yim ())
d0(
(11)
where Yem denote the spherical harmonic functions of natural order and integer degree |m| s l. Consider signals on the sphere bandlimited at L, such that fem = O for all é 2 L. In this setting the summation over & in Equation 10 may be truncated at L
1-1
f(0,0) = Σ Σ
femYen (0, 6).
(12)
t= m=-
A sampling theorem the sphere (e.g. Driscoll & Healy 1994 McEwen & Wiaux 2011) states how to sample a band-limited function [00 () at a finite number of locations, such that all of the information content of the continuous function is captured. Since the sphere S is compact mani- fold, its Fourier space is discrete. All of the information content of a continuous bandlimited signal thus captured in its finite of harmonic coefficients fem (for once the finite set of fem are known, the function can be evaluated for any continuous coordinate through Equation| 12 sampling theorem on the sphere is thus synonymous with the exact cal- culation of the spherical harmonic coefficients Sem of the continuous function from its samples. Consequently, sampling theorems effectively encode, often implicitly, an exact quadrature rule for evaluating the integral of band-limited function on the sphere. In many cases it useful to have an explicit representation of the quadrature rule associated with a given sampling theorem.
L and [m] S f). Indeed,
In this work we adopt the sampling theorem the sphere McEnn & Wiaux (2011). Specifically, we adopt a variant of the sampling scheme first considered in Daducci et (2.) that oversamples by one in both 0 and ò to recover samples that diametrically opposite and thus exhibit antipo- dal symmetry (see Section (.22 for explicit sample positions). This variant of the sampling scheme provides numerous practical advantages, such mapping closely onto the common equirect- angular sampling of 360 photos and videos and supporting dyadic downsampling and upsampling.
While the explicit quadrature rule for the standard sampling scheme of McEwen & Wiaux (2011) is presented therein, the quadrature rule for the variant with antipodal symmetry that we adopt in this work has not appeared any published literature; hence we present here. Consider the following integral of a function f(0, Φ) bandlimited at L:
21 Σ
I 1= f (00,))d((0,)
ΣV
f(O,, (p)g(0,.))
(13)
where q0,, Φ) denote quadrature weights such that the integral / is computed exactly from the sam- pled signal values f0,, o,). The derivation of the quadrature weights for the antipodally symmetric sampling scheme follows that given by (McEwen & Wiaux 2011 for the standard sampling scheme, with only some minor variations.
The derivation of the sampling scheme of (McEwen & Wiaux, 2011) is based the sphere to the torus through careful period extensions the function interest to the 9 domain [0, 2m): recall that the sphere 0 [0, r). Explicit quadrature weights can be determined by
extension of
12

---

folding back contributions from (It, 2T)) onto x0, function defined by sin on [0, π. and zero on (T, 2m):
x). Consider the Fourier transform of the periodic
tim/
m' = +1
w(m') L sin( exp(im 0))00
0,
m' odd, m' + #1 m' even
(14)
2/(1 m²²) )
The bandlimited representation of this function is given by
wy (,,
Σ
w (m)) exp(im'(8.)
(15)
(note the lower index of the summation of - L). The quadrature weights of the antipodally sym- metric sampling scheme are then given by folding back contributions from (T, 2T) onto (0, RT) with appropriate normalization:
2T (2T.)2 (v.(0) + (1 (1--d](1--6)))1-)1((0-)-)))
g0,,, db) (0.))
(16)
For completeness we present the result above for signals with spin 8, although the current article consider scalar signals with s 0 only.
B COMPUTATIONAL COST AND MEMORY USAGE
We compute the theoretical computational cost and empirical memory usage of the DISCO spher- ical convolution, contrasting it to the most efficient alternative equivariant spherical convolutional layers of an axisymmetric convolution computed via Fourier space (Esteves et al. 2018 Cobb al., 2021 Note that alternative equivariant spherical convolutional layers that are more expressive are considerably more costly, both in terms compute and memory (Cohen et al. 2018 Kondor al.,
2018 Cobb et al. 2021
Without loss of generality we consider layer, channel, and batch size of For the DISCO spherical convolutional layer we consider axisymmetric filters with 4 nodes and Peutott 3 L. For the axisymmetric harmonic layer we parameterize the filters with 10 nodes in harmonic space and use linear interpolation, similar to Esteves et al. (2018).
For the DISCO layer we compute computational floating point (FLOP) cost via analytic calculations of the sparse tensor products and linear interpolation. For memory usage we explicitly construct the sparse 3J tensor and then count the number of non-zero values; hence, the evaluation memory usage is empirical rather than purely theoretical. For the axisymmetric harmonic layer compute computational cost and memory usage via analytical calculations of the tensor products used the precompute-bas based spherical harmonic transforms, convolutions and linear interpolation.
Theoretical computational cost and empirical memory usage results are presented in Fig. 2 For the DISCO spherical convolution both computational costs and memory usage scales linearly in the number of spherical pixels, i.e. as O(N) = ( Furthermore, not only does the computa- tional cost of the DISCO spherical convolution scale more favorably than the most efficient alterna- tive equivariant spherical convolution layer but its absolute cost is also considerably lower. For 4k spherical images achieve saving in computational cost of 10 and saving in memory usage of 1
Inspecting the theoretical computational cost, as considered above, has the advantage that abstracts away details of a particular implementation. its level of optimization, and the hardware on which the model runs. Wall-clock compute time, while not independent of the aforementioned factors, is nevertheless typically of practical intersest and so is also evaluated. Due to limitations of the sparse tensor-vector product support in TensorFlow we explicitly batch the p' index of Equation 9 For computational evaluation we consider a p' batch size of 128 and evaluate the total compute time over all p'. iee. over all p' batches. On an NVIDIA RTX 3090 GPU we observe a wall-clock compute time of 0.0302 + 0.0018, 0.0898 + 0.0025, and 0.3255 + 0.0043 seconds for resolutions of = 1024, = 2048, and = 4096, respectively, when averaged over 10 experiments. With further improvements in sparse tensor support expect efficiency gains in future.
13

---

10ah
Axd symmetiic harmanic layer (tastest atternative)
10
AMt-symmetric hamonic layer
(tastest ttemativ)
(Durs)
(Ours)
10
Memory (MB)
10
FUDPS
10
20
10a
10a
10a
10a
Reduction
10a
10a
10a
10:
161
10a
Bendlimit
Bendlimit
(a) Computational cost
(b) Memory usage
Figure 2: Comparison of computational cost and memory usage of the DISCO spherical convolu-
tion and the most efficient alternative equivariant spherical convolution (an axisymmetric harmonic convolution). The DISCO spherical convolution achieves linear scaling the number spherical pixels, i.e. O(L²), for both computational cost and memory usage.
C CUSTOM SPARSE GRADIENTS
Since automatic differentiation of sparse tensor operations that include learnable parameters, in the DISCO convolution, is not supported in either TensorFlow or Py Torch we implement custom sparse gradients.
Consider the sparse tensor representation of the DISCO convolution
haj Σ V
(17)
where, for completeness, we also indicate the data instance by index Since CNN architectures typ- ically include many convolutional layers, with outputs of one layer providing the inputs for another, to compute gradients the loss function with respect to the parameters the model it is necessary to compute the gradient of the convolution with respect to both input and dense filter tensor:
Dh 1 dooss Ofrv
Thai 2. pv
fae
(18)
Algorithm
Function compute custom sparse gradients in TensorFlow.
Input: The upstream gradient
Th input signal fa and sparse filter tensors V TP 0o dh 0h 0f
do 0h
Output: Activation gradient
and sparse kernel tensor gradient
0h dV
1. (o dh Th 05 d
tf. sparse. spwrse_deenseeeeee aidensemmatmull V/' 0g
Oha
j = Wiindices
3. f/ tf. gather (aaterrJaa i')
Oa'
dg Thd 3
4.
tf tf. gather
Tha
0o ah
Oo' Ohd"
Σ tf. www.aahh..ulle. multipl
fa
0h 0y
14

---

α
α
()) R = ((a) E SO(3) rotation
(b) R = ((a)Y E (0(3)/SO(2) rotation
Figure 3: SO(3) and 50(3)/S0(2 rotations.
Pointwise activation functions a--) are typically included in CNN architectures to introduce non- linearity in an equivariant manner. Gradients can traced through activation functions by
Oa 0h Öh 0f d'
do That
da Sha'
Σ
V Da
(19)
di Jhas Ofav
Do Ōh
do Ob
do
Σ
Sax.
(20)
Th Ov
ahaj 2V y
abay
Equation 19 a sparse-dense tensor multiplication but with the V matrix transposed in input and output coordinates. Thus, we can use similar method to Section 4.2 to convert to compressed sparse tensor multiplication. Note also that we need only update the non-zero values V Vis. for which know the corresponding indices. Hence, need only compute Equation 20 for the sparse set of indices j't'. Pseudo code of our custom sparse gradient implementation is shown in Algorithm[]
We validate our implementation of custom sparse gradients presented here against gradients for dense tensor computed by automatic differentiation highly inefficient approach where the dense tensor has many entries set to zero), finding agreement to machine precision.
D EQUIVARIANCE TESTS
To test the equivariance of the DISCO spherical convolutions we perform similar equivariance tests to Cobb et al. 22021 Appendix D) but with rotations restricted to SO(3) sual illustration of (3(3) and S0(3)/SO(2 rotations in Fig. 3}. To demonstrate the equivariance of DISCO layers at high resolution perform equivariance tests for L E {128, 256, 512, 1024).
(we provide vi-
N with harmonic coefficients sampled from the standard normal distribution and NQ 20 random rotations { sampled uniformly SO(3)/SO(2). Note that for axisymmetric filters, this is equivalent tation since the first rotation about its axis leaves the filter unchanged. To demonstrate asymp- totic SO(3) equivariance for directional filters (see Section 3.2). we consider rotations of the form Qj = Y (7)) for random aj, ? and constant _ E {00 5", 10%). J. We measure the ex- tent to which our DISCO spherical convolution operator L (S + (2(2 is equivariant by evaluating the mean relative equivariance error
We consider N2 = 20 random spherical signals {
random SO(3)
No (D(2))) 2f(P(A))
(D(QN), 2(P()) 1
(21)
No
ID(2,)))
i=
resulting from pre-rotation of the signal, followed by application of D. opposed to post-rotation after application of D, where the operator norm (s ()9(9))
is defined by the inner product on the sphere
15

---

Table 5: Mean relative equivariance error expressed as a percentage (i.e. 100 x e) for axisymmetric and directional filters (best and worst cases are shown, where worst is shown in gray inside brackets).
Resolution I.
Axisymmetric filters
Directional filters
B
3
10"
128
0.01 = 0.01 (0.27 0.02) 0.02 0.00 0.003
0.86 = 0.05 (1.11 0.00)
3.27 + 0.16 (3.85 2.20)
256
0.04
0.01 (0.33 1 0.03)
0.01 0.00 0.00)
0.83
0.02 (1.18 0.72)
3.24 1 0.08
(4.53 1 2.31)
512
0.04 0.01 (0.48 4: 0.03)
0.00 0.00 (0.01 0.00
0.83
0.01 (1.31
0.85)
3.23
1 0.05
(4.30
1 2.43)
1024
0.01 (0.55 0.00) 0.00 0.00
0.82 = 0.00
(1.04
0.69)
3.23 + 0.02
(3.93 2.22)
We do not impose filters be bandlimited. As the smoothness of the filter reduces, spherical alias- ing is increased, and equivariance errors are increased. We therefore consider best and worst cases to bracket equivariance errors. For the worst case scenario we allow filter parameters to be ran- dom and consider Acuanr = 5 L, with nodes along 0 and ø. For the best case scenario we use the same nodes and cutoff but consider smoother filters, with the axisymmetric filter given by the smooth function ris (0) exp( (senone /0e )² at node positions and the directional filter by (0) (Φ)). Equivariance errors computed in this manner for the DISCO spherical convolution are presented in Table 5
For axisymmetric filters we achieve excellent rotational equivariance. For directional filters, for which equivariance is asymptotic since we consider SO(3) /SO(2) rotations, we achieve excellent equivariance in the limit of latitudinal rotation 8 = 0-. As 3 increases, equivariance errors increase as expected but, nevertheless, we observe only moderate equivariance error of just few percent at 3 10 (this compares favorably to the equivariance error of ~ 35% for a spherical ReLU at L = 32: Cohen et al. 2018 Cobb et al. 2021
E ADDITIONAL INFORMATION ON BENCHMARK EXPERIMENTS
E. MNIST
Data. We project the MNIST digits onto the sphere at resolution L = 1024, using the same projec- tion as in Cohen et al. (0118). We also standardize the data by subtracting the mean and dividing by the standard deviation (both computed from the entire dataset).
Architecture. We consider two models consisting of 7 convolutional layers, one with DISCO spher- ical convolutions and one with standard planar convolutions for comparison. Convolutional layers are followed by a global integration over the sphere (Cobb et al.) (02T) for the spherical model and a global pooling for the planar model, which are followed by 2 dense layers, where all layers include ReLU activations. As typical for CNNs, we continually reduce the resolution of feature maps and increase the number of channels through the model. The bandlimit and number of chan- nels (L, channels) considered for each convolutional layer are as follows when progressing through the model: (1024, 4), (512,8), (256, 16). (128, 32). (64, 64), (32, 128), (16, 256). For the DISCO model we consider a hybrid model containing DISCO spherical convolutions for early layers that operate at high-resolution and harmonic convolutions for subsequent layers operating at resolutions L < 64. For both the spherical and planar model we adopt depthwise separable convolutions for channel mixing for layers resolutions L S 64 and the standard channel treatment otherwise. For the DISCO spherical convolutions we adopt axisymmetric filters with Acusot = 3T / L and 4 nodes along& 8, while for the harmonic convolutions we adopt axisymmetric filters parameterized with 10 nodes in harmonic space, similar to Esteves et al. (2018). For the planar model we adopt 3x3 con- volutional filters. For both models the two proceeding dense layers contain 256 and 10 neurons, respectively. The resulting DISCO model has 449k learnable parameters, while the planar model has 4k The architectures of the spherical and planar models are closely matched; however, since there are differences in the structure of the spherical and planar layers this results in slight differ- ence in number of learnable parameters. In any case, this will not induce any difference in the main results of [
Training, We train for 10 epochs usign the ADAM optimizer (Kingma & Ba 2015). with a learning rate of 0.001 and batch size of 8.
16

---

Toput (256, 4)
(256, 32)
(128, 64)
(64, 128)
(32, 256)
oResBlook
oRtsBlack
oResBlock
oResBlock
oReaBlock
concat
cotica
comcat
coneat
(16, 256)
Outpus (206), 16)
RowBookk
(256, 64)
ReeBBook CUpCoou
(1225 125)
ResBliokk aUpCome
(644, 206)
RoaBlack
(82, 112)
pCoos
UpCome
DownCoon
UpCoor
EP
S? coav
a]
(L/2, a)
(L,
(2L, a)
BeeBlock
[L,, a)
(4,, a)
SP coune
(4,
conv
ReLU
(4, by
JRNoRelLU
RRNoRel
aHN
(4, b)
Figure DISCO spherical convolutional residual U-Net architecture.
RGB
Unknown
Beam
Bookcase
Ceiling
Chair
Clutter
Truth
Column
Door
Sofa
Table
Wall
Window
DISCO
Invalid
Figure 5: Example predictions for semantic segmentation 2D3DS data.
F.2 SEMANTIC SEGMENTATION: 2D3DS
Data. The 2D3DS dataset (Armeni al. 2017) consists of 1413 equirectangular RGB-Depth indoor 360 images, with each pixel belonging to one of 14 classes. We downsample the images to I, 256 with bilinear interpolation for the RGB-D images and nearest-neighhor interpolation for the classes. We standardize the RGB-D data by subtracting the mean and dividing by the standard deviation channel-wise. We also experiment with image augmentation via random mirroring, i.e. switching the co-ordinate + -.. which improves performance little (see Table 2
Architecture. We consider a U-Net style architecture with residual blocks as illustrated Fig- ure 4 Convolutional layers are typically composed of depthwise-separabl DISCO spherical con- volutions. Downsampling layers are DISCO spherical convolutions but with outputs at half the resolution, i.e. L/2. For upsampling layers we use transposed DISCO spherical convolutions sam- pled at double the resolution, i.e. 2L. We use spherical bauch-normalizations and consider similar residual blocks to Jiang et al. (2019). Note in the final layer we consider 14 channels to represent the different classes and also consider invalid channel. We consider models with axisymmet- iitreciioal-separrate
ric DDISCO-Axsymmette DOISCO-Directions filters for all convolution layers. For axisymmetric filters we set f = 3T L and 4 nodes along for L. > 64, else standard harmonic convolutions with 10 nodes in har-
(DISCO-Separable)
unconstrained directional
17

---

RGB
Vold
Building
Road
Sidewalk
Truth
Ferce
Vegetation
Pole
Traffic
Pedestrian
DISCO
cycle
Lanemarking
Figure 6: Example predictions for semantic segmentation of
data.
monic space, similar to Esteves al. (2018). For separable filters we set Octott = 3T/ and 4 nodes along 0 and σb Finally, we also use unconstrained filters where the nodes are square 3 x 3 pixel grid projected onto the sphere. The projection such that the central node is on the pole
0 and the edge nodes are at f = 7. Φ E = ) and A v2n Φ { + Sx }. We use bilinear inter- L.
polation on this regular grid to obtain values outside the nodes. For the DisCO-Directiona model we also consider the case with the mirror augmentation discussed above The models DISCO-AxIO Axismmmris,mmetrric, DISCO-Separe Sepaa-eee and DISCO-Directional have 1.78M, 1.47M and 1.68M learnable parameters respectively.
Training. We train for 120 epochs using the ADAM optimizer (Kingma Ba 2015) with a batch size of 8 and learning rate of 0.01 which is reduced plateau by 0.1 with patience 8. We use class-wise weighted cross-entropy loss to balance the class examples. We also weight the loss and metrics by the area of each pixel on the sphere. We the same 3-fold split for validation as in Jiang et al. (2019). Example segmentation results are shown in Figure
E.3 SEMANTIC SEGMENTATION: OMNI-SYNTHIA
Data. The Omni-SYNTHIA dataset (Ros et al. 2016) consists of 2269 panoramic RGB images of outdoor city sceneries with each pixel belonging to one of 13 classes. We downsample the images to = 256 with bilinear interpolation for the RGB images and nearest-neighhour for the classes. We also experiment with copy-and paste augmentation (Ghiasi et al. (021), which is straightforward to implement for our sampling scheme and improves performance a little (see Table 3 This is implemented by choosing random training image, then choosing random small object of pole, traffic sign, pedestrian, or bike and pasting this object on different training image for both RGB and segmented values. The pasted objects are also translated randomly by ø - n /L which can be performed straightforwardly for our sampling scheme; we also do random mirroring of the pasted object.
We consider the same backbone of a DISCO spherical convolutional residual U-Net architecture as for the 2D3DS data (as in Figure but with 13 output channels (+1 channel for in- valid) and 3 channels for the input. Here we consider only separable directional filters parameterized identically as for 2D3DS MASCO--Separalle We also consider the case with the sopy-and-paste augmentation discussed above (DISCO-Separablee parameters.
Architecture.
The resulting model has 1.47M learnable
Training. We train for 70 epochs using the ADAM optimizer Kingma & Ba 20155 with batch size of 8 and learning rate of 0.01, which is reduced on plateau by 0.1 with patience We use a class-wise weighted cross-entropy loss to balance the class examples. We also weight the loss and metrics by the area of each pixel on the sphere. We use the same training/test split as in Zhang et al. 2019). Example segmentation results are shown in Figure 6
18

---

RGB
6
Truth
4
2
6
DISCO
4
2
Figure Example predictions for depth estimation of Pano3D data (depth plotted in meters).
E.4 DEPTH ESTIMATION: PANO3D ON MATTERPORT3D
Data. The Matterport3D dataset (Chang et al. 2017) contains 7907 spherical RGB images from which we predict spherical depth maps. We downsample the RGB-Depth images to L 512 with bilinear interpolation.
Architecture. We consider the same backbone of DISCO spherical convolutional residual U-Net architecture as for the semantic segmentation problem but adapted for depth estimation. The archi- tecture is similar to Figure 4 but we replace each ResBlocks simply by DISCO spherical convolu- tion. All resolutions are doubled to start at resolution L = 512 for the input. We consider 3 input channels and 1 output channel (depth). The number of channels for the down leg of the model are (64, 64, 128, 256, 512), which is reversed for the up leg. For the upsampling layers we increase the bandlimit by 2 using nearest neighbor interpolation. We adopt this upsampling rather than trans- posed convolutions simplify transfer learning (discussed below) from planar to spherical convo- lutions. Due to the higher resolution we use smaller batch sizes, which means batch- normalization is unsuitable and apply a group- nouuppoomaliztiion and ReLU after every DISCO spherical convolution. We consider only unconstrained directional filters segmentation problems, as described above. The resulting model has 658k learnable parameters.
therefore group-nort normalizaiin (Wu & He 2018) with 16 groups. Here
adopting the same parameterization as for the
area weighted é loss and the same weighted metrics in Albanis et al. (202)) We use the same train/test/validation split as Albanis et al. (021). For faster training transfer learning by first training the above architecture with planar convolution layers with 3 x 3 filters, for 60 epochs using the ADAM optimizer (Kingma & Ba 2015). with batch size of 2 and learning rate of 0.0001. The projected on the sphere for the unconstrained directional DISCO filters. After the weight transfer, the DISCO model is then trained for further 60 epochs with the same hyperparameters. Example depth estimation results are shown in Figure]
Training. We
filter weights are then easily transferred to the filters
19


Reviews:
[{"review_id": "pon4h_EPeHD", "reviewer": "Reviewer_vRza", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a kind of scalable and equivariant spherical convolution by applying continuous filters to discrete spherical inputs. The authors claim that this method is more computation- and memory-efficient than previous works.", "summary_of_the_review": "This idea is trivial and the writing should be significantly improved. Also, their experiments cannot support their claims well.", "strength_and_weaknesses": "Strength:\n1. This method combines the advantages of continuous and discrete spherical convolutions, and is both rotation-equivariant and scalable. \n\nWeaknesses:\n1. This paper contains many long sentences, reads rather awkwardly and even has many fault wordings, e.g., \n\"the sphere is a homogeneous space with global symmetries on which act elements of the group of 3D rotations SO(3)\"\n\"The DISCO approximation of Equation 3 is highly accurate for bandlimited signals, which real-world signals can be well approximately by for sufficient bandlimit.\"\n\"The filter is represented continuously so also does not introduce any error.\"\nI think the writing should be significantly improved.\n\n2. To my understanding, the main idea is in Equation (3), i.e., applying continuous filters \\psi to the discrete inputs f, and then implementing group convolutions, which is rather trivial.\n\n3.  Restricting rotations to SO(3)/SO(2) will degrade the equivariance over arbitrary 3D rotations to that over z-rotations. Actually, the equivariance w.r.t z-rotations can be easily achieved by employing oriented-aware spherical CNNs, such as HexUNet. \n\n4. The baselines of spheical MNIST and Pano3D are both planar CNNs, which are very weak. The result for 2D3DS is also not the state-of-the-art, as the performance is much weaker than [1].\n\n5. The paper claims that this method is both memory- and computation-efficient. However, there is no practical experimental evaluation about that.\n\n[1] Eder et al. Tangent Images for Mitigating Spherical Distortion. CVPR 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d6Aoo9578Y9", "reviewer": "Reviewer_ef9H", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "There are now a wide crop of Spherical CNN models, which can be classified along several different axes e.g. fully Fourier/fully real/partly real and partly harmonic; or continuous models and discrete models. If we focus on the latter, existing continuous models guarantee rotational equivariance, while being computationally expensive (and this varies depending on how much equivariance error is tolerated). On the other hand, discrete approaches (such as those that represent spherical signals as a graph) are more computationally attractive, but don't handle equivariance in a satisfactory manner. Part of the expense of the continuous approaches comes from the prohibitive cost of the spherical/rotational Fourier transform. This paper proposes a framework that attempts to combine the advantages of both of these styles of models, leading to the proposed spherical CNN scaling up much better than existing models, and also becoming capable of doing dense prediction. \n\nThe paper first describes the basic idea of the continuous approach (exemplified by Cohen et al, Kondor et al., Esteves et al. and Cobb et al.), while describing some of the issues concerning sampling and the computational load. It then briefly discusses some of the discrete models (which tend to be graph based and are inherently limited in the symmetries that they can capture). It is also mentioned that recent scattering-based methods still don't manage to work quite well with high-resolution images -- something that I am in full agreement with. The approach of the paper is described, at a high-level, in sectoin 3.1. The key equations are equations 3 and 4. The main innovation of this section is to replace the integral by a quadrature (possible source of error), which is evaluated at some sample positions. However, the fllter, and its transformation is kept continuous -- and consequently does not lead to errors. The quadrature error is minimized (or in principle can be avoided) by appeal to an interesting sampling theorem from a decade ago. After this the authors describe a few area where the computational load can be further chopped off (and permit linear scaling with the pixels). This includes reducing the space of rotations SO(3) to SO(3)/SO(2) (the authors keep using equivariance to SO(3)/SO(2) -- which I am not sure what it means since it is not a group, it is quotient space). Filter parameterization also uses simple linear interpolation after sample points are selected and reduces the load a bit further (although possibly being a source of error). The authors also detail how they leverage the sparse representations that are obtained for further computational (and memory) gains. The memory gains are also augmented by making use of a symmetry in the sampling procedure. Eventually, the authors also describe an extremely well-engineered system that is able to scale to levels unlike other spherical CNNs and performs wells on some experiments (including dense segmentation). ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "- The paper is very well motivated and solves an important problem -- that is of scaling spherical CNNs so that they become useful for larger images and for dense prediction. \n- The main conceptual contribution is quite simple, but still significant. \n- The system is implemented very efficiently, and the engineering effort and attention to detail is admirable. \n- The dense prediction results are good, and unlike other spherical CNNs. \n- Paper is also very well written. \n\nWeaknesses\n- The paper seems to make no indication about code availability. I will definitely consider this to be a negative.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Pus7FfFzUv", "reviewer": "Reviewer_kRdr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel form of group convolution called Discrete-Continuous (DISCO) group convolution, which is targeted at making group convolutions scalable and at the same time retain equivariance properties. DISCO discretizes the input signal space while the filters and output space are maintained to be continuous. If the input signals are band-limited and there are sufficient samples for reconstruction, then the group convolution can be performed exactly using DISCO convolution. This means that the computation is much faster and also, the operation is exactly equivariant to group action. The authors then discuss the special case of signals on the $S^2$ sphere with the natural $SO(3)$ action as well as $SO(3)/SO(2)$ transformation. They also discuss in detail how to build a computationally scalable DISCO convolution using sparse tensors and other memory optimization. Experiments are performed on Rotated-MNIST recognition and semantic segmentation and depth estimation for omnidirectional images. ", "summary_of_the_review": "I think this paper has several very good ideas, implementation and experimental results. There are some crucial details that I am not able to understand by reading the paper which I hope the authors can explain. For now, I am giving this paper a \"marginally above threshold\", but would be more than happy to increase my rating to an accept or a strong accept depending on how the authors address the weaknesses I have listed. \n\nUPDATE AFTER AUTHOR RESPONSE:\n\nI thank the authors for their explanations and improvements to the paper. Based on this and the other reviews and response, I am recommending acceptance of the paper.", "strength_and_weaknesses": "Strengths:\n\n1. The paper addresses an important question which is how to make group convolution scalable while maintaining equivariance. Earlier attempts at scalability required full discretization which sacrificed equivariance. In contrast, here, only the band-limited signal is discretized to a level where all the information is still maintained, and the filters and the outputs are continuous. The authors then show that this maintains equivariance while making the computation much cheaper as expensive Fourier transforms are no longer required. This is a novel direction.\n\n2. These general ideas are made useful in practice for spherical images by presenting DISCO version of the spherical and $SO(3)$ convolution.\n\n2. The theoretical speedup is huge and the computation is linear in the number of pixels when the filters have local support and are small. \n\n3. The authors describe the implementation in detail using sparse tensor which shows linear scaling with number of pixels. This is very good result, in my opinion. And as the authors say, for 4k images, this shows a theoretical speedup of $10^9$.\n\n4. The authors also describe smart tricks for memory optimization by exploiting the patterns in the filter arrays and the convolution operation leading to a $10^4$ improvement. \n\n5. Experimental results on equivariance tests clearly show that the method maintains nearly perfect equivariance. The error mainly comes from having to eventually discretize the integral in order to compute the convolution on a computer. \n\n6. Although I am not familiar with the baselines, results on semantic segmentation and depth estimation datasets show very good results using DISCO. I am especially happy to see the results on depth estimation performing comparably with a U-Net that uses 40x the number of parameters.\n\nWeaknesses:\n\n1. There is a crucial detail that I am not following in the method which I hope the authors can explain. I understand that the sampling theorems on a compact group or its homogeneous space (e,g, sphere) show how many samples are needed to preserve all the information after discretization. But I don't see how this leads to Eq. (3) (and Eq(4)), Given a discrete signal, I would think we would still need to first reconstruct the underlying continuous signals by some low-pass filter (convolving with some form of a sinc function?) on the space for performing the convolution. I am not sure I understand how the result in Eq. (3) and Eq. (4) is possible. \n\n2. While the speedup and memory improvements are fantastic, they are still based on theoretical analysis. Can the authors show computation times observed empirically?\n\n3. Experimental results should ideally have error bars and statistical significance results. \n\n4. I urge the authors to try and release their software implementation to make the experiments easily reproducible, but of course, this is not required.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FnXXVUFjSwt", "reviewer": "Reviewer_D1fT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose the hybrid discrete-continuous group convolution on sphere that possess both equivariant property and computational scalability. The sparse tensor representation is used in implementation to further save computation cost and memory. Experiments are performed on spherical MNIST, Omni-SYNTHIA, 2D3DS and Pano3D, which shows some improvements over previous works. \n\n", "summary_of_the_review": "In summary, this paper is well written, but it still needs to be refined carefully. I recommend marginally below the acceptance threshold for this paper. ", "strength_and_weaknesses": "Two main advantages of this paper are as follows:\n\n1, The paper is well written and the key contribution is easy to understand. \n\n2, The analysis of equivariance error, computational cost and memory usage are provided to support the efficiency of the methods.\n\nTwo main disadvantages of this paper are as follows:\n\n1, In experiments, the state-of-the-art claim seems unprecise. For example, in 2D3DS dataset, the HoHoNet (https://arxiv.org/pdf/2011.11498v3.pdf) even achieve mIoU of 52%. Besides, the improvement is minor without data augmentation compared to previous methods.\n\n2, The sufficiency of the restriction from SO(3) to SO(3)/SO(2) equivariance need to be further developed. It seems that the method can’t capture the fully SO(3) equivariance as continuous approaches, which may lead to performance degradation on dataset defined on sphere with random rotation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster