Fully Steerable 3D Spherical Neurons

Emerging from low-level vision theory , steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations . In our work , we propose a steerable feed-forward learning-based approach that consists of spherical decision surfaces and operates on point clouds . Focusing on 3D geometry , we derive a 3D steerability constraint for hypersphere neurons , which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets . Exploiting the rotational equivariance , we show how our model parameters are fully steerable at inference time . We use a synthetic point set and real-world 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and , after online optimization , invariant class predictions for known point sets in unknown orientations . 1 INTRODUCTION . We present a novel feed-forward model consisting of steerable 3D neurons for point cloud classification , an important and challenging problem with many applications such as autonomous driving , human-robot interaction , and mixed-reality installations . We achieve the steerability by using geometric neurons ( Melnyk et al. , 2021 ) , leading to a rotation equivariant network . Besides becoming a geometrically explainable approach , further benefits of the proposed method include rotation invariant classification and , therefore , lowered data augmentation requirements during learning . We make use of a conformal embedding to obtain higher-order decision surfaces . Following the motivation in the recent work of Melnyk et al . ( 2021 ) , we focus on 3D geometry and spherical decision surfaces , arguing for their natural suitability for problems in Euclidean space . We show how a spherical neuron ( classifier ) , i.e. , the hypersphere neuron ( Banarer et al. , 2003b ) or its generalization for 3D input point sets — the geometric neuron ( Melnyk et al. , 2021 ) — can be turned into a steerable neuron . We prove that the aforementioned spherical neurons in any dimension require only up to first-degree spherical harmonics to accommodate the effect of rotation . This allows us to derive a 3D steerability constraint for the spherical neurons and to describe a recipe for creating a steerable model from a pretrained spherical classifier . Using the synthetic Tetris dataset ( Thomas et al. , 2018 ) and the skeleton data from the UTKinectAction3D dataset ( Xia et al. , 2012 ) , we first verify the derived constraint and check its stability with respect to perturbations in the input . We further conduct an experiment in a realistic setting , where we initialize the steerable model parameters using a noisy or imperfect prediction of the transformation applied to the input and optimize the parameters in an unsupervised way . The contributions of our work are as follows : ( a ) We prove that the activation of spherical neurons on rotated input only varies by up to first-degree spherical harmonics ( Section 4.1.1 ) . ( b ) Based on a minimal set of four spherical neurons that are rotated to the corresponding vertices of a regular tetrahedron ( Section 4.1.2 ) , we derive the main result of our paper ( Section 4.1.3 ) – a 3D steerability constraint for spherical neurons ( 13 ) . ( c ) We propose a method to turn a trained spherical classifier into a fully steerable model ( Section 5.2 ) that produces predictions invariant to 3D rotations , even in the presence of noise ( Section 5.3 ) , which we verify experimentally using both synthetic and real 3D data . 2 RELATED WORK . 2.1 STEERABILITY AND EQUIVARIANCE . Steerability is a powerful concept from early vision and image processing ( Freeman et al. , 1991 ; Knutsson et al. , 1992 ; Simoncelli et al. , 1992 ; Perona , 1995 ; Simoncelli & Freeman , 1995 ; Teo & Hel-Or , 1998 ) that resonates in the era of deep learning . The utility of steerable filters and the main theorems for their construction are presented in the seminal work of Freeman et al . ( 1991 ) . Geometric equivariance in the context of computer vision has been an object of extensive research over the last decades ( Van Gool et al. , 1995 ) . Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator . For this reason , Lie theory can be used to to study steerable filters for equivariance and invariance ( Reisert , 2008 ) . Nowadays , equivariance-related research extends into deep learning , e.g. , the SE ( 3 ) -equivariant models of Fuchs et al . ( 2020 ) , Thomas et al . ( 2018 ) , and Zhao et al . ( 2020 ) , and the SO ( 3 ) -equivariant network of Anderson et al . ( 2019 ) , as well as Marcos et al . ( 2017 ) and Kondor et al . ( 2018 ) . Also steerable filter concepts are increasingly used in current works , e.g. , Cohen & Welling ( 2016 ) considered image data and proposed a CNN architecture to produce equivariant representations with steerable features , which involves fewer parameters than traditional CNNs . They considered the dihedral group ( discrete rotation , translation , and reflection ) , and the steerable representations in their work are proposed as formation of elementary feature types . One limitation of their approach is that rotations are restricted to four orientations , i.e. , by π/2 . More recently , Weiler et al . ( 2018b ) utilized group convolutions and introduced steerable filter convolutional neural networks ( SFCNNs ) operating on images to jointly attain equivariance under translations and discrete rotations . In their work , the filter banks are learned rather than fixed . Further , the work of Weiler & Cesa ( 2019 ) proposed a unified framework for E ( 2 ) -equivariant steerable CNNs and presented their general theory . The steerable CNNs for 3D data proposed by Weiler et al . ( 2018a ) are closely related to our work . The authors employed a combination of scalar , vector and tensor fields as features transformed by SO ( 3 ) representations and presented a model that is equivariant to SE ( 3 ) transformations . They also considered different types of nonlinearities suitable for nonscalar components of the feature space . The novel SE ( 3 ) -equivariant approach by Fuchs et al . ( 2020 ) introduced a self-attention mechanism that is invariant to global rotations and translations of its input and solves the limitation of angularly constrained filters in other equivariant methods , e.g. , Thomas et al . ( 2018 ) . Noteworthy , the work of Jing et al . ( 2020 ) proposed the geometric vector perceptron ( GVP ) consisting of two linear transformations for the input scalar and vector features , followed by nonlinearities . The GVP scalar and vector outputs are invariant and equivariant , respectively , with respect to an arbitrary composition of rotations and reflections in the 3D Euclidean space . The key point distinguishing our approach from other equivariant networks is that we use conformal modeling ( Li et al. , 2001a ; Hitzer , 2008 ) , with which the scalar product of two points yields their Euclidean distance ( allowing us to construct spherical decision surfaces ) and nonlinear conformal transformations on Rn can be linearized ( allowing us to steer the decision spheres ) . Besides , we do not constrain the space of learnable parameters ( as opposed to , e.g. , Thomas et al . ( 2018 ) , Weiler et al . ( 2018a ) , and Fuchs et al . ( 2020 ) ) , but construct our steerable model from a freely trained base network , as we discuss in detail in Section 4 . 2.2 CONFORMAL MODELING AND THE HYPERSPHERE NEURON . The utility of conformal embedding for Euclidean geometry and the close connection to Minkowski spaces are thoroughly discussed by Li et al . ( 2001a ) . An important result is that one can construct hyperspherical decision surfaces using representations in the conformal space ( Li et al. , 2001b ) , as done in the work of Perwass et al . ( 2003 ) . The hypersphere neuron proposed by Banarer et al . ( 2003b ) is such a spherical classifier . Remarkably , since a hypersphere can be seen as a generalization of a hyperplane , the standard neuron can be considered as a special case of the hypersphere neuron . Stacking multiple hypersphere neurons in a feed-forward network results in a multilayer hypersphere perceptron ( MLHP ) , which was shown by Banarer et al . ( 2003a ) to outperform the standard MLP for some classification tasks . However , its application to point sets was not discussed . This motivated the work of Melnyk et al . ( 2021 ) on the geometric neuron , where the learned parameters were shown to represent a combination of spherical decision surfaces . Moreover , the geometric neuron activations were proved to be isometric in the 3D Euclidean space . In our work , we use the latter observation as the necessary condition for deriving the steerability constraint . Making spherical classifiers steerable adds to the practical value of the prior work by Melnyk et al . ( 2021 ) . 3 BACKGROUND . 3.1 STEERABILITY . As per Freeman et al . ( 1991 ) , a 2D function f ( x , y ) is said to steer if it can be written as a linear combination of rotated versions of itself , i.e. , when it satisfies the constraint fθ ( x , y ) = M∑ j=1 vj ( θ ) f θj ( x , y ) , ( 1 ) where vj ( θ ) are the interpolation functions , θj are the basis function orientations , and M is the number of basis function terms required to steer the function . An alternative formulation can be found in the work of Knutsson et al . ( 1992 ) . In 3D , the steering equation becomes fR ( x , y , z ) = M∑ j=1 vj ( R ) fRj ( x , y , z ) , ( 2 ) where fR ( x , y , z ) is f ( x , y , z ) rotated by R ∈ SO ( 3 ) , and each Rj ∈ SO ( 3 ) orients the corresponding jth basis function . Theorems 1 , 2 and 4 in Freeman et al . ( 1991 ) describe the conditions under which the steerability constraints ( 1 ) and ( 2 ) hold , and how to determine the minimum number of basis functions for the 2D and 3D case , respectively . 3.2 CONFORMAL EMBEDDING . We refer the reader to Section 3 in the work of Melnyk et al . ( 2021 ) for more details , and only briefly introduce important notation in this section . The conformal space for the Euclidean Rn counterpart can be formed as MEn ≡ Rn+1,1 = Rn ⊕ R1,1 , where R1,1 is the Minkowski plane ( Li et al. , 2001a ) with orthonormal basis defined as { e+ , e− } and null basis { e0 , e∞ } representing the origin e0 = 1 2 ( e− − e+ ) and point at infinity e∞ = e− + e+ . Thus , a Euclidean vector x ∈ Rn can be embedded in the conformal space MEn as X = C ( x ) = x + 1 2 ∥x∥2 e∞ + e0 , ( 3 ) where X ∈ MEn is called normalized . The conformal embedding ( 3 ) represents the stereographic projection of x onto a projection sphere in MEn and is homogeneous , i.e. , all embedding vectors in the equivalence class [ X ] = { Z ∈ Rn+1,1 : Z = γX , γ ∈ R \ { 0 } } are taken to represent the same vector x . Importantly , given the conformal embedding X and some Y = y + 12∥y∥2 e∞ + e0 , their scalar product in the conformal space corresponds to their Euclidean distance , X · Y = − 12∥x − y∥2 . This interpretation of the scalar product in the conformal space is the main motivation in constructing spherical classifiers .

Summary: ## Summary and contributions. Authors propose 3D "spherical neurons" leading to rotationally equivariant layers. They do so by building on the spherical and geometric neurons introduced in Melnyk et al. (2021), which leverages the conformal space (for $\mathbb{R}^n$) to perform operations. The authors then solve for the steerability constraint for this neuron and empirically show that the proposed approach overperforms Melnyk et al. (2021) on rotated 3D data.