Fully Steerable 3D Spherical Neurons

Emerging from low-level vision theory , steerable filters found their counterpart in prior work on steerable convolutional neural networks equivariant to rigid transformations . In our work , we propose a steerable feed-forward learning-based approach that consists of spherical decision surfaces and operates on point clouds . Focusing on 3D geometry , we derive a 3D steerability constraint for hypersphere neurons , which are obtained by conformal embedding of Euclidean space and have recently been revisited in the context of learning representations of point sets . Exploiting the rotational equivariance , we show how our model parameters are fully steerable at inference time . We use a synthetic point set and real-world 3D skeleton data to show how the proposed spherical filter banks enable making equivariant and , after online optimization , invariant class predictions for known point sets in unknown orientations . 1 INTRODUCTION . We present a novel feed-forward model consisting of steerable 3D neurons for point cloud classification , an important and challenging problem with many applications such as autonomous driving , human-robot interaction , and mixed-reality installations . We achieve the steerability by using geometric neurons ( Melnyk et al. , 2021 ) , leading to a rotation equivariant network . Besides becoming a geometrically explainable approach , further benefits of the proposed method include rotation invariant classification and , therefore , lowered data augmentation requirements during learning . We make use of a conformal embedding to obtain higher-order decision surfaces . Following the motivation in the recent work of Melnyk et al . ( 2021 ) , we focus on 3D geometry and spherical decision surfaces , arguing for their natural suitability for problems in Euclidean space . We show how a spherical neuron ( classifier ) , i.e. , the hypersphere neuron ( Banarer et al. , 2003b ) or its generalization for 3D input point sets — the geometric neuron ( Melnyk et al. , 2021 ) — can be turned into a steerable neuron . We prove that the aforementioned spherical neurons in any dimension require only up to first-degree spherical harmonics to accommodate the effect of rotation . This allows us to derive a 3D steerability constraint for the spherical neurons and to describe a recipe for creating a steerable model from a pretrained spherical classifier . Using the synthetic Tetris dataset ( Thomas et al. , 2018 ) and the skeleton data from the UTKinectAction3D dataset ( Xia et al. , 2012 ) , we first verify the derived constraint and check its stability with respect to perturbations in the input . We further conduct an experiment in a realistic setting , where we initialize the steerable model parameters using a noisy or imperfect prediction of the transformation applied to the input and optimize the parameters in an unsupervised way . The contributions of our work are as follows : ( a ) We prove that the activation of spherical neurons on rotated input only varies by up to first-degree spherical harmonics ( Section 4.1.1 ) . ( b ) Based on a minimal set of four spherical neurons that are rotated to the corresponding vertices of a regular tetrahedron ( Section 4.1.2 ) , we derive the main result of our paper ( Section 4.1.3 ) – a 3D steerability constraint for spherical neurons ( 13 ) . ( c ) We propose a method to turn a trained spherical classifier into a fully steerable model ( Section 5.2 ) that produces predictions invariant to 3D rotations , even in the presence of noise ( Section 5.3 ) , which we verify experimentally using both synthetic and real 3D data . 2 RELATED WORK . 2.1 STEERABILITY AND EQUIVARIANCE . Steerability is a powerful concept from early vision and image processing ( Freeman et al. , 1991 ; Knutsson et al. , 1992 ; Simoncelli et al. , 1992 ; Perona , 1995 ; Simoncelli & Freeman , 1995 ; Teo & Hel-Or , 1998 ) that resonates in the era of deep learning . The utility of steerable filters and the main theorems for their construction are presented in the seminal work of Freeman et al . ( 1991 ) . Geometric equivariance in the context of computer vision has been an object of extensive research over the last decades ( Van Gool et al. , 1995 ) . Equivariance is a necessary property for steerability as the group acting on the input space needs to be represented in the co-domain of the operator . For this reason , Lie theory can be used to to study steerable filters for equivariance and invariance ( Reisert , 2008 ) . Nowadays , equivariance-related research extends into deep learning , e.g. , the SE ( 3 ) -equivariant models of Fuchs et al . ( 2020 ) , Thomas et al . ( 2018 ) , and Zhao et al . ( 2020 ) , and the SO ( 3 ) -equivariant network of Anderson et al . ( 2019 ) , as well as Marcos et al . ( 2017 ) and Kondor et al . ( 2018 ) . Also steerable filter concepts are increasingly used in current works , e.g. , Cohen & Welling ( 2016 ) considered image data and proposed a CNN architecture to produce equivariant representations with steerable features , which involves fewer parameters than traditional CNNs . They considered the dihedral group ( discrete rotation , translation , and reflection ) , and the steerable representations in their work are proposed as formation of elementary feature types . One limitation of their approach is that rotations are restricted to four orientations , i.e. , by π/2 . More recently , Weiler et al . ( 2018b ) utilized group convolutions and introduced steerable filter convolutional neural networks ( SFCNNs ) operating on images to jointly attain equivariance under translations and discrete rotations . In their work , the filter banks are learned rather than fixed . Further , the work of Weiler & Cesa ( 2019 ) proposed a unified framework for E ( 2 ) -equivariant steerable CNNs and presented their general theory . The steerable CNNs for 3D data proposed by Weiler et al . ( 2018a ) are closely related to our work . The authors employed a combination of scalar , vector and tensor fields as features transformed by SO ( 3 ) representations and presented a model that is equivariant to SE ( 3 ) transformations . They also considered different types of nonlinearities suitable for nonscalar components of the feature space . The novel SE ( 3 ) -equivariant approach by Fuchs et al . ( 2020 ) introduced a self-attention mechanism that is invariant to global rotations and translations of its input and solves the limitation of angularly constrained filters in other equivariant methods , e.g. , Thomas et al . ( 2018 ) . Noteworthy , the work of Jing et al . ( 2020 ) proposed the geometric vector perceptron ( GVP ) consisting of two linear transformations for the input scalar and vector features , followed by nonlinearities . The GVP scalar and vector outputs are invariant and equivariant , respectively , with respect to an arbitrary composition of rotations and reflections in the 3D Euclidean space . The key point distinguishing our approach from other equivariant networks is that we use conformal modeling ( Li et al. , 2001a ; Hitzer , 2008 ) , with which the scalar product of two points yields their Euclidean distance ( allowing us to construct spherical decision surfaces ) and nonlinear conformal transformations on Rn can be linearized ( allowing us to steer the decision spheres ) . Besides , we do not constrain the space of learnable parameters ( as opposed to , e.g. , Thomas et al . ( 2018 ) , Weiler et al . ( 2018a ) , and Fuchs et al . ( 2020 ) ) , but construct our steerable model from a freely trained base network , as we discuss in detail in Section 4 . 2.2 CONFORMAL MODELING AND THE HYPERSPHERE NEURON . The utility of conformal embedding for Euclidean geometry and the close connection to Minkowski spaces are thoroughly discussed by Li et al . ( 2001a ) . An important result is that one can construct hyperspherical decision surfaces using representations in the conformal space ( Li et al. , 2001b ) , as done in the work of Perwass et al . ( 2003 ) . The hypersphere neuron proposed by Banarer et al . ( 2003b ) is such a spherical classifier . Remarkably , since a hypersphere can be seen as a generalization of a hyperplane , the standard neuron can be considered as a special case of the hypersphere neuron . Stacking multiple hypersphere neurons in a feed-forward network results in a multilayer hypersphere perceptron ( MLHP ) , which was shown by Banarer et al . ( 2003a ) to outperform the standard MLP for some classification tasks . However , its application to point sets was not discussed . This motivated the work of Melnyk et al . ( 2021 ) on the geometric neuron , where the learned parameters were shown to represent a combination of spherical decision surfaces . Moreover , the geometric neuron activations were proved to be isometric in the 3D Euclidean space . In our work , we use the latter observation as the necessary condition for deriving the steerability constraint . Making spherical classifiers steerable adds to the practical value of the prior work by Melnyk et al . ( 2021 ) . 3 BACKGROUND . 3.1 STEERABILITY . As per Freeman et al . ( 1991 ) , a 2D function f ( x , y ) is said to steer if it can be written as a linear combination of rotated versions of itself , i.e. , when it satisfies the constraint fθ ( x , y ) = M∑ j=1 vj ( θ ) f θj ( x , y ) , ( 1 ) where vj ( θ ) are the interpolation functions , θj are the basis function orientations , and M is the number of basis function terms required to steer the function . An alternative formulation can be found in the work of Knutsson et al . ( 1992 ) . In 3D , the steering equation becomes fR ( x , y , z ) = M∑ j=1 vj ( R ) fRj ( x , y , z ) , ( 2 ) where fR ( x , y , z ) is f ( x , y , z ) rotated by R ∈ SO ( 3 ) , and each Rj ∈ SO ( 3 ) orients the corresponding jth basis function . Theorems 1 , 2 and 4 in Freeman et al . ( 1991 ) describe the conditions under which the steerability constraints ( 1 ) and ( 2 ) hold , and how to determine the minimum number of basis functions for the 2D and 3D case , respectively . 3.2 CONFORMAL EMBEDDING . We refer the reader to Section 3 in the work of Melnyk et al . ( 2021 ) for more details , and only briefly introduce important notation in this section . The conformal space for the Euclidean Rn counterpart can be formed as MEn ≡ Rn+1,1 = Rn ⊕ R1,1 , where R1,1 is the Minkowski plane ( Li et al. , 2001a ) with orthonormal basis defined as { e+ , e− } and null basis { e0 , e∞ } representing the origin e0 = 1 2 ( e− − e+ ) and point at infinity e∞ = e− + e+ . Thus , a Euclidean vector x ∈ Rn can be embedded in the conformal space MEn as X = C ( x ) = x + 1 2 ∥x∥2 e∞ + e0 , ( 3 ) where X ∈ MEn is called normalized . The conformal embedding ( 3 ) represents the stereographic projection of x onto a projection sphere in MEn and is homogeneous , i.e. , all embedding vectors in the equivalence class [ X ] = { Z ∈ Rn+1,1 : Z = γX , γ ∈ R \ { 0 } } are taken to represent the same vector x . Importantly , given the conformal embedding X and some Y = y + 12∥y∥2 e∞ + e0 , their scalar product in the conformal space corresponds to their Euclidean distance , X · Y = − 12∥x − y∥2 . This interpretation of the scalar product in the conformal space is the main motivation in constructing spherical classifiers .

Summary: The paper aims to derive a steerability constraint for spherical neurons (3D point classifiers with spherical decision boundary). The steerability constraint enables test-time optimization of a pre-trained classifier to make predictions equivariant to 3D rotation perturbations applied to the input. When input rotation perturbations are unknown, the authors propose a method to recover the unknown rotations and therefore make rotation-invariant predictions. The experiments on a few small scale datasets verifies some of the claims.