Quaternion Equivariant Capsule Networks for 3D Point Clouds

1 INTRODUCTION . It is now well understood that in order to learn a compact representation of the input data , one needs to respect the symmetries in the problem domain ( Cohen et al. , 2019 ; Weiler et al. , 2018a ) . Arguably , one of the primary reasons of the success of 2D convolutional neural networks ( CNN ) is the translation-invariance of the 2D convolution acting on the image grid ( Giles & Maxwell , 1987 ; Kondor et al. , 2018 ) . Recent trends aim to transfer this success into the 3D domain in order to support many applications such as shape retrieval , shape manipulation , pose estimation , 3D object modeling and detection . There , the data is naturally represented as sets of 3D points or a point cloud ( Qi et al. , 2017a ; b ) . Unfortunately , extension of CNN architectures to 3D point clouds is non-trivial due to two reasons : 1 ) point clouds are irregular and unstructured , 2 ) the group of transformations that we are interested in is more complex as 3D data is often observed under arbitrary non-commutative SO ( 3 ) rotations . As a result , achieving appropriate embeddings requires 3D networks that work on points to be equivariant to these transformations , while also being invariant to the permutations of the point set . In order to fill this important gap , we propose the quaternion equivariant point capsule network or QE-Network that is suited to process point clouds and is equivariant to SO ( 3 ) rotations compactly parameterized by quaternions ( Fig . 2 ) , in addition to preserved translation and permutation equivariance . Inspired by the local group equivariance ( Lenssen et al. , 2018 ; Cohen et al. , 2019 ) , we efficiently cover SO ( 3 ) by restricting ourselves to the sparse set of local reference frames ( LRF ) that collectively characterize the object orientation . The proposed capsule layers ( Hinton et al. , 2011 ) deduces equivariant latent representations by robustly combining those local LRFs using the proposed Weiszfeld dynamic routing . Hence , our latent features specify to local orientations disentangling the pose from object existence . Such explicit storage is unique to our work and allows us to perform rotation estimation jointly with object classification . Our final architecture is a hierarchy of QE-networks , where we use classification error as the only training cue and adapt a Siamese version when the relative rotation is to be regressed . We neither explicitly supervise the network with pose annotations nor train by augmenting rotations . Overall , our contributions are : 1 . We propose a novel , fully SO ( 3 ) -equivariant capsule architecture that is tailored for simultaneous classification and pose estimation of 3D point clouds . This network produces in- Under review as a conference paper at ICLR 2020 ( a ) Input Point Cloud ( b ) Initial LRFs ( c ) LRFs After First Pooling ( d ) Multi-channel LRFs Prior to QE-Network-2 variant latent representations while explicitly decoupling the orientation into capsules , thus attaining equivariance . Note that equivariance results have not been previously achieved regarding the quaternion parameterization of the 3D special orthogonal group . 2 . By utilizing LRFs on points , we reduce the space of orientations that we consider and hence can work sparsely on a subset of the group elements . 3 . We theoretically prove the equivariance properties of our 3D network regarding the quaternion group . Moreover , to the best of our knowledge , we for the first time establish a connection between the dynamic routing of Sabour et al . ( 2017 ) and Generalized Weiszfeld iterations ( Aftab et al. , 2015 ) . By that , we theoretically argue for the convergence of the employed dynamic routing . 4 . We experimentally demonstrate the capabilities of our network on classification and orientation estimation of 3D shapes . 2 PRELIMINARIES AND TECHNICAL BACKGROUND . In this paper we will speak of the equivariance of point clouds under the actions of quaternions . We now provide the necessary background required for the grasp of this content . 2.1 EQUIVARIANCE . Definition 1 ( Equivariant Map ) . For a G-space acting on X , the map Φ : G × X 7→ X is said to be equivariant if its domain and co-domain are acted on by the same symmetry group ( Cohen & Welling , 2016 ; Cohen et al. , 2018a ) : Φ ( g1 ◦ x ) = g2 ◦ Φ ( x ) ( 1 ) where g1 ∈ G and g2 ∈ G. Equivalently Φ ( T ( g1 ) x ) = T ( g2 ) Φ ( x ) , where T ( · ) is a linear representation of the group G. Note that T ( · ) does not have to commute . It suffices for T ( · ) to be a homomorphism : T ( g1 ◦ g2 ) = T ( g1 ) ◦ T ( g2 ) . In this paper we use a stricter form of equivariance and consider g2 = g1 . Definition 2 ( Equivariant Network ) . An architecture or network is said to be equivariant if all of its layers are equivariant maps . Due to the transitivity of the equivariance , stacking up equivariant layers will result in globally equivariant networks e.g. , rotating the input will produce output vectors which are transformed by the same rotation ( Lenssen et al. , 2018 ; Kondor & Trivedi , 2018 ) . 2.2 THE QUATERNION GROUP H1 . The choice of 4-vector quaternions has multiple motivations : 1 . All 3-vector formulations suffer from infinitely many singularities as angle goes to 0 , whereas quaternions avoid those . 2 . 3-vectors also suffer from infinitely many redundancies ( the norm can grow indefinitely ) . Quaternions have a single redundancy : q = −q , a condition that is in practice easy to enforce . 3 . Computing the actual ‘ manifold mean ’ on the Lie algebra requires iterative techniques with subsequent updates on the tangent space . Such iterations are computationally harmful for a differentiable GPU implementation . Definition 3 ( Quaternion ) . A quaternion q is an element of Hamilton algebra H1 , extending the complex numbers with three imaginary units i , j , k in the form : q = q11 + q2i + q3j + q4k = Algorithm 1 : Quaternion Equivariant Dynamic Routing 1 input : Input points { x1 , ... , xK } ∈ RK×3 , input capsules ( LRFs ) Q = { q1 , . . . , qL } ∈ H1L , with L = N c ·K , N c is the number of capsules per point , activations α = ( α1 , . . . , αL ) T , trainable transformations T = { ti , j } i , j ∈ H1L×M 2 output : Updated frames Q̂ = { q̂1 , . . . , q̂M } ∈ H1M , updated activations α̂ = ( α̂1 , . . . , α̂M ) T 3 for All primary ( input ) capsules i do 4 for All latent ( output ) capsules j do 5 vi , j ← qi ◦ ti , j // compute votes 6 for All latent ( output ) capsules j do 7 q̂j ← A ( { v1 , j . . .vK , j } , α ) // initialize output capsules 8 for k iterations do 9 for All primary ( input ) capsules i do 10 wi , j ← αi · sigmoid ( − δ ( q̂j , vi , j ) ) // compute the current weight 11 q̂j ← A ( { v1 , j . . .vL , j } , w : ,j ) // see Eq ( 4 ) 12 α̂j ← sigmoid ( − 1K L∑ 1 δ ( q̂j , vi , j ) ) // recompute activations ( q1 , q2 , q3 , q4 ) T , with ( q1 , q2 , q3 , q4 ) T ∈ R4 and i2 = j2 = k2 = ijk = −1 . q1 ∈ R denotes the scalar part and v = ( q2 , q3 , q4 ) T ∈ R3 , the vector part . The conjugate q̄ of the quaternion q is given by q̄ : = q1 − q2i− q3j− q4k . A unit quaternion q ∈ H1 with 1 ! = ‖q‖ : = q · q̄ and q−1 = q̄ , gives a compact and numerically stable parametrization to represent orientation of objects on the unit sphere S3 , avoiding gimbal lock and singularities ( Busam et al. , 2017 ) . Identifying antipodal points q and −q with the same element , the unit quaternions form a double covering group of SO ( 3 ) . H1 is closed under the non-commutative multiplication or the Hamilton product : ( p ∈ H1 ) ◦ ( r ∈ H1 ) = [ p1r1 − vp · vr ; p1vr + r1vp + vp × vr ] . ( 2 ) Definition 4 ( Linear Representation of H1 ) . We follow Birdal et al . ( 2018 ) and use the T ( q ) , q1 −q2 −q3 −q4q2 q1 −q4 q3q3 q4 q1 −q2 q4 −q3 q2 q1 . To be concise we will use capital letters to refer to the matrix representation of quaternions e.g.Q ≡ T ( q ) , G ≡ T ( g ) . Note that T ( · ) , the injective homomorphism to the orthonormal matrix ring , by construction satisfies the condition in Dfn . 1 ( Steenrod , 1951 ) : det ( Q ) = 1 , Q > = Q−1 , ‖Q‖ = ‖Qi , :‖ = ‖Q : ,i‖ = 1 and Q − q1I is skew symmetric : Q + Q > = 2q1I . It is easy to verify these properties . T linearizes the Hamilton product or the group composition : g ◦ q , T ( g ) q , Gq . 2.3 3D POINT CLOUDS . Definition 5 ( Point Cloud ) . We define a 3D surface to be a differentiable 2-manifold embedded in the ambient 3D Euclidean space : M2 ∈ R3 and a point cloud to be a discrete subset sampled on M2 : X ∈ { xi ∈M2 ∩ R3 } . Definition 6 ( Local Geometry ) . For a smooth point cloud { xi } ∈ M2 ⊂ RN×3 , a local reference frame ( LRF ) is defined as an ordered basis of the tangent space at x , TxM , consisting of orthonormal vectors : L ( x ) = [ ∂1 , ∂2 , ∂3 ≡ ∂1 × ∂2 ] . Usually the first component is defined to be the surface normal ∂1 , n ∈ S2 : ‖n‖ = 1 and the second one is picked according to a modality dependent heuristic . Note that recent trends such as ( Cohen et al. , 2019 ) acknowledge the ambiguity and either employ a gauge ( tangent frame ) equivariant design or propagate the determination of a certain direction until the last layer ( Poulenard & Ovsjanikov , 2018 ) . Here , we will assume that ∂2 can be uniquely and repeatably computed , a reasonable assumption for the point sets we consider ( Petrelli & Di Stefano , 2011 ) . For the cases where this does not hold , we will rely on the network ’ s robustness . We will explain our method of choice in Sec . 4 and visualize LRFs of an airplane object in Fig . 1 . 3 SO ( 3 ) -EQUIVARIANT 3D CAPSULE NETWORKS Disentangling orientation from representations requires guaranteed equivariances and invariances . Yet , the original capsule networks of Sabour et al . ( 2017 ) can not achieve equivariance to general groups . To this end , Lenssen et al . ( 2018 ) proposed to use a manifold-mean and a special aggregation that makes sure that the trainable transformations get pose-aligned points as input . We will extend this idea to the non-abelian SO ( 3 ) and design capsule networks sparsely operating on a set of LRFs computed on local neighborhoods of points , parameterized by quaternions . In the following , we first explain our novel capusle layers , the main building block of our architecture . We then show how to stack those layers via a simple aggregation resulting in an SO ( 3 ) -equivariant 3D capsule network that yields invariant representations ( or activations ) as well as equivariant rotations ( latent capsules ) .

Summary: This paper presents a rotation-equivariant model for use with point-clouds.  Capsule networks with activation and rotation pose estimate in the form of a quaternion are applied to local patches of 3D points, and combined up through progressively more coarser receptive fields.  Rotation equivariance is obtained by looking for agreement between input point frames (or capsules), by iteratively clustering using the quaternion mean.  The method is evaluated on ModelNet40, obtaining much improved results in the case where rotation orientation is arbitrary.