Leveraging SE(3) Equivariance for Self-supervised Category-Level Object  Pose Estimation from Point Clouds

1 Introduction . Object pose estimation is a crucial computer vision task that is widely employed in robotics , humanobject interaction , and augmented-reality applications . Previous work can be categorized broadly into two genres : classic instance-level 6D object pose estimation ( e.g. , PoseCNN [ 25 ] ) , which assumes the availability of exact CAD models for all object instances ; and category-level 6D object pose estimation ( e.g. , NOCS [ 23 ] ) , which defines a category-level reference frame and is able to generalize to a category of object instances . The motivation underlying the category-level approach is to develop systems that can accommodate novel object instances that were unseen during training . This approach is therefore more applicable for general robotic vision systems ( e.g. , of a home robot ) that must face complex environments , such as those encountered in everyday situations . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To achieve the intra-category generalization , category-level 6D object pose estimation systems often require a significantly larger amount of annotated training data than classic instance-level systems . The reason for more training examples lies in the need to cover large intra-category shape variations among different object instances , in addition to the usual requirement for diversity in object poses , occlusion patterns , and environments . However , obtaining good 6D pose annotations is expensive and time-consuming . As a result , this line of research often relies heavily on synthetic training data . In order to scale up category-level object pose estimation systems to allow for more object categories , there is a strong demand for self-supervised methods that do not require pose annotations . Designing self-supervised methods for category-level pose estimation is very challenging . Although a few systems ( e.g. , Self6D [ 22 ] ) have recently been proposed for self-supervised instance-level 6D object pose estimation , all of those systems have access to the CAD model of each instance during training ( but without the pose annotations ) . These previous approaches are thus inapplicable to self-supervised category-level object pose estimation , for which no CAD model is available for any object instances during either training or testing . The lack of any CAD models and annotations further presents a formidable obstacle to the designer : the reference frame to define category-level pose is unspecified throughout the training . To the best of our knowledge , there is no existing work on fully self-supervised category-level pose estimation , probably due to this issue . We therefore need to find a way to allow the emergence of a category-level reference frame during training . A starting point is to consider the category-level reference frame known as Normalized Object Coordinate Space ( NOCS ) , which is manually defined in [ 23 ] . Within the NOCS regression framework , all shapes are aligned in this space ; also , dense canonical reconstruction of the visible points of the object of interest is performed . The relationship between the reference frame and canonical reconstruction inspires us to ask the following question : can a system learn to perform canonical reconstruction of object instances from the same object category , whose reconstruction space will naturally be a category-level reference frame ? This canonical reconstruction presents two requirements : 1 ) all of the object reconstructions need to be aligned , and 2 ) the reconstruction should not change if the input object undergoes a change in pose . The second requirement suggests an SE ( 3 ) -invariant reconstruction of an observed object , which inspires us to consider point cloud processing networks that are SE ( 3 ) -invariant/equivariant . Formally , given a set of point cloud transformations TA : RN×3 → RN×3 for A ∈ SE ( 3 ) , a neural network φ : RN×3 → F ( where F is an arbitrary feature domain ) is called equivariant if for each A there exists an equivariant transformation SA : F → F so that : SA [ φ ( X ) ] = φ ( TA [ X ] ) , ∀A ∈ SE ( 3 ) , X ∈ RN×3 Note that invariance is a special case of equivariance ; the network is called invariant under the transformations when SA is an identity mapping . Therefore , we propose to use an SE ( 3 ) -invariant network for shape reconstruction . To find a selfsupervision for this reconstruction task , we propose to jointly estimate the object pose P and use the predicted transformation in P to transform our reconstructions into the camera space so that a consistency loss can be formed between the proposed reconstruction and our input observation . In contrast to the pose invariance pursued by our shape reconstruction module , here the estimated 6D pose should naturally be equivariant with any SE ( 3 ) transformation applied to the input , which means that the estimated pose should undergo the same 6D transformation when the input object changes its 6D pose . We thus use an SE ( 3 ) -equivariant network for this pose estimation . Leveraging SE ( 3 ) invariance and equivariance , our proposed system is designed to disentangle shape and pose in a selfsupervised manner . Note that we do not explicitly enforce shape alignment in reconstruction . In other words , our network has the freedom to choose canonical poses of each instance for reconstruction during training . Even so , we find that our network automatically chooses to align all the instances for reconstruction ( except for certain symmetry ambiguities ) , because the network is relatively “ lazy ” and such reconstruction is the easiest thing to do . Within this aligned reconstruction space , a categorylevel reference frame emerges without any supervision , which further defines our category-level pose that is to be estimated by our equivariant pose estimation module . Extensive experiments show that this self-supervised disentanglement is successful for both complete and partial object observations . Our proposed method achieves very accurate 6D pose estimation on synthetic point clouds generated from ModelNet [ 24 ] , and it works well on the real-world NOCSREAL275 dataset [ 23 ] . An ablation study further shows the key role of equivariance and invariance in this self-supervised framework . 2 Related Work . Supervised Category-Level Object Pose Estimation . The pioneering work of Wang et al . [ 23 ] proposed Normalized Object Coordinate Space ( NOCS ) as a category-specific canonical reference frame , so that the category-level pose of a previously unseen object can be defined as the transformation from its NOCS . Several follow-up efforts have improved NOCS by considering articulated objects [ 11 ] , by incorporating object pose tracking [ 21 ] , by leveraging analysis-by-synthesis and shape generative models [ 4 , 6 ] , or by exploiting learnable deformation [ 19 ] . However , all of these approaches adopt fully-supervised training paradigms and assume that object poses are known at training time . As discussed previously , object pose annotations are very difficult to obtain for category-level learning at a large scale . Self-supervised Pose Estimation . To avoid the need for pose annotations in fully-supervised learning paradigms , several works have looked into self-supervision for instance-level object pose estimation . Self6D [ 22 ] and DTPE [ 15 ] obtain object poses by minimizing the difference between rendered depth ( or RGBD ) images and real observations in a self-supervised manner . AAE [ 16 ] trains a pose-augmented autoencoder on synthetic images , which can then be applied to pose estimation for real images . However , these methods still assume that a corresponding CAD model is available for the target object for which pose needs to be estimated . For category-level pose estimation without any CAD models , these approaches can not be applied . An alternative approach known as shape co-alignment [ 1 , 3 , 12 ] allows emergence of consistent category-level reference frames . These methods usually assume clean meshes as inputs , or they leverage strong priors like symmetry , making them unsuitable for more challenging shapes or even partial observations exhibiting high geometrical or topological variations . Moreover , these methods do not make the connection between shape co-alignment and category-level 6D pose estimation . QEC [ 27 ] has a problem scope closest to ours . However , instead of aiming for self-supervised category-level 6D pose estimation for both complete and partial observations , this system deals only with 3D pose ( 3D rotation ) of complete shapes . Also , QEC does not evaluate the predicted category-level 3D pose , and there is no guarantee regarding the pose quality . Other methods like [ 10 , 20 ] require multi-view RGB images as input , and the primary focus is still on shape reconstruction . In summary , our work is the first to target self-supervised category-level object 6D pose estimation . This paper demonstrates significant potential of our method using only a depth point cloud as the input . Equivariant Point Cloud Neural Networks . Recently , achieving SE ( 3 ) invariance and equivariance in point cloud processing has attracted a lot of attention . Early work [ 13 , 26 ] focused more on achieving invariance , while most recent work has explored equivariance . Recent work has introduced Tensor Field Networks [ 18 ] and SE3-transformers [ 9 ] , which leverage spherical harmonics ; quaternion equivariant capsule networks [ 27 ] ; and , most recently , Vector Neurons [ 7 ] . We note that EPN [ 5 ] is the only network for which equivariance does not entail a cost of additional network capacity . 3 Method . In section 3.1 , we consider the Equivariant Point Network [ 5 ] with emphasis on its SE ( 3 ) -equivariance properties . Section 3.2 describes how we disentangle shape and pose , jointly estimate them , and form a self-supervised learning pipeline . Section 3.3 describes how category-level pose is inferred and evaluated at test time . 3.1 Revisiting Equivariant Point Networks . To achieve shape and pose disentanglement , our framework leverages a state-of-the-art SE ( 3 ) equivariant network known as the Equivariant Point Network ( EPN ) [ 5 ] . Taking a point cloud X as input , EPN achieves equivariance regarding both 3D translation t ∈ R3 and 3D rotation g ∈SO ( 3 ) , and it can generate per-point localized equivariant features f ( x ) for points x ∈ X . We briefly review its translation and rotation equivariance properties . Translation Equivariance . Considering 3D translation Tt [ X ] = X + t that moves X to X ′ , a translation equivariant transformation can be defined as St = I ( identity mapping ) . This definition of translation equivariance is widely used in other SE ( 3 ) -equivariant networks , e.g. , Tensor Field Network [ 18 ] and SE ( 3 ) -transformer [ 9 ] . Under this equivariance , f′ ( x′ ) = f′ ( x+t ) = St [ f ( x ) ] = f ( x ) , which means the feature f of one point x does not change under translation of the whole point cloud . The simplest way to produce this translation equivariance is to avoid directly processing the absolute point coordinates but instead use relative point offsets . EPN sets up local frames around each input point x and uses only the relative point offsets x̃i = xi − x during feature extraction . These point offsets can still depict the geometry but do not change under translation of the input . Although any point network that uses only relative point offsets will be translation equivariant under this definition , EPN specifically chooses to use KPConv [ 17 ] as its convolution operator and performs convolution at each point x within its ball neighborhood Nx . More specifically , ( f ∗ h ) ( x ) = ∑ x̃i+x∈Nx f ( x̃i ) h ( −x̃i ) , where h ( x̃ ) is a continuously parameterized function in Nx . Rotation Equivariance . When an input point cloud is transformed by an SO ( 3 ) rotation matrix , equivariance is a desired property , especially for the sake of rotation estimation . A non-trival ( noninvariant ) definition of rotation equivariance thus becomes very important . The key idea behind the rotation equivariance of EPN is to introduce a feature vector field F on an SO ( 3 ) manifold after discretizing the continuous SO ( 3 ) rotation group . EPN first proposes a kernel-rotated version of KPConv , which rotates the convolution kernel h by a rotation g ∈ SO ( 3 ) . Using the rotated kernel hg to perform convolution on the input point cloud will result in a new set of per-point features f ( x , g ) , equivalent to rotating the input X by g−1 . In other words , its rotated kernel function satisfies hg ( x̃i ) = h ( Tg−1 [ x̃i ] ) where g−1 is the inverse rotation of g. Assume for any g ∈ SO ( 3 ) , EPN can compute per-point C-channel features f ( x , g ) ∈ RC using the corresponding rotated KPConv kernel hg , then the features for all g and all x form a per-point vector field F = φ ( X ) = { f ( x , g ) ∈ RC | g ∈ SO ( 3 ) , x ∈ X } . We therefore can define a rotation equivariant transformation Sg0 [ f ( x , g ) ] = f ( x , ( g0 ) −1 ◦ g ) . Then we could prove Sg0 [ φ ( X ) ] = { f ( x , g−10 ◦ g ) } = { f ( Tg0 [ x ] , g ) } = φ ( Tg0 [ X ] ) . Intuitively , Sg0 will be equivalent to applying a circular shift to the vector field F with the value of each entry unchanged . In practice , it is infeasible to compute F for all rotations on SO ( 3 ) , and EPN therefore leverages a discrete approximation and achieves equivariance for a finite subgroup of SO ( 3 ) . Concretely , EPN discretizes the SO ( 3 ) rotation group into the icosahedron rotation group Gg , which contains 60 different rotations originating from the 60 rotational symmetries of a regular icosahedron and computes a rotated KPConv for each g ∈ Gg . This KPConv-based multi-rotation convolution is named PointConv , effectively generating a per-point 2D feature f ∈ RC×|G| for all x ∈ X . When the input point cloud X is rotated by some g ∈ Gg , the rotation equivariance guarantees that each f will be permuted in its G dimension according to g . I.e. , its j-th feature fj ∈ RC will become the j ′ -th feature of f′ , satisfying gj′ = g ◦ gj . To further increase the expressivity of the network and to allow information exchange between different coordinates of F , EPN further introduces a GroupConv operation , which passes messages across different gs of f ( x , g ) . This message passing leverages a rotation-invariant graph convolutional kernel and therefore does not break the equivariant structure of F . For more detailed information , please refer to EPN [ 5 ] . SE ( 3 ) -equivariance . By combining translation equivariance and rotation equivariance , we can define a rotation- and translation-equivariant transformation SA = St ◦Sg , so that EPN is strictly equivariant to all SE ( 3 ) transformations in GA = { TA | TA [ X ] = Tt ◦ Tg [ X ] , ∀t ∈ R3 , g ∈ Gg } . Note that GA is a subgroup of SE ( 3 ) ; we will discuss in section 3.2 how this subgroup approximation affects our task when the input undergoes rotations that are not in Gg .

Summary: This paper proposes a self-supervised approach to 6 DoF pose estimation. Usually such methods are either instance-level, assuming known CAD models for all object instances; or category-level, assuming a reference frame that can be generalized to all instances of a certain category. The second approach is more generalizable, however it requires a substantial amount of annotated training data, often from synthetic datasets. So far, self-supervision in this setting has been limited to instance-level, with CAD models but without pose annotations. This paper proposed category-level self-supervision and claims to be the first to do so, eliminating the need of both CAD models and large-scale annotated data.