Leveraging SE(3) Equivariance for Self-supervised Category-Level Object  Pose Estimation from Point Clouds

1 Introduction . Object pose estimation is a crucial computer vision task that is widely employed in robotics , humanobject interaction , and augmented-reality applications . Previous work can be categorized broadly into two genres : classic instance-level 6D object pose estimation ( e.g. , PoseCNN [ 25 ] ) , which assumes the availability of exact CAD models for all object instances ; and category-level 6D object pose estimation ( e.g. , NOCS [ 23 ] ) , which defines a category-level reference frame and is able to generalize to a category of object instances . The motivation underlying the category-level approach is to develop systems that can accommodate novel object instances that were unseen during training . This approach is therefore more applicable for general robotic vision systems ( e.g. , of a home robot ) that must face complex environments , such as those encountered in everyday situations . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To achieve the intra-category generalization , category-level 6D object pose estimation systems often require a significantly larger amount of annotated training data than classic instance-level systems . The reason for more training examples lies in the need to cover large intra-category shape variations among different object instances , in addition to the usual requirement for diversity in object poses , occlusion patterns , and environments . However , obtaining good 6D pose annotations is expensive and time-consuming . As a result , this line of research often relies heavily on synthetic training data . In order to scale up category-level object pose estimation systems to allow for more object categories , there is a strong demand for self-supervised methods that do not require pose annotations . Designing self-supervised methods for category-level pose estimation is very challenging . Although a few systems ( e.g. , Self6D [ 22 ] ) have recently been proposed for self-supervised instance-level 6D object pose estimation , all of those systems have access to the CAD model of each instance during training ( but without the pose annotations ) . These previous approaches are thus inapplicable to self-supervised category-level object pose estimation , for which no CAD model is available for any object instances during either training or testing . The lack of any CAD models and annotations further presents a formidable obstacle to the designer : the reference frame to define category-level pose is unspecified throughout the training . To the best of our knowledge , there is no existing work on fully self-supervised category-level pose estimation , probably due to this issue . We therefore need to find a way to allow the emergence of a category-level reference frame during training . A starting point is to consider the category-level reference frame known as Normalized Object Coordinate Space ( NOCS ) , which is manually defined in [ 23 ] . Within the NOCS regression framework , all shapes are aligned in this space ; also , dense canonical reconstruction of the visible points of the object of interest is performed . The relationship between the reference frame and canonical reconstruction inspires us to ask the following question : can a system learn to perform canonical reconstruction of object instances from the same object category , whose reconstruction space will naturally be a category-level reference frame ? This canonical reconstruction presents two requirements : 1 ) all of the object reconstructions need to be aligned , and 2 ) the reconstruction should not change if the input object undergoes a change in pose . The second requirement suggests an SE ( 3 ) -invariant reconstruction of an observed object , which inspires us to consider point cloud processing networks that are SE ( 3 ) -invariant/equivariant . Formally , given a set of point cloud transformations TA : RN×3 → RN×3 for A ∈ SE ( 3 ) , a neural network φ : RN×3 → F ( where F is an arbitrary feature domain ) is called equivariant if for each A there exists an equivariant transformation SA : F → F so that : SA [ φ ( X ) ] = φ ( TA [ X ] ) , ∀A ∈ SE ( 3 ) , X ∈ RN×3 Note that invariance is a special case of equivariance ; the network is called invariant under the transformations when SA is an identity mapping . Therefore , we propose to use an SE ( 3 ) -invariant network for shape reconstruction . To find a selfsupervision for this reconstruction task , we propose to jointly estimate the object pose P and use the predicted transformation in P to transform our reconstructions into the camera space so that a consistency loss can be formed between the proposed reconstruction and our input observation . In contrast to the pose invariance pursued by our shape reconstruction module , here the estimated 6D pose should naturally be equivariant with any SE ( 3 ) transformation applied to the input , which means that the estimated pose should undergo the same 6D transformation when the input object changes its 6D pose . We thus use an SE ( 3 ) -equivariant network for this pose estimation . Leveraging SE ( 3 ) invariance and equivariance , our proposed system is designed to disentangle shape and pose in a selfsupervised manner . Note that we do not explicitly enforce shape alignment in reconstruction . In other words , our network has the freedom to choose canonical poses of each instance for reconstruction during training . Even so , we find that our network automatically chooses to align all the instances for reconstruction ( except for certain symmetry ambiguities ) , because the network is relatively “ lazy ” and such reconstruction is the easiest thing to do . Within this aligned reconstruction space , a categorylevel reference frame emerges without any supervision , which further defines our category-level pose that is to be estimated by our equivariant pose estimation module . Extensive experiments show that this self-supervised disentanglement is successful for both complete and partial object observations . Our proposed method achieves very accurate 6D pose estimation on synthetic point clouds generated from ModelNet [ 24 ] , and it works well on the real-world NOCSREAL275 dataset [ 23 ] . An ablation study further shows the key role of equivariance and invariance in this self-supervised framework . 2 Related Work . Supervised Category-Level Object Pose Estimation . The pioneering work of Wang et al . [ 23 ] proposed Normalized Object Coordinate Space ( NOCS ) as a category-specific canonical reference frame , so that the category-level pose of a previously unseen object can be defined as the transformation from its NOCS . Several follow-up efforts have improved NOCS by considering articulated objects [ 11 ] , by incorporating object pose tracking [ 21 ] , by leveraging analysis-by-synthesis and shape generative models [ 4 , 6 ] , or by exploiting learnable deformation [ 19 ] . However , all of these approaches adopt fully-supervised training paradigms and assume that object poses are known at training time . As discussed previously , object pose annotations are very difficult to obtain for category-level learning at a large scale . Self-supervised Pose Estimation . To avoid the need for pose annotations in fully-supervised learning paradigms , several works have looked into self-supervision for instance-level object pose estimation . Self6D [ 22 ] and DTPE [ 15 ] obtain object poses by minimizing the difference between rendered depth ( or RGBD ) images and real observations in a self-supervised manner . AAE [ 16 ] trains a pose-augmented autoencoder on synthetic images , which can then be applied to pose estimation for real images . However , these methods still assume that a corresponding CAD model is available for the target object for which pose needs to be estimated . For category-level pose estimation without any CAD models , these approaches can not be applied . An alternative approach known as shape co-alignment [ 1 , 3 , 12 ] allows emergence of consistent category-level reference frames . These methods usually assume clean meshes as inputs , or they leverage strong priors like symmetry , making them unsuitable for more challenging shapes or even partial observations exhibiting high geometrical or topological variations . Moreover , these methods do not make the connection between shape co-alignment and category-level 6D pose estimation . QEC [ 27 ] has a problem scope closest to ours . However , instead of aiming for self-supervised category-level 6D pose estimation for both complete and partial observations , this system deals only with 3D pose ( 3D rotation ) of complete shapes . Also , QEC does not evaluate the predicted category-level 3D pose , and there is no guarantee regarding the pose quality . Other methods like [ 10 , 20 ] require multi-view RGB images as input , and the primary focus is still on shape reconstruction . In summary , our work is the first to target self-supervised category-level object 6D pose estimation . This paper demonstrates significant potential of our method using only a depth point cloud as the input . Equivariant Point Cloud Neural Networks . Recently , achieving SE ( 3 ) invariance and equivariance in point cloud processing has attracted a lot of attention . Early work [ 13 , 26 ] focused more on achieving invariance , while most recent work has explored equivariance . Recent work has introduced Tensor Field Networks [ 18 ] and SE3-transformers [ 9 ] , which leverage spherical harmonics ; quaternion equivariant capsule networks [ 27 ] ; and , most recently , Vector Neurons [ 7 ] . We note that EPN [ 5 ] is the only network for which equivariance does not entail a cost of additional network capacity . 3 Method . In section 3.1 , we consider the Equivariant Point Network [ 5 ] with emphasis on its SE ( 3 ) -equivariance properties . Section 3.2 describes how we disentangle shape and pose , jointly estimate them , and form a self-supervised learning pipeline . Section 3.3 describes how category-level pose is inferred and evaluated at test time . 3.1 Revisiting Equivariant Point Networks . To achieve shape and pose disentanglement , our framework leverages a state-of-the-art SE ( 3 ) equivariant network known as the Equivariant Point Network ( EPN ) [ 5 ] . Taking a point cloud X as input , EPN achieves equivariance regarding both 3D translation t ∈ R3 and 3D rotation g ∈SO ( 3 ) , and it can generate per-point localized equivariant features f ( x ) for points x ∈ X . We briefly review its translation and rotation equivariance properties . Translation Equivariance . Considering 3D translation Tt [ X ] = X + t that moves X to X ′ , a translation equivariant transformation can be defined as St = I ( identity mapping ) . This definition of translation equivariance is widely used in other SE ( 3 ) -equivariant networks , e.g. , Tensor Field Network [ 18 ] and SE ( 3 ) -transformer [ 9 ] . Under this equivariance , f′ ( x′ ) = f′ ( x+t ) = St [ f ( x ) ] = f ( x ) , which means the feature f of one point x does not change under translation of the whole point cloud . The simplest way to produce this translation equivariance is to avoid directly processing the absolute point coordinates but instead use relative point offsets . EPN sets up local frames around each input point x and uses only the relative point offsets x̃i = xi − x during feature extraction . These point offsets can still depict the geometry but do not change under translation of the input . Although any point network that uses only relative point offsets will be translation equivariant under this definition , EPN specifically chooses to use KPConv [ 17 ] as its convolution operator and performs convolution at each point x within its ball neighborhood Nx . More specifically , ( f ∗ h ) ( x ) = ∑ x̃i+x∈Nx f ( x̃i ) h ( −x̃i ) , where h ( x̃ ) is a continuously parameterized function in Nx . Rotation Equivariance . When an input point cloud is transformed by an SO ( 3 ) rotation matrix , equivariance is a desired property , especially for the sake of rotation estimation . A non-trival ( noninvariant ) definition of rotation equivariance thus becomes very important . The key idea behind the rotation equivariance of EPN is to introduce a feature vector field F on an SO ( 3 ) manifold after discretizing the continuous SO ( 3 ) rotation group . EPN first proposes a kernel-rotated version of KPConv , which rotates the convolution kernel h by a rotation g ∈ SO ( 3 ) . Using the rotated kernel hg to perform convolution on the input point cloud will result in a new set of per-point features f ( x , g ) , equivalent to rotating the input X by g−1 . In other words , its rotated kernel function satisfies hg ( x̃i ) = h ( Tg−1 [ x̃i ] ) where g−1 is the inverse rotation of g. Assume for any g ∈ SO ( 3 ) , EPN can compute per-point C-channel features f ( x , g ) ∈ RC using the corresponding rotated KPConv kernel hg , then the features for all g and all x form a per-point vector field F = φ ( X ) = { f ( x , g ) ∈ RC | g ∈ SO ( 3 ) , x ∈ X } . We therefore can define a rotation equivariant transformation Sg0 [ f ( x , g ) ] = f ( x , ( g0 ) −1 ◦ g ) . Then we could prove Sg0 [ φ ( X ) ] = { f ( x , g−10 ◦ g ) } = { f ( Tg0 [ x ] , g ) } = φ ( Tg0 [ X ] ) . Intuitively , Sg0 will be equivalent to applying a circular shift to the vector field F with the value of each entry unchanged . In practice , it is infeasible to compute F for all rotations on SO ( 3 ) , and EPN therefore leverages a discrete approximation and achieves equivariance for a finite subgroup of SO ( 3 ) . Concretely , EPN discretizes the SO ( 3 ) rotation group into the icosahedron rotation group Gg , which contains 60 different rotations originating from the 60 rotational symmetries of a regular icosahedron and computes a rotated KPConv for each g ∈ Gg . This KPConv-based multi-rotation convolution is named PointConv , effectively generating a per-point 2D feature f ∈ RC×|G| for all x ∈ X . When the input point cloud X is rotated by some g ∈ Gg , the rotation equivariance guarantees that each f will be permuted in its G dimension according to g . I.e. , its j-th feature fj ∈ RC will become the j ′ -th feature of f′ , satisfying gj′ = g ◦ gj . To further increase the expressivity of the network and to allow information exchange between different coordinates of F , EPN further introduces a GroupConv operation , which passes messages across different gs of f ( x , g ) . This message passing leverages a rotation-invariant graph convolutional kernel and therefore does not break the equivariant structure of F . For more detailed information , please refer to EPN [ 5 ] . SE ( 3 ) -equivariance . By combining translation equivariance and rotation equivariance , we can define a rotation- and translation-equivariant transformation SA = St ◦Sg , so that EPN is strictly equivariant to all SE ( 3 ) transformations in GA = { TA | TA [ X ] = Tt ◦ Tg [ X ] , ∀t ∈ R3 , g ∈ Gg } . Note that GA is a subgroup of SE ( 3 ) ; we will discuss in section 3.2 how this subgroup approximation affects our task when the input undergoes rotations that are not in Gg .

Summary: The paper proposes a framework to learn 6D object pose estimation without GT pose annotations and CAD models (while GT object masks are available, and each object is already isolated). The key is to jointly learn canonical shapes and poses by a SE(3)-equivariance network. It shows promising performance on both synthetic and real depth data.