Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
Year: 2022

Abstract:
Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models. To reduce the huge amount of pose annotations needed for categorylevel learning, we propose for the first time a self-supervised learning framework to estimate category-level 6D object pose from single 3D point clouds. During training, our method assumes no ground-truth pose annotations, no CAD models, and no multi-view supervision. The key to our method is to disentangle shape and pose through an invariant shape reconstruction module and an equivariant pose estimation module, empowered by SE(3) equivariant point cloud networks. The invariant shape reconstruction module learns to perform aligned reconstructions, yielding a category-level reference frame without using any annotations. In addition, the equivariant pose estimation module achieves category-level pose estimation accuracy that is comparable to some fully supervised methods. Extensive experiments demonstrate the effectiveness of our approach on both complete and partial depth point clouds from the ModelNet40 benchmark, and on real depth point clouds from the NOCS-REAL 275 dataset. The project page with code and visualizations can be found at: dragonlong.github.io/equi-pose.

heading: 1 Introduction
text: Object pose estimation is a crucial computer vision task that is widely employed in robotics, humanobject interaction, and augmented-reality applications. Previous work can be categorized broadly into two genres: classic instance-level 6D object pose estimation (e.g., PoseCNN [25]), which assumes the availability of exact CAD models for all object instances; and category-level 6D object pose estimation (e.g., NOCS [23]), which defines a category-level reference frame and is able to generalize to a category of object instances. The motivation underlying the category-level approach is to develop systems that can accommodate novel object instances that were unseen during training. This approach is therefore more applicable for general robotic vision systems (e.g., of a home robot) that must face complex environments, such as those encountered in everyday situations.
† Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
To achieve the intra-category generalization, category-level 6D object pose estimation systems often require a significantly larger amount of annotated training data than classic instance-level systems. The reason for more training examples lies in the need to cover large intra-category shape variations among different object instances, in addition to the usual requirement for diversity in object poses, occlusion patterns, and environments. However, obtaining good 6D pose annotations is expensive and time-consuming. As a result, this line of research often relies heavily on synthetic training data. In order to scale up category-level object pose estimation systems to allow for more object categories, there is a strong demand for self-supervised methods that do not require pose annotations.
Designing self-supervised methods for category-level pose estimation is very challenging. Although a few systems (e.g., Self6D [22]) have recently been proposed for self-supervised instance-level 6D object pose estimation, all of those systems have access to the CAD model of each instance during training (but without the pose annotations). These previous approaches are thus inapplicable to self-supervised category-level object pose estimation, for which no CAD model is available for any object instances during either training or testing. The lack of any CAD models and annotations further presents a formidable obstacle to the designer: the reference frame to define category-level pose is unspecified throughout the training. To the best of our knowledge, there is no existing work on fully self-supervised category-level pose estimation, probably due to this issue.
We therefore need to find a way to allow the emergence of a category-level reference frame during training. A starting point is to consider the category-level reference frame known as Normalized Object Coordinate Space (NOCS), which is manually defined in [23]. Within the NOCS regression framework, all shapes are aligned in this space; also, dense canonical reconstruction of the visible points of the object of interest is performed. The relationship between the reference frame and canonical reconstruction inspires us to ask the following question: can a system learn to perform canonical reconstruction of object instances from the same object category, whose reconstruction space will naturally be a category-level reference frame? This canonical reconstruction presents two requirements: 1) all of the object reconstructions need to be aligned, and 2) the reconstruction should not change if the input object undergoes a change in pose. The second requirement suggests an SE(3)-invariant reconstruction of an observed object, which inspires us to consider point cloud processing networks that are SE(3)-invariant/equivariant.
Formally, given a set of point cloud transformations TA : RN×3 → RN×3 for A ∈ SE(3), a neural network φ : RN×3 → F (where F is an arbitrary feature domain) is called equivariant if for each A there exists an equivariant transformation SA : F → F so that:
SA[φ(X)] = φ(TA[X]), ∀A ∈ SE(3), X ∈ RN×3
Note that invariance is a special case of equivariance; the network is called invariant under the transformations when SA is an identity mapping.
Therefore, we propose to use an SE(3)-invariant network for shape reconstruction. To find a selfsupervision for this reconstruction task, we propose to jointly estimate the object pose P and use the predicted transformation in P to transform our reconstructions into the camera space so that a consistency loss can be formed between the proposed reconstruction and our input observation. In contrast to the pose invariance pursued by our shape reconstruction module, here the estimated 6D pose should naturally be equivariant with any SE(3) transformation applied to the input, which means that the estimated pose should undergo the same 6D transformation when the input object changes its 6D pose. We thus use an SE(3)-equivariant network for this pose estimation. Leveraging SE(3) invariance and equivariance, our proposed system is designed to disentangle shape and pose in a selfsupervised manner. Note that we do not explicitly enforce shape alignment in reconstruction. In other words, our network has the freedom to choose canonical poses of each instance for reconstruction during training. Even so, we find that our network automatically chooses to align all the instances for reconstruction (except for certain symmetry ambiguities), because the network is relatively “lazy” and such reconstruction is the easiest thing to do. Within this aligned reconstruction space, a categorylevel reference frame emerges without any supervision, which further defines our category-level pose that is to be estimated by our equivariant pose estimation module.
Extensive experiments show that this self-supervised disentanglement is successful for both complete and partial object observations. Our proposed method achieves very accurate 6D pose estimation on synthetic point clouds generated from ModelNet [24], and it works well on the real-world NOCSREAL275 dataset [23]. An ablation study further shows the key role of equivariance and invariance in this self-supervised framework.

heading: 2 Related Work
text: Supervised Category-Level Object Pose Estimation. The pioneering work of Wang et al. [23] proposed Normalized Object Coordinate Space (NOCS) as a category-specific canonical reference frame, so that the category-level pose of a previously unseen object can be defined as the transformation from its NOCS. Several follow-up efforts have improved NOCS by considering articulated objects [11], by incorporating object pose tracking [21], by leveraging analysis-by-synthesis and shape generative models [4, 6], or by exploiting learnable deformation [19]. However, all of these approaches adopt fully-supervised training paradigms and assume that object poses are known at training time. As discussed previously, object pose annotations are very difficult to obtain for category-level learning at a large scale.
Self-supervised Pose Estimation. To avoid the need for pose annotations in fully-supervised learning paradigms, several works have looked into self-supervision for instance-level object pose estimation. Self6D [22] and DTPE [15] obtain object poses by minimizing the difference between rendered depth (or RGBD) images and real observations in a self-supervised manner. AAE [16] trains a pose-augmented autoencoder on synthetic images, which can then be applied to pose estimation for real images. However, these methods still assume that a corresponding CAD model is available for the target object for which pose needs to be estimated. For category-level pose estimation without any CAD models, these approaches cannot be applied. An alternative approach known as shape co-alignment [1, 3, 12] allows emergence of consistent category-level reference frames. These methods usually assume clean meshes as inputs, or they leverage strong priors like symmetry, making them unsuitable for more challenging shapes or even partial observations exhibiting high geometrical or topological variations. Moreover, these methods do not make the connection between shape co-alignment and category-level 6D pose estimation. QEC [27] has a problem scope closest to ours. However, instead of aiming for self-supervised category-level 6D pose estimation for both complete and partial observations, this system deals only with 3D pose (3D rotation) of complete shapes. Also, QEC does not evaluate the predicted category-level 3D pose, and there is no guarantee regarding the pose quality. Other methods like [10, 20] require multi-view RGB images as input, and the primary focus is still on shape reconstruction. In summary, our work is the first to target self-supervised category-level object 6D pose estimation. This paper demonstrates significant potential of our method using only a depth point cloud as the input.
Equivariant Point Cloud Neural Networks. Recently, achieving SE(3) invariance and equivariance in point cloud processing has attracted a lot of attention. Early work [13, 26] focused more on achieving invariance, while most recent work has explored equivariance. Recent work has introduced Tensor Field Networks [18] and SE3-transformers [9], which leverage spherical harmonics; quaternion equivariant capsule networks [27]; and, most recently, Vector Neurons [7]. We note that EPN [5] is the only network for which equivariance does not entail a cost of additional network capacity.

heading: 3 Method
text: In section 3.1, we consider the Equivariant Point Network [5] with emphasis on its SE(3)-equivariance properties. Section 3.2 describes how we disentangle shape and pose, jointly estimate them, and form a self-supervised learning pipeline. Section 3.3 describes how category-level pose is inferred and evaluated at test time.

heading: 3.1 Revisiting Equivariant Point Networks
text: To achieve shape and pose disentanglement, our framework leverages a state-of-the-art SE(3) equivariant network known as the Equivariant Point Network (EPN) [5]. Taking a point cloud X as input, EPN achieves equivariance regarding both 3D translation t ∈ R3 and 3D rotation g ∈SO(3), and it can generate per-point localized equivariant features f(x) for points x ∈ X . We briefly review its translation and rotation equivariance properties.
Translation Equivariance. Considering 3D translation Tt[X] = X + t that moves X to X ′, a translation equivariant transformation can be defined as St = I (identity mapping). This definition of translation equivariance is widely used in other SE(3)-equivariant networks, e.g., Tensor Field
Network [18] and SE(3)-transformer [9]. Under this equivariance, f′(x′) = f′(x+t) = St[f(x)] = f(x), which means the feature f of one point x does not change under translation of the whole point cloud. The simplest way to produce this translation equivariance is to avoid directly processing the absolute point coordinates but instead use relative point offsets. EPN sets up local frames around each input point x and uses only the relative point offsets x̃i = xi − x during feature extraction. These point offsets can still depict the geometry but do not change under translation of the input.
Although any point network that uses only relative point offsets will be translation equivariant under this definition, EPN specifically chooses to use KPConv [17] as its convolution operator and performs convolution at each point x within its ball neighborhood Nx. More specifically, (f ∗ h)(x) =
∑ x̃i+x∈Nx f(x̃i)h(−x̃i), where h(x̃) is a continuously parameterized function in Nx.
Rotation Equivariance. When an input point cloud is transformed by an SO(3) rotation matrix, equivariance is a desired property, especially for the sake of rotation estimation. A non-trival (noninvariant) definition of rotation equivariance thus becomes very important. The key idea behind the rotation equivariance of EPN is to introduce a feature vector field F on an SO(3) manifold after discretizing the continuous SO(3) rotation group. EPN first proposes a kernel-rotated version of KPConv, which rotates the convolution kernel h by a rotation g ∈ SO(3). Using the rotated kernel hg to perform convolution on the input point cloud will result in a new set of per-point features f(x, g), equivalent to rotating the input X by g−1. In other words, its rotated kernel function satisfies
hg(x̃i) = h(Tg−1 [x̃i])
where g−1 is the inverse rotation of g. Assume for any g ∈ SO(3), EPN can compute per-point C-channel features f(x, g) ∈ RC using the corresponding rotated KPConv kernel hg , then the features for all g and all x form a per-point vector field F = φ(X) = {f(x, g) ∈ RC | g ∈ SO(3), x ∈ X}. We therefore can define a rotation equivariant transformation
Sg0 [f(x, g)] = f(x, (g0) −1 ◦ g).
Then we could prove
Sg0 [φ(X)] = {f(x, g−10 ◦ g)} = {f(Tg0 [x], g)} = φ(Tg0 [X]).
Intuitively, Sg0 will be equivalent to applying a circular shift to the vector field F with the value of each entry unchanged. In practice, it is infeasible to compute F for all rotations on SO(3), and EPN therefore leverages a discrete approximation and achieves equivariance for a finite subgroup of SO(3). Concretely, EPN discretizes the SO(3) rotation group into the icosahedron rotation group Gg , which contains 60 different rotations originating from the 60 rotational symmetries of a regular icosahedron and computes a rotated KPConv for each g ∈ Gg . This KPConv-based multi-rotation convolution is named PointConv, effectively generating a per-point 2D feature f ∈ RC×|G| for all x ∈ X . When
the input point cloud X is rotated by some g ∈ Gg, the rotation equivariance guarantees that each f will be permuted in its G dimension according to g. I.e., its j-th feature fj ∈ RC will become the j ′ -th feature of f′, satisfying gj′ = g ◦ gj . To further increase the expressivity of the network and to allow information exchange between different coordinates of F , EPN further introduces a GroupConv operation, which passes messages across different gs of f(x, g). This message passing leverages a rotation-invariant graph convolutional kernel and therefore does not break the equivariant structure of F . For more detailed information, please refer to EPN [5].
SE(3)-equivariance. By combining translation equivariance and rotation equivariance, we can define a rotation- and translation-equivariant transformation SA = St ◦Sg , so that EPN is strictly equivariant to all SE(3) transformations in GA = {TA | TA[X] = Tt ◦ Tg[X], ∀t ∈ R3, g ∈ Gg}. Note that GA is a subgroup of SE(3); we will discuss in section 3.2 how this subgroup approximation affects our task when the input undergoes rotations that are not in Gg .

heading: 3.2 Self-Supervised Joint Shape Reconstruction and 6D Pose Estimation
text: When we have only an input point cloud X ∈ RN×3 to provide supervision, jointly optimizing 3D shape and 6D pose is an ill-posed problem by nature. The network will need to jointly reconstruct an invariant canonical shape point cloud Z ∈ RNZ×3 and estimate a 6D object pose P = (q, t), whereN and NZ are the number of points in X and Z, respectively, and we choose to use quaternion q as our rotation representation. Here the invariant canonical shape reconstruction means the reconstruction should be invariant under all SE(3) transformations of the input point cloud. Furthermore, we want the reconstruction to be automatically aligned across different object instances under the same object category so that the reconstruction is under a category-level canonical reference frame. For equivariant 6D pose estimation, we want the estimated pose to change by the same SE(3) transformation A that is applied to the input object point cloud. I.e., P (TA[X]) = TA[P (X)], ∀A ∈ SE(3) (approximately). To achieve these two goals, we form a self-supervised loop by transforming the invariant canonical reconstruction Z by the 6D transformation TP involved in estimated P into a pose reconstruction Y = TP [Z] and enforcing consistency between input point cloud X and this transformed reconstruction Y . We will show that, although this self-supervised loop can be formed without leveraging SE(3) equivariance, it is crucial to use a SE(3)-equivariant network for creating a category-level canonical reference, which enables category-level 6D pose estimation.
SE(3)-equivariant Feature Backbone. Given an input object point cloud X (partial or complete), we use the EPN network as our SE(3)-equivariant backbone φ to extract per-point per-rotation SE(3)-equivariant features F ∈ Rn×C×|Gg| = φ(X) = {f(xi, gj)}, where |{xi}| = n and gj ∈ Gg. EPN comes with a stacked multi-scale encoder architecture, and will alternately perform PointConv and GroupConv in each convolution block while downsampling the points between two convolution blocks. We then feed the equivariant features over downsampled points to the following canonical shape reconstruction and 6D pose estimation branches, as shown in Figure 1.
SE(3)-invariant Canonical Shape Reconstruction Branch. The task of this branch is to learn an SE(3)-invariant canonical shape reconstruction function ρ : F ∈ Rn×C×|Gg| → Z ∈ RNZ×3. Ideally, we want Z to be invariant under arbitrary SE(3) transformations, i.e., Z(ρ(φ(X))) = Z(ρ(φ(TA[X]))), ∀A ∈ SE(3). In practice, we relax this constraint and instead require Z to be invariant under arbitrary transformations A ∈ GA. To do so, we only need this learned function ρ to be invariant under all transformations SA on F , i.e., Z = ρ(F ) = ρ(SA[F ]), ∀A ∈ SE(3), thanks to the equivariance of F .
To implement this invariant function ρ, we apply an elementwise MLPρ : f ∈ RC → fρ ∈ RCρ to F = {f(xi, gj)} that only operates on each f(xi, gj) individually, yielding a new set of features Fρ = {fρ(xi, gj) | fρ(xi, gj) = MLPρ(f(xi, gj)), ∀xi ∈ X ′, gj ∈ G}. To obtain a global SE(3)invariant feature Finv , we propose to perform max-pooling over {xi}, as done in PointNet [14]. Note that for all A ∈ GA, SA is equivalent to a permutation in the gj dimension of F (xi, gj) as well as that of Fρ(xi, gj). Therefore we only need Finv to be permutation-invariant to gj . This suggests another max-pooling over {gj}. Formally, Finv = max-poolingi,j({Fρ(xi, gj) ∈ RCρ}). Finally, we use a MLP-based point generator MLPZ to transform Finv into R3NZ and reshape into a point cloud Z ∈ RNZ×3. To facilitate a canonical reconstruction, we choose to use u() = Sigmoid()− 0.5 as the final activation so that the reconstruction can be as zero-centered as possible.
Although Z is only strictly invariant when g ∈ Gg , a rotation gx that is not in Gg can be decomposed into a main component gx0 = mingj∈Ggdist(gj , gx) and a small residual rotation ∆gx, so that gx = gx0 ·∆gx. We can then see that Z ′ = ρ(φ(Tgx [X])) = ρ(φ(Tgx0 [T∆gx [X]])) = ρ(φ(T∆gx [X])). In this way, to learn an invariant shape reconstruction, our network now only needs to be invariant to rotations in the quotient group SO(3)/Gg , which renders a much easier learning problem. This SE(3)invariance brings important consequences to the reconstructed point cloud Z: 1) if X represents a complete object point cloud, then the reconstructed Z will remain the same for this object under all 6D object poses; 2) for object instances with similar shapes from the same category, the network will lean towards emerging category-level aligned reconstructions, which may significantly simplify the task of the point cloud generation and resemble what happens in Quotient Network [12].
Note that when the input point cloud X only depicts a partial object, as in a depth image, a pose change in the object is not equivalent to a SE(3) transformation to the point cloud due to the change in point visibility. We find in our experiments that the emerging category-level alignment still holds, but there is no guarantee that the reconstructed point clouds will be complete.
SE(3)-equivariant 6D Pose Estimation Branch. The task of this branch is to learn a pose estimation function π from F to an equivariant 6D object pose P . By saying equivariance, we mean that the estimated pose of X ′ = TA[X] will undergo the same 6D transformation TA that applies to the pose of the input point cloud X . I.e., π(φ(X ′)) = π(φ(TA[X])) = TA[π(φ(X))] = TA[P ].
Inspired by the rotation estimation network introduced in EPN, we propose to predict a pose hypothesis |Gg| containing one 6D pose Pj for each gj ∈ Gg. We therefore propose to learn a function π : F ∈ Rn×C×|Gg| → {Pj} ∈ R(3+4)×|Gg|, where 3 is for translation, and 4 for a quaterion as our rotation representation. Here we parameterize each 6D pose Pj = (qj , tj) using four components (qj0 ,∆qj , t0,∆t), explained as follows: qj0 represents a major discrete quaternion corresponding to gj ∈ Gg and satisfies gj = minq∈Ggdist(g, gj); ∆qj represents the residual quaternion, which can be composed with qj to form the full quaternion qj = qj0∆qj ; t0 = X̄ is the weight center of the input point cloud X and is equivariant to the translation of X and remains invariant for different gs; and ∆tj represents the residual translation under rotation gj . This pose parameterization will result in qj = qj0∆qj and tj = qj0∆tq −1 j0
+ t0, which will pose a canonical object point cloud Z into Z ′
j = qjZqj−1 + tj .
To achieve pose equivariance, we adopt a similar strategy to our invariant reconstruction branch but without max-pooling over gj . We learn an elementwise MLPπ : RC → RCπ to process each f(xi, gj) ∈ F individually, which transforms F into Fπ. We then perform max-pooling on Fπ over {xi}, yielding an equivariant global feature Fequiv ∈ RCπ×|Gg| = {fequiv(gj) ∈ RCπ | gj ∈ Gg}. We further learn a per-rotation MLPP : RCπ → R7 to transform each fequiv(gj) to a residual quaternion ∆qj and a residual translation ∆tj . This pose estimation branch is equivariant with any SE(3) transformation A ∈ GA. Similar to our argument for the invariant reconstruction, this subgroup equivariance will significantly ease the learning of pose estimation equivariance with arbitrary SE(3) transformations.
This 6D pose estimation branch has several important differences to the rotation estimation branch used in EPN. First, due to the self-supervised nature of this work, we do not know which rotation gj is the closest one to the ground truth object rotation and we therefore choose not to predict a probability distribution over gj . Second, to enable 6D pose estimation, we additionally estimate a residual translation ∆t. Finally, EPN does not constrain the range of ∆qj because it knows which gj is the best main rotation component and can supervise the residual accordingly. Here, to avoid an overparameterization of rotation, we constrain the residual quaternion ∆qj to be a small quaternion that can only compensate for the rotation discretization introduced by Gg but should not go beyond the zone of each qj0 . Note that the relationship between a quaternion q = (qw, qx, qy, qz) and an axis-angle (ω, θ) allows us to constrain the rotation angle of one quaternion easily: qw = cos(θ/2), where θ is the rotation angle. Given that the rotation angle between any two nearest-neighboring elements from icosahedron rotational group G is π/5, we propose to set θ(∆q) < π/5 to cover the whole SO(3) or S3 in the quaternion space while avoding being too overparameterized. This sets a constraint to qw of ∆q to be qw ≥ cos(π/10). Also, for a unit quaternion, qw ≤ 1. We can implement these two constraints via a customized activation function: cos(π/10) + (1− cos(π/10)) ·Sigmoid().
Self-supervised Loss. Given the canonical reconstructed point cloud Z and a set of 6D pose hypotheses {Pj = (qj , tj)}, we can obtain pose Z using each Pj , rendering posed reconstructions Yj = qjZ(qj)−1 + tj . We then can enforce a minimum-of-N loss between the input observation X and the posed point clouds:
Lrec = min j∈{1,...,|Gg|} d(X,Yj),
where d : RNX×3 × RNY ×3 → R represents a point cloud distance function. For complete object observation, we can choose d to be bidirectional chamfer distance [8] dCD(X,Y ) = 1NX ∑ x∈X min y∈Y ||x−
y||2 + 1NY ∑
y∈Y min x∈X ||x− y||2. For partial point cloud observation, due to the incompleteness of X , we can only enforce a unidirectional chamfer distance dUnidirCD(X,Y ) = 1NX ∑
x∈X min y∈Y ||x− y||2, which
only enforces each point in the partial point cloud X can find a closed point in Y . Note that the angle constraint of ∆q implemented by constraining the range of its qw only makes sense when ∆q is a unit quaternion. We therefore additionally enforce a regularization loss Lreg = (||∆q|| − 1)2. The total loss is L = Lrec + λLreg, where λ is a regularization weight.

heading: 3.3 Category-Level 6D Object Pose Inference and Evaluation
text: Category-Level 6D Object Pose Inference. Given an object point cloud X (partial or complete), we feed it to our network and generate a reconstruction Z, a set of pose hypotheses {Pj}, and the corresponding posed reconstruction {Yj}. We can then obtain its category-level 6D object pose estimation P̂ = (q, t) via selecting the best pose hypothesis P̂ = (qj∗ , tj∗) that satisfies j∗ = argmin
j∈{1,...,|Gg|} d(X,Yj). This best pose hypothesis is the 6D object pose of the observed object
instance in the reference frame of reconstruction Z.
Evaluating Category-Level 6D Object Pose. To quantitatively evaluate the performance of our self-supervised category-level 6D object pose estimation algorithm on test point cloud data {Xk} from a given category, we need to have the ground truth category-level pose annotations P ∗k = (q∗k, t ∗ k) for all test data Xk, which uniquely defines a category-level pose reference frame R∗.
Because our estimated poses P̂k are in the reference frames Rk of reconstruction Zk, there can be center shift and orientation differences between Rk and R or even among different Rks. We therefore need to find a consistent reference frame R̂ and register it to R∗. We propose the following registration algorithms.
For each test dataXk, we transform it using the inverse transformation of P ∗k , yielding a canonicalized object point cloud Z∗k = (q∗k)−1Xkq∗k − t ∗ k. We then feed Z ∗ k to our network for pose estimation and obtain a pose misalignment (q̃k, t̃k). We propose to use a RANSAC-based method to find a majority consensus of q̃ and t̃. Then, for all test data, our final pose prediction can be computed as (q̂q̃−1, t̂− q̂q̃−1t̃q̃q̂−1) using our network’s raw pose prediction (q̂, t̂).

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation
text: Datasets. We evaluate our method using the following datasets. 1) Complete shape point clouds from ModelNet40 [24]. We first test our method on ModelNet40 with five categories, including airplane, car, chair, sofa and bottle. Random rotation is applied to each input data during training, and 5 different rotations are randomly sampled for each test instance. We follow the original train/test split from the ModelNet40 dataset. 2) Depth point clouds of objects from ModelNet40. We evaluate our method on partial depth observations by rendering the depth images of the complete object instances described above. For depth rendering, we first put the camera at a fixed distance facing the object, and the object is allowed to have small offsets and random rotations. For each category, we generate 30K training images with 6K testing images. The dataset is made public in our project page. 3) Real-world depth point clouds from the NOCS dataset [23]. We further evaluate our algorithm on the partial object depth point clouds from the NOCS-REAL275 dataset . This dataset contains a synthetic training set with 275K discrete frames generated with 1085 object models in the classes
from ShapeNet [2] along with 7 real training videos capturing 3 different instances of objects for each category. Its testing set, NOCS-REAL275, has 3,200 frames collected from 6 real videos containing novel instances. We assume that the ground truth object mask is available in our experiments and use them to crop the objects from the depth.
Metrics. We perform all pose evaluation on held-out instances per category. For each category, we report rotational error Rerr, and translational error Terr in the form of mean, and median values, together with 5° and 5cm accuracy. For symmetric objects such as bottles, we evaluate the angular error of the up-axis direction.
Baselines. We compare against two supervised methods: EPN and KPConv, which serve as performance oracles. We utilize a tuned implementation of the ICP algorithm as a traditional unsupervised method, where we use the same set of 60 rotations from icosahedron group for rotation initializations. Because we assume that we do not have complete ground-truth shapes for testing, for ICP we randomly select an example shape, and iteratively register the observation to the example shape. Note there are no existing learning-based unsupervised methods that fit our setting, especially on partial shapes with both 3D rotation and 3D translation.

heading: 4.2 Experiments on Synthetic Object Point Clouds
text: 3D pose estimation from complete object point clouds. We first consider rotation of complete shapes. As shown in Table 1, although our method is fully self-supervised, it achieves comparable results to supervised method using state-of-the-art point cloud learning models such as EPN and KPConv. For certain categories, our method even surpassed the supervised oracles such as the mean rotational error on chair, and median rotational error on bottle. EPN gives the best supervised performance on all the categories except for bottle, mainly due to the difficulty of rotational mode classification under rotational symmetry ambiguity. Without the equivariance property on full SO(3) space, KPConv struggles to give an accurate rotation estimation, and our self-supervised pipeline out-performs it on all 4 asymmetric categories, while performing well on symmetric bottle. For simple shapes like airplane and car, ICP algorithm could achieve a reasonable estimation after iterative alignment, but fails at chair and sofa category. It also does not work well for the bottle case as the given example shape does not match well with all different bottle instances.
6D pose estimation from partial object depth point clouds. When the input observation is partial, there will be more ambiguities in determining object canonical pose. Despite this challenge, we observe that our method is still able to infer well-aligned canonical shapes across different instances, as shown in Table 2 and Figure 2. For some object categories, our method is able to infer a complete
object shape by training on different view points. However, for bottle, the algorithm only considers the object shape as a ’half’ bottle due to the symmetry – partial bottle shape can already cover all possible view points. Quantitatively, results in Table 2 show that our method achieves the best accuracy on both partial airplane and chair data. It also handles the symmetric object like bottle well, while the two supervised methods suffer from performance degradation due to the increased difficulty.
Effects of equivariance. As shown in Table 3, we compare the method’s performance when we use equivariant or non-equivariant backbones. The results demonstrate the importance of using equivariant neural networks for canonical shape reconstruction. Leveraging SE(3)-equivariance makes it possible to generate consistently aligned canonical shapes, which in turn helps get accurate pose estimation, using non-equivariant backbone for shape would fail totally. We also show that using equivariant backbone for direct pose regression would further improve the pose estimation accuracy, especially on partial shapes. Different equivariant backbones. As shown in Table 4, we compare the performance of different equivariant backbones, with both SE(3)-equivariant networks like EPN and SE(3) transformer. SE(3) transformer also promises rotational and translational equivariance by leveraging properties of spherical harmonics.From our experiments, it could also get consistently aligned canonical reconstructions. However, compared to EPN, SE(3) transformer is way more computationally heavier, and SE(3) transformer struggles to generate accurate rotation estimation compared to EPN, especially on partial shapes. EPN-20 is another variant of EPN network, when the full SO(3) space is divided into 20 different sub-regions, which has smaller number of sub-spaces than our finalized EPN-60 backbone. From our experiment it doesn’t get better performance than EPN-60. Based on these experimental findings, we finally choose EPN-60 as our SE(3)-equivariant backbone. Failure cases. When heavy self-occlusions and shape symmetry present, the performance of our model tends to degrade. We observe both sofa and car categories have a lot of 180-degree-flip,
which means the model couldn’t distinguish dominant direction like front and back. The error of rotation estimation also contributes to the translational error for car and sofa. For the bottles, we also observe multiple cases where the reconstructed shape is upside down and an considerable offset to the dominant object center which results in the poor performance for translation estimation. See our appendix for the error percentile curves.

heading: 4.3 Experiments on Real-World Object Point Clouds
text: To test our method under a more realistic setting and explore the limits of its performance, we examine our method on the challenging real world depth data from NOCS-REAL275[23] dataset, where sensor noises and heavy occlusions in the real world cluttered scene further complicates the problem, e.g. the delicate handle structure of the mug category is often invisible, making it difficult to estimate its pose and shape. In addition, different from our randomly rotated synthetic data, this real dataset contains a biased pose distribution, e.g. bottles are always standing upright and the bottom part is never observed, adding difficulty to the reconstruction task. We consider five different experiment settings: 1) perform supervised pretraining on CAMERA[23] dataset, similar to EPN; 2) self-supervisedly finetune the pretrained weight of 1) using the small train split of NOCS; 3) self-supervisedly pretrain on CAMERA; and finally 4) self-supervisedly finetune the pretrained weight of 3) on NOCS-train, together with 5) ICP algorithm. We report a full evaluation of category-level 6D pose estimation on our NOCS-REAL275 dataset using different methods in Table 5. Our fully unsupervised method beats ICP on almost all categories and performs quite well on bottle, bowl, and can, without using any labelled or real data. Self-supervised finetuning on real data further improves its performance.

heading: 5 Conclusion
text: This paper has presented a novel self-supervised network that is capable of category-level 6D object pose estimation. The system operates without CAD models or annotations, and it achieves impressive accuracy on both synthetic and real-world category-level pose estimation benchmarks. This work is the first to demonstrate the power of SE(3)-equivariant point cloud networks for 6D pose estimation, especially on partial object point clouds. As limitations, the performance of SE(3) equivariant networks may degrade when objects exhibit symmetries, or viewpoint distribution is highly biased, or occlusions are always heavy. Extending this work to a semi-supervised setting may help to relieve these limitations, which we leave for future works. This work can reduce the demand of 6D pose annotation, therefore may impose negative impacts to people who work on object pose annotations.
Acknowledgement: This research is supported by a Vannevar Bush faculty fellowship, NSF grant IIS-1763268, and gifts from the Adobe and Autodesk Corporations. We also appreciate resources provided by Advanced Research Computing in the Division of Information Technology at Virginia Tech.

Review summaries:
The paper proposes a framework to learn 6D object pose estimation without GT pose annotations and CAD models (while GT object masks are available, and each object is already isolated). The key is to jointly learn canonical shapes and poses by a SE(3)-equivariance network. It shows promising performance on both synthetic and real depth data.

This paper proposes a self-supervised approach to 6 DoF pose estimation. Usually such methods are either instance-level, assuming known CAD models for all object instances; or category-level, assuming a reference frame that can be generalized to all instances of a certain category. The second approach is more generalizable, however it requires a substantial amount of annotated training data, often from synthetic datasets. So far, self-supervision in this setting has been limited to instance-level, with CAD models but without pose annotations. This paper proposed category-level self-supervision and claims to be the first to do so, eliminating the need of both CAD models and large-scale annotated data.