Leveraging SE(3) Equivariance for Self-supervised Category-Level Object  Pose Estimation from Point Clouds
Authors: Xiaolong Li
Yijia Weng
Li Yi
Leonidas Guibas
A. Lynn Abbott
Shuran Song
He Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: 6D Pose
self-supervision
SE(3) equivariance
3D point cloud
category-level
real data
shape reconstruction

Abstract: Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models. To reduce the huge amount of pose annotations needed for category-level learning, we propose for the first time a self-supervised learning framework to estimate category-level 6D object pose from single 3D point clouds. During training, our method assumes no ground-truth pose annotations, no CAD models, and no multi-view supervision. The key to our method is to disentangle shape and pose through an invariant shape reconstruction module and an equivariant pose estimation module, empowered by SE(3) equivariant point cloud networks. The invariant shape reconstruction module learns to perform aligned reconstructions, yielding a category-level reference frame without using any annotations. In addition, the equivariant pose estimation module achieves category-level pose estimation accuracy that is comparable to some fully supervised methods. Extensive experiments demonstrate the effectiveness of our approach on both complete and partial depth point clouds from the ModelNet40 benchmark, and on real depth point clouds from the NOCS-REAL 275 dataset. The project page with code and visualizations can be found at: dragonlong.github.io/equi-pose.

Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation
Xiaolong Virginia Tech
Yijia Weng
Li Tsinghua University ericyi01248 gmail com
Peking University
1xi 1xiaol9@vt.ed
hal Asmmmemeer gmmam............ com
Leonidas Guibas
A. Lynn Abbott Virginia Tech
Stanford University
guibas@cs. stanford.
abbott@@vt.edu
Shuran Song Columbia University shurans@cs. co www.oollliio. wwmluum.eee.. edu
He Wang
Peking University hewangêpku. edu. cn
Abstract
Category-level object pose estimation aims to find 6D object poses of previously unseen object instances from known categories without access to object CAD models. To reduce the huge amount of pose annotations needed for category- level learning, we propose for the first time self supervisdd learning framework to estimate category-level 6D object pose from single 3D point clouds. During training, method assumes ground-truth pose annotations, no CAD models, and no multi-view supervision. The key to our method is to disentangle shape and pose through invariant shape reconstruction module and equivariant pose estimation module, empowered by SE(3) equivariant point cloud networks. The invariant shape reconstruction module learns perform aligned reconstruc- tions, yielding a category-level reference frame without using any annotations. In addition, the equivariant pose estimation module achieves category-level pose estimation accuracy that is comparable to some fully supervised methods. Extensive experiments demonstrate the effectiveness of our approach on both complete and partial depth point clouds from the ModelNet40 benchmark, and on real depth point clouds from the NOCS-REAL 275 dataset. The project page with code and visualizations can be found at: dragonlong, wealooonngggtuiiiiiooo github.io/equi-pss..
Introduction
Object pose estimation is crucial computer vision task that is widely employed in robotics, human- object interaction, and augmented-reality applications. Previous work can categorized broadly into two genres: classic instance-level 6D object pose estimation (c.g., PoseCNN [25]), which assumes the availability exact CAD models for all object instances; and category-level 6D object pose estimation (e.g.. NOCS [23]), which defines category-level reference frame and is able to generalize to category of object instances. The motivation underlying the category-level approach to develop systems that accommodate novel object instances that were unseen during training. This approach therefore more applicable for general robotic vision systems (e.g., of home robot) that must face complex environments, such those encountered in everyday situations.
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To achieve the intra-category generalization, category-level 6D object pose estimation systems often require a significantly larger amount of annotated training data than classic instance-level systems. The reason for more training examples lies in the need to cover large intra-category shape variations among different object instances, in addition to the usual requirement for diversity in object poses, occlusion patterns, and environments. However, obtaining good 6D pose annotations is expensive and time-consuming. As a result, this line of research often relies heavily on synthetic training data. In order to scale up category-level object pose estimation systems to allow for more object categories, there is a strong demand for self-supervised methods that do not require pose annotations.
Designing sulf-super methods for category-level pose estimation is very challenging. Although a few systems (e.g.. Self6D [22]) have recently been proposed for self-supervvsed instance-level 6D object pose estimation. all of those systems have access to the CAD model of each instance during training (but without the pose annotations). These previous approaches are thus inapplicable to seff-supervised category- object pose estimation, for which no CAD model is available for any object instances during either training or testing. The lack of any CAD models and annotations further presents formidable obstacle to the designer: the reference frame to define category-level pose is unspecified throughout the training. To the best of our knowledge, there is no existing work on fully self-supervised category-level pose estimation, probably due to this issue.
We therefore need to find a way to allow the emergence of a category-level reference frame during training. A starting point is to consider the category-level reference frame known as Normalized Object Coordinate Space (NOCS), which is manually defined in [23]. Within the NOCS regression framework, all shapes are aligned in this space; also, dense canonical reconstruction of the visible points of the object of interest is performed. The relationship between the reference frame and canonical reconstruction inspires us to ask the following question: can system learn to perform canonical reconstruction of object instances from the same object category, whose reconstruction space will naturally be category-leve' reference frame? This canonical reconstruction presents two requirements: 1) all of the object reconstructions need to be aligned, and 2) the reconstruction should not change if the input object undergoes a change in pose. The second requirement suggests an SE(3)-invariant reconstruction of an observed object, which inspires us to consider point cloud processing networks that are SE(3) (ns333)nvaainn/qquuvvriinn
Formally, given a set of point cloud transformations Ta Nx3 N for A E SE(3), a neural network Φ Nx3 + F (where F is an arbitrary feature domain) is called equivariant if for each A there exists an equivariant transformation SA F F so that:
SalØ(x)] = (TAI[T)) AA E SE(3), X
Note that invariance is special case of equivariance; the network is called invariant under the transformations when SA is an identity mapping.
Therefore, we propose to use an SEE3)-invariant network for shape reconstruction. To find self- supervision for this reconstruction task, we propose to jointly estimate the object pose P and use the predicted transformation in P to transform our reconstructions into the camera space so that a consistency loss can be formed between the proposed reconstruction and our input observation. In contrast to the pose invariance pursued by our shape reconstruction module, here the estimated 6D pose should naturally be equivariant with any SE(3) transformation applied to the input, which means that the estimated pose should undergo the same 6D transformation when the input object changes its 6D pose. We thus use an SEu3)-equivarian network for this pose estimation. Leveraging SE(3) invariance and equivariance, our proposed system is designed to disentangle shape and pose in a self- supervised manner. Note that we do not explicitly enforce shape alignment in reconstruction. In other words, our network has the freedom to choose canonical poses of each instance for reconstruction during training. Even so, we find that our network automatically chooses to align all the instances for reconstruction (except for certain symmetry ambiguitics), because the network is relatively "lazy" and such reconstruction is the casiest thing to do. Within this aligned reconstruction space, a category- level reference frame emerges without any supervision, which further defines our category-level pose that is to be estimated by our equivariant pose estimation module.
Extensive experiments show that this self-supervised disentanglement is successful for both complete and partial object observations. Our proposed method achieves very accurate 6D pose estimation on synthetic point clouds generated from ModelNet [24], and it works well on the real-world NOCS- REAL2275 dataset [23]. An ablation study further shows the key role of equívariance and invariance in this self-supervised framework.
2

---

2 Related Work
Supervised Category-Leve Object Pose Estimation. The pioneering work of Wang et al. [23] proposed Normalized Object Coordinate Space (NOCS) as category-specific canonical reference frame, so that the category-level pose of previously unseen object can be defined the transfor- mation from its NOCS. Several follow-up efforts have improved NOCS by considering articulated objects [11], by incorporating object pose tracking [21], by leveraging
and shape generative models [4, 6], or by exploiting learnable deformation [19]. However, all of these approaches adopt fully-supervised training paradigms and assume that object poses are known at training time. As discussed previously, object pose annotations are very difficult to obtain for category-level learning at a large scale.
Self-supervised Pose Estimation. To avoid the need for pose annotations in fully-supervised learning paradigms, several works have looked into self-supervision for instance-level object pose estimation. Self6D [22] and DTPE [15] obtain object poses by minimizing the difference between rendered depth (or RGBD) images and real observations in a self-ssuer manner. AAE [16] trains pose- anse-augmented autoencoder on synthetic images, which can then be applied to pose estimation for real images. However, these methods still assume that a corresponding CAD model available for the target object for which pose needs to be estimated. For category-level pose estimation without any CAD models, these approaches cannot be applied. An alternative approach known as shape co-alignment [1, 3, 12] allows emergence of consistent category reference frames. These methods usually assume clean meshes as inputs, or they leverage strong priors like symmetry, making them unsuitable for more challenging shapes even partial observations exhibiting high geometrical or topological variations. Moreover, these methods do not make the connection between shape co-alignment and category-level 6D pose estimation. QEC [27] has problem scope closest ours. However, instead of aiming for self-supervised category-level 6D pose estimation for both complete and partial observations, this system deals only with 3D pose (3I) rotation) of complete shapes. Also, QEC does not evaluate the predicted category-level 3D pose, and there no guarantee regarding the pose quality. Other methods like [10, 20] require multi-view RGB images input, and the primary focus is still on shape reconstruction. In summary, our work is the first to target self-supervised category-level object 6D pose estimation. This paper demonstrates significant potential of our method using only depth point cloud as the input.
Equivariant Point Cloud Neural Networks. Recently, achieving SE(3) invariance and equiv- ariance in point cloud processing has attracted a lot of attention. Early work [13, 26] focused more on achieving invariance, while most recent work has explored equivariance. Recent work has intro- duced Tensor Field Networks [18] and SE3-transformers [9], which leverage spherical harmonics; quaternion equivariant capsule networks [27]; and, most recently, Vector Neurons [7]. We note that EPN [5] is the only network for which equivariance does not entail a cost of additional network capacity.
3 Method
In section 3.1, we consider the Equivariant Point Network [5] with emphasis on its SEuii-oquivariance properties. Section 3.2 describes how we disentangle shape and pose,jointlly estimate them, and form a self-supervised learning pipeline. Section 3.3 describes how category-level pose is inferred and evaluated test time.
3.1 Revisiting Equivariant Point Networks
To achieve shape and pose disentanglement, our framework leverages a state-of-the-art SE(3) equiv- ariant network known the Equivariant Point Network (EPN) [5]. Taking point cloud X as input, EPN achieves equivariance regarding both 3D translation E R and 3D rotation ESO(3), and generate per-poin localized equivariant features f(x) for points E X. We briefly review its translation and rotation equivariance properties.
Translation Equivariance. Considering 3D translation T[X] = X t that moves X to X', translation equivariant transformation can be defined as S, (identity mapping). This definition of translation equivariance is widely used in other SE33)--quivariant networks, e.g., Tensor Field
3

---

SE3-invariant canonical reconstruction branch
Per-rotation input
N2x3
reconstructions Vj
Input point cloud X E Wx3
Pointwise MLP max poo-lpoling over (No 0y²
14, (4) (4,44) Aq. + q Atq
Invariant global feature PI
SE(3)- equiv. features F [f((x, B
Equivariant Point Network
Equivariant global features
Per-rot:
residual poses (t)
MIP
Pointwise MLP max-pooling over [xi]
Feguiu
(feguio (0))
SE3-equivariant pose branch
Loss L min min v/) A(lAqq 1)²
Figure I: Self-supervised category-level pose estimation pipeline. Taking an input point cloud X (partial or complete), our network can construct invariant canonical shape Z and estimate equiv- ariant pose P. which together can form self-supervised consistency loss to the input observation.
Network [18] and trans-rransf [9]. Under this equivariance, f(x') ((x-)) - which means the feature f of one point x does not change under translation of the whole point cloud. The simplest way to produce this translation equivariance is to avoid directly processing the absolute point coordinates but instead use relative point offsets. EPN sets up local frames around each input point and uses only the relative point offsets X, offsets can still depict the geometry but do not change under translation of the input.
during feature extraction. These point
Although any point network that uses only relative point offsets will be translation equivariant under this definition, EPN specifically chooses to use KPConv [17] as its convolution operator and performs convolution at each point x within its ball neighborhood Nr. More specifically, * (x)x)= Σ f(x;))(--x;) where (xx) is continuously parameterized function in NV.
Rotation Equivariance. When an input point cloud is transformed by an SO(3) rotation matrix. cquivariance is desired property. especially for the sake of rotation estimation. A non-trival (non- invariant) definition of rotation equivariance thus becomes very important. The key idea behind the rotation equivariance of EPN is to introduce a feature vector field F on an SO(3) manifold after discretizing the continuous SO(3) rotation group. EPN first proposes a kernel-rotated version of KPConw, which rotates the convolution kernel h by rotation SO(3). Using the rotated kernel hg to perform convolution on the input point cloud will result new set of per-point features f(x, g). equivalent to rotating the input X by -1 In other words, its rotated kemnel function satisfies
hg (x,) h(T [x:))
where g the inverse rotation of g. Assume for any g SO(3), EPN can compute per-point C-channel features f(x,g) g) E R using the corresponding rotated KPConw kernel hg, then the features for all g and all x form a per-point vector field F = s(X) {f(x, .) E RC | E SO(3), x E X}. We therefore can define rotation equivariant transformation
Sg [[(x g)] f(x, (go) g))
Then could prove
S = (f(xx [[[(x]9''' 95
-0) -
= ( (x). = (TT
Intuitively, Sn, will be equivalent to applying a cireular shift to the vector field with the value of each entry unchanged. In practice, it is infeasible to compute F for all rotations on SO(3), and EPN therefore leverages discrete approximation and achieves equivariance for finite subgroup of SO(3). Concretely, EPN discretizes the SO(3) rotation group into the icosahedron rotation group Gg. which contains 60 different rotations originating from the 60 rotational symmetries of a regular icosahedron and computes rotated KPConv for each Gg. This KPConv- multi-rotation convolution is
named PointConv, effectively generating per-point 2D feature e CC|GG| for all X. When

---

the input point cloud X is rotated by some g E Gg. the rotation equivariance guarantees that each f will be permuted in its G dimension according to I.e. its j-th feature Ij E Re will become the j' o'tth feature of f', satisfying Φ g o 9j. To further increase the expressivity of the network and to allow information exchange between different coordinates of F, EPN further introduces a GroupConv operation, which passes messages across different gs of f(x: g). This message passing leverages a votation-invarial graph convolutional kernel and therefore does not break the equivariant structure of F. For more detailed information, please refer to EPN [5].
-quivaaaie, By combining translation equivariance and rotation equivariance, we can define rotation- and translation-equivariiee to all SE(3) transformations in GA {TA TX[X]= Ti o Tg [X], Vt E R², 9 E Gg). Note that G'A a subgroup of SE(3); we will discuss in section 3.2 how this subgroup approximation affects our task when the input undergoes rotations that are not in Gg-
transformation SA- = S Sg, so that EPN is strictly equivariant
3.2 Self-Supervised Joint Shape Reconstruction and 6D Pose Estimation
When we have only an input point cloud X E RNx3 to provide supervision, jointly optimizing 3D shape and 6D pose is an ill-posed problem
nature. The network will need to jointly reconstruct an invariant canonical shape point cloud Z E R R.N2 x3 and estimate 6D object pose P = (q. t). where N and Nz are the number points in X and Z, respectively, and we choose to use quaternion q as our rotation representation. Here the invariant canonical shape reconstruction means the reconstruction should be invariant under all SE(3) transformations of the input point cloud. Furthermore. we want the reconstruction to be automatically aligned across different object instances under the same object category so that the reconstruction is under a category-level canonical reference frame. For equivariant 6D pose estimation, we want the estimated pose to change by the same SE(3) transformation A that is applied to the input object point cloud. I.e., (XT) = [X)), A E SE(3) (approximately).
To achieve these two goals, we form a self-supervised loop by transforming the invariant canonical reconstruction by the 6D transformation Tp involved in estimated P into a pose reconstruction Y Tr(2 and enforcing consistency between input point cloud X and this transformed reconstruction Y. We will show that, although this self-supervised loop can be formed without leveraging SE(3) equivariance, it is crucial to use SE(3)-equivariant network for creating category-leve canonical reference, which enables category-level 6D pose estimation.
-E(3)-equivariant Feature Backbone. Given an input object point cloud X (partial or complete), we use the EPN network as our equivvraan backbone 6 to extract per-point per-rotation
SE(()--quivariant features F E Ra G, = x(X) X) (f(x., 9; where (xx)] = n and gj E Gg- EPN comes with a stacked multi-scale encoder architecture, and will alternately perform PointConv and GroupConv in each convolution block while downsampling the points between two convolution blocks. We then feed the equivariant features over downsampled points to the following canonical shape reconstruction and 6D pose estimation branches, as shown in Figure
SE(3)-inarriant Canonical Shape Reconstruction Branch. The task of this branch is to learn
an SE(3)-invarian canonical shape reconstruction function p F E |2xC'|G, + Z R x3 Ideally, we want to be invariant under arbitrary SE(3) transformations, i.e., Z(p(o())) (X)) = Z (x)))Ta(X))) AA E SE(3). In practice, we relax this constraint and instead require Z to be invariant under arbitrary transformations A E GA. To do so, we only need this learned function to be invariant under all transformations SA on F, Le.., Z pl (F) [F)) VA E SE(3), thanks to the equivariance of F.
To implement this invariant function p. apply an elementwise MLP, E RCC + + E R. to F = [f(x,, gj}} that only operates on each f(x,, 95) individually, yielding a new set of features F2- = [f, 93) fx(,,, 95) = MLP, (f(x,, ),, VX, E X', 9j E G). To obtain a global SE(3)- invariant feature Fines we propose to perform max- max- pooling over {x:}. as done in PointNet [14]. Note that for all A G., SA is equivalent to permutation in the 9, dimension of F F(x, 9j) as well as that of Fx(x,,95) Therefore we only need Fine to be permutation-invaritn
to gj- This suggests another max-pooling over {g;}. Formally, Fine = max-pooling,, ( (x,, ((xx,,9)) 95) E RC))). Finally, we use a MLP-based point generator MLPz to transform Fin. into RaNz and reshape into point cloud Z E N2x3 To facilitate a canonical reconstruction, we choose to use u() = Sigmoid() - 0.5 as the final activation that the reconstruction can be as zero-centered possible.
5

---

Although Z is only strictly invariant when E Gg. rotation Ir that is not in Gg can be decomposed into a main component 9xp min dist(g), 9z) and a small residual rotation Ag.. so that 9.-
Ie. Ag. We can then see that Z' p s. [x]) p this way, to learn an invariant shape reconstruction, our network now only needs to be invariant to rotations in the quotient group SO(3)/G,, which renders much easier learning problem. This SE(3- invariance brings important consequences to the reconstructed point cloud Z: 1) if X represents a complete object point cloud, then the reconstructed Z will remain the same for this object under all 6D object poses; 2) for object instances with similar shapes from the same category, the network will lean towards emerging category-level aligned reconstructions, which may significantly simplify the task of the point cloud generation and resemble what happens in Quotient Network [12].
(X]])) = p(o [X])). In
Note that when the input point cloud X only depicts partial object, as in depth image, a pose change in the object is not equivalent to a SE(3) transformation to the point cloud due to the change in point visibility. We find in our experiments that the emerging category-level alignment still holds, but there is no guarantee that the reconstructed point clouds will be complete.
SE(3)-eqivarriant 6D Pose Estimation Branch. The task of this branch is to learn pose estimation function π from F to an equivariant 6D object pose P. By saying equivariance, we mean that the estimated pose of X' Ta(X] will undergo the same 6D transformation TA that applies to the pose of the input point cloud X. I.e., T(((X))) = π (
= [T(T ( (())) = [P]
Inspired by the rotation estimation network introduced in EPN, we propose to predict pose hypothesis GyI containing one 6D pose P3 for each gj E Gg. We therefore propose to learn a function π= F E RTxCX|GGl - {P,} E R(344)x|Gol, where 3 is for translation, and 4 for a quaterion rotation representation. Here we parameterize each 6D pose P, = (q, t,) using four components (q), Aqj. to, At), explained as follows: 43 represents major discrete quaternion corre- sponding 9 E Gg and satisfies 9j minyeg, disi.isssts))) g;); Aq, represents the residual quaternion, which can be composed with q; to form the full quaternion q; = // Aqj; to X is the weight center of the input point cloud X and is equivariant to the translation of X and remains invariant for different gs; and At, represents the residual translation under rotation gy. This pose parameterization will result in q, 4jo Ag, and t, = 4jo Atq 1 + to. which will pose a canonical object point cloud Z into Z,- 9, Zq, +t,.
To achieve pose equivariance, we adopt a similar strategy to our invariant reconstruction branch but without max-pooling over 9j. We learn an elementwise MLP- RC + RC to process each f(x,,;)) E F individually, which transforms F into Fz. We then perform max-pooling on Fa over {x.). yielding an equivariant global feature Fequiv E gC, X = (g;) E RC 9j E G, We further learn a per-rotation MLPp RC* + R² to transform each (gj to a residual quaternion Aq, and a residual translation Aty. This pose estimation branch is equivariant with any SE(3) transformation A E G.. Similar to our argument for the invariant reconstruction, this subgroup cquivariance will significantly ease the learning of pose estimation equivariance with arbitrary SE(3) transformations.
This 6D pose estimation branch has several important differences to the rotation estimation branch used in EPN. First, due to the self-supervised nature of this work, we do not know which rotation 9j is the closest one to the ground truth object rotation and we therefore choose not to predict probability distribution over 9j- Second, to enable 6D pose estimation, we additionally estimate a residual translation At. Finally, EPN does not constrain the range of Ag, because it knows which 9j is the best main rotation component and can supervise the residual accordingly. Here, to avoid an overparameterizztion of rotation, constrain the residual quaternion Aq, to be small quaternion that can only compensate for the rotation discretization introduced by Gg but should not go beyond the zone of each 9,0- Note that the relationship between quaternion q (ge, (er Trc qy 9=) and an axis-angle (w, 6)) allows to constrain the rotation angle of one quaternion easily: qu = cos (0/2), where 9 is the rotation angle. Given that the rotation angle between any two neirest-neighboring elements from icosahedron rotational group G is π 7/5, we propose to set ((Aq) < x/5 cover the whole S0(3) or 53 in the quaternion space while avoding being too overparameterized. This sets a constraint to Zoe of Âg be qu 2 cos(=/ (00) Also, for a unit quaternion, qur 1. We can implement these two constraints via a customized activation function: cos(t/10) + (1 - cos(TT/ (10) Sigmoid().

---

Self-supervised Loss. Given the canonical reconstructed point cloud Z and a set of 6D pose hypotheses {P, = (q f, we can obtain pose Z using each P, rendering posed reconstructions Y = 9; 2(q,) + t, We then can enforce a minimum-of-N N loss between the input observation X and the posed point clouds:
Lree
min d(X,Y,), /E _E(I.-..Ga)) G2l
where d: Nxx3 Ny x3 - represents point cloud distance function. For complete object
observation, we can choose d to be bidirectional chamfer distance [8] dcp (X, Y) = Nx XEXXEE
1 Σ min |x
y|| + Ny yE
1 E minl|x y/12. For partial point cloud observation, due to the incompleteness of X, we
can only enforce unidirectional chamfer distance (omiddrcD(XXX))
* Σ min laanllx-ylll which xe XEXEY
only enforces each point in the partial point cloud X can find closed point in Y. Note that the angle constraint of &q implemented by constraining the range of its qu only makes sense when Aq is a unit quaternion. We therefore additionally enforce regularization loss Treg = (l|Ag) 1)a. The total loss is L = free + Xreg. where is a regularization weight.
3.3 Category-Level 6D Object Pose Inference and Evaluation
Lategory-Level 6D Object Pose Inference. Given an object point cloud X (partial or complete), we feed it to our network and generate a reconstruction Z, a set of pose hypotheses {P;), and the corresponding posed reconstruction {Y,}- We can then obtain its category-level 6D object pose estimation p (q. t) via selecting the best pose hypothesis f (q,-, that satisfies j = argmin d(x, Yy). This best pose hypothesis is the 6D object pose of the observed object jE(I.....G..)
instance in the reference frame of reconstruction Z.
Evaluating Lategory-Level 6D Object Pose. To quantitatively evaluate the performance of our self--uper category-Level 6D object pose estimation algorithm on test point cloud data {Xx} from a given category, need to have the ground truth category-level pose annotations Pt (q% t for all test data Xx. which uniquely defines a category-level pose reference frame R.,
Because our estimated poses Ps are in the reference frames R3 of reconstruction Zg. there can be center shift and orientation differences between Rs and R or even among different R&s. We therefore need to find consistent reference frame it and register it to Rº. We propose the following registration algorithms.
For each test data Xs. we transform it using the inverse transformation of Ph yielding a canonicalized object point cloud zi (qi))~xx00 X c t. We then feed Z to our network for pose estimation and obtain a pose misalignment (āc. tz). We propose to use a RANSAC-based method to find a majority consensus of i and t. Then, for all test data, our final pose prediction can be computed as (qq -1 i è qq -iqq "1) using our network's 's raw pose prediction (q.
4
Experiments
4.1 Datasets and Evaluation
Datasets. We evaluate our method using the following datasets. 1) Complete shape point clouds from ModelNet40 [24]. We first test our method on ModelNet40 with five categories, including airplane, car, chair, sofa and bottle. Random rotation applied each input data during training, and 5 different rotations are randomly sampled for each test instance. We follow the original train/test split from the ModelNet40 dataset. 2) Depth point clouds of objects from ModelNet40. We evaluate our method on partial depth observations by rendering the depth images of the complete object instances described above. For depth rendering, we first put the camera at a fixed distance facing the object, and the object is allowed to have small offsets and random rotations. For each category, we generate 30K training images with 6K testing images. The dataset is made public our project page. 3) Real-world depth point clouds from the NOCS dataset [23]. We further evaluate our algorithm on the partial object depth point clouds from the NOCS-REAL273 dataset. This dataset contains a synthetic training set with 275K discrete frames generated with 1085 object models in the classes
7

---

Table 1: Rotation estimation on complete point cloud. We report mean and median rotation error, and accuracy for 3D pose estimation on rotated ModelNet40 Dataset. The best performance is in bold and the second best is underscored.
Meant") / MMd.(") 175' MAPT
Airplane
Car
Chair
Sofa
Bottle
EPNospprrvieed
3.35/ 1.1//1.//000 0.96
9.48 / 1.85 / 00.95
8.56 / 3.87 / 0.68 20. 12/01
4.76 1,56 0.97 128 62, 134.40
49.72/43.19 19//4//0.90 /0.03
KPC (nvpppprrise)
10.12
38.16/ 19:26 00.06
1.33/
ICP tcmpllate shage
8.11 1,22 /090
22.76/2.94/ 0.69
NK. 966 /D:
39.001 10.28
76.11 0.04
Ours
23.09/ 1,66 7087
17.24/ /.89
7.015 1 D.56
8.82 3.22 0.68
4.42/ 0.83/ 0.98
Table 2: 6D pose evaluation on ModelNet40 partial depth point clouds. The evaluation metrics include mean and median rotational error, mean and median translational error, and 5 accuracy together with 5.0.55 accuracy. Oracle means the supervised. The best performance is in bold and the second best is underscored.
Mean(") / 1/5.(")
Airphane
Car
Chair
Sofa.
Botile
EPN(omele]
1987447061
92.3/91.//0 10
3.2/ 0.75
/300 /0.79
12/0/54
Rotation
KPConv(oracle)
11117747024
107.8/ 113.4/0.00
14.6/70777
35.8/ /0.04
6.2/2.00.89
ICP: + template shape
14.7/154/00
87.4/ 15.0 0.36
68.19 19/9 13:47 10.29
115.0/ 165.470.11
26.11 1.50 0881
Ous
3,31/ LL4//16//0 /
124.9/ 127. 1..12
10.4/ 3.84/0.22
72.2/19/ /D.00
46.8/23/00
Mcan Med. 5/55 0.05
EPNoralee
0.00 70.10
0.27 /0,27
0.30/
0.09/ /0.08/0/////0
0.14/0.11 /0.03
Transation
KPCoev(oracle)
0.09/ /0////00/0000000 70002
0.18/0.122000
/09//007 /0.02
0J07 0.06 /
0.302 /0.23
ICP + tomplate shape
0.09/ / /DII 70.05
0.64 / 70.04/ 0.34
D.nK 11.2 00.19
0.07 (1/2/////0 /.04
0.06 0.04 71.60
Qurs
0.04 / /0.76
0.077 00.06/000000
0.05 / /0.44
0.07/ (.07/0000 /0.10
0.13/0.1//0.19
from ShapeNet [2] along with real training videos capturing 3 different instances of objects for each category. Its testing set, FOCS-REAL271 has 3,200 frames collected from 6 real videos containing novel instances. We assume that the ground truth object mask is available our experiments and use them to crop the objects from the depth.
Metrics. We perform all pose evaluation on held-out instances per category. For each category, we report rotational error Rerr, and translational error Terr in the form of mean, and median values, together with 5 and 5cm accuracy. For symmetric objects such as bottles, we evaluate the angular error of the up-axis direction.
Baselines. We compare against two supervised methods: EPN and KPConv, which serve as perfor- mance oracles. We utilize a tuned implementation of the ICP algorithm as a traditional unsupervised method, where we use the same set of 60 rotations from icosahedron group for rotation initializa- tions. Because we assume that we not have complete ground- shapes for testing, for ICP we randomly select an example shape, and iteratively register the observation to the example shape. Note there are no existing learning-basee based unsupervised methods that fit our setting, especially on partial shapes with both 3D rotation and 3D translation.
4.2 Experiments on Synthetic Object Point Clouds
3D pose estimation from complete object point clouds. We first consider rotation of complete shapes. As shown in Table 1, although our method is fully self-supervised, it achieves comparable results to supervised method using otate-of-the-art point cloud learning models such as EPN and KPConv. For certain categories, our method even surpassed the supervised oracles such as the mean rotational error on chair, and median rotational error on bottle. EPN gives the best supervised performance on all the categories except for bottle, mainly due to the difficulty of rotational mode classification under rotational symmetry ambiguity. Without the equivariance property on full SO(3) space, KPConv struggles to give an accurate rotation estimation, and our self-supervised pipeline out- out-performs on all 4 asymmetric categories, while performing well on symmetric bottle. For simple shapes like airplane and car, ICP algorithm could achieve reasonable estimation after iterative alignment, but fails at chair and sofa category. It also does not work well for the bottle case as the given example shape does not match well with all different bottle instances.
6D pose estimation from partial object depth point clouds. When the input observation is partial, there will be more ambiguities in determining object canonical pose. Despite this challenge, we observe that our method is still able to infer well-aligned canonical shapes across different instances, as shown in Table 2 and Figure 2. For some object categories, our method is able to infer complete
8

---

airplane
car
chair
sofa
bottle
Figure 2: Visualization on the ModelNet partial shapes. Here we show four test instances for each categories: left: input X, middle: canonical reconstruction Z, and right: posed reconstruction Y. Note that Zs have been calibrated.
Table 3: Pose estimation with equivariant and non-equivariant bakebones.
Dataset
Shape
Pose
Mean Rey
Median Revr(")
Mean Ter
Median Ter
5
5" 0.05
EPN
EPN
14.24
1.53
0,91
Complete
EPN
KPConv
21.24
1.70
0.88
airplane
KPConv
EPN
133.89
169.96
0.00
KPConv
KPConv
134.27
172.21
0.02
EPN
EPN
3.47
1.58
0.02
0.02
0.95
0,88
Partial
EPN
KPConv
4.24
1.96
0.03
0.02
0.93
0.85
airplane
КРСопм
EPN
134.34
174.98
0.07
0.08
0.05
0.05
KPConV
KPConV
132.88
172.31
0.077
0.08
0.04
object shape by training different view points. However, for bottle, the algorithm only considers the object shape as a hhalf" bottle due to the symmetry partial bottle shape can already cover all possible view points. Quantitatively, results in Table 2 show that method achieves the best accuracy on both partial airplane and chair data. It also handles the symmetric object like bottle well, while the two supervised methods suffer from performance degradation due to the increased difficulty.
Effects of equivariance. As shown in Table compare the method's performance when use equivariant or non-equivariant backbones. The results demonstrate the importance of using cquivariant neural networks for canonical shape reconstruction. Leveraging SEqiveequvarriane makes it possible to generate consistently aligned canonical shapes, which in turn helps get accurate pose estimation, using non-equivariant backbone for shape would fail totally. We also show that using equivariant backbone for direct pose regression would further improve the pose estimation accuracy. especially on partial shapes.
Different equivariant backbones. As shown in Table 4, compare the performance of different cquivariant backbones, with both SE(3)-cquivarian networks like EPN and SE(3) transformer. SE(3) transformer also promises rotational and translational equivariance by leveraging properties of spherical harmonics. From our experiments, could also get consistently aligned canonical reconstructions. However, compared to EPN, SE(3) transformer way more computationally heavier, and SE(3) transformer struggles to generate accurate rotation estimation compared to EPN, especially on partial shapes. EPN-20 is another variant of EPN network, when the full SO(3) space is divided into 20 different sub-regions, which has smaller number of sub-spaces than our finalized EPN-60 backbone. From our experiment doesn't 't get better performance than EPN-60. Based on these experimental findings, we finally choose EPN-60 as our SE(3) backbone.
Failure cases. When heavy self-occlusions and shape symmetry present, the performance of our model tends to degrade. We observe both sofa and car categories have lot of 180-degree-flip

---

Table 4: Pose estimation with different equivariant backbones.
Dutaset
Backbone
Mean Rer
Median Rgr(()
Mean Terr
Median Tere
5
0.05
SE(3) Transformer
6.65
3.64
0.7
Complete
EPN-20
64.12
2.40
0.63
airplane
EPN-60
23.09
1.66
0.87
SE(3) Transformer
19.82
5.29
0.07
0.04
0.46
0.13
Partial
EPN-20
85.53
5.15
0.03
0.03
0.50
0.45
airplane
FPN-60
3.47
1.58
0.02
0.02
0,95
0.88
Table 5: 6D pose evaluation on NOCS-REAL 275 dataset. Full evaluation of category-level 6D
pose estimation our ModelNet40 depth dataset using different methods. The evaluation metrics include mean and median rotational error, mean and median translational error, and 5 accuracy together with 5 5cm accuracy, The best performance is in bold and the second best is underscored.
Meant 115"")
Boitle
Bowl
Camera
Can
Laplup
Mag
1)
14.8/3.001.2 79
5.1/3.500.73
67.8/9 5/0.06
2.9/2.400.8 86
718/4.190.55
7,7/4,70,55
Rotation
21
2.1/2,00:.66
5,5/5 14048
38.011538.00
19/2.500 MUS
$4.8%.10..3 190.9
12.1/5
10.54.5/0.57
9.027550022
152.3116..8000
4.6/3990.55
98,7113.6/0001
123.3/126.80
7.373.500.7
6.315
87.1/79 20000
3.2/2/700.87
86.475.300.
1/48/134./00/
11.09.90020
119.0/15..00000
2.7/2.3/0.90
18..1/188./0000
1/1.1/12..50000
Meanti) / MMddim) ./5 5emmem 1
1)
0.04N.040.73
6.04/8.501.55
Trandition
2)
0.0770.0700.6
0.05M.0480.08
0.0./7
0.130.1330.87
0.02A)
6.04/6.330 .9
3)
0L05AL.5500.77
6.048.040022
0.07000000000
0.14/0.1300
06660.660001
021/0.22/0.00
4)
0.05/U0551..3
0.08/0.0610.38
0,033007700000
0,16/01160877
0.02/0.020.45
0.120012/0.00
0.13001220.04
0.06000560.20
0.1180.1XX.00
0.0991.599.90
0.06/0.0601004
0L05/8.0780000
which means the model couldn't 't distinguish dominant direction like front and back. The error of rotation estimation also contributes to the translational error for car and sofa. For the bottles, we also observe multiple cases where the reconstructed shape is upside down and an considerable offset to the dominant object center which results in the poor performance for translation estimation. See our appendix for the error percentile curves.
4.3 Experiments on Real-World Object Point Clouds
To test method under a more realistic setting and explore the limits of its performance, we examine our method on the challenging real world depth data from (OCS-REAAL225(33) dataset, where sensor noises and heavy occlusions in the real world cluttered scene further complicates the problem, e.g. the delicate handle structure the mug category is often invisible, making it difficult to estimate its pose and shape. In addition, different from our randomly rotated synthetic data, this real dataset contains a biased pose distribution, e.g. bottles are always standing upright and the bottom part is never observed, adding difficulty the reconstruction task. We consider five different experiment settings: 1) perform supervised pretraining on CAMERA[23) dataset, similar to EPN: 2) seff-supervisedly finetune the pretrained weight of 1) using the small train split of NOCS; 3) self-superviseely pretrain on CAMERA; and finally 4) self-supervisedly finetune the pretrained weight of 3) on NOCS-train, together with ICP algorithm. We report a full evaluation of category-level 6D pose estimation on our NOCS- REAL275 dataset using different methods in Table 5. Our fully unsupervised method beats ICP on almost all categories and performs quite well on bottle, bowl, and can, without using any labelled or real data. Self-supervised finetuning on real data further improves its performance.
5 Conclusion
This paper has presented a novel self-supervised network that is capable of category-level 6D object pose estimation. The system operates without CAD models or annotations, and achieves impressive accuracy on both synthetic and weal-world category-level pose estimation benchmarks. This work is the first to demonstrate the power SE(3)-equivariant point cloud networks for 6D pose estimation, especially on partial object point clouds. As limitations, the performance of SE(3) equivariant networks may degrade when objects exhibit symmetries, or viewpoint distribution highly biased, or occlusions are always heavy. Extending this work to a semi-supervised setting may help to relieve these limitations, which we leave for future works. This work can reduce the demand of 6D pose annotation, therefore may impose negative impacts to people who work on object pose annotations.
10

---

This research is supported by a Vannevar Bush faculty fellowship, NSF grant 118-176,68,, and gifts from the Adobe and Autodesk Corporations. We also appreciate resources provided by Advanced Research Computing in the Division of Information Technology at Virginia Tech.
Acknowledgement:
References
[1] Melinos Averkiou, Vladimir Kim, and Niloy Mitra. Autocorrelation descriptor for efficient co-alignment of 3d shape collections. In Computer Graphics Forum, pages 261-271, Wiley Online Library, 2016.
[2] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich
3D model repository. arXiv preprint arXiv:1512.03012. 2015.
[3] Mohamed Chaouch and Anne Verroust-Blondet. Alignment of 3D models. Graphical Models, 71(2):63-76, 2009. 3
[4] Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6D object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11973-119999 11922, 2020.
[5] Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14514-14523, 2021. 3,5
[6] Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural maalyss-bb-syntheen In European Conference Computer Vision, pages 139-156. Springer, 2020. 3
[7] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for -quuiaequi networks. arXiv preprint 1:Xiv:2104112229 2021. 3
[8] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017. 7
[9] Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling, Se (3) transforrrr:::: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020. 3,
[10] Eldar Insafutdinov and Alexey Dosovitskiy. Unsupervised learning of shape and pose with differentiable point clouds. arXiv preprint arXiv:1810.098: 09381, 2018. 3
[11] Xiaolong Li, He Wang, Li Yi, Leonidas J. Guibas, A. Lynn Abbott, and Shuran Song. Category- level articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3706 3706-3715, 2020. 3
[12] Eloi Mehr, André Lieutier, Fernando Sanchez Bermudez, Vincent Guitteny, Nicolas Thome, and Matthieu Cord. Manifold learning in quotient spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9165-9174, 2018. 3,6
[13] Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effective rotation-invariant point with spherical harmonics kernels. In 2019 International Conference on 3D Vision (3DV), pages 47-56. JEEE, 2019. 3
[14] Charles R Qi. Hao Su. Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings the vision and pattern recognition, pages 652-660, 2017. 5
the IEEE conference on computer
[15] Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Domain transfer for 3D pose estimation from color images without manual annotations. Asian Conference on Computer Vision, pages 69 84- Springer, 2018.

---

[16] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, and Rudolph Triebel. Augmented autoencoders: Implicit 3D orientation learning for 6D object detection. Inter- national Journal of Computer Vision, 128(3):114-729, 2020. 3
[17] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6411-6420, 2019.
[18] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and canslationeqquivriian neural networks for 3D point clouds. arXiv preprint arXiv: 19, 2018. 3,
[19] Meng Tian, Marcelo H Ang, and Gim Hee Lee. Shape prior deformation for categorical 6D object pose and size estimation. European Conference on Computer Vision, pages 530-546. Springer, 2020. 3
[20] Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Multi-view consistency supervisory signal for learning shape and pose prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2897-2905, 2018. 3
[21] Chen Wang. Roberto Martin-Martin, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category-level 6d pose tracker with anchor-based keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages (0059-10066) IEEE, 2020.
[22] Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, and Federico Tombari. Self6D: Self-supervised monocular 6D object pose estimation. In European Conference on Computer Vision, pages 108-122 125. Springer, 2020. 2, 3
[23] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2642-2651, 2019. 1, 3, 7,
[24] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015. 2,7
[25] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convo- lutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017. 1
[26] Yang You, Yujing Lou, Qi Liu, Yu-Wing Tai, Lizhuang Ma, Cewu Lu, and Weiming Wang. Pointwise rotation-invarian network with adaptive sampling and 3d spherical voxel convolution. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12717-12724, 2020.
[27] Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Federico Tombari. Quaternion equivariant capsule networks for 3D point clouds. In European Conference Computer Vision, pages 1-19. Springer, 2020. 3
12


Reviews:
[{"review_id": "uu1jPeNeN44", "reviewer": "Reviewer_ojV5", "summary": "This paper introduces a self-supervised category-level 6D pose estimation to minimize ideally canonical reconstructed point cloud and input point cloud based on rotation-equivariant features. Experiment results can verify the effectiveness on both complete and partial point-based observations.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is difficult to digest due to the writing due to a large number of typos. For example, on lines 65-68, features for 6D pose estimation can be equivariant rather than the problem itself. \n\nTechnically, the design of canonical reconstruction and using residual pose predictions for progressive alignment to canonical space is not novel in the field. In the context of category-level 6D pose estimation, the proposed method can be considered to combine two branches of canonical reconstruction and pose estimation in a self-supervised manner. However, due to lack of canonical reference, ideally canonical reconstruction can be affected by intra-class shape variations and inaccurate pose predictions. The authors are suggested to explain its effects on pose estimation and do some ablation studies, which are missing in the paper.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The following items need to be addressed:\n1.  clarify main contribution of the paper;\n2. ablation studies on effects of quality of reconstructed shape on pose estimation\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "fYbkd2Vlk79", "reviewer": "Reviewer_VRcx", "summary": "The paper proposes a framework to learn 6D object pose estimation without GT pose annotations and CAD models (while GT object masks are available, and each object is already isolated). The key is to jointly learn canonical shapes and poses by a SE(3)-equivariance network. It shows promising performance on both synthetic and real depth data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strongness:\n- Clear writing\n- Experiments on multiple datasets, including both synthetic and real ones\n- Detailed analysis of failure cases and limitations\n\n---\n\nWeakness:\n1. The authors miss some important related works. Although relying on multi-view supervision, there is a line of previous works focusing on learning-shape-from-x without GT labels, e.g. [1], [2], [3]. Especially, [3] has proposed a quite similar framework, like disentangling canonical shapes and poses, self-supervised losses, as well as selecting the best hypothesis during inference. The major differences are the type of supervision (multi-view) and the network design (PointNet + regression). It also presents many observations mentioned in this paper. For example, symmetric objects like cars are difficult for such self-supervised frameworks. \n2. Although there are no existing approaches to compare directly, the authors could provide more ablation studies to verify the design choices, which is kind of missing now. Although Sec 2.4 in Supp has presented some results, can the authors analyze the impact of equivariant networks on the shape and pose separately? For example, KPConv (or EPN) for shapes and EPN (or KPConv) for poses. Compared to [3], one key improvement of this paper is to introduce SE(3)-equivariant networks to generate consistent canonical shapes. Another question is how important it is to predict multiple hypotheses for 6D poses. Can the authors study the impact of the number of hypotheses (predefined groups)? And can the authors directly regress poses rather than using any hypothesis?\n3. The second car in Figure 2 looks strange. The blue point cloud (canonical shape) in the 3rd column looks different from that in the 2nd column.\n4. How stable is the training process? It seems that the network can easily get stuck in local minima (imperfect shapes will lead to wrong poses, and imperfect poses can also lead to imperfect shapes. This phenomenon might be most obvious for cars). Can the authors report the result of multiple trials of experiments to show the variance of the method?\n\n---\n\nTypos:\n- L152: there are two \"j-th\", and I think one should be \"j'-th\".\n- L203-L204:  $g_x=\\Delta g_{x_0} \\cdot g_x$\n\n---\nReferences\n- [1] Unsupervised Learning of Shape and Pose with Differentiable Point Clouds\n- [2] Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction\n- [3] Weakly-supervised 3d shape completion in the wild", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4ZaSlKnTuEp", "reviewer": "Reviewer_8FBM", "summary": "This paper proposes a self-supervised approach to 6 DoF pose estimation. Usually such methods are either instance-level, assuming known CAD models for all object instances; or category-level, assuming a reference frame that can be generalized to all instances of a certain category. The second approach is more generalizable, however it requires a substantial amount of annotated training data, often from synthetic datasets. So far, self-supervision in this setting has been limited to instance-level, with CAD models but without pose annotations. This paper proposed category-level self-supervision and claims to be the first to do so, eliminating the need of both CAD models and large-scale annotated data. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI find this paper quite novel, and as far as the authors claim (I am not an expert in the field) this is the first method to achieve self-supervision on category-level 3D object pose estimation. The reasoning behind how the authors achieve this is also well-explained and motivated throughout the paper (although I found it difficult to read in some points, see below). This also makes the paper significant, as it opens up a lot of potential follow-up work in self-supervised learning for this another task. \n\nOverall, the paper should be proof-read and improved in terms of grammar, presentation and clarity. Particularly the Method section, probably due to lack of space a lot is introduced in long sentences and paragraphs, it is hard to parse all the information. Having sub-section and proper equations with numbers and details would have been helpful. Same thing with experiments, there are a lot of long sentences and paragraphs with combined information that makes it hard to parse. \n\nOther comments: \n\n---\n\nThe contributions of this paper are three-fold:\n\n- A general self-supervised framework for category-level 6D object pose estimation without CAD models or annotations.\n- Showing that the concept of category-level reference frame can be recovered naturally during self-supervised training\n- High accuracy in synthetic and real-world category-level datasets \n\nI think contributions 1 and 2 can be combined into one, as the authors are proposing 1 by means of 2. About 3, \"achieving high accuracy\" is vague, it would be better to report relative improvements and other real comparisons with other methods. Is this method outperforming other self-supervised methods, or competitive with supervised methods?\n\n---\n\nAbout experiments, I would like to have seen ablations with different amounts of training data. This is the biggest benefit of using self-supervised learning, so improvement with data is an important aspect to show as a way to potentially boost results even further. \n\n-- \n\nOn table 1, why does the proposed method outperforms EPN only for chair, and why does KPConv outperforms EPN only for bottle, but by a very large margin?\n\n--- Minor comments\n\nline 360: Unfinished sentence.\nline 117: \"only one network\" or \"the only network\"?  I didn't understand this sentence.  EPN achieves equivariance without increasing in complexity?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention certain limitations of their approach, such as degradation when objects exhibit symmetries, or viewpoint distribution is highly biased, or under heavy occlusions. There are no quantitative ablations or examples of such cases from the datasets considered in experiments, that would have been helpful to further illustrate these points. As for societal impact, the authors mention negative impact to people who work on object pose annotations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "I believe the ethical concerns of this paper is minimal and pertains to the ones inherent to self-supervised learning. For example, removing unwanted dataset biases, correct anonymization when learning from unlabeled data, ensuring the correct diversity so it includes different aspects of the data, and so forth.  "}}]

Author rebuttal:

Meta-review: 
This paper proposes a self-supervised category-level 6D pose estimation method in which canonical shapes for the input objects are predicted without explicit supervision of orientation. It introduces an  equivariant network architecture and shows its importance for canonical shape estimation.  Reviewers point out missing citations of previous works that consider similar setup (yet different neural architectures), specifically:
[1] Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
[2] Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction
[3] Weakly-supervised 3d shape completion in the wild
and the authors are encouraged to reference these and discuss in detail the difference of their method against the above. Reviewers agree regarding the good empirical performance of the method and the detailed ablations contained in the paper, and the paper is suggested for publication.

Decision: Accept (Poster)