Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild
Authors: Kaifeng Zhang
Yang Fu
Shubhankar Borse
Hong Cai
Fatih Porikli
Xiaolong Wang
Venue: ICLR 2023 poster (2023)
Keywords: Category-level 6D pose estimation
self-supervised learning
correspondence
computer vision

Abstract: While 6D object pose estimation has wide applications across computer vision and robotics, it remains far from being solved due to the lack of annotations. The problem becomes even more challenging when moving to category-level 6D pose, which requires generalization to unseen instances. Current approaches are restricted by leveraging annotations from simulation or collected from humans. In this paper, we overcome this barrier by introducing a self-supervised learning approach trained directly on large-scale real-world object videos for category-level 6D pose estimation in the wild. Our framework reconstructs the canonical 3D shape of an object category and learns dense correspondences between input images and the canonical shape via surface embedding. For training, we propose novel geometrical cycle-consistency losses which construct cycles across 2D-3D spaces, across different instances and different time steps. The learned correspondence can be applied for 6D pose estimation and other downstream tasks such as keypoint transfer. Surprisingly, our method, without any human annotations or simulators, can achieve on-par or even better performance than previous supervised or semi-supervised methods on in-the-wild images. Code and videos are available at https://kywind.github.io/self-pose.

Published as a conference paper at ICLR 2023
SELF-SUPERVISED GEOMETRIC CORRESPONDENCE FOR SIREGORY-EEVER 6D OBJECT POSE ESTIMATION IN THE WILD
Kaifeng Zhang' Yang Fu², Shubhankar Borse", Hong Cai*, Fatih Porikli", Xiaolong Wang" TTinghuu University, PCC San Diego, $ *Qualcomm AI Research
ABSTRACT
While 6D object pose estimation has wide applications across computer vision and robotics, it remains far from being solved due to the lack of annotations. The problem becomes even more challenging when moving to category-level 6D pose, which requires generalization unseen instances. Current approaches are restricted by leveraging annotations from simulation or collected from humans. In this paper, we overcome this barrier by introducing self-supervised learning approach trained directly on large-scale real-world object videos for category-level 6D pose estimation in the wild. Our framework reconstructs the canonical 3D shape of an object category and learns dense correspondences between input images and the canonical shape via surface embedding. For training, we propose novel geometrical coeee-conssteecyy losses which construct cycles across 2D-3D spaces, across different instances and different time steps. The learned correspondence can be applied for 61) pose estimation and other downstream tasks such as keypoint transfer. Surprisingly, our method, without any human annotations or simulators, can achieve on-par or even better performance than previous supervised or semi- supervised methods on in-the-wild images. Code and videos are available at https::/kywiiii g thub. tooselffee e-lopose..
INTRODUCTION
Object 6D pose estimation is a long-standing problem for computer vision and robotics. In instance- level 6D pose estimation, a model is trained to estimate the 6D pose for one single instance given its 3D shape template (He et al., 2020; Xiang et al., 2017: Oberweger et al., 2018). For generalizing to unseen objects and removing the requirement of 3D CAD templates, approaches for category- 6D pose estimation are proposed (Wang al., 2019b). However, learning generalizable model requires a large amount of data and supervision. A common solution in most approaches (Wang et al., 2019h, Tian al., 2020; Chen et al., 2020a; 2021; Lin et al., 2021) is leveraging both real-world (Wang et al., 2019b) and simulation labels (Wang et al., 2019b; Chang et al., 2015) at the same time for training. While there are limited labels from the real world given the high cost of 3D annotations, we can generate many annotations as we want in simulation for free. However, it is very hard to model the large diversity of in-the-wild objects with a simulator, which introduces a large sim-to-real gap when transferring the model trained with synthetic data.
Although the real-world labels are hard to obtain, the large-scale object data is much more achiev- able (Fu & Wang, 2022). In this paper, we propose self-supervised learning approach that directly trains on large-scale unlabeled object-centric videos for category-level 6D pose estimation. Our method does not require any 6D pose annotations from simulation or human labor for learning. This allows the trained model to generalize to in-the-wild data. Given a 3D object shape prior for each category, our model learns the 2D-3D dense correspondences between the input image pixels and the 3D points on the categorical shape prior, namely geometric correspondence. The object 6D pose can be solved with the correspondence pairs and the depth map using a pose fitting algorithm.
We propose a novel Categorical Surface Embedding (CSE) representation, which is a feature field defined over the surface of the categorical canonical object mesh. Every vertex of the canonical mesh is encoded into a feature embedding to form the CSE. Given an input image, we use an image encoder to extract the pixel features the same embedding space. By computing the similarity
Work done while intern UC San Diego

---

Published as a conference paper at ICLR 2023
220
CR
C1441
Figure 1: Examples of self- self-supervié supervised category- 6D pose estimation in the wild. We propose novel Categorical Surface Embedding representation learn categorical 2D-3D geometric correspondences. For each example, we visualize the ohject 6D pose and correspondence map.
between the 3D vertex embeddings and the pixel embeddings, we can obtain the 2D-3D geometric correspondence. With such correspondence, we lift the 2D image texture to a 3D mesh, and project the textured mesh back the 2D RGB object image, segmentation mask and depth using differentiable rendering. We use reconstruction losses by comparing them to the 2D ground-truth for training. However, the reconstruction tasks do not provide enough constraints on learning the high dimensional correspondence.
To facilitate the optimization in training, we propose novel losses that establish cycle-consistency across 2D and 3D space. Within a single instance, our network can estimate the 2D-3D dense correspondence á using CSE and the global rigid transformation a for projecting 3D shapes to using another encoder. Given 2D pixel. we can first find its corresponding 3D vertex with 6 and then project back to 2D with ग, the projected location should be consistent with the starting pixel location. We can design a similar loss by forming the cycle starting from the 3D vertex. This provides an instance cycle-consistency loss for training. Beyond single instance, we also design cycles that go across different object instances within the same category. Assuming given instance A and B, this cycle will include a forward pass across the 3D space, and a backward pass across the 2D space: (i) Forward pass: Starting from one 2D pixel in instance A, can find its corresponding 3D vertex using c. This 3D vertex can easily find its corresponding 3D point location in instance B given the mesh is defined in canonical space. The located 3D point is then projected back to 2D using π for instance B. (ii) Backward pass: We leverage the self-supervised pre-trained DINO feature (Caron al., 2021) to find the 2D correspondence between two instances A and B. The located 2D pixel in instance B during the forward pass can find its corresponding location in instance A using the 2D correspondence, which provides a cross- cross-instanc instance conessoons loss. The same formulation can be easily extended to videos, where we take A and B the same instance across time, and this additionally provides a cross-time time cycle-consistency loss.
We conduct our experiments the in-the-wild dataset Wild6D (Fu & Wang, 2022). Surprisingly, self-supervised approach that directly trains on the unlabeled data performs on par or even better than fate--ofttheart approaches which leverage both 3D annotations the 6D pose estimation and the geometric correspondence map in Figure 1. Besides Wild6D, we also train and evaluate our model in the REAL275 (Wang et al., 2019b) dataset and shows competitive results with the fully supervised approaches. Finally, we evaluate the CSE representation on keypoint transfer tasks and achieve sfate-of-the- the-tr- art results. We highlight our main contributions as follows:
well simulation. We visualize
To the best of our knowledge, this is the first work that allows self-supervised 6D pose estimation training in the wild.
We propose framework that learns novel Categorical Surface Embedding representation for 6D pose estimation.
We propose novel conesssennssteney losses for training the Categorical Surface Embedding.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Category leeeel 6D Pose Estimation. Compared instance-level 6D pose estimation (Xiang et al., 2017; Li et al., 2018; Wang et al., 2019a), the problem of category-level pose estimation is more under-constrained and challenging due to large intra-class shape variations. Wang et al. (2019b) propose the Normalized Object Coordinate Space (NOCS) which aligns different objects of the same category into unified normalized 3D space. With the NOCS map estimation, the 6D pose is solved using the Umeyama algorithm (Umeyama, 1991). Following this line of research, subsequent works (Tian et al., 2020; Chen et al., 2020a: Fu & Wang. 2022) are proposed to learn more accurate NOCS representation. Aside from using NOCS, 6D pose estimation can also be approached by directly regressing the pose (Chen et al., 2021; Lin et al., 2021), estimating keypoint locations (Lin et al., 2022), and reconstructing the object followed by an alignment process (Irshad et al., 2022a;b). However, all these approaches require training on numan-annotated real datasets (Wang et al., 2019b; Ahmadyan et al., 2021) or synthetic datasets (Wang et al., 2019b; Chang et al., 2015). Our work follows the same tetimate-then-solee pipeline as NOCS, while doing this in a self-supervised manner without using synthetic data or human annotations real data.
Self-supervision isssuervisionn for Pose Estimation. With the high cost of getting 3D annotations, different self-supervised training signals are proposed for better generalization. One effective strategy is to perform the sim-to-real adaptation on the model pre-trained on the synthetic data (Chen et al., 2020b; He et al., 2022; Gao et al., 2020; You et al., 2022). Another line of research focuses achieving better generalization ability by semi-supervised training on both synthetic datasets and unlabeled real data (Fu & Wang. 2022; Manhardt et al., 2020; Peng al., 2022). All of these works require first training with the synthetic data, which can provide good initialization for 6D pose estimation. To the best of our knowledge, our work is the first one that directly trains on in-the-will images without any annotations. This avoids dealing with the sim2real gap problem in training and it has the potential to generalize to a larger scale of data without carefully designing our simulator.
Self-supervised Mesh Reconstruction and Geometric Correspondence. Our work is related to recent methods for mesh reconstruction by applying differentiable renderers (Kato et al., 2018; Liu et al., 2019) together with the predicted object shape, texture and pose (Kanazawa et al., 2018; Goel et al., 2020; Tulsiani et al., 2020; Li et al., 2022: Wu et al., 2021; Hu et al., 2021; Yang et al., 2021a; Ye et al., 2021). However, the object pose estimation results from these works are still far from satisfactory, especially for daily objects with complex structures and diverse appearances. Compared these works, our method is more close to recent works on learning the 2D-3D geometric correspondence from RGB images in a self-supervised manner (Güler et al., 2018; Neverova et al., 2020; Kulkarni et al., 2019; 2020; Yang et al., 2021b). For instance, Yang al. (2021b) establishes the dense correspondences between image pixels and 3D object geometry by predicting surface embedding for each vertex in the object mesh. However, the learned surface embeddings are object- specific and hard to generalize to different objects. To supervise correspondence learning, most existing works design the geometric consistency loss (e.g., Kulkarni al. (2019)) within a single object. In this paper, we propose novel cross-instance and cross-time cycle-consistency losses that build 4-step cycle and achieve better geometric correspondence.
Learning from Cycle-consistency. To construct the 4-step cycle, we will need to establish corre- spondence across images, which is related to visual correspondence learning in semantics (Rocco et al., 2017; Truong et al., 2021; Jeon et al., 2018; Caron et al., 2021; Huang et al., 2019) and for tracking (Vondrick et al., 2018; Wang et al., 2019c; Li et al., 2019; Xu & Wang, 2021; Jabri et al., 2020). For example, Wang et al. (2019c) proposes perform cycle-consistency learning by tracking objects forward and backward in time. Different from these works, our method constructs cycles across 2D and 3D space. In this aspect, our work is more close (Zhou et al., 2016), where 3I) CAD models are used as intermediate representations to help improve 2D image matching. However, they are not learning the 3D model but using existing CAD model templates for learning better 2D keypoint transfer. Our work learns the 2D-3D geometric correspondence and the 3D structure jointly,
METHOD
To estimate the 6D pose, we propose to learn the Categorical Surface Embedding (CSE) for finding geometric correspondence (2D-3D correspondence) in a self- manner. We perform learning under a reconstruction framework with differentiable rendering as shown in Fig. 2. Our model takes single RGB image input, with ground truth object mask and depth as supervision. Given the
3

---

Published as a conference paper at ICLR 2023
Vertex (vi)
transfer texture
Correspordencee Mairix
Rendered imago.
Canonical Mesh
2D-30 comespondence
(Moan Shape)
AV F
Renderer
recon
En
Image
Rondered mask
Feature
L cycle
(u,)
30-20 correspondence
Input Image
[R, t]
transform mesh
depth (Oplional)
operation
Figure 2: Framework Overview. Given the input image, the image encoder and feature extractor (blue) predict the canonical mesh deformations AV, transformations [ and per-pixel features. mesh encoder (green) predicts mesh per-vertex features. We compute the feature similarity matrix and bidirectional correspondences (orange). which are used to transfer image pixel colors to mesh texture. Finally, we use differentiable rendering render the RGB image, segmentation mask and (optionally) depth map. The framework is trained with reconstruction loss Crecon, cynssscon loss Legele regularization loss Lreg-
input RGB image, we compute the image feature in the space of CSE. At the same time, we adopt the PointNet (Qi et al., 2017) to encode the deformed categorical mesh shape (Section 3.1) and extract the vertex features. We compute the geometric correspondence between these two CSE feature maps and the correspondence to transfer the texture from 2D image 3D shape. We project the textured 3D shape with a regressed pose to the 2D image, mask, and depth via differentiable rendering. We compute the reconstruction losses against the 2D ground-truths (Section 3.2). However, reconstruction alone does not provide enough constraint for learning the nigh-dimensional geometric correspondence. We propose multi-level cycle-consistency losses (Section 3.3) to enforce the consistency of geometric correspondence across instances and time. We summarize our training and inference in Section 3.4.
3.1 CANONICAL SHAPE PRIOR DEFORMATION
The first step of our model is learn categorical canonical shape prior. We choose to use triangular mesh S {V. F), where V E 3x are canonical vertices and F 3 M are canonical faces that encode the mesh topology. We use AV to model the deformation of each instance in the same category, and the shape of an instance can be represented as V V AV, where V is a categorical shape prior and AV is the instance-specific deformation. The shape prior not only describes the approximate shape of the category, but also defines the canonical pose, such that the 6D pose estimation problem can turn into predicting the relative pose between the observed object and the canonical meshes. We initialize V with selected shape, and it remains learnable during training. Given an input image, we predict an implicit shape code Ushape IR with our image encoder. Then, for each vertex the canonical mean mesh, we concatenate the vertex positions [xi Bis 24 with the shape code Ushpee and use a MLP I' Sahaaee R Ra+3 - Ra to predict the per-vertex offset amount along 3 dimensions x, y,, 2 obtain A V.
3.2 CATEGORICAL SURFACE EMBEDDING AND CORRESPONDENCE
Once we have the deformed mesh based the input image, we can extract the vertex features using a PointNet (Qi et al., 2017) encoder (green box Fig. 2) as (Vi,'", E Nxd where N is the number of vertices and d is embedding dimension. We name this per-vertex embedding as the Categorical Surface Embedding, for introduces a shared embedding space across all mesh instances in the same category. On the image side, we use feature extractor network to obtain the pixel-wise features [1, "., M IR hxwxd where h, w are the height and width.
After extracting image features and vertex features, measure the cosine distance between per- pixel features and vertvertex features, from where we obtain the image-mesi and mesh- -maage correspondence matrices via Softmax normalization over all mesh vertices and all pixels:
cxp (cos (u;, vz)/T) Σ, exp (cos( Vz)/7)*
cxp (cos(uc, Vz)/T) Σ, exp (cos( vy)/T)
W 2D-3D
W 3D-D-
(1)
4

---

Published as a conference paper at ICLR 2023
W and W entries in the correspondence matrices that represent the 2D-3D and 3D-2D correspondence values between mesh vertex and pixel j. T is the temperature parameter. With the matrices, construct the 2D-3D correspondence mapping and 3D-2D correspondence mapping:
4
Σ 32 b
1/ 2.),
P Σ
W W.I-2D [[3,
4 and p, represent the corresponding 3D location of pixel j and the corresponding 2D location of vertex i, respectively. We denote the 2D>33D mapping as : RR2 R&3 such that (([,, Y2]) = q and the 3D-2D mapping as t IR IK² such that ([ ys 2,]) = Pt- We emphasize that our computation of correspondence is based on the similarities between per-vertex surface embedding and per-pixel image feature, which different from previous work (Kulkarni et al., 2019) on direct repression of the 2D-3D mapping function neural network outputs.
Texture transfer. The geometric correspondence can serve as texture flow that transfers image pixel colors the mesh. Specifically, correspondence, which leads to textured mesh: tex; Σp=1 W3DD-2D, -1.. where tex; is the texture color at mesh vertex i, and 1 is the RGB value at image pixel
propose to perform texture transfer using the predicted 3D-2D
Reconstruction Loss. We perform reconstruction in 2D space for supervision. We first estimate mesh transformation [R]t] (R E 50(3), E IR$) using the image encoder. We use the continuous 6D representation for 3D rotation (Zhou et al., 2019), and 3D translation. After transforming the mesh with the estimated rotation and translation, we adopt SoftRas (Liu et al., 2019) (deep orange box in Fig. as the differentiable renderer generate the RGB image, segmentation mask and depth map. The reconstruction loss is defined as follows:
Lrecon
Brexture || S 1- S11 Smack 1|S S1|2 + Bdeph ID.S-
where T,S. D are the imput image, ground truth segmentation mask, and the depth map, respectively, and i, S. Đ are the corresponding rendering results. With the reconstruction loss, our model can learn shape deformation and mesh transformation with back-propagation. The texture loss further enforces the color consistency between the projected vertices and the pixels. However, color consistency not sufficient to learn fine-grained nigh-dimensional correspondence. To this end, in the next section, we introduce the novel cole-coonssteecy loss which learns a coordinate-leve correspondence based on the reconstructed shape and pose.
3.3 LEARNING WITH CYCLE- CON.R--ONNSSTENCY
We propose multiple sycle-consistency losses across 2D and 3D space for learning, including constructing a cycle within single instance, and cycles across different instances and time.
Instance cycle-consistency. In Eq.2 we introduced the correspondence formulation, the correspond- ing 2D-3D mapping Φ R2 - R and 3D-2D mapping vi R R2. We also introduced the mesh transformation [R]t] regressed by the image encoder. For instance cycle-consistency, we encourage the consistency between the predicted mapping and the camera projection. In an ideal situation, should be consistent with the camera projection (which we denote matrix), and de should be consistent with the inverse projection -- Naturally, this leads to 2 cycle losses, one combining á with the other combining vi with -- (See Fig.3 left for an illustration):
Ct C' is the camera
1
1 V
CID-5D
Σ (m((()) [ pef
2ll
LID-2D
Σ qeV
((b(())))) -912.
(4)
We incorporate a visibility constraint into this loss by only computing cycles that involve non-occluded vertices under the estimated projection. (See Appendix A.2 for more details.)
Cross-instance and cross-time cycle-consistency. appearance across instances in the same category, we design a loss that encourages the consistency of the geometric correspondence across instances. We propose a cross-instance cycle-consistency loss as illustrated in Fig.3. The loss is built by 4-step cycle, consisting of 2 images (i.e., source and target images) and their respective reconstructed meshes. It contains a forward pass through the 3D space, and backward pass through the 2D space. In the forward pass, given pixel p, in the source image, we first map the point to the 3D mesh via the predicted correspondence W2D-3D creating correspondence distribution over the source mesh surface. In this distribution, the value
handle the large variance of object shape and
5

---

Published as a conference paper at ICLR 2023
(a) instance codee-onsistenccy
(b) cross-instance eycle-consistency
(c) cross-time cycle-consistemn
W
W
Linst
Liine
3D-2D correspondence
mage
mage
2D
CaD-3D
W
WD10-0
2D-3D correspondence
Cross-instance Image/
Cross-time Image
Figure 3: An illustration of the cycle-consistency losses. Left: instance cycle-consisteney. Loss is defined by re-pre re-projection offsets between the image and the reconstructed mesh, using 3D-2D and 2D-3D mappings (brown) and estimated projection (bluc). Middle and right: the cross-instance cycle-consistency and cross-time cyele-consistency, established as 4-step cycle, using our learned correspondence (brown) and directly-extraatedd DINO correspondence pairs (green).
each vertex indicates the correspondence intensity with p:. Then, we directly transfer the distribution on the source mesh onto the target mesh using the one-to-one vertex correspondence induced by deformation (recall that both meshes are deformed from the same mesh prior). We then re-project the correspondence distribution back from the target mesh to the target image using W 3-DD-D, resulting in distribution over target image pixels. Concretely, this distribution equals the product of two correspondence matrices, from where we can calculate the correspondence:
hocw
W forward
Wffowardd WDDD- 3-
forward Σ
[X2,
(5)
Wffoward
where we calculate q sum over the locations of target pixels. This constructs the forward correspondence mapping ftorward P; + q forward To form a cycle, we need to connect it with a backward mapping fhackward which operates directly between the source image and the target image and finds correspondences. To achieve this, we leverage a self-supervised pretrained feature extraction network DINO (Caron et al., 2021), and follow prior work (Goodwin al., 2022) extract & high-confidential correspondence pairs between the target and source image, {(q forward P. } where p, sycle is the pixel location going through the whole cycle back in the source image. Our cross-instance cycle-consistency loss Linst is defined to be the L2 distance on position offsets over the DINO correspondence pairs:
forward as the corresponding pixel on the target image for p. by a weighted
oyele) )
& cycle -p.12-
Linst Σ 1
lp,
(6)
Linst operates on images of different object instances. This can be easily extended to a cross time setting by selecting source and target images to be different frames of the same video clip, leveraging the continuity in time to learn geometric consistency. We denote this cross-time consistency loss as Lume- An illustration of both losses can be found in Fig.3. We combine all cycle-consistency losses as Ceyele $20-3D L2D-aD + AaD-2D L3D-2D + Binst Linst + Bime Lime-
3.4 TRAINING AND INFERENCE
Training. All the operations in our model are differentiable, which allows us to train all components in end-to-end manner. Besides the reconstruction loss Creon and the cycle-consistency loss Leyeles we also apply regularization loss Leeg on the learned shape to enforee shape smoothness, symmetry and minimize deformation. The total training objective is weighted of them, Lunal X Lreeon + Aeyclee Loyele + >veg Lrg. (See Appendix A.2 for more details.)
Inference. During inference, model estimate object shape and correspondences with RGB images. Additionally, for inference of the 6D pose, depth is also required. We first extract the 2D-3D geometric correspondence mapping, forming correspondence pair set of M elements, where M is

---

Published as a conference paper at ICLR 2023
Figure 4: Visualizing pose estimation and mesh reconstruction on Wild6D. We show the correspondence
mapping, the estimated pose and the reconstructed meshes.
5 degree 2cm
5 degree 5em
degree 2cm
10 degree
Methods
Data
1OU0.25
IOUo.s
5cm
CASS (Chen et al., 2020a)
C+R
19.8
1.0
0
0
Shape-Prior (Tian al., 2020)
C+R
55.5
32.5
2.6
3.5
9.7
13.9
DualPoseNet (Lin et al., 2021)
C+R
90.0
70.0
17.8
22.8
26.3
36.5
GPV-Pose (Di et al., 2022)
C+R
91.3
67.8
14.1
21.5
23.8
41.1
RePoNet (Fu & Wang, 2022)
C+W*
84.7
70.3
29.5
34.4
35.0
42.5
Ours
W
92.3
68.2
32.7
35.3
38.3
45.3
Table 1: Comparison with the SOTA methods Wild6D. The "Data" column records the data for training with C=CAMERA25, R=REAL2755, W=Wild6D, =n"=not using pose annotation.
the number of pixels on the object foreground mask. To remove probable outliers, we first apply selection scheme the correspondence pairs based on the consistency between 2D-3D and 3D-2D correspondence, Concretely, we define a confidence measure based on the cycle-consistency of our 2D)-3D) and 3D>-D matching:
conf, exp - Vpel. I.
High confidence indicates agreement between both directions of correspondence. We sort correspon- dence pairs using this confidence, and select those above threshold 0.5. We also lift the depth map of the object into partial point cloud, forming 3D rigid transformation estimation problem. We use the Umeyama algorithm (Umeyama, 1991) to solve this problem, with RANSAC (Fischler & Bolles, 1981) for outlier removal.
4
EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Category-level 6D object pose estimation. We conduct the main body of experiments on Wild6D (Fu Wang, 2022), which contains rich collection of 5, 166 videos across 1722 different objects and categories (bottle, bowl, camera, laptop and mug) with diverse appearances and backgrounds. The training set of Wild6D provides RGB-D information and object foreground segmentation masks generated by Mask R-CNN (He et al., 2017), and the test includes 6D pose labels generated by human annotation. On this dataset, we compare with the current state-of-the-arr semi-supervised method, RePoNet (Fu & Wang. 2022) and several pretrained supervised methods transferred this dataset. Following Wang et al. (2019b), we report the mean Average Precision (mAP) for different thresholds of the 3D Intersection over Union (IoU) and the m degree, n cm metric.
We also train and evaluate our model on the widely-used REAL277 dataset (Wang et al., 2019b), which contains smaller set videos (7 training scenes and 6 testing scenes), posing a large challenge on training the model. We categorize prior works REAL275 into 3 categories based on supervision: 1) supervised; 2) self-supervised (with synthetic data and annotations); 3) self-supervised (without synthetic data). Our method tackles the problem in the most challenging setting.
Keypoint transfer. To thoroughly evaluate our novel CSE representation and 2D-3D geometric correspondence, evaluate the keypoint transfer (KT) task. We use the CUB-200-200 dataset (Wah et al., 201 1) containing 6,000 training and test images of 200 species of birds. Each bird is annotated with foreground segmentation and 14 keypoints. During training,
only the RGB
7

---

Published as a conference paper at ICLR 2023
SB
Figure 5: Visualizing pose estimation and correspondence on REAL275 (left) and CUB-200-2011 (right).
Supervision
Methods
Data
IOU
IOU
degree
degree
5cm
5cm
NOCS (Wang et al., 2019b)
78.0
10.0
Shape-Prior (Tian et al., 2020)
C+R
77.3
21.4
54.1
Supervised
FS-Net (Chen et al., 2021)
C+R
95.1
92.2
28.2
60.8
DualPoseNet (Lin al., 2021)
C+R
79.8
35.9
68.8
Peng (2022)
C+R
73.0
19.6
Self- supervisdd
RePoNet (Fu & Wang, 2022)
C+R
85.8
76.9
31.3
56.8
(with
UDA-COPI (Lee et al., 2022)
C+R*
84.0
82.6
34.8
66.0
synthetic data
Chen et al. (2020b)
15.5
1.3
0.9
2.4
and annotations)
CPPF (You et al., 2022)
78.2
26.4
16.9
Self-supervised
He ct al. (2022)
R*+Y*
83.5
58.7
5.6
17.4
(without
Ours-REAL
R
76.3
41.7
11.6
28.3
synthetic data)
W
89.3
49.5
13.7
33.7
Table 2: Comparison with SOTA methods on REAL275. In "Data" column, C-CAMERA25, R-REAL275. synthheti objects, Y-BCB (Xiang al., 2017). W Wild6,, "**"=nol using pose annotation.
images and segmentation maps, with depth keypoint annotation. For inference, following CSM (Kulkarni et al., 2019), we take the Percentage of Correct Keypoints (PCK) as the evaluation metric, which measures the percentage of correctly transferred keypoints from a source image to target image if the keypoint is visible in both. Please refer to Appendix A.3 for more details.
4.2 COMPARISON WITH SEATE-FF-THE-ARE
Pose estimation performance. The pose estimation performance on Wild6D is reported in Table 1. Surprisingly, our method outperforms all previous methods only except for the IoUn.5 metric, while being the only method that uses completely no synthetic data 3D annotated data. On pose estimation metric 5 degree, 2cm and 10 degree, 2cm, our method outperforms the previous state-of-the-as semi-supervised method RePoNet (Fu & Wang, 2022) by 3.2% and 3.3% respectively, A visualization of the predicted correspondence, pose and the reconstructed mesh is shown in Fig.4. 4. More visualizations are given Fig. and Appendix C.
The result REAL275 is reported in Table 2. In the table, Ours-REAL is trained only REAL275. and Ours Wild6D trained only on Wild6D. We outperform previous self-supervised methods on most metries. Compared with supervised methods and self-supervised methods using annotated synthetic data, our performance is lower mainly because 1) the REAL275 training set is small in size, and 2) synthetic data with ground truth shape and pose can provide a strong supervision. Compared Ours-WildoD ability across image domains. A visualization of the correspondence and pose estimation results on REAL275 is shown in 5
with Ours-REAL275, Ours- Wild6D achieves a higher performance, showing a strong generalization
Keypoint Transfer Performance. Our key- point transfer results on the CUB dataset of birds (Wah al., 201 1) is shown in Table Our model outperforms all previous methods, including UMR (Li et al., 2022) which lever- ages pre-trained co-part segmentation network for supervision. We outperform CSM (Kulkarni al., 2019) and A-CSM (Kulkarni et al., 2020) by over 20%. showing the efficacy of our CSE representation compared with the existing sur-
Methods
KP Mask PCK
CMR (Kanazawa et al., 2018)
47.3
CSM (Kulkarni et al., 2019)
V
45.8
CSM
36.4
A-CSM (Kulkarn et al., 2020)
42.6
IMR (Tulsiani al., 2020)
53.4
UMR (Li et al., 2022)
58.2
SMR (Hu et al., 2021)
62.2
Ours
64.5
Table 3: Keypoint transfer result on CUB-200-2011.
8

---

Published as a conference paper at ICLR 2023
Method
10Up.s
5"
2cm
5" 5em
10"
2cm
10" 5em
w/o correspondence
70.1
32.0
15.3
20.5
23.6
34.0
w/o surface embedding
87.3
57.1
24.0
30.1
31.8
42.1
w/o cross-instance and cross-time eycle loss
89.9
63.0
30.9
32.8
36.4
42.8
w/o depth loss
89.8
52.6
13.0
14.7
27.4
35.6
w/o deformation
87.5
55.3
21.2
26.8
32.8
44.0
Full model
92.3
68.2
32.7
35.3
38.3
45.3
Table 4: Ablation study on model designs and loss functions.
Cycle level
Laptop
Bird
Lime Lima
IOUp.s
5"
2cm
5"
5cm
10"
5em
PCK
94.6
9.2
10.2
38.8
33,9
94.7
10.4
11.0
38.4
61.0
95.9
11.6
13.5
39.7
63.4
96.0
12.7
16.0
42.9
64.5
Table 5: Ablation study on the cross-instance and cross-time cycle-consisteney loss.
face mapping approach. We visualize the 2D-3D correspondence maps in Fig. 5. Despite large shape and pose variance, our model can still estimate correct correspondence relationships across different images of birds.
4.3 ABLATION STUDY
CSE and geometric correspondence. We first evaluate the effectiveness of the proposed CSE representation and geometric correspondence. In w/o correspondence setting, ablate both the mesh encoder and image feature extractor, and use no correspondence computation. At test time, instead of solving for pose from correspondences, the image encoder. This results in a large performance drop. as shown in Table 4, which proves regressing pose leads more difficulties in generalization than solving pose from correspondence.
the regressed rotation and translation from
In w/o surface embedddigg setting, we ablate the mesh encoder and have the image feature extractor directly output surface mapping, which serves the 2D-3D correspondence map. The performance under this setting is also significantly lower, showing the efficacy of our CSE representation.
Cross-instance and cross-time cycle-consistency. In method, propose the novel cross- instance and cross-time cycle-consistency loss to facilitate semantic-aware correspondence learning. Thus we also ablate this loss to verify its effect. In Table 4, we show the comparison training with and without the cross-instance and cross-time cycle-consistency loss on Wild6D. With the novel loss, performances at all metries improve, with the mAP IOUg.5 improved by 5.2%. This proves that our novel cycle-consistency losses can effectively leverage image semantics and thus achieves an improvement over the single-image projection- cycle loss.
In Table 5, we provide more detailed ablation on the two cycle levels Limm and Lima- We take the laptop category as an example to show the effectiveness of the cross-instance and cross-time loss for it has large pose and appearance variances. From this table, we can observe that adding proposed wycle-consistenge loss constantly improves the pose estimation accuracy. Combining both eycle levels achieves the highest performance, and increases the mAP at 5 degrees, 5em by 5.8% in total. We also evaluate on the CUB dataset Table 5. The PCK increases significantly by applying the cross-instance and cross-time cycle-consistency. The best-performing model increases over the model without cross-instance and cross-time cycle-consistency by 30.6%.
Depth loss. We also ablate the depth loss, shown in Table 4. Results show that using depth data causes a performance drop. Empirically, we find that some categories are relatively depth-sensitive, e.g. bowl. Our depth loss can largely eliminate the misaligning of these shapes.
CONCLUSION
In this work, we consider the task of self-supervised category-level 6D object pose estimation in the wild. We propose framework that learns the Categorical Surface Embedding representation and constructs dense geometric correspondences between images and 3D points on a categorical shape prior. To facilitate training of CSE and geometric correspondence, we propose novel cycle- consistency losses. In experiments, our self-supervised approach can perform on-par or even better than state-of-the-art methods category-level 6l) object pose estimation and keypoint transfer tasks.

---

Published as a conference paper at ICLR 2023
Reproducibility statement. All experiments in this paper reproducible. In Appendix A.1, we describe the detailed model architecture. In Appendix A.2 and Appendix A.3, we provide details for training and inference. In Appendix A.4, we provide all hyperparameters for training model. The source code and training scripts are available at https: httpt/gttthhco..ooo com/ hywind// sel pose
Acknowledgement. Prof. Xiaolong Wang's group was supported, part, by gifts from Qualcomm.
REFERENCES
Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron: A large scale dataset of object-centric videos the wild with pose annotations. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7822- 7831, 2021. 3
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceddings of the IEEE/CVF International Conference on Computer Vision, pp. 9650 9660, 2021. 3, 6, 15,
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-ine 3d model repository. arXiv preprint arXiv:1512.03312. 2015. 3, 14
Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category- level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19973-19982, 2020a. 3,7
Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape- based network for category-level 6d object pose estimation with decoupled rotation mechanism. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1581-1590, 2021. 3,
Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural aallysis-by-synhhett In European Conference Computer Vision, pp. 139-156. Springer, 2020b. 3, 8
Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via geometry-guided point-wise voting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
6781-6791, 2022. 7
Martin Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
Yang Fu and Xiaolong Wang. Category-level 6d object pose estimation in the wild: A semi-supervised learning approach and a new dataset. arXiv preprint arXiv:2206. 15436, 2022. 1, 2, 3, 7, 8, 15
Ge Gao, Mikko Lauri, Yulong Wang, Xiaolin Hu, Jianwei Zhang, and Simone Frintrop. 6d object pose regression via supervised learning on point clouds. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3643-3649. IEEE, 2020.
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In European Conference on Computer Vision, pp. 88-104, Springer, 2020.
Walter Goodwin, Sagar Vaze, loannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. arXiv preprint arXiv:2204.00 03635, 2022.
Riza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7297-7306, 2018.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017. 15
Yisheng He, Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun. Pvn3d: A deep point-wise 3d keypoinnss voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1632-11641, 2020.
Yisheng He, Haoqiang Fan, Haibin Huang, Qifeng Chen, and Jian Sun. Towards self-supervised category-level object pose and size estimation. arXiv preprint arXiv:2203.0288
2022. 3,
Tao Hu, Liwei Wang, Xiaogang Xu, Shu Liu, and Jiaya Jia. Self-supervised 3d mesh reconstruction from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6002-6011 1, 2021. 3, 8
Shuaiyi Huang, Qiuyue Wang. Songyang Zhang, Shipeng Yan, and Xuming He. Dynamic context correspondence network for semantic alignment. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2010-2019, 2019.
Muhammad Zubair Irshad, Thomas Kollar, Michael Laskey, Kevin Stone, and Zsolt Kira. Centersnap: Single-shot multi-object 3d shape reconstruction and categorical 6d pose and size estimation. arXiv preprint arXiv:2203.0199 2022a.
Muhammad Zubair Irshad, Sergey Zakharov, Rares Ambrus, Thomas Kollar, Zsolt Kira, and Adrien Gaidon. Shapo: Implicit representations for multi-object shape, appearance, and pose optimization. arXiv preprint 22t.:::200.13399 2022b. 3
Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. In Advances in Neural Information Processing Systems, pp. 19545-19560, 2020.
Sangryul Jeon, Seungryong Kim, Dongbo Min, and Kwanghoon Sohn. Parn: Pyramidal affine regres- sion networks for dense semantic correspondence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 351-366 1366, 2018.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018. 3, 8
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Nilesh Kulkarni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019. 3, 5.
Nilesh Kulkamni, Abhinav Gupta, David F. Fouhey, and Shubham Tulsiani. Articulation-aware canonical surface mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 3,8
Taeycop Lee, Byeong- Lee, Inkyu Shin, Jacsung Choe, Ukcheol Shin, So Kweon, and Kuk-Jin Yoon. Uda-cope: Unsupervised domain adaptation for category-level object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14891-14900, 2022. 8
Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint- task self-supervised learning for temporal correspondence. Advances Neural Information Processing Systems, pp. 317-327, 2019. 3
Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, and Jan Kautz. Self-supervised single-view view 3d reconstruction via semantic consistency, February 2022. US Patent 11,238,650. 3.8
Yi Li. Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 683-698, 2018.
11

---

Published as a conference paper at ICLR 2023
Jiehong Lin, Zewei Wei, Zhihao Li, Songeen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category- level 6d object pose and size estimation using dual pose network with refined learning of pose consisteney. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3560-3569, 2021. 1,3,7 7,8
Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A Vela, and Stan Birchfield. Single-stage keypoint-based category-level object pose estimation from an rgb image. In 2022 International
Conference on Robotics and Automation (ICRA), pp. 1547-1553. IEEE, 2022. 3
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019. 3,5
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1111.05100,, 2017. 15
Fabian Manhardt, Gu Wang, Benjamin Busam, Manuel Nickel, Sven Meier, Luca Minciullo, Xi- angyang Ji, and Nassir Navab. Cps++: Improving class-level 6d pose and shape estimation from monocular images with self-supervised learning. arXiv preprint arXiv:2003.05845 2020. 3
Natalia Neverova, David Novotny, Mare Szafraniec, Vasil Khalidov, Patrick Labatut, and Andrea Vedaldi. Continuous surface embeddings. Advances in Neural Information Processing Systems, 13:17288-17270 2020. 3
Markus Oberweger, Mahdi Rad, and Vincent Lepetit. Making deep heatmaps robust to partial occlusions for 3d object pose estimation. In Proceedings of the European Conference Computer Vision (ECCV), pp. 12.1119 119-134, 2018.
Wanli Peng, Jianhang Yan, Hongtao Wen, and Yi Sun. Self-supervised category-level 6d object pose estimation with deep implicit shape representation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2082-2090. 2022. 3,8
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 652-660, 2017. 4, 14
Ignacio Rocco, Relja Arandjelovic, and Josef' Sivic. Convolutional neural network architecture for geometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6148-6157 3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxiv::409. 1556, 2014. 15
Meng Tian, Marcelo H. Ang, and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), ECCV 2020, 2020. 1, 3, 8
Prune Truong, Martin Danelljan, Fisher Yu, and Luc Van Gool. Warp consistency for unsupervised learning of dense correspondences. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10346610336, 2021. 3
Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unanno- tated image collections. arXiv preprint MxXiv:2007.08504 2020. 3,8
S. Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions Pattern Analysis and Machine Intelligence, 13(04))376-380, apr 1991. ISSN 1939-3539, doi: 10.1109/34.88335 88573. 3,7
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In Proceedings of the European conference computer vision (ECCV), pp. 391-408, 2018. 3
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011. 8
12

---

Published as a conference paper at ICLR 2023
Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin-Martin, Cewu Lu, Li Fei-Fei, and Silvio Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3343-3355 2019a.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR),
June 2019b. 1, 7,
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2019c.
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. arXiv preprini arXiv:2107. 10844, 2021. 3
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes, arXiv preprint arXiv:1711.0019: 2017. 3,
Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: video frame-level similarity perspective. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV). pp. 10075-10085, October 2021.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15980-15989, 2021a.
Gengshan Yang. Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19326-19338, Curran Associates, Inc., 2021b.
Yufei Ye, Shubham Tulsiani, and Abhinav Gupta. Shelf-supervised mesh prediction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0843-8852. 2021.
Yang You, Ruoxi Shi, Weiming Wang, and Cewu Lu. Cppf: Towards robust category-level 9d pose estimation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6866 6875, 2022.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2881-2890, 2017. 14
Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-126, 2016.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019. 5, 14
A IMPLEMENTATION DETAILS
A.1 MODEL ARCHITECTURE
Image encoder. We use ResNet-18 as our image encoder backbone. We apply average pooling on the output of the th res-bloc, resulting in an image global feature with dimension 512. This
13

---

Published as a conference paper at ICLR 2023
Figure 6: Shape priors. From left to right: bottle, mug, bowl, laptop, camera, can, bird.
Properties
Bottle
Mug
Bowl
Laptop
Camera
Can
Bird
Dataset of vertices
W. R
W. R
W, R
W, R
W. R
CUB
642
886
482
995
974
680
555
Type of symmetry rotation flip rotation
flip
rotation
flip
Table 6: Properties of each category. W=Wild6D, R-RFAL.275.
global feature then serves the input for the pose predictor, which uses 4- 4-yyyer MLP with hidden dimension 128 to regress the 6D rotation representation (Zhou et al., 2019), and l-layer translation predictor that predicts the 3D translation. The image global feature is also fed into a fully connected layer to get the shape code Ushape with dimension 64.
Shape predictor. We use coordinate- based MLP with 5 fully connected layers for shape deformation prediction. The input for the network is the concatenation the 3-dimensional vertex coordinate and the 64-dimensional global shape code, and the output is 3-dimensional x; y, 2 offsets.
Mesh encoder and image feature extractor. We choose feature dimension 64 for our CSE represen- tation. For the mesh encoder, surface embedding. For the image feature extractor, we use a PSPNet (Zhao et al., 2017) decoder which fuses different levels of the image encoder feature maps to get the final image features. The final feature map is of size 64 x 64 (4x downsampling).
3-layer PointNet (Qi et al., 2017) structure for extracting the
A.2 TRAINING DETAILS
Data Sampling. For each RGB image input, we crop the image to 256 x 256 and locate the object at the image center. In each training batch, we sample b videos randomly from the dataset and sample t frames from each video uniformly. For the videos, randomly sample bird species and images with each species instead. This sampling strategy ensures that there are multiple images from the same video/specie in a batch for cross-time cycle-consistency learning.
dataset, since the dataset contains no
Shape prior. We select our shape prior by choosing a synthetic dataset from the ShapeNet (Chang al., 2015) dataset, then simplify the mesh contain appropriate number of vertices. A visualization of all the shape priors we list is listed in Fig. 6. More properties of each category are listed in Table
Regularization. We apply regularization losses in training, including deformation loss, Laplacian loss and symmetry loss. The deformation loss the smooth LI loss on the deformation A V. The Laplacian loss is by applying Laplacian smoothing on the deformed mesh. For symmetry loss, given a reconstructed mesh we uniformly sample m points, then apply K rotate/flip transformations depending on symmetry types, creating point cloud with m x K points. The symmetry loss is defined as the Chamfer distance between the point cloud and the original mesh.
Details of instance cycle-consistency loss. In the instance cycle-consistency loss (Eq. 4), we use .. to denote the inverse projection operation. We implement this inverse projection by casting a camera ray starting from a pixel and finding its intersection with the mesh. The point becomes the inverse projection result of the pixel. The visibility mask of the instance cycle-consistency loss is applied over the vertices, with value 1 for the vertices that visible after the projection, and value for the vertices that are occluded.
Details of cruss-instance cycle-consistency loss. When applying cross-instance cycle-consistency loss, we group batches of size B into B/2 image pairs. For Limas we pair images from different videos; for Limes we pair images from different frames of the same video. To stabilize training,
14

---

Published as a conference paper at ICLR 2023
Hyperparameters # of iterations
Wild6D
REAL275
CUB
20,000
10,000
5,000
(grenue, Smask Paeyth)
(Eq. 3)
00.05,0.15,000 0.1)
(0.05, 0.15,0.11
00.05,0.15,
0.15,
(Bm.SD, BID-SD, Sust. Busss) (Sec. 3.3)
(0.02, 0.02, 0.05, 0.05) (0.02, 0.02, 0.05, 0.05) 0.1
0.00,,0001,..00 0.1,
T (Eq.
0.1
0.1
1s (Eq. 6)
200
200
200
(Arccon: Acyeles Aves) (Sec. 3.3)
(1, 1,,
(1, 1,
(1,
Table 7: Training hyperparameters.
Method
3D/2D Transfer PCK
VGG (Simonyan Zisserman, 2014)
2D
17.2
DINO (Caron et al., 2021)
2D
60.2
Ours-2D
2D
72.9
Ours
3D
64.5
Table 8: Keypoint transfer result. 2D Transfer indicates transferring keypoints directly with image-image feature distance. 3D Transfer indicates transferring keypoints with mapping into the 3D space.
also add an auxiliary cycle loss, which is defined as follows: rotating the source image by a random degree to get target image, and constructing similar cycle as Linst Correspondence pairs directly come from rotation.
Depth. For the category-level 6D pose estimation task, our model requires RGB-D image to train in order to achieve the best performance. The depth only used for constructing a depth loss. During inference, depth is also required; otherwise, the relative scale of the translation cannot be correctly estimated due to the scale ambiguity in perspective projection. For the keypoint transfer task, our model only requires RGB images during both training and inference, since the task does not require to estimate the absolute depth of the object.
A.3 INFERENCE DETAILS
Given two images, we apply our image and mesh encoder to obtain the 2D-3D geometric correspondence both images. Then, for each keypoint in the source image, first extract its corresponding 3D location from the mapping. Then, for each pixel on the target mesh we also map it into 3D. Finally, we choose the point with the closest 3D distance to the 3D location of the keypoint. The corresponding target pixel becomes the predicted keypoint. If the distance between our prediction and the ground truth is less than a x max (h, w) (h, w are the height and width the bounding box of the object), then the prediction is considered correct. In all experiments, we report PCK at a 0.1.
Keypoint transfer.
Symmetry in pose estimation. For prediction can be an arbitrary rotation of the ground truth pose along the symmetric axis. Therefore, uniformly sample K poses around the symmetric pose candidate ground truth poses, and choose the one with the minimum error with the predicted pose.
categories (e.g. bottle, bowl, can), a correct
A.4 HYPERPARAMETERS
For all our experiments, we use AdamW (Loshchilov & Hutter, 2017) as our optimizer with learning rate (r 1 x 10-4, and apply cosine learning rate decay. The learning rate for vertex deformation set to 1 x 10-5, We use batch size 64 (with 16 videas, 4 frames each video). More hyperparameters are listed in Table
A.5 WILD6D SEGMENTATION MASKS
We use the segmentation masks for Wild6D provided by Fu & Wang (2022). The segmentation is generated by applying Mask R-CNN (He et al., 2017). Although this auto titto-enerraton pipeline can produce some errors, we find that in most images the segmentation masks are satisfactory. Figure shows some image and mask instances in the dataset.
15

---

Published as a conference paper at ICLR 2023
Figure 7: Visualizing segmentation masks in Wild6D,
B ADDITIONAL EXPERIMENT RESULTS
B.I. COMPARISON WITH DINO ON KEYPOINT TRANSFER
In our work we use DINO (Caron et al., 2021), a self-supervised pre-trained image backbone, to find correspondence pairs between cross- cross-instance images. To verify whether model's performance gain solely relies on DINO, we conduct experiments to evaluate DINO on the task of keypoint transfer. The results are shown in Table 8. Compared with DINO, Ours-2D similarly uses direct 2D transferring, and results in higher performance than DINO (72.9% vs. 60.2%). Furthermore, our 3D-aware transferring method also outperforms DINO by 4.3%. These results show that our method is leveraging DINO correspondences to improve the feature representations, and we also learn 2D-3D correspondences, which DINO is not capable of.
C VISUALIZATION
We provide a more detailed visualization of our model on pose estimation, keypoint transfer, cor- respondence learning and mesh reconstruction. In Figure 8 and 9, we show the keypoint transfer result and the learned geometric correspondence on CUB-200-2011. visualizations of the reconstructed meshes on Wild6D. In Figure 11, we show the correspondence and pose estimation results on REAL275. In Figure 12, we show comparisons of our estimated pose and the ground truth pose the Wild6D dataset.
Figure 10, we provide some
16

---

Published as a conference paper at ICLR 2023
Source
Target
Target GT
Source
Target
Target GT
Figure Visualizing keypoint transfer on CUB-200-2011. The three columns in group represent ground truth keypoints on the source image, the transferred keypoints on the target image, and the ground truth keypoints on the target image, respectively.
Figure 9: Visualizing geometric correspondence on CUB-200-201L. color each pixel depicts corresponding 3D point location in the canonical space. Note that our model learns mostly continuous and consistent correspondences despite large pose and appearance changes.
17

---

Published as a conference paper at ICLR 2023
Figure 10: Visualizing pose estimation and mesh reconstruction Wild6D. We show the correspondence mapping, the estimated pose and the reconstructed meshes under 2 different viewpoints chosen randomly.
SO
KIT
Figure 11: Visualizing correspondence and pose estimation REAL275. We train the model only Wild6D and test on REAL.275, producing these results.
18

---

Published as a conference paper at ICLR 2023
BRANIN
Figure 12: Visualizing pose estimation on Wild6D, The ground truth bounding boxes are colored green. and the predicted bounding boxcs are colored in red.
19


Reviews:
[{"review_id": "rDPbP_93JK", "reviewer": "Reviewer_ohWa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes to learn a model for category-level object pose estimation. The proposed approach relies on a dataset of RGB-D images of a certain category of objects, e.g. mugs or bottles, with a template mesh of an object belonging to the category. The method does not require any pose annotation to train, however it leverages recent DINO features during training. The model can still be called self-supervised because DINO learning is self-supervised itself. The approach is evaluated on the Wild6D dataset and is shown to be on par with the current state-of-the-art.", "summary_of_the_review": "Summarising, the paper contains good and valuable ideas. The evaluation shows, that the method is the best on Wild6D in terms of IoU .25 metric as well as pose metrics. However, the situation is mixed on the REAL275 dataset, and here there are question regarding the adequate baselines, because the paper refuses to accept that the competitive methods also fall into the 'self-supervised' category, while the competitive methods are also called 'self-supervised'. This is a difficult situation. Also, the paper is evaluated on just one in-the-wild dataset with 5 simple object categories. I would suggest evaluation on a ore diverse set of object categories.", "strength_and_weaknesses": "I see at least three strong points in the submitted work. First of all, the idea to formulate 2D-3D pose regression through cross-attention between the image pixel features and the mesh vertex vertices seems new, original and interesting. It is one step forward compared to the direct regression of 3D coordinates from 2D images, which became a traditional tool in dense correspondence and pose estimation models wince the Vitruvian Manifold work of 2012. As the ablation study shows, without this element, the model starts to perform significantly worse.\n\nThe second strong point is that the evaluation is done on a big real dataset of objects (1722 videos of 5 object categories). The baselines are chosen as rather recent methods, and the metrics are standard and widely used.\n\nThe third strong point is the evaluation on a keypoint correspondence task, which shows the power of the learned model beyond the primary task.\n\nHowever, while the current dataset is diverse in terms of objects, it is very constrained in terms of object categories (only 5 of them, and all are rather simple). It is unclear whether a method that performs well on these categories will generalize to arbitrarily complex categories, e.g. a vehicle, etc., and there is a very popular and recent dataset (CO3D) that can be used as well.\n\nNext, it is not well described in the paper, in my opinion, that the method needs depth data to train. It would be very interesting to see what metrics can we have if we don't use depth for training. It is often the case that the dataset of RGB images is much easier to collect. As an intermediate baseline, maybe one may use some self-supervised depth estimation models here as well. I see no problem in relying on some machine learning models for annotation if they are self-supervised as well. \n\nWhile I see that the reconstruction loss is important, probably I'd suggest explaining clearly why is it needed if we have the instance consistency loss. The problem is, the reconstruction loss enforces the colour consistency constraint between the projected vertices and the pixels. The instance consistency loss basically enforces consistency between the projected vertices and the pixels as well. It is stronger, because colours may be the same or similar, but coordinates may be very different. It is also not very clear why removing the depth loss makes things worse, because in my opinion (correct me if I'm wrong) but the depth supervision is already inside the instance consistency loss.\n\nThe experiment with keypoint transfer is interesting. But I would like to see the result of keypoint transfer by DINO features as well, to be able to compare and understand if there are any gains compared to this model.\n\nMoreover, there is one question. Why are tthe DINO features used in the cycle consistency, but not the proposed 2D features?\n\nIn the Fig. 2, the cycle losses should be shown as well.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "coY69yRt4ph", "reviewer": "Reviewer_hRgU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a self-supervised method for Category-level 6D Object Pose Estimation. Specifically, the paper proposes an architecture and a number of geometry-based consistency losses allowing simultaneous estimation of the shape of specific instance and its pose. The experimental sections, demonstrates improvements over SOTA.", "summary_of_the_review": "I give positive rating because in my opinion the paper is well written and novel, and will be interesting to the community.", "strength_and_weaknesses": "The self-supervised nature of the method is attractive and makes it more applicable due to lower costs of dataset acquisition. The results improve the SOTA. Interestingly, the paper achieves better results than some supervised methods. Ablation study validates the architecture and loss choices. The methods is validated on two domains (images and videos) and two tasks (6d pose estimation itself and key points transfer)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DcQYwGpD-7", "reviewer": "Reviewer_pKTW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a self-supervised framework for category-level 3D object pose estimation. A novel category surface embedding module is proposed which can help establishing dense correspondences within a single instance, between two different instances of the same category, and instances in different time frames. Experiment shows that the proposed algorithm achieves impressive performance on 3D pose estimation and keypoint transfer.", "summary_of_the_review": "In summary, I recommend that the paper could be accepted if all the questions in the weakness part are addressed", "strength_and_weaknesses": "Strength:\n1. The proposed Categorical Surface Embedding module and the corresponding cycle-consistency loss function is interesting and novel. \n2. The experiment result on pose estimation in the wild and dense key-point transfer on CUB dataset is impressive and the ablation study clearly shows the effectiveness of suface embedding and cycle loss.\n\nWeakness:\n1. There are some hyperparameters which weight different items in the overall loss function, such as \\beta_texture, \\beta_mask, \\beta_2D-3D, etc. From Table 7 in Sec. A.3, it seems that such hyperparameters varies across Wild6D and CUB in training, does the algorithm's performance depends significantly on these hyper-parameters?\n2. In Table 2, the proposed algorithm get good performance on the 'Self-supervised' setting, but significantly lower performance than algorithms in 'Synth supervised' setting, since the labeled synthetic images can be obtained with limited effort, what is the benefit of self-supervised training on Wild6D without pose label? and is it possible to test the setting of W*+R* data settting and compare to the method of UDA-COPE (Lee et al., 2022) with C+R* data setting?\n3. In Figure 2, it is not clear whether and how the deformation \\delta_V is regularized, there should be some sentences in sec.3.1 referring to Appendix A.2 and the regularization loss should be reflected in Figure 2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Rg9ueUi-A", "reviewer": "Reviewer_SV21", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn category-level 6D object pose in an unsupervised manner. The key contribution lies in a framework to learn 2D-3D correspondences between image and a category-level canonical shape as a mesh. The correspondences are learned by regressing surface embedding on both image pixels and mesh vertices, which are then matched to obtain correspondences. The unsupervised losses mainly consist of differentiable rendering loss and various cycle-consistency losses.", "summary_of_the_review": "The paper is solving a practically important problem, i.e. unsupervised learning of 6D object pose. The overall framework makes senses and works reasoablely in the experiments. That said, some important details needs further clarification as discussed above. ", "strength_and_weaknesses": "Strength\n+ Overall the paper is clearly written and easy to follow.\n+ The problem of 6D object pose estimation in the wild is of importance in practice, such as in robotic applications. \n+ The major strength over prior arts lies in its unsupervised nature, which does not require any human annotation of object pose. \n+ The experiments demonstrate state-of-the-art performance.\n\nWeakness:\n- Some of the technical deteils are not clear.  For example, it is not clear whether the mean shape is optimized or fixed - Sec 3.1 states that the mean shape is learnable but according to Sec. A.2 the mean shape is selected from synthetic data in ShapeNet.  \n- If the method learns both the category-level mean shape and per-instance shape deformation, the decompotion of mean shape and deformation is inherently ambiguious. Any change in the mean shape can be compansated by its inverse change in the deformation. It seems there is not any regularization to resolve such ambiguity. \n- The paper has been vague about whether the depth is required in the input. My understanding is yes, because the method estimates pose by Umeyama algorithm. It is then a limitation if RGBD is required as input rather than RGB image only. The paper also acknowledges that removing depth loss leads to a drop in performance. Without depth data, how is the absolute scale of translation determined, given the ambiguity between objece size and distance caused by perspective projection?  \n- There is no ablation study on the impact of deformation compared to just using a canonical shape.\n-  Many of the objects shown in the paper are symetric, hence it is again ambiguious to define the pose. Does this cause instabability in training and how is it handled?  \n - The backprojection from 2D to 3D in the right equation of Eq.(4) needs depth as well. It is better to point out this for clarity.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster