Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset
Authors: Yang Fu
Xiaolong Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: 6D object pose estimation is one of the fundamental problems in computer vision and robotics research. While a lot of recent efforts have been made on generalizing pose estimation to novel object instances within the same category, namely category-level 6D pose estimation, it is still restricted in constrained environments given the limited number of annotated data. In this paper, we collect Wild6D, a new unlabeled RGBD object video dataset with diverse instances and backgrounds. We utilize this data to generalize category-level 6D object pose estimation in the wild with semi-supervised learning. We propose a new model, called Rendering for Pose estimation network RePoNet), that is jointly trained using the free ground-truths with the synthetic data, and a silhouette matching objective function on the real-world data. Without using any 3D annotations on real data, our method outperforms state-of-the-art methods on the previous dataset and our Wild6D test set (with manual annotations for evaluation) by a large margin.  Project page with Wild6D data:  \url{https://oasisyang.github.io/semi-pose/}.

Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset
Yang Fu
Xiaolong Wang
UC San Diego
Abstract
6D object pose estimation is of the fundamental problems in computer vision and robotics research. While a lot of recent efforts have been made on generalizing pose estimation to novel object instances within the same category, namely category- level 6D pose estimation, is still restricted in constrained environments given the limited number of annotated data. In this paper, collect Wild6D, new unlabeled RGBD object video dataset with diverse instances and backgrounds. We utilize this data generalize category-level 6D object pose estimation in the wild with semi-supervised learning. We propose a new model, called Rendering for Pose estimation network (RePoNet), that is jointly trained using the free ground- truths with the synthetic data, and silhouette matching objective function on the real-world data. Without using any 3D annotations on real data, our method outperforms state-of-the-art methods on the previous dataset and our Wild6D test set (with manual annotations for evaluation) by large margin. Project page with Wild6D data: https: / /oasisyang. gi thub. so/ssmi-- ppose/.
1 Introduction
Estimating the 6D object pose is one of the problems in computer vision and robotics. It predicts the full configurations of rotation, translation and size of a given object, which has wide applications including Virtual Reality (VR) [2], scene understanding [30], and [42, 57, 31, 49]. There are two directions in 6D object pose estimation. One is performing instance-level 6D pose estimation, where model is trained estimate the pose of one exact instance with an existing 3D model [13, 34, 22, 50, 32, 5, 14]. However, learning instance- model restricts its generalization ability unseen objects. To achieve generalization unseen instance, another direction is recently proposed to perform category-level 6D pose estimation using one model [48, 3, 41, 29, 6]. However, the large appearance and shape variance across instances largely increase the difficulty in learning.
To overcome this limitation, Wang et al. [48] take the initial step to collect real-world dataset and annotations for category-level 6D pose estimation. Combining synthetic data with free ground-truth annotations, they show the learned model can be generalized to unseen objects within the same category. While this result is encouraging, the generalization ability of the model is still limited by the number and the diversity of the data due to the challenges in annotating 6D object poses. Specifically, only 8,000 images across 13 scenes are collected and annotated from the dataset proposed in [48]. Thus it is still very challenging to generalize 6D pose estimation on diverse objects in complex scenes.
In this paper, we propose to generalize cetegory-level 61) object pose estimation in the wild. To achieve this goal, we introduce new dataset and new semi-supervised learning approach. Our key insight is that, while annotating the 6D object pose is challenging, collecting the RGBD videos for these objects without labels much easier and more affordable. On the other hand, there are infinite ground-truth annotations for synthetic data which come for free. We propose to leverage the benefits from both sides. We first collect rich object-centric video dataset with diverse backgrounds and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Training
Testing results Wild6D
Syathetic
Data
Reed-world
Data
Figure I: Left:: we train our model with both synthetic data and real- world data under our proposed semi-supervised setting. Right: during inference, given the RGBD images, the object pose can be estimated precisely. Green bounding boxes show the 3D bounding boxes projection results on 2D images.
object instances using an RGBD camera. We train our model jointly using synthetic data with the free ground-truth 6D pose annotations and the unlabeled real-world RGBD videos via a silhouette matching objective. In this way, our pose estimation model can be generalized to in-the-wild data with minimum human labor. We collect an RGBD video dataset for 6D object pose estimation in the wild, namely Wild6D. Each video in the dataset shows multiple views of one or multiple objects (see examples in Fig. 1). In total, there are 5166 videos (> 1.1 million images) over 1722 object instances and 5 categories, which is significantly (300x) larger than the previous 6D object pose estimation dataset [48]. For evaluation, we annotate 486 videos over 162 objects.
Given this dataset, we design novel model for semi-supervinn Pose estimation Network (RePoNet). The RePoNet is composed of two branches of networks with a Pose Network to estimate the 6D object pose and Shape Network to estimate the 3D object shape. Given an RGBD image of an object, our Pose Network first estimates the Normalized Object Coordinate Space (NOCS) map [48], which will be integrated with lower-layer features to regress the object 9D pose (rotation, translation, and size parameters). Meanwhile, the Shape Network takes a category-level 3D shape prior as an input, and estimates the object shape for the current input instance. During training, we utilize both the synthetic data with the full ground-truths and the real-world data from Wild6D with foreground segmentation masks (obtained by Mask R-CNN [12]). Given the inputs with synthetic data, we apply the regression losses on both the NOCS map, 6D pose parameters, and object shape. Additionally, given the estimated pose and object shape, perform differentiable rendering to obtain an object mask projected in 2D. A silhouette matching objective is proposed to compare the difference between the projected mask and the ground-trus mask. With the real RGBD data, although we do not have the 3D ground-truths, we can still learn with the silhouette matching objective by comparing the projected mask against the foreground segmentation. In this way, the gradients are back- back-propage propagated through the 6D pose and object shape to adjust both Pose Network and Shape Network for real RGBD data. During inference, we first apply our Pose Network to estimate the NOCS map with a given image. With the NOCS map output, the object pose can be computed by solving the Umeyama algorithm [43]. Some estimation results on Wild6D can be found in Fig.
learning, called Rendering for
In our experiments, we first evaluate our approach with the dataset proposed in [48]. Our RePoNet shows improvement over state-of-the approaches when trained in fully superviedd setting (using 3D ground-truths from both real and synthetic data). By using the real-world data without its 3D trouh with semi-supervised learning, our performance can still be on par with previous approaches using full annotations. We then experiment with our Wild6D dataset and consistently show a large performance gain over the baselines in-the-wild objects. We highlight our main contributions as follows:
A new large-scale dataset Wild6D of object-centric RGBD videos in-the-wild for category- level 6D object pose estimation.
A semi-supervised approach with RePoNet which leverages both synthetic data and real- world data without 3D ground- for category-level 6D object pose estimation.

---

Our approach outperforms baselines by a large margin, especially when deployed on in-the- wild objects. We commit release our dataset, annotation and code for benchmarking future research.
2 Related Work
Category-level 6D Object Pose Estimation. Recently, researchers have proposed to learn a single model for one category of objects instead of just one instance on pose estimation. For example, Wang et al. [48] propose a canonical shape representation of different objects called Normalized Object Coordinate Space (NOCS) to handle the instance variations. The object pose and size are calculated by the Umeyama algorithm [43] with predicted NOCS map and the observed points. Follow-up work using the NOCS representation has focused on improving the shape priors [41] and incorporating direct regression of object pose and size [3. 25, 6]. While these approaches can be deployed unseen object instances, the generalization ability of these models is still limited by the scale and the diversity of real-world annotated data in the REAL275 dataset [48]. Only 13 scenes with 18 real objects in total are presented in REAL2275. In contrast, we propose semi-supervised learning approach with RePoNet, which leverages a new large-scale unlabeled dataset Wild6D for training. Both our method and the new dataset are the key components that lead to our goal of generalizing category-level 6D pose estimation in the wild.
Large-scale 3D Object Datasets. Various large-scale 3D datasets have been proposed for different tasks including reconstruction and pose estimation. For example, the Objectron dataset are collected and studied with large-scale scale object-centric videos and 3D annotations [1, 15, 26]. However, there are no depth images provided in Objectron, which might lead to ambiguities in 6D object pose estimation. Similarly, the recent proposed CO3D [37] dataset with diverse instances in different categories is also collected without recording the depth. While we can perform 3D reconstruction or generate the depth map using COLMAP [38], the error in predicted depth makes it difficult be used for 6D pose estimation. Different from these datasets, the 3DScan [9] dataset records RGBD videos of different categories of objects. However, most objects are heavily occluded by hands only partially visible with large objects like cars. The diversity of objects and scenes is also relatively low given limited human labor. Thus 3DScan not suitable for pose estimation tasks. In this paper, we propose Wild6D, a dataset with RGBD videos containing diverse objects taken with diverse backgrounds. While the training set is not labeled, we provide 6D pose annotations for the test videos. To the best of our knowledge, Wild6D is the largest RGBD dataset for 6D object pose estimation in the wild.
Rendering in Object Pose Estimation. One effective refinement method for 6D object pose estimation is via rendering [24, 55, 20, 16]. For instance, Iwase et al. [16] propose to utilize differentiable rendering and learn the texture of a 3D model. However, it still depends a pre-defined CAD model with a high-quality texture map for instance-level pose estimation and it is not feasible for category-level. Additionally, object pose estimation can be conducted via mnalsis-by-Synttest [44, 8, 52, 46, 40J, but they usually require gradient descent for optimization leading to relatively slower inference speed. Critically, these works only focus the instance-level setting and none of them can generalize to different instances. Inspired by recent work on 3D reconstruction [17. 23, 7, 18], our RePoNet learns the object shape and pose (including both NOCS map and R/T/S parameters) simultancously using differentiable rendering to provide a loss function during learning [28], which allows the gradients to backprop through the whole network for training. While Manhardt et al. [29] also utilize differentiable rendering to improve category-leve 6D pose, it is only applied in test time to adjust the pose, instead of training the network end-to-end. We show substantial improvement over all baselines training with RePoNet.
3 Wild6D Dataset
Existing Category-level 6D Object Pose Datasets. NOCS [48] is the most common dataset for category-level 6D object pose estimation, which consists of the synthetic CAMERA25 dataset and the weal-world REAL275 dataset. The synthetic CAMERA25 dataset contains 300,000 RGBD images of 1,085 object instances from 6 categories for training and evaluation. The REAL275 dataset shares the same categories with CAMERA25 but is more challenging under the real-world background. contains 4,300 RGBD images of 7 scenes for training and 2,750 images of 6 scenes for testing. Due to the limited scale and diversity of real data, the learned models from this dataset cannot generalize to nn-the-wild scenarios. Objectron [1] is large-scale dataset for object pose estimation and tracking.
3

---

Object
Synthetic
Shape
RePoNet
NOCS
Synthetie
Data
map
Ansotations
Woal-world
R
Data
Lawe Flone of Syuthatic Data
Maek
Lews Real World Data
Figure 2: Our semi-supervised approach. For the synthetic data, we supervise it with all the annotations. While for the real-world data, we train it by comparing the binary mask generated by the rendering module with the object foreground segmentation.
Datasets RGBD Real #Categories Hinstances #images
Objectron [1]
I8K
4M
CAMERA25 [48]
184
300K
REAL275 [48]
24
8k
Wild6D
5
1.8K
1M
Table 1: Comparison between Wild6D and existing datasets. Wild6D significantly scale up the number of images, object instances, complexity of scenes compared to previous RGBD datasets.
Different from NOCS [48], it consists of collection of short object-centric video clips captured in the real world. However, depth maps provided in Objectron, which might lead ambiguities in pose estimation.
Wild6D Collection. To achieve categorical 6D pose for real objects. collect new large-scale RGBD dataset, named Wild6D. Each video in the Wild6D is recorded via the iPhone front camera showing multiple views of objects where RGB images and the corresponding depth images and point cloud are captured simultaneously. The videos are captured by different turkers with their own iPhones to guarantee the diversity of instances and background scenes. Three videos taken for each object under different scenes. In total, Wild6D consists of 5,166 videos (>1.1 million images) 1722 different object instances and 5 categories, i.e., bottle, bowl, camera, laptop, and mug. Among this data, we split 486 videos of 162 instances to use them as the test set. Table summarizes the statistics of Wild6D comparing previous datasets: Wild6D significantly improves the number of images, object instances, and scene complexity.
Wild6D Annotation. To annotate more than 10,000 images in the testing set efficiently, propose a wracking-based annotation pipeline. Inside a video, we manually annotate the 6D object poses every 50 frames as keyframes. Given the annotation of the keyframe, we implement TEASER+ [51] together with colored ICP [33] to achieve the registration between the keyframe and the following frame and compute the transformation matrix. The ground-truth object pose of the following frame can be obtained by applying the transformation matrix to the keyframe annotation. Following this pipeline, can obtain accurate ground-truths by only labeling around 5 keyframes per video.
Proposed Method
We propose the Rendering for Pose estimation network (RePoNet) using both synthetic data and large-scale unlabeled real-world data in semi-supervised manner.
RePoNet overview. The RePoNet is composed two networks including the Pose Network to directly estimate the object 6D pose parameters (with the NOCS map as intermediate representation) and the Shape Network to reconstruct the object shape. The outputs from both networks can go through a differentiable rendering [28] module which outputs a segmentation mask.
Semi-supervised learning setting, As shown in Fig. we utilize two sets of data for semi-supervised learning: (i) synthetic data with full annotations including NOCS maps. CAD models, foreground segmentation, and 6D pose parameters, denoted as Day and (ii) a sarge-scale real-world RGBD dataset Wild6D with estimated foreground masks (using pre-trained Mask R R-CNN [12]), denoted as Dreat Both the synthetic data and the real-world data are utilized train our RePoNet jointly. For the synthetic data, all the ground-truths are used as supervision signals for the Pose Network and the Shape Network. While for the real-world data, we directly feed the outputs from the Pose Network and the Shape Network into the differentiable rendering module to generate the binary mask.

---

then train the whole RePoNet by comparing the rendered mask with the object foreground mask (silhouette matching loss). While there can be a more delicate way to deal with the domain gap between simulation and real, we find the direct sharing of parameters during training already provides a simple yet effective solution for generalization.
One important contribution of this framework is to make the whole procedure with RePoNet differen- tiable, using the Shape Network, and ConvNet to connect NOCS map to 6D pose parameters in the Pose Network. This allows the gradients to backprop through the network end-to-end. We will explain the architecture and objective details in the following subsections.
4.1
Framework Details
Input data pre-processing. Given RGBD image, we first utilize the off-the-shelf Mask R- CNN [12] model to infer the segmentation mask for each object instance (See supplementary material for segmentation examples). We crop an object of interest from the RGB image and the corresponding depth map. The pixels in the depth map are back-projected as an observed point cloud. We denote the observation of an object instance as X R Hx for RGB channels and E RM for the point clouds, where W, H stands for cropped image resolution and Np is the number of sampled points.
Feature extraction. We employ the PSPNet [56] extract the image feature as fx (Fig. 3 green box), and a PointNet [36] to extract the point cloud geometry feature as fo (Fig. 3 blue box). Meanwhile, for each category, we first define a category-level mesh prior M E N,x3 to represent objects belong specific category, where N, is the number of vertices of the pre-defined mesh. We also use the PointNet to extract the shape prior feature as feate (Fig. 3 purple box).
4.1.1 Pose Network.
Unlike instance-level 6D object pose estimation, where the CAD model of specific instance is always given as the reference, the Pose Network is excepted to work well even without any instance models. To achieve this goal, we leverage the NOCS representation proposed in [48]. The architecture of the Pose Network is shown as the bottom part of Fig. 3. The Pose Network takes the RGB features fx extracted using PSPNet and the point cloud features fp extracted using PointNet as inputs. The two different modalities are combined as RGBD fcatures using the dense fusion approach proposed in [45]. Specifically, the geometry feature fg of the i th point is concatenated with its corresponding color feature f and it is fed into a Graph Convolution Network (GCN) [27] (Fig. 3 yellow box) to obtain the point-wise RGBD feature, denote as figha"
Meanwhile, recall that we utilize a PointNet to obtain the point-wise geometry feature of the category- level shape prior feate (on the Shape Network branch). The shape prior contains the prior knowledge of the category, i.e. i.e.oarse canonical shape and pose, but it does not align with the RGBD feature perfectly. To use the information of categorical shape prior and address the misalignment, we first apply a max-pooling operation on point dimension to obtain global representation, then concatenate with every point-wise RGBD feature, denoted as fúoes: which can be used to estimate the NOCS coordinate. As the size and geometry of real-world object can be quite diverse, we introduce an implicit function Pnocs (-) to deal with objects of the arbitrary shape (Fig. 3 gray box with MLP). Specifically, the (- is implemented as a MLP with the point position and the corresponding feature as inputs, and predicts its NOCS coordinate as,
Dnoes P P) Phocs :i 1...,..., Np-
(1)
where P is the set of input point clouds and N, is the number of point clouds. The output is the NOCS coordinate pl Precs for each point.
The NOCS map also explicitly reflects the geometric shape information of objects, similar propertics also shown with representations in [47]. Thus, the 6D pose, i.e., rotation (R), translation (T), and scale (S) can be inferred from the NOCS map via a simple neural network. In detail, we directly take the RGBD feature figbd and concatenate it with the intermediate results of NOCS coordinates. This concatenated feature fpose will be used to predict the object pose R, T and S by a 3-layer ConvNet (Fig. 3 gray box with CNN). Using neural network to connect the NOCS map and the 6D pose parameters makes the whole process differentiable, which is essential for training with the silhouette matching loss. More details of the network architecture can be found in the supplementary materials.
Inference for 6D pose. For 6D object pose estimation, only the NOCS map output of Pose Network is required for inference. The object pose can be computed by solving the Umeyama algorithm [43]
5

---

Shape Network
vurticss position
Cate: Feat
Object Slapp
Categorical Shape Ptice
PointNet
Dif Bender
Pose
Mask
R,7.5
RGB Feos
PaP
Bequestttiio.
Point-wise MLP
NOCS
Crop Image
Max pooling
Pose Network
Concatemation
Figure 3: Overview of the proposed method. Given the input image and depth map, RePoNet estimates the object pose, NOCS map, and shape simultancously via Pose Network and Shape Network. These two networks are bridged via the differentiable rendering module. By comparing the predicted binary mask with the input foreground mask. RePoNet can effectively leverage the real-world data without any annotations.
with the NOCS map. The 6D pose parameters R, T and only used perform differentiable rendering during training. The reason is that the directly estimated pose parameters from conv module accurate when fitting the training data, but not accurate as using NOCS map when generalizing to novel test instances.
4.1.2 Shape Network.
The Shape Network (top part of Fig. 3) aims to reconstruct the 3D shape of the input object from the given shape prior via mesh deformation. The major usage of this network for performing differentiable rendering to provide training signals. The Shape Network only used to estimate the object scale during inference time.
Given the categorical shape prior representation fente the Shape Network borrows the point-wise RGBD feature
input object fiza from Pose Network and concatenate them together, denoted as féhupe- Specifically, we max-pool the features figha on point dimension and repeat it back with the number of shape prior points for concatenation with feate- We then predict the per-vertices deformation for the input mesh given fshape. The deformation prediction network Pdcform (.) is also an implicit function taking fshape and mesh vertices positions M as the inputs, and outputs the 3D deformation as,
defiom Shmee m' M) m
== 1...N.
where m delta represents the deformation for each vertex, N, is the number of the vertices, and Maelta = (ml d Then the estimated mesh vertices of the given object is Maeform M Maelt-
N. delta j=1'
4.1.3 Differentiable rendering.
To obtain the supervision from 2D segmentation, we perform differentiable rendering using the outputs from the Shape Network and the Pose Network. Specifically, we feed the estimated object shape Maeform and estimated object pose (R, T, S) into differentiable rendering module [28] to generate the binary mask of the given object, as shown in Fig. Given the rendering output binary mask, we can compare it against the Mask R-CNN segmentations and supervise the training of both pose network and shape network without additional 3D annotations.
4.2 Learning Objectives
We define multiple objectives for training the RePoNet. including the loss for 6D pose parameters, NOCS regression loss, shape reconstruction loss using synthetic ground-truths for supervision, and silhouette matching loss which can be applied to both synthetic data and real data without 6D pose annotations. We introduce each loss the following.
Disentangled 6D Pose Loss. Inspired by [24, 47, 39], we implement disentangled pose loss via individually supervising the rotation R, translation T and scale S. Instead of directly computing

---

parametric distances based on rotation matrix, we employ the variant of Point-Matching loss' [24, 20]. Additionally, we decouple the translation into the 2D location (0x,0y) of the 3D centroid projection on the image plane and the object's distance to camera (0x. Oy) can be approximated as the bounding box center of the given object (Cx,Cy): Given the camera intrinsies K, the translation can be calculated via Eq. 3,
T= K-K 1010x. Oy. 11"
(3)
Therefore, given the ground truth pose annotations (R*, T", S*), the objective function of 6D pose is formulated Eq. 4,
Lpose LR + Leenter + L2 + Ls
(4)
Thereby.
LR = avg | R R'xl2 Leenter = -x 0y 0 I11 - Ls = -|S-S1111 (5)
xcMcan
where MCAD is the ground-truth CAD model of the given object and ox 0z. 12 can be computed via Eq. 3.
NOCS Regression Loss. To guarantee the pose estimation accuracy, we enforce the predicted NOCS
coordinates P Procs be closer to the ground truth ones P Pocss We use smooth-LI loss [11].
Luves = Σ 0.5 - /)2/B, /.// y <B
10055xx->)/1
(-5xx-))//BB 11x
(6)
x,y ( 0.5 B. otherwise
where x Pr nocs. y Pnoes
Reconstruction Loss. Given the ground-truth CAD model MCAD, the shape estimation is supervised by minimizing Chamfer Distance(CD) between Maeform and MCAD via Eq. 7,
Lrecon
Σ
min
|Xx
v12
min X
||-->lī
YEMCAD
XEMadrm
YEMCAD
Silhouette Matching Loss. The above three objective functions, Lpese, Lnocs and Lrecon- require the ground-truth information and cannot work with the real-world data without annotations. Thus, we further implement silhouette matching objective function based input foreground segmentation and the rendering results. We employ the negative IOU loss [54] to measure the difference between two masks and denote it Lmask-
Therefore, for the sully-supervised training with annotated synthetic data, the total loss function is described in Eq. 8, where the Ás are balance parameters.
Sup Ause + AdZon + A + 24
(8)
Meanwhile, under the semi-supervised setting with both annotated synthetic data and unlabeled data, the total loss function is then formulated as:
L 1 dDD (d Lpose A2 + AreCono da Lmakk
where LyeD,y is the indicator function.
5
Experiments
5.1
Implementation Details
Semi-supervised setting. We use the training data of CAMERA25 [48] along with the corresponding annotations and jointly train the model with images of REAL275 [48] Wild6D without any 6D pose annotations. After cropping the object from the RGBD image, we first resize it to 192 x 192 and then randomly sample 1,024 points from both color image and depth map. To obtain the categorical shape prior, we choose a CAD model per category from the CAMERA25 training set manually and reduce its number of vertices to 1,024 as well. More configurations of RePoNet have been specified in supplementary materials. We adopt Adam [19] to optimize our model with the initial learning rate of 0.0001. The learning rate is halved every 10 epochs until convergence. We empirically set the balance parameters d, A2, A3 and 24 to 0.2, 2.0, 5.0 and 0.2, respectively.
Fully-supervised setting. We share the same architecture and configurations with semi-supervised training but utilize all annotations of CAMERA25 [48] and REAL275 [48].
We predict the quaternion representation of rotation matrix and follow the strategy in [50, 47] to deal with the symmetry of the object.
7

---

NOCS Implicit
5 degrec
10 degree 5em
degree
10 degree 5cm
10Up.s
1.75
105
0.75
map function
5cm
5cm
Fully
supervised
Semi-superve supervised
77.0
55.7
35.4
63.2
71.7
47.9
22.1
47.5
79.1
59.1
36.9
65.3
74.6
49.9
29.7
57.1
79.2
60.0
39.8
66.6
72.2
47.7
24.3
51.6
81.1
63.7
40.4
68.8
76.0
52.2
33,9
63.0
Table 2:
Ablation Implicit function and intermediate NOCS map. We evaluate on REAL2275 and the hest results highlighted in bold. w./..o the noes map refers whether using the NOCS map intermediate result for pose regression. w./.oo the implicit function stands for whether including the point/mesh coordinates in NOCS regression and shape reconstruction.
silhouette pose loss
degree 5cm
10 degree 5cm
S degree 10 degree
10U
1.75
IOU0.
1OU0.7
loss
Scm
5cm
Fully
supervised
Seml-supervised
78.1
61.9
39.7
65.2
74.8
47.1
27.9
59.7
76.7
58.2
30.7
59.7
73.1
46.0
18.7
47.9
81.1
63.7
40.4
68.8
76.0
52.2
33.9
63.0
Table 3: Ablation study on pose regression loss and silhouette matching loss. We evaluate on REAL275 and the best results are highlighted in bold.
5.2 Ablation Study
Implicit function and NOCS map. We first validate the effectiveness of two components in our approach: implicit function and intermediate NOCS map as described in Sec. 4.1.1. We evaluate the performance under sully-sup setting and semi-supervised setting on REAL275 [48] and report the results in Table 2. Under the fully-supervised setting, by using the implicit function, the performance on IOUn.s and degree, 5cm is improved by 2.2% and 4.4%. respectively. A similar improvement can also be observed for the model with intermediate NOCS map. The best performance is achieved when combining the implicit function and NOCS map together. And the same conclusion can also be obtained under the semi-supervised setting as shown in Table
Pose Loss and Silhouette Matching Loss. Besides the network architecture, we also show the effectiveness of the disentangled 6D pose loss and silhouette matching loss. The performance of the model w./.oo the pose regression loss and w./w.o. the silhouette matching loss are listed in Table 3. Clearly, both pose loss and silhouette matching loss enhance the object pose estimation performance under both fully-supervised and semi-supervised settings. For instance, the silhouette matching loss improves the IOUg.75 and 10 degree 5cm by 5.5% and 9.1% compared with the model without it under the fully-supervised setting. This significant improvement shows the rendering module can effectively encourage communication between two networks, thereby boosting the pose estimation performance.
Leveraging unlabeled real data. Additionally, report the performance of the proposed semi- supervised approach with training data from different sources in Table 4. As the baseline model, we first train it only with the synthetic data from CAMERA25 in fully-supervised manner. Then, by using the unlabeled Wild3D RGBD videos, the performance is significantly improved by 6.8% and 5.4% IOUp,75 and degree, 2cm. Similarly, the performance gains can be observed when adopting the REAL275 data without its annotations. Furthermore, with both REAL275 and Wild3D data, model outperforms the baseline model by a large margin and approaches the fully-supervised model trained on REAL.255 which can be served as the upper bound. More ablation studies about the amount of unlabeled real data used can be found in the supplementary materials.
Amount of unlabeled real data. We analyze the effect of using different fractions of unlabeled real data used during semi-supervised learning. We uniformly sample every 10% fraction of collected Wild6D training data for semi-supervise learning and evaluate the performance on REAL275 and Wild6D testing sets. As shown in Fig. 4, with more real data used during training, the object pose estimation performance is getting better.
5.3 Comparison with State-of-the-art Methods
Performance on REAL225. We split all existing methods into two groups by whether using full annotations of real data and compare their performance with RePoNet under both fully-supervised setting and supervied setting, denoted as "RePoNet-sup" and "RePoNet-semi" respectively. As listed in Table 5, RePoNet-sup can achieve competitive results among all existing approaches under
8

---

Training Data
5 degree 5 degree 10 degree
10 degree
Methods
10
10U
1075
CAMERAZA REAL.275 Wild6D
2em
5em
2cm
5cm
83.8
73.5
45.7
18.2
21.8
35.6
47.9
85.0
74.3
52.5
23.6
26.1
45.2
53.8
RePoNet-semi
85.8
76.9
49.2
29.1
31.3
48.5
56.8
86.0
78.5
51.8
30.8
36.0
48.7
59.7
RePoNet-sup
86.6
83.0
63.22
31.1
37.0
51.6
64.6
Table 4: Ablation study of proposed semi-supervised method with different training data. We evaluate the performance on REAL277 test set. "RePoNet-seemi and "RePoNet-sup" represent for the model trained w./w.o the 3D annotations of NOCS REAL225 respectively. Here we conduct experiments five categories sharing by hoth REAL.275 and Wild6D, be.bottle, bowl, mug, laptop and camera.
5 degree 2cm
degree 5em
10 degree
Methods
1OU
5cm
NOCS [48]
78.0
7.2
10.0
25.2
CASS [3]
77.7
23,5
58.0
Shape-Prion Phape-PriorPie [41]
77.3
19.3
21.4
54.1
FS-Net [6]
92.2
28.2
60.8
DualPoseNet [25]
79.8
29.3
35.9
SGPA [4]
80.1
35.9
39.6
70.7
GPV-Pose [10]
83.0
32.0
42.9
73.3
RePoNet-sup
81.1
35.1
40.4
68.8
CPS++ [29] w/ ICP
72.8
S 58.6
SSC-6D [35] w/ ICP
72.7
28.6
33.4
62.9
CPPF [53]
26.4
16.9
44.9
UDA-COPE [21]
82.6
30.4
34.8
66.0
RePoNet-semi
76.0
30.7
33.9
63.0
Table 5: Comparison of our approach with the SOTA methods on REAL275. Note that all SOTA methods listed in the top part are trained with full annotations of NOCS dataset including CAMERAZ. and REAL1275, while the bottom part is methods without using any real annotations. The best results are highlighted in bold.
the sully-sd setting. Note our method is complimentary to the techniques proposed in SGPA, and our key contribution and focus lies on the semi-supervised counterpart for generalization. As shown in the bottom part of Table 5, the RePoNet-semi outperforms CPS++ [29] by large margin under the supervipe setting 2. Moreover, compared with the existing fully-supervised methods". RePoNet-semi achieves a comparable or even better performance without any 6D pose annotations. Additionally, our approach can perform shape reconstruction, but it's our goal and we report its performance in the supplementary material.
5 degree15 degree 10 degree
Methods
IOU
Performance Wild6D. Finally, evaluate the performance of RePoNet on the Wild6D test- ing set as reported in Table 6. For some existing works, we directly test their pre-trained models trained on CAMERA75 along with REAL227. Since FS-Net [6] does not release the model, cannot experiment it. It can be observed that the pre-trained models cannot generalize to Wild6D due to the limited diversity of real data during training. For example, the performance of Shape-Priee Prior [41] on three pose estimation metrics are all lower than 15%. On the other hand, the RePoNet with semi-supervised learn- ing on Wild6D achieves 29.5% and 34.4% on 5 degree, 2cm and 5 degree, 5cm, which is 10x better than Shape-Prior [41]. This significant improvement shows the better generalization ability of RePoNet. Also, the superior performance of RePoNet-semi over RePoNet-syn shows our proposed semi-supervised training can effectively leverage the in-the-wil data. We also show the qualitative results on Wild6D test set in Fig. 5.
2cm
5em
5em
CASS [3]
1.04
0
0
Shape-Prior [41]
32,5
2.6
3.5
13.9
DualPoseNet [25]
70.0
17.8
22.8
36.5
GPV-Pose [10]
67.8
14.1
21.5
41.1
RePoNet-syn
66.7
26.0
30.8
40.3
RePoNet-semi
70.3
29.5
34.4
42.5
Table 6: Comparison of our approach with the SOTA methods on Wild6D. "RePoNet-syn" the model trained only on CAMERA75, while "RePoNet-semi" stands for the RePoNet trained CAMERA75 and Wild6D. The best results are highlighed bold.
Pn Table 5, CPS++ only reports the performance under 10 degree, 10 em. 3FS-Net Net and GPV-Pose treated as the supervised one, since it requires all real data and annotations.

---

Pose Estimation performance on REAL275
Pose: Estimation performance on WildeD
50
Iougo 75
Sdegrec. 2cm
20
10 degice. Zcm
30
10ugo 75
Sdegroe
20
degree.
00
0.2
0.4
0.6
08
10
00
0.2
0.4
06
0.0 1.0
The percentage of unlableded data during training
The percentage unlableded data during training
Figure 4: Ablation study semi-supervised training with different number of unlabeled real data. Here, we only show the performance on bottle and evaluate both REAL275 and Wild6D dataset.
ALL
NTTTT
Figure 5: Visualization Results on Wild6D test set. Red 3D bounding boxes denote the ground truth, and the green boxes are estimation results via proposed method.
6 Discussion
Conclusion. In this work, we consider the problem of generalizing 6D pose estimation to in-the- wild objects. Most of existing work are restricted in constrained environments due to limited number of annotated data. In effort resolve this limitation, collect Wild6D, with diverse instances and backgrounds. Instead of annotating them, we also propose RePoNet, a network that leverage both the synthetic data and real-world RGBD data. Without using any 3DD annotations on real-world data, RePoNet outperforms ttate-of-the-ar methods on existing datasets and Wild6D test set by large margin.
RGBD dataset
Limitations. Our proposed RePoNet may be hard to generalize to unseen categories. This because RePoNet depends the categorical mesh prior which is pre-defined for each category leading to the learnt deformation model fixed to a specific category. Alternative way to address this problem is to represent object shape via deformation of unit sphere to remove the category-level prior. Our dataset is currently limited to 5 categories to align with previous work for evaluation. We will extend it more categories in the future.
Acknowledgement. This work was supported, in part, by grants from DARPA LwLL, NSF CCF- 12666 (TILOS), NSF 1730158 CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CD, NSF ACI-1541349 D Pacific Research Platform, and gifts from Meta, Google, Qualcomm.
10

---

References
[1] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann, Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7822-7831, 2021. 3, 4
[2] Frank Biocca. Virtual reality technology: A tutorial. Journal communication, 42(())23--72 72, 1992.
[3] Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11973- 11982, 2020. 1, 1,9
[4] Kai Chen and Qi Dou. Sgpa: crtucture-guideee prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2773 2782, 2021. 9
[5] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, and Ales Leonardis. G2l-net: Global to local network for real-time 6d pose estimation with embedding vector features. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4233-4244 2020.
[6] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape-based network for category-lewe level 6d object pose estimation with decoupled rotation mechanism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1581-1590.
2021. 1, 3.9
[7] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning predict 3d objects with interpolation-based differentiable renderer. Advances in Neural Information Processing Systems. 32:9609- 1919,99999999.111111119991111999 2019. 3
[8] Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural analysis-by-symhheiss. In European Conference on Computer Vision, pages 139-156. Springer, 2020. 3
[9] Sungjoon Choi, Qian-Y Yi Zhou, Stephen Miller, and Vladlen Koltun. A large dataset of object scans. 160iv 2016.
[10] Yan Di. Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-Ievel object pose estimation via geometry-guide point-wise voting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6781-67911 2022. 9
[11] Ross Girshick. Fast r-cnn. In Proceedings of the IEFE international conference on computer vision, pages 1440-1448, 2015.
[12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pages 2961-22222222 1669,,96,,00 2017. 5.
[13] Yisheng He. Wei Sun, Haibin Huang, Jianran Liu, Haoqiang Fan, and Jian Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and patlem recognition, pages 1 11632-11641, 2020. 1
[14] Yisheng Hc, Haibin Huang, Hanqiang Fan, Qifeng Chen, and Jian Sun. Ffb6d: A full flow bidirectional fusion network for 6d pose estimation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
[15] Tingbo Hou, Adel Ahmadyan, Liangkai Zhang, Jianing Wei, and Matthias Grundmann. Mobilepose: Real- time pose estimation for unseen objects with weak shape supervision. arXiv preprint arXiv:2003.03522. 2020. 3
[16] Shun Iwase, Xingyu Liu, Rawal Khirodkar, Rio Yokota, and Kris M Kitani. Repose: Fast 6d object pose refinement via deep texture rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3303 3312, 2021. 3
[17] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specifi mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pages 371-386, 2018.
[18] Hiroharu Kato and Tatsuya Harada. Learning view priors for single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9778-9787, 2019. 3
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6 69800, 2014. 7
11

---

[20] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. Cosypose: Consistent multi-view multi- object 6d pose estimation. In European Conference on Computer Vision, pages 574-591. Springer, 2020.
3.
[21] Taeyeop Lee, Byeong-Uk Uk Lee, Inkyu Shin, Jaesung Choe, Ukcheol Shin, In Kweon, and Kuk-Jin Yoon. Uda-cope: Unsupervised domain adaptation for category-level object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14891-14900, 2022. 9
[22] Chi Li, Jin Bai, and Gregory D Hager. A unified framework for multi-view multi-class object pose estimation. In Proceedings the european conference computer vixion (eccr), pages 254-269, 2018. I
[23] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang. and Jan Kautz. Self-supervised single-view reconstruction via semantic consistency. In European Conference on Computer Vision, pages 677-693. Springer, 2020. 3
[24] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 683-698, 2018. 7
[25] Jiehong Lin, Zewei Wei, Zhihao Li, Songeen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category-level object pose and size estimation using dual pose network with refined learning of pose consistency. arXiv preprint a/Xiv:2103.06526, 2021. 3,
[26] Yunzhi Lin, Jonathan Tremblay, Stephen Tyree, Patricio A Vela, and Stan Birchfield. Single-stage keypoint- based category-level object pose estimation from an rgb image. arXiv preprint arXiv:2109.061 2021.
[27] Zhi-Hao Lin, Sheng- Yu Huang, and Yu-Chiang Frank Wang. Convolution in the cloud: Learning de- formable kernels 3d graph convolution networks for point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1800-1809, 2020. 5
[28] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based
3d reasoning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7708-7717, 2019. 3,4,6
[29] Fabian Manhardt, Gu Wang. Benjamin Busam, Manuel Nickel, Sven Meier, Luca Minciullo, Xiangyang Ji, and Nassir Navah. Cps++: Improving class-level pose and shape estimation from monocular images with self-supervised learning. arXiv preprint arXiv:2003.05833 2020. 1, 3,
[30] Eitan Marder-Epppseein Project tango. In ACM SIGGRAPH 2016 Real-Time Livel, pages 25-25. 2016.
[31] Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF International Conference Computer Vision,
pages 2901-2910, 2019. 1
[32] Markus Oberweger, Mahdi Rad, and Vincent Lepetit. Making deep heatmaps robust to partial occlusions for 3d object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 119-134, 2018.
[33] Jaesik Park, Qian Yi Zhou, and Vladlen Koltun. Colored point cloud registration revisited. In Proceedings of the IEFE international conference computer vision, pages 143-152, 2017. 4
[34] Sida Peng. Yuan Liu, Qixing Huang. Xiaowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4561-452 3561-457,, 2019.
[35] Wanli Peng, Jianhang Yan, Hongtao Wen, and Yi Sun. Self-supervised category-level 6d object pose estimation with deep implicit shape representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2082-2090, 2022. 9
[36] Charles Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-666266 660, 2017. 5
Deep learning point sets for 3d
[37] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real -ife 3d category reconstruction. In International Conference Computer Vision, 2021. 3
12

---

[38] Johannes L Schonberger and Jan-Michael Frahm. Giucture-from-mmotiee revisited. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 4104-4113, 2016 3
[39] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel Antez-Anteuuera, and Peter Kontschieder. Disentangling monocular 3d object detection. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1991-1999, 2019.
[40] Juil Sock, Guillermo Garcia-llernando, Anil Armagan, and Tae- Kye-Kyun Kim. Introducing pose consistency and carp-alignment for self-supervised 6d object pose estimation in color images. In 2020 International Conference on 3D Vision (3DV), pages 291-300. IEEE, 2020. 3
[41] Meng Tian, Marcelo H Ang, and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In European Conference Computer Vision, pages 530-546. Springer, 2020. 1, 3,
[42] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of houschold objects. arXiv preprint 00t 2018.
[43] Shinji Umeyama. Least-squares estimation of transformation parameters hetween two point patters.. IEEE Transactions on Puttern Analysis & Machine Intelligence, 13(04))376-380, 1991. 3,
[44] Angtian Wang, Adam Kortylewski, and Alan Yuille. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. arXiv preprint arXiv:2101.1233 2021.
[45] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin, Cewu Lu, Fei-Fei, and Silvio Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE/CVF conference compuder vision and pattern recognition, pages 3343-3352, 2019. 5
[46] Gu Wang, Fabian Manhardt, Jianzhun Shao. Xiangyang Ji, Nassir Navab, and Federico Tombari. Self6d: Self-supervised monocular 6d object pose estimation. In European Conference on Computer Vision, pages 108-125. Springer, 2020. 3
[47] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16611-16621, 2021. 5, 6,
[48] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Lconidas J Guibas. Normal- ized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 2642-26511 2019. 2, 4, 5, 7,8.9
[49] Bowen Wen, Chaitanya Mitash, Sruthi Soorian, Andrew Kimmel, Avishai Sintov, and Kostas E Bekris. Robust, occlusion-aware pose estimation for objects grasped by adaptive hands. In 2020 IEEF International Conference on Robotics and Automation (ICRA), pages 62 6210-6217. IEFF, 2020.
[50] Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint 10t.1111..00999 2017. 1,7
[51] Heng Yang, Jingnan Shi, and Luca Carlone. Teaser: Fast and certifiahle point cloud registration. JEFE Transactions on Robotics, 37(2):314-333 2020.
[52] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. arXiv preprint arXi::0012.0587, 2020. 3
[53] Yang You, Ruoxi Shi, Weiming Wang, and Cewu Lu. Cppf: Towards robust category-level 9d pose estimation in the wild. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6866-68161 2022.
[54] Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang. Unitbox: An advanced object detection network. In Proceedings of the 24th ACM international conference on Multimedia, pages 516 (66-520,77 2016.7 7
[55] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilie. Dpod: 6d pose object detector and refiner. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1941-1950, 2019. 3
[56] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2800 2017. 5
13

---

[57] Menglong Zhu, Konstantinos G Derpanis. Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang. Cody Phillips. Matthieu Lecce, and Kostas Danillddis. Single image 3d object delection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3936-3943. IEEE,
2014.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplementary
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [Yes] See supplementary
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "w3aDelysmNm", "reviewer": "Reviewer_rj61", "summary": "This dataset-and-method paper proposed a new unlabeled large-scale dataset for category 6D pose estimation in the wild, along with a new network design that can utilize that dataset in a weakly supervised manner. Most notably, the weakly supervision branch is on top of the desired shape and pose branches, thus allowing training both even without or with only a sporadic strong supervision signal. The proposed method is evaluated on existing datasets and existing methods are evaluated on the new dataset, showing that the proposed method exceeds state of the art.", "questions": "(1) Since iPhones were used, presumably with an own app to also capture depth, the rather accurate IMU could have been used to track the phone's movement (compared to the data-dependend models TEASER++/ICP, l.145).\n\n(2) What is the estimated accuracy of the annotated poses, based on the errors introduced in labelling and tracking?\n\n(3) Is the instance segmentation network pre-trained (and if yes, how / on which data), or trained along with the rest of the network in an end-to-end manner? For the latter case, I'd expect it to be rather unstable in the beginning of the training.", "limitations": "Yes, limitations and impact were appropriately addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is well written and good to follow. All steps are well motivated and described, the network can likely be reproduced based on the detailed description and well-done figures.\n\n+ The dataset acquisition is well thought off. Using \"Turkers\" to take videos with their own phones - compared to scientists - is very likely to have a reduced bias regarding scene selection, acquisiton etc.\n\n+ The dataset is a significant improvement over prior datasets regarding size and variety. However, it is not fully annotated, thus potentially limiting its long-term impact.\n\n+ The semi-supervised loss is well thought off, especially that it is based on the 6DoF pose which is the actual target (compared to a parallel branch / head in the network).\n\n+ The proposed network and training protocol is well thought off. It is based on well established building blocks and combines them in a way that allows weakly supervised training. This allows it to efficiently leverage a large amount of unlabeled or weakly annotated real data.\n\n+ The experiments are convincing and exhaustive. The proposed method is validated against prior art based on established datasets. Ablation studies show the impact of different building blocks. Some prior art methods are evaluated on the proposed dataset, forming a baseline evaluation.\n\n+ The results are very good compared to prior art.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fap1Js30L8l", "reviewer": "Reviewer_1VmD", "summary": "The paper proposes a method that utilizing unlabeled data for training to enable the proposed category-level 6Dpose estimation algorithm generalised to new scenarios. A new dataset is collected for implementing the idea. Comparisonal experiments were conducted on the REAL275 dataset and the proposed Wild6D dataset. The proposed algorithm has certain advantages over the listed algorithms. \n", "questions": "please check the strengths and weaknesses.", "limitations": "I didn't see any negative societal impact bring by this paper.", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The new method is able to utlizing the unlabeled real-scene data.\n\n2. The authors give a large RGB-D real dataset and annotate the testing dataset, which may be useful for certain tasks.\n\n3. The authors promise to make the code publicly available\n\nWeaknesses and Questions:\n\n1. The value of IOU-0.5 of CPS++ reported in Table 5 is way much lower than the value reported in the paper [1], please explain this inconsistency.\n\n2. The author simply compared the performance of RePoNet with CPS++ when presenting the effectiveness of RePoNet on processing unlabeled dataset. While many SOTA unsupervised algorithms, for example, UDA-COPE [1], behaves better than RePoNet on the REAL275 dataset without any real annotations. It is necessary to compare more relevant SOTA methods for a fair comparison.\n\n3. In Table 6, the CASS, Shape-Prior, DualPoseNet are all trained on the CAMERA75+REAL275, while RePoNet-syn and RePoNet-semi are trained either on CAMERA75 or CAMERA75+Wild6D. The author validated all these algorithms on the Wild6D and then emphasize the superiority of RePoNet. The lack of consistency on training dataset makes this comparison, again, unfair and invalid.\n\n4. For line 327, the authors stated that 'FS-Net does not release the model, we cannot experiment on it.' I am not fully convinced by the explanation. Although the FS-Net doesn't release the pre-trained model, the code is available on GitHub (https://github.com/DC1991/FS_Net). Given the prominent performance of FS-Net on IOU-0.5, it is worth to try, which I believe it is also possible, to experiment with FS-Net on Wild6D dataset based on the published code. \n\n[1] 'UDA-COPE: Unsupervised Domain Adaptation for Category-level Object Pose Estimation', CVPR2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z4YMP-01Rvj", "reviewer": "Reviewer_wqvk", "summary": "The paper proposes a new dataset for 6D pose estimation and a method named RePoNet, for estimating 6D object pose and size. The method falls in the category-level approach that predicts an object pose only based on its category. \nIn particular, the presented method is defined by two main branches a Pose Network and a Shape Network. The Pose Network uses the NOCS map as an intermediate representation to get the object pose, and the Shape Network uses the specific category 3D shape prior to leveraging the intra-class variation with the object mask.\n\nThe new dataset, named Wild6D, collects 5166 videos for five object categories under multiple views. \nThe method utilises the NOCS synthetic dataset, with all 6Dpose annotations, and the real dataset together with the proposed Wild6D to obtain masks and corresponding depth maps for training. The authors perform a detailed ablation analysis of the method on the datasets used. Results show that the proposed method outperforms the considered competing methods. Comparisons with state-of-the-art follow the metrics of [44].\n", "questions": "Please explain the implicit functions \\Psi_{nocs}, and what is used to train the MLP.  \nPlease explain \\Psi_{deform}, how the accuracy of \\Psi_{deform} is evaluated and the loss used. \n\nEven if no qualitative results on the shape are provided,  line 320 reads ‘our approach can perform shape reconstruction’.  \nHow relevant is the shape to generate the object bounding box? Why it is not used in inference (lines 224-225).\n\nIf the shape network is not used in inference, is the mask required? How is it used?\nPlease provide the parameters of the whole network and possibly runtime.\n\nPlease report correctly the comparisons with the semi-supervised methods using only real data, such as FS-Net.\n", "limitations": "Limitations are discussed, in particular, the difficulty to generalize. ", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper poses the relevant question of mitigating the annotation effort in category-level 6D pose estimation. The authors propose to exploit a common strategy in many fields, based on training on synthetic images and resolving the domain gap with real images.\nThe authors refer to the NOCS dataset [44] (Camera25, with 300K  synthetic images, and Real275, with 8K real images). At the same time, the authors' contribution is to use only synthetic images from CAMERA25, with full annotations including  CAD models, segmentation and 6D pose parameters,  and taking from Real275 only the RGBD information (depth map and masks). Due to the domain gap between synthetic and real images, the authors claim that the method is semi-supervised.\n\nThe authors also contribute with a new annotated dataset, Wild6D, built from videos recorded with iPhone and a depth sensor. The Wild6D is annotated with 6D object poses every 50 frames, and the intermediate ones follow a tracking strategy.\nThe strength of the paper is in the competitive results on the 6D pose obtained by the semi-supervised method.   \n\nMajor weaknesses of the paper are:\n1. presentation.\n2. The motivation of using only the synthetic parts with all the annotations.\n3. A lack of comparison with most recent works and a discussion on them.\n\nThe description of the architecture is quite hard to follow as there are a huge amount of component networks, introduced via references, not always precise,  as in lines 207-209 ([43] does not even mention the NOCS, and it refers to a dense correspondence map, see also \"Cdpn: Coordinates-based disentangled pose network for real-time RGB-based 6-dof object pose estimation\").\n\nThe main difficulty in category-level 6D pose estimation is the intra-class object variation. As in previous work, the paper uses a 'shape prior' to face the problem. Adapting the shape-prior to the current object refers to a deformation network (supposedly as in [37]). No details on the accuracy of the deformation network and the loss used are provided, nor qualitative results.\n\nThe whole architecture, collecting several modules and networks, lacks substantial motivation.\nIndeed,  the two losses,  the one for fully supervised training and the one for semi-supervised training, differ only in the data they collect. The balance parameters are the same, too, implying that there is no specificity in the semi-supervised learning for pose-sensitive features and geometric relationships.\n\nThe choice of training with the fully annotated synthetic images seems to be necessary for dealing with the prior shape. The motivation that synthetic data are at no cost is not convincing.\n\nRecent works use only real data, following a different semi-supervised pattern to minimise the amount of training data. \nFor example, \"GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Voting \"(CVPR2022) uses just real data for training and uses for evaluation not only CAMERA25 and REAL275 but also LineMod (not considered by the paper). FS-Net [6] also uses only 'real data' for training, despite Table 5  treating it as fully supervised.  \n\nOn the other hand, there are recent works relying only on synthetic datasets such as \"CPPF: Towards Robust Category-Level 9D Pose Estimation in the Wild\" (CVPR2022), \"Self-Supervised Category-Level 6D Object Pose Estimation with Deep Implicit Shape Representation\" (AAAI2022), \"Category level object pose estimation via neural analysis-by-synthesis\" (ECCV202), which the authors do not consider.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "P-ACXnBn5HT", "reviewer": "Reviewer_gzai", "summary": "This paper presents a dataset and semi-supervised method for 6DoF object pose estimation with the aim of generalizing better. The first contribution is a dataset called Wild6D which is a dataset with 5166 RGBD videos of 1722 object instances from 5 categories. The second contribution is a neural network RePoNet that is trained to estimate 6DoF pose by training on both synthetic data (with full supervision) and real data (partial supervision). Experimental results indicate that the proposed method outperforms other methods on the 6DoF pose estimation task.", "questions": "It would be nice to see a discussion of the following points (in addition to the above limitations) in the rebuttal:\n\n- How does Mask R-CNN quality affect the performance of the differentiable renderer?\n- Could more categories be trained using the proposed approach?", "limitations": "The paper includes a short limitations section. It would have been nice to include a discussion of societal impacts -- this is missing currently.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper has a nice contribution and I would recommend acceptance. In the following, I will list strengths, weaknesses, and other questions.\n\n## Strengths\n\n- The paper addresses an important problem of great practical importance. Generalizing 6DoF pose estimation is one of the grand challenges in 3D vision.\n- The dataset contribution is important and can help make rapid progress in the community.\n- The results are better than other state-of-the-art approaches.\n- The idea of using full supervision from synthetic data and partial supervision from real data is obvious, but surprisingly no other paper has tried it as far as I am aware.\n\n## Weaknesses\n\n- Although the proposed dataset has lots of instances, it is limited to 5 categories. I presume this is to keep it consistent with the NOCS synthetic dataset. However, this limitation should be mentioned, in my view.\n- While the architecture proposed in Figure 3 makes sense, there is not much justification for *why* this particular architecture was chosen over alternatives. For instance, why PSPNet was chosen? What is the rationale for combining RGB and geometric features in that way?\n- The description of the Shape Network is missing details. It's not clear how the categorical prior is implemented. For instance, does it take a random vector as input?\n- The results from Mask R-CNN can often be noisy, especially at the boundaries. It's not clear how this influences the result. It would have been nice to have a discussion on this.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept