MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
Authors: Yann Labbé
Lucas Manuelli
Arsalan Mousavian
Stephen Tyree
Stan Birchfield
Jonathan Tremblay
Justin Carpentier
Mathieu Aubry
Dieter Fox
Josef Sivic
Venue: CoRL 2022 Poster (2022)
Keywords: Object Pose Estimation
Robot Manipulation
Computer Vision

Abstract: We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.

MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
Yann Labbé" !# Stan Birchfield 2
Lucas Manuelli
Arsalan Mousavian 2
Stephen Tyree" 2
Jonathan Tremblay? 2
Justin Carpentier' Josef Sivie 5
Mathieu Aubry?
Dieter Fox 24
ENS/Inria
NVIDIA
*LIGM/ENPC
University of Washington CHRC
megaposeed..itthut
Abstract: We introduce MegaPose, method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present 6D pose refiner based on a render & compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided inputs the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large scale synthetic dataset of pho- torcalistic images of thousands of objects with diverse visual and shape properties, and show that this diversity is crucial obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it withoul retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-an performance on the ModelNet and YCB- Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page [1].
1 Introduction
Accurate 6D object pose estimation is essential for many robotic and augmented reality applications. Current tate-of-the-art methods are learning-basee [2, 3, 4, 5, 6) and require 3D models of the objects of interest at both training and test time. These methods require hours (or days) to generate synthetic data for each object and train the pose estimation model. They thus cannot be used in the context of robotic applications where the objects are only known during inference (e.g. CAD models are provided by a manufacturer or reconstructed [7]), and where rapid deployment to novel scenes and objects is key.
The goal of this work is to estimate the 6D pose of novel objects, i.e., objects that are only available at inference time and are not known in advance during training. This problem presents the challenge of generalizing to the large variability in shape, texture, lighting conditions, and severe occlusions that can be encountered real-world applications. Some prior works [8, 9, 10, 11, 12, 13, 14] have considered category-level pose estimation to partially address the challenge of novel objects by developing methods that can generalize to novel object instances of a known class (e.g. mugs or shoes). These methods however do not generalize to object instances outside of training categories. Other methods aim at generalizing to any novel instances regardless of their category [15, 16, 17, 18, 19, 20, 21, 22]. These works present important technical limitations. They rely non-learning based components
Inria Paris and Département d' d'informatique de I'ENS, École normale supérieure, CNRS, PSL Research University, 75005 Paris, France.
LLIGM, École des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-vallée, France.
"Czech Institute of Informatics, Robotics and Cybernetics the Czech Technical University in Prague.
'Wor partially done while the author was an intern at NVIDIA.
6th Conference on Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

(a) Training
(b) Inference
(c) Visually guided robotic manipulation
Inpuss
Oueout
New CAD model
Rogion of Intorest
BD Pase
modela
Syntheso Images
Figure 1: MegaPose is a 6D pose estimation approach (a) that is trained on millions of synthetic scenes with
thousands of different objects and (b) can be applied without re-truiming to estimate the pose of any novel object, given a CAD model and a region of interest displaying the object. It can thus be used to rapidly deploy visually guided robotic manipulation systems in novel scenes containing novel objects (c).
for generating pose hypotheses [21] (e.g. PPF [23]), for pose refinement [17] (e.g. PnP [24] and
ICP [25, 26]), for computing photometric errors in pixel space [15], or for estimating the object depth [18, 16] (e.g. using only the size of 2D detection [27]). These components however inherently cannot benefit from being trained large amount of data to gain robustness with respect noise, occlusions,
or object variability. Learning-baat based methods also have the potential to improve as the quality and size of the datasets improve.
Pipelines for 6D pose estimation of known (not novel) objects that consist of multiple learned stages [4, 5] have shown excellent performance on several benchmarks [2] with various illumination conditions, textureless objects, cluttered scenes and high levels of occlusions. We take inspiration from [4, 5] which split the problem into three parts: (i) 2D object detection, (ii) coarse pose estimation, and (iii) iterative refinement via render & compare. We aim at extending this approach to novel objects unseen at training time. The detection of novel objects has been addressed by prior works [17, 28, 29, 30] and is outside the scope of this paper. In this work, we focus on the coarse and refinement networks for 6D pose estimation. Extending the paradigm from [4] presents three major challenges. First, the pose of an object depends heavily on both its visual appearance and choice of coordinate system (defined in the CAD model of the object). In existing refinement networks based render & compare [20, 4], this information is encoded in the network weights during training, leading poor generalization results when tested on novel objects. Second, direct regression methods for coarse pose estimation are trained with specific losses for symmetric objects [4], requiring that object symmetries be known in advance. Finally, the diversity of shape and visual properties of the objects that can be encountered in real- world applications immense. Generalizing to novel objects requires robustness to properties such as object symmetries, variability of object shape, and object textures (or absence of).
Contributions. We address these challenges and propose method for estimating the pose of any novel object in a single RGB or RGB-D image, as illustrated in Figure 1. First, we propose a novel approach for 6D pose refinement based on render & compare which enables generalization to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we propose a novel method for coarse pose estimation which does not require knowledge of the object symmetries during training. The coarse pose estimation is formulated as classification problem where compare renderings of random pose hypotheses with observed image, and predict whether the pose can be corrected by the refiner. Finally, we leverage the availability of large-scale 3D model datasets to generate a highly diverse synthetic dataset consisting of 2 million photorealistic [31] images depicting over 20K models in physically plausible configurations. The code, dataset and trained models are available on the project page [1].
We show that our novel-object pose estimation method trained on our large-scale synthetic dataset achieves state-of-the ootte--f-the-arr performance on ModelNet [32, 20]. We also perform extensive evaluation of the approach on hundreds of novel objects from all 7 core datasets of the BOP challenge [2) and demonstrate that our approach achieves performance competitive with existing approaches that require access to the target objects during training.

---

2
Related work
In this section, we first review the literature on 6D pose estimation of known rigid objects. We then focus on the practical scenario similar to ours where the objects are not known prior to training.
6D pose estimation of known objects. Estimating the 6D pose of rigid objects is fundamental computer vision problem (33,34, 35] that was first addressed using correspondences established with locally invariant features 355,6,37, 38,,8,,2 23] or template matching [39, 40J. These have been replaced by learning-ber based methods with convolutional neural networks that directly regress sets of sparse [41, 42, 27,4 43, 44, 45, 46] or dense [47, (47,48, 49, 50, 3, 44] features. All these approaches use non-learning stages relying on PnP+Ransac [51, 24] 2o recover the pose from correspondences in RGB images, or variants of the iterative closest point algorithm, ICP [25, 26], when depth available. The best performing methods rely on trainable refinement networks [52, 20, 4, 20, 5] based on render & compare [53, 54, 55, 20]. These methods render single image of the object, which is not sufficient to provide complete information the shape and coordinate system of 3D model to the network. This information is thus encoded in the networks weights when training, which leads to poor generalization when tested on novel objects unseen at training. Our approach renders multiple views of an object to provide this 3D information, making the trained network independent of these object-specific properties.
6D pose estimation of novel objects. Other works consider practical scenario where the objects are not known in advance. Category- level 6D pose estimation is a popular problem [8, 9, 10. 11, 12, 13, 14] in which CAD models of test objects are not known, but the objects are assumed to belong to known category. These methods rely on object properties that are common within categories (e.g. handle of a mug) to define and estimate the object pose, and thus cannot generalize to novel categories. Our method requires the 3D model of the novel object instance to be known during inference, but does not rely on any category-level information. Other works address a scenario similar to ours. [56, 19, 18, 16, 30] only estimate the 3D orientation of novel objects by comparing rendered pose hypotheses with the observed image using features extracted by network. They rely handcrafted [18, 16] or learning- based DeepIM [19] refiners to recover accurate 6D poses. We instead propose method that estimates the full 6D pose of the object and show our refinement network significantly outperforms DeepIM [20] when tested on novel object instances. The closest works to ours are OSOP [17] and ZePHyR [21]. OSOP focuses on the coarse estimation by explicitly predicting 2D-2D correspondences between a single ren- dered view of the object and the observed image, and solves for the pose using PnP or Kabsch [25] which makes inference slower and less robust compared to directly predicting refinement transforms with a net- work done in solution. ZePHyR [21] strongly relies the depth modality, whereas our approach can also be used in RGB-only images. Finally, [15, 58, 22, 59] investigate using set of real reference views of the novel object instead of using a CAD model. These approaches have only reported results on datasets with limited or no occlusions. Our use of a deep render & compare network trained on a large- scale synthetic dataset displaying highly occluded object instances enables us to handle highly cluttered scenes with high occlusions like in the LineMOD Occlusion, HomebrewedDB or T-LESS datasets.
3 Method
In this section we present our framework for pose estimation of novel objects. Our goal is to detect the pose Tco (the pose of object frame O expressed in camera frame C composed of 3D rotation and 3D translation) of a novel object given an input RGB (or RGBD) image, Lo, and a 3D model of the object. Similar to DeepIM [20] and CosyPose [4], our method consists of three components (1) object detection, (2) coarse pose estimation and (3) pose refinement. Compared to these works, our proposed method enables generalization to novel objects not seen during training, requiring novel approaches for the coarse model, the refiner and the training data. Our approach can accept either RGB or RGBD inputs, if depth is available the RGB and D images are concatenated before being passed into the network. Detection of novel-objects in an image an interesting problem that has been addressed in prior work [28, 60, 17, 22, 30] but lies outside the scope of this paper. Thus for our experiments we assume access to an object detector, but emphasize that our method can be coupled with any object detector, including shot methods such as those in [28, 60).
3

---

Coarse Estimator
Refiner
Input Crop
Coarse Pose Estimate
Input Crop
Refined Pose Estimate
Tco
+
&
Multiview Renderingg of Input Pose
Toua
Refiner Model
Random Pose Hypotheses
Too,a
Teo
Tooo
Too.s
Teoa
Toai
argmax E
Teas
Tco
fM
Tesa
Multiple Views of Input Pose
Coarse Model
(a)
(b)
Figure 2:
@ denotes concatenation. (a) Coarse Estimator: Given a cropped input image the coarse module
renders object multiple input poses {Too). ). The coarse network then classifies which rendered image best matches the observed image. (b) Refiner: Given initial pose estimate Teo the refiner renders the objects
the estimated pose Teo, = Teo (blue axes) along with additional viewpoints {Teo.s } J=2 (green axes) defined
such that the camera z-axis intersects the anchor point O. The refiner network consumes the concatenation of the observed and rendered images and predicts updated pose estimate T
3.1 Technical Approach
Coarse pose estimation. pose estimator is to provide an initial pose estimate Tco,coare which is sufficiently accurate that can then be further improved by the refiner. In order to generalize to novel-objects we propose novel classification based approach that compares observed and rendered images of the object in a variety of poses and selects the rendered image whose object pose best matches the observed object pose.
Given an object detection, shown in Figure 1(b), the goal of the coarse
Figure 2(a) gives an overview of the coarse model. At inference time the network consumes the observed image I, along with rendered images (T&o) of the object in many different poses (T For each pose Too the model predicts score (Lo.I)(TEo) +E that classifies whether the pose hypothesis is within the basin of attraction of the refiner. The highest scoring pose T " argmax, &jj is used the initial pose for the refinement step. Since are performing classification, our method can implicitly handle object symmetries, as multiple poses can be classified as correct.
Pose refinement model. Given an input image and an estimated pose, the refiner predicts an updated pose estimate. Starting from coarse initial pose estimate Tco,coarse we can iteratively apply the refiner produce improved pose estimate. Similar [4, 20] our refiner takes as input observed and rendered images T, (T&.) and predicts updated pose estimate T Figure 2 (b), where k refers to the ktth iteration of the refiner. Our pose update uses the same parameterization as DeepIM [20] and CosyPose [4] which disentangles rotation and translation prediction. Crucially this pose update AT depends on the choice of an anchor point O, see Appendix for more details. In prior work [4, 20] which trains and tests on the same set of objects, the network can effectively learn the position the anchor point O for each object. However order to generalize to novel objects we must enable network infer the anchor point O at inference time.
In order to provide information about the anchor point to the network we always render images (T&)) such that the anchor point O projects to the image center. Using rendered images from multiple distinct viewpoints (Tco, ) the network can infer the location of the anchor point as the intersection point of camera rays that pass through the image center, see Figure 2(b).
Additional information about object shape and geometry be provided to the network by rendering depth and surface normal channels in the rendered image I,. We normalize both input depth

---

(if available) and rendered depth images using the currently estimated pose To to assist the network in generalizing across object scales, see Appendix for more details.
Network architecture. Both the coarse and refiner networks consists of a ResNet-34 backbone followed by spatial average pooling. The coarse model has a single fully-connected layer that consumes the backbone feature and outputs classification logit. The refiner network has single fully-connected layer that consumes the backbone feature and outputs 9 values that specify the translation and rotation for the pose update.
3.2 Training Procedure
Training data. For training, both the coarse and refiner models require RGB(-D) images with ground-truth 6D object pose annotations, along with 3D models for these objects. In order for our approach to generalize to novel-objects we require a large dataset containing diverse objects. All of our methods are trained purely on synthetic data generated using BlenderProc [31]. We generate a dataset of million images using a combination ShapeNet [61] (abbreviated as SN) and Senell--Sanndd-OOOe
(abbreviated as GSO) [7]. Similar to the BOP [62] synthetic data, we randomly sampled objects from our dataset and dropped them on plane using a physics simulator. Materials, background textures, lighting and camera positions are randomized. Example images can be seen in Figure (a) and in the Appendix. Some of our ablations also use the synthetic training datasets provided by the BOP challenge [62]. We add data augmentation similar to CosyPose [4] to the RGB images which was shown to be a key to successful sim-to-real transfer. We also apply data augmentation to the depth images as explained the appendix.
Refiner model. The refiner model is trained similarly [4]. Given image with an object M at ground-truth pose Teo we generate perturbed pose Too by applying random translation and rotation to Têo Translation is sampled from a normal distribution with standard deviations of ...02.0.2.0005, centimeters and rotation is sampled as random Euler angles with a standard deviation of 15 degrees each axis. The network is trained to predict the relative transformation between the initial and target pose. Following [4, 20] we use loss that disentangles the prediction of depth, x-y translation, and rotation. See the appendix for more details.
Coarse model. Given an input image To of an object M and a pose Tco the coarse model is trained to classify whether pose Tco is within the basin of attraction of the refiner. In other words, if the refiner were started with the initial pose estimate Tco would it be able to estimate the ground-truth pose via iterative refinement? Given ground-truth, pose-annotation Tco we randomly sample poses Teo by adding random translation and rotation to Teo The positives are sampled from the same distribution used generate the perturbed poses the refiner network is trained to correct (see above), and other poses sufficiently distinct to this one (see the appendix for more details) are marked as negatives. The model is then trained with binary cross entropy loss.
4 Experiments
We evaluate our method for 6D pose estimation of novel objects using the seven challenging datasets of the BOP [2, 62] 6D pose estimation benchmark, and the ModelNet [20] dataset. The dataset and the standard 6D pose estimation metrics we use are detailed in Section 4.1 1. all our experiments, the objects are considered novel, i.e. they are only available during inference on new image and they are not used during training. In Section 4.2, evaluate the performance of our approach composed of coarse and refinement networks. Notably, we show that (i) our method is competitive with others that require the object models to be known in advance, and (iii) our refiner outperforms current state-of- the-art on the ModelNet and YCB-V datasets. Section 4.3 validates our technical contributions and shows the crucial importance of the training data in the success of our method. Finally, we discuss the limitations in Section 4.4.
Our method can consume either RGB or RGB-D images depending the input modalities that are available.
5

---

Pose Initialization
Pose Refinement
BOP Datasets
RGB-D Inpui
Method
obyeces
Method
objects
LM-0
T-LESS
TUD-L
IC-BIN
ITODD
HB
YCB-V
Mean
CosyPose [4]
CosyPose
63.3
64.0
68.5
58.3
21.6
65.6
57.4
57.0
SuriEmb
BFGS
66.3
73.5
71.5
58.8
41.3
79.1
64.7
65.0
SurlEmmb [3]
BFGS+ICP
75.8
82.8
85.4
63.6
49.8
86.7
80.6
75.2
OSOP [17]
Multi-Hyp.
31.2
49.2
33.2
OSOP [17]
MH-ICP
48.2
60.5
57.2
(PPF, Sift) Zephyr (21)
59.8
51.6
(PPF, SiFF,Sit++ + Our coarse
Our refiner
57.0
62.3
CosyPose [4]
53.6
52.0
57.6
53.0
13.1
33.5
33.3
42.3
CosyPose Pose [4]
Ours
65.5
72.0
70.1
57.3
28.4
67.0
56.8
59.6
CosyPose
Ours
71.2
63.8
85.0
55.1
39.9
73.2
69.2
66.0
Ours
18.7
19.7
20.5
15.3
8.00
13.9
16.2
12 Ours
Ours
53.7
62.2
58.4
43.6
30.1
72.9
60.4
54.5
13 Ours
Ours
58.3
54.3
71.2
37.1
40.4
75.7
63.3
57.2
Table 1: Results on the BOP challenge datasets. We report the AR score on each of the 7 datasets considered the BOP challenge and the mean score across datasets. With the exception of Zephyr (row 11), all approaches are trained purely on synthetic data. For each column, we denote the best over result in italics and the best novel-object pose estimation method in bold.
4.1 Dataset and metrics
We consider the seven core datasets of the BOP challenge [62, 2]: LineMod Occlusion (LM- O) [63], T-LESS [64], TUD-L [62], IC-BIN [65], ITODD [66], HomebrewedDDB (HB) [67] and VideVideo (YCB- V) [47]. These datasets exhibit 132 different objects in cluttered scenes with occlu- sions. These objects present many factors of variation: textured or untextured, symmetric or asymmetric, household or industrial (e.g. watcher pitcher, stapler, bowls, multi-socker plug adaptor) which makes them representative of objects that are typically encountered in robotic scenarios. The ModelNet dataset depicts individual instances of objects from 7 classes of the ModelNet [32] dataset (bathtub, bookshelf, guitar, range hood, sofa, stand and wardrobe). We use initial poses provided by adding noise to the ground truth, similar previous works [20, 16, 15]. The focus is on refining these inital poses. We follow the evaluation protocol of [2] for BOP datasets, and of DeepIM [20] for ModelNet.
4.2 6D pose estimation of novel objects
Performance of coarse+refiner. Table reports results of our novel-object pose estimation method on the BOP datasets. We first use the detections and pose hypotheses provided by a combination of PPF and SIFT, similar to the algorithms provide multiple pose hypotheses. We find the best hypothesis using the score of our coarse network, and apply 5 iterations of our refiner. Results are reported in row 7. On YCB-V V, our method achieves a + +0.. AR score improvement compared to Zephyr (row 6). Averaged across the YCB-V V and LM-O datasets, the AR score of our approach 59.7 compared to 55.7 for Zephyr (row 6). Next, provide complete set of results using the detections from Mask-RCNN networks. Please note that since detection of novel objects is outside the scope of this paper, we use the networks trained on the synthetic PBR data of the target objects [2] which are publicly available for each dataset. We report the results of our coarse estimation strategy (Table 1, row 11 11), and after running the refiner network, on RGB (row 12) or RGB-D (row 13) images. We observe that (i) our refinement network significantly improves the coarse estimates (+41.0 mean AR score for our RGBD refiner) and (ii) the performance of both models competitive with the learning- based refiner of CosyPose [4] (row while not requiring to be trained on the test objects. The recent SurfEmb [3] performs better than our approach, but heavily relies on the knowledge of the objects for training and cannot generalize novel objects.
method Zephyr [21]. For each object detection, these
Performance of the refiner. We now focus on the evaluation of our refiner which can be used to refine arbitrary initial poses. Our refiner is the only learning- based approach in Table I which can be applied to novel objects. In rows and 10, we apply our refiner to the coarse estimates of CosyPose [4] (row 11). Again, we observe that our refiner significantly improves the accuracy of these initial pose estimates (+23.7 in average for the RGB-D -D model). Notably, the RGB-only method (row 9) performs better than the CosyPose refiner (row 1) on average, while not having seen the BOP objects during

---

Average Recall
Method
RGB-D
(5. 5cm)
ADD (0.1d)
Proj2D (5px)
DeepIM [20] Multi-Path [16]
64.3
83.6
73.3
84,8
81.6
LatentFusion[155
85.5
94.7
Ours
88.6
90.5
88.9
Ours
97.6
98.9
97.5
Table 2: Evaluation of the refiner on the ModelNet [20] dataset. The mean average recall is computed over the seven classes of the dataset.
Coarse
Refined
Coarse
Refined
N
E
MiAP Milk
40
Mi
Figure 3: Qualitative results. For each pair of images, the left image is a visualization of our coarse estimate, and the right image is after applying iterations of our refiner. None of these objects from the YCBV, LMO, HB, or T-LESS datasets were used for training our approach. Please notice the high accuracy of MegaPose despite (i) severe occlusions and (ii) the varying properties of the novel objects (c.g. the texture-less industrial plug in the top-right example, textured mustard bottle in the tep-left).
training. This is thanks to our large-scale training on thousands of various objects, while CosyPose only trained on tens of objects for each dataset.
One iteration of our refiner takes approximately 50 milliseconds on RTX 2080 GPU, making it suitable for use in online tracking application. Five iterations of our refiner are also 5 times faster than the object-specific refiner of SurfEmb [3] which takes around I second per image crop. Finally, we evaluate our refiner on ModelNet and compare it with the state-of-the-an methods MP-AAE [16] and LatentFusion [15]. For this experiment, we remove the ShapeNet categories that overlap with the test ones in ModelNet from our training set order to provide fair comparison on novel instances and novel categories similar [15, 16, 20]. Results reported in Table 2 show that our refiner significantly outperforms existing approaches across all metrics.
4.3 Ablations
In this section we perform ablations of our approach to validate our main contributions. Additional ablations are in the appendix. For these ablations, we consider the RGB-only refiner and re-train several models with different configurations hyper-parameters and training data.
Encoding the anchor point and object shape. As discussed in Section 3.1 the refiner must have information about the anchor point O in order to generalize to novel objects. We accomplish this by using 4 rendered views pointing towards the anchor point, see Figure 2(b). Table 3(a) shows the performance of the refiner increases as we increase the number of views from to 4, validating our design choice. Multiple views may also help the network to understand the object's appearance from alternate viewpoints, thus potentially helping the refiner to overcome large initial pose errors. We also validate our choice to provide normal map of the object to the network. This information can help the network subtle object appearance variations that are only visible under different illumination like the details on the cross of guitar.
7

---

ModelNet
Training objects
Num. objects
BOP5
ADD(0.10)
GSO+ShapeNet
10+ 100
47.9
28.7
Rendered
Rendered normals
ModelNet
GSO+ShapeNet
100+ + 1000
49.3
80.3
BOP5
views
ADD(0.10)
GSO+ShapeNet
250 + 2000
56.9
82.0
GSO+ShapeNet
500 + 10000
59.3
89.3
52.0
83.3
GSO+ShapeNet
1000 20000
61.7
96.1
59.0
90.4
61.7
96.1
GSO
1000
62.2
95.7
59.1
83.1
BOP
132
62.6
93.4
(a)
(b)
Table 3: Ablation study. We study (a) using multiple rendered object views and normal maps as input to our RGB-only refiner model and (b) training the refiner on different variations of our large-scale synthetic dataset. Average recall is reported on BOP5 (mean of LM-O, T-LESS, TUD-L, IC-BIN and YCB-V) and ModelNet. Number of training objects. We now show the crucial role of the training data. We report in Table 3 (b) the results for our refiner trained on an increasing number CAD models. The performance steadily increases with the number of objects, which validates that training on a large number object models is important to generalize to novel ones. These results also suggest that the performance of our approach could be improved as more datasets of high-quality CAD models like GSO [7] become available.
Variety in the training objects. Next, we restrict the training different sets of objects. We observe in the bottom of Table 3(b) that models from the GoogleScannedOhjectss are more important to the performance of the method on the BOP dataset compared to using both ShapeNet and GSO. We hypothesize this due to the presence of high-quality textured objects in the GSO dataset. Finally, we train our model on the 132 objects of the BOP dataset. When testing on the same BOP objects, the performance benefits from knowing these objects during training is small compared using our GSO+ShapeNet GSO dataset.
4.4 Limitations
While MegaPose shows promising results robot experiments (please see the supplementary video) and 6D pose estimation benchmarks, there is still room for improvement. We illustrate the failure modes of our approach in the supplementary material. The most common failure mode is due to inaccurate initial pose estimates from the coarse model. The refiner model is trained to correct poses that are within a constrained range but can fail if the initial error is too large. There exist multiple potential approaches to alleviate this problem. We can increase the number of pose hypotheses M at the expense of increased inference time, improve the accuracy of the coarse model, and increase the basin of attraction for refinement model. Another limitation is the runtime of our coarse model. We use M 520 pose hypotheses per object which takes around 2.5 seconds to be rendered and evaluated by our coarse model. In tracking scenario however, the coarse model is run just once at the initial frame and the object can be tracked using the refiner which runs at 20Hz. Additionally, our refiner could also be coupled with alternate coarse estimation approaches such as [17, 18] to achieve improved runtime performance.
5 Conclusion
We propose MegaPose, method for 6D pose estimation of novel objects. Megapose can estimate the 6D pose of novel objects given a CAD model of the object available only test time. We quantitatively evaluated MegaPose on hundreds of different objects depicted in cluttered scenes, and performed ablation studies to validate our network design choices and highlight the importance of the training data. We will release our models and large-scale synthetic dataset to stimulate the development of novel methods that are practical to use in the context of robotic manipulation where rapid deployment to new scenes with new objects is crucial. While this work focuses on the coarse estimation and fine refinement of an object pose, detecting any unknown object given only CAD model is still difficult problem that remains to be solved for having complete framework for detection and pose estimation of novel objects. Future work will address zero-shot object detection using our large-scale synthetic dataset.

---

Acknowledgements
This work was partially supported by the HPC resources from GENCI-IDRIS (Grant 011011181R2 the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1 1.100...10000000/////. 003/0000468), EU Horizon Europe Programme under the project AGIMUS (No. 101070165), Louis Vuitton ENS Chair on Artificial Intelligence, and the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P31AA-001 (PRAIRIE 3IA Institute).
References
[1] Project page: htttpss:/meegapseddd.ggg
[2] T. Hodañ, M. Sundermeyer, Drost, Labbé, Brachmann, Michel, C. Rother, and J. Matas, BOP Challenge 2020 on 6D object localization. In ECCVW, 2020.
[3] R. L.. Haugaard and A. G. Buch. Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surfacc embeddings. In CVPR, 2022.
[4] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic. CosyPose: Consistent multi-view multi-object 6D pose estimation. In ECCV, 2020.
[5] L. Lipson, Z. Teed, A. Goyal, and J. Deng. Coupled iterative refinement for 6D multi-object pose estimation. In CVPR, 2022.
[6] G. Wang. F. Manhardt, F. Tombari, and X. Ji. GDR-Net: for monocular 6D object pose estimation. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 16611-1662) June 2021.
Geometry-guided direct regression network
[7] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. In ICRA, 2022.
[8] Y. Lin, Tremblay, S. Tyree, A. Vela, and Birchfield. Single-stage keypoint-based category-level object pose estimation from an RGB image. In ICRA, 2022.
[9] H. Wang. Sridhar, J. Huang. Valentin, S. Song, and L. J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642-2651, 2019.
[10] X. Chen, Z. Dong, J. Song, A. Geiger, and O. Hilliges. Category level object pose estimation via neural analysi--y-synttesi.. In European Conference Computer Vision, pages 139-156. Springer, 2020.
[11] F. Li, I. Shugurov, B. Busam, M. Li, S. Yang, and S. Tlic. Polarmesh: A star-convex shape approximation for object pose estimation. IEEE Robotics and Automation Letters, 7(2):416 4423,,4432222 2022.
[12] F. Manhardt, G. Wang, B. Busam, M. Nickel, S. Meier, Mineiullo, X. Ji, and N. Navab. CPS++: Improving class-level 6D pose and shape estimation from monocular images with self-supervised learning. arXiv 2020.
preprint arXiv:2003.055:
[13] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. KPAM: Keypoint affordances for category-level robotic manipulation. In ISRR, 2019.
[14] B. Wen, W. Lian, K. Bekris, and S. Schaal. Catgrasp: Learning category- task-relevant grasping in clutter from simulation. arXiv preprint (tXxx221090091632. 2021.
[15] K. Park, A. Mousavian, Y. Xiang, and D. Fox. LatentFusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation. CVPR, 2020.
[16] M. Sundermeyer, M. Durner, E. Y. Puang. Z.-. Marton, N. Vaskevicius, K. O. Arras, and R. Triebel. Multi Iearning for object posc estimation across domains. In CVPR, June 2020.
[17] I. Shugurov, Li, B. Busam, and S. Ilie. OSOP: A multi-stage one shot object pose estimation framework. In CVPR, 2022.
[18] V. Nguyen, Y. Hu, Xiao, Salzmann, and Lepetit. Templates for object pose estimation revisited: Generalization to new objects and robustness occlusions. In CVPR, 2022.
[19] Y. Xiao, X. Qiu, P.-A. Langlois, M. Aubry, and R. Marlet. Pose from shape: Deep pose estimation for arbitrary 3D objects. In BMVC, 2019.

---

[20] Y. Li. G. Wang, X. Ji, Y. Xiang, and D. Fox. DeepIM: Deep iterative matching for 6D pose estimation. In ECCV, 2018.
[21] B. Okom, Q Gu, M. Hchert, and D. Held. Zeroshot pose hypothesis rating. In ICRA, 2021.
Zero-shot
[22] Y. Liu, Y. Wen, S. Peng. C. Lin, X. Long. T. Komura, and W. Wang. Gen6D: Generalizable model-free 6-DoF object pose estimation from RGB images. In arXiv:2204.1011 10776, 2022.
[23] B. Drost, M. Ulrich, N. Navab, and S. Ilic. Model globally, match locally: Efficient and robust 3d object recognition. In 2010 IEEE computer society conference on computer vision and pattern recogmition, pages 998 198-1005, Icee, 2010.
[24] V. Lepetit, F. Moreno Moreno-Noguer, and P. Fua. EPnP: An accurate O(n) solution to the PnP problem. I/CV, 2009.
[25] J. Besl and N. D. McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[26] Z. Zhang. Iterative point matching for registration of free-form curves and surfaces. International jauiiiinaooounn journal of computer vision, (3(2):119-135) 152, 1994.
[27] W. Kehl, F. Manhardt, F. Tombari. S. Ilie, and N. Navab. SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1521-1529, 2017.
[28] A. Osokin, D. Sumin, and V. Lomakin. Os2d:
One-stage one-shot object detection by matching anchor features. In European Conference on Computer Vision. pages 635 -52.. Springer, 2020.
[29] J.PP. Mercier, M. Garon, P. Giguere, J.-. Lalonde. Deep template- ohject instance detection. In Pro- ceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1507-1516, 2021.
[30] Y. Xiao and R. Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. In European Conference on Computer Vision (ECCV), 2020.
[31] M. Demninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam. Blenderproc. arXiv preprint 1. 2019.
[32] Z. Wu, S. Song, A. Khosla, F. Yu, Zhang, X. Tang, and J. Xiao. 3d shapenets: deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912- 1920, 2015.
[33] L. G. Roberts. Machine perception of three-dimemsiooal solids. PhD thesis, Massachusetts Institute of Technology, 1963.
[34] D.G. Lowe. Three-dimensional object recognition from single two-dimensional images. Artif. Intell., 1987.
[35] D. G. Lowe. Object recoonition from local eealriinvariant features. In ICCV, 1999.
[36] H. Bay, T. Tuytelaars, and L. Van Gool. SURF: Speeded up robust features. In ECCV, 2006.
[37] A. Collet S. S. Srinivasa. Efficient multi-view object recognition and full pose estimation. ICRA,
[38] A. Collet, M. Martinez, and S. S. Srinivasa. The MOPED framework: Object recognition and pose estimation for manipulation. The International Journal of Robotics Research, 2011.
[39] S. Hinterstoisser, S. Holzer, Cagniart, Ilie, K. Konolige, N. Navab, and Lepetit. Multimodal templates for real-time detection of texture-less objects heavily cluttered scenes. ICCV, 2011.
[40] F. Jurie and M. Dhome. A simple and efficient template matching algorithm. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2. pages 544 549, IEEE, 2001.
[41] M. Rad and V. Lepetit. BB8: A scalable, accurate, robust to partial occlusion method for predicting the
3D poses of challenging objects without using depth. In CVPR, 2017.
[42] J. Tremblay, T. To. B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield. Deep object pose estimation
for semantic robotic grasping houschold objects. In Conference on Robot Learning (CoRL), 2018.
[43] B. Tekin, S. Sinha, and P. Fua. Real-time seamless single shot 6d object pose prediction. In CVPR, 2018.
[44] S. Peng. Y. Liu, Q. Huang, Zhou, and H. Bao. Pvnet: Pixel-wise voting network 6dof pose estimation. In CVPR, 2019.
[45] G. Pavlakos, X. Zhou, A. Chan, K. G. Derpanis, and K. Daniilidis. 6-dof object pose from semantic keypoints. In ICRA, 2017.
10

---

[46] Y. Hu, J. Hugonot, P. Fua, and M. Salzmann. Regmentation-driveer 6D object pose estimation. In CVPR, 2019.
[47] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes. In RSS, 2018.
[48] K. Park. T. Patten, and M. Vincze. Pix2Pose: Pixel-wise coordinate regression of objects for 6D pose estimation. In ICCV, 2019.
[49] C. Song, J.J.Soon Song, and Q. Huang. Hybridpose: 6D object pose estimation under hybrid representations. In CVPR, 2020.
[50] S. Zakharov, L. Shugurov, and S. Ilie. Dpod: 6d pose object detector and refiner. In CVPR, 2019.
[51] R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, 8, second edition, 2004.
[52] C. Wang. D. Xu, Y. Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and S. Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3343-3352, 2019.
[53] K. Pauwels and D. Kragic. Simtrack: A simulation-based framework for scalable real-time object pose detection and tracking. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1300-1307. IEEE, 2015.
[54] M. Oberweger, P. Wohlhart, and V. Lepetit. Training a feedback loop for hand pose estimation. In ICCV,,
[55] F. Manhardt, W. Kehl, N. Navab, and F. Tombari. Deep model-based 6d pose refinement in rgb. In ECCV, 2018.
[56] M. Aubry, D. Maturana, A. A. Efros, B. Russell, and J. Sivic. Seeing 3D chairs: Exemplar part-based 2D-3D alignment using large datasct of CAD models. In CVPR, 2014.
[57] Y. Xiao, Y. Du, and R. Marlet. PoseContrast: Class-agnostic object viewpoint estimation the wild with pose-aware contrastive learning. In 3DV, 2021.
[58] Y. He, Y. Wang, H. Fan, J. Sun, and Q Chen. FS6D: Few-shot 6D pose estimation of novel objects. In CVPR, 2022.
[59] J. Sun, Z. Wang. S. Zhang, X. Hc, H. Zhao, G. Zhang, and X. Zhou. OnePose: Onc-shot object pose estimation without CAD models. In CVPR, 2022.
[60] E. Olson. AprilTag: A robust and flexible visual fiducial system. ICRA, 2011.
[61] A. X. Chang. T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li. S. Savarese, M. Savva, S. Song. H. Su, Xiao, Yi, and F. Yu. ShapeNet: An Information- 3D Model Repository. Technical Report arXiv:1512.03012fff Jes. GR), Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
[62] T. Hodan, F. Michel, F. Brachmann, W. Kehl, A. GlentBuch, D. Kraft, B. Drost, Vidal, S. Ihrke, X. Zabulis, ct al. Bop: Benchmark for 6d object pose estimation. In ECCV, 2018.
[63] S. Hinterstoisser, V. Lepetit, S. llie, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training. detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian conference on computer vision (ACCV), 2012.
[64] T. Hodan, P. Haluza, S. Obdržálek, J. Matas, M. Lourakis, and X. Zabulis. T-LESS: An RGB-D dataset for 6D pose estimation of Texture-Less objects. In Winter Conference on Applications of Computer Vision (WACV), 2017.
[65] A. Doumanoglou, R. Kouskouridas, S. Malassiotis, and T.-K. Kim. Recovering 6d object pose and predicting next-hest-view in the crowd. In CVPR, 2016.
[66] B. Drost, M. Ulrich, Bergmann, P. Hartinger, and Steger. Introducing mytec itodd-a dataset for 3d object recognition industry. In ICCV Workshops, 2017.
[67] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilie. Homehreweddb: Rgh-d dataset for 6d pose estimation of 3d objects. In /CCV Workshops, 2019.
11


Reviews:
[{"review_id": "rmnzlL6hvjS", "reviewer": "Reviewer_VhYw", "summary": "This paper study the problem of 6D pose estimation. It adopts a coarse-refine approach, where it first predicts a coarse pose hypothesis and use a refiner based render&compare to refine the pose. It can be applied to objects not seen in the training phase.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of using classifier to acquire coarse pose is simple and robust and it can handle symmetry implicitly.\n2. The experiments demonstrate the efficacy of the refiner module.\n\nWeaknesses:\n1. The efficacy of the coarse pose estimation model is not validated. In Table 1, the comparison between 8,9.10 and 11,12,13 is unfair for the proposed coarse module. It would be stronger if there is a baseline with other pose initialization techniques that can handle novel objects and use the proposed refiner.\n2. Since both the coarse mode and refiner is based on rendering, how would rendering quality and the different lighting in the real world affect the performance?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The general idea itself is pretty straightforward and not very interesting. But the performance of the proposed method is strong, especially on the real-world experiment. So I would vote for weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "1. More results to demonstrate the advantage of the coarse model over other baselines.\n2. Discuss how robust the model is to sim2real gap in terms of rendering.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "coXzJP9uSnU", "reviewer": "Reviewer_sFyM", "summary": "A method is proposed with which the pose of novel/unseen objects are estimated without requiring CAD models at test time, in contrast to the majority of methods that train with those models. During inference, the input is a region of interest (from a detector) as well as the CAD model from which a coarse pose is estimated then refined using two separate networks. The coarse pose estimation network is a binary classifier indicating if the synthetic rendering of the object can be successfully refined w.r.t. the observation. The input from a set of renderings with highest score is then the initial pose to start the refiner. The refiner network regresses the translation and rotation to iteratively bring the model into alignment with the observation. Experiments on the BOP core datasets as well as ModelNet show the proposed approach achieves best performance for novel objects and competitive results compared to methods that know the object models apriori i.e. seen objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**STRENGTHS**\n\nThe paper addresses a relevant problem in the robotics community and sufficiently motivates the design choices in the context of real limitations i.e. not always having access to object models before deployment. The authors cover the literature well, enabling them to position their work in the context of prior work.\n\nEstimating the pose of unseen objects is very challenging. Currently there exist some solutions but they have limitations that are addressed in the proposed approach. Therefore, this paper offers novelty in terms of problem formulation and methodology.\n\nThe proposed approach achieves state-of-the-art performance on BOP and ModelNet in comparison to methods that also deal with unseen objects. Additionally, robotic grasping experiments are showcased in the supplementary video to demonstrate the real-world application.\n\n\n**WEAKNESSES**\n\nWhile the presented solution is fully-learned, I do not see this is a critical argument as to why the method is better than those discusses on lines 36-41. It is not a strong argument to say that non-learned components are unfavourable because they depend on hyperparameters, especially when the presented work also has its own set of tunable parameters e.g. M for the number of poses to render, k for the number of iterations and even the range of sampled standard deviations for the refiner dataset. A side not to this is that the method seems to be very sensitive to M, here large M means potentially better initial poses at the cost of computation time. It would be good to see an experiment to evaluate this trade-off to complement the statements made in Sec 4.4.\n\nI question whether the method itself contributes to the high accuracy or if a significant performance gain is attributed to the enormous dataset used to train the networks. As table 3(b) reveals, fewer object models lead to a performance drop and under 500 + 10000, the method no longer outperforms the competitors. A better experiment would be to control for the dataset (i.e. train all methods with the new data or an equivalent number of training samples) in order to fairly compare the methods. The authors themselves make the claim that their method performs well because of the dataset in comparison to others that have a limited dataset to train in (lines 227-229).\n\nWhile it is ok to declare the detection stage to be out of scope (and sufficiently handled by prior work), the experiments do not represent a realistic scenario of a fully unseen object pipeline since the detections (for BOP) come from a network trained on the relevant objects, hence they are known before. A more suitable method, in the context of the paper's goals, would be one of those mentioned on line 124. I suspect that the method could be sensitive to the initial detections, thus, it would be interesting to see an ablation on how the method performs w.r.t. detection inaccuracies. If the inaccuracy of zero-shot methods is below the tolerance of the proposed method, then a full pipeline is in fact not yet achieved. Lastly, it is not clear where the detections come from for the ModelNet experiments and the demonstrations in the supplementary video.\n\nTable 1 is misleading since so few results are provided for the most relevant comparisons (rows 4-7). One can see that PPF + Zephyr + ICP is the best on LM-O, thus, it is questionable if the proposed method is the best overall without the missing results. (I understand the authors do not provide results on other datasets but code for their method is provided, thus it seems feasible to generate results for this publication). Furthermore, the authors state it is difficult to directly compare to OSOP due to the lack of a detector. Is it possible to use their detections to initialise your approach? This would make the comparison much more fair as the masks generated by Mask RCNN trained on the known objects is a huge advantage.\n\nFinally, the demonstration of robotic grasping in the supplementary video is a good addition but it does not constitute a real \"experiment\" that evaluates the proposed method's performance. No details are provided about these demonstrations so it is difficult to draw any serious conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors have presented very interesting ideas that are novel for the purpose of unseen object pose estimation/refinement. Additionally, the method outperforms current SotA approaches with similar constraints. While the submission has some very solid strengths, unfortunately the evaluation has some flaws. Namely, there is a disparity in terms of the size of the dataset methods are trained on and the detections that are used for initialisation. Also, the most related methods are not evaluated on all datasets, which presents an incomplete picture. Thus, I consider some work still be done to fully convince readers of the practical significance of the work, which leads me to recommend a weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The inability to train the coarse model that regresses instead of classifies is speculated to be caused by symmetries. Could symmetric objects be removed from the dataset, and the model retrained, to support this hypothesis?\n\nThe authors should state what hardware is used to train the models and how long training time is.\n\nIt would be good to see some qualitative failure cases.\n\nIt is not clear how the grasp poses are determined for the grasping demonstrations in the supplementary."}}, {"review_id": "FrVTn6MR7Lo", "reviewer": "Reviewer_KfxR", "summary": "This manuscript presented a method to estimate the 6D pose of objects unseen during training but only requires a CAD model at testing. It is achieved by learning two key components. A pose hypothesis generator and classification that initializes some pose guesses and whether they can be handled by the refiner, the second component. The refiner follows the render-and-compare idea similar to the existing CAD-model based methods with a difference that it can refine a novel object unseen during training. A large amount of training data is generated to gain such inductive bias.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and the figures are presented nicely\n- The motivation is clear and the application of this work is obvious for robotic manipulation\n- The experiments are extensive, including multiple public benchmarks. The comparison baselines are also up to date.\n- The performance is strong considering that it didn’t train on the testing objects. \n- To generalize to the testing object with a new coordinate frame definition provided in the CAD model, a novel and smart adaptation mechanism is designed by rendering multiple different vides to disambiguate.\n\nWeakness:\n- Is the performance equal when applied to different types of objects or is there any failure modes on certain types of objects? An illustration on this or discussion would be helpful. It’d be also interesting to have some detailed result decomposition on the objects to reflect their performance variations.\n- The object detection is assumed to be available. While this is not the focus of this work, I’d imagine the accuracy of the 2D detection can play a crucial role. In addition, the current manuscript uses Mask-RCNN which needs training and inference on the same categories of objects. This contradicts with the claim of applying the method to out-of-class objects. Moreover, the Mask-RCNN is trained on the PBR data in this work, meaning many of the training instances are exactly the same as the testing data in BOP. This gives a huge advantage over the comparison methods (usually use natural detectors) from the beginning. It’d be more interesting to see a complete pipeline, by replacing with a class-agnostic object detector or at least a Mask-RCNN that is trained on some general dataset (eg. MS COO), or comparing with baselines that uses the same object detector to showcase the performance and compare.\n- While the current manuscript claims to estimate 6D pose of novel objects, a CAD model is still needed during testing. The ease of accessibility to such a high-quality CAD model during testing (especially with high-fidelity texture) is questionable in real world scenarios.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "weak accept", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- In figure 2 bottom right, what does the green and blue axis mean and why is the T_CO2 and T_CO1 overlapped (redundant)?\n- During testing for pose hypothesis generation, what if the object’s frame origin defined in the CAD model is far away from the object’s geometry center, and thus far away from the 2D bounding box center? Is the current initializing process still able to converge? \n"}}]

Author rebuttal:

Decision: Accept (poster)