PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation
Authors: Thomas Georg Jantos
Mohamed Amin Hamdad
Wolfgang Granig
Stephan Weiss
Jan Steinbrener
Venue: CoRL 2022 Poster (2022)
Keywords: 6D Pose Estimation
Transformer
Object-Relative Localization

Abstract: Accurate 6D object pose estimation is an important task for a variety of robotic applications such as grasping or localization. It is a challenging task due to object symmetries, clutter and occlusion, but it becomes more challenging when additional information, such as depth and 3D models, is not provided. We present a transformer-based approach that takes an RGB image as input and predicts a 6D pose for each object in the image. Besides the image, our network does not require any additional information such as depth maps or 3D object models. First, the image is passed through an object detector to generate feature maps and to detect objects. Then, the feature maps are fed into a transformer with the detected bounding boxes as additional information. Afterwards, the output object queries are processed by a separate translation and rotation head. We achieve state-of-the-art results for RGB-only approaches on the challenging YCB-V dataset. We illustrate the suitability of the resulting model as pose sensor for a 6-DoF state estimation task. Code is available at https://github.com/aau-cns/poet .

PoET: Pose Estimation Transformer for Single-View, Multi-Object 6D Pose Estimation
Thomas Jantos Control of Networked Systems Group University of Klagenfurt, Austria thomas. antosSaau. at
Mohamed Amin Hamdad
Wolfgang Granig
Infineon Technologies Austria AG Villach, Austria
Infineon Technologies Austria AG Villach, Austria
mohamedamin, htw.eeddaammn.aaniiii... ineon.neo..
wolf wolfgang- wwwgggagggggiggeeeee granigginn ineon.eoon.
Stephan Weiss Control of Networked Systems Group University Klagenfurt, Austria stephan. weiss@aau. at
Jan Steinbrener
Control of Networked Systems Group University of Klagenfurt, Austria jan. aweinbreneeraa.. at
Abstract: Accurate 6D object pose estimation is an important task for a variety of robotic applications such as grasping or localization. It is challenging task due to object symmetries, clutter and occlusion, but it becomes more challenging when additional information, such as depth and 3D models, not provided, We present waasformer-basee approach that takes RGB image as input and predicts 6D pose for each object in the image. Besides the image, our network does not require any additional information such as depth maps 3D object models. First, the image is passed through an object detector to generate feature maps and to detect objects. Then, the feature maps fed into transformer with the detected bounding boxes as additional information. Afterwards, the output object queries are processed by a separate translation and rotation head. We achieve state-of- the-art results for RGB-only approaches on the challenging YCB-V dataset. We illustrate the suitability of the resulting model as pose sensor for 6-DoF state estimation task. Code is available at https: //gpt/////hhbbb con//au-can cna/ cns/poet.
Keywords: 6D Pose Estimation, Transformer, Object-Relative Localization
1 Introduction
Accurately estimating the 6D pose objects from RGB images is essential for robotics tasks such grasping localization [1, 2]. Grasping tasks require the robot to know the exact position of the object such that can place its end-effector effectively. In autonomous driving, is critical that the vehicle has sufficient knowledge of its surroundings including the relative 6D pose of all objects in its vicinity. For unmanned aerial vehicle (UAV) navigation, especially in close proximity to people infrastructure, realizing precise control depends the estimation of 6D object poses. In recent years, basedn-based 6D pose estimation with deep learning [3] has been on the rise. Approaches differ in terms of input data, network architecture, post processing and number of viewpoints [4, 5, 6. 7]. Observing objects from multiple viewpoints introduces constraints to the pose of objects and improves estimation [7, 8]. Availability of 3D object models allows for an iterative refinement of an initial pose estimate by either iterative closest point (ICP) [9] matching of pointclouds or by matching keypoints with perspective-n-pol
(PnP) [10] algorithm. However, these algorithms are computationally demanding. Besides being used for post processing, 3D models can be utilized as an additional input to the network [5). This may include model keypoints and corresponding features or information about the object such as symmetry axes and planes. Additionally, prior information
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

about the object class or a depth map corresponding to the input RGB image, which improves the networks ability to estimate the objects' distance the camera [8], can be passed to the network. Although additional input information can greatly benefit the accuracy of the final estimated pose, apart from requiring more detailed data base containing accurate 3D models and depth maps corresponding to RGB images, it results in higher computational complexity and thus, longer run time in comparison to the same neural network- -based architecture taking only RGB images [8].
While the requirements for real-time performance depend on the specific application, more often the availability of high-quality, detailed 3D models or depth maps limited. Depending on the type of additional input information, additional sensor hardware is also required, which may not be available during inference. In this work, we focus on a pose estimation framework purely based on RGB images and information provided by a backbone object detector. To the best of our knowledge, we are the first to incorporate global image context information into the pose estimation task by passing multi-scale feature maps to transformer network and relying only on 2D) image information. Our approach does not depend on the number of objects present. Our framework, dubbed PoET (Pose Estimation Transformer), can be used on top of any 2D object detector. We evaluate our approach on the YCB-V [3] dataset and compare our results to state-of-the-art approaches. Finally, we illustrate the suitability of the obtained model as pose sensor for a 6-DoF state estimation task, where "pose sensor" means the combination of a camera and PoET providing information about the camera pose relative to objects, i.e. its relative position and orientation. Our contributions are the following:
We present a transformer- based framework that takes a single RGB-image as input, estimates the 6D pose for every object present in the image and can be trained top of any object detector framework. A detailed ablation study supports our design choices.
The framework is independent of any additional information which is not contained in the raw RGB image. In particular, it does not depend on depth maps, object symmetries or 3D object models. Hence, our results are achieved without iterative refinement and the whole network can be trained using 3D model independent loss functions.
We achieve state-of-the competitive results in comparison approaches utilizing 3D models.
results on the YCB-V [3] dataset for RGB-only methods and
We show the feasibility of the resulting model pose sensor in 6- localization task.
The rest of the paper is organized as follows: In Section 2, related work for 6D pose estimation is reviewed. Following the presentation of our method and implementation details in Section 3, the experiments and the corresponding results are discussed in Section 4 including an ablation study investigating our network architecture. Additionally, we illustrate how PoET and its relative 6l) pose estimates can be used for localization in Section 4.3. Finally, the limitations are discussed in Section 5 and the paper is concluded in Section 6. We refer the reader the supplementary material for an extensive ablation study and additional results on the LM-O [11] dataset.
2 Related Work
Classical image-based 6D pose estimation approaches can be split into feature- based methods and template-basede methods. For the latter, object pose is determined by matching object templates against the input image [12, 13]. While template-bar based approaches work well on texture-less objects, they are prone to fail in scenarios where objects are occluded. In feature-based methods, local features are extracted from the image and then matched to the 3D model to determine correspondences [14]. Based on these 2D-3D correspondences, the 6D pose of the object can be derived. While these approaches can handle occlusion of objects, they require textures in order to perform the matching. If RGB-D images are available, the additional information provided by the depth can be used to improve the initial pose estimate by iterative refinement [15, 16] such as e.g., ICP [9]. Even though those methods can achieve -fate-of-the-ar performance, they are computationally expensive and the availability of depth maps is not guaranteed.
In recent years, advancements in deep learning for computer vision tasks have been applied to single- view, image-based 6D pose estimation, either to replace components of classical approaches [17, 18, 19, 20), or end-to-end learned methods, where the 6D pose is directly estimated from the input using convolutional neural networks (CNNs). Xiang et al. [3] proposed with PoseCNN, a CNN- based method to regress the 3D translation and rotation of each object present in the image using

---

Object Query Embbddiing
Translation Head
Sp.
Posttiinal
f0 Sy)
Enceding
Rederence Poanss
(Multi-Senle] Feature Map
Atsention Eacoder
Attention Decoder
Rotation Head
Object Detector Backbone
PoET
Figure 1: Overview of the PoET network architecture for single-view, multi-object 6D pose estimation. Bounding box information for each detected object is passed to the transformer an object query. Afterwards, for each object query the egocentric 3D translation and 6D rotation
[25] is predicted.
an object symmetry-aware loss function. Li al. [8] proposed a framework that introduces prior knowledge about the object class into the network and perform pose estimation by discretizing the possible translation and rotation values unique bins resulting in classification task.
Aside from depth images, using 3D object models for pose estimation yielded promising results [21, 4,5, 6, 7]. The approaches differ in terms of how the 3D model is used. Kehl al. [21] and Li al. [4] predict the 6D pose of objects and then refine the estimated pose. With GDR-Net, Wang et al. [22] able integrate PnP into end-to-end trainable network. Similarly, et al. [6] use the 3D model of an object for iterative pose refinement. Given an initial pose estimate, a network is trained iteratively refine the pose by matching rendered image created from the 3D model to the original image. Labbé et al. [7] apply this approach to multiple viewpoints of the same scene resulting in improved pose estimation. In yet another approach, Billings and 3D model as an additional input to their network, dubbed SilhoNet. Their network predicts the object silhouette and 3D translation vector derived from the 2D bounding box position. Based on the former, the 3D rotation of the object is estimated and corrected for silhouette symmetric objects significant restriction, as the real error rotation for symmetric objects can be very large.
[5] provide the
Similar to our approach, Amini et al. [23] presented a transformer-based architecture to directly regress the 6D pose for multiple objects contained in single image. By extending the Detection Transformer [24] with translation and rotation heads, they are able to train the whole network in an end-to-end fashion. However, they require the object 3D model they use a symmetry aware loss [3]. Our approach does not require any additional information such as depth maps, 3D models or known object symmetries, but instead directly estimates the translation and rotation of objects in the camera coordinate frame from single RGB image. In contrast to other methods that work with regions of interest for pose estimation, we keep the complete image feature map and provide the regions of interest as an additional input to our transformer network.
3 Method
We present a novel transformer-based neural network for the 6D pose estimation task. Taking single RGB image as its input, the 6D pose of every object detected in the image is predicted simultaneously. After generating (multi-scale) feature maps by passing the image through an object detector, they are processed by a transformer architecture. At the end, translation and rotation estimated in a decoupled manner. In this section, first present the general structure of our network. Afterwards, we talk about specific implementation details and data preparation.
3.1 Network Architecture
Fig. shows a detailed overview of our network architecture, which consists of three steps. First, the input image is passed through a backbone object detector network. Both the generated (multi- scale) feature maps and the predicted bounding boxes are used in subsequent processing. PoET can be trained on top of any object detector architecture and thus extend a pre-trained object detection framework to include 6D object pose estimation. Second, the (multi-scale) feature maps used for the object detection step are passed to the encoder of multi-head irrention-based transformer.
3

---

Our transformer architecture is a modified version of the Deformable DETR transformer module proposed by Zhu et al. [26]. In contrast to the original DETR [24], the deformable transformer allows to process multi-scale feature maps similar to state-of-the-ard object detectors. By only attending to a limited number of feature map keypoints the decoder, the transformer achieves a higher pass- through rate. Additionally, a deformable transformer shows faster convergence rates than regular transformer architecture. The main idea behind using a transformer architecture for feature map refinement is to generate features that capture the global information contained in the image. For example, such additional information might be the image location of other objects present or general information of the overall scene extracted from the image.
While the encoder of the deformable transformer is kept unchanged, we modified the decoder to incorporate more information from the object detection step: First, the learned object query embedding is replaced by bounding box information. For each detected object, the bounding box center coordinates (Ca: Gy), the width and the height h are normalized and then position-encoded [27] to generate the object query embeddings. The embedding dimension I, is chosen such that 2 2- L equals the hidden dimension dA of the transformer. In our case, the number of parameters np equals to 4. Moreover, the inter-query attention heads ensure that information is properly propagated between the different object queries. Second, the Deformable DETR originally only attends to a limited number of keypoints which are randomly sampled around reference points. Instead of predicting reference points from query embeddings by a trainable fully connected layer. we directly feed the normalized center coordinates (Ca, as the reference points to the decoder. By feeding this additional information the decoder along with the encoder-refined image feature maps and the inter-query attention heads, the decoder generates new object query embeddings which not only contain local information regarding the object but also global information extracted from the image. Third, the object queries outputted by the transformer are passed through a translation and rotation head. This allows us to simultaneously estimate the pose for multiple objects independent of how many objects are present and which class they belong to. As we approach the pose estimation problem from a global image context perspective by extracting features from the whole image, the network directly estimates the translation and rotation with respect to the camera.
Our translation head is simple multi-layer perceptron (MLP) with input dimension dn, one hidden layer and output dimension 3. We directly predict the translation = (y..ty.f)) with respect to the camera frame. Given the ground-truth translation f, our translation head is trained with simple 1.2-loss defined as
(1) The rotation prediction head is identical to the translation head besides the output dimension. For estimating the rotation, we use the 6D rotation representation proposed by Zhou et al. [25] as this representation does not suffer from discontinuities with respect to learning as e.g., quaternions do. Hence, the network predicts 6-dimensional output vector. Afterwards, the 6D representation is used to determine the estimated rotation matrix R E SO(3) as described in [25]. The rotation head trained using a geodesic loss [28] given by
L - El12-
1 2
Tr RR -1).
Liot arccos
(2)
where R is the ground-truth rotation and Tr(-) is the matrix trace operator. To ensure numerical stability of the loss during training, the argument of the arecos is clamped between - 1 e and 1 c, where c = le 1 6. Our whole network is then trained with a weighted multi-task loss expressed as
L= d,L + Arot
(3)
where the loss is calculated for each object and then averaged across all objects present across all images in the batch. X and Arot are the weighting parameters for the translation and rotation loss respectively. PoFT can be trained either class-specific class-agnostic. In the class-specific case with Rets different classes, the translation and rotation output dimension are changed to 3 nels and 6 netts respectively. For each object query, one hypothesis for each class is regressed and the final output is chosen depending on the class predicted for the bounding box. However, no class information fed into the transformer.
3.2 Implementation Details & Data Preparation
While PoET can be trained on top of any object detector, we used Scaled- YOLOv4 [29] as the backbone object detector as it offers good trade-off between speed and accuracy. An MS-

---

OLARMOID
Figure 2: Qualitative results of the relative 6-DoF object poses predicted by PoET for the YCB-V dataset.
COCO [30] pre-trained Scaled-YOLOOve YOLOv4 is fine tuned for 10 epochs on the YCB-V dataset for the object detection task. During the training of POET. the weights of the object detector backbone are frozen.
We implement PoET using PyTorch [31] and train it for 50 epochs using AdamW [32] with learning rate of 2e and a batch size of 16. Our best performing network has five encoder and decoder layers, d, = 256, 16 attention heads and positional embedding dimension of L 32. The network is simultaneously trained for 3D translation and 3D rotation estimation and the weighting parameters are set to A = 2 and Xyot 1, such that both losses in the same value range.
Given the commonly used data split for YCB-V [3, 5, 6, 7, 8, 33], we train our network on 80 of the available 92 video sequences and reserve the remaining 12 sequences and their challenging keyframes for testing and evaluation. Moreover, we include 80,000 synthetic images generated from 2D projections of the 3D object models provided by the original authors [3]. We refer the reader to the supplementary material, for more details regarding our implementation.
4 Results & Experiments
In this section, we present PoET's performance on the YCB-V benchmark dataset for 6D pose estimation. For evaluation, use the AUC of ADD-S metric [3] and additionally report the average translation and rotation error in cm and degree, respectively, as done by [5]. We compare our results to state-of-ther oftthe-rrt, single-view, RGB-based approaches and list the results of other approaches as reported in the corresponding work. We conduct an ablation study on the network architecture and modifications to the transformer part. The ablation study investigates the influence of the object detector backbone, transformer modifications, data augmentation, network architecture and rotation representation on PoET's performance. While we present here our most important findings, refer the reader to our supplementary material for additional ablation results. Finally, we illustrate how PoET's relative pose estimates can be used for camera localization.
4.1
6D Pose Estimation
Our best performing network is class-specific, consists of 5 encoder and decoder layers with 16 attention heads, and was trained according to Section 3.2. Methods that rely on an object detector to predict regions of interest (ROIs) usually do not elaborate whether and, if so, how multiple predictions, bad predictions and missing predictions are treated within their approach. Therefore, to allow for fair comparison, we also report the results of PoET and other approaches given ground-truth bounding boxes in addition to the results on predicted bounding boxes. We present representative qualitative results in Fig. 2.
In Table report our results for the AUC of ADD-S metric. Both for predicted and ground-truth bounding boxes, PoFT outperforms (overall and also for most individual classes) other state-of-the- art RGB-based methods that either work on the whole input image [3, 23] or that predict ROIs for pose regression
[8]. We also outperform SilhoNet [5] which feeds additional information from the 3D model its network and reduces predicted rotations by known object symmetries. This shows that PoET and its feature maps containing global image context reduce the need for any additional inputs. This is especially highlighted when comparing the performance of PoET the ROI-based approaches SilhoNet and MCN [8] in the case where all networks are provided with ground-truth bounding boxes. Only models that explicititiy explicitly utilize 3D object models during inference, either through PnP during the estimation [22] or by performing iterative refinement after estimating an initial pose [7, 6), achieve slightly better results but this comes at the cost of significantly increased computational complexity and the need for accurate 3D models to be known priori.
5

---

Table 1: Comparison with state-of-the-art YCB-V. We report the AUC of ADD-S. gt denotes results achieved with providing ground-truth ROIs to network. The 3D model row indicates how the object model is used: either to calculate the loss function, as an additional input, for symmetry reduction or for PnP or iterative refinement (IR) based pose estimation. (*) denotes symmetric objects. Bold and italic values indicate state-of thee-----thtttt-t results for methods not based PnP/IR, using ground-trun or predicted ROIs, respectively.
Medhal
PouCNN [3]
SilloNes (5)
SillwNes,
MCNIS
MCN,,
TWD [23]
PIET
PoET
CDD-Net [22]
CooyPose (7] DespMM66
3D Midel
Loo
Input 4 Sym
Injut + 1:5m
23
20
Loos
283
20
Pul
IR
IX
chel
61.0
57.3
51.2
V1.9
91.5
H5.4
96.5
sracker bus
76.9
73.5
38-4
64.3
78.5
96.5
90.4
80.5
64.0
91.0
sugus box
64.3
36.6
53.8
62.4
85.1
90.3
94.5
92.4
98.3
96.2
tomato suup
80.9
38.7
39.4
87.9
93.3
68.9
94.0
91.8
96.1
92.4
meatant bottle
90.2
89.8
91.0
97.5
91.9
94.7
94.8
91.7
99.5
95.1 1
tura io
87.9
89.3
39.9
64.7
98.2
02.2
94.3
90.4
96.1
96.1
publing box
70.0
68.
39.1
51.0
84.0
K5.1
93.8
59.0
94.5
90.7
gelatin box
87.1
$2.7
93.6
80.4
92.1
56.9
92.77
91.7
95.3
94.3
ported meot can
78.5
78.8
$6.8
K3.1
90.8
K3.5
94.1
91.2
K2.9
85.6
bararia
85.9
30.7
$8.7
15.1
70.0
91.8
91.3
89.5
9600
71.5
phckes hase
76.8
91.7
9138
54.8
92.7
91.7
98.8
bleach deatserr
71.9
76.0
K3.0
85.4
94.4
bowd"
54.0
may
76.0
92.1
07.4
pows drill
72.8
76.0
89.8
91.9
block*
65.8
54.0
90.7
17.3
scisson
56.2
49.1
763
71.6
80.2
81D
75.2
68.4
75.0
meeker
71.4
75.0
831
60.1
66.4
74.9
61.5
A1.2
874
86.
large chaupe
40.9
69.2
84.5
66.8
86.5
78.1
95.7
68.6
60.3
74.
extn large clamg"
47.0
12.3
88.4
61.1
79.5
54.7
96.0
82.55
73.6
73.2
foom bricc*
67.8
77.9
85.4
60,9
79.2
9
89.7
81.3
50.4
81.9
All
75.9
79.6
85.8
75.1
86.9
85.2
91.8
67.
89.1
89.8
85.1
The improved performance of PoET compared to other RGB-based models in terms of the ADD-S score is due to better estimate of the 3D translation as can be seen in Table 2. Again, ground- truth-based results are also presented. In contrast to directly estimating the 3D translation like POET, PoseCNN as well as SilhoNet determine the 3D) translation by estimating the depth and center pixel coordinates of an object and then reprojecting them using the known camera intrinsic parameters, which is considered the easier task to learn [3]. MCN treats the translation estimation as a classification problem by binning the translation space. In addition to the global image information provided by the multi-scale feature maps, our approach also learns to model the camera intrinsics, which results in a more accurate estimation of translation.
For 3D rotation estimation, we achieve the same average error as regular PoseCNN. Not surprisingly, networks that reduce the possible rotation space based on known object symmetries achieve better result but with limited upplicability to real-world scenarios. The mean average 3D rotation error shown in Table 2. The influence of reducing the rotation by geometrical symmetries (t) is highlighted for PoseCNN. Since PoET makes use of the full multi-scale RGB feature maps, it outperforms SilhoNet for objects that have no symmetries in the silhouette space and only performs significantly worse for objects with rotational symmetries around an axis in 3D space, but without requiring a priori knowledge about the 3D shape of objects or restricting the rotation space based on symmetries. The main source for rotational errors are objects with rotational symmetries around one axis (master chef can, tomato soup can, tuna fish can). We have investigated the axes of our rotation errors and compared them to the symmetry axes for symmetric objects. The average tilt of rotation eттor axes with respect to symmetry axes across all test images and symmetric objects is only 15 degrees. If we ignore rotational eπτогr about symmetry axes, our average rotation error reduces to 1 11.24 degrees, outperforming all RGB-only competitors, see Table 2. We refer the reader to the supplementary material for additional ablation experiments as well as PoET's performance for the stricter BOP(33) and AUC of ADD [3] metrics and for the LM-O benchmark dataset [11].
4.2 Ablation Study
The ablation study investigates the influence of different components on the performance of our PoET framework. By assuming a perfect object detector that provides ground truth bounding boxes to POET, we ensure that the evaluation includes every object present in the image even for those which might be not detected by the object detector. During the ablation study, we focus on the AUC of ADD/ADD-S metric, the average translation error and rotation error. All results reported in this section are for the same hyperparameter configuration as described in Section 3.2, the same fixed seed and trained for the same number of epochs. The final results are summarized in Table 3. We compare our best performing network from Section 4.1 (Baseline) to class-agnostic version (Agnostic) and a network with less layers (Small). Finally, we investigate the influence of the integration of bounding box information into the transformer on the performance of PoET. We train PoET with learnable reference points (RP), trainable query embeddings (Q) or the combination of

---

Table 2: Comparison of average translation in em and rotation in degrees on YCB-V. Bold and italic values indicate the stale-of-the-arl for results, when working on ground-truh or predicted ROIs respectively. f
indicates that the rotation predictions are reduced by geometric symmetries as described in [S].
Meshed
PouCCNN11)
SilbNNa[[]]
SilhNNa,
PrFT
PowcCNN
PoweONN
SilboNe!
Nil
P.RT,
PxFT
clel
X29
5022
1.14
L.37
1.2%
50.7
1.57
1:31
1.11
19.22
10
cracker bux
4.02
5:24
2.38
1.48
3.14
F9.65
19.69
19.86
9.53
9.65
21.87
sugar box
306
2.10
1.67
0.94
144
9.39
9.20
12.25
11.50
3.95
4.40
2,40
1,09
18.25
8.40
189
1.82
50.97
49.29
mustard bottle
1665
0,94
1.42
9.94
9.59
5.78
5,07
23.71
27.53
fish
1.57
0.95
52.80
12.74
1.50
pudking bot
Trandlation Ezior (cm)
1.01
10.20
10.20
20,05
18.39
0.87
golatin
1.20
5.25
12.52
7.19
mear
20,40
pttcher cleamer
1.31
0.95
11.98
11.98
6.61
8,04
3.00
20.85
20.35
51.25
21.93
houl"
3.30
1.51
75.53
75.51
49.05
nug
1.28
19:44
1944
7,02
power dhill
4.39
6.77
3.19
0.98
2.29
9.91
9.91
30.54
10.56
5.92
6.45
wood bookk"
0.34
5.50
3,23
1,41
4.75
23.63
23,33
25.52
23.23
5.88
14.32
scissoes
0.40
9.91
2.59
1.38
3.72
43.98
43.98
155.53
158,82
3.19
5.27
large mus ler
3.89
3.24
2.31
2.68
2.75
92. 44
13.59
10.44
10.72
24.95
25.91
large clamp"
9.79
6.27
3.51
0.98
2.38
58.12
38.1
3.54
6.03
2.61
4.88
exin hrre champ*
8.36
4.86
2.12
0.91
1.10
34. 18
34. 18
29. 18
7,30
2,38
26.07
foom brick
2.48
3.98
2.31
1.90
3.40
22. 67
22.67
13.84
17.36
37.20
36. 34
All
416
349
2.45
1.30
272
20.79
17.53
76.04
13.45
23.65
27.35
Table 3: Ablation study results PoET on YCB-V. We report the AUC ADD/-S the average translation and rotation error. Ablation of class mode (Agnostic), network size (Small) and transformer modifications (RP, Q, RP Q). The exact meaning of the tags are described in the text. Agnostic
Metric
Baseline
Small
RP
RP Q
AUC of ADD-S
92.8
88.9
91.8
87.6
82.2
42.0
AUC of ADD
80.8
73.2
78.1
66.8
59.5
12.2
Avg. T. Error [em]
1.20
1.95
1.48
1.92
2.99
9.06
Avg. Rot. Error [*]
23.65
24.92
25.64
37.31
35.39
74,26
both (RP + Q). all three cases the performance is reduced comparison to a version of PoET that uses bounding box information to generate query embeddings and reference points. This shows that providing a transformer with bounding box information can greatly benefit training process leading to improved performance for the same training duration. We kindly refer the reader to the supplementary material for an in-depth analysis and discussion of the ablation study.
4.3 Localization
PoET is well suited for vision-based object-relative localization. A transformation between coordinate frames and B expressed in coordinate frame C is fully defined by the translation Ctab and the rotation Ras. Given a set of landmarks with known ground-truth pose (R mo a single frame that captures at least one of those landmarks can be used in combination with PoET to localize the camera by estimating the relative pose (Ri (%o to all landmarks present in the frame. For each landmark, the estimated camera pose (FP: wey ((po) can be determined by
R = R 5:a and
R
W inc
= We E R'po 5. CF
(4)
The final camera pose (Rowes Ewee can then be determined by taking the average over all landmarks present in the image. YCB-V's test sequences offer different camera trajectories and each with a different constellation of multiple objects serving as landmarks. For each individual frame we w
estimate the camera pose and compare the ground-truth. In Fig. 3 we show an example trajectory for single sequence. For further examples we refer the reader to the supplementary material.
We compare three different approaches: using all detected objects (all), perform simple outlier rejection by taking and choosing the hypothesis the majority of objects agree on (out) or by incorporating the camera pose estimate from the previous frame into the outlier rejection cases with multiple hypotheses having the same number of votes (prev). For the sake of comparison, we also calculate the camera pose by choosing the estimated camera pose being closest to the current ground-truth camera pose (best). Moreover, we differentiate between PoET being provided either object detections from the backbone (bb) or ground-truth bounding box information (gt). We calculate the sample error mean and standard deviation across all frames for the position and attitude and summarize them in Table Simple outlier rejection greatly benefits the localization emoor. This is due to wrong relative object pose estimates having large impact on the final estimated camera pose when taking simple average. Incorporating the camera pose estimate from the previous frame only slightly improves the localization as it only helps in multi-hypothesis
7

---

cases, In comparison to the estimated trajectories using the best estimated camera pose, our outlier rejection based trajectories perform worse with around factor two. Nonetheless, the localization results when considering only individual frames are remarkable. The difference in performance between backbone and ground-truth detections due to the backbone potentially missing an object or assigning wrong class and thus throwing off the estimate.
To further motivate the use of PoET pose sensor in mobile roboties, we have integrated it in a state-of-the-art sensor fusion framework [34] and performed state estimation experiments with YCB-V objects in our motion capture room using inertial data for propagation and PoET for the pose update. For representative trajectory, the average error in rotation was (roll, pitch, yaw) (4.0, 6.9, 14.8) degrees and in translation (x, y, z) = (0.084, 0.179, 0.032) meters. The performance for our real data is only slightly worse compared to the performance on the benchmark dataset, even though we were using completely different camera, than the one used to record the original dataset, and the objects have slightly changed in appearance in comparison to the original YCB objects. Nonetheless, PoET able to provide sufficiently accurate pose data. Summarizing, PoET achieves sufficient localization accuracy such that it can be used as pose sensor for robotics.
(x. (p... [mm]
Position
15,
0.8
bb
OUR
(56, 28) (100, 106, 36)
(12, 43, 26) 1 (56. 59, 27)
0.6
(28, 27, (31. 35,
(102, 138, + 117,111,,550 53)
E
(38, 31, 19) (80. 80, 28)
(32. 28, 19) 21)
(21.
12)
-0.6
Lylls piich)
500
1000
1500
2000
(19.6, 7.1, =
(18.6,
Attitude
(5.4. 222 (10.4.
yaw
bb
peev
(4.1. 14, 1 1 2.1)
50
ro11
hest
(2.6, 1.5, (3.0, 1.7.
degree
(18.0 6.7.
(16.5. 10.8, 12.9) (3.7. 1.2, 3.0) (8.9. 2.0.
0
roll
out
pitch
gt
(2.8. 1.2, + 2.2,133 1.5)
100
pitch
ext
(2.0, 13, 1300 1A,
500
1000
1500
2000
Table 4: Sample error mean and standard Figure 3: Example trajectory of single sequence for gt deviation of camera pose across all 12 and out. We plot the ground truth and estimated position sequence frames. Position and attitude are and attitude.
Trames
reported in mun and degree.
5 Limitations
In Section 4.1 it was discussed that PoET is outperformed by methods utilizing 3D object information for rotation estimation in particular for objects that have rotation-symmetric silhouettes. Wrongly estimated rotations lead to the assumption that the camera views the objects from a different angle resulting in wrong hypotheses for the localization task as discussed in Section 4.3. Moreover, the low resolution the images in the YCB-V dataset means that RGB textures are not as dominant, especially when augmentation is used, This is the main reason why PoET has difficulties to estimate silhouette amatiin-symmest objects that are not symmetric in RGB space, see e.g. Table 2.
Conclusion
In this work we presented novel, transformer-based framework for multi-object 6D pose estimation. It can be used on top of any object detector and the only input required is a single RGB image. The image is passed through an object detector backbone to create (multi-scale) image feature maps and to detect objects. Bounding box information of detected objects is fed into the transformer decoder which improves the learning. By taking the whole image into consideration during the estimation process, our framework does not rely on any additional information. We outperform other RGB-based methods by a wide margin on the YCB-V dataset. This especially important for scenarios where no detailed 3D models or prior object information is available, or where computational efficiency is required and thus any input besides the RGB image has to be dropped. Moreover, we highlighted how PoET's 's relative 6D object pose estimation can be used as pose sensor for robot localization tasks.
8

---

Acknowledgments
This work was supported by the Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) under the grant agreement 881082 (MUKISANO).
References
[1] H. Song, W. Choi, and H. Kim. Robust vision-based relative-localization approach using rgb-depth camera and lidar sensor fusion. IEEE Transactions on Industrial Electronics, 63(6): 2016.
[2] V. Loing, R. Marlet, and M. Aubry. Virtual training for real application: Accurate object- robot relative localization without calibration. International Journal of Computer Vision, 126 19):1045-1000, 2018.
[3] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Poseenn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Robotics: Science and Systems (RSS), 2018.
[4] Z. Li, G. Wang, and X. Ji. Cdpn: Coordinates based disentangled pose network for real- time rgb-based 6-dof object pose estimation. In Proceedings of the IÉEE/CVF International Conference on Computer Vision, pages 7678-7687, 2019.
[5] G. Billings and M. Johnsonn-Rbbesoon Silhonet: An rgb method for 6d object pose estimation. IEEE Robotics and Automation Letters, 4(4):3727-3734, 2019.
[6] Y. Li, G. Wang. X. Ji, Y. Xiang, and D. Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 683-698, 2018.
[7] Y. Labbé, J. Carpentier, M. Aubry, and Sivic. Cosypose: Consistent multi-view multi-object 6d pose estimation. In European Conference on Computer Vision, pages 574-591. Springer, 2020.
[8] C. Li, J. Bai, and G. D. Hager. A unified framework for multi-view multi-class object pose estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 254-269, 2018.
[9] P. J. Besl and N. D. McKay, Method for registration of 3-d shapes. In Sensor Fusion IV: Control Paradigms and Data Structures, volume 1611, pages 586-606. International Society for Optics and Photonics, 1992.
[10] M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395. 1981.
[11] A. Krull, E. Brachmann, F. Michel, M. Y. Yang. S. Gumhold, and C. Rother. Learning analysis- by-synthesis for pose estimation in rgb-d images. In Proceedings of the IEEE international conference computer vision, pages 954-962, 2015.
[12] S. Hinterstoisser, C. Cagniart, S. Ilic, P. Sturm, N. Navab, Fua, and V. Lepetit. Gradient response maps for real-time detection textureless objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(5):876-888, 2011.
[13] Z. Cao, Y. Sheikh, and N. K. Banerjee. Real-time scalable 6dof pose estimation for textureless objects. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 2441-2448. IEEE, 2016.
[14) G. Pavlakos, X. Zhou, A. Chan, K. G. Derpanis, and K. Daniilidis. 6-dof object pose from semantic keypoints. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2011-201 1-2018. IEEE, 2017.
[15] S. Hinterstoisser, V. Lepetit, S. llic, S. Holzer, G. Bradski, K. Konolige, and N. Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian Conference Computer Vision, pages 548-562. Springer, 2012.

---

[16] F. Michel, A. Kirillov, E. Brachmann, A. Krull, S. Gumhold, B. Savchynskyy, and C. Rother. Global hypothesis generation for 6d object pose estimation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 462-471, 2017.
[17] K. Park, T. Patten, and M. Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7668-7677 2019.
[18] Hodan, D. Barath, and J. Matas. Epos: Estimating 6d pose of objects with symmetries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11703-11712 2020.
[19] M. Rad and V. Lepetit. Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. In Proceedings of the IEEE International Conference on Computer Vision, pages 3828-3836, 2017.
[20] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4561-45700 2019.
[21] W. Kehl, F. Manhardt, F. Tombari, S. Ilie, and N. Navab. Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again. In Proceedings of the IEEE International Conference on Computer Vision, pages 1521-1529, 2017.
[22] G. Wang, F. Manhardt, F. Tombari, and X. Ji. Gdr-net: Geomelry-guided direct regression network for monocular 6d object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1661 16611-1661) 1-16621, 2021.
[23] A. Amini, A. S. Periyasamy, and S. Behnke. T6d-direct: Transformers for multi-object 6d pose direct regression. In DAGM German Conference on Pattern Recognition, pages 530-544. Springer, 2021.
[24] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213- 229. Springer, 2020.
[25] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9745-5753. 2019.
[26] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference Learning Representations (ICLR), 2021.
[27] B. Mildenhall, Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405 421. Springer, 2020.
P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf:
[28) S. Mahendran, H. Ali, and R. Vidal. 3d pose regression using convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2174-2182, 2017.
[29] C.-Y. Wang, A. Bochkovskiy, and H.-Y, M. Liao. Scaled-yolov4: Scaling cross stage partial network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13029-13038, 2021.
[30] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L., Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
[31] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, al. Pytorch: imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
10

---

[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), 2019.
[33] Hodañ, M. Sundermeyer, B. Drost, Labbé, Brachmann, F. Michel, C. Rother, and J. Matas. Bop challenge 2020 on 6d object localization. In European Conference on Computer Vision, pages 577-594. Springer, 2020.
[34] C. Brommer, R. Jung. J. Steinbrener, and Weiss. MaRS: A modular and robust sensor-fusion framework. IEEE Roboties and Automation Letters, 6()):359-366, 2020.


Reviews:
[{"review_id": "yQXwJf9izSn", "reviewer": "Reviewer_pB8h", "summary": "This paper tackles the problem of 6D pose estimation object instances. The method is pretty standard. It is a transformer based architecture that takes in the features at different scales of the object detector as the input along side the positionally encoded information for each detection and predicts rotation and translation for each object. The method is evaluated on YCB-V dataset. It achieves decent (but not SOTA) on pose estimation. \n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\n- Why both bbox coordinates and object centers are provided to transformer. What is the intuition behind having bbox info as object codes (as opposed to fixed learned object codes in other works) + providing center as the reference point? How would the results change if for example, object queries are fixed and learned and only reference points are provided?\n- Backbone effect: the method takes as input the features of Yolo. I would like to see if the same results hold if the detector is changed or the method is tied to YOLO object detector.\n- Loose connection to robotics application: I would like to see an experiment where the proposed method is used to estimate the pose of an object and then robot manipulate the object.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am more leaning toward the borderline and open to change based on authors' response to my comments.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses above.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "m-MVilKVnLcM", "reviewer": "Reviewer_v6yy", "summary": "This paper proposes a system for object pose estimation from RGB images. The only input to the system is the RGB image, it does not use depth maps or 3D models. The system first uses an object detector to generate feature maps and bounding boxes, which are passed to a transformer module, along with auxiliary information such as box center coordinates. The output of the transformer is then passed to a translation head and a rotation head, which together output a pose for each object present. The method performs comparably to methods that use 3D models such as PoseCNN, and outperforms other RGB-only methods. The authors also demonstrate that their method can be used for camera localization given landmarks of known ground truth poses.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach is RGB-only and does not require depth maps or 3D models of every object, so it is much more accessible as anyone with a regular camera could in theory use it\n\nWeaknesses:\n- While object pose estimation methods commonly struggle with accurate orientation prediction, the rotation error for PoET still seems a bit too high to truly be of use in robotic manipulation tasks\n- The authors compare with methods that are several years old (PoseCNN and SilhoNet). Are they still the best methods to compare against?\n- The method requires a large dataset of labeled images and is unlikely to generalize well to novel objects\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an interesting new method for object pose estimation using transformers with a simple and elegant architecture, and the authors show that it outperforms baselines. However, I have reservations about the baselines as well as the usefulness of the method for robotics given the high rotation errors.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses above", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "hlP4nkC3OGo", "reviewer": "Reviewer_6Xj1", "summary": "This paper presents a model, called PoET, for single-image pose estimation. They feed the image to a backbone object detector (Scaled-YOLO-v4), which produces multi-scale features and bounding boxes. An attention-decoder model subsequently processes each detected object, using an architecture based on Deformable DETR. The model then predicts rotation and translation, using geodesic loss and L2 distance. They evaluate the model on the YCB-V dataset, where their method gets better performance than many baselines, including T6d-direct and SilhoNet. Ablations in the supplementary material compare different design decisions, such as different model size and rotation representations. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The approach obtains good performance on YCB-V. It outperforms previous approaches that do not have 3D models available. It obtains similar performance to T6d-direction, without \n- The approach is a natural combination of object detection methods and transformers. It is arguably simpler than many of the baselines.\n\nWeaknesses:\n- The paper paper is not written very clearly. In particular, the paper does not very clearly explain its contribution, and the ablations are not well organized. \n- Only one qualitative result is shown, and it is in the supplementary material.\n- I think that the ablations should be moved to the main paper, and presented more concisely. Given that the contribution is the architecture (and a combination of existing ideas), and the improvement over previous methods is not large, I think that these ablations are very important. It would also be helpful to see these expanded to cover some of the other key contributions, such as the design of the Deformable DETR module (which is one of the main novelties). \n- The approach is a straightforward application of existing methods (object detection followed DETR-like attention and regression). In particular, it's similar to T6D-Direct (which requires a 3D model for its symmetry-aware loss).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this method gets good results with a simple method, the approach is quite incremental, and the paper is not very clearly written. Given that both the approach and the results are incremental, I would have expected there to be significantly more analysis. I think that this paper is an interesting step towards simpler, transformer-based pose regression, but it would benefit from additional revisions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "As mentioned above, it would be helpful for the authors to provide more explanation for how the the technical approach is a significant advance over T6d-direct, provide additional ablations related to DETR, and describe future writing revisions. "}}, {"review_id": "BoZj1ftqH6", "reviewer": "Reviewer_jp2U", "summary": "The paper presents a framework for single-view, multi-object 6-D pose estimation. The method is based on an object detector backbone, which receives a single RGB image as input, and a modified version of a Deformable DETR Transformer, which is provided with multi-scale features extracted by the object detector, as well as the center coordinates and extent of the bounding-box predicted by the latter. The \"object queries\" returned by the Transformer are passed through two separate MLP to regress a translation and rotation vector for each detected object. The object detector is finetuned from a pretrained model, while the Transformer and MLPs are trained end-to-end with pose supervision  and receiving ground-truth bounding box and object class at training time. The approach is novel and achieves state-of-the-art results on the YCB-V and LM-O datasets for RGB-only methods. The method is also demonstrated for the localization of a camera against a set of landmarks on scenes from a dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The ideas and the architecture proposed by the paper are mostly novel and provide interesting insights about the use of (Deformable DETR) Transformer networks for multi-object pose estimation. The experiments are thorough and extensive and the comparisons with previous work fair (e.g., all the baselines use synthetic data for augmentation during training, as the proposed method). The approach achieves state-of-the-art on the evaluated datasets for RGB-only methods, and runs at real-time rates. The supplementary material provides a large number of additional results and analyses.\n\nHowever, a substantial improvement in the quality of the presentation is required. In particular:\n\n- The quality of the visualizations is in my opinion not yet at the level of an international conference: Many writings in the figures and tables have an extremely small font size, and are hard to read even after zooming to 300%. Figure 1, as well as the line plots (Fig. 2, Suppl. Fig. 3. and 5) could be restyled to appear more professional, by, e.g.: making the font and its size uniform across the visual elements and w.r.t. the other figures (and ideally the text), making the colors more distinctive in the line plots for overlapping lines (so as to increase readability), and explaining the meaning of the legend and the other visual elements (e.g., the MLPs in Fig. 1).\n- A large amount of typos and formatting issues are currently present (see \"Issues -> Typos/formatting\").\n- Language is at times articulated or inaccurate (see \"Issues -> Language suggestions\" and partly \"Issues -> Points to clarify\")\n- Occasionally, symbols are used, either in the equations or in Tables, without being introduced in the previous text or caption respectively.\n\nCitations of related work sometimes lack preciseness (e.g., papers cited as examples of a class of method, but not actually matching the features mentioned in the text for these methods). A number of general statements made in the introduction and to provide context for the related work would need to be supported by references.\n\nFinally, while for the \"pose sensor\" experiments results are shown that could be applied in a robotic scenario for camera localization, no actual deployment on hardware is demonstrated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I recommend accepting the paper, given that the proposed ideas are mostly novel, that the task investigated is of concrete relevance for robotics and that the results demonstrated, as well as the running speed of the algorithm, could make the method particularly suitable for real-world robotic applications. However, a non-negligible amount of editing should be performed to address the issues detailed in the \"Issues\" section.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "# Points to clarify\n- L39: What does \"_detailed_ data _base_\" mean exactly? Also, this and the previous statements in the paragraph should be supported by references\n- L40: \"architecture\": Is this referring only to network-based solutions? If so, it should be specified\n- \"pose sensor\" (L52, 62, 299, 320): What exactly is meant by this? \"Sensor\" commonly refers to a piece of hardware\n- L74: [7] does not assume 3D models\n- L77: [9] does not perform 6D pose estimation, nor does it use a 3D model\n- L91: What does \"binning the translation and rotation\" mean?\n- L142: While represented with $\\gamma$, the encoding from NeRF is never referred to in [28] as \"gamma encoding\"\n- Eq. 2-3: The text should mention what R, $\\lambda_t$ and $\\lambda_{rot}$ are\n- L176: Missing meaning of $n_{cls}$​. This part could be clearer in general, e.g., specify that one pose hypothesis is regressed for each class\n- L193: How exactly is the match performed?\n- Tab. 3: How can the large discrepancy on \"scissors\" be interpreted?\n- L263: PoseCNN -> SilhoNet?\n- Eq. 4: Why \"W\" in the top-left corner of translation only? Why \"wo\" subscripts in the RHS of translation and \"wc\" in the LHS?\n- L294-295: How can this large discrepancy be interpreted?\n- L308: Can this be supported with results?\n- Suppl.\n  - Sec. 2: Why does RP + Q perform much worse?\n  - L67-69: If Jitter is defined in the text as having $\\sigma=0.02$, how can it be evaluated for different sigmas in Fig. 1? Clarify that it is different from Tab. 1\n  - L83: How can this be supported?\n  - L131, Tab. 6: What does GT mean for YOLO/R-CNN? The GT bounding boxes do not come from a network\n  - L138: Is Mask R-CNN or Faster R-CNN used?\n  - Fig. 5: What does \"out\" mean?\n  - L178: [13] -> [12]?\n\n# Visualization\n- Fig. 1: Unusual style: Very thin lines, not clear what the dots in the right part represent. Writings at the bottom are crucial for understanding, but barely visible. Almost all the text (e.g., $c_x, c_y$) is too small to be read without zoom\n- Table 2-4, Fig. 2, Suppl.: Fig 3.: Numbers, text and lines are way too small to be read without zoom\n- Suppl.\n  - Tab. 1: Symbols in the column headers are not defined in the caption. At least a sentence mentioning that these are defined in the text should be inserted\n  - Bold should be used in all Tables to highlight results, particularly in Tab. 4-5, in which Baseline does not always achieve the lowest error\n- Inconsistent fonts across figures (cf., e.g., Fig. 1 and Suppl. Fig. 1)\n\n# Typos/formatting\n- Dashes in multi-word adjectives\n  - state of the art -> state-of-the-art: L12, 60, 131, 223, 239. Suppl.: L71, 132, 140, 177, 181, 186, 204, Caption Tab. 6, 9\n  - 6 DoF -> 6-DoF: L14\n  - RGB only -> RGB-only: L60\n  - pass through -> pass-through: L133\n  - encoder refined -> encoder-refined: L150\n  - ground truth -> ground-truth: L160, 192, 195, 235, 246, 273, 287, 297, Caption Tab. 1. Suppl.: L58, 59, 131, 135, 155, 170, 185, 197, Caption Fig. 1, 4\n  - 6 dimensional -> 6-dimensional: L165\n  - class specific/agnostic -> class-specific/agnostic: L172, 173. Suppl.: L19\n  - real world -> real-world: L261\n  - rotation symmetric -> rotation-symmetric: L303, 308\n- L31: an -> a\n- Missing commas after the following words\n  - L36: \"image\"\n  - L49: \"framework\", \"Transformer)\"\n  - L88: \"PoseCNN\"\n  - L101: Before \"dubbed\"\n  - L111: \"symmetries\"\n  - L123: \"architecture\"\n  - L192: \"disregarded\"\n  - L194: \"inference\"\n  - L251: \"translation\"\n  - L257: \"intrinsics\"\n  - \"e.g.\" -> \", e.g., \": L81, 164. Suppl.: L120\n- L37: Missing apostrophe after \"objects\"\n- L38: apart of -> apart from\n- L43: What does \"it\" refer to?\n- L46: to the -> into the\n- L78: this -> these\n- L98: takes [...] applies -> take [...] apply\n- L120: section we, -> section, we\n- L139: \";\" at the end of paragraph -> \", as follows.\"?\n- positional encoded -> positionally-encoded: L142, Suppl.: L109\n- Exp. notation: $e$ -> $\\textrm{e}$, else it is Euler's number: L169, Suppl.: Caption Fig 6\n- L214: Remove comma after \"though\"\n- Missing period at the end of caption: Table 1-3. Suppl.: Table 1-9\n- L233: ROI -> ROIs\n- L243: reduces -> reduce\n- L254: \"it\" -> \"them\", or rephrase\n- L285: hypothesis -> hypotheses\n- Missing venue for ref. [5]\n- Ref. [27] and [33]: Replace arXiv with ICLR 2021/2019 resp.\n- Format of [32] is not uniform with other references (e.g., URL)\n- Suppl.\n  - L3: work -> works\n  - L17: is -> are\n  - Fig. 1: \"AuC\" -> \"AUC\" for consistency with text\n  - L140: detector -> detectors\n  - Caption Tab. 7: perfromance -> performance\n  - Caption Fig. 4: Remove \"as\"\n  - L199: PNP -> PnP\n\n# Language suggestions\n- L32-33: \"Besides using 3D object models [...], they can be\" -> \"Besides being used for post processing, 3D models can be\"\n- L89: \". Moreover, [...] employ\" -> \", which makes use of\"\n- L100: \"as they\" -> \"who\"\n- L129: \"attention-based\" is redundant. One could rather specify \"_vision_ transformer\"\n- Suppl.\n  - L23: \"layers,\" -> \"layers;\"\n  - L216: \"to further improve\" -> \"on further improving\""}}]

Author rebuttal:

Decision: Accept (poster)