3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation
Authors: Chen Zhao
Tong Zhang
Mathieu Salzmann
Venue: ICLR 2024 poster (2024)
Keywords: Object pose estimation; Unseen objects; 3D computer vision
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Prior methods that tackle the problem of generalizable object pose estimation highly rely on having dense views of the unseen object. By contrast, we address the scenario where only a single reference view of the object is available. Our goal then is to estimate the relative object pose between this reference view and a query image that depicts the object in a different pose. In this scenario, robust generalization is imperative due to the presence of unseen objects during testing and the large-scale object pose variation between the reference and the query. To this end, we present a new hypothesis-and-verification framework, in which we generate and evaluate multiple pose hypotheses, ultimately selecting the most reliable one as the relative object pose. To measure reliability, we introduce a 3D-aware verification that explicitly applies 3D transformations to the 3D object representations learned from the two input images. Our comprehensive experiments on the Objaverse, LINEMOD, and CO3D datasets evidence the superior accuracy of our approach in relative pose estimation and its robustness in large-scale pose variations, when dealing with unseen objects.

Published as a conference paper at ICLR 2024
3D AWARE HYPOTHESIS & VERIFICATION FOR GEN- ERALIZABLE RELATIVE OВJЕCT POSE ESTIMATION
Chen Zhao EPFL-CVVLab chen. Mhen.zhaa@epp1.co
Tong Zhang
Mathieu Salzmann EPFL-CVILab, ClearSpace SA mathi eac.ieeuaaa sal mmann@epp1
EPFL--VVRL
tong. eegggnhann@eepf.cc
ABSTRACT
Prior methods that tackle the problem of generalizable object pose estimation highly rely on having dense views of the unseen object. By contrast, we address the scenario where only single reference view of the object available. Our goal then is to estimate the relative object pose between this reference view and a query image that depicts the object in a different pose. In this scenario, robust generalization is imperative due to the presence of unseen objects during test- ing and the large-scale object pose variation between the reference and the query. To this end, we present new hypothesis-ande wnd-sss--n---riiiiii verifiiciinntion framework, in which we generate and evaluate multiple pose hypotheses, ultimately selecting the most reliable one as the relative object pose. To measure reliability, we introduce a 3D-aware verification that explicitly applies 3D transformations to the 3D object representations learned from the two input images. Our comprehensive experi- ments on the Objaverse, LINEMOD, and CO3D datasets evidence the superior accuracy of our approach in relative pose estimation and its robustness in large- scale pose variations, when dealing with unseen objects. Our project website is at: http://ssp....ztbbooooooooee//// 3DAA://
INTRODUCTION
Object pose estimation is crucial in many computer vision and robotics tasks, such as VR/AR (Azumal 1997), scene understanding (Geiger et al. 2012. Chen et al. 2017 Xu et al. 2018 Marchand al. 2015), a and robotic manipulation (Collet al. 201 Zhu al.
2014 Trem-
blay et al. 2018 Pitteri et al. 2019). Much effort has been made toward estimating object pose parameters either by direct regression (Xiang et al. 2017 Wang et al. 2019a Hu et al. 2020) or
by establishing correspondences (Peng et al. 2019 Wang et al. 2021 Su et al. 2022) which act as input to a PnP algorithm (Lepetit al. 2009). These methods have achieved promising results in the closed-set scenario, where the training and testing data contain the same object instances. However, this assumption restricts their applicability to the real world, where unseen objects from new categories often exist. Therefore, there has been growing interest in generalizable object pose estimation, aiming to develop models that generalize to unseen objects the testing phase.
In this context, some approaches (Zhao et al. 2022b; Shugurov et al. 2022) follow template- matching strategy, matching a query object image with reference images generated by rendering the 3D textured object mesh from various viewpoints. To address the scenario where the object mesh is unavailable, as illustrated in Fig. [[a] some methods take real dense-view images as references. The object pose in the query image is estimated either by utilizing a cemplate-matching mechanism (Liu let al. 2022) or by building 2D-3D correspondences (Sun et al., 2022). A computationally expensive 3D reconstruction (Schonberger & Frahm, 2016) is involved to either calibrate the reference images or reconstruct the 3D object point cloud. In any event, the requirement of dense-view references pre- cludes the use of these methods for individual sparse images, e.g., downloaded from the Internet. Intuitively, with sufficiently diverse training data, one could think of learning to regress the object pose parameters directly from single query image. However, without access to a canonical ob- ject frame, the predicted object pose would be ill-defined as it represents the relative transformation between the camera frame and the object frame.
"Corresponding author.

---

Published as a conference paper at ICLR 2024
Quety (Unseen
Reference (iiggee View)
Calibrated Reference
Quary (Unvean (Ui OHR)
Estimation
Object Pose
Ref. Pose
Dence Views
AP
Poss Estimation
AP (Relative Pose)
Object Pose
(a) Previous work
(b) Our method
Figure 1: Difference between previous work and our method. Previous approaches (a) estimate the pose of an unseen object building upon either template matching or 2D-3D correspondences, which requires dense views of the object as references. By contrast, our method (b) takes only one reference input and predicts the relative object pose between the reference and query. The object pose in the query can be derived when the pose of the reference is available.
To bypass this issue, we assume the availability of single reference image that contains the novel object. As shown in Fig. [(bb relative object pose between the reference view and the query view, which is thus well-defined. If the object pose in the reference is provided, the object pose in the query be derived. In this scenario, one plausible solution is to compute the relative object pose based on pixel-level correspondences (Lowe) 2004 Rublee et al. (011). However, the two views may depict a large- scale object pose variation, and our experiments will evidence that even the state-of-the-art feature- matching approaches (Sarlin et al. 2020, Sun al.] 2021 Goodwin et al. 2022) cannot generate reliable correspondences in this case, which thus results in inaccurate relative object pose estimates. As an alternative, Zhang et al. (2022): Lin al. (2023) predict the likelihood of pose parameters leveraging an energy-based model, which, however, lacks the ability to capture 3D information when learning 2D feature embeddings.
take this reference to be the canonical view and estimate the
paradigm, drawing inspiration from its remark- able success in robust estimation (Fischler & Bolles 1981). We randomly sample pose parameter hypotheses and verify the reliability of these hypotheses. The relative object pose is determined as the most reliable hypothesis. Since relative pose denotes a 3D transformation, achieving robust verification from two 2D images is non-trivial. Our innovation lies in 3D-aware verification mech- anism. Specifically, we develop a 3D reasoning module over 2D feature maps, which infers 3D) structural features represented as 3D volumes. This lets explicitly apply the pose hypothesis as a 3D transformation to the reference volume. Intuitively, the transformed reference volume should be aligned with the query one if the sampled hypothesis is correct. We thus propose verify the hypothesis by comparing the feature similarities of the reference and the query. To boost robustness, we aggregate the 3D features into orthogonal 2D plane embeddings and compare these embeddings to obtain a similarity score that indicates the reliability of the hypothesis.
By contrast, we adopt a
Our method achieves state-of-the-at performance on an existing benchmark of Lin al. (2033) Moreover, we extend the experiments to new benchmark for generalizable relative object pose estimation, which we refer as GROP. Our benchmark contains over 10,000 testing image pairs, exploiting objects from Objaverse (Deitke et al. (023) and LINEMOD (Hinterstoisser et al.) 2012) datasets, thus encompassing both synthetic and real images with diverse object poses. In the context of previously unseen objects, our method outperforms the feature-mane maturre and energy-based tech- niques by a large margin in terms of both relative object pose estimation accuracy and robustness. We summarize contributions follows:
We highlight the importance of relative pose estimation for novel objects in scenarios where only one reference image available for each object.
We present a new hypothesis-an and-verification paradigm where verification is made aware of 3DD by acting learnable 3D object representation.
2

---

Published as a conference paper at ICLR 2024
We develop benchmark called GROP, where the evaluation of relative object pose estimation is conducted on both synthetic and real images with diverse object poses.
RELATED WORK
Instance-Specifte Object Pose Estimation. The advancements in deep learning have revolution- ized the field of object pose estimation. Most existing studies have focused instance-level object pose estimation (Xiang et al. 2017| Peng et al. 2019 Wang et al. 202T] Su et al. 2022 Wang et al.) 20T9a), aiming to determine the pose of specific object instances. These methods have achieved markable performance in the closed-set setting, which means that the training data and testing data contain the same object instances. However, such an instance-level assumption restricts the appli- cations in the real world where previously unseen objects widely exist. The studies of /Zhao et al. (2022b): [iuu et al. (2022) have revealed the limited generalization ability of the instance-level ap- proaches when confronted with unseen objects. Some approaches (Wang et al., 2219b) Chen al.) 2020a Lin et al., 2022) relaxed the instance-level constraint and introduced category-level object pose estimation. More concretely, the testing and training datasets consist of different object in- stances but the same object categories. As different instances belonging to the same category depict similar visual patterns, the category- object pose estimation methods capable of generaliz- ing well to new instances. However, these approaches still face challenges in generalizing to objects from novel categories, since the object appearance could vary significantly.
Generalizable Object Pose Estimation. Recently, some effort has been made toward generalizable object pose estimation. The testing data may include objects from categories that have not been encountered during training. The objective is to estimate the pose of these unseen objects without retraining the network. In such a context, the existing approaches can be categorized into two groups, i.c., cemplate-matchine
methods (Sundermeyer et al.) 2020 Labbé et al. 2022 Zhao et al. 2022b
Liu et al.) 2022] [Sugurov et al.) 2022) and ceature-matchingg methods (Sun et al. [222]] He al., 2022b). Given query image of the object, the template- mate-ingt methods retrieve the most similar reference image from a pre-generated database. The object pose taken as that in the retrieved reference. The database is created by either rendering the 3D object model capturing images from various viewpoints. The feature-matching methods reconstruct the 3D object point cloud by performing SFM (Schonberger & Frahm 2016) over sequence of images. The 2D-3D matches are then built over the query image and the reconstructed point cloud, from which the object pose is estimated by using the PnP algorithm. Notably, these two groups both require dense-view reference images to be available. Therefore, they cannot be applied in scenarios where only sparse images are accessible.
Relative Object Pose Estimation. Some existing methods could nonetheless be applied for relative object pose estimation, even though they were designed for different purpose. For example, one could use traditional (Lowe 2004) or learning-based (Sarlin et al. 2020 Sun et al. 2021 Goodwin al. 2022) methods build pixel-pixel correspondences and compute the relative pose by using multi-view geometry (Hartley & Zisserman (203).. However, as only two views (one query and one reference) are available, large-scale object pose variations are inevitable, posing challenges to the correspondence-based approaches. Moreover, RelPose (Zhang al.) 2022) and RelPose++ (Lin jet al. 2023) build upon an energy-based model, which combines the pose parameters with the two- view images the input and predicts the likelihood of the relative camera pose. However, RelPose and RelPose++ exhibit limitation their ability to reason about 3D information, which we found crucial for inferring the 3D transformation between 2D images. By contrast, we propose explicitly utilize 3D information in a new hypothesis-and and-verification paradigm, achieving considerably better performance in our experiments.
3 METHOD
3.1 PROBLEM FORMULATION
We train network RGB images depicting specific object instances from set Otrain- During testing, we aim for the network to generalize to new objects in the set Otest with Ozest nO = 0. In contrast some previous methods which assume that Orrain and Orest contain the same cate-
3

---

Published as a conference paper at ICLR 2024
Feature Extraction
Latent Representation Learning
Hypothesis-und und-- Verification
F
Query
Hypothesis
3D Reascoing
AP E SD(3)
Featue Apgropaiion
Shired
Triining
3D
Trum
Reference
Figure 2: Overview of framework. Our method estimates the relative pose of previously unseen objects given two images, building upon a new pooothess--nndveriiiee pothesis AP is randomly sampled and its accuracy is measured 3D information, we perform the verification over a 3D object representation indicated as a learnable 3D volume. The sampled hypothesis is coupled with the learned representation via 3D transfor- mation over the reference 3D volume. We learn the 3D volumes from the 2D feature maps extracted from the RGB images by introducing 3D reasoning module. To improve robustness, randomly mask out some blocks colored in white during training.
paradigm. A hy- To explicitly integrate
gories, i.e., Ctrain = Creat, we work on generalizable object pose estimation. The testing objects Otest may belong to previously unseen categories, i.c., Crest * Ctrain- In such context, we propose to estimate the relative pose AP of the object depicted in two images Lg and I,.. As the 3D object translation can be derived by utilizing 2D detection (Saito et 2022 Wang al. 2023 Kirillov et al. 2023), we focus on the estimation of 3D object rotation AR E (O(3). which is more challenging. illustrated in Fig. 2 our method builds upon a hypothesis and-verification mecha- nism (Fischler & Bolles, 1981). Concretely, we randomly sample an orientation hypothesis AR, utilizing the 6D continuous representation of Zhou al. (2019). We then verify the correctness of AR, using verification score s, I,, I,,,, θ) where indicates network with learn- able parameters O. The expected A determined as the hypothesis with the highest verification score, i.e.,
AR
max I,)R,, ES,3))
To facilitate the verification, we develop a 3D transforming layer over a learnable 3D object repre- sentation. The details will be introduced in this section.
3.2 3D OBJECT REPRESENTATION LEARNING
Predicting 3D transformations from 2D images is inherently challenging, as necessitates the ca- pability of 3D reasoning. Furthermore, the requirement of generalization ability to unseen objects makes the problem even harder. Existing methods (Zhang et al.) [202]] Lin et al.] (003) tackle this challenge by deriving 3D information from global feature embeddings, which obtained through global pooling over 2D feature maps. However, this design exhibits two key drawbacks: First, the low-level structural features which are crucial for reasoning about 3D transformations are lost; Sec- ond, the global pooling process incorporates high-level semantic information (Zhao et al. 2022b), which coupled with the object category. Therefore, these approaches encounter difficulties in accurately estimating the relative pose of previously unseen objects.
To address this, we introduce a 3D object representation learning module that is capable of reason- ing about 3D information from 2D structural features. Concretely, the process begins by feeding the query and reference images into pretrained encoder (Ranftl al.) 2020), yielding two feature maps F1, F" E RC*H,,x, As no global pooling layer is involved, Fv and F" contain more structural information than the global feature embeddings of (Zhang et al. 2022; Lin et al.) 2023). Subsequently, F? and Fr serve inputs to a 3D reasoning module. Since each RGB im- age depicts the object from a particular viewpoint, inferring 3D features from a single 2D feature map is intractable. To address this issue, combine the query and reference views and utilize the transformer (Vaswani et al. 2017: Dosovitskiy et al. 2020). renowned for its ability to capture relationships among local patches.
4

---

Published as a conference paper at ICLR 2024
Our 3D reasoning block comprises self-attention layer and cross-attention layer, which account for the view and inter-view relationships, respectively. Notably, unlike the existing method Lin et al. (2023) that utilizes transformers an image level, i... treating global feature embed- ding as a token, our module takes FY and F" as input, thereby preserving more structural information throughout the process. Specifically, we compute
F = ( FT 52 gross) "sett
F = ( (S D Prosss
(3)
where g denotes the 3D reasoning block with learnable parameters { seits Raoss P D )- Let us take F9 as an example as the process over F" is symmetrie. We tokenize F9 by flattening from where N H Wy- A position embedding (Dosovitskiy et al. 2020)
RCCH,xW, to
added to the sequence of tokens, which accounts for positional information. To ensure a broader receptive field that covers the entire object, the tokens are fed into a self-attention layer, which formulated as Fy {(F): F/ where denotes the attention layer. As aforementioned. Ff only describes the object in L,. which captured from a single viewpoint. We thus develop a cross-attention layer, incorporating information from the other view I, into Fy. We denote the cross F7|2 where F serves the input of the next 3D reasoning block.
attention as F 1+
C F9 and F comprise both intra-view and inter-view object-related information. Nevertheless, it is still non-trivial ple the 3D transformation with the 2D feature maps, which is crucial in the following hypothesis- and-verification module. To handle this, we derive 3D object representation from the 2D fea- ture map simple yet effective manner. We lift F and F from 2D space to 3D space, i.e., RCx H XW, + RCaa D D x where C= Cad x Dj. The 3D representations are thus en- coded as 3D volumes V and Vr. Since the spatial dimensionality of V and V matches that of the 3D transformation, such a lifting process enables the subsequent 3D-aware verification.
We denote the output of the last 3D reasoning block as FT, F F
3.3 3D-AWARE HYPOTHESIS AND VERIFICATION
The hypothesis- www--andiiiaattioooooooo mechanism has achieved tremendous success as a robust estima- tor (Fischler & Bolles 1981) for image matching (Yi et al.) 2018 Zhao et al. 2021). The objective is to identify the most reliable hypothesis from multiple samplings. In such context, an effee- tive verification process is critical. Moreover, in the scenario of relative object pose estimation, expect the verification to be differentiable and aware of the 3D transformation, We thus tailor the hypothesis-an sanbsis--add-e-iiiatttt mechanism to meet these new requirements.
We develop a 3D masking approach in latent space before sampling hypotheses, drawing inspi- ration from the success of the masked visual modeling methods (He al.) 2022al Xie et al. 2022]. Instead of masking the RGB images, we propose to mask the learnable 3D volumes, which we empirically found more compact and effective. Specifically, we sample two binary masks Vg. Vī E RCax x H, xW, during training, initialized as all ones. h of the elements in each mask are randomly set to 0. 3D masking is performed V V @ Vg. V = V O V where @ denotes the Hadamard product. Note that the masking is asymmetric over V and VT. Such a
design enables the modeling of object motion between two images (Gupta al. 2023 offering potential benefits to the task of relative object pose estimation.
The hypothesis-andd-vrrfffe verifiiiiiiiiooon process begins by randomly sampling hypotheses, utilizing the 6D continuous representation of Zhou et al. (2019). Each hypothesis is then converted to a 3D rotation matrix AR,, i.e., R + 3 During the verification, we explicitly couple the hypothesis with the learnalle 3D representation by performing a 3D transformation. This is formulated as
(4) where X denotes the 3D coordinates of the elements in V with L= x Hy x W, and y indicates trilinear interpolation. We keep the query 3D volume unchanged and only transform the reference 3D volume. Intuitively, the transformed V should be aligned with V if the sampled hy- pothesis is correct. Conversely, an incorrect 3D transformation is supposed to result in a noticeable disparity between the two 3D volumes. Therefore, our transformation-base approach facilitates the verification of AR, which could be implemented by assessing the similarity between V9 and
V = (') X" E 3XLL
5

---

Published as a conference paper at ICLR 2024
VO. However, the transformed V tends to be noisy in practice because of zero padding during the transformation and some nuisances such as the background. We thus introduce feature aggrega- tion module, aiming to distill meaningful information for robust verification. More concretely, project V and V back three orthogonal 2D planes, i.e., IR Ca D2 xW, + R3CxH,xW, with C Csd D.. and aggregate the projected features A ((V') V) A' = VV, where A A' E RR xxW, represent the distilled feature embeddings and g is the aggregation module with learnable parameters V. The verification score is then computed as
A jk
Ai |
A)k AJk E C2
(5)
N
3
We run the hypothesis and verification M times in parallel and the expected AR" is identified as
AR = ARE: == arg max (8 i 112,...,My
(6)
Note that compared with the dynamic rendering method (Park al. 2020) which optimizes the ob- ject pose by rendering and comparing depth images, our approach performs verification the latent space. This eliminates the need for computationally intensive rendering and operates independently of depth information. An alternative to the popthesis-and-veriiee
mechanism consists of opti- mizing AR via gradient descent. However, our empirical observations indicate that this alternative often gets trapped in local optima. Moreover, compared with the mheeyy-baeed approaches (Zhang al. 2022, [Lin et al. 2023), our method achieves a 3D-aware verification. To highlight this, let us formulate the emergy-based model with some abuse of notation
AR' =
arg max Ss. s; = FC I,) ((R,)) AR,E50(3)
(7)
where FC denotes fully connected layers. In this context, the 2D image embedding and the pose embedding are learned as f(I,,,, I,) and ((),,, separately. By contrast, our framework, the volume features are conditioned on AR, via the 3D transformation, which thus facilitates the 3D> aware verification.
We train our network using an infoNCE loss (Chen et al. 2020b), which is defined as
exp( exp(s (s)
L = log
(8)
iiml exp( (s:/T)
where si denotes the score of a positive hypothesis, and 7 = 0.1 is a predefined temperature. The positive samples identified by computing the geodesic distance
[r (r(AAR,AR
D
arccos
//T,
(9)
2
where ARg is the ground truth. We then consider hypotheses with D < as positive samples.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
In our experiments, we employ 3D reasoning blocks. We set the number of hypotheses during training and testing to M = 9, 000 and M = 50, 000, respectively. We define the masking threshold
0.25 and the geodesic distance threshold À 15t (Zhang et al. 2022 Lin et al., (023). We train our network for 25 epochs using the Adam W (Loshchilov & Hutter 2017) optimizer with
batch size of 48 and a learning rate of 10-4, which is divided by 10 after 20 epochs. Training takes around 4 days on 4 NVIDIA Tesla V100s.
4.2 EXPERIMENTAL. SETUP
We compare our method with several relevant competitors including feature- matching methods, ie., SuperGlue (Sarlin al. 2020), LoFTR (Sun et al. 202T), and ZSP (Goodwin al. 2022

---

Published as a conference paper at ICLR 2024
SuperGlue
LoFTR
ZSP
Regress
RelPose
RelPose++
Ours
Angular Error J
67.2
77.5
87.5
46.0
50.0
38.5
28.5
Acc @ 30 (%)
45.2
37.9
25.7
60.6
64.2
77.0
83.5
Ace @ 15t (%)
37.7
33.1
14.6
42.7
48.6
69.8
71.0
Table 1: Experimental results CO3D.
SuperGlue
LoFTR
ZSP
Regress
RelPose
RelPose++
Ours
Angular Error
102.4
134.1
107.2
55.9
80.4
33.5
28.1
Acc @ 30 (%) f Acc @ 15t (%)
15.1
9.6
4.2
39.2
20.8
72.3
78.6
12.1
7.7
1.5
15.6
6.7
42.9
58.4
Table 2: Experimental results on Objaverse.
SuperGlue
LoFTR
ZSP
Regress
RelPose
RelPose++
Ours
Angular Error J
64.8
84.5
78.6
52.1
58.3
46.6
41.7
Acc @ 30 (%) Acc @ 15t (%)
26.2
24.2
10.7
26.5
26.1
42.5
61.5
14.3
13.5
2.7
7.6
7.0
15.8
29.9
Table 3: Experimental results on LINEMOD.
energy-based methods, i.e. RelPose (Zhang et al. 2022) and RelPose++ Lin et al. 2023). and regression method (Lin et al. 2023). We first pertorm evaluation using the benchmark defined in (Lin et al. 2023), where the experiments are conducted on the CO3D (Reizenstein et al. 202T) dataset. We report the angular error between the predicted AR and the ground truth, which is computed as in Eq. 9 and the accuracy with thresholds of 30 and 15t (Zhang et al.) 2022 Lin al. 2023). Furthermore, We extend the evaluation by introducing new benchmark called GROP To this end, we utilize the Objaverse (Deitke et al., (023) and LINEMOD (Hinterstoisser et al. 2012) datasets, which include synthetic and real data, respectively. We retrain RelPose, RelPose++, and the regression method in our benchmark, and use the pretrained models for SuperGlue and LoFTR since retraining these two feature feature-matching approaches requires additional pixel-level annotations. For ZSP, there is training process involved, evaluate using the code released by the authors. We derive AR from the estimated essential matrix (Hartley Zisserman 2003) for the feature-matching methods because only have access RGB images. We evaluate all methods on identical predefined query and reference pairs (8,304 on Objaverse and 5,000 on LINEMOD), which ensures fair comparison. Given our emphasis on relative object rotation estimation, we crop the objects from the original RGB image utilizing the ground-truth object bounding boxes (Xiao
al. 2019 Zhao et al. 2022b Park al.) 2020 Nguyen al. 2022 In Sec. 4.5 robustness against noise in the bounding boxes.
we evaluate
4.3 EXPERIMENTS ON CO3D
Let first evaluate approach in the benchmark used in (Zhang et al. 2022 Lin et al. 2023), which builds upon the CO3D dataset (Reizenstein et al.) 2021). All testing objects here are pre- viously unseen and the evaluation thus emphasizes the generalization ability. Table [ reports the results in terms of angular error and accuracy. Note that the results of SuperGiuee Regress, RelPose, and RelPose++ shown here align closely with the ones reported in (Lin et al.) (023). lending cred- ibility to the evaluation. More importantly, our method produces consistently more precise relative object poses, with improvements of at least 10% in angular error. This evidences the generalization ability of our approach to unseen objects.
4.4 EXPERIMENTS ON GROP
Let now develop the evaluation in our benchmark. Table 2 and Table provide the experimental results on Objaverse and LINEMOD, respectively. Our method also achieves superior generalization ability unseen objects, outperforming the previous methods by substantial margin. For instance,
7

---

Published as a conference paper at ICLR 2024
Reference
Query
SuperGlue
Regress
RelPose
RelPose++
Ours
2
2
S
Figure 3: Qualitative results Objaverse and LINEMOD. Here, assume the reference to be calibrated and visualize the object pose in the query. which derived from the estimated relative object pose. The predicted and ground-truth object poses indicated by blue and green arrows, respectively.
100
80
WVP
607
A 30 (%)
50
60
40
30
LaFTS
25P
Regress
RelPose
10
Reihose
Cara
15
30
60
90
105
120
135
150
165
180
0.10
0.15
0.25
Geodesic Distance
Noise Level
(a)
(b)
Figure 4: Robustness. (a) Acc @ 30 curves obtained with varying degrees of object pose variation between the reference and the query, measured by the geodesic distance. (b) Similar curves but for different levels of noise added to the object bounding boxes.
we achieve an improvement of at least 15.5% on Objaverse and 14.1 % on LINEMOD, measured in terms of Acc @ 15". Moreover, illustrate some qualitative results in Fig. To this end, assume the object pose R' in the reference to be available, and the object pose R in the query is computed R ARR". We represent the predicted and the ground-trui object poses blue and green arrows, respectively. This evidences that our method consistently yields better predictions. In the scenario where there is notable difference in object pose between the reference and query (as in the cat images in the third row), the previous methods struggle to accurately predict the pose for the unseen object, while our approach continues to deliver an accurate prediction.
4.5 ABLATION STUDIES
To shed more light on the superiority of our method, we develop comprehensive ablation studies on Objaverse and LINEMOD. Most of the experiments are conducted on LINEMOD since it is real dataset. As the two sparse views, i.e., a reference and query, might result in a large-scale object pose variation, we divide the Objaverse testing data into several groups based on the object pose variation between the reference and query, measured by geodesic distance. The task becomes progressively more
start the ablations by analyzing the robustness in such context. Specifically,
8

---

Published as a conference paper at ICLR 2024
w/o att.
w/o mask
w/ 2D mask
w/o agg.
RelPose=
Ours
Angular Error
41.9
42.1
42.6
41.9
59.7
41.7
Acc @ 30 (%) t
60.0
59.6
60.1
59,4
26.4
61.5
Acc @ 15 (%) 28.2
27.9
27.3
26.4
7.9
29.9
Table 4: Effectiveness of the key components in pipeline.
challenging as the distance increases. We developed this experiment on Objaverse because of its wider range of pose variations compared to LINEMOD. Fig. 4(b) shows the Acc @ 30 curves as the distance varies from 0 180" Note that all methods demonstrate satisfactory predictions when the distance is small, i.e., when the object orientations in the reference and query views are similar. However, the performance of feature-matching approaches, i.e., SuperGlue, LoFTR, and ZSP, dramatically drops as the distance increases. This observation supports our argument that the feature- matchin methods are sensitive to the pose variations. By contrast, our method consistently surpasses all competitors, thus showing better robustness.
As the object bounding boxes obtained in practice are inevitably noisy, we evaluate the robustness against the noise in this context on LINEMOD. Concretely, we add noise the ground-truth bound- ing boxes by applying jittering to both the object center and scale. The jittering magnitude varies from 0.05 to 0.30, which results in different levels of noise. The experimental results are shown in Fig. 4(b) where our method outperforms the competitors across all scenarios. This promising robustness underscores the possibility of integrating our method with existing unseen object detec- tors (Zhao et al. 2022a Liu et al. 2022). To showcase this, we extend our method to 6D unseen object pose estimation by combining with the detector introduced in (Liu al. 2022) and provide some results in the appendix.
Furthermore, we evaluate the effectiveness of the key components in our framework. The results on LINEMOD are summarized in Table where the evaluation of effectiveness encompasses four dis- tinct aspects: First, we develop counterpart by excluding self-attention and cross-attention layers (w/o att.) from the 3D reasoning blocks; Second, we modify the 3D masking by either omitting it (w/o mask) or substituting it with a 2D masking process over RGB images (w/ 2D mask); Third, directly compute the similarity of 3D volumes without utilizing the 2D aggregation module (w/o agg.): Fourth, we replace our 3D-aware verification mechanism with the energy- -based model (Zhang let 2022 Lin et al. 2023) (RelPose*),, while retaining our feature extraction backbone unchanged. The modified versions, namely w/o att., w/o mask, w/ 2D mask, and w/o agg., exhibit worse perfor- mance, which thus demonstrates the effectiveness of the presented components, i.e., attention layers, 3D masking, and the feature aggregation module. Additionally, the inferior results yield by RelPose* highlight that the 3D-aware verification mechanism contributes the high-accuracy predictions, in- stead of the feature extraction backbone in our framework. Consequently, this observation supports our claim that the proposed verification module facilitates the relative pose estimation for unseen objects by preserving the structural features and explicitly utilizing 3D information.
5 CONCLUSION
In this paper, we have tackled the problem of relative pose estimation for unseen objects. We assume the availability only one object image as the reference and aim to estimate the relative object pose between the reference and a query image. In this context, we have tailored the hypothesis- and-verification paradigm by introducing a 3D-aware verification, where the 3D transformation is explicitly coupled with learnable 3D object representation. We have developed comprehensive experiments on Objaverse, LINEMOD, and CO3D datasets, taking both synthetic and real data with diverse object poses into account. Our method remarkably outperforms the competitors across all scenarios and achieves better robustness against different levels of object pose variations and noise. Since our verification module incorporates local similarities when computing the verification scores, it could be affected by the occlusions. This stands as a potential limitation that we consider, and we intend explore and address this issue in our future research endeavors.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This work was funded in part by the Swiss National Science Foundation via the Sinergia grant CRSII5-180359 and the Swiss Innovation Agency (Innosuisse) via the BRIDGE Discovery grant 10B2-0_194729.
REFERENCES
Ronald T Azuma. A survey of augmented reality. Presence: leleoperators & virtual environments, 6(4):355-388) 13555-88,,10 1997.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.064450, 2016.
arXiv preprint
Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category- level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 11973-11982, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, 1597-1607. PMLR, 2020b.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa. The moped framework: Object recog- nition and pose estimation for manipulation. The International Journal of Robotics Research, 30 (10):1284-133) 2011.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Martin Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (0):381-395, 1981.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 3361- IEEE, 2012.
Walter Goodwin, Sagar Vaze, loannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. In Proceedings of the European Conference Computer Vision, pp. 516-532. Springer, 2022.
Agrim Gupta, Jiajun Wu. Jia Deng, and Li Fei-Fei. Siamese masked autoencoders. arXiv preprint arXiv:2305. 143444 2023.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022a.
10

---

Published as a conference paper at ICLR 2024
Xingyi He, Jiaming Sun, Yuang Wang. Di Huang, Hujun Bao, and Xiaowei Zhou. Onepose++: Keypoint-free one-shot object pose estimation without cad models. Advances in Neural Informa- tion Processing Systems, 35:35103-35115, 2022b
Stefan Hinterstoisssr,, Vincent Lepetit, Slobodan Ilie, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian Conference on Computer Vision, pp. 548-562. Springer, 2012.
Yinlin Hu, Pascal Fua, Wei Wang, and Mathieu Salzmann. Single-stage 6d object pose estimation. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2930-
2939, 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0006 2023.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Trem- blay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivie. MegaPose: 6D Pose Estima- tion of Novel Objects via Render & Compare. In CoRL, 2022.
Vincent Lepetit, Francese Moreno-Noguer. and Pascal Fua. Epnp: An accurate (n) solution to the pup problem. International Journal of Computer Vision, 81:155-166, 2009.
Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse- view observations. arXiv preprini urXiv:2305.04992. 2023.
Jiehong Lin, Zewei Wei, Changxing Ding, and Kui Jía. Category-level 6d object pose and size es- timation using self-supervised deep prior deformation networks. In Proceedings of the European Conference on Computer Vision, pp. 19-34. Springer, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 1-to--- Zero-shot one image to 3d object. arXiv preprint arXiv:2303.11328, 2023.
Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang- Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images. Proceedings of the European Conference on Computer Vision, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17Xiv:1711.05100 2017.
David G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91-110, 2004.
Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: hands-on survey. IEEE Transactions on Visualization and Computer Graphics, 22(12):2633- 2651, 2015.
Van Nguyen Nguyen, Yinlin Hu, Yang Xiao, Mathieu Salzmann, and Vincent Lepetit. Templates for 3d object pose estimation revisited: Generalization to new objects and robustness to occlusions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 6771-6780, 2022.
Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox. Latentfusion: End-to-end differen- tiable reconstruction and rendering for unseen object pose estimation. In Proceedings the conference on Computer Vision and Pattern Recognition, pp. 10710-10719, 2020.
the IEEE
Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4561-4570 2019.
11

---

Published as a conference paper at ICLR 2024
Giorgia Pitteri, Slobodan Ilce, and Vincent Lepetit. Comet: generic 3d corners for 6d pose estimation of new objects without retraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transac- tions on Pattern Analysis and Machine Intelligence, 44(3):1623-1637, 2020.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901-10911, 2021.
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2564 2571. Ieee, 2011.
Kuniaki Saito, Ping Hu, Trevor Darrell, and Kate Saenko. Learning to detect every thing in an open world. In Proceedings of the European Conference Computer Vision, pp. 268-284. Springer, 2022.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the JEEF/CVE Confer- ence on Computer Vision and Pattern Recognition, pp.
2020.
Johannes L Schonberger and Jan-Michael Frahm. Structure-front frotmmmmoooo-moooonn revisited. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4104 4113, 2016.
Ivan Shugurov, Fu Li, Benjamin Busam, and Slobodan Ilic. Osop: A multi-stage one shot object pose estimation framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6835-6844, 2022.
Yongzhi Su, Mahdi Salch, Torben Fetzer, Jason Rambach, Nassir Navab, Benjamin Busam, Didier Stricker, and Federico Tombari. Zebrapose: Coarse to fine surface encoding for 6dof object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoguition, pp. 6738 6738-6748, 2022.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. Onepose: One-shot object pose estimation without cad models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6825-6834, 2022.
Martin Sundermeyer, Maximilian Durner, En Yen Puang, Coltan-Csahle Marton, Narunas Vaskevi- cius, Kai O Arras, and Rudolph Triebel. Multi-path learning for object pose estimation across domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 13916-13925, 2020.
Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birch- field. Deep object pose estimation for semantic robotic grasping of household objects. In Confer- ence on Robot Learning, 2018. httthtts://aaaa org. wwtporgg////rrroo/eee//////////e 10790
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martin-Martin. Cewu Lu, Li Fei-Fei, and Silvio Savarese. Densefusion: 6d object pose estimation by iterative dense fusion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3343-3352, 2019a.
12

---

Published as a conference paper at ICLR 2024
Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Gdr-net: Ceometry-guidess di- rect regression network for monocular 6d object pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 16611-16621, 2021.
He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2642- 2651, 2019b.
Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao, and Shengjin Wang. Detecting everything in the open world: Towards universal object detection. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11433- 11443, 2023.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neu- ral network for 6d object pose estimation cluttered scenes. arXiv preprint arXiv:1711.0011 2017.
Yang Xiao, Xuchong Qiu, Fierre-Alain Langlois, Mathicu Aubry, and Renaud Marlet. Pose from shape: Deep pose estimation for arbitrary 3D objects. In British Machine Vision Conference (BMVC), 2019.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the TEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointiusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings of the IEEE conference on Computer Vision and Pattern Recog- nition, pp. 244 14-2553.4 2018.
Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1566-2674, 2018
Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In Proceedings of the European Conference on Computer Vision, pp. 592-611. Springer, 2022.
Chen Zhao, Yixiao Ge. Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann. Progressive correspondence pruning consensus learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6464-6477 2021.
Chen Zhao, Yinlin Hu, and Mathieu Salzmann. Loeposenet: Robust location prior for unseen object pose estimation, arXiv preprint 1xXiv:2211.162900 2022a.
Chen Zhao, Yinlin Hu, and Mathieu Salzmann. Fusing local similarities for retrieval-based 3d ori- entation estimation of unseen objects. In Proceedings of the European Conference on Computer Vision, pp. 106-122. Springer, 2022b.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2019.
Menglong Zhu, Konstantinos Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips. Matthieu Lecce, and Kostas Danillidis. Single image 3d object detection and pose es- timation for grasping. In Proceedings of the IEEE International Conference on Robotics and Automation, pp. 31139933399 3943. IEEE, 2014.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A ARCHITECTURE OF THE 3D REASONING MODULE
We show the architecture of the 3D reasoning module in Fig. Each 3D reasoning block consists of a self-attention layer and cross-attention layer, which excel at capturing intra-view and inter- view relationships, respectively. The input 2D feature map is flattened from RCXH,xW, to IR N where N H W.. A position embedding. denoted as PE, is added to the flattened feature map. Fig. 5() illustrates the attention layer. The context refers to the input feature map itself in the self-attention layer and represents the feature map of another view in the vross-attention layer. We use the standard multi-head attention (Vaswani et al. 2017) and layer normalization (Ba et al, 2016 in our attention layers.
B DATA CONFIGURATION
The synthetic images are generated by rendering objects of Objaverse from randomly sampled view- points (Liu et al. 2023). We attach these images to random backgrounds which are sampled from COCO Lin et al. 2014). We randomly sample 128 objects from Objaverse and use objects from LINEMOD sampled by Liu et al. (2022) as testing data, reserving the remaining objects for training. This design guarantees that all objects are previously unseen during the testing phase. We train the network on both synthetic and real data, alleviating the problem of domain gap.
Recall that we assume we have access to only one reference image and the objective is to estimate the relative object pose between the reference and the query. Therefore, the selection of the reference image is a crucial aspect of our benchmark. As multi-view images are available in Objaverse and LINEMOD datasets, one could randomly sample reference given query. However, such a strategy may yield an inappropriate reference. As shown in Fig. 66, the object depicted in the reference image barely overlaps with the one in the query, which makes the relative object pose estimation too challenging. Therefore, we filter out the inappropriate references from the datasets during training and testing, which makes our evaluation more reasonable.
Specifically, we convert the object rotation matrices R and RD to Euler angles (at,,, (Otr,,,, 7r) and (0g. Sg. 7/y,, which indicate azimuth, elevation, and in-plane rotation, respectively. Note that only azimuth and elevation lead to viewpoint changes, which thus determine the co-visible regions be- tween the reference and query. Consequently, we set the in-plane rotation and convert the Euler angle back the rotation matrix, i.e., R h(,, 8))). We then measure the difference of the new rotation matrices R' and πि by computing the geodesic distance. We exclude the image pair with a distance larger than a predefined threshold (90" by default in our experiments). As illustrated in Fig. 4 in our main paper, the retained image pairs display acceptable variations in object pose. Moreover, utilize the synthetic images Objaverse generated by Liu et al. (2023). Each 3D object model is rendered from 10 randomly sampled viewpoints, which yields synthetic images without in plapee rotations. To introduce in-plane rotations, rotate the reference and query images using randomly sampled 2D in-plane rotations during training and testing.
Fig. 7 shows the histograms of object pose variations between the reference and query images. We measure the variations based on the geodesic distance between the two object rotation matrices R2 and R The histograms show that the image pairs we used in our experiments exhibit diverse range of object pose variations, which makes our evaluation results convincing.
C QUALITATIVE RESULTS OF 6D OBJECT POSE ESTIMATION
We extend our method to 6D pose estimation for unseen objects by utilizing an off-the-shelf gener- alizable object detector (Liu et al. 2022). More concretely, instead of using dense-view reference images, we feed the one reference we have in our benchmark to the pretrained detection network, which predicts the object bounding box in the query image. We use the parameters of the object bounding box to compute 3D object translation, following the implementation in (Liu et al. 2022). Subsequently, we crop the object from the query and employ our approach to predict the relative 3D object rotation. The object rotation in the query derived as R ARR". Fssows some qual-
14

---

Published as a conference paper at ICLR 2024
PE
Solf
Cross Cross Att.
imput
FY
Fl+1
Linear Q
Multi-Head Attention
Layer Norm
Layer Norm
NXC
NXC
NxC
MLPs
context
Linear K
Self
F
F/
F(+11 NXC
NxC
NxC
Linear
(a) 3D reasoning block
(b) Attention layer
Figure 5: Architecture of the 3D reasoning module.
Reference Query
Figure 6: Examples of inappropriate references.
1400
3200
100
Pairs
800
600
600
40
200
700
20
60
160 100
70
40
67
10
100
131
140
160 185
60
60
100
Geodesic Distance
G Geodesic Distance
Geodesic Distance
(a) Objaverse
(b) LINEMOD
(c) CO3D
Figure 7: Histograms of the object pose variation between the reference and query. We measure the object pose variation the geodesic distance between the two object rotation matrices R and Rd. The histogram depicts the number of image pairs falling within different distance intervals.
itative results of 6D pose estimation for the unseen objects LINEMOD. We draw the 3D object bounding boxes in blue and green, using the predicted 6D object pose and the ground truth, respec- tively. The promising results demonstrate the potential of approach in terms of generalizable 6D object pose estimation.
D MORE DETAIL. ABOUT THE ABLATION STUDIES
As we introduced the main paper, we performed an ablation study, evaluating the robustness against the noise added to the 2D object bounding boxes. We simulate the bounding boxes in real- world applications by performing jittering to the ground truth with different levels of noise. We denote the object center and the size of the bounding box perturbed parameters from the intervals (c 0.5 n * c + 0.5 n * 8) and (T4m, (1 + n)), respectively, where n indicates the noise. We varied n from 0.05 to 0.3 in our experiments. Please refer Fig. (b) in main paper for the experimental results.
and We then randomly sample the
15

---

Published as a conference paper at ICLR 2024
donGer
DORS
Figure 8: Qualitative results of 6D pose estimation for unseen objects on LINEMOD. The blue and green 3D object bounding boxes are drawn using the predicted 6D object pose and the ground truth, respectively.
Verification Score
0.5
-0.5
-1
50
100
150
180
Geodesic Distance
Figure 9: Verification scores of all sampled pose hypotheses. The x-axis and y-axis represent the geodesic distance between the pose samplings and the ground-truth relative object pose, and the verification scores, respectively.
Method
RelPosc++
Ours
RelPosc++-5000
Ours-5000
MACs
94.6
54.7
11.3
16.3
Angular Error
38.5
28.5
50.7
35.3
Table 5: Efficiency. Relpose++ uses 500,000 pose samples by default, while we sample 50,000 poses for our method in our experiments. RelPose++-5000 and Ours-5000 denote RelPose++ and our method with 5,000 pose samples, respectively. The multiply-acumuulte operations (MACs) are used to measure the computation consumption.
E EFFICIENCY
is worth noting that during testing, our method utilizes 50,000 pose samples, while RelPose++ uses 500,000. Despite processing fewer samples, method achieves better accuracy in relative object pose estimation. To further evaluate the efficiency, we measure the computation cost in multiply- accumulate operations (MACs) and show the results in Table 5 All evaluated methods process the pose samples in parallel. "RelPosc++-5000 and "Ours-5000" refer to RelPose++ and our method with 5,000 samples, respectively. The results clearly show that our method achieves a better trade- off between efficiency and accuracy in relative object pose estimation. Additionally, our method with only 5,000 samples still delivers more accurate results than RelPoset+ with 500,000 samples.
16


Reviews:
[{"review_id": "YdmyFPMChk", "reviewer": "Reviewer_nmWB", "summary": "The paper addresses the problem of estimating the pose of an object observed in a query image relative to a reference image. A novel method is presented for estimating the relative 3D orientation of the object. Multiple orientation hypotheses are sampled and scored using the proposed verification strategy. The verification strategy relies on extracting 3D feature volumes from the query and reference image, and measuring similarity between the 3D volume of the query with the 3D volume of the reference transformed with the 3D orientation of the hypothesis. The comparison of the 3D volumes is done using a novel feature aggregation strategy to gain robustness with respect to the background. The paper also introduces a new benchmark for this task, relying on synthetic images of objaverse objects, and real images of the LINEMOD dataset.", "questions": "The method presented in this paper is well designed and demonstrated to be effective for estimating the relative 3D orientation of an object observed in two different images. I however currently have questions regarding the quantitative comparison with state-of-the-art (W3), and concerns regarding the practical applications of such approach (W1) and it’s runtime (W2). I will gladly increase my rating if these concerns can be addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "S1. The paper is well easy written and easy to understand.\n\nS2. The method can operate from RGB images, and only requires a single reference image.\n\nS3. The technical contributions are sound and demonstrated to be effective in an ablation study (Table 4 of the main paper). In particular, the attention layers to extract 3D features capturing information from both query and reference images; and the aggregation mechanism to gain robustness with respect to the background are well motivated. \n\nS4. The method combines several interesting design choices which are demonstrated to lead to higher relative rotation estimation accuracy compared to several baselines (Table 1). However it is not clear if the comparison with RelPose and RelPose++ is fair (see weakness W3).", "weaknesses": "W1. The object pose estimation setup addressed in this paper should be better motivated. If a CAD model of the object is available, multiple reference images can be rendered or other methods applied directly, e.g. [A,B,C,D,E,F]. In this paper, availability of a CAD model of the object is not assumed, but obtaining the pose of the object with respect to the camera in the query image would still require the pose of the object to be known in the reference image. It is not clear how the reference pose would be obtained in practice without assuming the CAD model is known. In addition, the method presented and evaluated in this paper does not estimates the absolute or relative 3D translation of the object between the query and reference images. Overall, the paper would be more convincing if practical applications of the presented approach were discussed.\n\n[A] DeepIM: Deep Iterative Matching for 6D Pose Estimation. Li et al, ECCV 2018.\n\n[B] Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects. Xiao et al, 3DV 2019.\n\n[C] Multi-path Learning for Object Pose Estimation Across Domains. Sundermeyer et al, CVPR 2020.\n\n[D] CorNet: Generic 3D Corners for 6D Pose Estimation of New Objects without Retraining. Pitteri et al, ICCW 2021.\n\n[E] OSOP: A Multi-Stage One Shot Object Pose Estimation Framework. Shugurov et al, CVPR 2022.\n\n[F] MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare. Labbe et al, CoRL 2022.\n\nW2. The runtime of the approach is not mentionned. 50000 hypotheses are evaluated during testing, what is the tradeoff between accuracy and runtime ?\n\nW3. Is it not clear if the RelPose, RelPose++ and the presented approach were all trained using the same images for the comparison on CO3D presented in Table 1. The supplementary material mentions that the approach is trained using objects from the objaverse dataset. Is it also the case for the experiments in Table 1 ? \n\nMinor details:\n- Some papers have incorrect citations, e.g. Gen6D is an ECCV paper, DOPE is a CoRL paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ZNRLanSJW", "reviewer": "Reviewer_c3Qb", "summary": "The paper proposes an object pose estimator that can estimate relative pose with a single reference view.\nTo tackle the large viewpoint change scenarios, they propose a hypothesis-and-verification framework for robust pose estimation.\nThe image features and 3D feature volumes are extracted for each image first.\nThen the poses are sampled and used to warp 3D feature volumes to the reference view, where the verification scores are regressed by transformer architecture. Pose is estimated by selecting the hypothesis with the largest similarity score.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed hypothesis-and-verification framework seems to be effective especially for large viewpoint change scenarios, compared with matching-based methods.\n2) The experiments and ablation studies are thorough.\n3) The overall writing is clear and easy to follow.", "weaknesses": "1) The generalizability across datasets. The proposed method is generalizable and can handle unseen objects. I'm curious about the generalizability across datasets. For example, the proposed method is trained on the CO3D dataset, but how about the generalizability of the LINEMOD dataset? I think the generalizability across datasets is important for the proposed method, especially for real-world applications.\n\n2) Running time comparisons. The paper needs to sample M=50000 poses for verification at test time. I'm concerned about the efficiency of estimating a single pose, even if the verifications are parallelized. I think the running time comparisons with other methods are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2CMpblNlG5", "reviewer": "Reviewer_jWeY", "summary": "This paper addresses the problem of relative pose estimation for unseen objects. The authors assume that only one object image as the reference is available and they aim to estimate the relative object pose between the reference and a query image. To this end, the authors propose a hypothesis-and-verification paradigm by introducing a 3D-aware verification. In particular, 3D transformation is explicitly coupled with a learnable 3D object representation. Experiments were performed on Objaverse, LINEMOD, and CO3D datasets, taking both synthetic and real data with diverse object poses into account.", "questions": "Questions for the author:\n\n(1) In what scenarios or applications is using only one reference image necessary for the pose estimation task? Why this is a meaningful problem setting?\n\n(2) How it is possible to obtain reliable 3D volume representation for an unseen object from only its two views, which may even present large variations in view angle and lighting?\n\n(3) Is it possible to show some visualizations of the intermediate results of the 3D reasoning component?\n\n(4) What are the remaining error cases of the model for the defined task? \n\n(5) What backbone network is used? Does the backbone network selection affect model performance?\n\n(6) What is the computational cost and speed of this method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Object pose estimation is a central task for 3D computer vision. Generalizable object relative pose estimation allows dealing with unseen object and is important to apply the technology to real world applications. This paper addresses this task along this direction and specifically focus on the cases that only one reference image is available. This case is difficult, especially when the object image appearance is view-dependent and the query and reference images have very different viewpoints. Despite of the difficulties, the proposed methods shows significant performance improvement than SOTA baselines on several public datasets including Objaverse, LINEMOD, and CO3D. The paper is well organized.", "weaknesses": "Weakness of the paper: \n\n(1) Although the task setting chosen by the paper is challenging for SOTA methods, the motivation of this task setting is not clearly explained. Using only one reference image can pose great difficulty for the pose estimation task. An extreme example is that the reference image is taken from frontal viewpoint and a query image is taken from rear viewpoint, which may share no common features to match at all. Meanwhile, with any modern camera device, one can actually easily capture multiple images as use them as reference. In what scenarios, and why using only one reference image is critical is not mentioned in the paper.\n\n(2) An important component of the method is the 3D reasoning model that learns the 3D volumes from 2D feature maps of two RGB image inputs. However, this is itself a very challenging research topic about open-set 3D volume representation reconstruction with two RGB inputs. It’s difficult to understand why it is possible to obtain reliable 3D representation for an unseen object from only its two views, which may even present large variations in view angle and lighting. Also, experimental verification for this component is missing. It would be informative to show some visualizations of the intermediate results of this component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7pot0b70Cz", "reviewer": "Reviewer_jg4A", "summary": "This paper proposed a 3D latent object representation and 3D-aware hypothesis verification framework for general object relative pose estimation from two 2D images. The latent 3D representation is obtained by lifting the 2D feature map into 3D space and the 3D-aware hypothesis are generated and verified in a RANSAC-like manner. Experiment has been done on public benchmark dataset CO3D, Objaverse and LINEMOD and the proposed method achieved superior performance compared with previous SOTA.", "questions": "1. The number of hypothesis during training and testing is huge, 9000 for training and 50000 for testing, how about the computational cost? why they are different?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method combined several modules to construct an interesting pipeline for general object relative pose estimation, including 2D-to-3D feature lifting, 6D continous rotation representation hypotheses sampling, RANSAC-style hypothsis verification. \n2. The achieved performance improvement over RelPos++ is impressive\n3. Pose estimation robustness in case of background clutter and object motion are handled with modules of 3D masking, feature aggregation, etc.", "weaknesses": "1. The paper claims that the method can be used for unseen object relative pose estimation, but it actually may rely on object detection to get relevant object area for pose estimation, and there is no reliable unseen object detector available\n2. The presentation is not clear and sometimes confusing, in 4.5 ablation study section, the method of 'RelPose*' should be explicitly explained, and it seems that the attention module, mask module and aggregation module donot result in significant performance improvement, it would be clearer if we start from baseline and add the proposed modules step by step to show their effectiveness\n3. There are some typos, Sec.3.1, 'as illustrated in Fig.3' -> 'as illustrated in Fig.2'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)