OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models
Authors: Xingyi He
Jiaming Sun
Yuang Wang
Di Huang
Hujun Bao
Xiaowei Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Object pose estimation
Local feature matching
Structure from motion

Abstract: We propose a new method for object pose estimation without CAD models. The previous feature-matching-based method OnePose has shown promising results under a one-shot setting which eliminates the need for CAD models or object-specific training. However, OnePose relies on detecting repeatable image keypoints and is thus prone to failure on low-textured objects. We propose a keypoint-free pose estimation pipeline to remove the need for repeatable keypoint detection. Built upon the detector-free feature matching method LoFTR, we devise a new keypoint-free SfM method to reconstruct a semi-dense point-cloud model for the object. Given a query image for object pose estimation, a 2D-3D matching network directly establishes 2D-3D correspondences between the query image and the reconstructed point-cloud model without first detecting keypoints in the image. Experiments show that the proposed pipeline outperforms existing one-shot CAD-model-free methods by a large margin and is comparable to CAD-model-based methods on LINEMOD even for low-textured objects. We also collect a new dataset composed of 80 sequences of 40 low-textured objects to facilitate future research on one-shot object pose estimation. The supplementary material, code and dataset are available on the project page: https://zju3dv.github.io/onepose_plus_plus/.

OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models
Xingyi Hel
Jiaming Sun Hujun Bao
Yuang Wang' Xiaowei Zhou
Di Huang
"Zeejiang University
Image Derivative Inc.
The University of Sydney
Abstract
We propose a new method for object pose estimation without CAD models. The previous feature-matching- www-mmaaatthaaaeeeeeme based method OnePose [48] has shown promising sults under a one-shot setting which eliminates the need for CAD models or object-specific training. However, OnePose relies on detecting repeatable im- age keypoints and is thus prone to failure on low-textured objects. We propose keypoint-free pose estimation pipeline remove the need for repeatable key- point detection. Built upon the detector-frre feature matching method LoFTR [47]. we devise new keypoint-free SfM method to reconstruct a semi-dense point-cloud model for the object. Given query image for object pose estima- tion, a 2D-3D matching network directly establishes 2D-3D correspondences between the query image and the reconstructed point-cloud model without first detecting keypoints in the image. Experiments show that the proposed pipeline outperforms existing one-shot CAD-model- methods by large margin and is comparable to CAD-model-bases methods on LINEMOD even for low-textured objects. We also collect a new dataset composed of 80 sequences of 40 low- textured objects to facilitate future research on one-shot object pose estimation. The supplementary material, code and dataset available the project page: https://2juuee wwtpasdd..itttub..ee io/ /owp/oeepeee.ee/// pateeeeppss____ __luus_pu/s....
Introduction
Object pose estimation is crucial for immersive human-object interactions in augmented reality (AR). The AR scenario demands the pose estimation arbitrary household objects in our daily lives. However, most existing methods [39, 29, 38, 55, 2, 37] either rely high-fidelity object CAD models or require training separate network for each object category. The instance- or category-specific nature of these methods limits their applicability in real-world applications.
To alleviate the need for CAD models or category-speeific training, OnePose [48] proposes a new setting of one-shot object pose estimation. It assumes that only video sequence with annotated object poses is available for each object and aims for its pose estimation in arbitrary environments. This setting eliminates the requirements for CAD models and the separated pose estimator training for each object, and thus more widely applicable for AR applications. OnePose adopts the feature- matching- based visual localization pipeline for this problem setting. It reconstructs sparse object point clouds with SfM [44] and establishes 2D-3D correspondences between keypoints in the query image and the point cloud model estimate the object pose. Being dependent on detecting repeatable
'The first authors contributed equally. The authors from Zhejiang University affiliated with the State Key Lab CAD&CG and the 7JU-SenseTimee Joint Lab of Vision. "Coresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Ref Images
Reconstructed SIM Point Cloud
Object Pase Estimation for Query Images
20113 poins
pontts
paints
972 palnts
Ours
OnePose
Ours
OnePose
Figure I: Comparsion Between Our Method and OnePose [48]. For low-textured objects that challenging for OnePose, our method can reconstruct their semi-dense point clouds with more complete geometry and thus achieves more accurate object pose estimation. Green and blue boxes represent ground truth and estimated poses, respectively.
keypoints, OnePose struggles with low-textured objects whose complete point clouds difficult to reconstruct with keypoint-baseh SIM. Without complete point clouds, pose estimation is prone to failure for many low-textured household objects.
We propose to use keypoint- free feature matching pipeline on top of OnePose to handle low-textured objects. The keypoint- frree remi-dense feature matching method LoFTR [47] achieves outstanding performance on matching image pairs and shows strong capabilities for finding correspondences in low-textured regions. It centers of regular grids sub-pixel accuracy matches on the right image in coarse-to-fine manner. However, this two-view- dependent nature leads to inconsistent "keypoints" and fragmentary feature tracks, which go against the preference of modern SfM systems. Therefore, keypoint-free feature matching cannot be directly applied to OnePose for object pose estimation. We will further elaborate this issue in Sec. 3.1.
left image as "keypoints" and extracts
To get the best of both worlds, we devise a novel system to adapt keypoint-free matching for one-shot object pose estimation. We propose two-stage pipeline for reconstructing 3D structure, striving for both accuracy and completeness. For testing, we propose a sparse-to- to-se-s 2D-3D matching network that efficiently establishes accurate 2D-3D correspondences for pose estimation, taking full advantage of our keypoint-free frree design.
More specifically, to better adapt LoFTR [47] for SfM, we design a coarse-to-fine scheme for accurate and complete semi-dense object reconstruction. We disassemble the coarse-to-fine structure of LoFTR and integrate them into reconstruction pipeline. In the coarse reconstruction phase, use less accurate yet repeatable LoFTR coarse correspondences to construct consistent feature tracks for SfM and yield an inaccurate but complete semi-dense point cloud. Then, our novel refinement phase optimizes the initial point cloud by refining "keypoint" locations in coarse feature tracks to sub-pixel accuracy. As shown in Fig. 1, our framework can reconstruct accurate and complete semi-dense point clouds even for low-textured objects, which lays the foundation for building high-quality 2D-3D correspondences for pose estimation.
At test time, we draw inspiration from the sparse-to- torrse-to-desse matching strategy in visual localization [12], and further adapt it to direct 2D-3D matching in coarse-to- fine manner for efficiency. Additionally, we use self- and cross-altention to model long-range dependencies required for robust 2D-3D matching and pose estimation of complex real-world objects, which usually contain repetitive patterns or low-textured regions.
We evaluate our framework on the OnePose [48] dataset and the LINEMOD [16] dataset. The experiments show that our method outperforms all existing one-shot pose estimation methods [48, 33] by a large margin and even achieves comparable results with instance-level methods [39, 29] which are trained for each object instance with CAD model. To further evaluate and demonstrate the capability of our method in real-world scenarios, we collect new dataset named OnePose-Lood Loweettue,, which comprises 80 sequences of 40 low-textured objects.
Contributions.
A keypoint-free SfM method for semi-dense reconstruction of low-textured objects.
A sparse-to-dense 2D-3D matching network for accurate object pose estimation. challenging real-world dataset OnePose-LLow Lew'uuure composed of 40 low-textured objects with ground-truth object pose annotations.

---

2
Related work
CAD-Model-Based Object Pose Estimation. Many previous methods leverage known object CAD models for pose estimation, which can be further categorized into instance-level, category-level. and generalizable methods by their generalizability. Instance- level methods estimate object poses either by directly regressing poses from images [58, 20, 29] or construct 2D-3D correspondences and then solve poses with PnP [39, 59]. The primary deficiency is that these methods need to train a separate network for each object. Category-level methods, such as [55, 51, 21, 54, 56, 4], learn the shape prior shared within a category and climinate the need for CAD models in the same category at test time. However, these methods cannot handle objects in unseen categories. Some recent methods leverage the generalization power of 2D feature matching for the pose estimation of unseen objects. Reference images are rendered with CAD models and then matched with the query image, using either sparse keypoints matching [61] or dense optical flow [45]. All methods mentioned above depend on high-fidelity textured CAD models for training or rendering, which are not easily accessible in weal-world applications. Our framework, instead, reconstructs a 3D object model from pose-annotated images for object pose estimation.
CAD-Model-Free Object Pose Estimation. Some recent methods get rid of the CAD model completely. RLLG [2] uses correspondences between image pairs as supervision for training, without known object model. However, it still requires accurate object masks supervision, which are not casily accessible without a CAD model. NeRF-Pose [24] reconstructs a neural representation NeRF [36] for an object first and train an object coordinate regression network for pose estimation. These methods are not generalizable to unseen objects. The recently proposed Gen6D [33] and OnePose [48] only require set of reference images with annotated poses estimate object poses and can generalize to unseen objects. Gen6D uses detection and retrieval to initialize the pose of a query image and then refine it by regressing the pose residual. However, Gen6D requires accurately detected 2D bounding box for pose initialization and struggles with occlusion scenarios. OnePose reconstructs objects' sparse point cloud and then extracts 2D-3D correspondences for solving poses. It performs poorly on low-textured objects because of its reliance on repeatably detected keypoints. Our work is inspired by OnePose but climinates the need for keypoints in the entire framework, which leads to better performance on both textured and low-textured objects.
Notably, leveraging feature matching for object pose estimation is a previous methods [9, 35, 14, 17, 46] extract keypoints on the query image first and perform matching with reference images SfM model to obtain 2D-3D matches for pose estimation. The main challenges are the ambiguous matches incurred by low-textured and repetitive patterns. They either rely on the ratio test in the matching stage [35, 14, 46) or leverage prioritized hypothesis testing in the outlier filtering stage [9, 17] to reject ambiguous matches. Different from them, our framework climinates the keypoint detection for the query image by directly performing matching between the 2D feature map and the 3D model, which benefits pose estimation for low-textured objects. Moreover, we leverage the attention mechanism to disambiguate 2D and 3D features for matching, while the direct feature disambiguation is not explored by these methods.
problem. Some
Structure from Motion and Visual Localization. Visual localization estimates camera poses of query images relative to a known scene structure. The scene structure is usually reconstructed with Structure-from-Morroe (SfM) [44], relying on the feature matching methods [34, 6, 62, 25, 41, 13]. The localization problem is then solved by finding 2D-3D correspondences between the 3D scene model and query image. HLoc [40] scales up visual localization in coarse-to-fine manner. It is based on image retrieval and establishes 2D)3D correspondences by lifting 2D-2D matches between query images and retrieved database images to 3D space. However, HLoc is not suitable for our setting since it is slow during pose estimation because it depends on 2D-2D matching of multiple image pairs as the proxy to locate one query image. Our framework is more relevant to the previous visual localization methods which are based on efficient direct 2D-3D matching [43, 26, 49, 32, 3, 60, 27, 5, 7, 50, 42, 12]. To boost matching efficiency between the large-scale point cloud and query images, some of them narrow the searching range by leveraging the priors [43, 26, 5] or compress 3D models by quantizing features [43, 32, 42]. However, these strategies contribute little to the disambiguation of features. They often use priors [43, 32, 27, 42] or geometric verification [49, 3, 60, 50] in the outlier filtering stage to cope with the challenges from low-textured regions or repetitive patterns. In
contrast, our method works on the 2D-3D matching phase but focuses disambiguating features
3

---

(Py)
Object Point Cloud
Matching Network
MaD
2D-30 correspondencee
Pose
Coarse Reconstruction
Refinement
EL 1. Keypoint-Free Structure from Motion
Query Image
Object Pose Estimation
Overview. 1. For each object, given reference image sequence {I, } with known object poses } our keypoint-free SiM framework reconstructs the semi-dense object point cloud in a coarse-to-fine manner. The coarse reconstruction yields the initial point cloud (+) which is then optimized to obtain an accurate point cloud (+) in the refinement phase. 2. At test time, our 2D-3D matching network directly matches reconstructed object point cloud with a query image I, to build 2D-3D correspondences MaD, and then the object pose &. is estimated by solving PnP with Man.
Figure 2:
before matching. Our idea is to directly disambiguate and augment dense features by implicitly encoding their spatial information and relations with other features in learning- based manner.
Our keypoint- free SfM framework is related to SfM refinement methods PatchFlow [8] and PixSMM [31]. They improve keypoint-basedd SfM for more accurate 3D reconstructions by re- fining imaccuraeely-deteeted sparse local features with local patch flow [8] or dense feature maps [31]. Different from them, we leverage fine-level matching with Transformer [53] to refine the 2D locations of coarse feature tracks and then optimize the 3D model with geometric error. Please refer to the supplementary for more detailed discussions. [57] also works keypoint-free SfM. However, refines the coarse matches by the keypoint relocalization, which is single-view dependent and faces the issue of inaccurate keypoint detection. In contrast, our refinement leverage two-view patches to find accurate matches. Note that there are also methods proposed by keypoint-free matchers [47, 62] adapt themselves for SfM. They either round matches grid level [47] or merge matches within a grid to the average location [62] to obtain repeatable "keypoints" for SIM. However, all these ap- proaches trade-off point accuracy for repeatability. On the contrary, our framework obtains repeatable features while preserving the sub-pixel matching accuracy by the refinement phase.
3 Methods
An overview of our method is shown in Fig. 2. Given reference image sequence with known object poses [I,, E. our objective is to estimate the object poses {E,) for the test images, where i and q denote the indices of the reference images and test images, respectively. To achieve this goal, we propose a novel two-stage pipeline, which first reconstructs the accurate semi-dense object point cloud from reference images (Section 3.2), and then solves the object pose by building 2D-3D correspondences in coarse-to-fine manner for test images (Section 3.3). Since our method is highly related to the keypoint-free matching method LoFTR [47], we give it short overview in Section 3.1.
3.1 Background
Keypoint-Free Feature Matching Method LoFTR [47). Without keypoint detector, LoFTR builds semi-dense matches between image pairs (noted as left and right images) in a coarse-to-fine pipeline. First, dense matches between two coarse-level feature maps (V/ resolution in LoFTR) built and upsampled, yielding coarse semi-dense matches in the original resolution. With the locations of all left matches fixed, the right matches are refined maps. Thanks to the keypoint-frre design and the global receptive field of Transformers, LoFTR is capable of building correspondences low-textured regions.
sub-pixel level using fine-level feature
Problem of Using LoFTR for Keypoint- SfM. Directly combining LoFTR with modern keypoint-basee StM systems such as COLMAP [44] is not applicable since they rely on fixed keypoints detected on each image to construct feature tracks for estimating 3D structures. However, for LoFTR, matching locations on a right image depend on its pairing left images. Therefore, right matching locations are not consistent when paired with multiple left images. Due to this reason, keypoint-free feature matching cannot establish feature tracks across multiple views for effective 3D structure optimization in SIM and is thus not directly applicable in OnePose.

---

P
1. Coarse Matching Pairs 2. Coarse Reconstruction
3. Feature Track Refinement 4.Point Cloud Optimization
Figure 3: Keypoint- Free SfM. 1. We first build repeatable coarse semi-dense 2D matches between image pairs. 2. Then, we feed coarse matches to COLMAP [44] to build a coarse feature track T and coarse 3D point P (+). 3. To refine Te', we fix a reference node u, (O) and search around the local window (C) of each source node u (O) for sub-pixel correspondences ú (O). 4. Finally, we optimize the depth d, of u, by minimizing reprojection errors. We back-prope pack-project u, with its refined d, to the object coordinate to obtain an optimized accurate object point cloud pá (0).
3.2 Keypoint-Free Structure from Motion
To better adapt LoFTR for SIM, we design coarse-to-fine SfM framework leveraging the properties of LoFTR's s coarse and fine stages separately. Our framework constructs the coarse structure of the feature tracks (T2) } and point cloud (P.) in the coarse reconstruction phase. Then in the refinement phase, the coarse structures are refined to obtain the accurate point cloud (P'). J. For clarity, in this part, use : to denote the coarse matching results and use to denote fine matching results. We consider the feature track T = {u e &²I = I...NN}} } as a set of matched 2D points observing a 3D point P R23 j denotes the index of the feature track and its corresponding 3D point.
Coarse Reconstruction. We first strive for the completeness of the initially reconstructed 3D structure. We propose to use the inaccurate yet repeatable coarse correspondences of LoFTR for COLMAP [44] to reconstruct the coarse 3D structure. The coarse correspondences, as shown in Fig. (1), can be seen as pixel-wise dense correspondences on downsampled image pairs. Every pixel in the downsampled image can be regarded "keypoint" in the original image. Therefore, performing coarse matching can provide repeatable semi-dense correspondences for COLMAP to reconstruct coarse feature tracks {T²} and semi-dense point cloud [P?]. as shown in Fig. 3 (2).
Refinement. Due to the limited aceuracy of performing matching on downsampled images, the point cloud from the coarse reconstruction is inaccurate and thus insufficient for the object pose estimation. Therefore, we further refine the object point cloud [P/] with sub-pixel correspondences. To achieve this, we first fix the position of one node for each feature track Te' and refine other nodes within the track. Then,
the refined tracks (7 to optimize the (P!)-
For the refinement of {T2), we draw the idea from the fine-level matching module in LoFTR and adapt it to the multi-view scenario. As shown in Fig. 3 (3), we first select and fix one node in each T as the reference node u,, and then perform fine matching with each of the remaining source node u The fine matching searches within a local region around each ü for a sub-pixel correspondence ú so the nodes" locations in the coarse feature track are refined. We denote the refined feature tracks as {T/). J. Details about the selection reference nodes u, are provided in the supplementary material.
We now treat the refined feature tracks (Ty) fixed measurements, and optimize the 3D locations of the coarse point cloud (P!} using reprojection errors as shown Fig. (4). To accelerate the convergence, inspired by SVO [11], we further decrease the DoF of each P by only optimizing the depth d, of each reference node u, Specifically, we transform each point P to the frame of u, and use its coordinate of z-axis to initialize dr. Then optimize each reference node depth d, by minimizing the distance between each reprojected location and the refined feature location út:
d; argmin
Σ | (E--4 n (u,) 1f=
KEW,11
E-- is the where π is the projection determined by intrinsic camera parameters, and E E relative pose between the frame the reference node and k-thh source node.
Finally, the optimized depth d; of each reference node is transformed to the canonical object coordinate to get the refined 3D point P Notably, when applying the proposed system in practical AR applications, we can optimize inaccurate camera poses obtained from ARKit along with the 3D points, i.e., solving bundle adjustment problem. For the later 2D-3D matching at test time, we calculate and store each 3D point feature by averaging the 2D features of its associated 2D points.
5

---

FaD
Self
Cross
MaD
F
(P)
FaD
Fa
MiD
sult Cross Attention
Correlation
Object Point Cloud
PnP
FED
crog
Expectation
u
Frep
Fltten
Coarse Matching
2. Fine Matching
Query Image
22
Pose
Figure 4: Object Pose Estimation. At test time, first extract multi- query image features (F2D, F2D). Coarse Matching module transforms coarse 2D and 3D features N, times with self- and cross-attention modules and then build their coarse 2D-3D correspondences MaD Next, we crop the local window F crop the fine feature map around each coarse 2D) match. Fine Matching module transforms the 3D feature and cropped 2D features and calculates each 2D fine match location i with feature correlation and expectation. The object pose E,y is then solved using PnP with MaD
Note that we store coarse and fine 3D features separately, which are extracted from multi rultiireesoluion feature maps of LoFTR's 's feature backbone.
3.3 Object Pose Estimation
At test time, establish 2D-3D matches between the object point cloud {P'} and the query image I, to estimate object pose Eg. Inspired by [47], we first extract hierarchical feature maps of I, and then perform matching in coarse-to-fine manner for efficiency, as illustrated in Fig. 4.
Coarse 2D-3D Matching. We first perform dense matching between the pre-calculated coarse 3D point features F3D E NxC. and the extracted coarse image feature map F2D # xC. This phase globally searches for a rough correspondence of each 3D object point in the query image, which also determines whether the 3D point is observable by Ly.
We augment 3D and 2D features FAD, F2D) with positional encodings, to make them position- dependent, and thus facilitates their matching. Please refer to the supplementary material for more details. Then we flatten the 2D feature map and apply self- and cross-attention layers by No times to yield the transformed features [F Fr F Linear Attention [19] is used in our model reduce the computational complexity, following [47]. A score matrix S is calculated by the similarity between two sets of features F1 and FiD- We then apply dual-softmax operation [52] on S to get the matching probability matrix p.:
(2) (.-)) is the inner product, J is scale factor, and and q denote the indices of a 3D point and pixel the flattened query image, respectively. The coarse 2D-3D correspondences MDD are established from pe by selecting correspondences above confidence threshold 0:
P 9) softmax (S softmax (S (8(,))), where S (jj = (FF) Fin(1)).
MGp (G. 4) V(i.g) MNN (P"), P (i. 2 0).
MNN refers to finding the mutual nearest neighbors. We use this strict criterion to suppress potential false matches.
Fine Matching. For a visible 3D point Pi determined by MaD, our fine matching module searches for its sub-pixel 2D correspondence ń within the local region of its coarse correspondence i Similar to [47], we crop a local window W with a size w x w around i the fine feature map FED E IR: # xC,, Then the cropped feature map F Forop E R= and corresponding 3D fine feature FID RC, transformed by N, self- and cross- attention layers. We correlate the transformed 3D fine feature vector F% with all elements in transformed 2D feature F Foropp and apply softmax to get the probability distribution of its 2D correspondence in the cropped local window:
exp (P) F
p(ulj, F F Frrop) Euew exp (P) (ro (u))
The fine correspondence P on the query image is then obtained with an expectation:
G o Eue w p(u P Ferop).
(
After building the 2D-3D correspondences M between the query image and the object point cloud, solve the object pose E, with the cerppecivve--Poin (PnP) [22] algorithm and RANSAC [10].

---

Data Annotate
2. Example Images
3. Scanned Models
& Capture
Figure 5: Data capture and example images of the proposed LesPoss-LLwTexturr dataset.
Supervision. We jointly train the coarse and fine modules of 2D-3D matching framework with different supervisions, following [47]. The ground-truth 2D-3D correspondences M and the coarse matching probability matrix Ps are obtained by projecting observable SIM points to the 2D frame with the ground-truth object pose. We optimize the coarse matching module by minimizing the focal loss [30] between the predicted pe and P For the fine module, we minimize the & loss between the predicted 2D coordinate ú and the ground truth ig The total loss is the weighted sum of coarse and fine losses. More details are provided in the supplementary material.
3.4 Implementation Details
Keypoint-Free SIM. The COLMAP [44] triangulation is used to construct the coarse 3D struc- ture. In the refinement phase, the local window with size of 9 is searched for the sub-pixel correspondence of each reference node. The Levenberg-Marquard algorithm [23] implemented in DeepLM [18] is used to optimize the coarse object point cloud. We use the LoFTR [47] outdoor model pre-trained on MegaDepth [28]. Running time analyses are given in the supplementary material.
Object Pose Estimation. We use ResNet-18 [15] as the image backbone and set N₂ = 3, Ny = for the 2D-3D attention module. The scale factor is 0.08, the cropped window size w in the fine level is 5, and the confidence threshold 0 is set to 0.4. As for training, the backbone of our model initialized with the LoFTR outdoor model. and the remaining parts of our model use randomly initialized weights. The entire model is trained on the OnePose training set, and we randomly sample or pad the reconstructed point cloud to 7000 points for training. We use the Adam W optimizer with an initial learning rate of 4 x 10-3 The network training takes about 20 hours with batch size of 32 on NVIDIA- V100 GPUs. During testing, we use all reconstructed 3D points (~ 15000) for building 2D-3D correspondences, and our 2D-3D matching module takes 88ms for a 512 x 512 query image on a single V100 GPU.
4 Experiments
4.1 Datasets
OnePose and LINEMOD Datasets. We validate our method on the OnePose [48] and LINEMOD [16] datasets. The OnePose dataset is newly proposed, which contains around 450 real-world video sequences of 150 objects. LINEMOD is a broadly used dataset for object pose estimation. For both datasets, we follow the train-test split previous methods [48, 29].
OnePose-LoWTextur Dataset. Since the original OnePose evaluation set mainly comprises tex- tured objects, we collected an additional test set, named original OnePose dataset. The proposed dataset is composed of 40 household low-textured objects. For each object, there are two corresponding videos captured with different backgrounds, one as the reference video and the other for testing. Besides, to evaluate and compare our method with CAD--odellbbase methods, we further obtain high-fidelity 3D models of eight randomly selected objects with a commercial 3D scanner. Some example images are shown Fig. Please refer the supplementary material for more details.
LowTexture to supplement the
7

---

Table 1: Comparison with One-shot Baselines. Our method is compared with HLoc [40] combined with different feature matching methods and OnePose [48], using the cm-degree pose success rate with different thresholds.
UneFuse dataset
OnePose- Low Texture
Time (ms)
Tem- Ideg
Scm Sileg
Sem Sdeg
Icm- Tdleg
Jcm. Jdeg Sem--tdeg
of (SPP SPG)
31.1
75.9
82.0
13.8
36.1
42.2
835
[LLae (LoFTR))
39.
72.3
80.4
13.2
41.3
52.3
909
OnePose
49.7
77.5
35.7
45.4
66.4
Ours
51.1
80.8
87.7
16.8
57.7
72.1
88.2
Table 2: Comparison with Instance-level Baseline. Our method is compared with PVNet[39] on objects with CAD models in the OnePose-Low Texture dataset using the ADD(S)-0.1d metric.
Obj. ID
0700
0706
0714
0721
0727
0732
0736
0740
Avg.
PVNet
12.3
90.0
68.1
67.6
95.6
57.3
49.6
61.3
62.7
Ours
89.5
99.1
97.2
92.6
98.5
79.5
97.2
57.6
88.9
4.2 Experiment Settings and Baselines
Baselines. We compare the proposed method with the following baselines in two categories: 1) One-shot baselines [48, 33, 40] that hold the same setting as ours. OnePose [48] and HLoc [40] are most relevant to our method in leveraging feature matching for reconstruction and pose estimation. To be specific, we compare with HLoc combined with different feature matching methods including SuperGlue [41] and LoFTR [47]. 2) Instance-level baselines [39, 29] that require CAD-models and need to be trained separately for each object. These methods achieve high accuracy through training on many rendered images with extensive data augmentation. We compare our method with them to demonstrate that our method achieves competitive results while not relying on CAD models and eliminating per-object pose estimator training.
Evaluation Protocols. We compare method with OnePose and HLoc using the same set of reference images. Since HLoc's original retrieval module is designed for the outdoor scenes, we use uniformly sampled 10 reference views for 2D-2D matching for pose estimation, following [48]. For the comparison with PVNet [39], we follow its original training setting, which first samples 8 keypoints the object surface and then trains a network using 5000 synthetic images for each object. In contrast, our method only uses around 200 reference images to reconstruct the object point cloud. We evaluate our method and PVNet on the same real-world test sequences, while our matching model has never seen the test objects before. As for the experiments on LINEMOD, we compare our method with OnePose by running their open-soure code. Our method and OnePose share the same 2D bounding boxes from an off-the-shelf object detector YOLOv5 [1]. Note that the object detector is trained on
images only to provide rough bounding boxes. We use the real training images (~ 180) for object reconstruction and all test images for evaluation. The results of other baselines on LINEMOD are from the original papers.
Metrics. We use metrics including the cm-degree pose success rate, the ADD(S)-0. Id average distance with threshold of 10% of the object diameter, and the 2D projection error Proj2D with threshold of pixels. The definitions of these metrics are detailed in the supplementary material.
4.3 Results on the OnePose and OnePose-LowTTextus Datasets
Comparison with One-shot Baselines. The cm-degree success rate with different thresholds are used for evaluation. As shown in Tab. 1, our method substantially outperforms OnePose [48] and HLoc [40]. Objects in the OnePose dataset have rich textures, benefiting keypoint detection. Therefore, keypoint-basee methods OnePose and HLoc (SPP+SPG) perform reasonably well. Our method achieves even higher accuracy thanks to the keypoint- design, effectively utilizing both texture-rich and low-textured object regions for pose estimation. On the OnePose- LowTexture dataset, our method surpasses OnePose and HLoc by a large margin. This further demonstrates the capability of our keypoint-free design for object reconstruction and the sparse-to-dense 2D-3D matching for object pose estimation. Hl (LoFTR") uses LoFTR coarse matches for SIM and uses full LoFTR to match the query image and retrieved images for pose estimation. does not rely on keypoints, similar to our design. Our method significantly outperforms it on accuracy and runs 10x faster.
8

---

Table 3: Results on LINEMOD. Our method is compared with Instance-level and One-shol baselines. Note that Gen6D is fine-tuned on selected subset of objects and uses the rest for testing. Gen6D is the version without fine-tuning on LINEMOD. Symmetric objects are indicated by
Type
Name
ape
benchwise
driller
Object Name Name
dack egghox' gluc*
bolepuncher
iron lamp phone
Avg.
ADDYY3-00.d
Instunco-level
CDPN
67.3
98.8
92.8
96.6
86.6
95.1
75.2
99.6
99.6
89.7
97.9
97.8
80.7
91.4
PVNet
79.3
95.7
98.9
Gen6D!
62.1
45.6
40.9
48.8
16.2
One-shot
OncPosc
Gen6D)
11.8
92.6
66.1
88.1
77.2
60.7
47.9
67.4
74.5
40.5
34.2
71.3
95.7
87.2
37.5
$1.9
89.2
87.6
60.6
63.6
Ours
31.2
97.3
88.0
89.8
70.4
92.5
42.3
99.7
48.0
69.7
97.4
97.8
76.9
76.9
FrajDD
Instance-level
CDPN
97.5
98.3
98.6
99.6
99.3
94.9
98.4
99.1
98.4
99.3
91.9
95.7
96.8
98.0
PVNet
99.2
99.8
99.2
99.9
99.3
96.9
98.0
99.3
98.5
100.0
99.2
98.3
99.4
99.0
One-shot
OncPosc
35.2
94.4
96.8
87.4
77.2
76.0
73.0
89.9
55.1
79.1
92.4
58.9
69.4
78.1
Ours
97.3
99.6
99.6
99.2
98.7
93.1
97.7
98.7
51.8
98.6
98.9
98.8
94.5
94.3
The improved accuracy and speed come from the accurate point cloud reconstructed by our novel SfM framework and the efficient 2D-3D matching module.
Comparison with Instance-level Baseline PVNet. On the OnePose-Low LexTerure dataset, the proposed method is compared with PVNet [39] on the subset objects with scanned models. The ADD(S)-0. Id results presented in Tab. 2. Even though is trained large number (~ 5000) of rendered images covering almost all possible views, our method still outperforms it on most objects without additional training. We attribute this to PVNet's susceptibility to domain gaps and our matching module's 's robustness and generalizability, thanks to its large-scale pre-training.
4.4 Results on LINEMOD
We compare the proposed method with OnePose [48] and Gen6D [33] which are under the One-shot setting, and Instance-level methods PVNet [39] and CDPN [29] on ADD(S))- Id and Proj21) metrics. As shown in Tab. 3, our method outperforms existing one-shot baselines significantly and achieves comparable performance with instance-level methods. Notably, our method and OnePose are only trained on the OnePose training set and tested on LINEMOD without additional training.
Since LINEMOD is mainly composed of low-textured objects, our method outperforms OnePose significantly thanks the keypoint-free design. Gen6D [33] is CAD-model-free and can generalize to unseen objects similar to our method. However, it relies on detecting accurate object bounding boxes for pose initialization, which is hard on LINEMOD because of the poor image quality and slight object occlusion. In contrast, our method only needs rough object detection to reduce possible mismatches, which more robust to detection error. Moreover, the performance of Gen6l) drops significantly without training on a subset of LINEMOD, while our method requires no extra training and achieves much higher accuracy than Gen6D. The experiment demonstrates the superiority of our method over existing methods under the one-shot setting.
Our method has lower or comparable performance with instance-level methods [39, 29], which trained to fit each object instance, and thus perform well naturally, at the expense of the tedious training for each object. In contrast, our method is grounded in highly generalizable local features and generalizes to unseen objects with comparable performances.
4.5 Ablation Studies
We conduct several experiments on the OnePose dataset and the OnePose- LowTTxttree dataset to validate the efficacy of the point cloud refinement in the SfM framework and the attention module in our 2D-3D matching module. More ablation studies are detailed in the supplementary material.
Point Cloud Refinement in the Keypoint-Frie StM. We validate the effectiveness of the point cloud refinement from two perspectives, as shown in Tab. (Left). One is evaluating the accuracy of the reconstructed point cloud with ground truth object mesh on LreePose-LwTexturr dataset following evaluations [31] and the other is quantifying the impact of reconstruction accuracy on the cm-degree pose success rate. Compared with the coarse point cloud reconstructed with coarse-level LoFTR only, the accuracy of our refined point cloud increased significantly, especially under strict threshold (1mm). Moreover, using the refined point cloud for object pose estimation brings around

---

Table 4: Ablation Studies. We quantitatively validate the effectiveness of the point cloud refinement in the keypoint-free SfM and the attention module the 2D-3D matching network, using the point cloud accuracy metric and cm-degree pose success rate with different thresholds.
Point Cloud Accuracy
Success Rale OnePose Dataset
Success on One Low Texture
Imm
3mm
Smmn
lcm- Idog
3om-3dog
Semr-Sdeg
Icm- Ideg
Jem-Jdeg
Sem--Sdeg
26.6
K5.7
43.9
78.3
85.9
allention.
12.2
55.3
reline.
30,9
75.8
85.7
51.1
80.8
87.7
atlention.
16.8
57.7
72.1
LINEMODE Dataset
OnePose-LowTexture Dataset
Ablation: Feature PCA
Atrention
Figure 6: Qualitative Results showing the reconstructed semi-dense object point clouds and the estimated object poses. The ablation part visualizes the 2D and 3D features before and after our 2D-3D attention module. Features become more discriminative as shown by the color contrast.
7% improvement on the strict lcm-Ideg metric. These experiments demonstrate that the point cloud refinement improves the reconstructed point clouds' precision, thus benefiting the pose estimation.
Attention module in the Pose Estimation Network. We validate the attention design in our matching network quantitatively and qualitatively. is shown Tab. (Right) that compared with directly matching the backbone features, using the transformed features for 2D-3D correspondences obtains 15% improvement on the Scm-5deg metric on OnePose- LowTexture dataset. The visualization of features in Fig. 6 shows that the transformed 2D and 3D features become more discriminative for establishing correspondences. The ablation study demonstrates that the attention module provides the global receptive field and plays a critical role in the pose estimation of low-textured objects.
5
Conclusion
We propose a keypoint-free SIM and pose estimation pipeline that enables pose estimation of both texture-rich and low-textured objects under the one-shot CAD-model-frre setting. Our method can efficiently reconstruct accurate and complete 3D structures of low-textured objects and build robust 2D-3D correspondences with the test image for accurate object pose estimation. The experiments show that our method achieves significantly better pose estimation accuracy compared with existing modelldeleee methods, and even achieves comparable results with CAD-model-basse instance- level methods. Although we do not see the immediate negative societal impact of our work, we do note that accurate object pose estimation can be potentially used for malicious purposes.
Limitations. Being dependent on local feature matching, our method inherently suffers from very low-resolution images and extreme scale and viewpoint changes. In the current pipeline, we still need a separate object detector to provide rough regions of interest, In the future, we envision more tight integration with the object detector, where object detection can also be carried out through local feature matching.
The authors would like to acknowledge the support from the National Key Research and Development Program of China (No. 2020AAA0108901), NSFC (No. 62172364), the ZJU-SenseTime Joint Lab of 3D Vision, and the Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
Acknowledgements.
10

---

References
[1] Ultralytics/yolov5. httpa://githe hem/ulltralltica/yolt 2021. 8
[2] Ming Cai and Ian Reid. Reconstruct locally, localize globally: A model free method for object pose estimation. In CVPR, 2020. 1.3
[3] Federico Camposeco, Torsten Sattler, Andrea Cohen, Andreas Geiger, and Mare Pollefeys. Toroidal constraints for two-por wo-poont localization under high outlier ratios. CVPR, 2017. 3
[4] Xu Chen, Zijian Dong. Jie Song, Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural analysis-by- -yynthssss., ECCV, 2020. 1,3
[5] Siddharth Choudhary and P. Narayanan. Visibility probability structure from sfm datasets and applica-
tions. ECCV, 2012.
[6] Daniel Tone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperPoint: Self-supervised interest point
detection and description. CVPRW, 2017. 3
Michacl Donoser and Dieter Schmalstieg. Diseriminative feature-to-point matching in image-based
localization. CVPR, 2014. 3
[8] Mihai Dusmanu, Johannes L. Schönberger. and Mare Pollefeys. Multi-Vies Optimization of Local Feature Geometry. In ECCV, 2020.
[9] Michele Fenzi, Ralf Dragon, Laura TealeTaixé, Bodo Rosenhahn, and Jöm Ostermann. 3d object recog- nition and pose estimation for multiple objects using multi-prioritized ransac and model updating. In
DAGM/OAGM Symposium, 2012.
[10] Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24, 1981, 6
[11] Christian Forster, Matia Pizzoli, and Davide Scaramuzza. Svo: Fast semi-direct monocular visual odometry.
ICRA, 2014. 5
[12] Hugo Germain, Guillaume Bourmaud, and Vincent Lepetit. Sparse-to-dense hypercolumn matching for
long-term visual localization. 3DV, 2019. 2. 3
[13] Hugo Germain. Guillaume Bourmaud, and Vincent Lepetit. S2dnet: Learning image features for accurate sparse-to-dense matching. ECCV, 2020.
[14] Iryna Gordon and David G. Lowe. What and where: 3d object recognition with accurate pose. In Toward Category-Leve Object Recognition, 2006.
[15] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR,
2016. 7
[16] Stefan Hinterstoißerr Vincent Lopetit, Slobodan Ilie, Stefan Holzer, Gary R. Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In ACCV, 2012. 7
[17] Edward Hsiao, Alvaro Collet, and Martial Hebert. Making specific features less discriminative to improve point-based based 3d object recognition. CVPR, 2010.
[18] Jingwei Huang, Shan Huang, and Mingwei Sun. Deeplm: Large-scale nonlinear least squares deep
learning frameworks using stochastic domain decomposition. CVPR, 2021. 7
[19] A. Katharopoulos, A. Vyas, N. Pappas. and F. Fleuret. Transformers are mns: Fast autoregressive
transformers with lincar attention. In ICML, 2020. 6
[20] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilie, and Nassir Navab. SSD-6D: making rgh-based detection and 6d pose estimation great again. In ICCV, 2017.
[21] Taeyeop Lee, Byeong- Lee. Myungchul Kim, and In Kweon. Category-Leve metric scale object
shape and pose estimation. JEFE Robotics and Automation Letters, 2021. 3
[22] Vincent Lepetit, Francesc Moreno-Noguce and P. Fua. Epnp: An accurate o(n) solution to the pup problem.
JCC, 81, 2008. 6
[23] Kenneth Levenberg. A method for the solution of certain non linear problems in least squares. Quarterly
of Applied Mathematics. 2, 1944. 7
[24] Fu I., Hao Yu, Ivan Shugurov, Benjamin Busam, Shaowu Yang, and Slobodan Tlic. Nerf-pose: A sebrrecoosttuut--be--rregesss approach for weakly-supervised 6d object pose estimation. arXiv preprint
arXiv:2203.0488 04802, 2022. 3
[25] Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu. Dual-rcsolution correspondence networks. In
NeurIPS, 2020.
[26] Yunpeng Li, Noah Snavely, and Daniel P. Huttenlocher. Location recognition using prioritized feature matching. ECCV, 2010.
[27] Yunpeng Li, Noah Snavely, Daniel P. Huttenlocher, and Pascal Fua. Worldwide pose estimation using
3d point clouds. In ECCV, 2012. 3
[28] Zhenggi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos.
CVPR, 2018. 7
[29] Zhigang Li, Gu Wang, and Xiangyang Ji. Cdpn: Coordinates- disentangled pose network for real-time
rgh-based 6-dof object pose estimation. ICCV, 2019. 1, 2, 3, 7, 8,
[30] Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[31] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-perfect structure-
from-motion with featuremetric refinement. ICCV, 2021. 4,9
[32] Liu Liu, Hongdong Li. and Yuchao Dai. Efficient global 2d-3d matching for camera localization in a large-scale 3d map. ICCV, 2017.
11

---

[33]
Yuan Liu, Yilin Wen, Sida Peng. Cheng Lin, Xiaoxiao Long, Taku Komura, and Wenping Wang. Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images. In ECCV, 2022. 2, 3. 8,9
[34]
David G. Lowe. Distinctive image features from scale-invariant keypoints. ICV, 2004. 3
[35]
Manuel Martinez, Alvaro Collet, and Siddhartha S. Srinivasa. Moped: A scalable and low latency object
recognition and pose estimation system. ICRA, 2010.
[36] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 3
[37] Keunhong Park, Arsalan Mousavian, Yu Xiang. and Dieter Fox. LatentFusion: End-to-end differentiable
reconstruction and rendering for unseen object pose estimation. In CVPR, 2020.
[38] Kiru Park, Timothy Patten, and Markus Vineze. Pix2Pose: Pixel-wise coordinate regression of objects for 6d pose estimation.
ICCV, 2019. 1
[39] Sida Peng, Xiaowei Zhou, Yuan Liu, Haotong Lin, Qixing Huang, and Hujun Bao. PVNet: pixel-wiss wise
voting network for 6dof object pose estimation. T-PAMI, 2020. 1, 1,2, 3, 8. 9
[40] Paul- Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse to fine: Robust
hierarchical localization at large scale. In CVPR, 2019. 3, 8
[41] Paul- aulu-Ed Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperOlue: Learning
feature matching with graph neural networks. In ICCV, 2020. 3.8
[42] Torsten Sattler, Michal Havlena, Filip Radenović, Konrad Schindler, and Mare Pollefeys. Hyperpoints and
fine vocabularies for large-scale location recognition. ICCV, 2015. 3
[43] Torsten Sattler, B. Leibe, and Leif P. Kobbelt. Efficient & effective prioritized matching for large-scale amage-based localization. T-PAMI, 39, 2017.
[44] Johannes I.. Schönberger and Jan-Michael Frahm. Structureffrommmootte revisited. In CVPR, 2016. 1, 3,
4, 5,5,7 7
[45] Ivan S. Shugurov, Fu Li, Benjamin Busam, and Slobodan Ilie. Osop: A multi-stage one shot object pose estimation framework. CVPR, 2022. 3
[46] Iryna Skrypnyk and David G. Lowe. Scene modelling, recognition and tracking with invariant image features. ISMAR, 2004. 3
[47] Jiaming Sun. Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local
feature matching with transformers. CVPR, 2021. 1, 2, 4, 6, 7, 8
[48] Jiaming Sun, Zihao Wang. Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou.
OnePose: One-shot object pose estimation without CAD models. CVPR, 2022. 1, 2. 3, 7, 8, 9, 13
[49] Linus Svärm, Olof Enqvist, Fredrik Kahl, and Magnus Oskarsson. City-scale localization for cameras with known vertical direction. T-PAMI, 39, 2017.
[50] Linus Svärm, Olof Enqvist, Magnus Oskarsson, and Fredrik Kahl.
Accurate localization and pose
estimation for large 3d models. CVPR, 2014.
[51] Meng Tian, Marcelo H. Ang Jr, and Gim Hee Lec. Shape prior deformation for categorical 6D object pose and size estimation. ECCV, 2020, 3
[52] Michal J. Tyszkiewicz, Pascal Fua, and Eduard Trulls. DISK: learning local features with policy gradient. In NeurlPS, 2020. 6
[53] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. 4
[54] Gu Wang. Fabian Manhardt, Federico Tombari, and Xiangyang Ji. GDR-Net: Geometry-guided direct regression network for monocular 6D object pose estimation. In CVPR, 2021.
[55] He Wang, Srinath Sridhar, Jingwei Huang. Julien Valentin, Shuran Song, and Leonidas Guibas. Nor- malized object coordinate space for category-level 6d object pose and size estimation. CVPR, 2019. 1,
[56] Jiaze Wang. Kai Chen, and Qi Dou. Category-level 6D object pose estimation via cascaded relation and recurrent reconstruction networks. IROS, 2021. 3
[57] Aji Resindra Widya, Akihiko Torii, and M. Okutomi. Structure from motion using dense ean features with Computer Vision and Applications, 10, 2018. 4
keypoint relocalization. IPSJ Transactions
[58] Yu Xiang. Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes. RSS, 2018. 3
[59] Sergey Zakharov, Ivan S. Shugurov, and Slobodan Ilie. Dpod: 6d pose object detector and refiner. ICCV, 2019.
3
[60] Bernhard Zeisl, Torsten Sattler, and Marc Pollefeys. Camera pose voting for large-scale based localization. ICCV, 2015.
[61] Chengliang Zhong, Chao Yang, Ji Qi, Fuchun Sun, Huaping Liu, Xiaodong Mu, and Wenbing Huang. Sim2real object-centric keypoint detection and description. AAAI, 2022.
[62] Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixé, Patch2pix: suiiolar-guided pixel-level correspon- dences. CVPR, 2021. 3,
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main perimental results (either in the supplemental material or as URL)? [Yes] The OnePose [48] dataset and the LINEMOD [? dataset used in the paper are pub- lic; our code and the collected OncPose-Low LowTextureaatase will be published.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 3.4 and Sec. 4.1
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] Unfortunately, we did not provide the error bars due to resource constraints. We will make this up in the future.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sec. 3.4
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "w-rI0KbKqIW", "reviewer": "Reviewer_BWks", "summary": "The authors proposed 6-DoF object pose estimation algorithms which does not require CAD models of target objects.  They require only video sequence of the target object with camera pose.  The keypoint-free SfM build 3D models in training and the keypoint-free 2D-3D matching network can estimate the correspondences between the models and image query. The proposed method handles low-textured objects and achieved state-of-the-art accuracy on public dataset.", "questions": "- How about processing time for training and testing?\n- Is the pose estimation network trained for each object?", "limitations": "- CAD model or its equivalent can be reconstructed from the movie of target object with surrounded AR markers.  This might decrease the advantage of the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed algorithm can handle low-textured objects and does not require CAD model for training, those are useful for real applications.\n- The ablation study shows the effectiveness of each component.\n\nWeaknesses:\n- The proposed method needs off-the-shelf 2D object detector and real images for training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dmZVK7RrOZ", "reviewer": "Reviewer_hB2N", "summary": "The paper looks at the problem of one-shot object pose estimation on textureless objects where previous keypoint-based methods fail to perform well. The main contribution is using the keypoint-free SFM pipeline to create a repeatable semi-dense point cloud, which automatically helps improve 3D-2D correspondence to estimate the object pose. Comparisons show that the method performs better than a previous method which uses sparse point cloud reconstructions.", "questions": "How is the current method's dense reconstruction different from Multi-View Stereo paradigms. Why are evaluations not compared to these paradigms to show the accuracy improvement if the dense reconstruction is very good?\n\n", "limitations": "The limitations and potential negative impact are well studied in the paper. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Moving away from keypoint-based methods helps in automatic object pose estimation of objects in the wild.\n- The semi-dense reconstruction of the objects seems to be very helpful in the pose estimation due to better 2D-3D correpondences.\n- The results show that the method is more robust to occlusions than previous methods and a live demo helps show the method's accuracy.\n- Ablation study shows the advantages of the refinement step and the attention module in the pose estimation network.\n- Openpose-Hard dataset is useful for research in pose estimation of textureless objects.\n\nWeaknesses:\n- The major advantage or the difference between the proposed network and the previous baseline (onepose [30]) is the dense object reconstruction of the objects from videos. The contribution in the dense reconstruction from videos is well studied in the Multi-View Stereo frameworks. So I feel that the method doesn't have a lot of novelty in terms of reconstruction.\n- The paper assumes that the object videos are given aprior, so one-shot object pose estimation might be a misleading term as the method will fail if the objects videos are not provided beforehand.\n-  Literature review of Multi-view stereo needs to be well studied and the difference between these methods and the proposed keypoint-free methods need to be well established.\n- The evaluations are not substantial as comparisons to other methods like CAD-model based pose estimation have not been well studied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XUE5-0wKa9D", "reviewer": "Reviewer_BV2E", "summary": "The paper proposes an improvement of the one-shot pose estimation system OnePose to better estimate the pose of low-texture objects. In particular, the keypoint-based matching component of OnePose is replaced with the key-point free matching method LoFTR. For the evaluation of the improved functionality, the authors propose the new dataset OnePose-HARD, which contains low-texture objects along with their pose annotations.\n", "questions": "\n- Please consider including example images of the OnePose-HARD dataset in the paper.\n- Please clarify the source of the results of other methods that you compare your method to: Where are these numbers taken from or did you evaluate the methods yourself? In particular, the results of OnePose on the OnePose dataset and on LINEMOD, as described above. Please also describe why some results are underlined in Table 1.\n- Please consider including the runtimes of your method and the methods you compare to, in order to support the claim that your method is 10× faster.\n\n\nAdditional remarks (typos, suggestions etc.), no need to address in the rebuttal:\n- line 49: \"establish\" -> \"establishes\"\n- line 70: \"OnePose-HARD ,\" -> \"OnePose-HARD,\"\n- line 104: \"keypoints .\" -> \"keypoints.\"\n- Figure 2, caption: \"a reference image sequences\" -> \"a reference image sequence\"\n- Figure 2, caption: \"point cloud which are\" -> \"point cloud which is\"\n- line 131: \"build\" -> \"builds\"\n- line 295: \"number(~ 5000)\" -> \"number (~ 5000)\"\n- On the naming of the proposed dataset: The suffix \"HARD\" is very generic and does not tell what the difficulties actually are. Please consider a more informative suffix, such as \"LowTexture\".\n", "limitations": "Yes, limitations and impact were discussed where applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n+ The idea of replacing keypoint-based matching with keypoint-free matching for improved low-texture performance is straightforward and a logical extension of the prior work.\n\nQuality:\n+ The paper is well written and structured.\n+ The paper precisely adheres to the formatting requirements and the page limit.\n+ Related work is described sufficiently and it is made clear how the proposed method is positioned in the existing landscape.\n+ The proposed method is evaluated on appropriate datasets and compared to relevant state-of-the-art methods.\n\nClarity:\n+ The language is clear and easy to follow.\n+ Methods used in the paper, e.g., LoFTR, are explained briefly, but well understandable.\n- The description of the proposed OnePose-HARD dataset is quite abstract. Example images of the new dataset are not included in the paper.\n- In the evaluation, it is not always clear where the results of the other methods come from. E.g., in Table 1, the results of the OnePose method on the OnePose dataset is given as an overall single value for the whole dataset, while the original OnePose paper lists the results of the three categories \"large\", \"medium\", and \"small\" separately, but no overall single value. Similarly, in Table 3, the paper gives results for OnePose on the LINEMOD dataset, but the original OnePose paper contains no such evaluation.\n- The authors claim that their method is \"~10× faster\" [line 290] than other methods, but the actual runtimes of the other methods are not listed in the paper.\n\nSignificance:\n+ The addressed issue with low-texture objects is relevant in practice.\n+ The results of the proposed method seem to be significantly better than comparable methods, particularly in low-texture scenarios.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OSrNdlttdmZ", "reviewer": "Reviewer_iDEM", "summary": "The paper considers the problem of object pose estimation in scenarios where CAD model of the objects are not available. The paper describes OnePose++, a variant of the recently proposed OnePose approach that uses densely extracted descriptors (via LoFTR) rather than the SuperPoint keypoints used by OnePose. LoFTR provides matches between pairs of images, where the 2D positions of matching points vary depending on the image pair. As the resulting 2D positions are not repeatable, the paper uses a SfM approach designed to handle this scenario in order to build the 3D model of the object used for pose estimation. 2D-3D matches between a query image and the SfM model are established by directly matching descriptors against the 3D model in a coarse-to-fine manner. In addition to the OnePose++ method, the paper introduces a harder variant of the OnePose dataset, named OnePose-HARD. Experimental results show that OnePose++ outperforms most baselines by a wide margin (in particular, OnePose++ consistently outperforms OnePose).", "questions": "In order to consider raising my score, I would like to see the following points addressed in a rebuttal:\nQ1) Please describe in detail how the proposed keypoint-free SfM approach differs from prior work in this area.\nQ2) Why were Patch2Pix, Dual RCNet, etc. not considered as baselines?\nQ3) Please describe the relation of OnePose++ to prior work on SfM-based object pose estimation and visual localization (see W3 above).", "limitations": "The paper adequately discusses limitations and potential negative social impact.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has multiple strengths:\nS1) The proposed OnePose++ approach is a natural extension of OnePose that shows how to swap keypoint-based features with keypoint-free features. Since the latter have shown promise in challenging scenes, e.g., for weakly textured objects or under strong illumination conditions, this is interesting.\n\nS2) OnePose++ clearly outperforms OnePose and also most of the baseline methods. The strong results are a strength of the paper.\n\nS3) The paper provides a detailed ablation study that analyzes the impact of the individual components of OnePose++.\n\nS4) The proposed dataset, OnePose-HARD, seems very challenging and thus has the potential to drive research in the field. It will be of interest to the community.\n\nOn the negative side, there are also multiple weaknesses:\nW1) Both OnePose and OnePose++ are misclassified as being generalizable object pose estimation approaches. It is true that the underlying LoFTR features and the coarse and fine matching stages generalize beyond the data they were trained on. Yet, OnePose++ requires that \"a video sequence with annotated poses is available for each object\" is available and builds a SfM model for this particular type of object. I don't see how this is not an instance-level method. The SfM model for one object might give reasonable results for another object if both objects have very similar shapes (and textures). But a single model for one particular type of an object, e.g., a particular chair, will not generalize over the full class (e.g., all potential chairs).\nGiven that OnePose++ is an instance-level method, it should be more closely compared to other instance-level methods. It is not too surprising that it outperforms the generalizable baselines as it is able to train (in the form of building a SfM model) per object.\n\nW2) The paper claims proposing a \"keypoint-free SfM framework for accurate and complete semi-dense reconstruction\" as one of its main contributions. It is not clear to my how the described framework contributes novelty to the literature:\na) Besides LoFTR, there are other keypoint-free descriptors, e.g., Patch2Pix [Zhou et al., Patch2Pix: Epipolar-Guided Pixel-Level Correspondences, CVPR'21], Sparse NCNet [Rocco et al., Efficient ´neighbourhood consensus networks via submanifold sparse\nconvolutions, ECCV'20], and Dual RCNet [Li et al., Dual-Resolution Correspondence Networks, NeurIPS'20], that are evaluated in a visual localization setting that requires an underlying SfM model. They thus also provide approaches for keypoint-free SfM. All of them are inherently applicable to object pose estimation (since they do not make any assumption on the type of scenes). Similarly, LoFTR also uses a keypoint-free SfM approach (based on Dual RCNet) (see https://github.com/zju3dv/LoFTR/issues/9). Another approach to keypoint-free SfM based on dense feature matching between images is [Widya et al., Structure from motion using dense CNN features with keypoint relocalization, IPSJ Transactions on Computer Vision and Applications 2018]. Yet, this prior work is not discussed. The differences between this prior work and the proposed approach should be clearly described. Furthermore, comparisons with other keypoint-free approaches, e.g., Patch2Pix or Dual RCNet, are missing.\nb) As far as I can see, the coarse reconstruction stage for keypoint-free SfM is the same as for LoFTR (based on the description provided here: https://github.com/zju3dv/LoFTR/issues/9). The refinement stage seems identical to [18]. The paper states that \"Note that our keypoint-free SfM framework is also related to PixSfM [18] but comes with different motivations. PixSfM improves keypoint-based SfM for more accurate 3D reconstructions by refining inaccurately-detected sparse local features with dense feature maps. Different from PixSfM, we aim to adapt the keypoint-free method LoFTR [29] to SfM for object pose estimation.\" However, I disagree with this statement. As stated in the paper, \"Every pixel in the downsampled image can be regarded as a “keypoint” in the original image.\" The goal of the refinement stage is to \"refine the object point cloud with sub-pixel correspondences.\" In other words, the motivation for the refinement stage is to obtain a more accurate 3D model by refining initially inaccurate keypoint positions. This is achieved using dense feature maps to detect more accurate keypoint positions.\n\nW3) As in the case of keypoint-free SfM, there are other directions of highly related work that omitted:\na) Work on object pose estimation using SfM rather than CAD models certainly predates OnePose. Examples include [Gordon & Lowe, What and Where: 3D Object Recognition with Accurate Pose, Toward Category-Level Object Recognition, 2006], [Rothganger et al., 3D Object Modeling and Recognition Using Local Affine-Invariant Image Descriptors and Multi-View Spatial Constraints, 3D Object Modeling and Recognition Using Local Affine-Invariant Image Descriptors and Multi-View Spatial Constraints, IJCV 2006], [Hsiao et al., Making specific features less discriminative to improve point-based 3D object recognition, CVPR 2010[, [Bhat et al., Visual words for 3D reconstruction and pose computation, 3DIM/3DPVT 2011], and [Fenzi et al., 3D Object Recognition and Pose Estimation for Multiple Objects using Multi-Prioritized RANSAC and Model Updating, DAGM/OAGM 2012]. This prior work should be properly acknowledged.\nb) The paper states that \"HLoc is slow during pose estimation because it depends on multiple 2D-2D image matchings as the proxy for building 2D-3D correspondences.\" Yet, there is quite some literature on visual localization algorithms that do not use image retrieval but directly match 2D features against 3D points via associated feature descriptors. Examples include: [Arth et al., Wide Area Localization on\nMobile Phones. ISMAR 2009], [Li et al., Location Recognition using Prioritized Feature Matching, ECCV 2010], [Li et al., Worldwide Pose Estimation Using 3D Point Clouds, ECCV 2012], [Choudhary & Narayanan, Visibility probability structure from sfm datasets and applications, ECCV 2012], [Donoser & Schmalstieg, Discriminative featureto-point matching in image-based localization, CVPR 2014], [Cao & Snavely, Minimal scene descriptions from structure from motion models, CVPR 2014], [Lim et al., Real-time monocular image-based 6-dof localization, IJRR 2015], [Lynen et al., Get out of my lab: Largescale, real-time visual-inertial localization, RSS 2015], [Zeisl et al., Camera Pose Voting for Large-Scale Image-Based Localization, ICCV 2015], [Camposeco et al., Toroidal Constraints for TwoPoint Localization under High Outlier Ratios, CVPR 2017], [DuToit et al., Consistent map-based 3d localization on mobile devices, ICRA 2017], [Liu et al., Efficient Global 2D-3D Matching for Camera Localization in a Large-Scale 3D Map, ICCV 2017], [Sattler et al., Efficient & effective prioritized matching for large-scale image-based localization, PAMI 2017], [Svarm et al., City-scale localization for cameras with known vertical direction, PAMI 2017], and [Lynen et al., Large-scale, real-time visual-inertial localization revisited, IJRR 2019]. Many of these approaches are directly applicable to the object pose estimation setting based on SfM models and should thus be discussed.\n\nThe following additional comments did not affect my recommendation:\n* References for RANSAC and the PnP solver used are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept