SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference
Authors: Jiahui Wang
Haiyue Zhu
Haoren Guo
Abdullah Al Mamun
Cheng Xiang
Tong Heng LEE
Venue: NeurIPS 2025 poster ()
Keywords: Depth Estimation
6D Pose Estimation
Novel Object Pose Estimation
Primary area: deep_learning

Abstract: Recent 6D pose estimation methods demonstrate notable performance but still face some practical limitations. For instance, many of them rely heavily on sensor depth, which may fail with challenging surface conditions, such as transparent or highly reflective materials. In the meantime, RGB-based solutions provide less robust matching performance in low-light and texture-less scenes due to the lack of geometry information. Motivated by these, we propose **SingRef6D**, a lightweight pipeline requiring only a **single RGB** image as a reference, eliminating the need for costly depth sensors, multi-view image acquisition, or training view synthesis models and neural fields. This enables SingRef6D to remain robust and capable even under resource-limited settings where depth or dense templates are unavailable. Our framework incorporates two key innovations. First, we propose a token-scaler-based fine-tuning mechanism with a novel optimization loss on top of Depth-Anything v2 to enhance its ability to predict accurate depth, even for challenging surfaces. Our results show a 14.41% improvement (in $\delta_{1.05}$) on REAL275 depth prediction compared to Depth-Anything v2 (with fine-tuned head). Second, benefiting from depth availability, we introduce a depth-aware matching process that effectively integrates spatial relationships within LoFTR, enabling our system to handle matching for challenging materials and lighting conditions. Evaluations of pose estimation on the REAL275, ClearPose, and Toyota-Light datasets show that our approach surpasses state-of-the-art methods, achieving a 6.1% improvement in average recall.

SingRef6D:: Monocular Novel Object Pose Estimation with a Single RGB Reference
Jiahui Wang'
Haiyue Zhu
Haoren Guo'
Abdullah Al Mamun'
Cheng Xiang!
Tong Heng Leed
College of Design and Engineering, National University of Singapore 2SMMTech, Agency for Science, Technology and Research (A*STAR) wjiahui @u. nus. zhu_hai yuels wettecttcch.cc....ccm..ec..cccee a a-star r. edu. .gg
Abstract
Recent 6D pose estimation methods demonstrate notable performance but still face some practical limitations. For instance, many of them rely heavily on sen- sor depth, which may fail with challenging surface conditions, such as transparent or highly reflective materials. In the meantime, less robust matching performance in low-light and texture-less scenes due to the lack of geometry information. Motivated by these, we propose SingRef6D, a lightweight pipeline requiring only a single RGB image as reference, climinat- ing the need for costly depth sensors, multi-view image acquisition, or training view synthesis models and neural fields. This enables SingRef6D to remain ro- under resource-limited settings where depth or dense tem-
solutions provide
bust and capable plates are unavailable. Our framework incorporates two key innovations. First, we propose a sesen-scaler-ber tion loss on top of Depth-Anything v2 to enhance its ability to predict accu- rate depth, even for challenging surfaces. Our results show a 14.41 % improve- ment (in 61.05) REAL275 depth prediction compared to Depth-Anything v2 (with fine-tuned head). Second, benefiting from depth availability, we introduce a depth-aware matching process that effectively integrates spatial relationships within LoFTR, enabling our system to handle matching for challenging mate- rials and lighting conditions. Evaluations of pose estimation ClearPose, and Toyota-Light datasets show that our approach surpasses state-of- the-art methods, achieving 6.1% improvement in average recall. Project page: htEps: plusgrey github 10/ singgrefed
fine-tume ane-tunnng mechanism with a novel optimiza-
the REAL275.
1 Introduction
Determining the 6D pose of an object in three-dimensional space is essential task for various ap- plications in robotics, industrial automation, and augmented reality [П-[]]. Recent computer vision approaches for 6D pose estimation have achieved remarkable progress, enabling machines under- stand object orientation and position in space and interact with the physical world with increasing sophistication [-T)
Among these methods, one prevalent paradigm relies on accurate geometric information of object CAD models and scene depth maps. By matching observed scenes with pre-obtained 3D models, it achieves notable 6D pose estimation performance [6. &. 9. II. I2]. Although these solutions have shown significant effectiveness, they come with some practical limitations. First, obtaining CAD models for new objects is costly, requires specialized scanning equipment, and involves tedious
*Corresponiing author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

manual refinement. In addition, bassor-based depth sensing struggles with challenging surface con- ditions, particularly for black, transparent, and highly reflective materials. Our study shows that it leads to failure rate exceeding 85% in the ClearPose [0:3] dataset that focuses scenes with transparent objects. Alternatively, some methods use multiview image matching [ZZ. EL [ for 6D pose estimation, either from object projections [TT, B], video sequences [9, [[]] or leveraging neu- ral fields [|ا]] for view rendering. However, multiview matching relies on an extensive template
library for high accuracy, while constructing neural fields is computationally intensive and restricted to per-instance training.
In contrast, the human visual system excels at object pose estimation with remarkable efficiency and udaptability across diverse environments. It operates without explicit 3D models, exhaustive view- point sampling, or even strict binocular vision, instead relying on cognitive mechanisms for depth perception and shape understanding [0G--22]. Inspired by the human vision system, we propose SingRef6D, simple-yet-effciien pose estimation pipeline that requires only a Single Reference RGB image, eliminating the need for explicit 3D models, precise depth sensing, or any form of novel view synthesis, explicit implicit, while maintaining robustness and versatility. Moreover, Singref6D requires neither the training of costly generative models (e.g.. VAE [23], diffusion [[El]) nor the construction of scene-specific neural fields (e.g., NeRF [25]), yet achieves competitive per- formance.
Specifically, SingRef6D tackles the pose estimation problem by independently addressing their cor- responding challenges in both depth perception and pose solving stages. To enhance depth per- ception, we develop new fine-tuning approach for Depth-Anything v2 (DPAv2) [26] by using token scaler (a network to re-weight features from transformer layers) to scale hierarchical features dynamically. This refinement mimics the mechanism of human spatial perception in a stratified man- ner (20, 22 allowing our depth prediction module to reliably extract depth cues from a single RGB image, even under adverse conditions such as high reflectivity and transparency. Although our depth model is trained with supervision, effectively distills spatial priors into a compact model, enabling inference-time operation with only an RGB input. In this sense, our method implicitly expands the reference space without incurring the cost of dense geometry or view synthesis. With LoFTR's s strengths [2Z]], our second stage introduces depth-awe
matching module by fusing RGB and depth cues into a unified latent space. By encoding spatial priors during training, our depth model expands the effective view space, allowing LoFTR [27] to match across challenging appearances with only a single- reference. This integrated approach significantly refines pose estimates and ensures precise alignment even under challenging environmental conditions. The contributions of this work can be summarized as follows:
We introduce SingRef6D, a novel monocular 6D pose estimation pipeline requiring only single reference RGB image under a strictly minimal reference setup, without relying on CAD models, multi-view collections, novel-view synthesis.
We developed a lakeen-caale-bbsee fine-tuning approach for DPAv2 [26]] which enables our metric depth estimation to handle challenging surface conditions. Results on the ClearPose dataset [[33]] show a boosted accuracy from 31.23% to 54.30% for transparent objects.
We propose a depth-aware matching on top of LoFTR [27]], and our results show an im- provement of 6.1 1% in average recall across three pose estimation benchmarks.
2 Related Work
2.1 Novel Object 6D Pose Estimation
Novel object 6D pose estimation aims to identify the pose of previously unseen object. Current methods typically adhere to the following pipeline: identification of the object, integration of 3D information, matching, and pose solving. These methods fall into two main splits, one is feature matching- based methods
PX
8 IT.
G & ITO, Ki]] the other is template matching- For each category, investigate several representative
12. 52
based methods [64] 14
5. G%
59)).
models and discuss their limitations.
Based. MatchU [32] simultaneously extracts features from RGB, depth, and CAD models. Then, fine-grained matching with the decoded descriptor will be conducted. However, its heavy dependence CAD models and substantial training overhead limit its practicality in real-world

---

Table 1: Comparison of input data requirement: (1) Reference Required: the format and utilization of input data as reference required for pose estimation; (2) Extra cost: cost to obtain sufficient spatial information; and (3) Localization: the method for object localization.
Method
Reference Required
Extra Cost
Localization
OVE6D [GM
Procise CAD models
Mask
MegaPose |+
Precise CAD models
Bounding Box
GenoD [1
RGB video sequence
Image level Comparison
Bounding Box
OnePose [
RGB video sequence
Structure from Motion (SfM)
Bounding Box
Single RGB image and
Train a U-Net to synthesis 342
NOPE [A
poses new viewpoints
novel views
(One object scene)
PoseDifiusion [B)]
Multiview RGB templates Multiview RGBD templates
Train a diffusion network to generate poses Train 3D-bused matching model
Bounding Box
SAM6D [+]
Mask
FoundationPose
Multiview RGB temllates
Train neural fields to represent 3D spaces Train maichhng and segmenlation model Image- to-3D model
Mask
Oryon [i]
RGBD image text promut
Any-6D 6D [[]]
Single RGBD image
Mask
Synihesize novel
with diffusion model.
Zerol23 6D [[]]
RGB images
train neural fields to represent 3D spaces
(One object scene)
3DAHY.DVMNeL B2
Single RGB imuge
Large-scale training for 2D-3D latent space
SingRefSD (nurs)
Single RGB image
Fine tune a lightweight token scaler (only for depth)
Mask
applications. Oryon [6] uses vision- Lasion-language model to boost matching with text embedding. This approach fails to deal with transparent objects due to the invalid values of sensor-based -based depth.
Template-Based. NOPE [+] determines the pose of the object by image-level matching that requires extensive viewpoints to synthesize templates from the reference image to represent 3D space ade- quately. FoundationPose [S] trains neural fields, such as BundleSDF [[S]], generate a number of pose hypotheses. However, NOPE [[]] is limited to scenarios involving single textured object, as it predicts the object's appearance from novel viewpoints. This approach becomes incapable for complex scenes with multiple objects and suffers significant performance degradation when the multi-
viewpoints are spatially sparse. Such a reliance on view synthesis limits generalization object or low-texture sceneschallenges that SingRef6D explicitly tackles via a learned depth prior. The hypothesis generation with refinement in FoundationPose [S]] incurs computational overhead, and the neural field may fail when facing challenging light conditions.
Table MI summarizes the data requirements of different models for novel object 6D pose estimation. Although some models [[A B8, A-]] appear to use single RGB image as a reference, they incorporate techniques such as diffusion- based synthesis or 3D reconstruction. For instance, NOPE [M trains a network to synthesize image embeddings of novel views and requires inputting the pose of new viewpoints. 123-6D [_4]] trains a NeRF-like [ network to reconstruct the 3D shape with new views generated by a diffusion model. GigaPose [38] directly uses an image-to-3D model to obtain
a mesh from an image, which might result in large shape inaccuracy.
Our SingRef6D, in contrast, does not involve any format of multi-view rendering or novel view synthesis; all accessible reference information comes from single RGB image. This is one of the minimal and strictest requirements for the reference input that ensures strong adaptability even in data-scarce and resource-scarce scenarios. This sharply contrasts with approaches that require synthesizing dozens or hundreds of virtual views, which incur high training and inference costs. Such an efficiency makes well-suited for real-world applications where limited annotations or sparse reference data are available.
2.2 Monocular Metric Depth Estimation
We investigate several monocular metric depth estimation methods [97--2. 26. 53 MiDas [ the
first transformer-basee dense depth model, introduced top-to-bottom fusion for precise predictions. ZeeDepth [[X] requires extensive fine-tuning for accurate zero-shot depth estimation. UniDepth [[0] employs a self-promptable camera module for depth conditioning but struggles with generalization due to full supervision. ScaleDepth [50]] models metric-relative depth relationships based on the assumption that all transformations are linear, which has achieved efficiency but relatively low accu- racy. HybridDepth [5]]] performs well across various datasets but relies on an additional focal stack for spatial priors Depth-Anything v2 [26]), structurally similar to MiDas [E7]], benefits from class- agnostic self-supervised training, ensuring adaptability and fine-tuning ease. Given these insights, we fine-tune a pre-trained DPAv2 [26]] as our depth estimator. Details of our pipeline are provided
in Section Bl
3

---

- =T
T
Image
Depth
Target Object
Depth-Aware
(A)
Robust Metric Depth
Prediction
(B)
Deptl-Aware Matching
[CI Reletive 6D Pase-Solving
RGB Pair
Depih Pair
Flae Foatures
Dinos
Mukiriie Tempitto
Multi Scale
Encoder
Featurr
Pridetion
Fine LoFTR
Procée CAD Maddl
Training for Maicher
Llaston
Depih
Courne oFTR
Tikeen
Zero
Scalie
Cazs
Stogle BGB
Les Draining Burdel!
Fine-buning Depth iodell
Matching
Carrsputsenss
Highlights
Figure 1: Visualized pipeline for inference (upper), the details of our depth model, matching pro- cess, and highlights (lower). During inference, our fine-tuned depth model first estimates the metric depth accurately, which can deal with challenging surfaces. Subsequently. the proposed depth-aware aware matching utilizes depth value as spatial cues to establish correspondences even in low-textured re- gions. Then, the relative pose Ty-r can solved with a point cloud registration model and the 61) pose for the query object can be calculated with T-1 T, TT
3 Method
3.1
Overview
The 6D pose estimation problem addressed in this work is defined as follows. Given pair of RGB images, a query, and reference, both containing the same object, the objective is estimate the 6D relative pose between the query and reference. Unlike previous methods [ ITC. g. our approach does not rely on CAD shapes, multi-view images, or any extra costly training to implicitly obtain novel-view information. This "low" reference requirement improves the applicability in real-world scenarios, making our method highly practical and versatile.
Figure M outlines our pipeline, which begins by processing query and reference image pairs to gener- depth predictions using our robust depth prediction module (Figure mA). Next, query-reference correspondence is established through the depth-aware matching module (Figure IBB. With the avail- ability of off-the-shelf models like SAM [54], target object localization is simplified, enabling corre- spondence by cropping the scene to region of interest. After that, we compute the 3D relative pose using PointDSC [55]] and
point clouds.
3.2 Robust Metric Depth Prediction
Accurate geometry perception relies on precise metric depth estimation. While depth foundation models, such as DPAv2 [26], provide an effective solution for depth prediction, we observed that direct metric depth estimation from DPAv2 [26] is hindered by scaling inconsistencies. To address this limitation, fine-tuning can be employed mitigate the scaling issue. However, direct fine- tuning is either computationally demanding or often degrades clarity, leading to distorted boundaries that diminish overall performance, illustrated in Figure To overcome these challenges, we introduce a novel fine-tuning approach based token-scaler mechanism, designed to enhance the accuracy and robustness of metric depth prediction based on DPAv2 [26]. Our approach integrates ControlNet-like structure [56]] to dynamically scale and modulate features at different levels (low- level, mid-level, high-level, and global-level) of DPAv2 [26].. As shown in Figure BL the depth prediction pipeline incorporates the token-scaler mechanism for hicrarchical feature modulation,

---

144
Disketio
Siale Incouddaaayy
grozed trath
Preficioo
Shape Deternalon
RGB Scume
DeplirAuyythng
OKE'S (D
Depth:Auyttingga (D)
Oues (I)
Depth Sersor
Figure 2: Visualized comparison of projected point clouds with depth maps. The depth sensors were unable detect transparent objects and limited the device's inherent field of view. The predicted depth ensures a valid value in all pixels. Compared to vanilla DPAv2 [2ih] a fine-tuned head), point clouds using depths from our method are geometrically consistent and scale-correct.
which adaptively adjusts feature representations at each level without compromising computational efficiency and pre-training knowledge.
Our method performs well in pre- serving geometric characteristics and overall depth map quality, as demonstrated in Figure M Math- ematically, consider a backbone network (e.g., DINOv2 [57]] in DPAv2 [26]) for feature extrac- tion, let F, denote the features stage I, where E 21,2,3, 3, 4} cor- responds low-level, mid-level, high-level, and global-level fea- tures, respectively. We introduce novel token scaler that adap- tively re- weights the feature in each level and then fuses with the upper level as:
TLus-lsee
High level Pused Fenture
Pretraited D
Fusion Layer Ft
F.
E
Slage
Token Scaler Scaen,
Pusion
E
Lager F
F Ellicivnt Atlretion
Fusien Layer
Stage 2
Fusion
F. Inception CoRy
Stage
F
1-1
Up-sample
Depth
Laper
Figure 3: Visualized structure of the depth prediction pipeline (Left) and fusion layer with our token scaler (Right).
= (F), Scaler(F(+))))
E {1,2,3}
where Scaler()) is scale function, Fi is the fusion comv layer and F Upsample(Scaleel (F4)). Specifically, for low- and middle-level features (F, and F2), which retain fine-grained details and local features with high-frequency information, global awareness while minimizing the impact of potential noise. Conversely, for high- and global- level features (Fa and Ft). which primarily capture low-frequency global scene and context-level information, we employ an InceptConv-basee network [59] as the scaler function, emphasizing local features enhance the high-level feature map. More details are illustrated in Sections and of the Appendix.
apply efficient attention layer [58] to boost
In addition the token-scaler mechanism, fine-tuning approach also employs novel loss scheme that combines two key components: a global loss term to regulate the scale and shift of the depth map and local loss term to improve object geometry and surface reconstruction, i.e., feeept Llocat + Leglobal-
Global Loss. We build on the Scale-Shift Invariant (SSI) Loss with regularization, denoted as Lass Creg. has proven effective in models like MiDaS [[ and DPAv2 [26] for overall supervision. To further enhance performance, penalizing large residuals, especially in high-error regions (default loss in many depth estimation models), and the final global loss is,
incorporate BerHu [60]] loss as it complements SSI by better
Lglobal Lest + Creg + D))
(2)
5

---

where a: is a mathematical expression of our global loss illustrated in the supplementary material Section C
D is the ground truth depth and D is the prediction. The detailed
The local loss consists of three parts, Liocal = Lasaate + Ledge + Loorm-, where each is described in the following.
Scale Alignment Loss. The SSI loss [0>]]] effectively handles global scale and shift parameters but does not directly enforce object-level scale alignment, potentially reducing the accuracy of relative scale representations. To address this, we introduce a scale alignment loss
1/4
(d, d,)² M 1+ del
1 Σ
Lxcale
where d; and d, denotes the ground truth and predicted depth value of the i-th pixel, respectively, This loss quantifies the discrepancy between the ground truth and the prediction within an object. To enhance robustness against outliers, introduce an extra term with v into the loss. For large errors, it reduces the gradient, mitigating noise sensitivity, while for small errors, it approximates MSE, enabling flexible supervision.
Edge-emphasize Loss. Our analysis indicates that the geometry of the shape is predominantly determined by its edges, whereby inaccurate edge reconstruction in depth maps results in substantial 3D distortions, as illustrated in Figure Utilizing predictions from DPAv2 [52]], the textures are generally acceptable: however, the boundaries exhibit pronounced discontinuities. To this end, we propose an edge-emphasized loss:
Ledge Σ VIl || - Vd.
where VI, is the gradient of the corresponding position in the RGB image, and a is the weight. This loss allows for depth variations in regions with sharp texture changes, which typically indicate boundaries while constraining depth variations in areas with locally similar textures.
Normal Consistency Loss. While existing loss functions effectively address object scale and edge detection, they fail to account for surface deformation. Although precise edge detection aids object localization, errors in 3D surface projection can still compromise relative pose accuracy. To address this, we introduce a normal consistency loss that preserves geometric structure,
77
Coorm
Σ e E-AAi/d:ll KEM
A/.1l: (1
(5)
M
n7a
where 7ł represent the norm vector. Optimizing this loss enforces directional consistency of surface normals in the predicted depth map, ensuring alignment with ground truth and maintaining surface coherence for more accurate geometric reconstruction. The calculation details of the norm vector are discussed in the supplementary material Section C4L, In our approach, the enhanced depth prediction not only contributes to the geometry perception but also enhances matching performance for shape understanding, detailed in Section (3
3.3 Depth-Aware Matching and Pose-Solving
As Figure MB illustrates, RGB-only matching relies heavily on texture and local brightness, result- ing in two potential limitations: frequent mismatches between similarly textured foreground and background regions and poor performance in low-light areas such as shadows. These issues can de- grade the accuracy of pose solving by introducing unsatisfactory matches. To address this challenge, we propose fine-tuning-feet depth-aware matching module that effectively combines metric depth with RGB input to enhance spatial context understanding. We extend LoFTR [2Z]] by incorporat- ing corresponding depth maps as additional inputs, which enables the fusion of features between depth and RGB representations in the latent space (as shown in Figure ME). To preserve the well- trained feature extraction capability of LoFTR [27]]. we keep its parameters frozen while leveraging its coarse-tu torse-to-fii to-fine matching strategy on the combined features. Section C5 of the supplementary ma- terial provides a detailed mathematical formulation of this process. Finally, we use PointDSC [[5]] to refine the matched correspondences and estimate the relative pose T Tyery thus solving the desired 6D pose through T = T- 'T TT q-

---

Table 2: Quantitative result of metric depth estimation various benchmark datasets. The base- lines are fine-tuned with the corresponding training data for fair comparison. Our main results are highlighted with colored fonts and the best result of each column is shown in bold fonts.
Dataset
Method
81.05
61.10
61.25
RMSEL
log10
Abs. Rel..
Rq.Rel.J
MAEL
Unidepth (FT)
11.80
31.24
60.85
0.422
1.022
1.640
0.557
0.371
Depth-Anything (FT)
14.64
33.65
64.23
0.264
0.249
0.252
0.211
Light
Ours(25% data)
54.06
79.64
91.16
0.143
0,552
0.049
0.025
0.061
Ours(50% data)
62.57
86.96
95.39
0.137
0.521
0.044
0.025
0.055
Ourx(75% 75% data)
70.55
90.84
96.21
0.136
0.518
0.043
0.025
0.051
Ours
80.09
96.75
98.64
0.112
0.326
0.039
0.018
0.046
Unidepth (FT)
33.81
61.97
88.08
0.152
0.118
0.134
0.036
0.138
Depth-Anyyllan (FT)
29.87
44.82
0.288
0.216
Qurs(25% data)
28.64
53.12
86.23
0.184
0.127
0.142
0.036
0.138
REAL275
Ours(50% data)
32.75
56,96
87.83
0.175
0,109
0.108
0.122
755% data)
36.50
64,83
88.77
0.138
0.097
0.103
0.025
0.103
Ours
44,28
79.18
90.62
0.107
0.085
0.082
0.022
0.091
Unidepth (FT)
12.73
27.36
40.18
0.175
0.249
0.118
0.167
0.247
Depth-Anything (FT)
31.23
56.95
82.21
0.133
0.076
0.092
0.101
Ourst 25% data)
29.59
53.12
82.02
0.132
0.075
0.087
0.159
0.071
ClearPose
Ours(50% data)
42.69
63.75
86.55
0.080
0.066
77%% data)
49.92
70.61
92.76
0.106
0.062
0.073
0.138
0.060
Ours
54,30
79.65
96.87
0,103
0.064
0.066
0.129
0.059
RGB Sceneo
GT
Ours
SPIDepth
UniDepth
ScaleDetth
MetricD
Figure 4: Visualized depth prediction of ours and other metric depth estimation models: SPID- peth [53], UniDepth [89], ScaleDepth [S0]] and Metrie3D [63]] Ours performs more clearly than other basclines, preserving all valid pixels compared to the ground truth, which misses key values.
Experiment
4.1 Data Preparation and Experimental Setup
We evaluate our approach on multiple 6D pose estimation datasets, each selected for its unique chal- lenges. REAL227 [6]]] is chosen for its complex scenes with various objects. Toyota-Light [62], which is included in the BOP [[2]] challenge suite, provides a standardized testbed for evaluating robustness under challenging lighting conditions. To validate our method on transparent objects, we include ClearPose [[3]] and downsample the whole training set with step size of 100. During fine-
tuning, we freeze parameters in DPAv2 [26] and LoFTR [[2]] modules, training only our token scaler. Additional details on preprocessing and experimental setup can be found in the supplementary ma- terial (Sections [I]] and D.2)..
4.2 Baselines and Metrics
For monocular metric depth prediction, two state-of-the-art basclines, UniDepthv1 - and DPAv2 [26]], selected for their strong performance and widespread use. To evaluate metric depth quality, we choose various key metrics: absolute relative error (Abs.Rel.) and square relative error RSq. to measure prediction errors. The threshold accuracy (5,) gauges the proportion of predictions within max D/ D/D) < æ. The root mean squared error (RMSE) and mean absolute enror (MAF) are selected to judge overall quality. We also include MAE in log space (logı0) to
7

---

Table 6D pose estimation results on Real275 and Tyo-L dataset. We highlight our method with the colored font The best and the second best values of each column are reported with bold and underlined fonts, respectively.
REAL275 Dataset
Tyo-L Dataset
Matcher
Depth
AR 1
VSD
MSSD 1
MSPD 1
ADD
AR 1
VSD 1
MSSD
MSPD 1
ADD
Oracle
34.1
16.5
37.9
48.0
16.4
30.3
7.3
39.6
44.1
14.1
SIFT
DPAV2
1.2
3.6
Ours
12.2
4.9
12.2
19.7
5.1
20.5
5.6
22.1
33,9
11.0
Oracle
46.5
32.1
50.9
56.7
34.9
34.1
13.9
42.9
45.5
22.9
Oryon [
DPAV2
1.6
4.0
1.5
4.1
11.3
3.5
20.4
6.9
22.3
10.1
24.1
7.8
24.7
39.9
11.7
Oracle
41.7
28.6
44.9
53.8
30.0
36.8
19.6
44.7
46.1
24.5
RoMA [S
DPAV2
3.7
1.6
4.1
5.4
1.2
5.9
3.2
3.3
11.2
3.8
Oracle
56.8
41.2
63.0
66.2
56.9
42.0
34.2
44.1
47.6
35,0
Ours
DPAV2
4.1
2.1
3,5
6.1
1.4
5.8
2.8
3.5
11.1
2.9
Ours
28.7
29.9
11.6
31.7
13.9
33.8
47.3
13.1
vs. Oryon
+8.3
+0,9
+1.5
+7.6
+6.1
+9.1
better capture depth variations in distant objects. For 6D pose estimation, we adopt SIFT [64] and Oryon [[]] as reference-based baselines. We report Average Recall (AR) across VSD, MSSD, and MSPD [62] and compute ADD(S)-0.I) to assess 3D position error. Section D provides detailed mathematical definitions of each metric.
4.3 Quantitative Results
Table D presents the comprehen- sive results of depth prediction, and the visual comparison with more models is illustrated Fig- ure M Our method demonstrates strong performance. Using only half of the fine-tuning data, match Unidepth's 's [А performance. Nat- urally, performance improves with the scene scale when more training data is available. With all fine-tuning data, we achieve a 65% improve-
Table 4: 6D pose estimation results on ClearPose. We high-
light our method with the colored font. The best and the second best values of each column are reported with bold and underlined fonts, respectively.
Matcher
Depth
AR
VSD
MSSD
MSPD f
ADD 1
Manual
19.9
4,8
18.9
37.1
16.1
SIFT +
Raw
0.8
0.1
0.9
0.1
DPAV2
3.1
0.5
3.8
0.8
Ours
9.2
1.4
10.0
16.1
2.2
better comprehension
Manual
31.1
10.0
37.9
45.3
32.2
Raw
1.2
0.2
2.2
1.9
Oryon
DPAV2
8.8
1.5
10.5
14.3
3.3
17.1
ment in 61.05 over DPAv2 [26] in
Manual
32.3
9.3
39.0
48.5
33.1
Tyo-L [62]. In REAL275 [6]]] and ClearPose [[3]] we outperform DPAv2 [26] by 14.41% and 23.07%,
RoMA
Raw
1.5
0.3
1.2
3.0
2.2
DPAV2
8.9
4.0
Ours
17.7
4.0
19.1
29.7
8.6
Manual
32.4
10.1
38.8
48.2
33.4
respectively. Table B and g show the performance of pose estimation across the three benchmarks. Our method surpasses both SIFT [644] and Oryon [KI], achieving average AR im- provements of +15.3% and +6.5% with ground truth depth and +12.6% and +6.1% 1% with predicted depth, respectively. This results from our depth-aware matching. which better utilizes spatial infor- mation. Compared to DPAv2 [[6] (with fine-tuned head), our depth prediction produces substantial improvements: +14. 4% in accuracy with Oryon [6] matching and +20.3% with LoFTR- based matching approach. This improvement benefits from the proposed losses, which enhance the recon- struction of geometric characteristics.
Raw
1.4
0.2
2.0
Ours
DPAV2
9.1
1.6
11.1
14.4
3.9
Ours
19.4
5.1
19.7
33.5
10.0
vs. Oryon
+2.3
+1.3
+0.8
+3.9
+2.9
the Toyota-Light [62] dataset, our performance is lower than Oryon [6] with DPAv2 [26] depth, as depth prediction enrors affect pose estimation. Oryon [R]] benefits from textual prompts and VLM, achieving slightly better results. This highlights the importance of precise depth prediction for accurate pose estimation. Figure compares the 6D pose predictions of Oryon [6] and method.

---

Query
Refarace
Oryan
Ours
Tyo-L
Real275
1
Clear
Figure 5: A visual comparison of the predicted 6D pose across three datasets: the red point cloud with a 3D bounding box shows the prediction, and the green represents the ground truth. Our estimated pose exhibits less rotation emor and translation shift than the baselines.
From the figure, we can tell that SIFT [64] is unable to conduct precise 6D pose, while Oryon's depth-agnostic matching [B] increases rotation errors and translation drift.
Table B illustrates the efficiency compar- ison between our methods and the base-
Table 5:
Comparison of the computational require- ments of different matching methods.
lines. Our approach reduces computa- tional costs remarkably. This because Oryon [6] needs CLIPP [66]] to extract features, while RoMA [6S] is based on high-resolution feature maps processed by ViT [62]] and VGG [[S]]
ID
Method
#Params.
GFLOPs
Memory(GBB)
Oryon
264.3M
120.1
5,90
RoMA 55
111.3M
111.8
5.10
III
Ours [4
11.6M
13.9
0.74
Table 6: Ablation study on the effectiveness of Table 7: Ablation study on the different fine-tuning individual loss function REAL.275 dataset.
paradigms on REAL275 dataset. Full represents all pa- rameters in the depth head that will be updated.
Ledge Enseen
Abs.Rel_j
RMSE,
A (6 as)
31.16
0.279
0
-13.12
FT Part
FT Type
61.00
Abs.Rel.
RMSF,
v
35.18
0.188
0.215
-9.10
head
LoRA
24.91
0.318
0.371
V
V.
40.23
0.139
0.162
-4.05
head
Full
29.87
0.201
0.288
35.14
0.191
0,220
-9.14
head+scalor
LoRA 6
43.97
0.091
0.117
head+scaler
Full
44.37
0.080
0.102
44.28
0
0.107
scaler
N.A.
44.28
0.082
0.107
4.4 Ablation Study
Table shows the effectiveness of our loss components through ablation studies. Removing all three losses (Lsnales Ledges and Lnorm) leads to a substantial drop of -13.12% ō1.05 performance, demonstrating the importance of our composite loss function for accurate geometric estimation. Table @ presents the performance across various fine-tuning setups. Fine-tuning only the depth head DPAv2 [26], whether updating all parameters using LoRA [69], yields poor results. Integrating our token scaler into DPAv2 [[6]] markedly increases performance, proving the feasibility of our method. Considering the trade-off between the extra training burden and the marginal improvement, we chose to keep the depth head frozen.
Table M compares our depth-aware matching against standard approaches with Oryon [[]] RoMA [6S], and our LoFTR [RZ]]- matcher. The second column determines how depth in- formation is integrated with RGB information. In the absence of latent fusion, RGB-based and depth-based any) matching are performed independently, with the final correspondence obtained through the intersection of their respective results. Our method improves AR by +4.5% through

---

Table Ablation study on the effectiveness of individ- Table 9: Ablation study of diffcrent depth fu-
ual matching strategies on the Tyo-L dataset.
sion mechanisms for matching (Up), effective-
Use depth Latent fusion
Matcher
ART
ADD?
A(mean)
ness of token scaler and depth-aware matching
(Down) on the Tyo-L dataset.
RoMA x
19.4
6.7
-9.35
RoMA
20.8
7.2
-8.40
Coarse Feasure
Fine Feeaure
ART
ADD*
3.mmaa))
/
RoMA
30.6
13.0
-0.60
PE NI
PE P00
30.3
12.1
0.85
PE NI
Addiive
31.3
12.8
0.35
Oryon [
24.1
11.7
-4.50
Adddvv
PE M
31.1
12.5
4.60
Oryon
23.6
-5.15
Addivve
Addiivv
31.7
13.1
Oryon
29.7
12.6
-1.25
Depth Marching
AR1
21mean)
LoFTR 46
27.2
11.4
-3.10
-19.1
LoFTR
26.6
-3.40
LoFTR LB
31.7
13.1
latent fusion, which effectively incorporates depth as spatial cues to improve matching precision. Table E compares different feature fusion methods for depth-aware matching. We evaluated depth features as positional encoding (PE) signals [70] for LoFTR [M transformer layers and through direct latent space addition. Our experiments reveal that simple additive fusion outperforms the PE approach while being computationally more efficient. Table El reports the effectiveness of each mod- ule. Without the robust depth prediction, the projected 3D point clouds are unsatisfactory, leading to decline in performance. We also investigate the influence of other factors, such the view- point gap between query and reference, and the weights of each loss term. The detailed results are illustrated in the appendix.
5 Conclusion
We present a novel approach for 6D pose estimation that requires only single RGB reference image. We innovatively adapt DPAv2 [26] by fine-tuning it with a token scaler to predict met- ric depth. eschewing unreliable LiDeR- depth data. We neither render multiple novel views nor build object-specific 3D templates, perform sparse reconstruction with SfM, or NeRF [25]] In- stead, we directly predict dense depth map from single image using a frozen depth model (with lightweight tunable components), and fuse features in the 2D space. Integrating depth information into matching pipeline substantially reduces mismatches and improves correspondence density in low-texture regions, leading to more accurate pose estimation. As our method does not rely on synthesized views or neural fields, it has a notable generalization ability in different environments. Extensive experimental results demonstrate the effectiveness and versatility our method in various scenarios.
Limitations. Our approach uses object masks localize targets and constrain correspondence matching, limiting its applicability to scenarios with available segmentation masks. Additionally, its generalization is bounded by DPAv2 [26]] and pre-trained matching networks such as LoFTR [26]]. Failures may occur in extremely dark conditions, where the RGB camera captures little meaningful information.
Future Work. Our token scaler can be utilized to fine-tune other ViT-based models. Furthermore, depth-aware matching potentially enhances applications like scene reconstruction by providing geometric priors for multi-view images. Additionally, integrating VLMs for object localization can enhance accessibility and efficiency, ensuring smoother user experience for a wider audience.
Acknowledgement
This research is supported by the National University of Singapore under the NUS College of De- sign and Engineering Industry-focused FingeFenced PhD Scholarship programme. This research is supported by the National Research Foundation (NRF) "Centre for Advanced Roboties Technology Innovation (CARTIN)". and also supported by the National Robotics Programme (NRP) 2.0 funding initiative "Domain-specifie Robotics Foundation Models for Manufacturing DFM-RFM., The authors would like to acknowledge useful discussions with Dr. Bruce Engelmann from Hexagon, Manufac- turing Intelligence Division, Simufact Engineering GmbH.
10

---

References
[1] E. Marchand, H. Uchiyama, and F. Spindler. "Pose estimation for augmented reality: hands-on survey." IEEE transactions on visualization and computer graphics, vol. 22, no. 12, pp. 2633-2651, 2015.
[2] D. Bauer, P. Hönig, J.-B. Weibel, J. Garcfa- Rodriguez, M. Vincze et al. "Challenges for monocular 6D object pose estimation in robotics," IEEE Transactions Robotics, 2024.
[3] J. Guan, Y. Hao, Q. Wu, S. Li, and Y. Fang. "A survey of 6DoF object pose estimation methods for different application scenarios," Sensors, vol. 24, no. 4, p. 1076, 2024.
[4] J. Lin, L. Liu, D. Lu, and K. Jia, "SAM-6D: Segment anything model meets zero-shot 6d object pose estimation," " in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 27 279627916.
[5] B. Wen, W. Yang. J. Kautz, and S. Birchfield, "FoundationPose: Unified 6D pose estimation and tracking of novel objects," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2024, pp. 17 868 168-1785 17879.
[6] Corsetti, D. Boscaini. C. Oh, A. Cavallaro, and F. Poiesi, "Open-vocabulary object pose estimation." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). June 2024.
[7] V. N. Nguyen, T. Groueix, G. Ponimatkin, Y. Hu, R. Marlet, M. Salzmann, and V. Lepetit, "NOPE: Novel Object Pose Estimation from Single Image." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[8] H. Zhao, S. Wei, D. Shi, W. Tan, Z. Li. Y. Ren, X. Wei, Y. Yang. and S. Pu, "Learning symmetry -ware geometry correspondences for 6d object pose estimation." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 1404--14054,
[9] J. Sun, Z. Wang, S. Zhang. X. He, H. Zhao, G. Zhang, and X. Zhou, "OnePosc: One-shot object posc estimation without CAD models," in Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022, pp. 6825-6834.
[10] Y. Hc, Y. Wang, H. Fan, J. Sun, and Q. Chen, "FS6D: Few-shot 6D pose estimation of novel objects," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2022,
pp. 68 1824..224.
[11] F. Hagelskjær and R. L.. Haugaard, "KeyMatchNet: Zero-shot pose estimation in 3D point clouds by generalized keypoint matching," " in 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE). IEEE, 2024, pp. 870-877.
[12] A. Caraffa, D. Boscaini, A. Hamza, and F. Poiesi, "Freeze: Training free zero-shot 6D pose estimation with geometric and vision foundation models," in European Conference on Computer Vision (ECCV). Springer, 2024, pp. 431.
[13] X. Chen, H. Zhang. Z. Yu, A. Opipari, and O. Chadwicke Jenkins, "ClearPose: Large-scale transparent object dataset and benchmark," in European Conference Computer Vision (ECCV). Springer, 2022, pp. 381-396.
[14] Y. Liu, Y. Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang. "Gen6d: Generalizable model-free 6-DoF object pose estimation from RGB images," " in European Conference on Computer Vision (ECCV). Springer, 2022, pp. 298-315.
[15] B. Wen, J. Tremblay, V. Blukis, S. Tyree, T. Müller, A. Evans, D. Fox, J. Kautz, and S. Birchfield, "BundleDDF: Neural 6-DoF tracking and reconstruction unknown objects," Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023, pp. 606-617.
[16] D.N. White and J. Burge, "How distinct sources nuisance variability natural images and scenes limit human stereopsis," " bioRxiv, pp. 2024-02, 2024.
[17] Burge, C. C. Fowlkes, and Banks, "Natural-scene statisties predict how the figure- ground eue of convexity affects human depth perception." Journal of Neuroxcience, vol. 30, no. 21, pp. 7269-7280, 2010.
[18] E. Marinoiu, D. Papava, and C. Sminchisescu, "Pictorial human spaces: A computational study on the human perception of 3d articulated poses," International Journal of Computer Vision, vol. 119, pp. 194- 215, 2016.
11

---

[19] N. Cheng, Q. Dong, Z. Zhang, L. Wang, X. Chen, and C. Wang, "Egocentric processing of items in spines. dendrites, and somas in the retrosplenial cortex," Neuron, vol. 112, 4, pp. 646-660, 2024.
[20] Y. Shi, K. Barton, A. De Maria, J. M. Petrash, A. Shiels, and S. Bassnett, "The stratified syncytium of the vertebrate lens," Journal of Cell Science, vol. 122, no. 10, pp. 1607-1615, 2009.
[21] A. A. Bharath and M. Petrou, Nex generation artificial vision systems: Reverse engineering the human visual system. Artech House, 2008.
[22] M. Ramamurthy and V. Lakshminarayanan, "Human vision and perception." Handbook of advanced light- ing technology, pp. 1-23, 2015,
[23] D. Kingma and M. Welling, "Auto-encading variational bayes," arXiv preprint arXt: 2013.
[24] Ho, A. Jain, and Abbecl, "Denoising diffusion probabilistic models," Advances in neural information processing systems, vol. 33, pp. 6840-68511 2020.
[25] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. "NeRF: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM, vol. 65, no. 1, pp. 99 106, 2021.
[26] L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, "Depth Anything V2." arXiv:2406. 09414, 2024.
[27] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, "LoFTR: Detector-free free local feature matching with transformers," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 8922-8931.
[28] G. Pitteri, S. Ilie, and V. Lepetit, "CorNet: generie 3D corners for 6D pose estimation of new objects with- out retraining," in Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 0-0.
[29] G. Pitteri, A. Bugeau, S. Ilie, and V. Lepetit, "3D object detection and pose estimation of unseen objects in color images with local surface embeddings," in Proceedings of the Asian Conference on Computer Vision, 2020.
[30] M. Gou, H. Pan, H.-S. Fang, Z. Liu, C. Lu, and Tan, "Unseen object 6D pose estimation: benchmark and baselines," arXiv preprint urXiv:2206.1 11808. 2022.
[31] Chen, Sun, T. Bao, R. Zhao, L. Wu, and Z. He, "ZeroPose: Cad-model-baseed zero-shot pose estimation," arXiv preprint 2rXiv:2005.17934 2023.
[32] J. Huang, H. Yu, K.-T. Yu, N. Navab, S. Ilie, and B. Busam, "MatchU: Matching unseen objects for 6D pose estimation from RGB-D images." in Proceedings of the IEER/CVF Conference Computer Vision and Pottern Recognition (CVPR), 2024, pp. 10095-10 105.
[33] Lee, Y. Cabon, R. Brégier, S. Yoo, and J. Revaud, "MFOS: Model-free & one-shot object pose es- timation." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 4, 2024, pp- 2911-2919.
[34] Y. Du. Y. Xiao, M. Ramamonjisoa, V. Lepetit et al., "Pizza: A powerful image-only zero-shot zero-cad approach 6 DoF tracking." in 2022 International Conference 3D Vision (3DV). IEFE, 2022, pp. 515-525.
[35] N. Gao, V. Ngo, H. Ziesche, and G. Neumann, "SA6D: Self-adaptive few-shot 6D pose estimator for novel and occluded objects." in 7th Annual Conference on Robot Learning (CoRL), 2023.
[36] D. Cai, J. Heikkilä, and E. Rahtu, "GS-Pose: Cascaded framework for generalizable segmentation-basse 6D object pose estimation," arXiv preprint arXiv:2403. 10683, 2024.
[37] P. Pan, Z. Fan, B. Y. Feng, P. Wang, C. Li, and Z. Wang. "Learning to estimate 6DoF posc from limited data: A few-shot, generalizable approach using RGB images," in 2024 International Conference on 3D Vision (3DV). IEEE, 2024, pp. 1059-1071.
[38] V. N. Nguyen, T. Groueix. M. Salzmann, and V. Lepetit, "GigaPose: Fast and robust novel object pose estimation via one correspondence." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 9903 9913.
12

---

[39] D. Cai. J. Heikkilä. and E. Rahtu, "0VE6D: Object viewpoint encoding for depth-based 6D object pose estimation," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022, pp. 6803-6813.
[40] Y. Labbé, L. Manuelli, A. Mousavian, S. Tyree, S. Birchfield, J. Tremblay, J. Carpentier, M. Aubry, D. Fox, and J. Sivic, "MegaPose: 6D pose estimation of novel objects via render & compare," in CoRL 2022 Con--Coneeeecce on Robot Learning. 2022.
[41] Y. Liu, Y. Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang, "Gen6D: Generalizable model-free 6-DoF object pose estimation from RGB images," in ECCV, 2022.
[42] J. Wang, C. Rupprecht, and D. Novotny, "PoseDiffusion: Solving pose estimation via diffusion-aided bun- dic adjustment." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 9773-9783.
[43] T. Lee, B. Wen, M. Kang, G. Kang, I. S. Kweon, and K.-J. Yoon, "Any6D: Model-free 6D Pose Estimation of Novel Ohjects." in Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025, pp. 11 (33-11666 643.
[44] F. Di Felice, A. Remus. S. Gasperini, B. Busam, L. Oit, F. Tombari. R. Siegwart, and C. A. Aviz- zano, "Zero123-6D: Zero-shot novel view synthesis for RGB category-level 6D pose estimation," in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 14204 14211.
[45] C. Zhao, I. Zhang, and M. Salzmann, "3d-aware hypothesis & verification for generalizable relative object pose estimation," Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[46] C. Zhao, T. Zhang, 7. Dang, and M. Salzmann, "DVMNet: beyond hypotheses." in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2024, pp. 20 485-20444099 495.
Computing relative pose for unseen objects
[47] R. Ranftl, A. Bochkovskiy, and V. Koltun, "Vision transformers for dense prediction," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. pp. 179-12 188.
[48] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller, "ZoeDepth: Zero-shot transfer by combining relative and metric depth," arXiv preprint 2302v 12288, 2023.
[49] I. Piceinelli, Y.-H. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool, and Yu, "UniDepth: Universal monocular metric depth estimation." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[50] R. Zhu, C. Wang. Z. Song, L. Liu, T. Zhang, and Y. Zhang, "ScaleDepth: Decomposing metric depth estimation into scale prediction and relative depth estimation," arXiv preprint arXiv:2407.08187, 2024.
[51] A. Ganj. H. Su, and T. Guo, "HybridDepth: Robust metric depth fusion by leveraging depth from focus and single-image priors," in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IRFF, 2025, pp. 973-982.
[52] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, "Depth Anything: Unleashing the power of large-seale unlabeled data," " Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2024.
[53] M. Lavreniuk and A. Lavreniuk, "SPIDepth: strengthened pose information for self-supervised monocular depth estimation," in Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025, pp. 874-884.
[54] A. Kirillov, F. Mintun, N. Ravi, H. Mao, C. Rolland, L., Gustafson, Xiao, S. Whitchead, A, C. Berg, W. Y. Lo, P. Dollár, and R. Girshick, "Segment anything." arXiv:2304.0264: 2023.
[55] X. Bai, Z. Luo, L. Zhou, H. Chen, L. Li, Z. Hu, H. Fu, and C.-L. Tai, "PoinDDCC Robust point cloud registration using deep spatial consistency." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 15859 869.
[56] L. Zhang, A. Rao, and M. Agrawala, "Adding conditional control to text-to-image diffusson models." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3836- 3847.
13

---

[57] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez. D. Haziza. F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. "DINOv2: Learning robust visual features without supervision." 2023.
[58] Z. Shen, M. Zhang, H. Zhao, S. Yi, and H. Li, "Ffficient attention: Attention with linear complexities," in WACV, 2021.
[59] W. Yu, P. Zhou, S. Yan, and X. Wang, "Inceptiooneext When inception meets ConvNeXt," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 5672- 5683.
[60] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab, "Deeper depth prediction with fully convolutional residual networks," " in 2016 Fourth international conference on 3D vision (3DV). IEEÉ, 2016. pp. 239-248.
[61] H. Wang, Sridhar, J. Huang. Valentin, S. Song, and L. J. Guibas, "Normalized object coordinate space for category-level 6D object pose and size estimation," in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[62] T. Hodan, E. Michel, E. Brachmann, W. Kehl, A. GlentBuch, D. Kraft, B. Drost, J. Vidal, S. lhrke, X. Zabulis el al., "BOP: Benchmark for 6d object pose estimation," in Proceedings of the European conference computer vision (ECCV), 2018, pp. 19-34,
[63] W. Yin, C. Zhang, H. Chen, 7. Cai, G. Yu, K. Wang. X. Chen, and C. Shen, "Metric3D: Towards zero-shot metrie 3D prediction from single image," in Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023, pp. 9043-9054 9053.
[64] D. G. Lowe, "Object recognition from local scale-invariant features," in Proceedings of the seventh IEEE International Conference on Computer Vision (ICCV), vol. 2. lcce, 1999. pp. 1550-1157.
[65] Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, and M. Felsherg, "RoMa: Robust dense feature matching," " in Proceedings of the IEEE/CVF Conference on Compuder Vision and Puttern Recognition (CVPR), June 2024. pp. 19 790-19 800.
[66] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning (ICML). PMLR, 2021, pp. 8748-8763.
[67] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in International Conference on Learning Representations, 2020.
[68] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," " 2014.
arXiv preprint a409v;1409.1
[69] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., "LoRA: Low-rank adaptation of large language models." ICLR, vol. I, no. 2. p. 3, 2022.
[70] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
[71] Y. Li and T. Harada, "Lepard: Learning partial point cloud matching in rigid and deformable scenes." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022, pp. 5554 5564.
[72] S. Umeyama, "Least-squares estimation of transformation parameters between two point patterns," IEEE Transactions on Puttern Analysis & Machine Intelligence, vol. 13, no. 04, pp. 376-380, 1991
[73] P. J. Besl and N. McKay, "Method for registration of 3-D shapes." Sensor fusion IV: control paradigms and data structures, vol. 1611. Spie, 1992 pp. 386-606.
[74] H. Chen, P. Wang, E. Wang, W. Tian, L. Xiong, and H. Li, "EPro-PnPP PnP: Generalized end-to-end prob- abilistic perspective-nnpoint for monocular object pose estimation." " in Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition (CVPR), 2022, pp. 2781-2790.
[75] D. Brunet, E. R. Viscay, and Z. Wang, "On the mathematical properties of the structural similarity index." IEEF Transactions on Image Processing, vol. 21, no. 4, pp. 1488-1499, 2011.
14

---

[76] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, Girshick, P. Dollár, and C. Feichtenhofer, "SAM
2: Segment anything in images and videos," arXiv preprint urXiv:2408.0077 2024.
[77] D. Chetverikov, Svirko, D. Stepanov, and P. Krsek, "The trimmed iterative closest point algorithm,"" 2002 International Conference on Pattern Recognition, vol. 3. IEEE, 2002, pp. 545-548.
[78] Q. Liu, H. Zhu, Z. Wang, Y. Zhou, S. Chang, and M. Guo, "Extend your own correspondences: Unsuper- vised distant point cloud registration by progressive distance extension," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024, pp. 20 20816-20 826.
[79] Jung. S.-C. Wu, P. Ruhkamp, G. Zhai, H. Schieber, G. Rizzoli, P. Wang. H. Zhao, L. Garattoni, Meier, D. Roth, N. Navab, and B. Busam, "HouseCat6D large-scale multi modal category level 6D object per- ception dataset with household objects in realistic scenarios," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, pp. 22 22498- 22 508.
[80] D. Q. Huynh, "Metrics for 3D rotations: Comparison and analysis." " Journal of Muthematical Imaging and Vision, vol. 35, no. pp. 155 164,, 2009.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We develop novel framework for 6D pose estimation with single RGB reference. Our method mimics the mechanism layered human vision to fine-tune a depth estimation model that reweights the features of each stage. Thus, the 3D information is able to be obtained through metric depth estimation. Moreover, we propose fine-tuning-Fis method that fuses the spatial depth information into feature matching, which boosts the ultimate performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion section of the main body, we elaborate on some limitations of our work. These limitations can be further investigated to find suitable solutions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
15

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations their approach to ad- dress problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: In this work, we develop a novel architecture of the 6D pose estimation pipeline. There additional newly proposed theories, such optimization rep- resentation, etc.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: In the experiment section and appendix, we provide detailed settings for fine- tuning and data processing.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
16

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, deseribing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Partially]
All data we use in this paper comes from publicly available datasets. The code, for privacy reasons, included in the initial submission stage. Upon the situation of acceptance, we will consider releasing the code.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c
pub Lublic//an
semuisssionn ssionpolicyy for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (ittps:
www.isssssoooiiiiiioo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
17

---

Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes)
Justification: the experiment section and appendix, we provide detailed settings for fine- tuning and data processing.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We use the mean value as the justification metric. For each dataset, the final metric value is the mean across all samples. As the parameters of the model are frozen during the inference, therefore, the standard deviation of the same model inference on the same dataset is small enough.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We illustrate all hardware resources we use for this work in the experimental section and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///eerrr...cc////ollll/la
Answer: [Yes]
Justification: Our work is about novel object 6D pose estimation. After reviewing the NeurIPS Code of Ethics, our work conforms in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is about novel object 6D pose estimation. As it is standard task and does not involve any social-related issues, there no essential societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses
(e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system leams from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: All data and pre-trained models used by this work are publicly available and tested in many applications. There are such risks for this paper.
19

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use publicly available code resources
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcord com/ Gatasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re- e-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
20

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work focuses on 6D pose estimation with single RGB reference, which does not involve language models.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanstand components.
Please refer to our LLM policy (mm))) for what should should not be described.
21

---

Appendices of SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference
A Overview
The supplementary material starts here. Section E illustrates the hyperparameters we used for train- ing and inference. In Section KI some clarifications are illustrated in detail for definitions and cal- culations in Section El In Section D added more experimental details and clearly defined each metric mathematically. last, we visualize the influence cach layer's 's weight.
B Hyper-parameter Setting
In this section, we illustrate the detailed hyper-paramene parameter settings for our fine-tuning process in Table III
Table 10: Hyperparameter Settings based on the Tyo-L dataset.
Hyperparameter
Value
General
Epochs
80
RGB Resolution
640 x 480
Learning Rate
0.0005 3 0.001
Warmup Epochs
Weight Decay
Optimizer
Adamw
LR Scheduler
Cosine Annealing
Data
D min D max
0.3,8.0
Depth Estimator
Depth-Anything Depth Scale
19.0
Efficient Attention Head
4
Efficient Attention Hidden Dim
128
Inception Conv Branch Ratio
0.125
Token Dimension
256
Batch Size
16
LoFTR Matcher
Backbone
ResNet
Hidden Dims
256
0.
0.2
Attention Head Number
8
0.1
Loss Functions
Cscate Weight
1.0
1j in Lacale
0.2
Ledge Weight
0.7
a in Ledge
1.0
Enorm Weight
0.6
Creg Weight
0.5
a for BerHu loss
0.9
C Supplementation of Method
C.1 Details of Depth Estimation
Depth maps are crucial for extracting 3D information, especially for 6D pose estimation with single RGB reference. However, sensor-based based depth maps often contain invalid pixels, particularly
22

---

when capturing transparent black objects. Although human refinement of these maps can address such issues, requires significant investment in time and labor. To overcome these challenges, we propose novel pipeline that leverages pre-trained DPAv2 [26]] harnessing its superior general- ization capabilities and scene understanding to generate high-quality depth maps without manual intervention. To enhance metric depth prediction accuracy, we augment the pre-trained model with learnable token scalers, which are fine-tuned using a curated set of labeled data, resulting in signifi- cant performance improvements.
In DPAv2 [26]], for an RGB image I Hx It uses pre-trained DINOv2 [5>]]
extract multi- scale features denoted as {F, E R= 1i-1- F, is the feature output from the i-th stage. To obtain the feature map containing high-level semantic and low-level details, DPAv2 [26]] uses top-
to-bottom fusion to integrate the features. Specifically, a trainable fusion network F;, i E [1, k] is
adopted. Mathematically:
= FFF,,FF
(6)
where F. is the fused feature. Inside the fusion network. it interpolates the higher-level feature to the size of the lower-level feature and then uses convolution network to fuse two feature maps. For F',, since there no F the network will interpolate it to the size F--1 which can be treated as upsampling. After fusing the features all the way down to F1, the final logits D E IR HxW is obtained with conv-based depth-head:
D head(F'))
(7)
For our proposed method, the fusion process can be expressed as:
Fi (F,, Scaler(F(+.))
(8)
Moreover, we introduce a dynamic scale layer that predicts a scalar using F to F4' and trainable
linear layer.
H-IW-1
Tinear Σ W
ΣΣ F
k,,,
H
1=1
h=0 w==
where Linear(-) is a single linear layer, is a learnable parameter. Therefore, the estimated depth D is the prediction from the depth head multiplied by 8a.
Table I presents the impact of incorporating the dynamic scale prediction layer on model perfor- mance. Notably, this layer pro- vides an additional boost our models accuracy. Specifically, while the token scaler elevates 61.05 from 14.64% to 72.77% compared to DPAv2 [2G]], the inclusion of the dynamic scale layer further enhances performance by an additional 7.32%. To isolate its effect, we also evalu- ate framework that employs only the dynamic scale layer, omitting our token scaler. In this case, the improvement over DPAv2 [26] is limited to just 10.3%.
Table 11: Performance on whether using the dynamic scale layer on the Tyo-L dataset.
ID
8a
input
81.05 t
Abs.Rel.
RMSE4
72.77
0.051
0.148
II III
F'
75.43
0.042
0.142
Mcan([F',FFI)
80.09
0.039
0.112
Alternatively, the raw depth map Draw can serve as an optional input to our model. This approach stems from the idea that, despite potentially losing fine details, the raw depth map retains an accept- able overall sense of scale. By incorporating it, we can refine the global scale of our prediction more effectively. This process discussed in Section
C.2 Optional Depth Prior
In practice, the raw depth is usually available, although it might inconsistent and inaccurate. Our pipeline can seamlessly take the raw depth as additional input. We assume that the overall scale captured by the depth sensor acceptable. Therefore, if the depth prior is given, we can obtain the true minimum and maximum depth values from it. Thus, refining prediction with this "pseudo
23

---

ground truth" to obtain D,:
norm (D)m(DD min D mar D (raww)
Dj =
+
(10)
mar (Drau
where (D) is the normalized depth map to [0, 1]. Note that many raw depth maps, the minimal value is 0 due to depth deficiency. Therefore, only consider the valid depth pixels whose raw depth is larger than
C.3 Details of Global Loss
In this section, we conduct comprehensive analysis of existing loss functions to inform our global loss design, examining their relative strengths and limitations. The SSI loss MiDas [87]]] is defined as:
2H W Σ MSF ( xxx++t-00
Lasi
(11)
where MSE(-) is the mean square eπrоr, are predicted global scale and shift. Since our final goal to minimize the difference between D and D we optimize the t in closed-form:
H-W
(s, = arg min 8,t
Σ
sd. + -tddd
(12)
By defining = 1J 1D, 11. and g - [s, t]. the optimal can be obtain through solving the least-square problem in Eq. I12 The optimized vector g opt is:
H-W
H-W
opt
D.D
D,D.).
(13)
Alternatively. we can use Mean Absolute Error (MAE) to replace the MSE for more robust per- formance on outliers. Besides, the prediction and the ground truth depth map are then normalized through:
D D - median(D)
(14)
EE D - median(D)i
M
where M is the pixel number in the valid mask. Although this more tolerant when facing more outlines, the normalization operation destroys the natural metric depth output; therefore, the MSE version of the SSI Loss is preferred for metric depth estimation. The SiLogLoss, unlike SSI Loss, supervises the overall scale in log space.
Σ E(10g) (log (e*d;") log (d:'))²
Lalog min 2M
(15)
However, the SiLogLoss ignores the potential unknown global shift where SSI Loss is not absent. Therefore, it is not selected in our pipeline.
We additionally use gradient matching term 47 to match the depth change between prediction and ground truth.
Greg M
(
d:)* V aidd-aal d.)+l)
-
(16)
where (d. dh denotes the difference of disparity maps scale k. We K scale levels, halving the image resolution at each level.
For the BerHu term [60]. it is calculated with:
DD D
if D DJ c
BerHu(D.D)
(D-
(17)
if D >
2c
where c is a threshold and set to 0.1 multiplied by the max value of [D D in our experiments.

---

C.4 Details of Normal Consistency Loss
As mentioned in Section E. the normal consistency loss focuses on optimizing the surface repre- sentation in the predicted depth map. Since the camera intrinsic is available, the projected point cloud P can be obtained from the corresponding depth D. Then, we can calculate the difference vector of adjacent points, i.e., calculate the fire-dimeeniionl position difference corresponding to the adjacent horizontal and vertical pixels of each point:
Pa P 11 Pu, Pu,es
P. P pu,+11 Pu..:
where Pu,. YX is the three-dimensional point corresponding pixel (u, v). The normal vector can then be computed with the cross product:
= P X
The weight w; aims to reduce potential outliers due to edge, occlusion, distortion and to ensure a smooth, consistent surface representation. Therefore, it is calculated as: exp (=)||[[DD,))) This exponential-boe based format ensures lower values in regions with large local depth gradients (typically representing object boundaries) and higher values in areas with small local depth gradients (usually corresponding to continuous object surfaces). Figure & illustrates the visualized weight map. It is obvious that the weight for Forrm on the objects' edges is small, while the surfaces maintain large weights. Note that didn't filter the map with the objects' mask. In the real implementation, only foreground objects are considered.
Depth- based Normal Consistency Loss Weight Map
0.9
0.8
0.7
0.6
Figure 6:
The RGB and the predicted depth map (Left) and visualized weight map for Cnorm (Right). The green color represents large-weight area, while the white and pink colors represent weight areas.
C.5 Details of Depth-Aware Matching
Specifically, two RGB images I, I,,, and corresponding depth map D,, D, are provided. They first extracted by an image encoder simultaneously for feature representation with two output scales (one is 1 of the input resolution coarse feature, the other of the input resolution fine feature):
Pi (p enc(I.) + norm(enc(D:)):t e (g.r),
coarse
(18)
where enc(-) represents the pre-trained encoder. Note that the input depth maps are normalized to maintain a consistent scale. To ensure that depth information complements rather than dominates the matching process, we empirically normalize the depth features so they serve as auxiliary signal. Experimental results demonstrate that without proper normalization, depth information overwhelms the matching process, leading to suboptimal performance.
25

---

w/o depth
with depth
Figure 7: A comparison of matching results between RGB-only (left) and our proposed depth-aware
(right) matching pipeline. The colored boxes with connected lines represent obvious mismatches.
The coarse feature is first processed by a transformer decoder to calculate the similarity:
S(i.)) (doo(y dec((p )5)}
(19)
where dec(-) transformer decoder. The matching probability is then obtained with dual- softmax [27]:
P(i.j) = softmax(S(i, -)) (oftmax(S(..))).
(20)
The coarse matching correspondence is then calculated with defined threshold 0.:
M = ((, v(,J) MNN (P) P(.,))
(21)
where MNN(-) stands for the mutual nearest neighbor. Following the approach outlined by of local windows from the fine feature maps based the coarse matches M.. We then compute the correlation between the center vector of dee((slin)) ); and all vec- tors in dec(yp!ine),, ),, generating heatmap that represents matching probabilities between pixel and each pixel in the neighborhood j. By calculating the expectation over this probability distribution, we determine the pixel position on I,.. The aggregation of all these correspondences yields our final set of fine-level matches M,-
LoFTR [27]], we extract
Figure Ø presents a detailed comparison between our depth-awand matching and the RGB-only matching. As shown the figure, RGB-only matching struggles differentiate between foreground and background pixels (indicated by yellow and red lines). This is because it primarily relies on tex- ture and color similarity, neglecting the spatial relationship. Additionally, the density of matched points is sparse in regions with low RGB values, such as the black screen of the computer, due to the
26

---

lack of valid information in those areas, In contrast, our depth-aware matching effectively mitigates these issues, resulting improved performance.
With such a correspondence, we can obtain the relative pose ---r through deterministic point cloud registration methods [55,, [Z]] rigid transformation-solvingg algorithms [72-774]. Specifically, within the given coordinate system, the ground-truth 3-D point cloud P, 9 (under the world coordi- nates) can be transformed the 3-D query point cloud Pa and reference point cloud P, with
P3 T P T.
(22)
where the T and T, the pose matrix (from camera coordinate to world coordinate) for query and reference, respectively. As the relative pose can be used to transform the query point cloud to the reference point cloud as: P, To-r q. we can compute the 6D pose of the query object with:
T TT
TT T 9-rr
(23)
D Supplementation of Experiment
D.1 Details of Dataset Preparation
For the REAL275 [6]]and Tyo-Light [62] datasets, we randomly select 80% of the separated scenes as fine-tuning data, and the rest of the scenes are processed as testing data. The ClearPose [[33] dataset was constructed using sets 1, 5, and Scene from set and scene from sets 4-7 were designated as the test set due to their distinguished visual environments, while the remaining samples were randomly allocated to the training and testing sets. We treat the scene with a distinct background as a novel scene, although some of the objects are similar (This is because the appear- ance of the transparent object is heavily influenced by the background). As the original clearpose provides dense, continuous views for scene, we use the sparse version of it, which is downsampled 100 times. The processed clearpose dataset contains 3129 images for training and 643 images for testing. For testing, we assign objects with distinct backgrounds and shapes as the targets. For each dataset, the ground truth depth maps are clamped to given range [D min: D Dmaz] and the specific value setting is illustrated in the supplementary material Section [B]
Table 12: Average SSIM between fine-tuning data and test data
Dataset
SSIM
Tyo-L
0.2476 (# 0.1372)
REAL275
0.1676 (# 0.0299)
ClearPose
0.1812 (= 0.0410)
During the experiments, we manually split the data for fine-tuning and testing to ensure the test scenes not seen in any format by the model during fine-tuning. To quantify the similarity be- tween test and fine-tuning data for each dataset, we use the Structural Similarity Index Measure (SSIM) [5]] and then report the mean and standard deviation for each dataset.
From Table I2 we can observe that the SSIM [A scores between the fine-tuning scenes and test scenes are relatively low, indicating limited information overlap and acceptable separation.
D.2 Details of Experimental Setup
We fine-tune our model for 80 epochs with a batch size of 16 for all datasets except ClearPose [[]] which is fine-tuned with 40 epochs for a smaller time consumption. We adopt an Adam W optimizer with an initial learning rate of 0.0005, updated by cosine annealing scheduler with 3 warmup epochs. To avoid overfitting, we assign weight decay of 0.01 to the optimizer. For our proposed token scaler, the hidden dimension of each layer is 256, the head number of the efficient attention layer is 4, and the intermediate dimension is 128. For the inception-conv unit, the channel number of a convolution branch set to 32, and the intermediate dimension is set to 128 as well. All the token scalers are followed by a zero-initialized residual convolution layer, which is inspired by ControlNet [56]. The detailed hyperparameters and experiment environment settings are illustrated in the supplementary material Section B. We fine-tune the baseline models for 30 epochs using their
27

---

respective original loss functions while keeping all layers frozen except for the depth prediction heads. In Table Q these fine-tuned variants are denoted as UniDepth (FT) [99] and Depth-Anything (FT) [26].
For the relative pose-solving part, the crop ROI size of the target object is x 256 for efficiency. We adopt a pre-trained PointDSC [[S]] to solve the rigid transformation between point clouds with given correspondence points. Point correspondences within the object region were extracted using the matching result and filtered by an object mask provided by the dataset or segmentation mod- els [54]]. In REAL275 [6]]]] we select objects from distinct scenes to construct 2000 image pairs. In Tyo-L [62]], we select 2000 image pairs where each query and reference image is captured under different lighting conditions. In ClearPose [CC] we construct 1000 image pairs which the query and reference images have different backgrounds. using 6 different objects. All of our experiments are conducted on an Ubuntu 22.04 server with two Nvidia RTX3090 GPUs. The deep learning framework is PyTorcc 2.2.0 with CUDA 12.4.
D.3 Details of Evaluation Metrics
The absolute relative error (Abs.Rel.)
D
Rel.
D +
and its quadratic variant, the square relative erron(Sq.Rel.)
D D-D
Sq.Rel.
D +
quantify the local deviation between predicted (D) and ground-truth (D) depths. e is a small value to avoid division by O.
The root mean squared error (RMSE) and mean absolute error (MAE) are defined as:
H
W
RMSE
(d.
dzy))
HW
j=1
MAE
|duz disl
HW
f=1
The (logı)) metric is calculated with:
H
W
logi0
Σ
1og (day)
- 1o
HW
1=
For the 6D pose estimation metrics [62], Visible Surface Discrepancy (VSD) quantifies the spatial gap between an object's surfaces when comparing its true position to estimated position.
VSD ((P Σ Dp(p)) - Dp(p))
where P is the predicted pose, P is the ground-truth, and 8 is distance threshold. ((),)) repre- sents the set of pixels visible under pose P and whose depth difference is less than 8.
The greatest distance between two poses' surfaces, accounting for symmetry, measured by the Maximum Symmetry-Aware Surface Distance (MSSD):
MSSD(P, P. SM, VM) min max SESx XEVx
| P PPX-PSX|
where the set Sa contains global symmetry transformations of the object model M, Var is set of the model vertices.
28

---

The Maximum Symmetry-ANwe Aware Projection Distance (MSPD) aims to measure how visually differ- ent an object appears (measured pixels) when it's rendered using an incorrect pose while account- ing for all possible symmetries.
MSPD min max SESM XEVM
proj( - PPo
where proj(-) represents the operation of projecting three-dimensional point onto an image plane.
ADD(S)-0.14 measures the recall in pose estimation based threshold: pose is considered correctly estimated when its error less than 10% of the object's diameter. For an asymmetric object, the error is computed as:
ADD 1
Σ XEVM
Px Px
VM
where |Vrl represents the size of the vertice set, Px and Px represent the position of vertice x in the estimated and true poses respectively. For symmetric object:
ADD-S
Σ
min
Px
Px
VVM
X'EVar
x VM
where x' represents the closest symmetry point to x in the 3D model.
D.4
Sensitivity Analysis
We conduct the sensitivity analysis the weight of Lnorms Ledges and Lscate- Figure N illustrates the curve the results. We consolidate the final value of weights through extensive experiments.
D.5
More Experimental Result
D.5.1
Cross-domain Depth Fine-tuning
To further validate the cross-domain generalization capability of our method, we conducted addi- tional experiments involving domain transfer evaluation. Specifically, we fine-tuned the depth esti- mation model on single source dataset (e.g.. REAL275 [6]]], which primarily contains industrial and everyday objects under static lighting conditions) and directly evaluated its performance on target datasets with different characteristics, ClearPose [T3]] (transparent objects in complex spatial environments) and Tighta-Light [62]] (dynamic lighting variations)
any domain adaptation or additional fine-tuning. These three datasets exhibit distinct scales and environmental characteristics with substantial variations in camera viewpoints and intrinsic parameters, thereby enabling com- prehensive validation of our method's cross-domain generalization capabilities. The cross-domain evaluation results are presented in Table IK!
Table 13: Quantitative result of metric depth estimation with cross-validation. The first column denotes the generalization direction from the fine-tuned dataset to the evaluation datasets.
Setting
Method
6100
6,10 1
6,20
RMSEL
log10
Abs.Rel.j
Sq.Rel.L
MAEJ
Unidepth (FT)
8.35
22.61
42.01
0.983
L419
1.855
0.702
0.500
REAL275 Tyo-L
DPA v2 (FT)
10.55
28.93
50.23
0.519
0.764
0.399
0.177
0.326
Ours
28.13
34,91
55.72
0.377
0.471
0.223
0.119
0.218
Unidepth (FT)
8.74
14.02
25.91
0.364
0.268
0.294
0.312
0.273
REAL275 ClearPose
DPA
20.85
44.48
62.36
0.258
0.232
0.207
0.280
0.181
Ours
42.19
66.25
79.70
0.182
0.106
0.131
0.228
0.109
The experimental results show that cross-domain performance drops expected, reflecting the do-
main sensitivity inherent in depth estimation tasks. Nevertheless, our method consistently outper- forms fine-tuned baselines (e.g., DPA v2 [26] and UniDepth [49]] under the same cross-domain settings. This superior generalization stems from our token scaler's design philosophy. Unlike meth- ods tailored for specific datasets, it adaptively modulates intermediate features in data-agnostic manner. This generic approach enables better generalization across diverse visual inputs without being constrained by particular dataset characteristics.
29

---

61.00 05
44.2
44.1
0.2
0.4
0.8 Ledge Weight
0.6
1.0
1.2
(a) Performance differnt weights of Ledge
100544 (
61. 61.05'
44.0
43.5
0.2
0.4
0.8 Scaale Weight
0.6
1.0
1.2
(b) Performance to differnt weights of Lscate-
61.05
44.2
44.0
0.2
0.4
0.8 n Weight
0.6
1.0
1.2
(c) Performance to differnt weights Lnorm-
Figure 8: Sensitivity analysis the weights of our proposed losses REAL275.
Moreover, the introduced loss functions are designed to regularize the model for consistent geomet- ric understanding. By leveraging cues such as edges and surface normals, they focus on capturing generalizable geometric structures rather than overfitting to the distribution of a particular dataset. Therefore, this suggests that the improvements brought by our approach are not solely due domain- specific tuning but also stem from its advanced intrinsic mechanism. Besides, the proposed modules are inherently transferable and can be seamlessly integrated into other visual tasks (such as depth- based navigation, etc.), demonstrating strong potential for broader applicability beyond the current setting for other directions.
D.5.2 Results On Additional Pose Benchmarks
We conducted two additional experiments on LM-O [62] and YCB-V [62],, which are representa- tive benchmarks featuring occlusions and weakly textured objects, such as monochromatic, single- material objects. We evaluate pose estimation on three different target objects, generating 120 image pairs for each. Table reports the quantitative results.
The experimental results show that heavy occlusion still causes a significant drop in performance. However, our method consistently outperforms the baseline significantly within the single RGB ref- erence setting. This improvement stems from our framework's human vision-inspired mechanism, which adaptively adjusts features across different layers using both appearance and spatial informa- tion. While conventional approaches like Oryon [[]] or SIFT [64]] rely primarily on RGB similarity, our matching strategy incorporates spatial context to improve alignment accuracy. Additionally, we maintain spatial consistency throughout both coarse and fine-grained matching stages (see Figure MI.. These designs result in substantially better robustness, particularly challenging scenarios where appearance cues alone prove inadequate, such under heavy occlusion.
30

---

Table 14: 6D pose estimation results on YCB-V and LM-O dataset.
Dataset
Matcher
Depth
AR 1
VSD 1
MSSD f
MSPD 1 ADD
Oracle
6.1
3.1
6.9
8.3
5.5
Oryon
Ours
3,8
1.9
4.2
5,4
2.7
LM-O
Oracle
9.9
4.3
11.5
13.9
8.7
Ours
Ours
5.3
2.6
5.7
7.3
3.8
vs. Oryon
A
+2.7
+0.95
+3.1
+3.8
+2.2
Oracle
8.6
4.7
9.3
11.8
7.5
Oryon
Ours
5.8
2,4
6.6
8.4
5.2
YCB-V
Oracle
11.4
5.2
12.3
15.7
8.9
Ours
Ours
7.9
3.7
8.5
11.6
6.4
vs. Oryon +2.45
+0.90
+2.45
+3.55
+1.30
For multi-object situations, pipeline naturally extends to these scenarios by simply incorporat- ing an off-the-shelf object detector or segmenter, such as SAM [5_ or SAM2 [766]. This design highlights the generality and scalability of our approach, making it readily adaptable to diverse real- world applications without requiring major architectural changes. The experiment results illustrate the robustness of our method under occlusion scenarios.
D.5.3 Comparison With Additional Baselines
We find that Any-6D a FS6D [IС,, and the original LoFTR [27] comparable methods. There- fore, to further investigate the performance of them compared to our SingRef6D, We conduct several experiments and report the results in Table
Table 15: More 6D pose estimation results on the Tyo-L dataset.
Matcher
Depth
AR
VSD
MSSD
MSPD
ADD
Oracle
14.1
5.2
17.9
19.2
10.0
FS6D [00]
DPAV2
1.6
0.5
1.9
2.4
0.3
Ours
9.3
1.8
6.3
7.8
3.2
Oracle
43.3
15.8
55.8
58.4
32.2
Any-6D
DPAV2
5.2
1.3
4.3
11.0
2.2
Ours
31.6
13.5
33.5
47.8
13.2
Oracle
42.0
34.2
44.1
47.6
35.0
Ours
DPAV2
5.8
2.8
3.5
11.1
2.9
Ours
31.7
13.9
33.8
47.3
13.1
Our method outperforms FS6D [00 across the selected evaluation metrics. Any-6D [+ benefiting from its strong 3D reconstruction backbone, achieves slightly better results than ours when using annotated depth. However, it exhibits strong dependency on the quality of depth inputits perfor- mance drops significantly when using DPAv2-predicted depth. This is because Any-6D [X3] adopts a more global optimization strategy (e.g., point cloud registration or global pose supervision), which effectively constrains the maximum deviation (results in higher recall on MSSD and MSPD). How- ever, the accumulation of depth errors in local regions leads to lower recall of VSD. In contrast, method emphasizes local pixel-wied wise alignment (e.g., fine-grained optimization based depth maps), resulting in higher recall on VSD. Nonetheless, it is less robust handling certain keypoints or symmetries, which leads to lower recall MSSD and MSPD.
The performance gap between Any-6D differences seen when switching from oracle to predicted depth within each method, show that our pipeline more robust and overall superior for both pose estimation and depth prediction. Further- more, our method delivers more stable depth reconstruction and enhanced geometric awareness for pose estimation. Designed modularly, our methods can integrated seamlessly into downstream pose estimation pipelines or incorporated as components within other depth estimation models. On
[ and our method with different depth inputs, and the
31

---

the Tyo-L [62] dataset, vanilla LoFTR [27]] for RGB-basei matching yields an AR of 27.2, while our method achieves 31.7. This demonstrates more robust and accurate feature matching our method under identical conditions.
In addition, conduct the comparison studies with DVMNet [46] and 3DAHV 15 by iteratively inferring from the image pair. Table L reports the corresponding results.
Table 16: Comparison with other models on LM-O dataset
Method
AR
ADD
Angular Error
3DAHV [55]
7.7
6.3
52.71
DVMNet [66]
8.6
7.4
51.66
Ours
9.9
8.7
51.89
While method exhibits slightly higher angular compared to DVMNet [[6], it achieves superior performance in terms of AR and ADD, reflecting more accurate translation estimation. This difference stems from the fact that DVMNet [[6] and 3DAHV [[S] are supervised with full ground-truth metric poses and trained on large-scale datasets with extensive pose annotations. In contrast, our method is only fine-tuned using predicted depth and does require additional pose supervision.
Furthermore, DVMNet [96 and 3DAHV |5| focus relative pose estimation under the same- scene conditions with encoded latent 3D representation, whereas our method is designed to handle cross-scene situations and estimate 6D poses. This highlights key distinction: DVMNee [46] is suitable for applications requiring accurate SO(3) rotation estimation in stable environments, while approach provides more generalizable and lightweight solution for full-pose estimation without dependence on dense pose labels. This advantage facilitates the implementation of our method in real-world cases.
D.5.4 Comparison With Additional Registration Methods
Iterative Closest Point (ICP) M is well-known algorithm for point registration; we admit ICP [[Z]] baseline worthy of comparison. We conducted ICP [[Z]] experiments using point clouds from both reference and query views. The results are shown in the Table [77
Table 17: 6D Pose estimation result on the ClearPose dataset compared to ICP.
Matcher
Depth
AR t
VSD t
MSSD 1
MSPD f
ADD) 1
Manual
19.8
5.5
23.2
32.9
20.6
ICP
Raw
0.3
0.04
0.25
0.61
0.9
Ours
8.1
2.6
9.4
12.4
4.5
Manual 32.4
10.1
38.8
48.2
33.4
Ours
Raw
1.4
0.2
1.2
2.8
2.0
Ours
19.4
5.1
19.7
33.5
10.0
ICP [12]] performs unsatisfactorily due its sensitivity to initialization and noise, making correspon- dence unreliable. In contrast, our depth-aware matching operates in latent space, preserving pixel- level consistency and mitigating the pitfalls of noisy 3D reconstructions. Our method is enriched with spatial cues in coarse-to-fine manner, further boosted following pose solving, performing better under transparent and low-texture conditions where ICP struggles.
We also consider learning- based registration methods [78, [7]] to substitute ICP [77]] Table 18 shows that although LePard [Z]] and EYOC [ are notable 3D registration methods, their performance under the proposed setting is lower than our method. Directly applying registration methods to raw reconstructed point clouds yields suboptimal performance due to two key limitations. First, in our experimental setup, there is only reference sample, which is not guaranteed to originate approaches
from the same scene as the query. This makes it challenging for registration- to infer reliable relative spatial relationships from scene geometry alone, a single cross-scene
32

---

Table 18: 6D pose estimation results on three benchmarks compared learning -based registration methods.
Method
Real275
Tyo-L
ClearPose
AR
ADD
AR
ADD
AR
ADD
LePard [|]]
11.9
5.1
14.8
6.5
8.0
4.2
EYOC [78]]
13.1
5.5
15.2
7.0
8.3
4.4
Ours
28.7
11.6
31.7
13.1
19.4
10.0
RGB reference inherently provides a limited overlapped region (even using ground-truth depth). Second, point clouds derived from depth estimation are frequently incomplete due to occlusions and viewpoint variations, which significantly undermine registration effectiveness.
In contrast, our method first employs depth-aware matching to establish initial correspondences between query and reference images. These correspondences directly guide subsequent pose es- timation, enabling our pipeline to reduce dependence on the completeness of reconstructed point clouds. This design allows method to maintain robust performance even in cross-scene scenar- ios or when dealing with partial or noisy point cloudsa key advantage over approaches that rely exclusively on learning- based registration.
Additional experiments reveal that applying ICP [77]] refinement after the initial registration leads to only marginal gains (for LePard [Z]]]: +0.12 AR Real275 [[]]]. +0.06 AR ClearPose [[3]] and +0.11 AR on Tyo-L. [62]]) with negligible differences ADD. This underscores the role of accurate initial correspondence and further substantiates the effectiveness of approach. Notably, despite utilizing a lightweight architecture, our method surpasses learning-basse registration models under our single-RGB cross-scene reference settings, demonstrating both the practicality and superiority of the proposed framework.
D.5.5 Results on Benchmark with Reflective Objects
To provide analysis on reflective objects specifically, we fine-tuned our depth model subset of the training data and evaluated 6D pose estimation on the validation subset of HouseCat6D [79]] We manually selected three reflective objects (teapoint, knife, and cup) as targets and generated 100
image pairs for each.
Table 19: More 6D pose estimation results on the HouseCat6D dataset.
Matcher
Depth
AR 1
VSD f
MSSD t
MSPD 1
ADD f
Oracle
57.4
41.0
62.6
68.7
68.8
Oryon +
DPAV2
26.6
9.1
33.6
37.3
39.5
Ours
38.1
20.8
44.4
49.3
49.0
Oracle
63.2
48.2
73.0
78.6
76.2
Ours
DPAV2
30.1
12.0
41.9
46.3
43.8
Ours
44.5
26.1
51.8
55.5
55.2
vs. Oryon
A
+6.4
+5.3
+7.4
+6.2
+6.2
As shown in Table [ method outperforms the bascline and achieves higher depth quality pared to the vanilla fine-tuned DPAv2 [26]]. This is because baseline methods like Oryon [K]] heavily rely on RGB information, while the HouseCat6D [79] dataset is specifically designed to challenge such dependence. provides dynamic variations in RGB cues through high-resolution images and carefully selected reflective objects, which naturally exhibit different surface appearances from vary- ing viewpoints. This makes it difficult for RGB-dependent methods like Oryon [[S] to establish reliable correspondences.
In contrast, our approach employs coarse-to-fine matching strategy that effectively leverages depth cues. This enables robust matching even when RGB signals unreliable, as method can rely
33

---

Figure 9: The rendered images obtained from different viewpoints are referenced with respect the Z axis.
geometric structures and spatial consistency. Furthermore, the of high high-resolution feature maps allows for refined correspondence estimation, reducing errors in challenging cases. These results not only demonstrate the architectural advantage of our method but also highlight its practical applicability in real-world scenarios where RGB signals may dynamically vary.
D.5.6 Influences of Query-Referenne Viewpoint Gap
We conduct experiment that manually rotates the scene with specific angle [15", 150%] along the X,y, and Z axes, respectively. Specifically, we use accessible 3D models (e.g.. from BOP [62]] datasets) to construct scene with BlenderProc and rotate the camera by a (The reference scene and the query differ in background and contain different objects, except for the target object). We define right-handed coordinate system, where the Z-axis points vertically upward, perpendieu- lar to the object surface, and the X and Y axes lie in the horizontal plane. The camera is positioned (1, 1, 1), measured in meters, looking down the scene, with all objects placed on the XY plane. Since most existing benchmarks adopt a top-down perspective, we follow the same setup. Figure B illustrates an example of this albation environment. To evaluate the effect of viewpoint variation, we report results under different camera rotations along the X and Z axes. Rotation along the Y axis omitted, as we observed strong correlation between rotations around the X and Y axes during the experiments, with Spearman coefficient exceeding 0.86. This correlation is likely due to the symmetry of the setup. Therefore, we report only the Xaxxis rotation in the Table horizontal-axia variations. In future revisions, we plan to include finer mer-ggained angular distinctions and explicit values for Y-axis rotations on more scenes.
represent
Table 20: Results of rotation differences between reference and query viewpoints along three axes.
Asislu
15a
30"
45"
60'
75'
90"
120
150"
AR
ADD
AR
ADD
AR
ADD
ADD
ADD
AR
ADD
AR
ADD
AR
ADD
X
69.84
65.70
61.17
60.42
50.52
54.38
42.36
46.91
33.71
38.64
30.12
33.47
22.83
24.36
20.54
21.90
74.26
72.19
71.50
68.13
67.21
65.80
64.02
62.42
60.57
58.31
39.76
36.73
31.95
28.54
From the table, we observe that pose estimation performance degrades with increasing viewpoint differences between the reference and query images, especially when the rotation is along the X- axis. This is because X-axis rotations significantly alter perspective and occlusion patterns, making more challenging for the model to match spatial features. In contrast, Z-axis rotations mainly induce in-plane transformations with relatively mild geometric distortion, leading to a more gradual performance drop. However, when the rotation exceeds 90%, even Z-axis changes cause noticeable degradation. This is likely due to decreased visual discriminability for example, the front of mug may have a distinct texture or logo, while the back plain, reducing the object's appearance discriminability and making it harder for the model rely appearance and forcing it to depend solely on geometry and spatial consistency.
Despite this challenge, our method exhibits smaller performance drop (65%) under such extreme conditions, while baselines like Oryon [6]] and SIFT [[4]] fail almost entirely, with performance degradation exceeding 90%. This robustness stems from our models ability to effectively exploit spatial consistency and adapt spatial representations through the proposed token scaler, thus boosting geometric understanding in the coarse-to-fine matching process. This mechanism helps to improve matching accuracy when RGB cues are unreliable. Furthermore, our loss functions impose strong geometric constraints such as normal consistency and edge emphasis, enabling the model to maintain
34

---

Table 21: Pose Estimation Result (AR) with Binned Query-Reference Viewpoint Difference
Bin
Real275
Tyo-L
ClearPose
0,
x
33.1
41.5
31.1
4
27.2
34.2
20.1
[ 3x
23.3
30.1
16.6
ix .
19.9
26.8
11.8
a reasonable estimation accuracy even under large viewpoint changes. These results and analyses demonstrate the robustmess of our model and highlight its potential for real-world applications.
To further investigate the influence of the viewpoint gap statistically, we provide quantitative break- down of the pose estimation results these three datasets, binned by the query-reference viewpoint difference with the geodesic distance [80]. We adopt the same definition as in [80]] where geodesic distance unit sphere is calculated with SO(3) rotations and ranges from to π. Based on this, we performed a bin-wise statistical analysis on the three benchmark datasets used in our study. The ARs are shown in Table An Note that all statistical numbers are calculated based ground-truth pose annotations. We conducted basic stability check and removed a few outlier samples with abnormally low overlap, ensuring that the resulting subset maintained similar distribution to the overall dataset. Additionally, we examined the distribution of ground-truth distances and found that the proportions across different intervals were generally consistent, with no single range overwhelm- ingly dominating the distribution.
The performance of pose estimation deteriorates the geodesic distance between the query and ref- erence increases. On the ClearPose [|]]]] dataset, we observe that when the geodesic distance range shifts from [0, 3 to [ 37), the Average Recall (AR) of our method decreases by approximately 29.6%, whereas AR of Oryon [6] declines by 50.3%. This further substantiates the superiority of our approach over the baseline under the proposed evaluation setting. It is important to note that, since our query query-reference pairs originate from different scenes, additional factors-such variations in background, occlusions caused by surrounding objects, and changes in scene layout-also influ- ence performance fluctuations. For instance, on the REAL225 [[]]] dataset, the AR decreases by
approximately 29.6% for the same geodesic distance shift [0. 7) - T 3x )), compared to a 66.6% drop on ClearPose [I33], which contains glass objects. This is because glass objects in Clear- Pose [T3]]] are more difficult for the model to capture common appearance features, indicating that the intrinsic material properties of objects also impact this performance trend.
2: 4
D.6 More Visualizations
In this section, we visualize the prediction from two notable metric depth estimation models, SPIdepth [5:3] and Unidepth [[9]]. Figure @ illustrates the comparison between ours and theirs. UniDepth [39], though it estimates acceptable scales, lacks enough representation of objects' de- tails. SPIDepth [53], while identifying the object well, outputs an unsatisfactory overall scale.
In addition, we present visualizations of depth predictions using different weight distributions across F1 F,. The term Sealed represents the ratio of the mean value of the feature processed by the token scaler to that of the original feature. The rows corresponding to 0.0 represent cases where we manually initialized the output convolution layer weights to zero, resulting in the network effectively ignoring these layers during initial training iterations. Figure III. Figure I. and Figure 122 shows the visualization results of REAL275 [[]], ClearPose [I3]] and Tyo-L. [62]] dataset, respectively. The special cases are illustrated the upper right of each figure. "0.0 for all" means the token scaler a zero constant, which deactivates the fusion process, and the prediction is obtained purely from the vanilla Fi. 1-1.5 for all" or 1-1.0 for all" are special cases in which we manually clamp the scaler output to negative values and deploy such strategy to all layers. obvious from the figure that "0.0 for all" results in low-quality output that lacks clear representation and distinguishment of objects and background. 1-5 for all" or 1-1.0 for all" leads to meaningless output since a too large weight may break the distribution within the feature map, thus impairing the inference process in the depth head.
F
35

---

RGB
GT Depth
0.0 For All
-1.5 For All
2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
Scaled F.
Scaled F,
Scaled F
Scaled F
F
F
Figure 10: Comparison different fusion weights on Real275 dataset. Each column is independent and varies the corresponding weight from 1.0 to -2.5.
36

---

RGB
GT Depth
0.0 For All
-1.0 For All
2.5
2.0
1.5
0.5
0.0
-0.5
-1.0
Scaled F,
Scaled F,
Scaled F
Scaled
Figure 11: Comparison of different fusion weights on ClearPose dataset. Each column indepen- dent and varies the corresponding weight from 1.5 to -2.0.
37

---

RGB
GT Depth
0.0 For For
-1.5 For All
2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
Scaled F. F
Scaled F,
Scaled F,
Sealed F
F
F
F
Figure 12: Comparison of different fusion weights on the Tyo-L. dataset. Each column indepen- dent and varies the corresponding weight from 1.0 to -2.5.
38


Reviews:
[{"review_id": "IThDimi8o5", "reviewer": "Reviewer_ctwM", "summary": "The paper  proposes SingRef6D, a lightweight pipeline using a single RGB image as a reference, in contrast to some other 6D  object pose models relying on CAD models, depth sensors or novel view synthesis. Given a  query, and a reference image , both containing the same object, the objective is to estimate the 6D relative pose between them, thus all accessible reference information comes from a single RGB image.\nThe method targets transparent and multi object scenarios, with the former posing problems to depth sensors. \n\nThe main claimed innovation is a token-scaler-based fine-tuning mechanism with a novel optimization loss applied to Depth-Anything v2 and improving its depth estimation in challenging conditions. The results show improvements in depth prediction. Another claimed contribution is depth aware matching that integrates spatial relationships within LoFTR. The evaluation on REAL275, ClearPose, and Toyota-Light shows significant improvement for transparent objects.", "questions": "See a-d weaknesses above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The pipeline is relatively simple using DPA to predict depth, SAM to crop objects of interest and PointDSC with LoFTR to match two RGBD views.  There are a number of methods for 6D object pose estimation that follow similar chain of modules and require a single image, therefore the main contribution of this work is in scaler token for fine-tuning to mitigate the scaling inconsistencies of the mono depth estimator.\nIt attempts to integrate a ControlNet like structure  to dynamically scale and modulate DINO features at coarse and global levels with InceptConv-based network during DPA finetuning.\n\nMany loss terms are combined together including standard SSI loss in DPA with BerHu loss. Proposed scale alignment  loss that is robust to noise, which is pixel level depth loss within object, edge emphasizing loss and normal consistency loss for surface deformation. \n\nThe proposed scaler and the choice of loss functions are interesting and it should be possible to make use of these ideas in other methods.\n\nAblation study shows individual contributions of the main loss components validating their usefulness. Similarly the proposed scaler makes a positive difference in finetuning of DPA.\n\n\n\na) Although transparent and untextured objects are mentioned as the main targets these are not explicitly addressed by the proposed approach. The robustness to the lack of texture and reflections comes from the depth maps of the objects that complement the RGB images. Depth estimation is claimed to be improved by finetuning of DPA with the combination of various loss functions.\n\nb) Evaluation results demonstrate significant improvements over DPA on REAL275, ClearPose, and Toyota-Light. It is unclear if this is due to fine tuning on the specific domain and whether the general performance of DPA was not compromised as the evaluation is limited to object pose benchmarks only.  Generalisation across domains is not validated well.\n\n\nc) The choice of methods to compare against and the benchmarks could be more comprehensive.\nThe paper claims multi object pose but the benchmarks are focused on relatively uncluttered scenarios.\nREAL275 has some minor occlusions but other benchmarks could have been used to test robustness to textureless objects partial occlusion and large variations of viewpoints like in HOD3, LM-O, YCB-Video, LineMod.\n\nd) In terms of comparison to SOTA methods, SIFT, RoMA and Oryon are relatively weak baselines.  Including results for stronger methods would have made the evaluation more convincing. Any6D, Fs6d, FoundationPose could have been reported too for completeness of evaluation as well as original LoFTR which is used in this method too and makes part of a contribution. \nOne2Any is also relevant although from cvpr25.", "was_revised": true, "final_justification": "The rebuttal addressed my questions and provided additional results which look convincing.  Hopefully the code will be released and  used by others.", "rebuttal": "Thank you for the constructive feedback. We're glad you recognized the core contribution of the scaler token in addressing scale inconsistency, the effectiveness of ControlNet-style modulation in DPA fine-tuning, and the broader applicability of our loss functions. The positive ablation results further validate each component. Here, we summarize and respond to your questions and hope our answers address your concerns.\n\n> **W1:** Although transparent and untextured objects are mentioned as the main targets, these are not explicitly addressed by the proposed approach. The robustness to the lack of texture and reflections comes from the depth maps of the objects that complement the RGB images. Depth estimation is claimed to be improved by fine-tuning DPA with the combination of various loss functions.\n\n**A1:** We appreciate your insightful comment. Our approach avoids specialized modules or classifiers for transparency and textureless regions, achieving robustness through core design principles. We introduce a token scaler that adjusts hierarchical features in the depth model, enabling dynamic adaptation to varied visual conditions. For instance, it enhances feature contrast at object boundaries, improving depth estimation for transparent objects like glass bottles by better capturing spatial relationships. Our proposed loss functions promote geometric consistency by focusing on object surfaces, edges, and normals, enhancing spatial coherence in low-texture and low-reflectance scenarios. This is effective in real-world settings, such as unevenly lit reflective surfaces, where our consistency loss ensures robust surface representation. Table 2 shows our method achieves a $\\delta_{1.05}$ of 54.30 on ClearPose, outperforming fine-tuned DPAv2 (31.23), demonstrating strong generalization in challenging environments. Future work will further explore transparent and textureless objects, with expanded experimental discussion.\n\n\n> **W2:**  Evaluation results demonstrate significant improvements over DPA on REAL275, ClearPose, and Toyota-Light. Generalisation across domains is not well validated.\n\n**A2:** We appreciate the reviewer’s insightful comment. Our method prioritizes practical scenarios by enabling quick adaptation through efficient fine-tuning on relevant scenes. We agree with you that cross-domain generalization capability is important; thus, we conducted additional experiments involving domain transfer evaluation. Specifically, we fine-tuned the depth estimation model on REAL275, which primarily contains industrial and everyday objects under static lighting conditions, and directly evaluated its performance on ClearPose (transparent objects in complex spatial environments) and Tyo-L (dynamic lighting variations)—without any domain adaptation. The three datasets vary in scale, background, camera setting, and objects, enabling a comprehensive assessment of the cross-domain generalization. The results are shown in the table below.\n\n|Setting|Method|$\\delta_{1.05}$↑|$\\delta_{1.10}$↑|$\\delta_{1.25}$↑|RMSE↓|log10↓|Abs.Rel↓|Sq.Rel↓|MAE↓|\n|-|-|-|-|-|-|-|-|-|-|\n||Unidepth|8.35|22.61|42.01|0.983|1.419|1.855|0.702|0.500|\n|REAL275→Tyo-L|DPAv2|10.55|28.93|50.23|0.519|0.764|0.399|0.177|0.326|\n||**Ours**|**28.13**|**34.91**|**45.72**|**0.377**|**0.471**|**0.223**|**0.119**|**0.218**|\n||Unidepth|8.74|14.02|25.91|0.364|0.268|0.294|0.312|0.273|\n|REAL275→ClearPose|DPAv2|20.85|44.48|62.36|0.258|0.232|0.207|0.280|0.181|\n||**Ours**|**42.19**|**66.25**|**79.70**|**0.182**|**0.106**|**0.131**|**0.228**|**0.109**|\n\nThe experimental results show that cross-domain performance drops as expected. However, our method consistently outperforms fine-tuned baselines (e.g., DPA v2 and UniDepth) under the same settings. This superior generalization arises from our Token Scaler, which adaptively modulates intermediate features in a data-agnostic way, enabling better generalization across diverse visual inputs.\n\nMoreover, the introduced loss functions are designed to regularize the model for consistent geometric understanding. By emphasizing edges and surface normals, they focus on capturing generalizable geometric structures rather than overfitting to the distribution of a particular dataset. Therefore, this suggests that the improvements brought by our approach are not solely due to domain-specific tuning but also stem from its advanced intrinsic mechanism. Besides, the proposed modules are inherently transferable and can be seamlessly integrated into other visual tasks (such as depth-based navigation, etc.), demonstrating strong potential for broader applicability beyond the current setting for other directions. We will include these cross-domain results in the revised version to support our claims.\n\n> **W3:** The choice of methods to compare against and the benchmarks could be more comprehensive. The paper claims multi-object pose, but the benchmarks are focused on relatively uncluttered scenarios. REAL275 has some minor occlusions, but other benchmarks could have been used to test robustness to textureless objects, partial occlusion, like in LM-O, YCB-Video.\n\n**A3:** We express our gratitude to your insightful advice. This is certainly a valuable comparison. We conducted two additional experiments on LM-O and YCB-V, which are representative benchmarks featuring occlusions and weakly textured objects, such as monochromatic, single-material objects. Testing on these benchmarks is able to show the ability of our model to handle textureless objects and partial occlusion. We evaluate pose estimation on three different target objects, generating 120 image pairs for each. The table below shows the quantitative results. \n|Dataset|Matcher|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|-|-|\n||Oryon|Oracle|6.1|3.1|6.9|8.3|5.5|\n|||Ours|3.8|1.9|4.2|5.4|2.7|\n||Ours|Oracle|9.9|4.3|11.5|13.9|8.7|\n|LM-O||Ours|**5.3**|**2.6**|**5.7**|**7.3**|**3.8**|\n||vs.Oryon|Δ|+2.7|+0.95|+3.1|+3.8|+2.2|\n||Oryon|Oracle|8.6|4.7|9.3|11.8|7.5|\n|||Ours|5.8|2.4|6.6|8.4|5.2|\n||Ours|Oracle|11.4|5.2|12.3|15.7|8.9|\n|YCB-V||Ours|**7.9**|**3.7**|**8.5**|**11.6**|**6.4**|\n||vs.Oryon|Δ|+2.45|+0.90|+2.45|+3.55|+1.30|\n\nDespite heavy occlusion causing performance drops, our method consistently outperforms the baseline in the single RGB reference setting. This improvement arises from our human vision-inspired framework, which adaptively adjusts features using both appearance and spatial cues. Unlike traditional methods like Oryon or SIFT that rely mainly on RGB similarity, our approach leverages spatial context to enhance alignment accuracy and maintains consistency across matching stages (see Figure 1). These designs boost robustness, especially when appearance cues are unreliable.\n\nFor multi-object situations, our pipeline naturally extends to these scenarios by simply incorporating an off-the-shelf object detector or segmenter, such as SAM or SAM2. This design highlights the generality and scalability of our approach, making it readily adaptable to diverse real-world applications without major architectural changes. The experiment results prove the robustness of our method under occlusion scenarios. We are willing to add more discussion on this in the revised version.\n\n> **W4:** Including results for stronger methods would have made the evaluation more convincing. Any6D, Fs6d, FoundationPose could have been reported too for completeness of evaluation, as well as the original LoFTR, which is used in this method too, and is part of a contribution. One2Any is also relevant, although from CVPR25.\n\n**A4:** Thanks for the valuable feedback. We agree with you that Any-6D, FS6D, and vanilla LoFTR are comparable baselines. As for FoundationPose, it requires multiple reference images to train a neural field, which conflicts with our assumption of using only one reference RGB image. This difference shows our method’s applicability in real-world scenarios with limited input, in contrast. Regarding One2Any, we were unable to reproduce results due to the lack of released code. However, we analyzed the method thoroughly—it estimates pose by encoding object appearance and predicting ROC maps using a trainable U-Net and decoder. While effective, it relies on accurate depth and fully labeled pose datasets, resulting in heavier training costs. In contrast, our method requires only lightweight depth fine-tuning and is more flexible in data-limited settings. We consider One2Any to target a different scope, but we’re open to including it in future comparisons once the code becomes available.\n|Matcher|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|-|\n||Oracle|43.3|15.8|55.8|58.4|32.2|\n|Any-6D|DPAv2|5.2|1.3|4.3|11.0|2.2|\n||Ours|31.6|13.5|33.5|47.8|13.2|\n||Oracle|14.1|5.2|17.9|19.2|10.0|\n|FS6D|DPAv2|1.6|0.5|1.9|2.4|0.3|\n||Ours|9.3|1.8|6.3|7.8|3.2|\n||Oracle|42.0|34.2|44.1|47.6|35.0|\n|Ours|DPAv2|5.8|2.8|3.5|11.1|2.9|\n||Ours|31.7|13.9|33.8|47.3|13.1|\n\nThe table shows the result on the Tyo-L dataset. Our method outperforms FS6D across various metrics. While Any-6D achieves slightly better results with ground truth depth due to its strong 2D-3D model, it heavily relies on depth quality, as its performance drops with predicted depth. This is because its global optimization (e.g., point cloud registration) improves MSSD/MSPD but suffers from accumulated local depth errors, lowering VSD recall. In contrast, our method focuses on pixel-wise alignment, achieving higher VSD recall.\n\nOur method shows greater robustness when switching from oracle to predicted depth. It delivers more stable depth reconstruction and improved geometric awareness for pose estimation. With a modular design, it integrates well into pose estimation or depth prediction pipelines. Besides, vanilla LoFTR for RGB-based matching yields an AR of 27.2, while our method achieves 31.7 (in Table 9). This shows more robust and accurate feature matching of our method under identical conditions. In the revision, we will add these results to the experiment section."}, {"review_id": "uScsW4W4b9", "reviewer": "Reviewer_Xucn", "summary": "This paper presents a complete pipeline for novel object pose estimation from a single RGB image. The core methodology consists of three main stages: （1）Scaled Depth Estimation: The authors first propose a fine-tuning strategy based on a token-scaler architecture. This method adapts a monocular depth estimation model to recover a scene depth map with a correct metric scale from a single RGB input. （2）Depth-guided Feature Matching: Subsequently, the paper employs a training-free feature matching technique (LoFTR) to establish correspondences for the object instance between a reference and a query image. A key point here is the use of the previously estimated depth map to guide this matching process, which aims to improve the accuracy of the resulting keypoint matches. （3）Pose Recovery: Finally, using these depth-guided matches, the relative pose between the reference and query views is solved for, which in turn yields the final pose of the object in the query image. The proposed method was evaluated on three benchmark datasets. Through comparisons with various combinations of different depth estimation models and feature matching techniques, the experimental results demonstrate that the proposed integrated approach achieves promising results for the task of novel object pose estimation.", "questions": "1. How about the performance of the proposed metric depth estimation model on standard depth benchmarks？Within the presented object pose estimation experiments, what is the degree of similarity or difference between the scenes used for fine-tuning the depth model and the scenes used for testing?\n\n2. Could you please provide a quantitative comparison of the proposed method against recent works[A,B] on a standard instance-level benchmark (e.g., LINEMOD)?\n\n3. Could you please provide an analysis of your method's robustness to the choice of the reference view? Specifically, in your current quantitative results, what is the typical pose difference (e.g., in terms of rotation) between the reference and query images? How does performance degrade as this difference increases?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros：\n\n+ The paper outlines a complete framework for novel object pose estimation using a single RGB image. The methodology is presented as a sequence of three main modules: monocular metric depth estimation, depth-guided feature matching, and final pose recovery. This modular structure clearly defines the paper's technical approach.\n\n+ The authors introduce a fine-tuning strategy for a monocular depth model aimed at recovering the depth metric. They conduct experiments comparing this method against other approaches, and the paper reports that their method leads to improved depth accuracy and reduced geometric distortion.\n\n+ The proposed pipeline is evaluated on three public datasets. The paper presents quantitative results for the pose estimation task, including experiments on the ClearPose dataset, which contains challenging transparent objects.\n\nCons:\n\n- The core technical contribution appears to be the fine-tuning strategy for metric depth recovery. The proposed depth estimation method is a solution to a well-established problem (metric depth from a single image). As such, it should be also evaluated on standard depth estimation benchmarks. The current evaluation, which is confined to the three object pose datasets, does not provide a clear picture of how the method performs in a general context or how it compares to state-of-the-art depth estimation techniques. Besides, the paper seems to imply that the depth model is fine-tuned separately for each of the three datasets. This raises significant concerns about the model's generalization capabilities. It is unclear how different the fine-tuning scenes are from the testing scenes. A crucial unanswered question is: how well would a model fine-tuned on a general indoor dataset (e.g., ScanNet) perform when applied to a completely different scene with a similar scale? Without this analysis, the practicality of the proposed depth recovery method remains questionable.\n\n- From a pose estimation perspective, the paper's contribution is limited. The overall pipeline that uses feature matching followed by a PnP/relative pose solver is a standard and well-established practice. The main novelty claimed for the pose estimation part is the integration of depth to guide the LoFTR feature matcher. While this addition appears effective, the idea of using geometric priors (like depth) to refine feature matching is not new. The specific implementation is straightforward and can be seen as an incremental improvement rather than a novel conceptual contribution. Given that the method explicitly recovers depth for both the reference and query images, how does the proposed 2D matching with depth guidance compare to methods that perform keypoint matching directly on the generated point clouds, or use techniques like ICP for pose refinement? This comparison is currently missing.\n\n- The paper's experimental setup is more aligned with instance-level relative pose estimation. Therefore, the evaluation should also be extended to standard instance-level benchmarks like LINEMOD and LINEMOD-Occlusion. In addition,  the comparisons should also include recent and highly relevant instance-level methods [A,B].\n\n[A] \"3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation\", ICLR 2024.\n\n[B] \"DVMNet: Computing Relative Pose for Unseen Objects Beyond Hypotheses\", CVPR 2024.\n\n- As a reference-based method, the choice of the reference image is critical to performance, yet this aspect is not analyzed. The paper does not provide any analysis on how the performance varies with the choice of the reference view. What is the typical viewpoint difference (e.g., in terms of rotation and translation) between the reference and query images in the reported experiments? The robustness of the feature-matching-based approach is highly dependent on the visual overlap between the two views. It is expected that the performance will degrade significantly as the viewpoint difference increases. A thorough ablation study is needed to quantify this effect and understand the method's limitations, which is a standard practice for evaluating reference-based methods.", "was_revised": true, "final_justification": "The rebuttal and discussion addressed most of my concerns. Therefore, I will raise my score to Borderline Accept. At the same time, I maintain my position that the paper is only suitable for acceptance as a contribution to object pose estimation if, and only if, the authors incorporate the following crucial experiments from their rebuttal into the main manuscript. Specifically, the paper must include: （1）A cross-domain generalization experiment for pose estimation.\n（2）A direct comparison with recent relevant methods on instance-level pose estimation benchmarks.\n（3）A comparison with point cloud-based registration methods.\n（4）A study on the robustness of pose accuracy to the selection of the reference view. My recommendation is contingent upon these revisions, assuming such changes are permissible within the NeurIPS camera-ready policy.", "rebuttal": "Thank you for the thoughtful feedback. We’re glad our modular framework for novel object pose estimation is clear and technically sound. We appreciate your recognition of our improvements in depth accuracy, geometric consistency, and thorough evaluation across diverse datasets, including challenging transparent objects in ClearPose. We summarize and address each of the questions and hope they can address your concerns.\n\n> **W1:** How well would a model fine-tuned on a general indoor dataset perform when applied to a completely different scene with a similar scale? \n\n**A1:** Thank you for the comment. Although we aim to enable quick adaptation through efficient fine-tuning on relevant scenes, we agree that cross-domain generalization is important. We fine-tuned the depth model on REAL275 and evaluated it on ClearPose and Tyo-L without further adaptation. These datasets differ in scale, background, camera settings, and objects, offering a rigorous test of generalization (can be treated as distinct benchmarks). \n|Setting|Method|$\\delta_{1.05}$↑|$\\delta_{1.10}$↑|$\\delta_{1.25}$↑|RMSE↓|log10↓|Abs.Rel↓|Sq.Rel↓|MAE↓|\n|-|-|-|-|-|-|-|-|-|-|\n||Unidepth|8.35|22.61|42.01|0.983|1.419|1.855|0.702|0.500|\n|REAL275→Tyo-L|DPAv2|10.55|28.93|50.23|0.519|0.764|0.399|0.177|0.326|\n||Ours|**28.13**|**34.91**|**45.72**|**0.377**|**0.471**|**0.223**|**0.119**|**0.218**|\n||Unidepth|8.74|14.02|25.91|0.364|0.268|0.294|0.312|0.273|\n|REAL275→ClearPose|DPAv2|20.85|44.48|62.36|0.258|0.232|0.207|0.280|0.181|\n||Ours|**42.19**|**66.25**|**79.70**|**0.182**|**0.106**|**0.131**|**0.228**|**0.109**|\n\nThe experimental results show the expected drop in cross-domain performance. However, our method consistently outperforms baselines under identical conditions. This benefits from our Token Scaler’s data-agnostic feature modulation, which improves generalization ability across diverse inputs.\n\nOur loss functions enhance geometric consistency by leveraging generalizable cues like edges and surface normals, avoiding overfitting to specific datasets. These improvements stem from the model’s intrinsic design rather than domain-specific tuning. Besides, the proposed modules are easily transferable to other visual tasks (e.g., depth-based navigation), demonstrating broad applicability. We will include these cross-domain results in the revised version to support our claim.\n\n\n> **W2:**  The specific implementation is straightforward and can be seen as an incremental improvement. How does the proposed matching with depth guidance compare to methods that perform keypoint matching directly on the generated point clouds, like ICP?\n\n**A2:** Thank you for the helpful advice. We admit ICP is a baseline worthy of comparison. Our setting focuses on pose estimation from a single reference view, which often yields incomplete point clouds, so we initially considered ICP out of scope. Nevertheless, we conducted ICP experiments using point clouds from both reference and query views. The results are shown in the table below.\n|Matcher|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|-|\n||Manual|19.8|5.5|23.2|32.9|20.6|\n|ICP|Raw|0.3|0.04|0.25|0.61|0.9|\n||Ours|8.1|2.6|9.4|12.4|4.5|\n||Manual|32.4|10.1|38.8|48.2|33.4|\n|Ours|Raw|1.4|0.2|1.2|2.8|2.0|\n||Ours|**19.4**|**5.1**|**19.7**|**33.5**|**10.0**|\n\nICP performs unsatisfactorily due to it is sensitive to initialization and noise, making correspondence unreliable. In contrast, our depth-aware matching operates in latent space, preserving pixel-level consistency and mitigating the pitfalls of noisy 3D reconstructions. Our method is enriched with spatial cues in a coarse-to-fine manner, further boosted following pose solving, performing better under transparent and low-texture conditions where ICP struggles.\n\nWhile we do not introduce a new matching conception—still following feature matching followed by pose estimation—our key contribution is embedding predicted dense depth into the latent space of the matching network. In the challenging setting of single RGB reference input, no CAD models, and pose-supervised training, this modular depth fusion consistently improves robustness to occlusion, weak textures, and transparency, all without retraining. We will include related discussions and results in the revised version.\n\n\n> **Q1:** How about the performance of the proposed metric depth estimation model on standard depth benchmarks? What is the degree of similarity or difference between the scenes used for fine-tuning the depth model and the scenes used for testing?\n\n**A3:** Thanks for your valuable comment. Our depth perception relies on the strong foundation model DPAv2. The proposed token scaler preserves its knowledge while dynamically modulating layer weights to adapt to different scenes. To avoid information leakage, we fine-tuned the token scaler (with DPAv2 pretrained on KITTI) on a downsampled NYU-D training set. On its test set, our method improves $\\delta_{1.05}$ by **0.14** and reduces Abs.Rel by **0.09** compared to DPAv2 with a fine-tuned head. This shows our approach is effective for general depth prediction. We will include more results in the revised version.\n\n\nIn our experiments, we manually split the data to ensure that test scenes remained entirely unseen during fine-tuning. To clarify this, we will add a data description section with visualizations in the revised version. To quantify the similarity between fine-tuning and test data, we compute the Structural Similarity Index Measure and report the mean and std for each dataset.\n|Dataset|SSIM|\n|-|-|\n|Tyo-L|0.247(±0.137)|\n|REAL275|0.167(±0.029)|\n|ClearPose|0.181(±0.041)|\n\nThe SSIM scores are low, suggesting limited overlap and sufficient separation between scenes. In future work, we plan to explore more semantic-aware metrics, such as CLIP similarity or learned perceptual measures, to better assess cross-scene similarity.\n\n> **W3/Q2:** Could you provide a quantitative comparison of the proposed method against recent works[A, B] on an instance-level benchmark?\n\n**A4:**  Thanks for your insightful suggestion. The referenced works are indeed great works for instance-level relative pose estimation. They encode images into a trainable latent voxel space and achieve impressive results without relying on explicit depth inputs by learning from large-scale pose-annotated datasets. We will include a detailed discussion of these methods in the revised Related Work section. We compare with DVMNet and 3DVHA by performing iterative inference on image pairs. Results are summarized in the table below.\n\n|Method|AR|ADD|Angular Error|\n|-|-|-|-|\n|3DVHA|7.7|6.3|52.71|\n|DVMNet|8.6|7.4|51.66|\n|Ours|9.9|8.7|51.89|\n\nWhile our method shows slightly higher angular error than DVMNet, it achieves better AR and ADD, indicating more accurate translation estimation. This is because DVMNet and 3DVHA are supervised with full ground-truth poses for a latent voxel space, which might encounter translation shifts in an unseen environment.\n\nThis shows a key difference: DVMNet is suited for stable scenes needing precise $SO(3)$ estimation, whereas our approach offers a more generalizable and lightweight solution for full-pose estimation, making it more practical for real-world applications. We will include these discussions and explore further experiments inspired by these notable works in the revised version.\n\n> **Q3:** Could you provide an analysis of your method's robustness to the choice of the reference view? In your current quantitative results, what is the typical pose difference (rotation) between the reference and query? How does performance degrade as this difference increases?\n\n**A5:** We appreciate your feedback. The table below reports the angle differences between reference-query pairs across datasets, showing reasonable variation rather than fixed or easily inferred values. This highlights the practical relevance of our setup and demonstrates our ability under various angular conditions.\n\n|Dataset|Range|Mean|Std.|\n|-|-|-|-|\n|REAL275|1.5-179.6|51.7|31.9|\n|ClearPose| 1.4-169.2|49.3|29.8|\n|Tyo-L|2.9-180.0|98.8|57.0|\n\nTo assess the impact of viewpoint gaps between reference and query images, we conduct additional experiments in BlenderProc by constructing scenes using 3D models from BOP and applying rotations along the X, Y, and Z axes by angles $\\alpha$. We report results for rotations along the X and Z axes. Y-axis rotations are omitted due to their strong correlation with X-axis rotations (Spearman>0.86), likely due to the symmetry of the setup. Future revisions will include finer angular splits and explicit Y-axis rotation results on more scenes. \n|Axis\\α|15°||30°||45°||60°||75°||90°||120°||150°||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|\n|X|69.84|65.70|61.17|60.42|50.52|50.38|42.36|42.91|33.71|33.64|30.12|30.47|26.83|22.36|20.54|19.90|\n|Z|74.26|72.19|71.50|69.83|68.13|67.21|65.80|64.02|62.42|60.57|58.31|56.48|39.76|36.73|31.95|28.54|\n\nThe table shows that pose estimation performance declines with increasing viewpoint differences, especially along X-axis rotations, which introduce significant perspective and occlusion changes. Z-axis rotations mainly cause in-plane transformations with milder distortion, leading to a slower performance drop, though accuracy still degrades when rotations exceed 90°, likely due to reduced visual discriminability (e.g., textured front with logos vs. plain back of a cup).\n\nDespite these challenges, our method shows a 65% drop under extreme conditions, while baselines like Oryon and SIFT degrade by over 90%. This robustness comes from our model’s ability to exploit spatial consistency and adapt feature representations via the token scaler. Additionally, our loss functions enforce strong geometric constraints, helping maintain accuracy under large viewpoint changes. These findings underscore our method’s robustness and its potential for real-world applications, offering a promising direction for future research."}, {"review_id": "Ov3qQB8JQo", "reviewer": "Reviewer_dpvV", "summary": "This paper presents SingRef6D, a method for novel object 6D pose estimation that only requires a single RGB reference image. The authors propose a lightweight pipeline that combines a fine-tuned Depth-Anything v2 model with a novel token-scaler mechanism for depth prediction and integrates depth-aware matching into LoFTR to improve correspondence estimation. The proposed method does not require CAD models, novel-view synthesis, or video-based supervision, making it efficient and practical. Experimental results on REAL275, ClearPose, and Toyota-Light demonstrate strong performance and improvements over existing works.", "questions": "- How does the proposed method perform when there is a large viewpoint gap between the reference and query images—specifically, in cases where the two views have minimal overlap? \n- How robust is the proposed approach to heavy occlusion, such as in Occlusion-Linemod dataset?", "limitations": "I do not see any major limitation in this work. While the authors mention the use of object masks for target localization, I do not consider this to be a significant limitation. In practice, object masks can be easily obtained using existing tools such as SAM or SAM2. Moreover, since there are typically multiple objects present in a scene, it is entirely reasonable to specify which object the model should focus on in the reference image.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The problem addressed in this paper is well motivated. Estimating 6D pose from a single RGB image is very challenging but highly practical for real-world applications.\n- I like the idea of using depth models for this pose estimation task, as recovering object scale from only a single RGB reference is highly ambiguous. The proposed token-scaler mechanism further improves the robustness and accuracy of metric depth prediction on challenging surfaces, such as transparent or reflective materials. The reported gains in depth metrics across multiple datasets are significant.\n* The model is computationally efficient compared to recent baselines like Oryon and RoMA, requiring fewer parameters and less memory while achieving better or comparable accuracy.\n* Extensive evaluations and ablation studies are provided, which clearly demonstrate the effectiveness of each proposed component, including the depth losses and feature fusion strategies.\n\nWeaknesses:\n* One major concern in estimating 6D pose from a single RGB reference is when there is a large viewpoint gap between the reference and the query image—i.e., when the two views share only a small overlap. Although the authors show promising qualitative examples (e.g., the first row of Figure 5), I would expect to see a more thorough quantitative evaluation that directly studies this challenge. If the proposed method is robust in small-overlap conditions, that would be a very strong result to highlight in the paper.\n* While the paper includes difficult examples such as transparent objects, most of the presented samples involve objects that appear relatively clear, with minimal occlusion. It is unclear how well the proposed method handles more severe occlusions, such as those found in the Occlusion-LM dataset. The authors could consider using the single-image setup introduced in GigaPose [39] for a more comprehensive comparison.", "was_revised": true, "final_justification": "I have read through all the reviews and the rebuttal. Overall, I believe the authors have responded well to the points raised, particularly with the addition of new supporting experiments. Therefore, I keep my score and recommend accepting this paper.", "rebuttal": "We thank the reviewer for the encouraging feedback. We're encouraged that you found our problem motivation clear and recognized the practical value of our SingRef6D. We appreciate your recognition of our use of depth models to resolve scale ambiguity and the token-scaler mechanism for improving depth prediction on challenging surfaces. Your acknowledgement of our significant performance gains, computational efficiency over strong baselines like Oryon and RoMA, and the thoroughness of our evaluations is truly appreciated. We summarize and respond to each question below, and hope our answers address your concerns.\n\n> **W1/Q1:** How does the proposed method perform when there is a large viewpoint gap between the reference and query images, specifically, in cases where the two views have minimal overlap?\n\n**A1:** We appreciate your insightful suggestion. We agree with you that quantifying such a relationship is beneficial for our work. To this end, we conduct an experiment that manually rotates the scene with a specific angle $\\alpha \\in [15^\\circ,150^\\circ]$  along the $X$,$Y$, and $Z$ axes, respectively. Specifically, we use accessible 3D models (e.g., from BOP datasets) to construct a scene with BlenderProc and rotate the camera by $\\alpha$ (The reference scene and the query differ in background and contain different objects, except for the target object). We define a right-handed coordinate system, where the $Z$-axis points vertically upward, perpendicular to the object surface, and the $X$ and $Y$ axes lie in the horizontal plane. The camera is positioned at (1, 1, 1), measured in meters, looking down at the scene, with all objects placed on the $X$–$Y$ plane. Since most existing benchmarks adopt a top-down perspective, we follow the same setup. To evaluate the effect of viewpoint variation, we report results under different camera rotations along the $X$ and $Z$ axes. Rotation along the $Y$ axis is omitted, as we observed a strong correlation between rotations around the $X$ and $Y$ axes during the experiments, with a Spearman coefficient exceeding 0.86. This correlation is likely due to the symmetry of the setup. Therefore, we report only the $X$-axis rotation in the table below to represent horizontal-axis variations. In future revisions, we plan to include finer-grained angular distinctions and explicit values for $Y$-axis rotations on more scenes. \n\n|Axis\\α|15°||30°||45°||60°||75°||90°||120°||150°||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|AR|ADD|\n|X|69.84|65.70|61.17|60.42|50.52|50.38|42.36|42.91|33.71|33.64|30.12|30.47|26.83|22.36|20.54|19.90|\n|Z|74.26|72.19|71.50|69.83|68.13|67.21|65.80|64.02|62.42|60.57|58.31|56.48|39.76|36.73|31.95|28.54|\n\nFrom the table, we observe that pose estimation performance degrades with increasing viewpoint differences between the reference and query images, especially when the rotation is along the X-axis. This is because X-axis rotations significantly alter perspective and occlusion patterns, making it more challenging for the model to match spatial features. In contrast, Z-axis rotations mainly induce in-plane transformations with relatively mild geometric distortion, leading to a more gradual performance drop. However, when the rotation exceeds 90°, even Z-axis changes cause noticeable degradation. This is likely due to decreased visual discriminability — for example, the front of a mug may have a distinct texture or logo, while the back is plain, reducing the object's appearance discriminability and making it harder for the model to rely on appearance and forcing it to depend solely on geometry and spatial consistency.\n\nDespite this challenge, our method exhibits a smaller performance drop (65%) under such extreme conditions, while baselines like Oryon and SIFT fail almost entirely, with performance degradation exceeding 90\\%. This robustness stems from our model’s ability to effectively exploit spatial consistency and adapt spatial representations through the proposed token scaler, thus boosting geometric understanding in the coarse-to-fine matching process. This mechanism helps to improve matching accuracy when RGB cues are unreliable. Furthermore, our loss functions impose strong geometric constraints such as normal consistency and edge emphasis, enabling the model to maintain a reasonable estimation accuracy even under large viewpoint changes. These results and analyses demonstrate the robustness of our model and highlight its potential for real-world applications. We also believe this direction presents a promising avenue for future research.\n\n> **W2/Q2:**  It is unclear how well the proposed method handles more severe occlusions, such as those found in the Occlusion-LM dataset. The authors could consider using the single-image setup introduced in GigaPose for a more comprehensive comparison. How robust is the proposed approach to heavy occlusion, such as in the Occlusion-Linemod dataset?\n\n\n**A2:** We appreciate your insightful advice. The authors agree with you that considering occlusion is important. We conducted two additional experiments on LM-O and YCB-V, which are representative benchmarks featuring occlusions and weakly textured objects, such as monochromatic, single-material objects. We evaluate pose estimation on three different target objects, generating 120 image pairs for each. The table below illustrates the experiment results.\n|Dataset|Matcher|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|-|-|\n||Oryon|Oracle|6.1|3.1|6.9|8.3|5.5|\n|||Ours|3.8|1.9|4.2|5.4|2.7|\n||Ours|Oracle|9.9|4.3|11.5|13.9|8.7|\n|LM-O||Ours|**5.3**|**2.6**|**5.7**|**7.3**|**3.8**|\n||vs.Oryon|Δ|+2.7|+0.95|+3.1|+3.8|+2.2|\n||Oryon|Oracle|8.6|4.7|9.3|11.8|7.5|\n|||Ours|5.8|2.4|6.6|8.4|5.2|\n||Ours|Oracle|11.4|5.2|12.3|15.7|8.9|\n|YCB-V||Ours|**7.9**|**3.7**|**8.5**|**11.6**|**6.4**|\n||vs.Oryon|Δ|+2.45|+0.90|+2.45|+3.55|+1.30|\n\nFrom experimental outcomes, heavy occlusion still results in a notable performance decline. Nonetheless, our method has all-around superiority compared to the baseline. This is due to our framework introducing a human vision-inspired mechanism that adaptively adjusts features across different layers, guided by both appearance and spatial cues. Unlike conventional methods such as Oryon or SIFT that rely heavily on RGB similarity, our matching process integrates spatial information to enhance alignment fidelity. Moreover, it considers spatial consistency in both coarse-matching and fine-grained matching (in Figure 1). These designs lead to significantly improved robustness, especially under challenging conditions where appearance-based cues alone are insufficient (e.g., occlusion), which validates that our performance is not accidental but benefits from the designed mechanism.\n\nMethods like GigaPose and Any-6D rely on a single-image 3D reconstruction model, which makes them highly sensitive to object texture and the quality of the reconstruction model. In contrast, our approach is lightweight and based on cross-image matching, making it more robust to texture variations and easily integrable with other multi-view pose estimation pipelines. We are open to including additional comparisons with such methods in the revised version and plan to explore their integration in future work."}, {"review_id": "IQgFs72tTT", "reviewer": "Reviewer_QjZ6", "summary": "The paper proposes SingRef6D, a novel object pose estimation method based on monocular image and corresponding monocular depth estimation. The method is composed of two parts. Depth estimation part Pose estimation part.\n\nDepth estimation is based on Depth Anything V2, but includes an extra part that is meant for improving depth quality as well as adapting the scale of the depth to the given training scenes.\n\nThe pose estimation part is composed of two parts, LoFTR-based matching and PointDSC-based refinement. The proposed method modifies LoFTR to take depth, which improves the quality of matching further, especially on the photometrically challenging material object, like glasses.\n\nThe paper shows an experiment on depth estimation to prove the effectiveness of the fine-tuning method proposed in the paper. also pose estimation, mainly based on a matching pipeline, to prove the performance of the proposed matching method with depth.", "questions": "1. Misleading term with \"Metric Depth Prediction\"\n\n- It feels like fine-tuning is meant to overfit the depth prediction on each dataset's distance instead of actually managing to predict the correct metric depth for different conditions. For example, the scaler is fine-tuned on the ClearPose dataset again before running the validation, which means that depth prediction is only a metric when it's fine-tuned to the specific scenario. It's rather \"overfitting\" on the specific scenario than predicting the correct metric depth.\n- An interesting experiment will be using C.2's depth prior to make the pipeline compatible with different scenes without finetuning to show the robustness of the proposed network on different scenarios.\n\n2. Missing quantitative evaluation\n\n- I am curious why some of the experiments shown in Fig.4 are missing in Tab.2. For example, SPIDepth, ScaleDepth, Metric3D results are missing in Tab.2. Is it because they are significantly worse? It would be great to show their number regardless.\n\n3. The experiment on the reflective objects is missing, although it is mentioned in the abstract and intros.\n\n- There is one depth dataset [1] that is based on the same tabletop scenario with household objects, including reflective items, as well as high-quality depth ground truth, so that the author can fine-tune the proposed method. Also, a 6D pose dataset [2] that contains photometrically challenging objects, both transparent and reflective. It will be great to run an experiment on the reflective items by first fine-tuning the method on the depth dataset [1] and running pose estimation on the pose dataset [2].\n\n4. The experiment is weak on the 6D pose estimation part.\n\n- In lines 47-48, it is mentioned that the proposed method can perform competitively compared to generative or reconstruction-based methods. However, the experiment only covers matching-based methods. It would be great to include a comparison of such methods in the experiment section (e.g,  Any-6D, Zero123-6D, etc) by selecting two frames (i.e., one frame for generation, the other frame for getting pose). It's possible that the proposed method underperforms compared to such methods, but it's fair, as the proposed method is light. This experiment can give a better idea of the trade-off between complexity and performance.  Also, the experiment section lacks experiments on methods that are specifically targeted and trained on pose estimation. Including such experiments will strengthen the experiment section of the paper.\n\n\n[1] On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks, Jung et.al.\n\n[2] Housecat6D: a large-scale multi-modal category level 6D object perception dataset with household objects in realistic scenarios, Jung et. al.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength :\n1. The paper is easy to read, the figures are intuitive, and shows good extra details in the supplementary material.\n2. Both qualitative and quantitative experiments show the superior performance of the proposed method\n3. The proposed method specifically focuses on the photometrically challenging material, such as transparent and reflective.\n\nWeakness (Suggestions on each points are written in the Questions section): \n1. Misleading term with \"Metric Depth Prediction\"\n2. Missing quantitative evaluation on few depth methods shown in qualitative evaluation\n3. The experiment on the reflective objects is missing, although it is mentioned in the abstract and intros.\n4. The experiment is weak on the 6D pose estimation part.\n5. Contribution is slightly weak - The paper's main contribution is rather upgrading and fine-tuning depth prediction, adding depth in LoFTR matching. The rest of the pipeline is rather combination of existing works.", "was_revised": true, "final_justification": "The authors ran extra experiments requested by reviewers with reasonable results, and also address the questions fairly well.", "rebuttal": "Thank you for the positive feedback. We’re glad our clear presentation, intuitive figures, and detailed supplementary material were well received. We appreciate the recognition of our method’s strong performance, especially on challenging materials like transparent and reflective surfaces often overlooked in prior work. Below, we summarize and respond to each question, hoping to address your concerns.\n\n> **Q1:** The term \"Metric Depth Prediction\" seems misleading — the model is fine-tuned to each dataset's scale, effectively overfitting to specific scenarios rather than truly predicting metric depth across varied conditions. An interesting experiment will be using C.2's depth prior to make the pipeline compatible with different scenes without fine-tuning to show the robustness of the proposed network on different scenarios. \n\n**A1:** We appreciate your insightful comment. We fine-tuned our model on the ClearPose training set, but no calibration or optimization was performed on the test set scenes. This ensures that the backgrounds and environments in the test set remain novel and unseen during training. Since the model directly outputs metric depth, we refer to it as metric depth prediction. We appreciate your point; it might be more accurate to describe our approach as metric-aware adaptation, and we will consider using such terminology in revisions. \n\nTo investigate overfitting, as shown in Table 2 of the main paper, directly fine-tuning a vanilla DPAv2 on REAL275 significantly degrades performance ($\\delta_{1.05}$ drops to 29.87), whereas our method achieves a higher result of 44.28 (even with less data), demonstrating better robustness. This highlights the advantage of our token scaler in preserving generalizable representations by adaptively integrating hierarchical features rather than overfitting to dataset-specific distributions. Moreover, since our approach reuses the pre-trained model via a zero-initialized projection and layer-wise modulation, the original knowledge is well retained, mitigating significant alterations for the weights. Given the strong spatial understanding capabilities of transformer-based models like VGGT, we are also interested in extending our method to such architectures, as our token scaler is readily applicable to other hierarchical vision transformers.\n\nWe agree with you that it can be used for tuning-free inference. Therefore, we conducted experiments using the depth prior defined in Section C.2 to scale DPAv2's predicted depth without dataset-specific fine-tuning. \n|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|\n|Manual|32.4|10.1|38.8|48.2|33.4|\n|Raw|1.4|0.2|1.2|2.8|2.0|\n|DPAv2|9.1|1.6|11.1|14.4|3.9|\n|DPAv2(Raw as Prior)|11.8|2.4|10.9|22.1|6.4|\n|Ours|**19.4**|**5.1**|**19.7**|**33.5**|**10.0**|\n\n\nAs shown in the table, using raw depth priors improves performance over original DPAv2 predictions by enforcing global scale consistency. However, it still lags behind our full pipeline, as raw depth often fails to capture accurate object geometry and spatial layout, particularly in challenging cases like transparent objects in ClearPose. Even with depth alignment, DPAv2 struggles with surface reconstruction errors. That said, it is highly affected by the sensor and pretraining. \n\nIn contrast, our method enhances geometric understanding through losses such as normal consistency and edge emphasis. Our token scaler adapts features based on scene structure, improving robustness across diverse object categories. These advantages with a depth-aware matching yield stronger performance, critical for accurate 6D pose estimation. To the best of our knowledge, we are the first that propose using depth prior to scale the predicted depth for pose estimation within our settings. During the practical implementation, it is beneficial to first try depth prior. In the revised version, we will include additional experiments using various depth models and scaling strategies to validate the generalization of our method. We also aim to explore a tuning-free approach adaptable to diverse scenarios.\n\n\n> **Q2:**   I am curious why some of the experiments shown in Fig.4 are missing in Tab.2. For example, SPIDepth, ScaleDepth, and Metric3D results are missing in Tab.2. Is it because they are significantly worse? \n\n\n**A2:** Thank you for your valuable suggestion. In Table 2, we have fine-tuned these methods for fair comparison during the experiment history. For example, on the REAL275 dataset, the fine-tuned SPIDepth achieves a $\\delta_{1.05}$ of 11.09, likely due to its reliance on large-scale outdoor training data. The core idea of ScaleDepth is to decompose metric depth estimation into two stages: scale prediction and relative depth estimation. However, under our current data budget and scene types, this strategy is more sensitive to the convergence of scale distribution, which leads to weaker performance on $\\delta_{1.05}$ of only 11.48, which is lower than other methods. Metric3D uses a CSTM module to address scale ambiguity caused by varying camera intrinsics. However, depth estimation for transparent objects is challenging due to background interference and refraction, leading to additional scale ambiguity. As shown in Figure 4, Metric3D performs poorly on the ClearPose dataset. Thus, Metric3D is better suited for depth estimation of regular objects, while transparent and highly reflective objects (which are considered in our task) are beyond its scope. Moreover, its heavier architecture makes the fine-tuning challenging, which only achieves 10.6 in $\\delta_{1.05}$ on REAL275. Therefore, we initially omitted these results. However, based on your comment, we are willing to include them in the revision.\n\n> **Q3:**   It will be great to run an experiment on the reflective items in dataset [1] and dataset [2]. \n\n**A3:** We sincerely thank the reviewer for the insightful suggestion. In the supplementary material, we include videos demonstrating depth prediction results on reflective objects (e.g., a metal spoon), which partially showcase our method's robustness to challenging surfaces. We have carefully reviewed the two referenced works and their corresponding datasets. Both the HAMMER dataset [1] and HouseCat6D [2] are high-quality resources. We agree with you that incorporating them would further validate our approach. To provide a quick analysis, we fine-tuned our depth model on a subset of the training data and evaluated 6D pose estimation on the validation subset of HouseCat6D. We manually selected three reflective objects (teapoint, knife, and cup) as targets and generated 100 image pairs for each. \n|Matcher|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|-|\n||Oracle|57.4|41.0|62.6|68.7|68.8|\n|Oryon|DPAv2|26.6|9.1|33.6|37.3|39.5|\n||Ours|38.1|20.8|44.4|49.3|49.0|\n||Oracle|63.2|48.2|73.0|78.6|76.2|\n|Ours|DPAv2|30.1|12.0|41.9|46.3|43.8|\n||Ours|**44.5**|**26.1**|**51.8**|**55.5**|**55.2**|\n|vs.Oryon|Δ|+6.4|+5.3|+7.4|+6.2|+6.2|\n\nOur method outperforms the baseline and yields higher depth quality than fine-tuned DPAv2. Baselines like Oryon rely solely on RGB, which the HouseCat6D challenges through images with reflective objects, causing varying appearances. This makes matching solely based on RGB unreliable.\n\nIn contrast, our coarse-to-fine matching leverages depth cues for spatial consistency, enabling the model to access geometric relationships even when RGB is unreliable, thus boosting accuracy. High-resolution feature maps further refine correspondences, reducing errors in tough cases. These results showcase both the architectural strength and real-world applicability of our approach under varying RGB conditions.\n\nWe plan to further validate our approach by systematically fine-tuning on the full HAMMER dataset and evaluating pose estimation across a broader range of categories in HouseCat6D. All scripts and splits will be released to reproduce the evaluation.\n\n\n> **Q4:**   It would be great to include a comparison of such methods in the experiment section (e.g., Any-6D). Also, the experiment section lacks experiments on methods that are specifically trained on pose estimation. It's possible that the proposed method underperforms compared to such methods, but it's fair, as the proposed method is light.\n\n**A4:**  We appreciate your valuable advice. We admit works like Any-6D are high-quality contributions. In response, we have conducted additional comparisons on Tyo-L, and the results are in the table below. \n|Matcher|Depth|AR↑|VSD↑|MSSD↑|MSPD↑|ADD↑|\n|-|-|-|-|-|-|-|\n||Oracle|43.3|15.8|55.8|58.4|32.2|\n|Any-6D|DPAv2|5.2|1.3|4.3|11.0|2.2|\n||Ours|31.6|13.5|33.5|47.8|13.2|\n||Oracle|42.0|34.2|44.1|47.6|35.0|\n|Ours|DPAv2|5.8|2.8|3.5|11.1|2.9|\n||Ours|31.7|13.9|33.8|47.3|13.1|\n\nWhile Any-6D achieves slightly better results with ground truth depth, it benefits from its strong image-to-3D reconstruction model; it heavily relies on depth quality—its performance drops with DPAv2-predicted depth. Due to its global optimization (e.g., point cloud registration) improves MSSD/MSPD, but might accumulate local errors due to predicted depth, lowering VSD recall. In contrast, our method conducts a depth-aware image-based alignment, avoiding too much dependency on 3D registration with all points, achieving higher VSD recall.\n\nWe also reproduced results of methods like FoundationPose (77.9 AR on LM-O) and SAM6D (68.0 AR), which outperform ours but under distinct settings. FoundationPose needs a neural field over a full 3D scene and trains for a week on 4 V100 GPUs, and SAM6D uses dense multi-view templates from CAD models with training on 2 GPUs for 6 days. In contrast, our lightweight method fine-tunes on a single GPU using only one RGB reference image, possibly from a different scene, without implicit representation or CAD templates, and it greatly reduces computational and data requirements. We appreciate the advice and will include this discussion in the revised manuscript to clarify our method’s positioning and strengths."}]

Author rebuttal:

Meta-review: Following the discussion phase, all reviewers recommended acceptance (3 Accepts, 1 Borderline Accept), noting that the paper is well-motivated, novel, and thoroughly evaluated. The rebuttal addressed many of the reviewers’ concerns—for example, by adding cross-domain generalization experiments and other additional analyses. As a result, the AC has decided to accept the paper. Please take the reviewers’ feedback into account when preparing the camera-ready version.
Decision: Accept (poster)