Title: SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Depth Estimation

6D Pose Estimation

Novel Object Pose Estimation
Abstract: Recent 6D pose estimation methods demonstrate notable performance but still face some practical limitations. For instance, many of them rely heavily on sensor depth, which may fail with challenging surface conditions, such as transparent or highly reflective materials. In the meantime, RGB-based solutions provide less robust matching performance in low-light and texture-less scenes due to the lack of geometry information. Motivated by these, we propose **SingRef6D**, a lightweight pipeline requiring only a **single RGB** image as a reference, eliminating the need for costly depth sensors, multi-view image acquisition, or training view synthesis models and neural fields. This enables SingRef6D to remain robust and capable even under resource-limited settings where depth or dense templates are unavailable. Our framework incorporates two key innovations. First, we propose a token-scaler-based fine-tuning mechanism with a novel optimization loss on top of Depth-Anything v2 to enhance its ability to predict accurate depth, even for challenging surfaces. Our results show a 14.41% improvement (in $\delta_{1.05}$) on REAL275 depth prediction compared to Depth-Anything v2 (with fine-tuned head). Second, benefiting from depth availability, we introduce a depth-aware matching process that effectively integrates spatial relationships within LoFTR, enabling our system to handle matching for challenging materials and lighting conditions. Evaluations of pose estimation on the REAL275, ClearPose, and Toyota-Light datasets show that our approach surpasses state-of-the-art methods, achieving a 6.1% improvement in average recall.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SingRef6D, a novel object pose estimation method based on monocular image and corresponding monocular depth estimation. The method is composed of two parts. Depth estimation part Pose estimation part.\n\nDepth estimation is based on Depth Anything V2, but includes an extra part that is meant for improving depth quality as well as adapting the scale of the depth to the given training scenes.\n\nThe pose estimation part is composed of two parts, LoFTR-based matching and PointDSC-based refinement. The proposed method modifies LoFTR to take depth, which improves the quality of matching further, especially on the photometrically challenging material object, like glasses.\n\nThe paper shows an experiment on depth estimation to prove the effectiveness of the fine-tuning method proposed in the paper. also pose estimation, mainly based on a matching pipeline, to prove the performance of the proposed matching method with depth.\n\nSTRENGTHS AND WEAKNESSES:\nStrength :\n1. The paper is easy to read, the figures are intuitive, and shows good extra details in the supplementary material.\n2. Both qualitative and quantitative experiments show the superior performance of the proposed method\n3. The proposed method specifically focuses on the photometrically challenging material, such as transparent and reflective.\n\nWeakness (Suggestions on each points are written in the Questions section): \n1. Misleading term with \"Metric Depth Prediction\"\n2. Missing quantitative evaluation on few depth methods shown in qualitative evaluation\n3. The experiment on the reflective objects is missing, although it is mentioned in the abstract and intros.\n4. The experiment is weak on the 6D pose estimation part.\n5. Contribution is slightly weak - The paper's main contribution is rather upgrading and fine-tuning depth prediction, adding depth in LoFTR matching. The rest of the pipeline is rather combination of existing works.\n\nQUESTIONS:\n1. Misleading term with \"Metric Depth Prediction\"\n\n- It feels like fine-tuning is meant to overfit the depth prediction on each dataset's distance instead of actually managing to predict the correct metric depth for different conditions. For example, the scaler is fine-tuned on the ClearPose dataset again before running the validation, which means that depth prediction is only a metric when it's fine-tuned to the specific scenario. It's rather \"overfitting\" on the specific scenario than predicting the correct metric depth.\n- An interesting experiment will be using C.2's depth prior to make the pipeline compatible with different scenes without finetuning to show the robustness of the proposed network on different scenarios.\n\n2. Missing quantitative evaluation\n\n- I am curious why some of the experiments shown in Fig.4 are missing in Tab.2. For example, SPIDepth, ScaleDepth, Metric3D results are missing in Tab.2. Is it because they are significantly worse? It would be great to show their number regardless.\n\n3. The experiment on the reflective objects is missing, although it is mentioned in the abstract and intros.\n\n- There is one depth dataset [1] that is based on the same tabletop scenario with household objects, including reflective items, as well as high-quality depth ground truth, so that the author can fine-tune the proposed method. Also, a 6D pose dataset [2] that contains photometrically challenging objects, both transparent and reflective. It will be great to run an experiment on the reflective items by first fine-tuning the method on the depth dataset [1] and running pose estimation on the pose dataset [2].\n\n4. The experiment is weak on the 6D pose estimation part.\n\n- In lines 47-48, it is mentioned that the proposed method can perform competitively compared to generative or reconstruction-based methods. However, the experiment only covers matching-based methods. It would be great to include a comparison of such methods in the experiment section (e.g,  Any-6D, Zero123-6D, etc) by selecting two frames (i.e., one frame for generation, the other frame for getting pose). It's possible that the proposed method underperforms compared to such methods, but it's fair, as the proposed method is light. This experiment can give a better idea of the trade-off between complexity and performance.  Also, the experiment section lacks experiments on methods that are specifically targeted and trained on pose estimation. Including such experiments will strengthen the experiment section of the paper.\n\n\n[1] On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks, Jung et.al.\n\n[2] Housecat6D: a large-scale multi-modal category level 6D object perception dataset with household objects in realistic scenarios, Jung et. al.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SingRef6D, a method for novel object 6D pose estimation that only requires a single RGB reference image. The authors propose a lightweight pipeline that combines a fine-tuned Depth-Anything v2 model with a novel token-scaler mechanism for depth prediction and integrates depth-aware matching into LoFTR to improve correspondence estimation. The proposed method does not require CAD models, novel-view synthesis, or video-based supervision, making it efficient and practical. Experimental results on REAL275, ClearPose, and Toyota-Light demonstrate strong performance and improvements over existing works.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem addressed in this paper is well motivated. Estimating 6D pose from a single RGB image is very challenging but highly practical for real-world applications.\n- I like the idea of using depth models for this pose estimation task, as recovering object scale from only a single RGB reference is highly ambiguous. The proposed token-scaler mechanism further improves the robustness and accuracy of metric depth prediction on challenging surfaces, such as transparent or reflective materials. The reported gains in depth metrics across multiple datasets are significant.\n* The model is computationally efficient compared to recent baselines like Oryon and RoMA, requiring fewer parameters and less memory while achieving better or comparable accuracy.\n* Extensive evaluations and ablation studies are provided, which clearly demonstrate the effectiveness of each proposed component, including the depth losses and feature fusion strategies.\n\nWeaknesses:\n* One major concern in estimating 6D pose from a single RGB reference is when there is a large viewpoint gap between the reference and the query image—i.e., when the two views share only a small overlap. Although the authors show promising qualitative examples (e.g., the first row of Figure 5), I would expect to see a more thorough quantitative evaluation that directly studies this challenge. If the proposed method is robust in small-overlap conditions, that would be a very strong result to highlight in the paper.\n* While the paper includes difficult examples such as transparent objects, most of the presented samples involve objects that appear relatively clear, with minimal occlusion. It is unclear how well the proposed method handles more severe occlusions, such as those found in the Occlusion-LM dataset. The authors could consider using the single-image setup introduced in GigaPose [39] for a more comprehensive comparison.\n\nQUESTIONS:\n- How does the proposed method perform when there is a large viewpoint gap between the reference and query images—specifically, in cases where the two views have minimal overlap? \n- How robust is the proposed approach to heavy occlusion, such as in Occlusion-Linemod dataset?\n\nLIMITATIONS:\nI do not see any major limitation in this work. While the authors mention the use of object masks for target localization, I do not consider this to be a significant limitation. In practice, object masks can be easily obtained using existing tools such as SAM or SAM2. Moreover, since there are typically multiple objects present in a scene, it is entirely reasonable to specify which object the model should focus on in the reference image.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a complete pipeline for novel object pose estimation from a single RGB image. The core methodology consists of three main stages: （1）Scaled Depth Estimation: The authors first propose a fine-tuning strategy based on a token-scaler architecture. This method adapts a monocular depth estimation model to recover a scene depth map with a correct metric scale from a single RGB input. （2）Depth-guided Feature Matching: Subsequently, the paper employs a training-free feature matching technique (LoFTR) to establish correspondences for the object instance between a reference and a query image. A key point here is the use of the previously estimated depth map to guide this matching process, which aims to improve the accuracy of the resulting keypoint matches. （3）Pose Recovery: Finally, using these depth-guided matches, the relative pose between the reference and query views is solved for, which in turn yields the final pose of the object in the query image. The proposed method was evaluated on three benchmark datasets. Through comparisons with various combinations of different depth estimation models and feature matching techniques, the experimental results demonstrate that the proposed integrated approach achieves promising results for the task of novel object pose estimation.\n\nSTRENGTHS AND WEAKNESSES:\nPros：\n\n+ The paper outlines a complete framework for novel object pose estimation using a single RGB image. The methodology is presented as a sequence of three main modules: monocular metric depth estimation, depth-guided feature matching, and final pose recovery. This modular structure clearly defines the paper's technical approach.\n\n+ The authors introduce a fine-tuning strategy for a monocular depth model aimed at recovering the depth metric. They conduct experiments comparing this method against other approaches, and the paper reports that their method leads to improved depth accuracy and reduced geometric distortion.\n\n+ The proposed pipeline is evaluated on three public datasets. The paper presents quantitative results for the pose estimation task, including experiments on the ClearPose dataset, which contains challenging transparent objects.\n\nCons:\n\n- The core technical contribution appears to be the fine-tuning strategy for metric depth recovery. The proposed depth estimation method is a solution to a well-established problem (metric depth from a single image). As such, it should be also evaluated on standard depth estimation benchmarks. The current evaluation, which is confined to the three object pose datasets, does not provide a clear picture of how the method performs in a general context or how it compares to state-of-the-art depth estimation techniques. Besides, the paper seems to imply that the depth model is fine-tuned separately for each of the three datasets. This raises significant concerns about the model's generalization capabilities. It is unclear how different the fine-tuning scenes are from the testing scenes. A crucial unanswered question is: how well would a model fine-tuned on a general indoor dataset (e.g., ScanNet) perform when applied to a completely different scene with a similar scale? Without this analysis, the practicality of the proposed depth recovery method remains questionable.\n\n- From a pose estimation perspective, the paper's contribution is limited. The overall pipeline that uses feature matching followed by a PnP/relative pose solver is a standard and well-established practice. The main novelty claimed for the pose estimation part is the integration of depth to guide the LoFTR feature matcher. While this addition appears effective, the idea of using geometric priors (like depth) to refine feature matching is not new. The specific implementation is straightforward and can be seen as an incremental improvement rather than a novel conceptual contribution. Given that the method explicitly recovers depth for both the reference and query images, how does the proposed 2D matching with depth guidance compare to methods that perform keypoint matching directly on the generated point clouds, or use techniques like ICP for pose refinement? This comparison is currently missing.\n\n- The paper's experimental setup is more aligned with instance-level relative pose estimation. Therefore, the evaluation should also be extended to standard instance-level benchmarks like LINEMOD and LINEMOD-Occlusion. In addition,  the comparisons should also include recent and highly relevant instance-level methods [A,B].\n\n[A] \"3D-Aware Hypothesis & Verification for Generalizable Relative Object Pose Estimation\", ICLR 2024.\n\n[B] \"DVMNet: Computing Relative Pose for Unseen Objects Beyond Hypotheses\", CVPR 2024.\n\n- As a reference-based method, the choice of the reference image is critical to performance, yet this aspect is not analyzed. The paper does not provide any analysis on how the performance varies with the choice of the reference view. What is the typical viewpoint difference (e.g., in terms of rotation and translation) between the reference and query images in the reported experiments? The robustness of the feature-matching-based approach is highly dependent on the visual overlap between the two views. It is expected that the performance will degrade significantly as the viewpoint difference increases. A thorough ablation study is needed to quantify this effect and understand the method's limitations, which is a standard practice for evaluating reference-based methods.\n\nQUESTIONS:\n1. How about the performance of the proposed metric depth estimation model on standard depth benchmarks？Within the presented object pose estimation experiments, what is the degree of similarity or difference between the scenes used for fine-tuning the depth model and the scenes used for testing?\n\n2. Could you please provide a quantitative comparison of the proposed method against recent works[A,B] on a standard instance-level benchmark (e.g., LINEMOD)?\n\n3. Could you please provide an analysis of your method's robustness to the choice of the reference view? Specifically, in your current quantitative results, what is the typical pose difference (e.g., in terms of rotation) between the reference and query images? How does performance degrade as this difference increases?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper  proposes SingRef6D, a lightweight pipeline using a single RGB image as a reference, in contrast to some other 6D  object pose models relying on CAD models, depth sensors or novel view synthesis. Given a  query, and a reference image , both containing the same object, the objective is to estimate the 6D relative pose between them, thus all accessible reference information comes from a single RGB image.\nThe method targets transparent and multi object scenarios, with the former posing problems to depth sensors. \n\nThe main claimed innovation is a token-scaler-based fine-tuning mechanism with a novel optimization loss applied to Depth-Anything v2 and improving its depth estimation in challenging conditions. The results show improvements in depth prediction. Another claimed contribution is depth aware matching that integrates spatial relationships within LoFTR. The evaluation on REAL275, ClearPose, and Toyota-Light shows significant improvement for transparent objects.\n\nSTRENGTHS AND WEAKNESSES:\nThe pipeline is relatively simple using DPA to predict depth, SAM to crop objects of interest and PointDSC with LoFTR to match two RGBD views.  There are a number of methods for 6D object pose estimation that follow similar chain of modules and require a single image, therefore the main contribution of this work is in scaler token for fine-tuning to mitigate the scaling inconsistencies of the mono depth estimator.\nIt attempts to integrate a ControlNet like structure  to dynamically scale and modulate DINO features at coarse and global levels with InceptConv-based network during DPA finetuning.\n\nMany loss terms are combined together including standard SSI loss in DPA with BerHu loss. Proposed scale alignment  loss that is robust to noise, which is pixel level depth loss within object, edge emphasizing loss and normal consistency loss for surface deformation. \n\nThe proposed scaler and the choice of loss functions are interesting and it should be possible to make use of these ideas in other methods.\n\nAblation study shows individual contributions of the main loss components validating their usefulness. Similarly the proposed scaler makes a positive difference in finetuning of DPA.\n\n\n\na) Although transparent and untextured objects are mentioned as the main targets these are not explicitly addressed by the proposed approach. The robustness to the lack of texture and reflections comes from the depth maps of the objects that complement the RGB images. Depth estimation is claimed to be improved by finetuning of DPA with the combination of various loss functions.\n\nb) Evaluation results demonstrate significant improvements over DPA on REAL275, ClearPose, and Toyota-Light. It is unclear if this is due to fine tuning on the specific domain and whether the general performance of DPA was not compromised as the evaluation is limited to object pose benchmarks only.  Generalisation across domains is not validated well.\n\n\nc) The choice of methods to compare against and the benchmarks could be more comprehensive.\nThe paper claims multi object pose but the benchmarks are focused on relatively uncluttered scenarios.\nREAL275 has some minor occlusions but other benchmarks could have been used to test robustness to textureless objects partial occlusion and large variations of viewpoints like in HOD3, LM-O, YCB-Video, LineMod.\n\nd) In terms of comparison to SOTA methods, SIFT, RoMA and Oryon are relatively weak baselines.  Including results for stronger methods would have made the evaluation more convincing. Any6D, Fs6d, FoundationPose could have been reported too for completeness of evaluation as well as original LoFTR which is used in this method too and makes part of a contribution. \nOne2Any is also relevant although from cvpr25.\n\nQUESTIONS:\nSee a-d weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}