F2M-Reg: Unsupervised RGB-D Registration with Frame-to-Model Optimization

Authors: Zhinan Yu, Zheng Qin, Yijie Tang, Yongjun Wang, Renjiao Yi, Chenyang Zhu, Kai Xu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: RGB-D registation, unsupervised learning, frame-to-model optimization

Abstract:
This paper focuses on training a robust RGB-D registration model without ground-truth pose supervision.
Existing methods usually adopt a pairwise training strategy based on differentiable rendering, which enforces the photometric and the geometric consistency between the two registered frames as supervision. However, this frame-to-frame framework suffers from poor multi-view consistency due to factors such as lighting changes, geometry occlusion and reflective materials. In this paper, we present F2M-Reg, a novel frame-to-model optimization framework for unsupervised RGB-D registration. Instead of frame-to-frame consistency, we leverage the neural implicit field as a global model of the scene and use the consistency between the input and the rerendered frames for pose optimization. This design can significantly improve the robustness in scenarios with poor multi-view consistency and provides better learning signal for the registration model. Furthermore, to facilitate the neural field optimization, we create a synthetic dataset, Sim-RGBD, through a photo-realistic simulator to warm up the registration model. By first training the registration model on Sim-RGBD and later unsupervisedly fine-tuning on real data, our framework enables distilling the capability of feature extraction and registration from simulation to reality. Our method outperforms the state-of-the-art counterparts on two popular indoor RGB-D datasets, ScanNet and 3DMatch. Code and models will be released for paper reproduction.

Decision: reject
Reviewer scores: 5, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730534192828

thread_type:
official_review

<thread_object>
creator: reviewer_bx3g

<content>
summary:
The paper introduced a method for RGB-D registration using frame-to-model optimization. The pipeline incorporates a pretrained registration model suing PointMBF and Co-SLAM for point cloud registration. The experiments on real-world ScanNet and 3DMatch datasets present the superiority of the proposed method. The ablation studies investigated the effectiveness of each components.

strengths:
- The writing is comprehensive and easy to follow
- The proposed method demonstrates high performance compared to baseline methods on two real-world RGB-D datasets.
- The ablation study on the main text and supplementary materials are thorough and effectively showcases the efficacy of the proposed framework.

weaknesses:
- The proposed method should not be characterized as unsupervised learning as it claimed. Instead, it involves a registration model that has been pretrained on a synthetic RGB-D dataset and subsequently adapted to real-world RGB-D scenes during inference. Strictly speaking, the method aligns more precisely with a zero-shot learning approach. Please ensure the claims are accurately represented.

- The experiments could benefit from utilizing more recent datasets. The ScanNet-v2 dataset, while widely used, is dated and known for sensor noise that results in unreliable depth maps with many gaps containing zero or infinity values. More accurate and high-resolution RGB-D streams are available in newer datasets like ScanNet++ and TUM RGB-D. Conducting additional experiments on these recent datasets is recommended.

- The proposed pipeline incorporates Co-SLAM, which utilizes a tracking system for camera pose estimation, requiring that the input RGB-D stream strictly follow a time series. However, this requirement shows limitations for general RGB-D registration tasks that handle multi-view data, where the input does not necessarily adhere to a time-series format. In scenarios involving unordered data, the effectiveness of a SLAM system may be compromised.

questions:
- The paper states that the frame-to-frame framework experiences difficulties in maintaining multi-view consistency due to issues like lighting variations, geometric occlusions, and reflective surfaces; however, NeRF encounters similar challenges under these conditions. It should be clarified how the frame-to-model approach addresses these limitations in comparison to the frame-to-frame method.

- In the proposed framework, initial poses are generated using a bootstrap method trained on synthetic RGB-D data. Considering that Co-SLAM, utilized for tracking within the proposed framework, also incorporates its pose initialization strategy based on constant movement assumption, how do these initial poses from the bootstrap method align or integrate with the initial pose assumptions in Co-SLAM? Specifically, please detail any adaptations or refinements made to ensure consistency between the poses initialized by F2M-Reg and the subsequent pose tracking performed by Co-SLAM,

- Please also discuss the limitations of the proposed methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573506611

thread_type:
official_review

<thread_object>
creator: reviewer_qbae

<content>
summary:
The paper introduces an unsupervised method for robust RGB-D registration without ground-truth pose supervision. Unlike prior methods that rely on frame-to-frame photometric and geometric consistency, which are often affected by lighting changes, occlusion, and reflective materials, F2M-Reg employs a frame-to-model approach. The method begins with pre-training the model on a synthetic dataset, Sim-RGBD, with ground-truth poses, and subsequently fine-tunes it on real-world datasets without ground-truth poses by leveraging a neural implicit field as a 3D scene representation for pose optimization. This approach enhances robustness against multi-view inconsistencies, as demonstrated by experimental results comparing F2M-Reg with existing methods. In summary, F2M-Reg contributes a new unsupervised RGB-D registration framework, a synthetic dataset for initial model training, and an effective frame-to-model approach, setting new benchmarks on popular RGB-D datasets.

strengths:
The paper identifies the limitations of frame-to-frame matching in unsupervised learning, particularly due to instabilities arising from lighting variations, occlusions, and reflective surfaces. The proposed frame-to-model matching, supported by a neural implicit field (NeRF), effectively mitigates these issues, demonstrating a significant improvement over traditional methods.

weaknesses:
- The use of the term bootstrap in the paper is potentially misleading. In deep learning, bootstrapping generally refers to iterative self-training, where a model refines itself by generating pseudo-labels and learning from them. The training on synthetic data described in this paper aligns more with pre-training. However, the fine-tuning on real datasets with initial poses refined through NeRF could be called bootstrapping.

questions:
- The performance gains from Sim-RGBD bootstrapping across different datasets is not quite consistent. Can the authors provide insights into why bootstrapping appears less critical for ScanNet compared to 3DMatch?

- NeRF optimization, particularly joint pose and neural field optimization, typically assumes static scenes. This assumption might limit the method's performance in dynamic environments. If this is indeed a constraint, it should be acknowledged in the paper. If not, could the authors clarify why the method remains effective in dynamic scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641671708

thread_type:
official_review

<thread_object>
creator: reviewer_sxy2

<content>
summary:
This paper presents F2M-Reg, an unsupervised RGB-D registration framework that addresses the frame-to-frame registration task by dealing with multi-view inconsistencies with bootstrapping with a synthetic dataset.

strengths:
1. F2M-Reg stands out by shifting from a frame-to-frame to a frame-to-model approach for RGB-D registration, which is an extension of existing approaches in the context of unsupervised 3D vision tasks. 
2. This use of a neural implicit field as a global scene model to capture broader scene-level information is a possible direction to handle complex conditions, such as low overlap and lighting changes, where traditional methods often fall short.
 3. The introduction of a synthetic bootstrapping dataset, Sim-RGBD, bridges the gap between synthetic and real-world performance in unsupervised settings, which is a notable improvement in unsupervised model initialization.

weaknesses:
1. Although F2M-Reg is compared with several baselines, it is unclear where the improvement comes from. According to Table 4, the results without bootstrapping are not exciting enough.
2. In order to evaluate the effectiveness of the neural implicit field-guided mechanism, this paper needs additional experiments and comparisons with SOTA approaches without bootstrapping.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643825559

thread_type:
official_review

<thread_object>
creator: reviewer_4zed

<content>
summary:
This paper proposes a frame-to-model optimization framework guided by a neural implicit field for unsupervised RGB-D registration. By introducing the differential rendering capabilities of neural radiance fields, better pose supervision can be achieved.  Meanwhile，this paper creates a synthetic dataset for warming up registration model.

strengths:
1. The paper introduces neural radiance fields to provide global information for the registration training
2. It builds a synthetic dataset to pretrain the model, ensuring the rationality of pose initialization.

weaknesses:
1. The comparison with existing work seems unfair:  

   - sota methods like PointMBF are trained purely without supervision while the proposed method requires extra dataset with pose labels. In Table 1, the metrics of the proposed method looks slight better than existing unsupervised data. Does the improvement come from the benefit of including extra dataset? Please provide results without the boostrap using the process of training the registration model on the Sim-RGBD dataset for Table 1. Or please consider other ways to make a fair comparison. Will other work benefit the extra dataset with pose labels as well?

   -  In line 368-370, the authors propose to evaluate more difficult setting “evaluating view pairs sampled 50 frames apart” and mentioned “However, due to insufficient overlap in some segments of the data for pairs sampled 50 frames apart, the evaluation significantly distorts both the mean and median values. As a result, we have chosen not to include these results in our experimental presentation.” My question: why this 50 frames apart setting? Also if the insufficient overlap pairs are excluded, 

2. The influence of the warming up dataset is not studied. How large the warming up dataset should be? How similar the warming up dataset should be to the target datasets? What’s is the influence of number objects of ShapeNet on the final metrics? For the construction of datasets, there are already many synthetic datasets for indoor scenes, such as Replica and Habitat. This paper does not demonstrate the advantages of the custom dataset compared to other datasets. 

3. The training efficiency. As NeRF is very slow in training, the proposed method should require much more computation resources than PointMBF and hard to scale.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732368595337

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewer for their valuable comments, recognizing the interestring problem being solved(Reviewer Sxy2), the technical soundness of the method(All Reviewers), the good writing quality (Reviewer 4zeD), and intuitive illustration (Reviewer Sxy2, Reviewer Qbae), the comprehensive experiments (All Reviewers), the outstanding performance (All Reviewers), and the application value(Reviewer 4zeD). We will address each reviewer's concerns individually below.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732738179079

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of changes on rebuttal revision

comment:
Dear reviewers,
We have made several modifications based on the your feedbacks and re-uploaded the PDF. The changes are summarized as follows:
- For **Reviewer 4zeD**, we updated all tables in the paper to report both the mean and median values of the registration models.
- For **Reviewer Sxy2**, we added an **Ablation on Different Module Combinations** in the Appendix to emphasize the paper's core contribution. We demonstrate that the F2M approach enables more effective unsupervised training of the registration model, with a synthetic warm-up mechanism addressing the sensitivity of F2M's neural field to initial poses.
- For **Reviewer Qbae**, we replaced the term "bootstrap" with "warm-up" throughout the paper to better describe the pre-training process. This modification is more pertinent to the claim that pre-training on a synthetic dataset provide a better initialization for F2M approach to achieve better performance.
- For **Reviewer bx3G**, we added a comparison table between F2M-Reg and PointMBF on TUM RGB-D in the Appendix. Additionally, we included a **Limitation** section in Section 6 to acknowledge and discuss potential weaknesses in our method.

We will continue to address the reviewers' concerns comprehensively and provide detailed responses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302967248

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Disccusion

comment:
Dear Chairs and Reviewers,

Hope this message finds you well. 

As the discussion period comes to a close, we would like to provide a brief summary of our discussions with the reviewers for reference. First and foremost, we sincerely thank all the reviewers for their insightful comments and valuable suggestions.

We summarize the main concerns raised by the reviewers along with our corresponding responses as follows:

- **About the performance between F2M-Reg w/o warm up and PointMBF**.  As detailed in the main paper, the frame-to-model approach is highly sensitive to model initialization, which we address with the synthetic warm-up mechanism. Additional results from various training datasets (ScanNet, 3DMatch) and evaluation settings (20/50-frame apart) further demonstrate the superiority of our method.
- **About the writing.** We offer a deeper explanation of the pipeline and its individual modules, including further discussion on the differences between zero-shot learning and unsupervised registration in our method.  We also clarify and modify ambiguous terms and provide a more detailed description of the advantages of the frame-to-model approach over the frame-to-frame approach, supported by extensive experimental evidence.
- **About the usage of Co-SLAM like approach in the fine-tuning stage.** This issue extends to the question of whether training datasets need to be sequential and the scope of our method. Notably, our approach can train the registration model under low-overlap conditions, where the assumption of constant movement no longer holds. The registration model then provides the initial pose necessary for constructing the neural implicit field.
- **About more ablation studies.**  We attribute the improvement to both warming up and fine-tuning stages. To identify the specific contributions of each, we conduct ablation studies on warming-up dataset size, object count per scene, model generalizability, and rendering strategies. Furthermore, we evaluated our method and PointMBF under the 20-frame-apart and 50-frame-apart settings. 

---

Based on the discussion with reviewers, we also present a brief summary of our paper  as follows:

- **Observation:** Existing frame-to-frame methods usually adopt a pairwise training strategy based on differentiable rendering, which suffers from poor multi-view consistency due to factors such as lighting changes, geometry occlusion and reflective materials.
- **Solution:** F2M-Reg addresses these issues by introducing a frame-to-model optimization framework, which leverages the neural implicit filed as a global model of the scene and use the consistency between the input and the re-rendered frames for pose optimization. To facilitate the neural field optimization, F2M-Reg warms up the registration model on synthetic dataset, Sim_RGBD.
- **Results:** F2M-Reg outperforms the state-of-the-art counterparts on two popular indoor RGB-D datasets, ScanNet and 3DMatch. Also F2M-Reg achieves significantly better performance than previous methods in more challenging scenarios with lower overlap or severe lighting changes.
- **Highlights:**  Focusing on unsupervised RGB-D registration task, our work has the following highlights:
  - **Frame-to-model optimization framework:** The infusion of global reconstruction information enhances the reliability of re-rendering, which fortifies the robustness of registration model.
  - **Synthetic warm-up mechanism:** The warming up mechanism  provide high-quality initial poses for neural implicit field optimization.

Thank you once again for your efforts in reviewing and discussing our work. We greatly appreciate all the valuable feedback that contributed to enhancing our submission.

Sincerely
Authors of Submission 7404
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an unsupervised method for robust RGB-D registration without ground-truth pose supervision where a frame-to-model approach is employed. The proposed method begins with pre-training the model on a synthetic dataset, Sim-RGBD, with ground-truth poses, and subsequently fine-tunes it on real-world datasets without ground-truth poses by leveraging a neural implicit field as a 3D scene representation for pose optimization.