GS-CPR: Efficient Camera Pose Refinement via 3D Gaussian Splatting

Authors: Changkun Liu, Shuai Chen, Yash Sanjay Bhalgat, Siyan HU, Ming Cheng, Zirui Wang, Victor Adrian Prisacariu, Tristan Braud
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Visual Localization, Camera Pose Estimation, 3D Gaussian Splatting

Abstract:
We leverage 3D Gaussian Splatting (3DGS) as a scene representation and propose a novel test-time camera pose refinement (CPR) framework, GS-CPR. This framework enhances the localization accuracy of state-of-the-art absolute pose regression and scene coordinate regression methods. The 3DGS model renders high-quality synthetic images and depth maps to facilitate the establishment of 2D-3D correspondences. GS-CPR obviates the need for training feature extractors or descriptors by operating directly on RGB images, utilizing the 3D foundation model, MASt3R, for precise 2D matching. To improve the robustness of our model in challenging outdoor environments, we incorporate an exposure-adaptive module within the 3DGS framework. Consequently, GS-CPR enables efficient one-shot pose refinement given a single RGB query and a coarse initial pose estimation. Our proposed approach surpasses leading NeRF-based optimization methods in both accuracy and runtime across indoor and outdoor visual localization benchmarks, achieving new state-of-the-art accuracy on two indoor datasets.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730322770033

thread_type:
official_review

<thread_object>
creator: reviewer_uwqs

<content>
summary:
This proposes a novel visual localization method that utilizes 3D Gaussian Splatting. This approach replaces the point cloud in the pipeline of visual localization with a 3DGS model as the 3D representation. The camera pose of a query image is computed in a coarse-to-fine manner. The coarse pose is predicted by using a pretrained pose estimator. The pose refinement is carried out by neural rendering and image matching. The method is evaluated across indoor and outdoor datasets, outperforming state-of-the-art NeRF-based methods and traditional localization methods.

strengths:
The idea of using 3DGS for visual localization is simple yet effective. It replaces the key components in the localization pipeline with advanced alternatives. For instance, the image retrieval module is replaced with a pose estimator for a coarse pose prediction. The 2D-3D matching is performed based on a prebuilt 3DGS model instead of the point cloud. The authors take the challenges, such as the appearance gap, into account and introduce an exposure adaption module, which enhances the appearance consistencies. The proposed method achieves more accurate camera pose estimates than the existing localization approaches on several datasets. The paper is well-written and easy to follow.

weaknesses:
-	The superiority of 3DGS in the framework of visual localization is a bit unclear. The scalability of 3DGS seems questionable. For instance, in a large-scale and unbounded scenario, the 3D map might be built from thousands of images in the traditional localization pipeline. The 3DGS appears to be inapplicable in this case. If this is the case, the requirement of the 3DGS model might be a limitation in real applications.

-	The authors argue that in HLoc, the top-1 retrieved image may have limited overlap with the query. However, this is the limitation of the current image retrieval techniques rather than a shortcoming of HLoc itself. In the introduced pipeline, the coarse pose is computed using an off-the-shelf approach, which might be more powerful than image retrieval. Therefore, this approach could also benefit HLoc. For example, the reference can be selected based on the predicted camera pose. The advantage of 3DGS is not convincing enough.

-	The evaluation metrics are inconsistent across different datasets. The authors report accuracy in some cases and pose error in others.

-	The comparison with traditional localization methods like HLoc on Cambridge Landmarks is missing.

questions:
What is the major difference compared with 3DGS SLAM methods since they can also optimize the camera poses?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730351860231

thread_type:
official_review

<thread_object>
creator: reviewer_tsxc

<content>
summary:
The paper pays attention on camera pose refinement with 3D Gaussian Splatting as a scene representation, which aims to enhance the localization accuracy of state-of-the-art absolute pose regression and scene coordinate regression methods. A refinement framework, called GSLoc, is proposed to establish 2D-3D correspondences by an exposure-adaptive module. Experiments demonstrate that GSLoc improves the localization accuracy over different relocaliztion methods.

strengths:
1 The paper proposes a novel test-time pose refinement framework, termed GSLoc, which employs 3D Gaussian Splatting for scene representation and utilizes the 3D vision foundation model MASt3R for precise matching.

2 GSLoc enhances the pose estimation accuracy of both APR and SCR methods across these benchmarks.

weaknesses:
1 The main contribution is incremental. 

(1)Firstly, the motivation of GSLoc is less interesting. From my viewpoint, the usage of feature matching for pose refinement is not novel. Meanwhile, the paper just uses 3D Gaussian Splatting for scene representation instead of traditional hand-crafted point cloud. Compared with Gaussian Splatting SLAM systems (like Gaussian SLAM [1, 2]), the idea of this paper is less innovative.

(2)Secondly, the innovative contributions are few. The paper seems a combination of current hot technologies, like 3D Gaussian Splatting and the 3D vision foundation model MASt3R. 

[1]Gaussian Splatting SLAM, CVPR 2024

[2]GS-SLAM: Dense Visual SLAM with 3D Gaussian Splatting

[3]LoFTR: Detector-Free Local Feature Matching with Transformers, CVPR 2021

[4]DKM: Dense Kernelized Feature Matching for Geometry Estimation

2 The localization accuracy comparisons with feature matching based methods are missing. The core idea behind GSLoc is feature matching with 3D Gaussian Splatting. It is curious that whether GSLoc is also effective for pose refinement of leaned feature matching based methods, like LoFRT[3], DKM[4] or others. It is suggested to add these results. 


3 The overlap of Figure 2 and 3 is somwhat high. It is suggested to modify Figure 3 to express the difference of GSLocrel in comparison with GSLoc.

questions:
The question and suggestions are listed in the part of Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554180020

thread_type:
official_review

<thread_object>
creator: reviewer_hmwq

<content>
summary:
This paper proposes a new approach to optimize the initial camera pose in visual localization. Using 3DGS as the scene representation, RGB images and depth maps are rendered based on the initial camera pose of the query image estimated by other methods. Then, 2D-2D matches between the query image and the rendered results are computed via MASt3R and elevated to 2D-3D matches using the depth maps. Finally, the relative pose is solved through PnP to optimize the initial camera pose. Compared to previous camera pose optimization methods based on cost functions and optimization, the GSLoc pipeline achieves one-shot pose optimization, offering advantages in both speed and accuracy. The effectiveness of GSLoc has been validated on indoor and outdoor visual localization datasets.

strengths:
- The GSLoc pipeline combines the advantages of speed and accuracy. Through high-quality 3DGS reconstruction and rendering, it directly calculates optimized camera poses using image matching, which enhances the robustness and usability of the system compared to optimization methods based on specific cost functions.
- The writing of the paper is easy to follow.
- Comprehensive experiments have been conducted, demonstrating advantages over other camera pose optimization methods and traditional visual localization approaches.

weaknesses:
- The paper lacks certain experiments to demonstrate the necessity of the proposed method:
    - Table 2 shows the effectiveness of GSLoc compared to HLoc. However, it does not pair HLoc with an improved initial pose estimation method, such as ACE, as done for GSLoc. A reasonable baseline would be to use ACE to estimate the initial camera pose, retrieve the top-K images with the closest poses from the database, and then compute the localization result using HLoc and optionally refinet it with PixLoc.
- The novelty of the method is limited, as the effectiveness of the pipeline primarily relies on improvements in reconstruction and rendering quality provided by Scaffold-GS as the scene representation.
- There is a lack of analysis on the method’s applicability:
    - Feasibility of high-quality 3DGS reconstruction: Reconstructing high-quality 3DGS to render the images needed for GSLoc requires densely sampled viewpoints. Moreover, modeling requires images with viewpoints very close to that of the query. How robust is GSLoc when there are no views close to the query viewpoint (i.e., in cases of wide baselines)?
    - Impact of significant appearance changes on the method (e.g., on datasets like Aachen-day-night or similar ones of smaller scenes).
    - Feasibility of the method for large-scale visual localization (e.g., Aachen-day-night).

questions:
- What are the feature detector and matcher used by HLoc in the comparison shown in Table 2?
- In the runtime analysis, there seem to be some issues with the time calculation for GSLoc and GSLoc_rel.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605101851

thread_type:
official_review

<thread_object>
creator: reviewer_rrxl

<content>
summary:
This method applies 3DGS to address the test-time camera pose refinement task. The manuscript first renders RGB and depth images from 3DGS, then matches the query image with the rendered images. To bridge the domain gap between the query image and 3DGS, an exposure-adaptive module is incorporated. Additionally, MASt3R is used as a training-free feature extractor for accurate matching. Extensive experiments demonstrate the effectiveness of the proposed GSLoc method.

strengths:
S1 - The task appears meaningful, and the results presented in the manuscript are quite promising.

S2 - The Affine Color Transformation module is a noteworthy component, effectively addressing the RGB domain gap between the query image and the rendered images.

S3 - The experiments are well-designed and effectively demonstrate the superiority of the proposed GSLoc method over various baseline approaches.

weaknesses:
W1 - The method appears to lack comparisons with state-of-the-art approaches, such as GLACE + GSLoc and ACE/GLACE + NeFes. Including these comparisons would provide a clearer benchmark for evaluating its performance.

W2 - I’m curious about the quality of the depth images rendered by 3DGS and whether they are consistent enough to serve as a reliable information source.

questions:
Q1 - The sota comparision such as GLACE + GSLoc.

Q2 - The quality of the rendered depth image.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730623666531

thread_type:
official_review

<thread_object>
creator: reviewer_bjdv

<content>
summary:
This paper presents a test-time refinement method, GSLoc, for improving camera localization accuracy. With a single RGB query, the method leverages ACT (exposure-adaptive affine color transformation) enhanced 3DGS and MASt3R to establish 2D-3D correspondences and then uses PnP + RANSAC for pose estimation. On two indoor and one outdoor datasets, the method achieved SOTA results.

strengths:
++ The paper shows that, with introducing any new techniques, simply combining existing techniques can achieve state-of-the-art results for camera localization.

weaknesses:
-- It appears that the authors misused \citep and \citet. This creates some difficulty when reading the paper.

-- Formulas should be followed by punctuation marks.

-- Please provide the detailed derivation of Eq. (3). It seems the translation part is incorrect.

-- Please provide analyses on the failure cases and potential limitations of the work.

-- The title is GSLoc, where GS stands for Gaussian Splatting, but the meaning of Loc is not mentioned in the paper. Since the focus of this work is on pose/localization refinement, Loc may not be the most suitable term.

questions:
What are the failure cases and potential limitations of the work? Please provide some quantitative and qualitative results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732095551179

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Revision:

comment:
Dear Reviewers,

We sincerely appreciate your recognition of the effectiveness of our approach and its state-of-the-art performance among neural render pose (NRP) estimation methods. Thank you for your thorough and insightful reviews, which have significantly contributed to improving our paper.

In this revision, we have implemented several changes in response to your valuable suggestions. These updates are highlighted in blue in the revised manuscript.

* Following Reviewer BjDV’s suggestion regarding the title, we carefully considered and updated the **title and framework name** from **GSLoc** (Localization) to **GS-CPR** (**C**amera **P**ose **R**efinement) to better reflect the focus of our work on camera pose/localization refinement. For clarity in discussions with reviewers, we continue to use the original name GSLoc, while the title change has been implemented in the rebuttal version. We also addressed the minor writing issues raised by Reviewer BjDV.

* Based on Reviewer UwQs’s feedback, we updated Table 4 to ensure consistent evaluation metrics across datasets.  We also included the configuration details of HLoc in Table 2, as recommended by Reviewer HMwQ.

* We added a new paragraph in Appendix A.3 to clarify our system design and included a limitation analysis in Appendix A.5. In addition, in response to Reviewer rRXL’s suggestion, we included the results of GLACE + our framework in Tables 12 and 13 of the Appendix A.6.

Finally, we have provided detailed responses to all your comments and concerns in the individual review threads. We hope that our revisions and responses address any remaining issues and help clarify the contributions of our work. We kindly request that you reconsider your evaluation of our paper in light of these changes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for optimizing the initial camera pose in visual localization. Using 3DGS as the scene representation, RGB images and depth maps are rendered based on the initial camera pose of the query image estimated by other methods. Then, 2D-2D matches between the query image and the rendered results are computed via MASt3R and elevated to 2D-3D matches using the depth maps. Finally, the relative pose is solved through PnP to optimize the initial camera pose.  The idea of using 3DGS for visual localization is simple yet effective. Through high-quality 3DGS reconstruction and rendering, the proposed method directly computes optimized camera poses using image matching, which enhances the robustness and usability of the system.  On the other hand, the reviewers raised concerns regarding lacking certain experiments, insufficient analysis of the method’s applicability, incremental contribution, and limitation.  Some important SOTA methods are suggested for comparison.  The choice of the baseline is also argued. The combination of technologies is raised as a concern. Failure cases and/or clarifying potential limitations are also pointed out.  The authors have provided additional experiments to address the concerns on experiments in the rebuttal with arguing analysis of the results. The authors’ rebuttal and following discussion between the authors and the reviewers have resolved the raised concerns, leading to convincing contribution of the paper.  The reviewers are all positive for the paper.  This paper should be accepted, accordingly.  It is highly suggested for the authors to release the code in public so that the paper becomes more citable.