Title: 3D Gaussian Splatting based Scene-independent Relocalization with Unidirectional and Bidirectional Feature Fusion
Venue: NeurIPS 2025
Accepted: yes
Keywords: Camera relocalization

3D Gaussian Splatting

Feature Fusion
Abstract: Visual localization is a critical component across various domains.
The recent emergence of novel scene representations, such as 3D Gaussian Splatting (3D GS), introduces new opportunities for advancing localization pipelines.
In this paper, we propose a novel 3D GS-based framework for RGB based, scene-independent camera relocalization, with three main contributions.
First, we design a two-stage pipeline with fully exploiting 3D GS.
The pipeline consists of an initial stage, which utilizes 2D-3D correspondences between image pixels and 3D Gaussians,
followed by pose refinement using the rendered image by 3D GS.
Second, we introduce a 3D GS based Relocalization Network, termed GS-RelocNet, to establish correspondences for initial camera pose estimation.
Additionally, we present a refinement network that further optimizes the camera pose.
Third, we propose a unidirectional 2D-3D feature fusion module and a bidirectional image feature fusion module, integrated into GS-RelocNet and the refinement network, respectively, to enhance feature sharing across the two stages.
Experimental results on public 7 Scenes, Cambridge Landmarks, TUM RGB-D and Bonn demonstrate state-of-the-art performance.
Furthermore, the beneficial effects of the two feature fusion modules and pose refinement are also highlighted.
In summary, we believe that the proposed framework can be a novel universal localization pipeline for further research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a camera relocalization system coupled with unidirectional and bidirectional feature fusion module on 3DGS scenes. It extends the state of the art in neural rendering and pose estimation by allowing robust relocalization in unseen environments without scene-specific training. \nThe pipeline is structured in two stages: \n1. A coarse pose estimation module, GS-RelocNet, is used to obtain the confidence matrix between image pixels and randomly selected Gaussians. The confidence matrix is further processed with PnP with RANSAC algorithm to obtain the initial camera pose. GS-RelocNet is trained by comparing with the groundtruth confidence matrix. \n2. A rendered view under the predicted camera pose is processed together with the groundtruth rendering by a refinement network. This network outputs the residual coordinate map, also followed by a PnP method with RANSAC to predict the relative camera pose. Refined pose is calculated based on the original pose and relative pose. The refinement network is trained with coordinate map loss and an auxiliary loss for camera’s position and orientation.\n\nPerformance on existing datasets claim high localization accuracy and efficiency (~15 fps). The refinement network appears to improve significantly compared to scene-independent methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Scene-independent relocalization for 3DGS scenes: This paper achieves zero-shot generalization based solely on renderings and 3DGS features, unlike prior work 3DGS-Reloc[1], which relied on scene-specific LiDAR scans.\n2. Feature fusion architecture is well-motivated. Outperforms single-path approaches in GS-CPR [2].\n\nWeaknesses:\n1. In this paper, there are 4096 selected 3DGS for training and inference. For large-scale scenes, this resolution may reduce relocation accuracy.\n2. The method proposed in this paper may require higher gpu memory usage compared to other models.\n\n[1] Jiang P, Pandey G, Saripalli S. 3dgs-reloc: 3d gaussian splatting for map representation and visual relocalization[J]. arXiv preprint arXiv:2403.11367, 2024.\n\n[2] Liu C, Chen S, Bhalgat Y S, et al. GS-CPR: Efficient camera pose refinement via 3d gaussian splatting[C]//The Thirteenth International Conference on Learning Representations. 2025.\n\nQUESTIONS:\n1. The paper samples 3D Gaussian components uniformly within grid cells to construct correspondences. However, different 3D Gaussian components may have different scales and sizes, which in turn may influence their respective projected area on the image plane. This variation could affect the confidence scores within the confidence matrix and subsequently the final pose estimation. How sensitive is your method’s performance to the sampling strategy of 3D Gaussian components with different sizes?\n\n2. The pipeline utilizes rendered views from 3D Gaussian. However, rendered images may be sensitive to environment conditions, (e.g. lighting directions or intensities, artifacts from the original 3DGS scene). How robust is your method to these variations in rendered feature maps?\n\n3. This pipeline is trained on limited datasets. Under what conditions does the system fail (e.g., low-texture regions, reflective surfaces)?\n\nLIMITATIONS:\nThe paper assumes the scene is static. The method’s ability to deal with moving objects, structural modifications, or other dynamic components — a common scenario in real-world applications — is currently unclear or unaddressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel scene-independent camera relocalization framework based on 3D GS, comprising two stages: initial pose estimation via 2D-3D correspondences between image pixels and 3D Gaussians, and pose refinement using rendered images from 3D GS. The framework introduces GS-RelocNet to establish correspondences and integrates a unidirectional 2D-3D feature fusion module and a bidirectional image feature fusion module. Experiments on 7 Scenes and Cambridge Landmarks demonstrate state-of-the-art performance, particularly in scene-independent relocalization, with significant improvements over existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The first scene-independent relocalization framework using 3D GS.\n\n2. The unidirectional and bidirectional fusion modules enhance feature sharing, validated by ablation studies showing substantial performance gains.\n\n3. Runs at 15.4 FPS, outperforming other 3D GS-based methods, making it suitable for real-world applications.\n\nWeaknesses\n1. The paper emphasizes scene-independent methods. However, the paper only includes one benchmark on 7Scenes under the scene-independent setting. Therefore, more experiments (such as Cambridge Landmarks) under scene-independent conditions should be included.\n\n2. The evaluated datasets are relatively simple and structured, where the current method already demonstrates excellent performance. It is suggested to conduct evaluations on more challenging datasets.\n\n2. The Introduction is somewhat confusing. It is suggested to refine it to clearly express the logical links among the challenges, motivation, and methodology.\n\nQUESTIONS:\n1. It is suggested to evaluate the generalization performance of the proposed method across multiple test datasets used for scene-independent methods, similar to how methods like DUSt3R and VGGT are evaluated.\n\n2. It is suggested to briefly explain the connections of each method module in the caption of Figure 1 to enhance readability.\n\n3. There is a contradiction between \"Scene-independent\" and \"Relocalization\" in the title, as Relocalization inherently implies scene-dependent settings. It is suggested to replace \"Relocalization\" with \"Localization\".\n\nLIMITATIONS:\nThe experimental scenes in the paper are relatively simple and structured. It is suggested to enhance the method's performance in large-scale scenes with extreme appearance variations, which are more significant for real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a camera localization method based on 3D Gaussian splatting.\n\nIn the first stage, the method generates 2D–3D correspondences. A neural network, GS-RelocNet, takes as input the RGB image and the 3D Gaussians, and outputs a confidence matrix. 2D–3D correspondences are derived from this matrix, and the pose is then estimated using PnP+RANSAC. Within GS-RelocNet, a unidirectional feature fusion module is proposed. GS-RelocNet is trained to regress a ground-truth confidence matrix, where each entry encodes the Mahalanobis distance between pixel coordinates and the reprojected 3D points.\n\n\nThe second stage is a refinement step. A refinement network takes the real RGB image and a rendered RGB image and outputs a residual coordinate map, following Wang & Qi. Here, a bidirectional feature fusion module is proposed. The network is trained using a coordinate loss map and a pose loss.\n\nThe approach is evaluated on 7-Scenes, Cambridge Landmarks, and an additional dataset (12-Scenes, in the supplementary material).\nIn the scene-dependent setting, the results are comparable to the state-of-the-art STDLoc (CVPR 2025).\nIn the scene-independent setting, the proposed method is the first to use Gaussian splatting and significantly outperforms existing methods on 7-Scenes.\nMoreover, the method is considerably faster than its competitors: 65 ms versus 143 ms for STDLoc.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper proposes the first scene-independent camera localization method based on 3D Gaussian splatting.\n2. The method is computationally efficient.\n3. The outperforms state-of-the-art methods on 7-Scenes in the scene-independent setting.\n4. An extensive ablation study supports the architectural and design choices.\n\n## Weaknesses\n1. Only a single NeurIPS paper (Zhu, 2024) is cited, which raises concerns about whether the paper is fully within the scope of NeurIPS.\n\n2. The authors do not plan to release their code and provide no justification. Reproducing the results may be challenging, as some implementation details are missing. For example, in the description of the Unidirectional Feature Fusion module, it appears that RGB features are reshaped to apply the residual connection, but the input and output dimensions are not specified.\n\n3. Two main contributions are the unidirectional and bidirectional feature fusion modules, but it is unclear why classical cross-attention layers were not considered instead. I would have expected a comparison with more standard cross-attention architectures (e.g., Dust3R’s ViT variant) in the ablation study.\n\nQUESTIONS:\n1. **Ablation study** -  Could you include an architecture using cross-attention layers in the ablation study, or explain why this was not considered appropriate? \n2. **Technical details** - Since the code will not be released, could you provide the missing details (e.g., tensor dimensions) to make re-implementation easier?\n3. **Auxiliary loss** - For the auxiliary loss, how did you backpropagate through the predicted camera pose produced by the RANSAC module?\n4. **Computation time** - Could you provide a breakdown of the computation time for each module (GS-RelocNet, PnP+RANSAC, and refinement iterations)?\n\nLIMITATIONS:\nAlthough the authors indicate in the checklist that limitations are discussed, I was unable to find any such discussion in the main paper or the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task visual localization which consists in estimating the 6DoF pose of a query image within a known environment. It is a crucial task for AR/MR and autonomous-driving systems. One popular way of solving pose estimation is regressing the 3D position (in world coordinates) of each pixel within an image and estimating the pose through RANSAC+PnP. However this requires fitting the regression network on each scene independently and the underlying 3D point cloud representation lacks dense texture information. The authors hence leverage 3DGS representations which jointly encodes scene geometry and texture by introducing a scene independent visual localization pipeline. It consists of GS-RelocNet which establishes pixel to 3D Gaussians correspondences (used with a PnP+RANSAC loop to estimate an initial pose) and a refinement network that regresses residual coordinate maps between the query image and the image rendered from the initial pose (used to update the initial pose by estimating the relative pose with PnP+RANSAC). Both modules use fusion mechanisms to share information between different modalities in their two branches. The resulting pipeline is evaluated on two real world standard relocalization datasets.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1) Conceptually, learning to regress point to 3D Gaussian correspondences is a good intermediate solution between feature matching and scene coordinate regression. GS-RelocNet combines two powerful backbones for image and point clouds through a feature fusion module which allows it to predict such correspondences with high accuracy. It is further supported by an ablation study.  \n2) 3DGS scene independent visual localization is a very relevant research direction and GS-RelocNet generalizes to unseen indoor scenes during training yielding very accurate pose estimation. \n3) The relocalization pipeline achieves state of the art results against rendering-based baselines and regression baselines on two standard relocalization datasets. \n4) The feedforward regression nature of the method makes the inference very efficient and allows the relocalization localization pipeline to almost run at real time. \n\n### Weaknesses\n1) Given that the scene-independent nature of the pipeline is advertised as the main contribution of the paper I find experiments around generalization to be too limited. Notably, the scene independent version is evaluated only in indoor environments which inherently facilitates generalization as room layouts and the semantic content of rooms are relatively similar between scenes. Furthermore, 7-Scenes is not a challenging dataset for visual localization given the small scale of the scenes and the dense distribution of training cameras.\n\n2) l.466 it is mentioned that for 7-Scenes, “ground truth poses were obtained using the KinectFusion system”. However, it seems like your pipeline (and some other methods in Tab. 1) is evaluated against the SfM pseudo ground truth poses from [1] while the rest of the methods in the table report results evaluated against the original Dslam pseudo ground truth poses. It is important that all methods within the table are evaluated against the same pseudo ground truth as results can significantly vary between both ground truths. Could you clarify which pseudo ground truth was used and if necessary evaluate your method against both ground truths.     \n\n[1] On the Limits of Pseudo Ground Truth in Visual Camera Re-Localisation. Eric Brachmann, Martin Humenberger, Carsten Rother, Torsten Sattler\n\n3) Matching based structure based methods ought to at least be mentioned in the introduction as, overall, they remain the visual localization approach with the best robustness and accuracy on real world scenes.    \n\n[2] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From Coarse to Fine: Robust Hierarchical Localization at Large Scale. In CVPR, 2019.\n\n[3] Martin Humenberger, Yohann Cabon, Nicolas Guerin, Julien Morat, Jérôme Revaud, Philippe Rerole, Noé Pion, César Roberto de Souza, Vincent Leroy and Gabriela Csurka. Robust Image Retrieval-based Visual Localization using Kapture.\n\n4) There is a total lack of information regarding the underlying 3DGS model. \nSimilarly there is no information regarding RANSAC hyperparameters (number of iterations, thresholds) which can heavily influence the runtime.\nFor reproducibility purposes, this information should be added in the supplementary material.\n\nQUESTIONS:\n1) Do you have additional results to evaluate the generalization capabilities of GS-RelocNet?\n\n2) Which underlying 3DGS model is used? Which hyperparameters in the 3DGS model? How are the point clouds used to initialize the 3DGS models obtained?\nThere are some illumination variations in Cambridge Landmark, is this explicitly handled somewhere in the pipeline?\n\n3) For a query image, could you break down the runtime between the different steps of your pipeline?\n\n4) How many descriptors Ni per image or what is the patch embedding size?\n\nAdditional references for 3DGS visual localization:  \nGennady Sidorov, Malik Mohrat, Ksenia Lebedeva, Ruslan Rakhimov, and Sergey Kolyubin. “GSplatLoc: Grounding Keypoint Descriptors into 3D Gaussian Splatting for Improved Visual Localization.”  \nPietrantoni, Maxime, Gabriela Csurka, and Torsten Sattler. \"Gaussian Splatting Feature Fields for (Privacy-Preserving) Visual Localization.\" In CVPR 2025  \nHongjia Zhai, Xiyu Zhang, Boming Zhao, Hai Li, Yijia He, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang. “SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Reality.” In TVCG 2025\n\nLIMITATIONS:\nLimitations not addressed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}