SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We conduct experiments on two public indoor benchmark datasets. The 7-Scenes [35] contains seven indoor scenes recorded by a KinectV1 camera, the data includes RGB-D images, poses, and ground truth 3D models. The 12-Scenes [39] contains twelve indoor scenes with RGB-D images and poses. The recorded environments of the 12-Scenes are significantly larger than those in the 7-Scenes. We use fewer posed images compared to baseline localization methods, and the specific number of training data for each scene is configured as shown in Table 1 and Table 2.
Table 1: The number of training data on the 7-Scenes dataset.
Chess	Fire	Heads	Office	Pumpkin	Kitchen	Stairs	All
Baseline	4000(100%)	2000(100%)	1000(100%)	6000(100%)	4000(100%)	7000(100%)	2000(100%)	26000(100%)
Our	260(6%)	280(14%)	240(24%)	300(5%)	320(8%)	350(5%)	340(17%)	2090(8%)
Table 2: The number of training data on the 12-Scenes dataset.
Kitchen1	Living1	Bed	Kitchen2	Living2	Luke	Gates362	Gates381	Lounge	Manolis	Office5a	Office5b	All
Baseline	744(100%)	1036(100%)	868(100%)	768(100%)	725(100%)	1370(100%)	3536(100%)	2950(100%)	925(100%)	1613(100%)	1001(100%)	1391(100%)	16927(100%)
Our	185(25%)	170(16%)	215(25%)	190(25%)	180(25%)	340(25%)	350(10%)	295(10%)	230(25%)	270(17%)	250(25%)	280(20%)	2955(17%)
Baseline. Baseline methods can be classified into two main categories based on whether or not using ground truth 3D labels. Specifically, we utilized the following methods, each with its unique characteristics and requirements. 1). Methods without ground truth 3D label supervision: DFNet [10] and Direct-PoseNet [11] is the absolute pose regression method based on NeRF; FeatLoc++ [2] is the absolute pose regression method using sparse features. Furthermore, featLoc++Au utilizes sparse 3D structure to enhance visual localization performance; TransPoseNet [33] achieve visual localization based on attention mechanism; 2). Methods with ground truth 3D label supervision: SA [5] is a recently proposed absolute pose regression method based on 3D structure. HACNet [21], DSAC* [8] and DSAC++ [7] are state-of-the-art visual localization methods based on coordinate regression. PixLoc [30] is an advanced structure-based visual localization method.
Implementation Details. Our approach utilizes Nerfacto [38], an accelerated variant of NeRF. Nerfacto was trained for 100,000 iterations, and all camera poses are only centralized, with no scale adjustment. As errors in the ground truth pose of 7-Scenes dataset impair the training quality of Nerfacto [38], we turn on the pose-optimization component of the Nerfacto for 7-Scenes dataset. We set the far plane to 8 meters for office5a and office5b in the 12-Scenes dataset and set the far plane to 6 meters for other scenes. We set the near plane to 0 meters for all scenes. The coordinate regression network is trained for 40 epochs with a learning rate of 0.001. We empirically set  $\tau = 200$  for the threshold of coordinate distance. In pose optimization, we use SuperPoint with SuperGlue and their official weights to detect correspondences, set  $\lambda = 0.5$ ,  $N_{1} = 100$  for IBVS iterations. For training time, the Nerfacto and coordinate regression network are trained on one NVIDIA RTX3090 GPU for about 2 days. Please see the supplementary material for more implementation details.
Table 3: Comparison of median position and orientation error  $\left( {m,{}^{ \circ  }}\right)$  on the 7-Scenes dataset.
Method	Chess	Fire	Heads	Office	Pumpkin	Kitchen	Stairs	Average
w/o ground truth 3D labe	PoseNet [19]	0.32/8.12	0.47/14.4	0.29/12.0	0.48/7.68	0.47/8.42	0.59/8.64	0.47/13.8	0.44/10.4
Direct-PoseNet[11]	0.10/3.52	0.27/8.66	0.17/13.1	0.16/5.96	0.19/3.85	0.22/5.13	0.32/10.61	0.20/7.26
TransPoseNet [33]	0.08/5.68	0.24/10.6	0.13/12.7	0.17/6.34	0.17/5.6	0.19/6.75	0.30/7.02	0.18/7.81
FeatLoc++Au [2]	0.07/3.66	0.17/5.95	0.10/7.57	0.16/5.20	0.11/3.86	0.20/6.43	0.16/8.57	0.14/5.89
DFNet[10]	0.05/1.88	0.17/6.45	0.06/3.63	0.08/2.48	0.10/2.78	0.22/5.45	0.16/3.29	0.12/3.71
Ours	0.03/1.18	0.03/1.17	0.02/1.12	0.04/1.39	0.08/2.36	0.07/2.21	0.08/2.07	0.05/1.64
w/ ground truth 3D label	SA[5]	0.08/2.17	0.21/6.14	0.13/7.93	0.11/2.65	0.14/3.34	0.12/2.75	0.29/6.88	0.15/4.55
DSAC++[7]	0.02/0.5	0.02/0.9	0.01/0.8	0.03/0.7	0.04/1.1	0.04/1.1	0.09/2.6	0.04/1.1
DSAC*[8]	0.02/1.10	0.02/1.24	0.01/1.82	0.03/1.15	0.04/1.34	0.04/1.68	0.03/1.16	0.03/1.36
HACNet[21]	0.02/0.7	0.02/0.9	0.01/0.9	0.03/0.8	0.04/1.0	0.04/1.2	0.03/0.8	0.03/0.9
PixLoc[30]	0.02/0.80	0.02/0.73	0.01/0.82	0.03/0.82	0.04/1.21	0.03/1.20	0.05/1.30	0.03/0.98
Comparisons. We report the median translation (cm) and rotation  $(^{\circ})$  errors [19] in each scene, which is a main evaluation metric for localization accuracy. The localization accuracy is reported in Table 3 for 7-Scenes and Table 4 for 12-Scenes. As can be observed, our method consistently achieves the best performance among visual localization methods without ground truth 3D label
Table 4: Comparison of median position and orientation error  $\left( {m,{}^{ \circ  }}\right)$  on the 12-Scenes dataset.
w/o ground truth 3D label	w/ ground truth 3D label
Scene	PoseNet[19]	FeatLoc++[2]	FeatLoc++Au[2]	Ours	SA[5]	HACNet[21]
Kitchen1	0.29/15.48	0.53/14.0	0.32/5.19	0.01/0.60	0.08/4.45	0.01/0.4
Living1	0.29/15.31	0.55/9.75	0.26/3.89	0.01/0.57	0.08/2.50	0.01/0.4
Bed	0.57/17.85	0.60/10.2	0.37/5.39	0.02/0.82	0.07/3.87	0.01/0.4
Kitchen2	0.21/18.18	1.21/29.1	0.73/6.37	0.02/0.79	0.08/2.96	0.01/0.3
Living2	0.31/23.58	0.73/11.4	0.40/5.71	0.03/1.07	0.09/3.13	0.01/0.4
Luke	0.35/20.07	0.59/11.4	0.33/4.85	0.03/1.20	0.12/4.82	0.01/0.5
Gates362	0.27/16.71	1.03/8.95	0.52/5.22	0.02/0.75	0.06/2.04	0.01/0.4
Gates381	0.37/20.52	0.73/12.9	0.42/6.23	0.02/0.82	0.12/5.02	0.01/0.6
Lounge	0.29/18.42	0.97/8.30	0.39/4.50	0.02/0.83	0.07/2.53	0.01/0.5
Manolis	0.22/17.45	0.66/11.9	0.30/4.67	0.01/0.62	0.09/3.52	0.01/0.5
Office5a	0.57/14.55	0.61/8.05	0.31/4.32	0.06/2.63	0.10/5.08	0.01/0.5
Office5b	0.47/15.49	0.51/7.29	0.23/4.14	0.02/0.74	0.09/2.57	0.02/0.5
Average	0.35/17.80	0.73/11.9	0.38/5.04	0.02/0.95	0.09/3.54	0.01/0.5
supervision. Meanwhile, our method has comparable performance with methods using ground truth 3D labels as supervision, which closes the gap between the two types of visual localization methods. It is important to note that we use fewer posed images for training compared to the baseline methods, which shows the high value in practical application. Compared with the methods using NeRF for data augmentation, our method (0.05m,  $1.64^{\circ}$ ) outperforms DFNet [10] (0.12m,  $3.71^{\circ}$ ) and Direct-PoseNet [11] (0.20m,  $7.26^{\circ}$ ), thus providing a more efficient way to take advantage of the scenes prior provided by NeRF. Moreover, our method outperforms attention-based MS-Transformer [27] and FeatLoc++Au [2], which utilize sparse 3D structure augmentation, highlighting the potential of NeRF in visual localization.
Figure 3: Qualitative results on 12-Scenes dataset. (a) the correspondences between images rendered using coarse poses and query images. (b) the pose optimization trajectories in 3D space and image plane. (c) the correspondences between images rendered using optimized poses and query images.
Qualitative results. We also qualitatively analyze the NeRF-IBVS on 12-Scenes dataset. Our coarse pose estimation can obtain the poses near the target as shown in Figure 3(a), which provides a good starting point for pose optimization. The pose optimization trajectories in both 3D space and image plane show that the optimized pose can accurately reach the desired pose as shown in Figure 3(b). Figure 3(c) shows that the images rendered using the optimized poses are accurately aligned with the
query images. Figure 3(b)-(c) demonstrate that our method can achieve accurate localization. Please see the supplementary material for more qualitative results.
5.2 Simulation for Navigation
To verify the effectiveness of our method in enabling IBVS-based navigation without using custom markers and the depth sensor, we randomly selected a pair of images from the real 12-Scenes dataset with co-views to serve as the initial state and the desired state for navigation. We first establish the correspondences between the initial state and the desired state as shown in Figure 4(a). Appropriate 2D and 3D correspondences are selected as navigation prior as shown in Figure 4(b)-(c). A 6-degree-of-freedom navigation simulation is then performed using the camera model. Figure 4(d) shows that the navigation trajectory in both 3D space and image plane can accurately reach the desired state. Figure 4(e) shows that the coordinate error of correspondences can converge. The coordinate error of correspondences can reflect errors in the pose. The simulation results in Figure 4(d)-(e) demonstrate that our IBVS-based navigation without using custom markers and the depth sensor can achieve accurate navigation. Please see the supplementary material for more simulation results.
Figure 4: Simulation navigation results on 12-Scenes dataset. (a) candidate correspondences are established between initial states and desired states. (b) and (c) the appropriate 2D and 3D correspondences are selected by NeRF-IBVS. (d) navigation trajectory in both 3D space and image plane. (e)  $e(x_{i}), e(y_{i})$  is the image coordinate error of correspondences.
In this section, we conduct ablation experiments on the 12-Scenes dataset to evaluate the effectiveness of the major design of the proposed NeRF-IBVS.
Effectiveness of coarse pose estimation. To guarantee excellent performance for subsequent pose optimization, coarse pose estimation is required to output coarse poses and coarse 3D coordinates near the target. Therefore, we evaluate the localization performance of coarse pose estimation on the 12-Scenes dataset. As shown in Table 6, our coarse pose estimation obtains poses and 3D coordinates near the target (0.27m,  $13.10^{\circ}$ ), providing a good starting point for pose optimization.
We consider that the coarse poses and coarse 3D coordinates near the target when the position and orientation error of the coarse pose reduces after pose optimization. To further verify the effectiveness of coarse pose estimation, we count how often this happens. The specific quantitative results on the 12-Scenes dataset as shown in Table 5, the average frequency of coarse pose error reduction is greater than  $95\%$ . Therefore, the coarse pose estimation consistently outputs results near the target.
Effectiveness of pose optimization. To verify the effectiveness of pose optimization, we compare the overall NeRF-IBVS with a coarse pose estimation pipeline on 12-Scenes dataset. Localization result in Table 6 shows that pose optimization  $(0.02\mathrm{m}, 0.95^{\circ})$  can significantly improve the localization performance compared to coarse pose estimation  $(0.27\mathrm{m}, 13.10^{\circ})$ .
Table 5: Frequency of the coarse pose reduction after pose optimization on the 12-Scenes dataset.
12Scenes	Kitchen1	Living1	Bed	Kitchen2	Living2	Luke	Gates362	Gates381	Lounge	Manolis	Office5a	Office5b	Average
Frequency	96.88%	99.59%	94.61%	96.67%	95.42%	97.76%	100.00%	95.16%	99.69%	90.46%	90.14%	91.60%	95.66%
Table 6: Average of median position and orientation error  $\left( {m,{}^{ \circ  }}\right)$  on the 12-Scenes dataset.
coarse pose estimation	NeRF-IBVS w/0 correspondence selection	NeRF-IBVS w/ correspondence selection
12-Scenes	0.27/13.10	0.09/3.73	0.02/0.95
Effectiveness of correspondence selection. To verify the effectiveness of correspondence selection, we evaluate the localization performance of NeRF-IBVS without correspondence selection. The results presented in Table 6 show that the NeRF-IBVS without correspondence selection downgrade localization performance  $(0.09\mathrm{m}, 3.73^{\circ})$  compared to NeRF-IBVS with correspondence selection  $(0.02\mathrm{m}, 0.95^{\circ})$ . It demonstrates that correspondence selection can select the appropriate correspondences for NeRF-IBVS and enhance the robustness of NeRF-IBVS to noise.
We also qualitatively analyze the effectiveness of correspondence selection. Coordinate error and depth error of correspondences are generally prone to occur in areas with poor rendering quality. As shown in Figure 5, the correspondence selection tends to choose correspondences that occur in areas with good rendering quality, which can enhance the robustness of pose estimation.
Figure 5: Qualitative comparison on correspondence selection. (a) the correspondences are randomly selected. The dashed red circles indicate the correspondences in the areas with poor rendering quality. (b) the rendered depth is used to indicate depth error of correspondences. (c) the correspondences are selected using correspondence selection algorithm.
We propose a novel localization framework that can achieve precise localization performance using only a few posed images compared to other localization methods. We first use a few posed images with coarse 3D labels provided by NeRF to train a coordinate regression network. The coarse pose of the query image can be estimated by a coordinate regression network with the PNP algorithm. Then, we optimize the coarse pose using IBVS based on NeRF representation to achieve accurate localization performance. Further, our method can provide effective navigation prior, which can enable IBVS-based navigation without using custom markers and the depth sensor. This advantage greatly expands the application scope of IBVS-based navigation. To the best of our knowledge, NeRF-IBVS is the first method that can accomplish both visual localization and navigation based on IBVS, providing a novel idea for pose optimization by controlling camera motion.
Limitations. The quality of NeRF rendering is crucial for our method. Due to the poor rendering quality of NeRF in large outdoor scenes compared to indoor scenes, our method is mainly applied in indoor scenes. Moreover, due to the slow rendering speed of NeRF, our method cannot achieve real-time performance. The proposed method also struggles with non-Lambertian and dynamic objects since the NeRF performs poorly in these scenarios. In the future, the improved NeRF variants can be used to improve the performance of our method.
This work is supported by the National Natural Science Foundation of China (No. 91948303), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), and CCF-Alibaba Innovative Research Fund For Young Scholars.
[1] Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEEE Robotics Autom. Lett., 7(2):4606-4613, 2022.
[2] Thuan Bui Bach, Tuan Tran Dinh, and Joo-Ho Lee. FeatLoc: Absolute pose regressor for indoor 2D sparse features with simplistic view synthesizing. ISPRS Journal of Photogrammetry and Remote Sensing., 189:50-62, 2022.
[3] Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced feature points: Optimizing feature detection and description for a high-level task. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947-4956, 2020.
[4] Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced feature points: Optimizing feature detection and description for a high-level task. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4947-4956, 2020.
[5] Hunter Blanton, Scott Workman, and Nathan Jacobs. A structure-aware method for direct pose estimation. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 205-214, 2022.
[6] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dsac — differentiable ransac for camera localization. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 2492–2500, 2017.
[7] Eric Brachmann and Carsten Rother. Learning less is more - 6d camera localization via 3d surface regression. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 4654-4662, 2018.
[8] Eric Brachmann and Carsten Rother. Visual camera re-localization from RGB and RGB-D images using DSAC. IEEE Trans. Pattern Anal. Mach. Intell., 44(9):5847-5865, 2022.
[9] Francois Chaumette and Seth Hutchinson. Visual servo control. i. basic approaches. IEEE Robotics Automation Magazine., 13(4):82-90, 2006.
[10] Shuai Chen, Xinghui Li, Zirui Wang, and Victor Adrian Prisacariu. Dfnet: Enhance absolute pose regression with direct feature matching. In European Conference on Computer Vision, pages 1-17, 2022.
[11] Shuai Chen, Zirui Wang, and Victor Prisacariu. Direct-posedet: Absolute pose regression with photometric consistency. In International Conference on 3D Vision, pages 1175–1185, 2021.
[12] Ondrej Chum and Jiri Matas. Optimal randomized RANSAC. IEEE Trans. Pattern Anal. Mach. Intell., 30(8):1472-1482, 2008.
[13] Peter I Corke, Witold Jachimczyk, and Remo Pillat. Robotics, vision and control: fundamental algorithms in MATLAB, volume 73. Springer, 2011.
[14] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 224-236, 2018.
[15] Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi-object dynamics with compositional neural radiance fields. In Conference on Robot Learning, pages 1755-1768, 2022.