NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation
Authors: Yuanze Wang
Yichao Yan
Dianxi Shi
Wenhan Zhu
Jianqiang Xia
Tan Jeff
Songchang Jin
KE GAO
XIAOBO LI
Xiaokang Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: NeRF
Image-Based Visual Servoing (IBVS)
visual localization
visual navigation
TL;DR: NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation.

Abstract: Visual localization is a fundamental task in computer vision and robotics. Training existing visual localization methods requires a large number of posed images to generalize to novel views, while state-of-the-art methods generally require dense ground truth 3D labels for supervision. However, acquiring a  large number of posed images and dense 3D labels in the real world is challenging and costly. In this paper, we present a novel visual localization method that achieves accurate localization while using only a few posed images compared to other localization methods. To achieve this, we first use a few posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. Then a coarse pose is estimated from the regression network with PNP. Finally, we use the image-based visual servo (IBVS) with the scene prior provided by NeRF for pose optimization. Furthermore, our method can provide effective navigation prior, which enable navigation based on IBVS without using custom markers and depth sensor. Extensive experiments on 7-Scenes  and 12-Scenes datasets demonstrate that our method outperforms state-of-the-art methods under the same setting, with only 5\% to 25\% training data.  Furthermore, our framework can be naturally extended to the visual navigation task based on IBVS, and its effectiveness is verified in simulation experiments.

NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation
Yuanze Wang Yichao Yan'* 1 Dianxi Shi Wenhan Zhu' Jianqiang Xia
Tan Jeff Songchang Jin Ke Gao Xiaobo Li Xiaokang Yang'
MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University Intelligent Game and Decision Lab (IGDL., Beijing, China TTinjin Artificial Intelligence Innovation Center AAlibaba Group
[yz. wang, laang.ynniiie panyichao, zhuwenhan823, xsyang) tu edu. en
dxshi@nudt. edu. en, jianqiiaanan ang xia@f com, jwcosssssssssii innpttsiinnuuc.om org (xiaobo. lixb, gaoke. .go) gallbbaabaa com
Abstract
Visual localization is a fundamental task in computer vision and robotics. Training existing visual localization methods requires a large number of posed images generalize to novel views, while state-of-the-art methods generally require ground truth 3D labels for supervision. However, acquiring large number of posed images and 3D labels in the real world is challenging and costly. In this paper, we present novel visual localization method that achieves accurate localization while using only a few posed images compared to other localization methods. To achieve this, we first to train a coordinate regression network. Then a coarse pose is estimated from the regression network with PNP. Finally, we use the image-based visual servo (IBVS) with the scene prior provided by NeRF for pose optimization. Furthermore, our method can provide effective navigation prior, which enables navigation based on IBVS without using custom markers and the depth sensor. Extensive experiments on 7-Scenes and 12-Scenes datasets demonstrate that our method outperforms state-of-the the--rh-rttt art methods under the same setting, with only 5% to 25% training data. Furthermore, our framework can be naturally extended to the visual navigation task based on IBVS, and its effectiveness is verified in simulation experiments.
few posed images with coarse pseudo-3D labels provided by NeRF
Introduction
The estimation of camera positions and orientation based query image is fundamental task in computer vision. It has wide applications in various fields such as robotics, virtual reality, and autonomous driving. Existing works that tackle this task be generally divided into two categories. The first line of approaches are structure-based 6 28 31 37 (66,, which first establish correspondences between 3D points and 2D pixels, and then compute camera pose with PnP (16] solver utilizing RANSAC [12]. These methods typically achieve satisfactory results at the cost of a complex pipeline. The other line of approaches regression- based 2 B 40
42 41), which either regress the 3D coordinates or directly regress the absolute pose. These methods yield a more compact framework and have fast inference performance. All the above methods require substantial number of posed images (generally thousands of posed images per scene) to guarantee the generalization ability to novel viewpoints. Moreover, state-or oftee-of-te-arr art methods 66 4 21 also require dense ground truth 3D labels for supervision, such as depth and 3D model. However, acquiring large number
2
'These authors contributed equally to this work.
i "ooresponniig authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

NeRF
Query Image
Visual Localization
NeRF-IBVS
Visual Navigation
Desired State
Initial State
Figure 1: Demonstration of the proposed NeRF-IBVS. Our method can achieve visual localization based on single query image. Moreover, our method can obtain navigation prior after processing the images of the initial state and desired state, which can better enable IBVS-basee navigation.
of posed images and dense 3D labels is challenging and costly. naturally raises a question: possible to design a framework that achieves accurate localization results while only requiring small number of posed images?
Recent advances in Neural Radiance Fields (NeRF) 25 provide a promising solution. Trained from hundreds of posed images, NeRF can render realistic images and give coarse depth maps for novel viewpoints as prior knowledge of 3D scenes. Intuitively, this prior knowledge can assist visual localization methods in generalizing to novel viewpoints. Current NeRF-based visual localization methods [10 11 [6] only employ NeRF for data augmentation purposes, which still requires substantial number of images for training. The estimated pose will be coarse when using a few data. Considering this issue, it would be desirable if the coarse pose in NeRF could be further refined to meet the need for visual localization.
In this paper, we propose a novel visual localization framework called NeRF-IBVS, which achieves accurate localization while using only few posed images. NeRF-IBVS adopts coarse-to-fine paradigm to estimate the pose of the query image. In the coarse stage, we first train a NeRF with hundreds of posed images, which is significantly fewer than typical visual localization methods. Then, we use the same posed images with coarse pscudo-3D labels provided by NeRF to train coordinate regression network. The coarse pose is estimated from the regression network with PNP. In the refining stage, we design an optimization algorithm to effectively optimize the coarse pose. Motivated by the image-based visual servo (IBVS) [9) which navigates the camera from the initial pose the desired pose using coordinate eтror and depth of correspondences between the current image and the target image. Therefore, we first use NeRF to render the image and depth corresponding to current coarse poses. Then, we establish the correspondences between the rendered image and the query image and query the approximate depth of the correspondences based on the rendering depth map. Finally, the correspondences with depth are used launch IBVS to guide pose optimization.
Due rendering quality errors, directly applying IBVS NeRF-rendered images and depth is not feasible. Moreover, IBVS requires non-collinear correspondences during d-degree-of-freeee freedom navigation. In order to achieve robust pose optimization, we design correspondence selection algorithm using spatial information provided by NeRF and the coordinate regression network. To speed up the pose optimization, we further design a strategy that reduces the rendering frequency of NeRF. Further, the effective 2D and 3D correspondences obtained by NeRF-IBVS can be used as the navigation prior to enhance IB V-bbadd navigation. Specifically, this navigation prior can enable IBVS -VS-based navigation without using custom markers and the depth sensor. This advantage greatly expands the application scope.
To the best of our knowledge, NeRF-IBVS is the first method that can accomplish both visual localization and visual navigation based on IBVS as shown in Figure L providing a novel idea for pose optimization by controlling camera motion. Employing only 5% to 25% of the posed images required by other localization methods on 7-Scenes dataset and 12-Scenes dataset, our method outperforms state-of-the-as methods without using ground truth 3D labels and has comparable performance with methods using ground truth 3D labels. Furthermore, our method can enable visual navigation based on IBVS without using custom markers and the depth sensor, which greatly expands the application scope of IB basee navigation. We also verified the effectiveness of visual navigation in simulation environment.
2

---

2 Related Work
Visual localization.
46 use complex pipelines to achieve precise localization. Firstly, correspondence is established between 3D coordinates and 2D pixels. Secondly, a camera pose is computed through a PnP solver with RANSAC. The first stage requires complex pipeline consisting of image retrieval, feature extraction, and matching. To achieve a more compact frame compared to structure-
Structure-based methods [6 28 37
methods. In recent years, regression based localization methods 3 40 [2] 42) [4] (24] attract increasing attention, which either regresses the 3D coordinates or directly regress the absolute pose. Specifically, coordinate regression methods, such as DSAC++ 77] and HACNet [21] neural networks to learn the first stage of the muuutuue-bbsed methods, which directly regress 3D scene coordinates from an image. These coordinates regression methods typically use ground truth 3D labels supervision to achieve better localization performance. Unlike coordinate regression, absolute pose regression methods 19 40 generally use only posed images train the neural network. MS-Transformer (32) and AtLoc 41 use attention mechanisms to guide the regression process and have substantially improved the performance of absolute pose regression methods. Recently, some methods (TO, [ 26 use NeRF as data enhancement to improve the performance of visual localization methods. The above methods require a substantial number of posed images to guarantee the generalization ability novel viewpoints. Our method can achieve accurate localization while using only a few posed images compared to other localization methods.
Robotics with neural radiance fields. Recently, the neural radiance field (NeRF) 25 has shown great potential in robotics applications 22] 15). iNerf 44 fixes the network weights of NeRF and directly optimizes the camera pose parameters by computing the backpropagation of the photometric loss. However, this method requires an initial pose near the ground truth pose. Methods based on photometric loss tend fail when blur and artifacts appear in the rendered images. iMAP (36) and NICE-SLAM 4 are a pose estimator in the SLAM setting that requires continuous sequential data.
BARF 23 optimize camera poses during the training of NeRF. Therefore they cannot estimate the pose of single query image. Moreover, NeRF can provide coarse scene geometry, which is useful for navigation tasks. NeRF-nav [ designs a smooth and collision-proo navigation strategy based on the density provided by NeRF. RNR-Map (20) achieves precise visual navigation performance based on NeRF, but is limited to three degrees of freedom and requires RGBD pictures and odometer information. Our method can accomplish both visual localization and 6 degrees of freedom navigation based on visual servo and using only RGB images.
3 Preliminaries
NeRF. NeRF uses MLP network Fe to encode the 3D scene, which predicts the RGB color e E R33 and volume density R for each input 3D point p E IR and its view direction d E R3, It can be summarized as (p, d) = (c, o), where is the parameters of MLP network. Then, NeRF uses volume rendering techniques to generate realistic rendered images C(r) and coarse depth maps ((r) from any viewpoint:
M C(r) - Σ T1(1-ep((-0,0) D(r) = Σ Ti(1 exp(-o;ū,))t 111
M
1=1
where and M indicates the rendering ray and the number of points along the ray, G and & indicates the color and distance of the samples in the rendering ray, T; (- ii (-E o5 indicates the
accumulated transmittance from the near bound of the rendering ray to 4, and 8, = 4,+1 4, is the distance between adjacent samples in rendering ray.
IBVS. The aim of Image- image-Based Visual Servoing (IBVS) is to control the camera to move toward the desired pose based on vision information while minimizing the coordinate error of image features between the current image and the target image. Correspondences are used as image features in this paper. To achieve this goal, IBVS first calculates the desired image coordinate velocity of correspondences based on the coordinate error. Then, the Jacobi matrix is constructed based on the correspondences, which establishes the relationship between the image coordinate velocity and the camera velocity. Finally, the desired camera velocity to control the camera motion towards the target solved based on the Jacobi matrix and the desired image coordinate velocity. The specific details of the VS are presented in the supplemental materials.
3

---

Coarse Pose Estimation
NeRt
Nermalized
Hundreds Posed Images Per
Rendered Depih
Pseudb-3D
'f
Coarse Pose
Course 3D Labels
Inforence Training
Query Image
Coordinate regression
Pose Optimization
Navigation Prior
NeRF
Query Image
2D
Correspondenes
IBVS Medakwill Correppnniinn Schecun
Navigation Trajecteries
HAVS Modale
Conespondences
3D Correspondenes
Figure 2: The pipeline of method, NeRF-IBVS. In the coarse pose estimation, first train NeRF with hundreds of posed images, which is significantly fewer than typical visual localization methods. Then, use the same posed images with coarse pscudo-3D labels provided by NeRF to train a coordinate regression network. The coarse pose is estimated from the regression network with PNP. In the pose optimization, the correspondences with depth between the rendered image and the target image are obtained to launch IBVS to guide pose optimization. Moreover, our method can provide effective navigation prior, which enhances navigation based on IBVS.
4 Method
Given a few posed images compared to the typical localization method, we aim to achieve accurate localization. To achieve this, first few posed images with pseudo-3D labels provided by NeRF to train coordinate regression network. Then the coarse pose is estimated from the regression network with PNP. Finally, we introduce IBVS gradually refine the coarse pose by leveraging the 3D scene prior provided by NeRF. Furthermore, our method can provide effective navigation prior, which enhances navigation based on IBVS. Figure demonstrates the overall pipeline.
4.1 Coarse Pose Estimation
We use few posed images with pseudo-31) labels provided by pre-training NeRF to train coordinate regression network. Then the coarse pose is estimated from the regression network with PNP. Specifically, we convert the pixel coordinates of training images into pseudo-3D labels Pw using depth D provided by NeRF:
Pw = DT-
where denote internal parameter and pose of camera. Following common settings 43 these pseudo-3D labels Pw are then normalized to fit within the range of unit cube. Then, we use the advanced coordinate regression component GDR-Net 42 as our coordinate regression network, which is trained by pseudo-3D labels P To estimate coarse pose, the coordinate regression network

---

processes the query image to generate 3D coordinates and establishes a 2D-3D correspondence. This 2D-3D correspondence is then fed into PnP [16] to determine the coarse pose T of the query image. We empirically found that the lack of multi-view constraints in the boundary regions of the scene leads to significant depth error in image edges. To mitigate the impact of the rendering quality error, we exclude the edge areas the image of our pseudo-3D labels P w when training our coordinate regression network.
4.2 Pose Optimization
To optimize the coarse poses, our method views pose optimization as the process of navigating the camera from the coarse pose towards the target pose using scenes prior provided by NeRF.
4.2.1 IBVS Module
IBVS initialization using NeRF, To start pose optimization, first initialize IBVS using NeRF. This process mainly initializes the Jacobian matrix L of IBVS using the image coordinates and depth of correspondences, which establishes the relationship between the velocity v. of the camera and the velocity s of image coordinates:
Lvc.
where the L corresponds to per correspondence can be formulated as:
Z.
ry
(1 x²) V
1 y
zy
where x, and Ze denote image coordinates and depth of the correspondence in the currently rendered image. To initialize the Jacobian matrix L, we first utilize NeRF to render an image C, and depth map D, that corresponds to the current coarse pose Í. Then the off-the-shelff correspondences establishment algorithm, such as SuperPoint [14] with SuperGlue [29], is employed extract the pixel coordinate (ma; m,) of correspondences between the query image C. and the rendered image Cr. The depth Z. of correspondences m, can be obtained by querying the rendered depth D,. Image coordinates n, = (x. y) in the rendered image can be obtained using the pixel coordinates m, and the internal parameter K follow (9). Finally, n. with depth Z. used to initialize L.
Optimization iteration. After the IBVS initialization, we started IBVS iteration to navigate the camera from the coarse pose toward the target pose. At each iteration, IBVS needs to calculate the velocity V. of the camera update the pose. To control the DOF velocity v. of the camera, four non-collinear correspondences are selected and the velocity á of the image coordinates are set to ->(nq - n2)) follow [9. According to Equation
3 we can obtain the camera velocity Ve as:
Ve XL (nq nz);
where X, Lt and ng denote proportional factor, generalized inverse matrix, and image coordinates in the query image. Then, we update the current pose by integrating the camera velocity ve during unit iteration to obtain the variation of pose (T)v.(t)). The coarse poses are updated as follows:
T(t + 1) = (tv
The IBVS iteration proceeds until either the pixel coordinate error of correspondences is less than a certain threshold or the maximum number of iterations is reached.
Theoretically, each iteration of IBVS requires NeRF rendering to update the parameters of the Jacobian matrix L. However, utilizing NeRF rendering for each IBVS iteration is highly time- consuming and impractical. Therefore, we design a new fast iteration strategy that reduces the rendering frequency of NeRF. We first calculate the 3D coordinates of correspondences using the depth information provided by NeRF, while assuming the 3D coordinates of the correspondences are precise. At each iteration, We project the 3D coordinates of correspondences to the image plane by using the updated poses. This projection can be an approximate substitute for the correspondence in the rendered image. In this way, skip the time-consuming NeRF rendering process, and NeRF rendering is only executed at the initialization IBVS.
Due to rendering errors of NeRF, the assumption that the 3D coordinates of correspondences are accurate is violated, so accumulation errors will occur. Accumulated errors may lead to the optimiza- tion terminating with large error, In order to eliminate accumulated errors, render the image
5

---

again based on the most recently updated poses to relaunch IBVS iterations. Typically, the IBVS iterations are launched no more than four times obtain a precise localization.
The first IBVS iterations launch aims to move the camera a significant distance toward the target pose, while the subsequent IBVS iterations launch only fine-tune the pose around the target pose. To improve the optimization efficiency, We dynamically set the number of iteration steps. Specifically, the number N (i) of iterations steps in the ith IBVS iterations launch is set dynamically based on coordinate error of correspondences:
N1. w(t)>1 Ni w(i){}
where wii) || [,,
Dg 12
(w(())
ng
wi otherwise
where N1 and n. , na denote the fixed number of iteration steps in the first IBVS iteration launch and image coordinates of correspondences in the ith IBVS iterations launch, respectively.
4.2.2 Correspondence Selection
The Jacobian matrix L of IBVS is highly sensitive to the selection of the correspondences, and requires accurate and non-collinear correspondences to update parameters during iteration. Due to rendering errors of NeRF, which may cause inaccurate correspondences. In order to increase the robustness of IBVS, we designed correspondence selection algorithm to select appropriate corre- spondences for IBVS. We first query coarse 3D coordinates estimated by the coordinate regression network to obtain 3D coordinate Pq of correspondences mq in the query image. Then, we use coarse pose T to project the 3D coordinate P. q onto the rendered image to obtain image coordinates âr. Finally, we can obtain the coordinate distances of correspondences d = l| ny Since the coarse pose T and 3D coordinates Pq near the target, the accurate correspondences generally have a smaller distance. Therefore, we empirically set threshold 7 for the coordinate distance d to filter out large number of outliers. Finally, we use RANSAC [12 to further select accurate and non-collinear correspondences. Please see the supplementary material for more details.
4.3 Visual Navigation
NeRF-IBVS can provide navigation prior which enhances IBVS-based navigation. To enable IBVS navigation, it is necessary to use correspondences that are non-collinear and remain in the camera's field of view during navigation. However, acquiring such correspondences is challenging. current usually rely custom markers which limit the applications of IBVS. The navigation prior provided by our method can solve this problem. Specifically, We first use NeRF- IBVS to locate the initial pose of the navigation task. Then the initial pose is viewed to start the NeRF-IBVS locating the target state of the navigation task. After that, the correspondence selection algorithm can obtain accurate and non-collinear correspondences and the IBVS module can obtain navigation trajectories in the image plane. The current correspondences are not yet guaranteed to remain in the camera's field of view during navigation. Benefiting from the navigation trajectory, the appropriate 2D correspondences that fall outside of the image plane are automatically filtered out. It ensures that the correspondences selected remain in the camera's 's field of view during navigation. We backproject 21) correspondences based on depth provided by NeRF to obtain coarse 3D coordinates as 3D correspondences. During the navigation iteration, the 3D correspondences are projected using the current pose to obtain the depth prior and the image coordinates prior. The image coordinates prior can assist in locating and associating the 2D correspondences in the current image, which enables IBVS- based navigation without custom markers. Moreover, the depth prior can be used update the parameters of the Jacobian matrix in each iteration, which enables -eVe-b navigation without using a depth sensor. Although there is small eтror in the depth obtained by projection, IBVS has the advantage of being robust to small depth errors
methods 18 45 34]
coarse pose
Experiments
In this section, we first compare the proposed NeRF-IBVS with state-of-the-an visual localization methods. Furthermore, our method extended to -BVS-based navigation without using custom markers and the depth sensor, and its effectiveness is verified in simulation experiments.

---

5.1 Visual Localization
Datasets, We conduct experiments on two public indoor benchmark datasets. The 7-Scenes contains seven indoor scenes recorded by 1 camera, the data includes RGB-D images, poses, and ground truth 3D models. The 12-Scenes 39 contains twelve indoor scenes with RGB-D images and poses. The recorded environments of the 12-Scenes are significantly larger than those in the 7-Scenes. We use fewer posed images compared to baseline localization methods, and the specific number of training data for each scene is configured as shown in Table Mableeaan and Table 2
35
Table 1: The number of training data on the 7-Scenes dataset.
Chess
Fire
Heads
Office
Pumpkin
Kitchen
Stairs
All
Baseline
4000(100%)
2000
100%)
6000 100%)
4000 100%) 320(8%)
7000 100%%) 350(5%)
2000X 100%)
26000 100%%
Our
260(6%)
280(14%) )
240124%)
30055%)
340(7%)
209018%)
Table 2: The number of training data on the 12-Scenes dataset.
Kidhel
Litteg
Krichez
Liveag
Labe
Ga
Leunge
Manote
Offcctt
Oftecto
All
Beadiac
1441 (009))
(62001009)
7251 300000
(S0R6)
2333 SEOPG)
(13185096)
2003))
L391 (0000)))))
(9922113006)
Baseline. Baseline methods can be classified into two main categories based on whether or not using ground truth 3D labels. Specifically, we utilized the following methods, each with its unique characteristics and requirements. 1). Methods without ground truth 3D label supervision: DFNet [10] and Direct-PoseNV [1]] is the absolute pose regression method based on NeRF; FeatLoc++ 2 is the absolute pose regression method using sparse features. Furthermore, fcatLoc++Au utilizes sparse 3D structure to enhance visual localization performance; TransPoseNet [33 achieve visual localization based on attention mechanism; 2). Methods with ground truth 3D label supervision: SA [5] is a recently proposed absolute pose regression method based on 3D structure. HACNet (21), DSAC* [ and +SAC++ 7 are state-of-thh--ee visual localization methods based on coordinate regression. PixLoc 30 is an advanced structure-based visual localization method.
Implementation Details. Our approach utilizes Nerfacto 38 an accelerated variant of NeRF Nerfacto was trained for 100,000 iterations, and all camera poses are only centralized, with no scale adjustment. As errors in the ground truth pose of 7-Scenes dataset impair the training quality of Nerfacto (38), we turn on the pose-optimization component of the Nerfacto for 7 7-Scenes dataset. We set the far plane to meters for office5a and office5b in the 12-Scenes dataset and set the far plane to 6 meters for other scenes. We set the near plane to 0 meters for all scenes. The coordinate regression network is trained for 40 epochs with a learning rate of 0.001. We empirically set r = 200 for the use SuperPoint with SuperGlue and their
threshold of coordinate distance. In pose optimization, official weights to detect correspondences, set À = 0.5, Ni 100 for IBVS iterations. For training time, the Nerfacto and coordinate regression network are trained one NVIDIA RTX3090 GPU for about 2 days. Please see the supplementary material for more implementation details.
Table 3: Comparison of median position and orientation error (m,) 9) on the 7-Scenes dataset.
Method
Chess
Fire
Heads
Office
Pumpkin
Kitchen
Starrs
Average
PoseNet
19
0.32/8.12
0.47/14.4
0.29/12.0
0.48/7.68
0.47/8.42
0.59/8.64
0.47/13.8
0.44/10.4
Direct-PoseNe
0.10/3.52
0.27/8.66
0,17/13.1
0.16/5.96
0.19/3885
0.22/5.13
0.32/10.61
0.20/7.26
wlo
TransPoseNet
33
0.08/5.68
0.24/10.6
0.13/12.7
0.17/6.34
0.17/5.6
0.19/6.75
0.30/7.02
0.1877.81
ground truth
AcatLoc+=Au
0.07/3.66
0.17/5.95
0.1077.57
0.16/5.20
0,11/3.86
0.20/6.43
0.16/8.57
0.14/5.89
3D labe
D10
0.05/1.88
0.17/6.45
0.06/3.63
0.08/2.48
0.10/2.75 78
0.22/5.45
0.16/3.29
0.12/3.71
Ours
0.03/1.18
0.03/1.17
0.02/1.12
0,04/1.39
0.08/2.36
0,07/2.21
0.08/2.07
0.05/1.64
SA
0.08/2.17
0.21/6.14
0.1377.93
0.11/2.65
0,14/3.34
0.12/2.75
0.29/6.88
0.13/4.55
w/
D7A
0.02/0.5
0.02/0.9
0.01/188
0.03/0.7
0.04/1.1
0.04/1.1
0.09/2.6
0.04/1.1
3D label
ground
DSAC* X
0.02/1.10
0.02/1 24
0.01/1.82
0.03/1.15
0.04/1 34
0.04/1.68
0.03/1.16
0.03/1.36
HACNet 21
0.02/0.7
0.02/0.9
0.01/0.9
0.03/0.8
0.04/1.0
0.04/1.2
0.03/0.8
0.03/0.9
PixLod 30)
0.02/0.80
0.02/0.73
0.0100822
0.03/0.82
0.04/1.21
0.03/1.20
0.05/1.30
0.03/0988
Comparisons. We report the median translation (em) and rotation (*) errors 119 in each scene, which is a main evaluation metric for localization accuracy. The localization accuracy is reported in Table 3 for 7-Scenes and Table 4 for 12-Scenes. As can be observed, our method consistently achieves the best performance among visual localization methods without ground truth 3D label
7

---

Table 4: Comparison of median position and orientation error (m, ) on the 12-Scenes dataset.
w/o ground truth 3D label
w/ ground truth 3D label
Scene
PoseNet
FeatLoc++
FeatLoc++Au
Ours
SA
HACNet
Kitchen
0.29/15.48
0.53/14.0
0.32/5. 19
0.01/0.60
0.08/4.45
0.01/0.4
Livingl
0.29/15.31
0.35/9.75
0.26/3.89
0.01/0.57
0.08/2.50
0.01/0.4
Bed
0.57/17.85
0.60/10.2
0.37/5.39
0.02/0.82
0.07/3.87
0.01/0.4
Kitchen2
0.21/18.18
1.21/29.1
0.73/6.37
0.02/0.79
0.08/2.96
0.01/0.3
Living?
0.31/23.58
0.73/11.4
0.40/5.71
0.03/1.07
0.09/3.13
0.01/0.4
Luke
0.35/20.07
0.59/11.4
0.33/4.85
0.03/1.20
O. 0.12/4.82
0.01/0.5
Gates362
0.27/16.71
1.03/8.95
0.52/5.22
0.02/0.75
0.06/2.04
0.01/0.4
Gates381
0.37/20.52
0.73/12.9
0.42/6.23
0.02/0.82
0.12/5.02
0.01/0.6
Lounge
0.29/18.42
0.97/8.30
0.39/4.50
0.02/0.83
0.07/2.53
0.01/0.5
Manolis
0.22/17.45
0,66/11,9
0.30/4.67
0.01/0.62
0.09/3.52
0.01/0.5
Office5a
0.57/14.55
0.61/8.05
0.31/4.32
0.06/2.63
0.10/5.08
0.01/0.5
Officc5b
0.47/15.49
0.51/7.29
0,23/4.14
0.02/0.74
0.09/2.57
0.02/0.5
Average
0.35/17.80
0.73/11.9
0.38/5.04
0.02/0.95
0.09/3.54
0.01/0.5
supervision. Meanwhile, our method has comparable performance with methods using ground truth 3D labels supervision, which closes the gap between the types of visual localization methods. It is important to note that we use fewer posed images for training compared to the baseline methods, which shows the high value in practical application. Compared with the methods using NeRF for data augmentation, our method (0.05m, 1.64%) outperforms DFNet [IO] (0. 12m, 3,71 and Direct-PoseNet (II) (0.20m, 7.26%), thus providing more efficient way to take advantage of the scenes prior provided by NeRF. Moreover, our method outperforms ditention-basee MS- Transfommrrr (7] and FeatLoc++Au 2) which utilize sparse 3D structure augmentation, highlighting the potential of NeRF in visual localization.
Readered Image
uery Image
Pose Optimization Trajectory
Rendered Image
Query Image
2
Figure 3: Qualitative results on 12-Scenes dataset. (a) the correspondences between images rendered using coarse poses and query images. (b) the pose optimization trajectories in 3D space and image plane. (c) the correspondences between images rendered using optimized poses and query images.
Qualitative results. We also qualitatively analyze the NeRF-IBVS on 12-Scenes dataset. Our coarse pose estimation can obtain the poses near target as shown in Figure Ja), a), which provides good starting point for pose optimization. The optimization trajectories in both 3D space and image plane show that the optimized pose can accurately reach the desired pose as shown in Figure 3bb). Figure B(c) shows that the images rendered using the optimized poses accurately aligned with the

---

query images. Figure (c) (c) demonstrate that our method can achieve accurate localization. Please see the supplementary material for more qualitative results.
5.2 Simulation for Navigation
To verify the effectiveness of our method in enabling IB LBVS-b navigation without using custom markers and the depth sensor, we randomly selected a pair of images from the real 12-Scenes dataset with co-views to serve as the initial state and the desired state for navigation. We first establish the correspondences between the initial state and the desired state as shown Figure a). Appropriate 2D and 3D correspondences are selected as navigation prior shown in Figure 4 (c) A 6-degree- of of-freedom navigation simulation is then performed using the camera model. Figure ( shows that the navigation trajectory in both 3D space and image plane can accurately reach the desired state. Figure 4 e) shows that the coordinate error of correspondences can converge. The coordinate error of correspondences can reflect errors in the pose. The simulation results in Figure demonstrate that our IB VS-based navigation without using custom markers and the depth sensor can achieve accurate navigation. Please see the supplementary material for more simulation results.
Deired State
Initial State
Deired State
Initial State
NeRF-IBVS
hp)
601
Zexted post Inttll
d1d)
641
3P
( Correpaneeesss
(d) Navigation Trajectories
(c) Pixrl Coordinate Frrers
Figure 4: Simulation navigation results on 12-Scenes dataset. (a) candidate correspondences are established between initial states and desired states. (b) and (c) the appropriate 2D and 3D correspon- dences are selected by NeRF-IBVS. (d) navigation trajectory in both 3D space and image plane. (e) e(a:), e(ys) is the image coordinate error of correspondences.
5.3 Ablation Studies
In this section, we conduct ablation experiments on the 12-Scenes dataset to evaluate the effectiveness of the major design of the proposed NeRF-IBVS.
Effectiveness of coarse pose estimation. To guarantee excellent performance for subsequent pose optimization, coarse pose estimation is required to output coarse poses and coarse 3D coordinates near the target. Therefore, we evaluate the localization performance of coarse pose estimation on the 12-Scenes dataset. As shown in Table 6 our coarse pose estimation obtains poses and 3D coordinates near the target (0.27m, 13%) providing a good starting point for pose optimization.
We consider that the coarse poses and coarse 3D coordinates near the target when the position and orientation error of the coarse pose reduces after pose optimization. To further verify the effectiveness of coarse pose estimation, 12-Scenes dataset as shown in Table 5| the average frequency of coarse pose error reduction is greater than 95%. Therefore, the coarse pose estimation consistently outputs results near the target.
count how often this happens. The specific quantitative results on the
Effectiveness of pose optimization. To verify the effectiveness of pose optimization, we compare overall NIRF-IBVS with a coarse pose estimation pipeline on 12-Scenes dataset. Localization result in Table 6 shows that pose optimization (0.02m, 0.95%) can significantly improve the localization performance compared to coarse pose estimation (0.27m, 13.10").

---

Table 5: Frequeney of the coarse pose reduction after pose optimization on the 12-Scenes dataset.
12Scemes
Kitchenl
Livingl
Bed
Kitchen2
Living!
Luke
Gates362
Gotes381
Lounge
Manolis
OfficeSa
Office5b
Average
Prequency
96.88%
99.59%
94.61%
96.67%
95.42%
91,76%
100.00%
95.16%
99.59%
90.46%
90.14%
91.00%
95.66%
Table 6: Average of median position and orientation error (m, on the 12-Scenes dataset.
pose estimation
NcKF-IBVS
corrssooudence sclection
NcRF-IBVV w/ correspandence solection
12-Somes
0.27/13.10
Effectiveness of correspondence selection. To verify the effectiveness of correspondence selection, we evaluate the localization performance of NeRF-IBVS without correspondence selection. The results presented in Table show that the NeRF-IBVS without correspondence selection downgrade localization performance (0.09m, 3.73%) compared to NeRF-IBVS with correspondence selection (0.02m, 0.95%). ). It demonstrates that correspondence selection can select the appropriate correspon- dences for NeRF-IBVS and enhance the robustness of NeRF-IBVS to noise.
We also qualitatively analyze the effectiveness of correspondence selection. Coordinate error and depth error of correspondences are generally prone to occur in areas with poor rendering quality. As shown in Figure 5 the correspondence selection tends to choose correspondences that occur in areas with good rendering quality, which can enhance the robustness of pose estimation.
Query Image
Rendered Image
Rendered Depth
Query Image
Rendered Image
(a)
(c)
Figure 5: Qualitative comparison on correspondence selection. (a) the correspondences are randomly selected. The dashed red circles indicate the correspondences in the areas with poor rendering quality. (b) the rendered depth is used to indicate depth error of correspondences. (c) the correspondences are selected using correspondence selection algorithm.
6 Conclusion
We propose a novel localization framework that can achieve precise localization performance using only a few posed images compared to other localization methods. We first use few posed images with coarse 3D labels provided by NeRF train a coordinate regression network. The coarse pose of the query image can be estimated by coordinate regression network with the PNP algorithm. Then, we optimize the coarse pose using IBVS based on NeRF representation to achieve accurate localization performance. Further, our method can provide effective navigation prior, which can enable VB navigation without using custom markers and the depth sensor. This advantage greatly expands the application scope of -VS-baeed navigation. To the best of our knowledge, NeRF-IBVS is the first method that can accomplish both visual localization and navigation based on IBVS, providing novel idea for pose optimization by controlling camera motion.
Limitations. The quality of NeRF rendering is crucial for our method. Due to the poor rendering quality of NeRF in large outdoor scenes compared to indoor scenes, our method is mainly applied indoor scenes. Moreover, due to the slow rendering speed of NeRF, our method cannot achieve real-time performance. The proposed method also struggles with non-Lambertian and dynamic objects since the NeRF performs poorly in these scenarios. In the future, the improved NeRF variants can be used to improve the performance of method.
10

---

Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. 91948303), Shanghai Municipal Science and Technology Major Project (2021SHZDZZX010)), and CCF-Alibaba Innovative Research Fund For Young Scholars.
References
[1] Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in neural radiance world. IEEE Robotics Autom. Lett., 7(2):4606 2022.
[2] Thuan Bui Bach, Tuan Tran Dinh, and Joo-Ho Lee. FeatLoc: Absolute pose regressor for indoor
2D sparse features with simplistic view synthesizing. ISPRS Journal of Photogrammetry and Remole Sensing.. 189:00-62, 2022.
[3] Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced feature points:
Optimizing feature detection and description for a high-level task. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4947-4956, 2020.
[4] Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced feature points:
Optimizing feature detection and description for high-level task. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9947-495,, 2020.
[5] Hunter Blanton, Scott Workman, and Nathan Jacobs. A structure-aware method for direct pose estimation. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 205-214, 2022.
[6] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dsac differentiable ransac for camera localization. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 2492-2500, 2017.
[7] Eric Brachmann and Carsten Rother. Learning less more 6d camera localization via 3d surface regression. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 4654-4662, 2018.
[8] Eric Brachmann and Carsten Rother. Visual camera re-localization from RGB and RGB-D images using DSAC. IEEE Trans. Pattern Anal. Mach. Intell., 44(9):5877-5865 2022.
[9] Francois Chaumette and Seth Hutchinson. Visual servo control. i. basic approaches. IEEE Robotics Automation Magazine,, 13(4):82-90 90, 2006.
[10] Shuai Chen, Xinghui Li, Zirui Wang, and Victor Adrian Prisacariu. Dfnet: pose regression with direct feature matching. In European Conference on Computer Vision, pages 1-17, 2022.
Enhance absolute
[11] Shuai Chen, Zirui Wang. and Victor Prisacariu. Direct-posenet: Absolute pose regression with photometric consistency. In International Conference on 3D Vision, pages 1175-1185, 2021.
[12] Ondrej Chum and Jiri Matas. Optimal randomized RANSAC. IEEE Trans. Pattern Anal. Mach. Intell., 3078):4422-1482 2008
[13] Peter Corke, Witold Jachimezyk, and Remo Pillat. Robotics, vision and control: fundamental algorithums MATLAB, volume 73. Springer, 2011.
[14] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 224-236, 2018.
[15] Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multi- object dynamics with compositiona neural radiance fields. In Conference on Robot Learning, pages 1755-1768, 2022.

---

[16] Xiao-Shan Gao, Xiaorong Hou, Jianliang Tang, and Hang-Fei Cheng. Complete solution classification for the perspective-three three-point problem. IEEE Trans. Pattern Anal. Mach. Intell., (5(8):930-943, 2003.
[17] Jeffrey Ichnowski, Yahav Avigal. Justin Kerr, and Ken Goldberg, Dex-nerf: Using neural radiance field to grasp transparent objects. In Conference on Robot Learning, pages 526-536, 2021.
[18] Zhehao Jin, Jinhui Wu, Andong Liu, Wen-An Zhang, and Li Yu. Policy-based deep reinforce- ment learning for visual servoing control of mobile robots with visibility constraints. IEEE Trans. Ind. Electron., (9(2))1898-119))) 2022.
[19] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: convolutional network for real-time 6-dof camera relocalization. In 2015 IEEE International Conference on Computer Vision, pages 2938 2946, 2015.
[20] Obin Kwon, Jeongho Park, and Song Hyun Oh. Renderable neural radiance map for visual navigation. ArXiv, abs/2303.00304, 2023.
[21] Xiaotian Li, Shuzhe Wang, Yi Zhao, Jakob Verbeek, and Juho Kannala. Hierarchical scene coordinate classification and regression for visual localization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11980-11989, 2020.
[22] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 123-123, 2021.
[23] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: bundle-adjusting neural radiance fields. In 2021 IEEE/CVF International Conference on Computer Vision, pages 5721-5731, 2021.
[24] Jianlin Liu, Qiang Nie, Y. Liu, and Chengjie Wang. Nerf-loc: Visual localization with condi- tional neural radiance field. ArXiv, abs/2304.07979, 2023.
[25] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421, 2020.
[26] Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle. LENS: localization enhanced by nerf synthesis. In Conference on Robot Learning, pages 1347-1356, 2021.
[27] Le Qi, Yu Zhang, Qingyu Yin, and Ting Liu. Ms-transformer: Introduce multiple structural priors into unified transformer for encoding sentences. Comput. Speech Lang., 72:101304, 2022.
[28] Paul-Edoo EdloaEd Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse to fine: Robust hierarchical localization large scale. IEEE Conference on Computer Vision and Pattern Recognition, pages 12716-12725, 2019.
[29] auu-Edd Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4937-4946, 2020.
[30] Paul-Edouard Sarlin, Ajaykumar Unagar, Mäns Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Mare Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the feature: Learning robust camera localization from pixels to pose. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3247-3257, 2021.
[31] Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Efficient & effective prioritized matching for large-scale image-based localization. IEEE Trans. Pattern Anal. Mach. Intell, 1745 2017.
12

---

[32] Yoli Shavit, Ron Ferens, and Yosi Keller. Learning multi-scene absolute pose regression with transformers. In 2021 IEEE/CVF International Conference on Computer Vision, pages 2713-2722, 2021.
[33] Yoli Shavit, Ron Ferens, and Yosi Keller. Learning multi-scene absolute pose regression with transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2713-2722 2021.
[34] Haobin Shi, Lin Shi, Gang Sun, and Kao-Shing Hwang. Adaptive image-based visual servoing for hovering control of quad-rotor. IEEE Transactions on Cognitive and Developmental Systems., 12(3):417-426, 2020.
[35] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and An- drew W. Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 2930- 2937, 2013.
[36] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison. imap: Implicit mapping and positioning in real-time. In 202/ IEEE/CVF International Conference on Computer Vision, pages 2209-6218, 18, 2021.
[37] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mírcea Cimpoi, Mare Pollefeys, Josef Sivic, Tomás Pajdla, and Akihiko Torii. Inloc: Indoor visual localization with dense matching and view synthesis. IEEE Trans. Pattern Anal. Mach. Intell., 43(4): 1293)):123-130 2021.
[38] Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang. Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja. David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2330. 202404444, 2023.
[39] Julien P. C. Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip H. S. Torr, Shahram Izadi, and Cem Keskin. Learning to navigate the energy landscape. In Fourth International Conference 3D Vision, pages 323-332, 2016.
[40] Walch, C. Hazirbas, L.. Taaxe T. Sattler, S. Hilsenbeck, and D. Cremers. Image- based localization using Istms for structured feature correlation. In 2017 IEEE International Conference on Computer Vision, pages 627-637, 2017
[41] Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham. Atloc: Attention guided camera localization. In The Whirty-Fourts AAAI Conference on Artificial Intelligence, pages 10393-10401, 2020.
[42] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Gdr-net: Geodetry-guided direct regression network for monocular 6d object pose estimation. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition, pages 16611-166211 2021.
[43] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2642-2651, 2019.
[44] Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, pages 1323-1330, 2021.
[45] Kunwu Zhang, Yang Shi, and Huaiyuan Sheng. Robust nonlinear model predictive control based visual servoing of quadrotor uavs. JEEE/ASME Transactions on Mechatronics, 26(2):700-708, 2021.
[46] Qunjie Zhou, Sérgio Agostinho, Aljosa Osep, and Laura Leal-Taixé. Is geometry enough for matching in visual localization? In European Conference on Computer Vision, pages 407-425, 2022.
[47] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SIAM:: neural implicit scalable encoding for SLAM. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12776-12786. 2022.
13


Reviews:
[{"review_id": "awOEy3ErNu", "reviewer": "Reviewer_iS7u", "summary": "The paper introduces image based visual servo to refine the coarse poses estimated using a coordinate regression network trained on neural rendering based 3D labels. The paper also uses correspondences in depth between neural rendered image and query image to iteratively refine the pose of the image. The paper also shows that NeRF-IBVS can be used as a navigation prior and subsequently use IBVS navigation. ", "questions": "1. The paper could use better neural rendering techniques for latency and accuracy of depth and image rendering.  \n2. No details about the RANSAC algorithm used.  \n3. Can the superglue correspondence network used be made trainable? ", "limitations": "1. The paper may not be able to handle instances where there are a lot of specular objects in the scene. This can lead to bad coarse pose and in turn IBVS may not converge.\n2. The paper doesn't compare runtimes while comparing performance of other visual navigation methods. Some of the methods might have compromised on quality to reduce latency\n3.The authors claim IBVS iterations are launched no more than 4 times. How was this arrived at ? Again this really depends on the quality of rendering. Any evidence to support this will help.\n\nTypos:\n-'Deired' -> Desired in caption for Figure 4.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper uses only a few images for visual localization compared to other methods out there \n2. The paper does both visual localization and navigation based on visual servoing. The authors claim to be the first to do that.\n3. The paper shows SOTA results on multiple datasets for both visual localization and navigation. \n4. The paper provides and effective navigation prior which improves IBVS based navigation.", "weaknesses": "1. The paper trains the coordinate regression network using the approximate 3D labels obtained through neural rendering which is not ideal. A NERF trained with a small number of posed images has a lot of rendering noise, as discovered by the authors and shown in Fig 3.\n2. It is not clear how the authors decide which pixels in the NERF-rendered images are used to train the coordinate regression net. They do mention that image boundaries are not used, but this is not a satisfactory description. Image boundaries from NERF may well be crisp, and parts of the image near the centre might have noise.\n3. The coordinate regression network name is a bit confusing. At first glance, it appears as if it is a network that regresses pose of the camera from the 2D image. Actually, it is regressing 3D positions for each image pixel in the global coordinate frame.\n4. The IBVS module is not clear. The authors suddenly introduce camera velocity and image velocity and a Jacobian without mentioning any prior information about this - which is presumably from the visual servoing literature. This material is provided in the supplementary, but should be included in the main paper.\n5. The method is not real-time due to the time taken by neural-rendering, during the optimization step\n6. The authors have not explored the joint training of NERF and the coordinate regression net, which might be an avenue to pursue.\n7. The Quality of NERF rendering is crucial even for getting coarse poses that are subsequently used to train the coordinate regression net, and the PnP on the 3D poses and the 2D images. This will struggle with non-Lambertian and dynamic objects, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions.\n\n**W1 and W2: The paper trains the coordinate regression network using the approximate 3D labels provided by NeRF which is not ideal. It is not clear how the authors decide which pixels in the NERF-rendered images are used to train the coordinate regression net.**\n\nThanks for your insightful comment.\n\n1) Since the pose optimization module will correct the coarse poses provided by the coordinate regression network with PnP. The coordinate regression networks only need to ensure that the estimated result is in the vicinity of the target, which does not need to be a very accurate result. Therefore the approximate 3D labels with rendering noise obtained through neural rendering are enough to train the coordinate regression network.\n2) We empirically found that the rendering noise of NeRF mainly appears in the scene edge region (lack of multi-view constraints), which generally appear in the edge region of images, and the rendering noise in the center region of the images has little effect on the goal that the coordinate regression network only needs to estimate results near the target. Therefore, we empirically crop out information within 40 pixels from the boundary to minimize the effect of the rendering error.\n3) To verify the above arguments, we perform quantitative experiments on the 12Scenes dataset. We consider that the coarse results provided by the coordinate regression network are in the vicinity of the target when the position and orientation error of the coarse pose reduces after pose optimization. Therefore, we count how often this happens, and the specific quantitative results are shown in the following table:\n\n| 12Scenes | kitchen1 | living1 | bed    | kitchen2 | living2 | luke   | gates362 | gates381 | lounge | manolis | 5a     | 5b     | Average    |\n|----------|----------|---------|--------|----------|---------|--------|----------|----------|--------|---------|--------|--------|--------|\n| **Frequency**     | 96.88%   | 99.59%  | 94.61% | 96.67%   | 95.42%  | 97.76% | 100.00%  | 95.16%   | 99.69% | 90.46%  | 90.14% | 91.60% | 95.66% |\n\nAs shown in the table, the average frequency of coarse pose error reduction is greater than 95\\% in all scenes. Therefore, the coordinate regression network consistently outputs results near the target despite only cropping the boundary information of the image.\n\n**W3: The coordinate regression network name is a bit confusing.**\n\nThanks for your valuable suggestions.\nFollowing HACNet [20] and DSAC* [8] in the field of visual localization, which defines the coordinate regression network as regressing dense 3D coordinates directly from 2D images. We add the definition of coordinate regression networks in the methods section.\n\n**W4: The IBVS module is not clear.**\n\nThanks for your valuable suggestions. We have rewritten the method section to make IBVS module clearer for readers and make this paper self-contained. Specifically, We add a subsection for the methods section called **“Preliminaries”** to introduce the core concepts of IBVS. Please refer to **Q2** in the **\"General Response\"** for specific modification details.\n\n**W5: The method is not real-time due to the time taken by neural rendering.**\n\nThanks for your insightful comment.\nThe efficiency is not the main problem addressed in the paper, we mainly take full advantage of the prior knowledge of the scene provided by NeRF to enhance visual localization and visual navigation.\nFor example, in the 7Scenes dataset, the proposed method obtains better localization performance than the advanced real-time methods such as FeatLoc++Au [2] (our: 0.05m/1.55$^{\\circ}$ vs FeatLoc++Au: 0.14m/5.89$^{\\circ}$) and uses fewer posed images for supervision (our: 2090 vs FeatLoc++Au: 26000). Furthermore, our method enables navigation based on IBVS without using custom markers and the depth sensor compared to general IBVS-based navigation methods [9,17].\nIn the future, we can try to use faster and more accurate variants of NeRF to improve the overall efficiency of the proposed method.\n\n**W6: The authors have not explored the joint training of NERF and the coordinate regression net.**\n\nThanks for your valuable suggestions.\nThe coordinate regression network needs the approximate labels provided by NeRF for training. Therefore, it is necessary to train NeRF first and then train the coordinate regression network. If the NeRF and coordinate regression network are trained jointly, it may be necessary to focus on the NeRF training in the early stage and focus on the coordinate regression network training in the later stage, which might be a worthy future direction.\n\n**W7: The Quality of NERF rendering is crucial. The proposed method will struggle with non-Lambertian and dynamic objects, etc.**\n\nIndeed, the quality of NeRF rendering is crucial for our method. A fundamental solution to this problem (caused by non-Lambertian and dynamic objects, etc.) requires further improvements in NeRF. In the future, the improved NeRF variants can be used to improve the performance of our method.\n\n**Q1: The paper could use better neural rendering techniques for latency and accuracy of depth and image rendering.**\n\nTheoretically, a better NeRF does lead to performance improvements, which is a good direction for future research.\n\n**Q2: No details about the RANSAC algorithm used.**\n\nSorry for the unclear details. Actually, RANSAC is a common module in the field of visual localization. We have explained it in detail in **“1.4 Details of Pose Optimization and Correspondence Selection”** of the supplementary material.\n\n**Q3: Can the superglue correspondence network used be made trainable?**\n\nYes, superglue can be trained, but generating the training labels requires a lot of manual annotations. Therefore, we do not train superglue in our dataset and only use its official weights."}, {"review_id": "tLCcUldYqQ", "reviewer": "Reviewer_oADp", "summary": "This paper tackles the task of visual localization, i.e. estimating a camera pose from a query image. Approaches in the literature require a large number of posed images and even dense 3D supervision for some of them. Authors propose to leverage Neural Radiance Fields (NeRF) to solve the problem of visual localization while requiring fewer posed images and no 3D labels.\n\nThe method can be decomposed into sequential steps: (1) A NeRF model is trained from a set of posed images, (2) a coordinate regression network is trained on the posed images using depth predictions from the NeRF model to provide coarse 3D labels, (3) a coarse camera pose is predicted by the coordinate regression network from the query image, and is refined by performing Image-Based Visual Servoing. Camera pose refinement is turned into a navigation problem from the coarse to the target pose.\n\nThe paper compares the localization performance of the proposed method with different baselines of the literature on 2 datasets: 7-scenes and 12-scenes. Authors show that their method outperforms reported baselines trained without 3D labels and is on par with the ones trained with dense 3D supervision while using less training posed images.\n\nFinally, the authors show their NeRF-IBVS framework can also be used to perform camera navigation from an initial state to a target state, both specified as images.", "questions": "1. [Major] Paper writing could be improved. Some sentences could be made smoother (e.g. some sentences are split in half while they should not). Writing could also be made more concise and efficient, as some parts are quite hard to follow. I am sorry to provide only vague directions here, but I have the feeling there is not one specific part of the paper to improve, but rather smoothness and conciseness should be improved globally to allow a better understanding. Technical sections (e.g. IBVS, correspondence selection) could be presented more progressively by defining some of the core notions early on and diving into the details more smoothly.\n\n2. [Major] An experiment involving a comparison between IBVS and iNeRF-like camera optimization in the last step, i.e. refinement from coarse to target pose, should be performed by authors.", "limitations": "Authors mentioned two relevant limitations of their work:\n1. NeRF rendering quality can be low in large outdoor scenes, which might have an impact on the visual localization performance. Experiments are only conducted in indoor scenes in this paper.\n2. The low rendering speed of NeRF models prevents their solution from running in real time.\n\nBoth limitations rather come from the capabilities of current NeRF models than the proposed method. Future improvements in the field of NeRFs might allow better localization in large scenes and with faster runtime.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles an important problem: performing visual localization from fewer posed images (compared with current state-of-the-art methods) and no required 3D dense supervision.\n\n2. NeRF models are an interesting tool to perform visual localization and it is great to have papers studying this application.\n\n3. The paper considers a decent number of baselines from the literature in the experimental study. Ablation studies are also performed to evaluate the gain brought by the different proposed contributions.", "weaknesses": "1. [Major] Paper writing could be improved. The current version of the paper is fine when it comes to understanding the main contributions but several reads were needed to grasp the details of the work. I suggest authors go through the paper again and improve the coherence and writing.\n\n2. [Major] NeRF is a differentiable function mapping and thus, as done in previous work (iNeRF [43] as mentioned by authors in the paper), pose refinement can be performed by freezing the NeRF weights and optimizing camera location based on a rendering loss. It would be interesting that authors compare using IBVS and optimization as in iNeRF to perform the last pose refinement step, i.e. from the coarse to the target position. Optimizing camera pose based on NeRF rendering can be considered as a simpler alternative to the IBVS framework (e.g. no need to build the $L$ matrix).\n\n3. [Minor] This paper introduces a limited set of technical novelties. However, it tackles an important problem and nicely combines previous works, proposing simple yet efficient ideas to perform the target task. Thus, I consider this not an important issue with this work, but should still be mentioned in this review. For example, the authors mention they “design a new fast iteration strategy that reduces the rendering frequency of NeRF”. To the best of my knowledge, this is mainly about querying the NeRF model every n iterations, which might not be considered as a strong contribution. Authors might want to reconsider how they introduce this part of their work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions.\n\n**W1 and Q1: Paper writing could be improved.**\n\nThanks for your valuable suggestion. We have polished the whole paper to make it more smooth and concise for readers. Please refer to **Q2** in the **\"General Response\"** for specific modification details.\n\n\n**W2 and Q2: Optimizing camera pose based on NeRF rendering can be considered as a simpler alternative to the IBVS framework. An experiment involving a comparison between IBVS and iNeRF-like camera optimization in the last step.**\n\nThanks for your valuable suggestion.\n1) Since our other contribution is to enhance IBVS-based visual navigation. Therefore, our method will not accomplish the visual navigation task, if we replace IBVS framework with iNeRF.\n2) Since iNeRF performs time-consuming neural rendering at each optimization iteration, iNeRF spends more time relative to IBVS framework (15.65 seconds vs 3.58 seconds). Therefore, replacing IBVS with iNeRF in our method will substantially increase the time cost of pose optimization.\n3) Implementing the replacement of IBVS with iNeRF in our method and testing it on 7Scenes and 12Scenes datasets is not achievable in a short period of time. We think this replacement is a good direction for future research. \n\n**W3: “fast iteration strategy” might not be considered as a strong contribution.**\n\nThanks for your valuable suggestion. The fast iteration strategy is simple but greatly accelerates the iteration of pose optimization. It provides a good starting point for possible future work related to IBVS utilizing the scene prior information provided by NeRF."}, {"review_id": "Ic7rCZ38Il", "reviewer": "Reviewer_k2yC", "summary": "This paper proposes a visual localization pipeline that uses NeRF and image-based visual servoing. The main contribution is that the proposed method uses much fewer images with pose annotations and no 3D ground-truth labels (only uses pseudo ground-truth provided by NeRF to train a coordinate regression network). The method first relies on the coordinate regression network to provide an initial coarse pose, which is then refined with image-based visual servoing.", "questions": "See weaknesses.", "limitations": "The authors mention a couple important limitations in the main paper. I could not find discussion on potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper presents a creative approach towards the visual localization problem. I believe the utilization of NeRF to learn a coordinate regression model and as pose initialization is novel, considering that prior works used NeRFs mostly for data augmentation. Furthermore, formulating the pose refinement process as a visual servoing problem that uses state-of-the-art image matching methods and NeRF is an interesting idea which fits well the problem. Overall, the paper proposes something new and useful to an important task.", "weaknesses": "I have a few concerns/comments. Not all of these are necessarily weaknesses.\n\nThere is discussion over iNeRF in the related work. Authors claim that limitations of iNeRF are that it requires a very good initial pose and relies on photometric loss which is sensitive to artifacts in the rendered images. This is somewhat true for the proposed approach as well as visual servoing, by definition, requires visual overlap between observation and query image and any image matching method is going to be affected by rendering artifacts. In fact, the paper acknowledges this fact and has an extra verification step (section 3.2.2) to filter out outliers from unreliable Superpoint+Superglue correspondences.\n\nIn my understanding using NeRF to initialize IBVS is analogous to the structured methods (such as [27,36]) retrieving a set of candidate database images to establish an initial pose(s). The proposed method mentions that only renders a single image C_r, but how do you ensure that C_r has visual overlap with the query? How often does this occur? Why not sample around the coarse pose T and generate multiple images?\n\nThe paper uses the phrase \"velocity s of the image coordinates\". Isn't that just the optical flow?\n\nIn terms of pipeline complexity, the method is actually closer to the structured methods (than the regression-based). I am curious to see runtime required for a single query compared to the baseline methods.\n\nCan you provide more details as to the number of posed images required to train the proposed method vs the baselines?\n\nWhat is the difficulty of the queries in the visual servoing experiments? i.e. what is the pose difference / overlap between initial and desired state? It would be interesting to divide the test set into easy/hard queries based on their pose diff / ovelap in order to show the limitations of the method. The authors do show the performance of the coarse pose estimation but it is not clear how sensitive IBVS is to that coarse estimate. \n\nMinor issue: Please include references of baselines in Table 1, and use \"/\" instead of \",\" to separate numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions.\n\n**W1: The proposed method mentions that only renders a single image $C_r$, but how do you ensure that $C_r$ has visual overlap with the query? How often does this occur? Why not sample around the coarse pose T and generate multiple images?**\n\nThanks for your valuable suggestion.\nThe coordinate regression network is able to ensure that the single rendered image $C_r$ corresponding to the estimated coarse pose overlaps with the query image. To verify this argument, we perform quantitative experiments on the 12Scenes dataset. We consider that the single rendered image corresponding to the coarse pose has a good visual overlap with the query image when the position and orientation error of the coarse pose reduces after pose optimization. Therefore, we count how often this happens, and the specific quantitative results are shown in the following table:\n\n| 12Scenes  | kitchen1 | living1 | bed    | kitchen2 | living2 | luke   | gates362 | gates381 | lounge | manolis | 5a     | 5b     | Average    |\n|-----------|----------|---------|--------|----------|---------|--------|----------|----------|--------|---------|--------|--------|--------|\n| **Frequency** | 96.88%   | 99.59%  | 94.61% | 96.67%   | 95.42%  | 97.76% | 100.00%  | 95.16%   | 99.69% | 90.46%  | 90.14% | 91.60% | 95.66% |\n\n\nAs shown in the table, the visual overlap between rendered and query images occurs with high frequency.\n\nSince there is a high probability that the rendered image corresponding to the coarse pose has visual overlap with the query image and rendering multiple images spends a lot of time. Therefore, we do not sample around the coarse pose T and generate multiple images.\n\n**W2: The paper uses the phrase \"velocity s of the image coordinates\". Isn't that just the optical flow?**\n\nIndeed, both the velocity of the image coordinates in our paper and optical flow represent the velocity of pixels, but they are solved differently. In IBVS, the velocity of the image coordinates indicates the desired velocity to control the correspondences motion towards the target coordinate which is the image coordinates error of correspondences multiplied by a scaling factor. The optical flow is primarily solved based on the pixel intensity changes between consecutive frames.\n\n**W3: In terms of pipeline complexity, the method is actually closer to the structured methods (than the regression-based). I am curious to see the runtime required for a single query compared to the baseline methods.**\n\nIndeed, our method is more time-consuming due to the slow rendering speed of NeRF. For example, the time comparison of our method with state-of-the-art methods HACnet [20] in our device is (3.61 seconds vs 0.07 seconds). In the future, faster and more accurate variants of NeRF can be used to further improve the overall efficiency of the proposed method.\n\n\n**W4: Can you provide more details as to the number of posed images required to train the proposed method vs the baselines?**\n\nPlease refer to **Q1** in **“General Response”** for the number of training data. Actually, we have included these tables in the section **\"1.1 Amount of Training Data for Each Scene\"** of the supplementary material. And we will include these tables in the paper to make it clear to readers.\n\n\n**W5: It would be interesting to divide the test set into easy/hard queries based on their pose diff / ovelap in order to show the limitations of the method. The authors do show the performance of the coarse pose estimation but it is not clear how sensitive IBVS is to that coarse estimate.**\n\nThanks for your valuable suggestion.\nIn order to illustrate the sensitivity of IBVS to coarse estimate, we conduct quantitative experiments on the 12Scenes dataset. There is no relevant literature in the field of IBVS to define easy and hard cases, we assume that the test image whose position and orientation errors of the coarse pose are greater than the median error is the hard case, and vice versa as the easy case. \nWe consider that the IBVS module optimizes the pose successfully when the error of the coarse pose is reduced. We count the frequency of successful optimization of the coarse pose in the easy case and hard case respectively. Specifically, the quantitative experimental results are shown in the following table:\n\n|                  | kitchen1   | living1   | bed        | kitchen2   | living2    | luke       | gates362  | gates381   | lounge    | manolis    | 5a         | 5b         | Average        |\n|------------------|------------|-----------|------------|------------|------------|------------|-----------|------------|-----------|------------|------------|------------|------------|\n| **median error** | 0.23/13.41 | 0.19/9.65 | 0.38/15.26 | 0.28/14.79 | 0.34/14.32 | 0.27/11.52 | 0.11/8.31 | 0.23/13.12 | 0.39/9.78 | 0.32/15.60 | 0.27/16.24 | 0.25/15.21 | 0.27/13.10 |\n| **easy case**         | 100.00%    | 100.00%   | 99.02%     | 100.00%    | 100.00%    | 99.36%     | 100.00%   | 99.05%     | 100.00%   | 98.25%     | 99.20%     | 98.03%     | 99.41%     |\n| **hard case**         | 93.75%     | 99.19%    | 90.20%     | 93.33%     | 90.86%     | 96.15%     | 100.00%   | 91.27%     | 99.39%    | 82.71%     | 81.12%     | 85.22%     | 91.93%     |\n\nThe results in the table show that IBVS can be run successfully almost all the time in the easy case and fails with a small probability in the hard case. Thus IBVS is robust to most of the coarse estimation of poses.\n\n**Minor issue: Please include references of baselines in Table 1, and use \"/\" instead of \",\" to separate numbers.**\n\nThanks for pointing this out.\nWe have added references to baselines in Table 1, and corrected \"/\" to \",\" to separate numbers in Table 1 and Table 2."}, {"review_id": "d3gYnwUtNd", "reviewer": "Reviewer_91GX", "summary": "The paper presents a novel visual localization method combined with the NeRF technique to address the issue of training a large number of pose images in existing visual localization models. The proposed method has two advantages: (1) the paper trains a coordinate regression network using a few posed images with coarse 3D labels generated by NeRF. (2) The paper uses an image-based visual servo to explore 3D scenes for pose optimization.  The simulation experiments show superior performance in comparison with other methods.", "questions": "See Weakness part", "limitations": "Due to the poor rendering quality of NeRF in large outdoor scenes compared to indoor scenes, our method is mainly applied in indoor scenes. Moreover, due to the slow rendering speed of NeRF, the proposed method cannot achieve real-time performance.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed method, NeFT-IBVS uses a coarse-to-fine strategy to accurately localize the camera with a few posed images being adopted.  The method equips with two leading edges: (1) a NeRF-based posed images are used to train a coordinate regression network in the coarse stage. (2) an optimization algorithm is presented to effectively optimize the coarse pose.  Furthermore, the authors also design a new pipeline to reduce the rendering frequency of NeRF to speed up the pose optimization. ", "weaknesses": "1. The authors claim the proposed method is able to localize the camera with fewer posed images, can the authors give a rigorous description of it? For instance, how many posed images can be used, or can the authors show the reduced ratio of used posed images compared to state-of-the-art methods?\n2. Can the authors give a comparison with DSAC method[1] on the 7-Scene dataset? It seems that the proposed method performs worse than it.\n[1] Brachmann, Eric and Carsten Rother. “Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC.” IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (2020): 5847-5865.\n\n3. For statistical significance, the experiments should be conducted several times and the statistical significance of the results should be determined. Furthermore, some typos are necessary to be corrected. For example, “We” should be rewritten “we” in Line 103. “enables” should be rewritten “enable”. The writing is obligated to be further polished.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions.\n\n**W1: The authors claim the proposed method is able to localize the camera with fewer posed images, can the authors give a rigorous description of it? For instance, how many posed images can be used, or can the authors show the reduced ratio of used posed images compared to state-of-the-art methods?**\n\nThanks for your valuable suggestion.\nPlease refer to **Q1** in **“General Response”** for the number of training data. Actually, we have included these tables in the section **\"1.1 Amount of Training Data for Each Scene\"** of the supplementary material. And we will include these tables in the paper to make it clear to readers.\n\n\n**W2: Can the authors give a comparison with DSAC method[1] on the 7-Scene dataset? It seems that the proposed method performs worse than it. [1] Brachmann, Eric and Carsten Rother. “Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC.” IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (2020): 5847-5865.**\n\nThanks for your valuable suggestion.\nActually, We have performed the comparison in the 7-Scene dataset and the results are given in Table 1 of the submitted manuscript. The mentioned method DSAC is denoted as DSAC* [8]. Although the localization performance of our method is slightly lower than DSAC*, we use fewer data to train the model **(2090 vs 26000)** and achieved comparable performance **(0.05m/1.55$^{\\circ}$ vs 0.03m/1.36$^{\\circ}$)**. Furthermore, our method can enhance IBVS-based navigation, which enables navigation based on IBVS without using custom markers and the depth sensor. We have included references to baseline methods in all tables to make it more clear for readers. \n\n**W3: For statistical significance, the experiments should be conducted several times and the statistical significance of the results should be determined. Furthermore, some typos are necessary to be corrected. For example, “We” should be rewritten “we” in Line 103. “enables” should be rewritten “enable”. The writing is obligated to be further polished.**\n\nThanks for your valuable suggestion.\n1) We conduct five experiments on the 7Scenes and 12Scenes datasets and calculate the mean and variance of the results, which are shown in the following table:\n\n|              | 7Scenes               | 12Scenes              |\n|--------------|-----------------------|-----------------------|\n| **Mean**     | 0.05m/1.61$^{\\circ}$  | 0.02m/0.92$^{\\circ}$  |\n| **variance** |   3.28$\\times 10^{-6}$/0.00407                    |   6.45$\\times 10^{-7}$/0.00079                     |\n\nAs shown in the table, the mean is almost identical to the results presented in the paper and the variance is very small. Therefore the performance of the proposed method is very stable.\n\n2) We have corrected “We” to “we” in Line 103. \nWe have corrected “enables” to “enable” in Figure 1 and Line 62. In addition, we further polished the whole paper and performed grammatical checks. Please refer to **Q2** in the **\"General Response\"** for specific modification details."}, {"review_id": "zfzbLP19Vk", "reviewer": "Reviewer_F153", "summary": "This paper addresses visual localisation (i.e. the estimation of the\n6-degrees-of-freedom pose of a camera) from scene coordinate regression.\n\nThe problem tackled is the need for a large amount of pose and depth labels to\ntrain scene coordinate regression networks.\n\nThe paper proposes a solution to reduce the label requirements by producing\npseudo-groundtruth depth labels with a NeRF trained on the scene.\n\nThe lower data requirement comes at the cost of a lower accuracy of the scene\ncoordinate regression network. This is compensated by a pose refinement step\nbased on visual servoing: the coarse pose is refined until the view rendered by\nthe nerf on the estimated pose matches the target pose. More specifically, the\npose is estimated until the local features in the rendered view are close to\ntheir corresponding points in the target image. The pose updates are estimated\nto reduce the discrepency between these feature positions.\n\nThe method proceeds as follows:\n- Given a set of calibrated images of the scene, train a nerf.\n- Generate pseudo-groundtruth depth with the nerf.\n- Train the scene coordinate regression with the pseudo-groundtruth depth.\n- Infer a coarse pose from the trained scene coordinate regression network.\n- Refine the coarse pose using visual-based servoing \n\nThe method is compared against state-of-the-art localisation method from\nseveral categories: scene coordinate regression network based (e.g. DSAC++),\npose estimation network based (e.g. PoseNet, Direct-PoseNet,  DFNet) and\nfeature based (PixelLoc). ", "questions": "Q1:  L146,147: what is the difference between $m_r$ and $n_r$? from reading, one\n  gets that $m_r$ is a pixel.\n\nQ2: L171: does the next ibvs run starting from the output pose of the previous\n  ibvs? Or do all the ibvs start from the coarse pose from the scene coordinate\n  regression network?\n\nQ3:  L187: where is $\\hat{n}^r$ used?\n\nQ4: L188: where is $d$ used?\n", "limitations": "Figure 1 would gain from having a technical description in addition to the\ncurrent caption. Something along the lines of: \"Left: The coarse pose estimated with\nthe scene coordinate regression network is refined with IBVS. Right: Illustration of the pose updates / navigation in the NerF scene.\"\n\nTechnical typos:\n- L4: \"require groundtruth 3D labels for supervision\": only regression-based\n  methods require 3D labels (i.e. depth). Structure-based appraoches do not.\n- L39: the reference [25] indeed uses nerf for data augmentation but [10,11]\n  uses nerf to refine camera poses with a render-and-compare strategy (10 with\n  photometric loss and 11 with a feature loss). This sentences need to be\n  updated accordingly.\n- Unsuported claim that the proposed method need less training images:\n  - L46: \"which is significantly fewer than typical visual localization method\".\n    This claim is not supported.\n  - L86-87: \"Our method ...\"\n  - L103: \"We utilize few posed images\"\n  - L241: \"It is important to note that we use fewer ...\"\n- The paper claims that RNR-Map [19] requires RGBD but the original paper\n  claims that it is based on RGB vision only. It is would be useful to have\n  this clarified.\n- Eq1: M is undefined even though one can guess that it is the number of points\n  along the ray\n- L116: $t_n$ is undefined\n- Eq2: Isn't the depth $D$ already defined in the camera coordinate frame? If\n  so, then there is no need to transform the points with $T^{-1}$ i.e. it\n  should be $P_W = D K^{-1} p$.\n- L146: \"depth of correspondences\" -> \"depth of pixels\"\n- Eq7 assumes that the set of correspondences remain the same for all IBVS,\n  which is an assumption only introduced later in 3.2.2. Instead, the\n  assumption should be mentionned before, even if only in a short sentence. The\n  reader can then be referred to 3.2.2 for more details.\n\n- L201: \"we can obtain accurate ...\": how are these correspondences derived?\n  one can guess that is with L144-147 but it would be helpful to the reader if\n  it was explicitly mentionned.\n\n- L226: Pixloc is actually a feature-based refinement only method so it does\n  not fully qualify as a structure-based method.\n\n- Fig4: e(x) ... are undefined even though one can guess that these are\n  distance errors\n\n\n\nConfusing writing that impedes with the understanding of the technical details:\n- L7: \"only a few\". It might be better to either give a quantification of how\n  much less or only write \"less\" images.\n\n- L8: \"To achieve this, we first use a few posed images with coarse 3D\n  labels provided by NeRF to train a coordinate regression network, which is\n  used to provide the coarse pose for unseen view. \"\n  It would be easier to read if written with a different order:\n  - pseudo ground-truth 3d labels are computed with nerf\n  - the scene coordinate regression network is trained\n  - a coarse pose is estimated from the regression network\n  - the coarse pose is refined with IBVS\n\n- L53: \"the correspondences with depth\": the reader can guess the derivation\n  bein run i.e. select pixels in one image and find their match in the second\n  image using the depth and the image poses. But the reader should not have to\n  guess the technical derivations so it is better to write it down explicitly.\n  - Same comment for L141: the \"depth correspondences\"\n\n- L61: \"navigation prior\" is undefined. Does this refer to the \"simulated\"\n  navigation run in the NeRF scene?\n\n- L125: \"geometric constraints\" is undefined\n\n- L127: Does \"edge areas\" refer to the scene's edge or the image's edge? One\n  could guess that it is the scene's edge but it is better to specify it.\n\n- L136: \n  - \"we first initialize IBVS ...\". At this stage, IBVS has not been\n    introduced and it is not part of the common knowledge in the visual\n    localisation community (this is more of a robotic common knowledge). So it\n    would be helpful to the reader to have a high level description of IBVS\n    before the low-level one that is already in the paper. This will help the\n    reader understand what \"initializing the IBVS\" means (e.g. \"The initial pose\n    in the IBVS optimisation is the coarse pose and the 3D points that will\n    support the optimisation are generated with the NeRF depth.\").\n  - the term pixel velocity is not introduced and it is not a term common in\n    the visual localisation community which is more used to the\n    render-and-compare notations than the visual servoing notations.\n\n- L168: this sentence is paradoxial. An alternative formulation could be: \"The\n  assumption is violated so accumulation errors will occur.\"\n\n\nA non exhaustive list of typos:\n- L26: the other line of approach *is* [...] regresse*s*\n- L75: \"To achieve ...\". It seems this sentence should not be there.\n- L76: has attracted\n- L103: \"Then the same ...\": this sentence misses a verb\n- Fig4: Deired -> Desired\n\n\n### Post-Rebuttal\n\n#### Addressed weaknesses:\nW3: The evaluation on 12 scenes compares only against methods that do not require\ngroundtruth depth labels, whereas the 7 scenes evaluation also compared against\nmethod that do use groundtruth depth labels.\n- Update: The paper does evaluate methods using ground-truth depth (only DSAC is missing in the 12 scenes experiments).\n\nW2: The paper claims to contribute not only to localisation but also navigation but\nthere is no comparison to previous navigation work (RNR-MAP). Also, the claim\nthat RNR-MAP requires depth information seems opposite to the RNR-MAP's claim\nto be RGB-vision based only (in the related work section).\n- Update: the comparison to rnr-map is not applicable so this limitation is obsolete.\nHowever, the contribution to the navigation task should be clarified.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1: The problem of reducing the data requirement to train scene coordinate\nregression network is an interesting problem.\n\nS2: The approach to use NeRF to produce pseudo-groundtruth 3D labels and the use of\nthis synthetic depth inside a localisation experiment is insightful.\n\nS3: The paper cites and compares against relevant works.", "weaknesses": "W1: The writing in general is good and one get the high-level description of the\ntechnical derivations. However, some parts are confusing which requires the\nreader to re-read a paragraph multiple times or to guess the technical\nderviation being described.  (See Limitations for detailed comments and\nsuggested updates). Also, there are several english typos that can easily be\ncaught with an automatic spell-grammar checker.\n\nW4: The paper claims that one advantage of the method is that it requires less\nlabeled data (calibrated images with depth) than other methods but there is no\nexperiments to support this claim (e.g. gradually reducing the number of\ntraining images and comparing the performance drop).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments and suggestions.\n\n**W1: Paper writing could be improved.**\n\nThanks for the valuable suggestion. We have polished the whole paper to make it more smooth and concise for readers. Please refer to **Q2** in the **\"General Response\"** for specific modification details.\n\n**W2: The paper claims to contribute not only to localization but also navigation but there is no comparison to previous navigation work (RNR-MAP). Also, the claim that RNR-MAP requires depth information seems opposite to the RNR-MAP's claim to be RGB-vision based only.**\n\nThanks for the valuable suggestion.\n1) RNR-MAP [1] indeed uses depth maps in the construction of the map, which is described in the section “3. RNR-Map” of RNR-MAP. Also, the inputs of the navigation system use depth information, which is described in \"Figure 3. Navigation System Overview\".\n2) Our IBVS-based navigation (6-degree-of-freedom navigation, without visual odometry information) is not configured in the same way as RNR-MAP (3-degree-of-freedom navigation, with visual odometry information). Therefore, the comparison between IBVS-based navigation and RNR-MAP is unfair.\n3) Our main contribution to visual navigation is to enhance IBVS-based navigation. The main point is that our method enables navigation based on IBVS without using custom markers and the depth sensor compared to general IBVS-based navigation methods [9,17]. It is obvious to enhance the general IBVS-based navigation. Therefore, it does not need to be compared with other visual navigation methods to illustrate enhancement.\n\n**W3: The evaluation on 12 scenes compares only against methods that do not require groundtruth depth labels, whereas the 7 scenes evaluation also compared against method that do use groundtruth depth labels.**\n\nThanks for your valuable comment. Actually, our method is compared with both types of methods in 7Scenes and 12Scenes.\nSpecifically, in the 7Scenes dataset, our method is compared to both methods that without using groundtruth depth labels (DFNet [10], FeatLoc++Au [2], MS-Transformer [26, TransPoseNet [32], PoseNet [18]) and methods that use groundtruth depth labels (PixLoc [29], HACNet [20], DSAC* [8], DSAC++ [7], SA [5]).\nIn the 12Scenes dataset, our method is compared to both methods that without using groundtruth depth labels (FeatLoc++Au and FeatLoc++ [2], PoseNet [18]) and methods that use groundtruth depth labels (SA [5], HACNet [20]).\n\n**W4: There are no experiments to support this claim that one advantage of the method is that it requires less data than other methods.**\n\nThanks for your valuable comment.\nPlease refer to **Q1** in **“General Response”** for the number of training data. Actually, we have included these tables in the section **\"1.1 Amount of Training Data for Each Scene\"** of the supplementary material. And we will include these tables in the paper to make it clear to readers.\n\n**Q1: L146,147: what is the difference between $m_r$ and $n_r$ from reading, one gets that $m_r$ is a pixel.**\n\n$n_r$ is defined in L147 as the image coordinate of the correspondences. Specifically, $n_r$ is the pixel coordinate $m_r$ minus the pixel coordinate of the camera's optical center. We will add the explanation of $n_r$ to the paper.\n\n**Q2: L171: does the next ibvs run starting from the output pose of the previous ibvs?**\n\nThe next ibvs run starts from the output pose of the previous ibvs and we will add this detail to the paper.\n\n**Q3: L187: where is $\\mathbf{\\hat{n}_r}$ used?.**\n\nIn L188, $\\mathbf{\\hat{n}_r}$ is used to compute the coordinate distance $d = ||\\mathbf{\\hat{n}_r} - \\mathbf{n_r}||_2$\n\n**Q4: L188: where is $d$ used?**\n\nIn \"3.2.2 Correspondence Selection\", we empirically set a threshold $\\tau$ for the coordinate distance $d$. We will filter out the outliers in the correspondences when the coordinate distance $d$ is greater than the threshold $\\tau$. We will add the symbol $d$ after the coordinate distance in L191 to make it clear for readers.\n\n**Limitations: Technical typos**\n\nThanks for the valuable suggestion, we have carefully studied all the comments. Below we will provide point-by-point responses to all the suggestions.  \n1) We update \"while state-of-the-art regression-based methods require dense ground truth 3D labels for supervision\" in L4.\n2) We update \"Current NeRF-based visual localization methods employ NeRF for data augmentation purposes [25], or use nerf to refine camera poses with a render-and-compare strategy [10,11]\" in L39.\n3) Please refer to **W4**.\n4) Please refer to **\"1.\"** in **W2**.\n5) We add \"M is the number of points along the ray\" for Eq1.\n6) We add \"$t_n$ is the near bound of the camera ray\" in L116.\n7) Depth $D$ is defined in the camera coordinate frame. But $D\\mathbf{K}^{-1}\\mathbf{p}$ only transforms points from the image coordinates frame to the camera coordinates frame. We require camera pose $\\mathbf{T}$ to transform the point from the camera coordinate frame to the world coordinate frame: $P_w=D\\mathbf{T}^{-1}\\mathbf{K}^{-1}\\mathbf{p}$  in Eq2.\n8) In L146, the depth of the correspondences is used to initialize the Jacobi matrix of IBVS. Therefore, depth $Z_c$ is the depth of the correspondences.\n9) We add \"The set of correspondences remains the same for all IBVS\" in the \"3.2.1 IBVS Module\" for Eq7.\n10) In L136, we have rewritten the method section to make IBVS module clearer for readers. \nPlease refer to the beginning of **Q2** in the **\"General Response\"** for specific modification details.\nWe add \"In IBVS, the pixel velocity indicates the desired velocity to control the correspondences motion towards the target coordinate which is the image coordinates error of correspondences multiplied by a scaling factor\".\n11) For issues in L201, Fig4, L7, L8, L53, L61, L125, L127, and L168, please refer to **Q2** in the **\"General Response\"** for specific modification details.  \n12) For the typos. We perform grammatical checks on the whole paper and correct all grammatical errors."}]

Author rebuttal:
# General Response: 
We thank all reviewers for their insightful and constructive suggestions, which help a lot in further improving our paper. 

**Q1: The number of training data.**

We notice that several reviewers are concerned about the number of data required to train the proposed method compared to the baseline. We provide the specific number of training data as follows: 

| 7Scenes  | Chess             | Fire               | Heads              | Office            | Pumpkin           | Kitchen           | Stairs             | All                |
|----------|-------------------|--------------------|--------------------|-------------------|-------------------|-------------------|--------------------|--------------------|
| **Baseline** | 4000(100\%)       | 2000(100\%)        | 1000(100\%)        | 6000(100\%)       | 4000(100\%)       | 7000(100\%)       | 2000(100\%)        | 26000(100\%)       |
| **Our**      | **260(6\%)** | **280(14\%)** |**240(24\%)** | **300(5\%)** | **320(8\%)** | **350(5\%)** | **340(17\%)** | **2090(8\%)** |

| 12Scenes | kitchen1   | living1     | bed        | kitchen2   | living2    | luke        | gates362    | gates381    | lounge     | manolis     | 5a          | 5b          | All          |
|----------|------------|-------------|------------|------------|------------|-------------|-------------|-------------|------------|-------------|-------------|-------------|--------------|
| **Baseline** | 744(100\%) | 1036(100\%) | 868(100\%) | 768(100\%) | 725(100\%) | 1370(100\%) | 3536(100\%) | 2950(100\%) | 925(100\%) | 1613(100\%) | 1001(100\%) | 1391(100\%) | 16927(100\%) |
| **Our**      | **185(25\%)**  | **170(16\%)**   | **215(25\%)**  | **190(25\%)**  | **180(25\%)**  | **340(25\%)**   | **350(10\%)**   | **295(10\%)**   | **230(25\%)**  | **270(17\%)**   | **250(25\%)**   | **280(20\%)**   | **2955(17\%)**   |

Actually, we have included these tables in the section **"1.1 Amount of Training Data for Each Scene"** of the supplementary material. And we will include these tables in the paper to make it clear to readers.

**Q2: The writing issue.**

We notice that several reviewers suggest that the paper writing could be further polished. Therefore, we have worked on both readability and language and have also involved native English speakers for language improvement. We also added a lot of technical details to make our paper more understandable to the readers. We list the major changes as follows:

We have rewritten the method section to make IBVS module clearer for readers and make this paper self-contained. Specifically, we add a subsection for the methods section called **“Preliminaries”** to introduce the core concepts of IBVS: “The aim of Image-Based Visual Servoing (IBVS) [9] is to control the camera to move towards the desired pose based on vision information while minimizing the error of correspondences between the current image and the target image. To achieve this goal, IBVS first calculates the desired image coordinate velocity of correspondences based on the coordinate error of correspondences. Then, the Jacobi matrix is constructed based on the correspondences which establishes the relationship between the image coordinate velocity and the camera velocity. Finally, the desired camera velocity to control the camera motion towards the target is solved based on the Jacobi matrix and the desired image coordinate velocity. The specific details of the IBVS are presented in the supplemental materials”. 

We add details and fix typos in the paper. We add "M is the number of points along the ray" for Eq1. We add the explicit mention in L201: "After that, the correspondence selection algorithm can obtain accurate and non-collinear correspondences and the IBVS module can obtain navigation trajectories. Therefore we can obtain accurate and non-collinear correspondences with navigation trajectories". 
We add “$e(x_i), e(y_i)$ is the image coordinate error of correspondences” and correct “Deired” to “Desired ” in Fig4.
We update "only a few images" to "fewer images" in L7.
We update "To achieve this, we first use NeRF to generate pseudo-3D labels which are used to train the scene coordinate regression network. Then a coarse pose is estimated from the regression network. Finally, we use the image-based visual servo (IBVS) to utilize 3D scenes provided by NeRF for pose optimization" in L8.
We update "Then, we establish the correspondences between the rendered image and the query image and query the approximate depth of the correspondences based on the rendering depth map. Finally, the correspondences with depth are used to launch IBVS to guide pose optimization" in L53.
We update "where $x, y$ and $Z_c$ denote image coordinates and depth of correspondences in the currently rendered image" in L141.
We add "Navigation prior is accurate and non-collinear correspondences and navigation trajectories corresponding to the correspondences" in L61.
We rewrite the "geometric constraints" to "multi-view constraints" which are the basic concepts in multi-view stereo in L125.
We rewrite the "edge areas" to "edge areas in the image" in L127.
We update "Due to rendering errors of NeRF, the assumption that the 3D coordinates of correspondences are accurate is violated so accumulation errors will occur" in L168.
We correct “enables” to “enable” in  Fig 1 and L 62. We correct “We” to “we” in L 103. We add  "$t_n$ is the near bound of the camera ray" in L 116. 


Meta-review: This paper proposes a method for visual localization which generates images and 3D labels with a NERF, followed by visual servoeing.

Overal, the paper had received 4 positive out of 5 ratings and was disscussed by reviewers and authors and then reviewers and AC. The reviewers appreciated novelty, strong results, the usage as a navigation prior. Issues raised were on clarity (the main issue) and some minor discussions happened on comparisons with differentiable baselines like iNERF. The rebuttal could answer many of the questions. Although some issues on clarity remained, the AC judges the paper to be ready for publication.
Decision: Accept (poster)