Title: NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation
Venue: NeurIPS 2023
Accepted: yes
Keywords: NeRF

Image-Based Visual Servoing (IBVS)

visual localization

visual navigation
Abstract: Visual localization is a fundamental task in computer vision and robotics. Training existing visual localization methods requires a large number of posed images to generalize to novel views, while state-of-the-art methods generally require dense ground truth 3D labels for supervision. However, acquiring a  large number of posed images and dense 3D labels in the real world is challenging and costly. In this paper, we present a novel visual localization method that achieves accurate localization while using only a few posed images compared to other localization methods. To achieve this, we first use a few posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. Then a coarse pose is estimated from the regression network with PNP. Finally, we use the image-based visual servo (IBVS) with the scene prior provided by NeRF for pose optimization. Furthermore, our method can provide effective navigation prior, which enable navigation based on IBVS without using custom markers and depth sensor. Extensive experiments on 7-Scenes  and 12-Scenes datasets demonstrate that our method outperforms state-of-the-art methods under the same setting, with only 5\% to 25\% training data.  Furthermore, our framework can be naturally extended to the visual navigation task based on IBVS, and its effectiveness is verified in simulation experiments.

NeRF-IBVS: Visual Servo Based on NeRF for Visual Localization and Navigation

Abstract

Visual localization is a fundamental task in computer vision and robotics. Training existing visual localization methods requires a large number of posed images to generalize to novel views, while state-of-the-art methods generally require ground truth 3D labels for supervision. However, acquiring a large number of posed images and 3D labels in the real world is challenging and costly. In this paper, we present a novel visual localization method that achieves accurate localization while using only a few posed images compared to other localization methods. To achieve this, we first use a few posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. Then a coarse pose is estimated from the regression network with PNP. Finally, we use the image-based visual servo (IBVS) with the scene prior provided by NeRF for pose optimization. Furthermore, our method can provide effective navigation prior, which enables navigation based on IBVS without using custom markers and the depth sensor. Extensive experiments on 7-Scenes and 12-Scenes datasets demonstrate that our method outperforms state-of-the-art methods under the same setting, with only 5% to 25% training data. Furthermore, our framework can be naturally extended to the visual navigation task based on IBVS, and its effectiveness is verified in simulation experiments.

1 Introduction

The estimation of camera positions and orientation based on a query image is a fundamental task in computer vision. It has wide applications in various fields such as robotics, virtual reality, and autonomous driving. Existing works that tackle this task can be generally divided into two categories. The first line of approaches are structure-based [6, 28, 31, 37, 46], which first establish correspondences between 3D points and 2D pixels, and then compute camera pose with PnP [16] solver utilizing RANSAC [12]. These methods typically achieve satisfactory results at the cost of a complex pipeline. The other line of approaches is regression-based [2, 3, 40, 21, 42, 41], which either regress the 3D coordinates or directly regress the absolute pose. These methods yield a more compact framework and have fast inference performance. All the above methods require a substantial number of posed images (generally thousands of posed images per scene) to guarantee the generalization ability to novel viewpoints. Moreover, state-of-the-art methods [6, 4, 21] also require dense ground truth 3D labels for supervision, such as depth and 3D model. However, acquiring a large number

-NeRF

QueryImage

Visual

NeRF-IBVS

Localization

Visual

Navigation

DesiredState

of posed images and dense 3D labels is challenging and costly. It naturally raises a question: is it possible to design a framework that achieves accurate localization results while only requiring a small number of posed images ?

Recent advances in Neural Radiance Fields (NeRF) [25] provide a promising solution. Trained from hundreds of posed images, NeRF can render realistic images and give coarse depth maps for novel viewpoints as prior knowledge of 3D scenes. Intuitively, this prior knowledge can assist visual localization methods in generalizing to novel viewpoints. Current NeRF-based visual localization methods [10, 11, 26] only employ NeRF for data augmentation purposes, which still requires a substantial number of images for training. The estimated pose will be coarse when using a few data. Considering this issue, it would be desirable if the coarse pose in NeRF could be further refined to meet the need for visual localization.

In this paper, we propose a novel visual localization framework called NeRF-IBVS, which achieves accurate localization while using only a few posed images. NeRF-IBVS adopts a coarse-to-fine paradigm to estimate the pose of the query image. In the coarse stage, we first train a NeRF with hundreds of posed images, which is significantly fewer than typical visual localization methods. Then, we use the same posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. The coarse pose is estimated from the regression network with PNP. In the refining stage, we design an optimization algorithm to effectively optimize the coarse pose. Motivated by the image-based visual servo (IBVS) [9], which navigates the camera from the initial pose to the desired pose using coordinate error and depth of correspondences between the current image and the target image. Therefore, we first use NeRF to render the image and depth corresponding to the current coarse poses. Then, we establish the correspondences between the rendered image and the query image and query the approximate depth of the correspondences based on the rendering depth map. Finally, the correspondences with depth are used to launch IBVS to guide pose optimization.

Due to rendering quality errors, directly applying IBVS to NeRF-rendered images and depth is not feasible. Moreover, IBVS requires non-collinear correspondences during 6-degree-of-freedom navigation. In order to achieve robust pose optimization, we design a correspondence selection algorithm using spatial information provided by NeRF and the coordinate regression network. To speed up the pose optimization, we further design a strategy that reduces the rendering frequency of NeRF. Further, the effective 2D and 3D correspondences obtained by NeRF-IBVS can be used as the navigation prior to enhance IBVS-based navigation. Specifically, this navigation prior can enable IBVS-based navigation without using custom markers and the depth sensor. This advantage greatly expands the application scope.

To the best of our knowledge, NeRF-IBVS is the first method that can accomplish both visual localization and visual navigation based on IBVS as shown in Figure 1, providing a novel idea for pose optimization by controlling camera motion. Employing only 5% to 25% of the posed images required by other localization methods on 7-Scenes dataset and 12-Scenes dataset, our method outperforms state-of-the-art methods without using ground truth 3D labels and has comparable performance with methods using ground truth 3D labels. Furthermore, our method can enable visual navigation based on IBVS without using custom markers and the depth sensor, which greatly expands the application scope of IBVS-based navigation. We also verified the effectiveness of visual navigation in a simulation environment.

2 Related Work

Visual localization. Structure-based methods [6, 28, 31, 37, 46] use complex pipelines to achieve precise localization. Firstly, correspondence is established between 3D coordinates and 2D pixels. Secondly, a camera pose is computed through a PnP solver with RANSAC. The first stage requires a complex pipeline consisting of image retrieval, feature extraction, and matching. To achieve a more compact frame compared to structure-based methods. In recent years, regression-based localization methods [2, 3, 40, 21, 42, 41, 24] attract increasing attention, which either regresses the 3D coordinates or directly regress the absolute pose. Specifically, coordinate regression methods, such as DSAC++ [7] and HACNet [21] use neural networks to learn the first stage of the structure-based methods, which directly regress 3D scene coordinates from an image. These coordinates regression methods typically use ground truth 3D labels as supervision to achieve better localization performance. Unlike coordinate regression, absolute pose regression methods [19, 40, 2] generally use only posed images to train the neural network. MS-Transformer [32] and AtLoc [41] use attention mechanisms to guide the regression process and have substantially improved the performance of absolute pose regression methods. Recently, some methods [10, 11, 26] use NeRF as data enhancement to improve the performance of visual localization methods. The above methods require a substantial number of posed images to guarantee the generalization ability to novel viewpoints. Our method can achieve accurate localization while using only a few posed images compared to other localization methods.

Robotics with neural radiance fields. Recently, the neural radiance field (NeRF) [25] has shown great potential in robotics applications [22, 17, 15]. iNerf [44] fixes the network weights of NeRF and directly optimizes the camera pose parameters by computing the backpropagation of the photometric loss. However, this method requires an initial pose near the ground truth pose. Methods based on photometric loss tend to fail when blur and artifacts appear in the rendered images. iMAP [36] and NICE-SLAM [47] are a pose estimator in the SLAM setting that requires continuous sequential data. BARF [23] optimize camera poses during the training of NeRF. Therefore they cannot estimate the pose of a single query image. Moreover, NeRF can provide coarse scene geometry, which is useful for navigation tasks. NeRF-nav [1] designs a smooth and collision-proof navigation strategy based on the density provided by NeRF. RNR-Map [20] achieves precise visual navigation performance based on NeRF, but is limited to three degrees of freedom and requires RGBD pictures and odometer information. Our method can accomplish both visual localization and 6 degrees of freedom navigation based on visual servo and using only RGB images.

3 Preliminaries

NeRF. NeRF uses MLP network F θ to encode the 3D scene, which predicts the RGB color c ∈ R 3 and volume density σ ∈ R for each input 3D point p ∈ R 3 and its view direction d ∈ R 3 . It can be summarized as F θ ( p , d ) = ( c , σ ) , where θ is the parameters of MLP network. Then, NeRF uses volume rendering techniques to generate realistic rendered images C ( r ) and coarse depth maps D ( r ) from any viewpoint:

where r and M indicates the rendering ray and the number of points along the ray, c i and t i indicates the color and distance of the samples in the rendering ray, T i = ( -∑ i -1 j =1 σ j δ j ) indicates the accumulated transmittance from the near bound of the rendering ray to t i , and δ i = t i +1 -t i is the distance between adjacent samples in rendering ray.

IBVS. The aim of Image-Based Visual Servoing (IBVS) is to control the camera to move toward the desired pose based on vision information while minimizing the coordinate error of image features between the current image and the target image. Correspondences are used as image features in this paper. To achieve this goal, IBVS first calculates the desired image coordinate velocity of correspondences based on the coordinate error. Then, the Jacobi matrix is constructed based on the correspondences, which establishes the relationship between the image coordinate velocity and the camera velocity. Finally, the desired camera velocity to control the camera motion towards the target is solved based on the Jacobi matrix and the desired image coordinate velocity. The specific details of the IBVS are presented in the supplemental materials.

NeRF

Normalized

Hundreds of Posed

Rendered Depth

Pseudo-3DLabelsi

ImagesPerScene

PNP

Coarse3DLabels

Inference

Coordinate regression network

Training

Navigation Prior

NeRF

Query

Nmage

Appropriate 2D

Correspondenes

Select

Candidate points

-0.65

Correspondences Selection

IBVSModulewith

Navigation

Trajecteries

IBVSModulewith

Correspondences Selection

Appropriate3D

Correspondenes

4 Method

Given a few posed images compared to the typical localization method, we aim to achieve accurate localization. To achieve this, we first use a few posed images with pseudo-3D labels provided by NeRF to train a coordinate regression network. Then the coarse pose is estimated from the regression network with PNP. Finally, we introduce IBVS to gradually refine the coarse pose by leveraging the 3D scene prior provided by NeRF. Furthermore, our method can provide effective navigation prior, which enhances navigation based on IBVS. Figure 2 demonstrates the overall pipeline.

4.1 Coarse Pose Estimation

We use a few posed images with pseudo-3D labels provided by pre-training NeRF to train a coordinate regression network. Then the coarse pose is estimated from the regression network with PNP. Specifically, we convert the pixel coordinates p of training images into pseudo-3D labels P w using depth D provided by NeRF:

where K , T denote the internal parameter and pose of camera. Following common settings [43], these pseudo-3D labels P w are then normalized to fit within the range of a unit cube. Then, we use the advanced coordinate regression component of GDR-Net [42] as our coordinate regression network, which is trained by pseudo-3D labels P w . To estimate coarse pose, the coordinate regression network

processes the query image to generate 3D coordinates and establishes a 2D-3D correspondence. This 2D-3D correspondence is then fed into PnP [16] to determine the coarse pose ˆ T of the query image. We empirically found that the lack of multi-view constraints in the boundary regions of the scene leads to a significant depth error in image edges. To mitigate the impact of the rendering quality error, we exclude the edge areas in the image of our pseudo-3D labels P w when training our coordinate regression network.

4.2 Pose Optimization

To optimize the coarse poses, our method views pose optimization as the process of navigating the camera from the coarse pose towards the target pose using scenes prior provided by NeRF.

4.2.1 IBVS Module

IBVS initialization using NeRF. To start pose optimization, we first initialize IBVS using NeRF. This process mainly initializes the Jacobian matrix L of IBVS using the image coordinates and depth of correspondences, which establishes the relationship between the velocity v c of the camera and the velocity ˙ s of image coordinates:

where the L corresponds to per correspondence can be formulated as:

where x, y and Z c denote image coordinates and depth of the correspondence in the currently rendered image. To initialize the Jacobian matrix L , we first utilize NeRF to render an image C r and depth map D r that corresponds to the current coarse pose ˆ T . Then the off-the-shelf correspondences establishment algorithm, such as SuperPoint [14] with SuperGlue [29], is employed to extract the pixel coordinate ( m q , m r ) of correspondences between the query image C q and the rendered image C r . The depth Z c of correspondences m r can be obtained by querying the rendered depth D r . Image coordinates n r = ( x, y ) in the rendered image can be obtained using the pixel coordinates m r and the internal parameter K follow [9]. Finally, n r with depth Z c are used to initialize L .

Optimization iteration. After the IBVS initialization, we started IBVS iteration to navigate the camera from the coarse pose toward the target pose. At each iteration, IBVS needs to calculate the velocity v c of the camera to update the pose. To control the 6 DOF velocity v c of the camera, four non-collinear correspondences are selected and the velocity ˙ s of the image coordinates are set to -λ ( n q -n r ) follow [9]. According to Equation (3), we can obtain the camera velocity v c as:

where λ, L + , and n q denote proportional factor, generalized inverse matrix, and image coordinates in the query image. Then, we update the current pose by integrating the camera velocity v c during unit iteration to obtain the variation of pose ∆ T ( v c ( t )) . The coarse poses are updated as follows:

The IBVS iteration proceeds until either the pixel coordinate error of correspondences is less than a certain threshold or the maximum number of iterations is reached.

Theoretically, each iteration of IBVS requires NeRF rendering to update the parameters of the Jacobian matrix L . However, utilizing NeRF rendering for each IBVS iteration is highly timeconsuming and impractical. Therefore, we design a new fast iteration strategy that reduces the rendering frequency of NeRF. We first calculate the 3D coordinates of correspondences using the depth information provided by NeRF, while assuming the 3D coordinates of the correspondences are precise. At each iteration, We project the 3D coordinates of correspondences to the image plane by using the updated poses. This projection can be an approximate substitute for the correspondence in the rendered image. In this way, we skip the time-consuming NeRF rendering process, and NeRF rendering is only executed at the initialization of IBVS.

Due to rendering errors of NeRF, the assumption that the 3D coordinates of correspondences are accurate is violated, so accumulation errors will occur. Accumulated errors may lead to the optimization terminating with a large error, In order to eliminate accumulated errors, we render the image

again based on the most recently updated poses to relaunch IBVS iterations. Typically, the IBVS iterations are launched no more than four times to obtain a precise localization.

The first IBVS iterations launch aims to move the camera a significant distance toward the target pose, while the subsequent IBVS iterations launch only fine-tune the pose around the target pose. To improve the optimization efficiency, We dynamically set the number of iteration steps. Specifically, the number N ( i ) of iterations steps in the i th IBVS iterations launch is set dynamically based on coordinate error of correspondences:

where N 1 and n r i , n q i denote the fixed number of iteration steps in the first IBVS iteration launch and image coordinates of correspondences in the i th IBVS iterations launch, respectively.

4.2.2 Correspondence Selection

The Jacobian matrix L of IBVS is highly sensitive to the selection of the correspondences, and it requires accurate and non-collinear correspondences to update parameters during iteration. Due to rendering errors of NeRF, which may cause inaccurate correspondences. In order to increase the robustness of IBVS, we designed a correspondence selection algorithm to select appropriate correspondences for IBVS. We first query coarse 3D coordinates estimated by the coordinate regression network to obtain 3D coordinate ˆ P q of correspondences m q in the query image. Then, we use coarse pose ˆ T to project the 3D coordinate ˆ P q onto the rendered image to obtain image coordinates ˆ n r . Finally, we can obtain the coordinate distances of correspondences d = ∥ ˆ n r -n r ∥ 2 . Since the coarse pose ˆ T and 3D coordinates ˆ P q near the target, the accurate correspondences generally have a smaller distance. Therefore, we empirically set a threshold τ for the coordinate distance d to filter out a large number of outliers. Finally, we use RANSAC [12] to further select accurate and non-collinear correspondences. Please see the supplementary material for more details.

4.3 Visual Navigation

NeRF-IBVS can provide navigation prior which enhances IBVS-based navigation. To enable IBVS navigation, it is necessary to use correspondences that are non-collinear and remain in the camera's field of view during navigation. However, acquiring such correspondences is challenging, current methods [18, 45, 34] usually rely on custom markers which limit the applications of IBVS. The navigation prior provided by our method can solve this problem. Specifically, We first use NeRFIBVS to locate the initial pose of the navigation task. Then the initial pose is viewed as a coarse pose to start the NeRF-IBVS locating the target state of the navigation task. After that, the correspondence selection algorithm can obtain accurate and non-collinear correspondences and the IBVS module can obtain navigation trajectories in the image plane. The current correspondences are not yet guaranteed to remain in the camera's field of view during navigation. Benefiting from the navigation trajectory, the appropriate 2D correspondences that fall outside of the image plane are automatically filtered out. It ensures that the correspondences selected remain in the camera's field of view during navigation. We backproject 2D correspondences based on depth provided by NeRF to obtain coarse 3D coordinates as 3D correspondences. During the navigation iteration, the 3D correspondences are projected using the current pose to obtain the depth prior and the image coordinates prior. The image coordinates prior can assist in locating and associating the 2D correspondences in the current image, which enables IBVS-based navigation without custom markers. Moreover, the depth prior can be used to update the parameters of the Jacobian matrix in each iteration, which enables IBVS-based navigation without using a depth sensor. Although there is a small error in the depth obtained by projection, IBVS has the advantage of being robust to small depth errors[13].

5 Experiments

In this section, we first compare the proposed NeRF-IBVS with state-of-the-art visual localization methods. Furthermore, our method extended to IBVS-based navigation without using custom markers and the depth sensor, and its effectiveness is verified in simulation experiments.

5.1 Visual Localization

Datasets. We conduct experiments on two public indoor benchmark datasets. The 7-Scenes [35] contains seven indoor scenes recorded by a KinectV1 camera, the data includes RGB-D images, poses, and ground truth 3D models. The 12-Scenes [39] contains twelve indoor scenes with RGB-D images and poses. The recorded environments of the 12-Scenes are significantly larger than those in the 7-Scenes. We use fewer posed images compared to baseline localization methods, and the specific number of training data for each scene is configured as shown in Table 1 and Table 2.

Baseline. Baseline methods can be classified into two main categories based on whether or not using ground truth 3D labels. Specifically, we utilized the following methods, each with its unique characteristics and requirements. 1). Methods without ground truth 3D label supervision: DFNet [10] and Direct-PoseNet [11] is the absolute pose regression method based on NeRF; FeatLoc++ [2] is the absolute pose regression method using sparse features. Furthermore, featLoc++Au utilizes sparse 3D structure to enhance visual localization performance; TransPoseNet [33] achieve visual localization based on attention mechanism; 2). Methods with ground truth 3D label supervision: SA [5] is a recently proposed absolute pose regression method based on 3D structure. HACNet [21], DSAC* [8] and DSAC++ [7] are state-of-the-art visual localization methods based on coordinate regression. PixLoc [30] is an advanced structure-based visual localization method.

Implementation Details. Our approach utilizes Nerfacto [38], an accelerated variant of NeRF. Nerfacto was trained for 100,000 iterations, and all camera poses are only centralized, with no scale adjustment. As errors in the ground truth pose of 7-Scenes dataset impair the training quality of Nerfacto [38], we turn on the pose-optimization component of the Nerfacto for 7-Scenes dataset. We set the far plane to 8 meters for office5a and office5b in the 12-Scenes dataset and set the far plane to 6 meters for other scenes. We set the near plane to 0 meters for all scenes. The coordinate regression network is trained for 40 epochs with a learning rate of 0.001. We empirically set τ = 200 for the threshold of coordinate distance. In pose optimization, we use SuperPoint with SuperGlue and their official weights to detect correspondences, set λ = 0 . 5 , N 1 = 100 for IBVS iterations. For training time, the Nerfacto and coordinate regression network are trained on one NVIDIA RTX3090 GPU for about 2 days. Please see the supplementary material for more implementation details.

Comparisons. We report the median translation (cm) and rotation ( ◦ ) errors [19] in each scene, which is a main evaluation metric for localization accuracy. The localization accuracy is reported in Table 3 for 7-Scenes and Table 4 for 12-Scenes. As can be observed, our method consistently achieves the best performance among visual localization methods without ground truth 3D label

supervision. Meanwhile, our method has comparable performance with methods using ground truth 3D labels as supervision, which closes the gap between the two types of visual localization methods. It is important to note that we use fewer posed images for training compared to the baseline methods, which shows the high value in practical application. Compared with the methods using NeRF for data augmentation, our method (0.05m, 1.64 ◦ ) outperforms DFNet [10] (0.12m, 3.71 ◦ ) and Direct-PoseNet [11] (0.20m, 7.26 ◦ ), thus providing a more efficient way to take advantage of the scenes prior provided by NeRF. Moreover, our method outperforms attention-based MS-Transformer [27] and FeatLoc++Au [2], which utilize sparse 3D structure augmentation, highlighting the potential of NeRF in visual localization.

(a)

(b)

(c)

Rendered Image

Pose Optimization Trajectory

Rendered Image

Initial pose

0.00

Desired pose

0.10

500

600

Initial pose

0.00 005

Desired pose

Initial pose

0.10

0.15

600

Initial pose

0.36

Desired pose

0.32

0.30

0.20

9.25

Initial pose

Desired pose

0.460

jaxod)n

Qualitative results. We also qualitatively analyze the NeRF-IBVS on 12-Scenes dataset. Our coarse pose estimation can obtain the poses near the target as shown in Figure 3(a), which provides a good starting point for pose optimization. The pose optimization trajectories in both 3D space and image plane show that the optimized pose can accurately reach the desired pose as shown in Figure 3(b). Figure 3(c) shows that the images rendered using the optimized poses are accurately aligned with the

query images. Figure 3(b)-(c) demonstrate that our method can achieve accurate localization. Please see the supplementary material for more qualitative results.

5.2 Simulation for Navigation

To verify the effectiveness of our method in enabling IBVS-based navigation without using custom markers and the depth sensor, we randomly selected a pair of images from the real 12-Scenes dataset with co-views to serve as the initial state and the desired state for navigation. We first establish the correspondences between the initial state and the desired state as shown in Figure 4(a). Appropriate 2D and 3D correspondences are selected as navigation prior as shown in Figure 4(b)-(c). A 6-degreeof-freedom navigation simulation is then performed using the camera model. Figure 4(d) shows that the navigation trajectory in both 3D space and image plane can accurately reach the desired state. Figure 4(e) shows that the coordinate error of correspondences can converge. The coordinate error of correspondences can reflect errors in the pose. The simulation results in Figure 4(d)-(e) demonstrate that our IBVS-based navigation without using custom markers and the depth sensor can achieve accurate navigation. Please see the supplementary material for more simulation results.

NeRF-IBVS

Deired State

Initial State

Deired State

Initial State

(c) 3D Correspondences

(d)  Navigation Trajectories

(e)  Pixel Coordinate Errors

Desired pixel

Initial pixel

(a)

(b)

Candidate points

Initial pose

150

Appropriate points

Desired pose

100

0.10

100

50

-0.55g

-0.50

0.05

(w)

0

0.60N

0.65

0.05

0.00

N

V(pixel)

200

jaxid

100

50

e(x1)

e(y1)

e(x2)

0.70

300

150

e(y2)

e(x3)

1.6

1.8

1.18

1.20

400

200

e(y3)

e(x4)

1.4

X(m)

1.6

1.8

2.0

1.0

1.2

1.4

0.35

0.40

X（m)

0.45

0.50

0.55

1.14

1.16

100

200

u (pixel)

300

400

500

600

250

20

iterations

40

60

80

e(y4)

5.3 Ablation Studies

In this section, we conduct ablation experiments on the 12-Scenes dataset to evaluate the effectiveness of the major design of the proposed NeRF-IBVS.

Effectiveness of coarse pose estimation. To guarantee excellent performance for subsequent pose optimization, coarse pose estimation is required to output coarse poses and coarse 3D coordinates near the target. Therefore, we evaluate the localization performance of coarse pose estimation on the 12-Scenes dataset. As shown in Table 6, our coarse pose estimation obtains poses and 3D coordinates near the target (0.27m, 13.10 ◦ ), providing a good starting point for pose optimization.

We consider that the coarse poses and coarse 3D coordinates near the target when the position and orientation error of the coarse pose reduces after pose optimization. To further verify the effectiveness of coarse pose estimation, we count how often this happens. The specific quantitative results on the 12-Scenes dataset as shown in Table 5, the average frequency of coarse pose error reduction is greater than 95%. Therefore, the coarse pose estimation consistently outputs results near the target.

Effectiveness of pose optimization. To verify the effectiveness of pose optimization, we compare the overall NeRF-IBVS with a coarse pose estimation pipeline on 12-Scenes dataset. Localization result in Table 6 shows that pose optimization (0.02m, 0.95 ◦ ) can significantly improve the localization performance compared to coarse pose estimation (0.27m, 13.10 ◦ ).

Effectiveness of correspondence selection. To verify the effectiveness of correspondence selection, we evaluate the localization performance of NeRF-IBVS without correspondence selection. The results presented in Table 6 show that the NeRF-IBVS without correspondence selection downgrade localization performance (0.09m, 3.73 ◦ ) compared to NeRF-IBVS with correspondence selection (0.02m, 0.95 ◦ ). It demonstrates that correspondence selection can select the appropriate correspondences for NeRF-IBVS and enhance the robustness of NeRF-IBVS to noise.

We also qualitatively analyze the effectiveness of correspondence selection. Coordinate error and depth error of correspondences are generally prone to occur in areas with poor rendering quality. As shown in Figure 5, the correspondence selection tends to choose correspondences that occur in areas with good rendering quality, which can enhance the robustness of pose estimation.

Rendered Image

Rendered Image

Rendered Depth

(a)

(b)

(c)

6 Conclusion

We propose a novel localization framework that can achieve precise localization performance using only a few posed images compared to other localization methods. We first use a few posed images with coarse 3D labels provided by NeRF to train a coordinate regression network. The coarse pose of the query image can be estimated by a coordinate regression network with the PNP algorithm. Then, we optimize the coarse pose using IBVS based on NeRF representation to achieve accurate localization performance. Further, our method can provide effective navigation prior, which can enable IBVS-based navigation without using custom markers and the depth sensor. This advantage greatly expands the application scope of IBVS-based navigation. To the best of our knowledge, NeRF-IBVS is the first method that can accomplish both visual localization and navigation based on IBVS, providing a novel idea for pose optimization by controlling camera motion.

Limitations. The quality of NeRF rendering is crucial for our method. Due to the poor rendering quality of NeRF in large outdoor scenes compared to indoor scenes, our method is mainly applied in indoor scenes. Moreover, due to the slow rendering speed of NeRF, our method cannot achieve real-time performance. The proposed method also struggles with non-Lambertian and dynamic objects since the NeRF performs poorly in these scenarios. In the future, the improved NeRF variants can be used to improve the performance of our method.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 91948303), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), and CCF-Alibaba Innovative Research Fund For Young Scholars.

References

Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEEE Robotics Autom. Lett. , 7(2):4606-4613, 2022.

Thuan Bui Bach, Tuan Tran Dinh, and Joo-Ho Lee. FeatLoc: Absolute pose regressor for indoor 2D sparse features with simplistic view synthesizing. ISPRS Journal of Photogrammetry and Remote Sensing. , 189:50-62, 2022.

Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced feature points: Optimizing feature detection and description for a high-level task. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4947-4956, 2020.

Aritra Bhowmik, Stefan Gumhold, Carsten Rother, and Eric Brachmann. Reinforced feature points: Optimizing feature detection and description for a high-level task. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4947-4956, 2020.

Hunter Blanton, Scott Workman, and Nathan Jacobs. A structure-aware method for direct pose estimation. In IEEE/CVF Winter Conference on Applications of Computer Vision , pages 205-214, 2022.

Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. Dsac - differentiable ransac for camera localization. In 2017 IEEE Conference on Computer Vision and Pattern Recognition , pages 2492-2500, 2017.

Eric Brachmann and Carsten Rother. Learning less is more - 6d camera localization via 3d surface regression. In 2018 IEEE Conference on Computer Vision and Pattern Recognition , pages 4654-4662, 2018.

Eric Brachmann and Carsten Rother. Visual camera re-localization from RGB and RGB-D images using DSAC. IEEE Trans. Pattern Anal. Mach. Intell. , 44(9):5847-5865, 2022.

Francois Chaumette and Seth Hutchinson. Visual servo control. i. basic approaches. IEEE Robotics Automation Magazine. , 13(4):82-90, 2006.

Shuai Chen, Xinghui Li, Zirui Wang, and Victor Adrian Prisacariu. Dfnet: Enhance absolute pose regression with direct feature matching. In European Conference on Computer Vision , pages 1-17, 2022.

Shuai Chen, Zirui Wang, and Victor Prisacariu. Direct-posenet: Absolute pose regression with photometric consistency. In International Conference on 3D Vision , pages 1175-1185, 2021.

Ondrej Chum and Jiri Matas. Optimal randomized RANSAC. IEEE Trans. Pattern Anal. Mach. Intell. , 30(8):1472-1482, 2008.

Peter I Corke, Witold Jachimczyk, and Remo Pillat. Robotics, vision and control: fundamental algorithms in MATLAB , volume 73. Springer, 2011.

Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 224-236, 2018.

Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, and Marc Toussaint. Learning multiobject dynamics with compositional neural radiance fields. In Conference on Robot Learning , pages 1755-1768, 2022.

Xiao-Shan Gao, Xiaorong Hou, Jianliang Tang, and Hang-Fei Cheng. Complete solution classification for the perspective-three-point problem. IEEE Trans. Pattern Anal. Mach. Intell. , 25(8):930-943, 2003.

Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. In Conference on Robot Learning , pages 526-536, 2021.

Zhehao Jin, Jinhui Wu, Andong Liu, Wen-An Zhang, and Li Yu. Policy-based deep reinforcement learning for visual servoing control of mobile robots with visibility constraints. IEEE Trans. Ind. Electron. , 69(2):1898-1908, 2022.

Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In 2015 IEEE International Conference on Computer Vision , pages 2938-2946, 2015.

Obin Kwon, Jeongho Park, and Song Hyun Oh. Renderable neural radiance map for visual navigation. ArXiv , abs/2303.00304, 2023.

Xiaotian Li, Shuzhe Wang, Yi Zhao, Jakob Verbeek, and Juho Kannala. Hierarchical scene coordinate classification and regression for visual localization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11980-11989, 2020.

Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning , pages 112-123, 2021.

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: bundle-adjusting neural radiance fields. In 2021 IEEE/CVF International Conference on Computer Vision , pages 5721-5731, 2021.

Jianlin Liu, Qiang Nie, Y. Liu, and Chengjie Wang. Nerf-loc: Visual localization with conditional neural radiance field. ArXiv , abs/2304.07979, 2023.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , pages 405-421, 2020.

Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle. LENS: localization enhanced by nerf synthesis. In Conference on Robot Learning , pages 1347-1356, 2021.

Le Qi, Yu Zhang, Qingyu Yin, and Ting Liu. Ms-transformer: Introduce multiple structural priors into a unified transformer for encoding sentences. Comput. Speech Lang. , 72:101304, 2022.

Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In IEEE Conference on Computer Vision and Pattern Recognition , pages 12716-12725, 2019.

Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4937-4946, 2020.

Paul-Edouard Sarlin, Ajaykumar Unagar, Måns Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, and Torsten Sattler. Back to the feature: Learning robust camera localization from pixels to pose. In IEEE Conference on Computer Vision and Pattern Recognition , pages 3247-3257, 2021.

Torsten Sattler, Bastian Leibe, and Leif Kobbelt. Efficient & effective prioritized matching for large-scale image-based localization. IEEE Trans. Pattern Anal. Mach. Intell. , 39(9):1744-1756, 2017.

Yoli Shavit, Ron Ferens, and Yosi Keller. Learning multi-scene absolute pose regression with transformers. In 2021 IEEE/CVF International Conference on Computer Vision , pages 2713-2722, 2021.

Yoli Shavit, Ron Ferens, and Yosi Keller. Learning multi-scene absolute pose regression with transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2713-2722, 2021.

Haobin Shi, Lin Shi, Gang Sun, and Kao-Shing Hwang. Adaptive image-based visual servoing for hovering control of quad-rotor. IEEE Transactions on Cognitive and Developmental Systems. , 12(3):417-426, 2020.

Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew W. Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In 2013 IEEE Conference on Computer Vision and Pattern Recognition , pages 29302937, 2013.

Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison. imap: Implicit mapping and positioning in real-time. In 2021 IEEE/CVF International Conference on Computer Vision , pages 6209-6218, 2021.

Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomás Pajdla, and Akihiko Torii. Inloc: Indoor visual localization with dense matching and view synthesis. IEEE Trans. Pattern Anal. Mach. Intell. , 43(4):1293-1307, 2021.

Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04264 , 2023.

Julien P. C. Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip H. S. Torr, Shahram Izadi, and Cem Keskin. Learning to navigate the energy landscape. In Fourth International Conference on 3D Vision , pages 323-332, 2016.

F. Walch, C. Hazirbas, L. Leal-Taixé, T. Sattler, S. Hilsenbeck, and D. Cremers. Imagebased localization using lstms for structured feature correlation. In 2017 IEEE International Conference on Computer Vision , pages 627-637, 2017.

Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham. Atloc: Attention guided camera localization. In The Thirty-Fourth AAAI Conference on Artificial Intelligence , pages 10393-10401, 2020.

Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16611-16621, 2021.

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2642-2651, 2019.

Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, and TsungYi Lin. inerf: Inverting neural radiance fields for pose estimation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 1323-1330, 2021.

Kunwu Zhang, Yang Shi, and Huaiyuan Sheng. Robust nonlinear model predictive control based visual servoing of quadrotor uavs. IEEE/ASME Transactions on Mechatronics. , 26(2):700-708, 2021.

Qunjie Zhou, Sérgio Agostinho, Aljosa Osep, and Laura Leal-Taixé. Is geometry enough for matching in visual localization? In European Conference on Computer Vision , pages 407-425, 2022.

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. NICE-SLAM: neural implicit scalable encoding for SLAM. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12776-12786, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses visual localisation (i.e. the estimation of the\n6-degrees-of-freedom pose of a camera) from scene coordinate regression.\n\nThe problem tackled is the need for a large amount of pose and depth labels to\ntrain scene coordinate regression networks.\n\nThe paper proposes a solution to reduce the label requirements by producing\npseudo-groundtruth depth labels with a NeRF trained on the scene.\n\nThe lower data requirement comes at the cost of a lower accuracy of the scene\ncoordinate regression network. This is compensated by a pose refinement step\nbased on visual servoing: the coarse pose is refined until the view rendered by\nthe nerf on the estimated pose matches the target pose. More specifically, the\npose is estimated until the local features in the rendered view are close to\ntheir corresponding points in the target image. The pose updates are estimated\nto reduce the discrepency between these feature positions.\n\nThe method proceeds as follows:\n- Given a set of calibrated images of the scene, train a nerf.\n- Generate pseudo-groundtruth depth with the nerf.\n- Train the scene coordinate regression with the pseudo-groundtruth depth.\n- Infer a coarse pose from the trained scene coordinate regression network.\n- Refine the coarse pose using visual-based servoing \n\nThe method is compared against state-of-the-art localisation method from\nseveral categories: scene coordinate regression network based (e.g. DSAC++),\npose estimation network based (e.g. PoseNet, Direct-PoseNet,  DFNet) and\nfeature based (PixelLoc).\n\nSTRENGTHS:\nS1: The problem of reducing the data requirement to train scene coordinate\nregression network is an interesting problem.\n\nS2: The approach to use NeRF to produce pseudo-groundtruth 3D labels and the use of\nthis synthetic depth inside a localisation experiment is insightful.\n\nS3: The paper cites and compares against relevant works.\n\nWEAKNESSES:\nW1: The writing in general is good and one get the high-level description of the\ntechnical derivations. However, some parts are confusing which requires the\nreader to re-read a paragraph multiple times or to guess the technical\nderviation being described.  (See Limitations for detailed comments and\nsuggested updates). Also, there are several english typos that can easily be\ncaught with an automatic spell-grammar checker.\n\nW4: The paper claims that one advantage of the method is that it requires less\nlabeled data (calibrated images with depth) than other methods but there is no\nexperiments to support this claim (e.g. gradually reducing the number of\ntraining images and comparing the performance drop).\n\nQUESTIONS:\nQ1:  L146,147: what is the difference between $m_r$ and $n_r$? from reading, one\n  gets that $m_r$ is a pixel.\n\nQ2: L171: does the next ibvs run starting from the output pose of the previous\n  ibvs? Or do all the ibvs start from the coarse pose from the scene coordinate\n  regression network?\n\nQ3:  L187: where is $\\hat{n}^r$ used?\n\nQ4: L188: where is $d$ used?\n\nLIMITATIONS:\nFigure 1 would gain from having a technical description in addition to the\ncurrent caption. Something along the lines of: \"Left: The coarse pose estimated with\nthe scene coordinate regression network is refined with IBVS. Right: Illustration of the pose updates / navigation in the NerF scene.\"\n\nTechnical typos:\n- L4: \"require groundtruth 3D labels for supervision\": only regression-based\n  methods require 3D labels (i.e. depth). Structure-based appraoches do not.\n- L39: the reference [25] indeed uses nerf for data augmentation but [10,11]\n  uses nerf to refine camera poses with a render-and-compare strategy (10 with\n  photometric loss and 11 with a feature loss). This sentences need to be\n  updated accordingly.\n- Unsuported claim that the proposed method need less training images:\n  - L46: \"which is significantly fewer than typical visual localization method\".\n    This claim is not supported.\n  - L86-87: \"Our method ...\"\n  - L103: \"We utilize few posed images\"\n  - L241: \"It is important to note that we use fewer ...\"\n- The paper claims that RNR-Map [19] requires RGBD but the original paper\n  claims that it is based on RGB vision only. It is would be useful to have\n  this clarified.\n- Eq1: M is undefined even though one can guess that it is the number of points\n  along the ray\n- L116: $t_n$ is undefined\n- Eq2: Isn't the depth $D$ already defined in the camera coordinate frame? If\n  so, then there is no need to transform the points with $T^{-1}$ i.e. it\n  should be $P_W = D K^{-1} p$.\n- L146: \"depth of correspondences\" -> \"depth of pixels\"\n- Eq7 assumes that the set of correspondences remain the same for all IBVS,\n  which is an assumption only introduced later in 3.2.2. Instead, the\n  assumption should be mentionned before, even if only in a short sentence. The\n  reader can then be referred to 3.2.2 for more details.\n\n- L201: \"we can obtain accurate ...\": how are these correspondences derived?\n  one can guess that is with L144-147 but it would be helpful to the reader if\n  it was explicitly mentionned.\n\n- L226: Pixloc is actually a feature-based refinement only method so it does\n  not fully qualify as a structure-based method.\n\n- Fig4: e(x) ... are undefined even though one can guess that these are\n  distance errors\n\n\n\nConfusing writing that impedes with the understanding of the technical details:\n- L7: \"only a few\". It might be better to either give a quantification of how\n  much less or only write \"less\" images.\n\n- L8: \"To achieve this, we first use a few posed images with coarse 3D\n  labels provided by NeRF to train a coordinate regression network, which is\n  used to provide the coarse pose for unseen view. \"\n  It would be easier to read if written with a different order:\n  - pseudo ground-truth 3d labels are computed with nerf\n  - the scene coordinate regression network is trained\n  - a coarse pose is estimated from the regression network\n  - the coarse pose is refined with IBVS\n\n- L53: \"the correspondences with depth\": the reader can guess the derivation\n  bein run i.e. select pixels in one image and find their match in the second\n  image using the depth and the image poses. But the reader should not have to\n  guess the technical derivations so it is better to write it down explicitly.\n  - Same comment for L141: the \"depth correspondences\"\n\n- L61: \"navigation prior\" is undefined. Does this refer to the \"simulated\"\n  navigation run in the NeRF scene?\n\n- L125: \"geometric constraints\" is undefined\n\n- L127: Does \"edge areas\" refer to the scene's edge or the image's edge? One\n  could guess that it is the scene's edge but it is better to specify it.\n\n- L136: \n  - \"we first initialize IBVS ...\". At this stage, IBVS has not been\n    introduced and it is not part of the common knowledge in the visual\n    localisation community (this is more of a robotic common knowledge). So it\n    would be helpful to the reader to have a high level description of IBVS\n    before the low-level one that is already in the paper. This will help the\n    reader understand what \"initializing the IBVS\" means (e.g. \"The initial pose\n    in the IBVS optimisation is the coarse pose and the 3D points that will\n    support the optimisation are generated with the NeRF depth.\").\n  - the term pixel velocity is not introduced and it is not a term common in\n    the visual localisation community which is more used to the\n    render-and-compare notations than the visual servoing notations.\n\n- L168: this sentence is paradoxial. An alternative formulation could be: \"The\n  assumption is violated so accumulation errors will occur.\"\n\n\nA non exhaustive list of typos:\n- L26: the other line of approach *is* [...] regresse*s*\n- L75: \"To achieve ...\". It seems this sentence should not be there.\n- L76: has attracted\n- L103: \"Then the same ...\": this sentence misses a verb\n- Fig4: Deired -> Desired\n\n\n### Post-Rebuttal\n\n#### Addressed weaknesses:\nW3: The evaluation on 12 scenes compares only against methods that do not require\ngroundtruth depth labels, whereas the 7 scenes evaluation also compared against\nmethod that do use groundtruth depth labels.\n- Update: The paper does evaluate methods using ground-truth depth (only DSAC is missing in the 12 scenes experiments).\n\nW2: The paper claims to contribute not only to localisation but also navigation but\nthere is no comparison to previous navigation work (RNR-MAP). Also, the claim\nthat RNR-MAP requires depth information seems opposite to the RNR-MAP's claim\nto be RGB-vision based only (in the related work section).\n- Update: the comparison to rnr-map is not applicable so this limitation is obsolete.\nHowever, the contribution to the navigation task should be clarified.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel visual localization method combined with the NeRF technique to address the issue of training a large number of pose images in existing visual localization models. The proposed method has two advantages: (1) the paper trains a coordinate regression network using a few posed images with coarse 3D labels generated by NeRF. (2) The paper uses an image-based visual servo to explore 3D scenes for pose optimization.  The simulation experiments show superior performance in comparison with other methods.\n\nSTRENGTHS:\nThe proposed method, NeFT-IBVS uses a coarse-to-fine strategy to accurately localize the camera with a few posed images being adopted.  The method equips with two leading edges: (1) a NeRF-based posed images are used to train a coordinate regression network in the coarse stage. (2) an optimization algorithm is presented to effectively optimize the coarse pose.  Furthermore, the authors also design a new pipeline to reduce the rendering frequency of NeRF to speed up the pose optimization.\n\nWEAKNESSES:\n1. The authors claim the proposed method is able to localize the camera with fewer posed images, can the authors give a rigorous description of it? For instance, how many posed images can be used, or can the authors show the reduced ratio of used posed images compared to state-of-the-art methods?\n2. Can the authors give a comparison with DSAC method[1] on the 7-Scene dataset? It seems that the proposed method performs worse than it.\n[1] Brachmann, Eric and Carsten Rother. “Visual Camera Re-Localization From RGB and RGB-D Images Using DSAC.” IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (2020): 5847-5865.\n\n3. For statistical significance, the experiments should be conducted several times and the statistical significance of the results should be determined. Furthermore, some typos are necessary to be corrected. For example, “We” should be rewritten “we” in Line 103. “enables” should be rewritten “enable”. The writing is obligated to be further polished.\n\nQUESTIONS:\nSee Weakness part\n\nLIMITATIONS:\nDue to the poor rendering quality of NeRF in large outdoor scenes compared to indoor scenes, our method is mainly applied in indoor scenes. Moreover, due to the slow rendering speed of NeRF, the proposed method cannot achieve real-time performance.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a visual localization pipeline that uses NeRF and image-based visual servoing. The main contribution is that the proposed method uses much fewer images with pose annotations and no 3D ground-truth labels (only uses pseudo ground-truth provided by NeRF to train a coordinate regression network). The method first relies on the coordinate regression network to provide an initial coarse pose, which is then refined with image-based visual servoing.\n\nSTRENGTHS:\nThe paper presents a creative approach towards the visual localization problem. I believe the utilization of NeRF to learn a coordinate regression model and as pose initialization is novel, considering that prior works used NeRFs mostly for data augmentation. Furthermore, formulating the pose refinement process as a visual servoing problem that uses state-of-the-art image matching methods and NeRF is an interesting idea which fits well the problem. Overall, the paper proposes something new and useful to an important task.\n\nWEAKNESSES:\nI have a few concerns/comments. Not all of these are necessarily weaknesses.\n\nThere is discussion over iNeRF in the related work. Authors claim that limitations of iNeRF are that it requires a very good initial pose and relies on photometric loss which is sensitive to artifacts in the rendered images. This is somewhat true for the proposed approach as well as visual servoing, by definition, requires visual overlap between observation and query image and any image matching method is going to be affected by rendering artifacts. In fact, the paper acknowledges this fact and has an extra verification step (section 3.2.2) to filter out outliers from unreliable Superpoint+Superglue correspondences.\n\nIn my understanding using NeRF to initialize IBVS is analogous to the structured methods (such as [27,36]) retrieving a set of candidate database images to establish an initial pose(s). The proposed method mentions that only renders a single image C_r, but how do you ensure that C_r has visual overlap with the query? How often does this occur? Why not sample around the coarse pose T and generate multiple images?\n\nThe paper uses the phrase \"velocity s of the image coordinates\". Isn't that just the optical flow?\n\nIn terms of pipeline complexity, the method is actually closer to the structured methods (than the regression-based). I am curious to see runtime required for a single query compared to the baseline methods.\n\nCan you provide more details as to the number of posed images required to train the proposed method vs the baselines?\n\nWhat is the difficulty of the queries in the visual servoing experiments? i.e. what is the pose difference / overlap between initial and desired state? It would be interesting to divide the test set into easy/hard queries based on their pose diff / ovelap in order to show the limitations of the method. The authors do show the performance of the coarse pose estimation but it is not clear how sensitive IBVS is to that coarse estimate. \n\nMinor issue: Please include references of baselines in Table 1, and use \"/\" instead of \",\" to separate numbers.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors mention a couple important limitations in the main paper. I could not find discussion on potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of visual localization, i.e. estimating a camera pose from a query image. Approaches in the literature require a large number of posed images and even dense 3D supervision for some of them. Authors propose to leverage Neural Radiance Fields (NeRF) to solve the problem of visual localization while requiring fewer posed images and no 3D labels.\n\nThe method can be decomposed into sequential steps: (1) A NeRF model is trained from a set of posed images, (2) a coordinate regression network is trained on the posed images using depth predictions from the NeRF model to provide coarse 3D labels, (3) a coarse camera pose is predicted by the coordinate regression network from the query image, and is refined by performing Image-Based Visual Servoing. Camera pose refinement is turned into a navigation problem from the coarse to the target pose.\n\nThe paper compares the localization performance of the proposed method with different baselines of the literature on 2 datasets: 7-scenes and 12-scenes. Authors show that their method outperforms reported baselines trained without 3D labels and is on par with the ones trained with dense 3D supervision while using less training posed images.\n\nFinally, the authors show their NeRF-IBVS framework can also be used to perform camera navigation from an initial state to a target state, both specified as images.\n\nSTRENGTHS:\n1. The paper tackles an important problem: performing visual localization from fewer posed images (compared with current state-of-the-art methods) and no required 3D dense supervision.\n\n2. NeRF models are an interesting tool to perform visual localization and it is great to have papers studying this application.\n\n3. The paper considers a decent number of baselines from the literature in the experimental study. Ablation studies are also performed to evaluate the gain brought by the different proposed contributions.\n\nWEAKNESSES:\n1. [Major] Paper writing could be improved. The current version of the paper is fine when it comes to understanding the main contributions but several reads were needed to grasp the details of the work. I suggest authors go through the paper again and improve the coherence and writing.\n\n2. [Major] NeRF is a differentiable function mapping and thus, as done in previous work (iNeRF [43] as mentioned by authors in the paper), pose refinement can be performed by freezing the NeRF weights and optimizing camera location based on a rendering loss. It would be interesting that authors compare using IBVS and optimization as in iNeRF to perform the last pose refinement step, i.e. from the coarse to the target position. Optimizing camera pose based on NeRF rendering can be considered as a simpler alternative to the IBVS framework (e.g. no need to build the $L$ matrix).\n\n3. [Minor] This paper introduces a limited set of technical novelties. However, it tackles an important problem and nicely combines previous works, proposing simple yet efficient ideas to perform the target task. Thus, I consider this not an important issue with this work, but should still be mentioned in this review. For example, the authors mention they “design a new fast iteration strategy that reduces the rendering frequency of NeRF”. To the best of my knowledge, this is mainly about querying the NeRF model every n iterations, which might not be considered as a strong contribution. Authors might want to reconsider how they introduce this part of their work.\n\nQUESTIONS:\n1. [Major] Paper writing could be improved. Some sentences could be made smoother (e.g. some sentences are split in half while they should not). Writing could also be made more concise and efficient, as some parts are quite hard to follow. I am sorry to provide only vague directions here, but I have the feeling there is not one specific part of the paper to improve, but rather smoothness and conciseness should be improved globally to allow a better understanding. Technical sections (e.g. IBVS, correspondence selection) could be presented more progressively by defining some of the core notions early on and diving into the details more smoothly.\n\n2. [Major] An experiment involving a comparison between IBVS and iNeRF-like camera optimization in the last step, i.e. refinement from coarse to target pose, should be performed by authors.\n\nLIMITATIONS:\nAuthors mentioned two relevant limitations of their work:\n1. NeRF rendering quality can be low in large outdoor scenes, which might have an impact on the visual localization performance. Experiments are only conducted in indoor scenes in this paper.\n2. The low rendering speed of NeRF models prevents their solution from running in real time.\n\nBoth limitations rather come from the capabilities of current NeRF models than the proposed method. Future improvements in the field of NeRFs might allow better localization in large scenes and with faster runtime.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces image based visual servo to refine the coarse poses estimated using a coordinate regression network trained on neural rendering based 3D labels. The paper also uses correspondences in depth between neural rendered image and query image to iteratively refine the pose of the image. The paper also shows that NeRF-IBVS can be used as a navigation prior and subsequently use IBVS navigation.\n\nSTRENGTHS:\n1. The paper uses only a few images for visual localization compared to other methods out there \n2. The paper does both visual localization and navigation based on visual servoing. The authors claim to be the first to do that.\n3. The paper shows SOTA results on multiple datasets for both visual localization and navigation. \n4. The paper provides and effective navigation prior which improves IBVS based navigation.\n\nWEAKNESSES:\n1. The paper trains the coordinate regression network using the approximate 3D labels obtained through neural rendering which is not ideal. A NERF trained with a small number of posed images has a lot of rendering noise, as discovered by the authors and shown in Fig 3.\n2. It is not clear how the authors decide which pixels in the NERF-rendered images are used to train the coordinate regression net. They do mention that image boundaries are not used, but this is not a satisfactory description. Image boundaries from NERF may well be crisp, and parts of the image near the centre might have noise.\n3. The coordinate regression network name is a bit confusing. At first glance, it appears as if it is a network that regresses pose of the camera from the 2D image. Actually, it is regressing 3D positions for each image pixel in the global coordinate frame.\n4. The IBVS module is not clear. The authors suddenly introduce camera velocity and image velocity and a Jacobian without mentioning any prior information about this - which is presumably from the visual servoing literature. This material is provided in the supplementary, but should be included in the main paper.\n5. The method is not real-time due to the time taken by neural-rendering, during the optimization step\n6. The authors have not explored the joint training of NERF and the coordinate regression net, which might be an avenue to pursue.\n7. The Quality of NERF rendering is crucial even for getting coarse poses that are subsequently used to train the coordinate regression net, and the PnP on the 3D poses and the 2D images. This will struggle with non-Lambertian and dynamic objects, etc.\n\nQUESTIONS:\n1. The paper could use better neural rendering techniques for latency and accuracy of depth and image rendering.  \n2. No details about the RANSAC algorithm used.  \n3. Can the superglue correspondence network used be made trainable?\n\nLIMITATIONS:\n1. The paper may not be able to handle instances where there are a lot of specular objects in the scene. This can lead to bad coarse pose and in turn IBVS may not converge.\n2. The paper doesn't compare runtimes while comparing performance of other visual navigation methods. Some of the methods might have compromised on quality to reduce latency\n3.The authors claim IBVS iterations are launched no more than 4 times. How was this arrived at ? Again this really depends on the quality of rendering. Any evidence to support this will help.\n\nTypos:\n-'Deired' -> Desired in caption for Figure 4.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}