SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanze Wang $^{1,2*}$  Yichao Yan $^{1*}$  Dianxi Shi $^{1,2\dagger}$  Wenhan Zhu $^{1}$  Jianqiang Xia $^{2,3}$  Tan Jeff $^{3}$  Songchang Jin $^{2}$  Ke Gao $^{4}$  Xiaobo Li $^{4}$  Xiaokang Yang $^{1}$
$^{1}$ MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University  $^{2}$ Intelligent Game and Decision Lab (IGDL), Beijing, China
$^{3}$ Tianjin Artificial Intelligence Innovation Center  $^{4}$ Alibaba Group
{yz.wang,yanyichao,zhuwenhan823,xkyang}@sjtu.edu.cn
dxshi@nudt.edu.cn, jianqiang.xia@foxmail.com, jsc04@tsinghua.org.cn
{xiaobo.lixb,gaoke.gao}@alibaba-inc.com
Visual localization is a fundamental task in computer vision and robotics. Training existing visual localization methods requires a large number of posed images to generalize to novel views, while state-of-the-art methods generally require ground truth 3D labels for supervision. However, acquiring a large number of posed images and 3D labels in the real world is challenging and costly. In this paper, we present a novel visual localization method that achieves accurate localization while using only a few posed images compared to other localization methods. To achieve this, we first use a few posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. Then a coarse pose is estimated from the regression network with PNP. Finally, we use the image-based visual servo (IBVS) with the scene prior provided by NeRF for pose optimization. Furthermore, our method can provide effective navigation prior, which enables navigation based on IBVS without using custom markers and the depth sensor. Extensive experiments on 7-Scenes and 12-Scenes datasets demonstrate that our method outperforms state-of-the-art methods under the same setting, with only  $5\%$  to  $25\%$  training data. Furthermore, our framework can be naturally extended to the visual navigation task based on IBVS, and its effectiveness is verified in simulation experiments.
The estimation of camera positions and orientation based on a query image is a fundamental task in computer vision. It has wide applications in various fields such as robotics, virtual reality, and autonomous driving. Existing works that tackle this task can be generally divided into two categories. The first line of approaches are structure-based [6, 28, 31, 37, 46], which first establish correspondences between 3D points and 2D pixels, and then compute camera pose with PnP [16] solver utilizing RANSAC [12]. These methods typically achieve satisfactory results at the cost of a complex pipeline. The other line of approaches is regression-based [2, 3, 40, 21, 42, 41], which either regress the 3D coordinates or directly regress the absolute pose. These methods yield a more compact framework and have fast inference performance. All the above methods require a substantial number of posed images (generally thousands of posed images per scene) to guarantee the generalization ability to novel viewpoints. Moreover, state-of-the-art methods [6, 4, 21] also require dense ground truth 3D labels for supervision, such as depth and 3D model. However, acquiring a large number
Figure 1: Demonstration of the proposed NeRF-IBVS. Our method can achieve visual localization based on a single query image. Moreover, our method can obtain navigation prior after processing the images of the initial state and desired state, which can better enable IBVS-based navigation.
of posed images and dense 3D labels is challenging and costly. It naturally raises a question: is it possible to design a framework that achieves accurate localization results while only requiring a small number of posed images?
Recent advances in Neural Radiance Fields (NeRF) [25] provide a promising solution. Trained from hundreds of posed images, NeRF can render realistic images and give coarse depth maps for novel viewpoints as prior knowledge of 3D scenes. Intuitively, this prior knowledge can assist visual localization methods in generalizing to novel viewpoints. Current NeRF-based visual localization methods [10, 11, 26] only employ NeRF for data augmentation purposes, which still requires a substantial number of images for training. The estimated pose will be coarse when using a few data. Considering this issue, it would be desirable if the coarse pose in NeRF could be further refined to meet the need for visual localization.
In this paper, we propose a novel visual localization framework called NeRF-IBVS, which achieves accurate localization while using only a few posed images. NeRF-IBVS adopts a coarse-to-fine paradigm to estimate the pose of the query image. In the coarse stage, we first train a NeRF with hundreds of posed images, which is significantly fewer than typical visual localization methods. Then, we use the same posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. The coarse pose is estimated from the regression network with PNP. In the refining stage, we design an optimization algorithm to effectively optimize the coarse pose. Motivated by the image-based visual servo (IBVS) [9], which navigates the camera from the initial pose to the desired pose using coordinate error and depth of correspondences between the current image and the target image. Therefore, we first use NeRF to render the image and depth corresponding to the current coarse poses. Then, we establish the correspondences between the rendered image and the query image and query the approximate depth of the correspondences based on the rendering depth map. Finally, the correspondences with depth are used to launch IBVS to guide pose optimization.
Due to rendering quality errors, directly applying IBVS to NeRF-rendered images and depth is not feasible. Moreover, IBVS requires non-collinear correspondences during 6-degree-of-freedom navigation. In order to achieve robust pose optimization, we design a correspondence selection algorithm using spatial information provided by NeRF and the coordinate regression network. To speed up the pose optimization, we further design a strategy that reduces the rendering frequency of NeRF. Further, the effective 2D and 3D correspondences obtained by NeRF-IBVS can be used as the navigation prior to enhance IBVS-based navigation. Specifically, this navigation prior can enable IBVS-based navigation without using custom markers and the depth sensor. This advantage greatly expands the application scope.
To the best of our knowledge, NeRF-IBVS is the first method that can accomplish both visual localization and visual navigation based on IBVS as shown in Figure 1, providing a novel idea for pose optimization by controlling camera motion. Employing only  $5\%$  to  $25\%$  of the posed images required by other localization methods on 7-Scenes dataset and 12-Scenes dataset, our method outperforms state-of-the-art methods without using ground truth 3D labels and has comparable performance with methods using ground truth 3D labels. Furthermore, our method can enable visual navigation based on IBVS without using custom markers and the depth sensor, which greatly expands the application scope of IBVS-based navigation. We also verified the effectiveness of visual navigation in a simulation environment.
Visual localization. Structure-based methods [6, 28, 31, 37, 46] use complex pipelines to achieve precise localization. Firstly, correspondence is established between 3D coordinates and 2D pixels. Secondly, a camera pose is computed through a PnP solver with RANSAC. The first stage requires a complex pipeline consisting of image retrieval, feature extraction, and matching. To achieve a more compact frame compared to structure-based methods. In recent years, regression-based localization methods [2, 3, 40, 21, 42, 41, 24] attract increasing attention, which either regresses the 3D coordinates or directly regress the absolute pose. Specifically, coordinate regression methods, such as DSAC++ [7] and HACNet [21] use neural networks to learn the first stage of the structure-based methods, which directly regress 3D scene coordinates from an image. These coordinates regression methods typically use ground truth 3D labels as supervision to achieve better localization performance. Unlike coordinate regression, absolute pose regression methods [19, 40, 2] generally use only posed images to train the neural network. MS-Transformer [32] and AtLoc [41] use attention mechanisms to guide the regression process and have substantially improved the performance of absolute pose regression methods. Recently, some methods [10, 11, 26] use NeRF as data enhancement to improve the performance of visual localization methods. The above methods require a substantial number of posed images to guarantee the generalization ability to novel viewpoints. Our method can achieve accurate localization while using only a few posed images compared to other localization methods.
Robotics with neural radiance fields. Recently, the neural radiance field (NeRF) [25] has shown great potential in robotics applications [22, 17, 15]. iNerf [44] fixes the network weights of NeRF and directly optimizes the camera pose parameters by computing the backpropagation of the photometric loss. However, this method requires an initial pose near the ground truth pose. Methods based on photometric loss tend to fail when blur and artifacts appear in the rendered images. iMAP [36] and NICE-SLAM [47] are a pose estimator in the SLAM setting that requires continuous sequential data. BARF [23] optimize camera poses during the training of NeRF. Therefore they cannot estimate the pose of a single query image. Moreover, NeRF can provide coarse scene geometry, which is useful for navigation tasks. NeRF-nav [1] designs a smooth and collision-proof navigation strategy based on the density provided by NeRF. RNR-Map [20] achieves precise visual navigation performance based on NeRF, but is limited to three degrees of freedom and requires RGBD pictures and odometer information. Our method can accomplish both visual localization and 6 degrees of freedom navigation based on visual servo and using only RGB images.
NeRF. NeRF uses MLP network  $F_{\theta}$  to encode the 3D scene, which predicts the RGB color  $\mathbf{c} \in \mathbb{R}^3$  and volume density  $\sigma \in \mathbb{R}$  for each input 3D point  $\mathbf{p} \in \mathbb{R}^3$  and its view direction  $\mathbf{d} \in \mathbb{R}^3$ . It can be summarized as  $F_{\theta}(\mathbf{p}, \mathbf{d}) = (\mathbf{c}, \sigma)$ , where  $\theta$  is the parameters of MLP network. Then, NeRF uses volume rendering techniques to generate realistic rendered images  $C(r)$  and coarse depth maps  $D(r)$  from any viewpoint:
$$
C (\mathbf {r}) = \sum_ {i = 1} ^ {M} T _ {i} (1 - \exp (- \sigma_ {i} \delta_ {i})) c _ {i}, D (r) = \sum_ {i = 1} ^ {M} T _ {i} (1 - \exp (- \sigma_ {i} \delta_ {i})) t _ {i}, \tag {1}
$$
where  $\mathbf{r}$  and  $M$  indicates the rendering ray and the number of points along the ray,  $c_{i}$  and  $t_i$  indicates the color and distance of the samples in the rendering ray,  $T_{i} = (-\sum_{j = 1}^{i - 1}\sigma_{j}\delta_{j})$  indicates the accumulated transmittance from the near bound of the rendering ray to  $t_i$ , and  $\delta_{i} = t_{i + 1} - t_{i}$  is the distance between adjacent samples in rendering ray.
IBVS. The aim of Image-Based Visual Servoing (IBVS) is to control the camera to move toward the desired pose based on vision information while minimizing the coordinate error of image features between the current image and the target image. Correspondences are used as image features in this paper. To achieve this goal, IBVS first calculates the desired image coordinate velocity of correspondences based on the coordinate error. Then, the Jacobi matrix is constructed based on the correspondences, which establishes the relationship between the image coordinate velocity and the camera velocity. Finally, the desired camera velocity to control the camera motion towards the target is solved based on the Jacobi matrix and the desired image coordinate velocity. The specific details of the IBVS are presented in the supplemental materials.
Figure 2: The pipeline of our method, NeRF-IBVS. In the coarse pose estimation, we first train a NeRF with hundreds of posed images, which is significantly fewer than typical visual localization methods. Then, we use the same posed images with coarse pseudo-3D labels provided by NeRF to train a coordinate regression network. The coarse pose is estimated from the regression network with PNP. In the pose optimization, the correspondences with depth between the rendered image and the target image are obtained to launch IBVS to guide pose optimization. Moreover, our method can provide effective navigation prior, which enhances navigation based on IBVS.
Given a few posed images compared to the typical localization method, we aim to achieve accurate localization. To achieve this, we first use a few posed images with pseudo-3D labels provided by NeRF to train a coordinate regression network. Then the coarse pose is estimated from the regression network with PNP. Finally, we introduce IBVS to gradually refine the coarse pose by leveraging the 3D scene prior provided by NeRF. Furthermore, our method can provide effective navigation prior, which enhances navigation based on IBVS. Figure 2 demonstrates the overall pipeline.
4.1 Coarse Pose Estimation
We use a few posed images with pseudo-3D labels provided by pre-training NeRF to train a coordinate regression network. Then the coarse pose is estimated from the regression network with PNP. Specifically, we convert the pixel coordinates  $\mathbf{p}$  of training images into pseudo-3D labels  $\mathbf{P}_{\mathbf{w}}$  using depth  $D$  provided by NeRF:
$$
\mathbf {P} _ {\mathbf {w}} = D \mathbf {T} ^ {- 1} \mathbf {K} ^ {- 1} \mathbf {p}, \tag {2}
$$
where  $\mathbf{K}$ ,  $\mathbf{T}$  denote the internal parameter and pose of camera. Following common settings [43], these pseudo-3D labels  $\mathbf{P}_{\mathbf{w}}$  are then normalized to fit within the range of a unit cube. Then, we use the advanced coordinate regression component of GDR-Net [42] as our coordinate regression network, which is trained by pseudo-3D labels  $\mathbf{P}_{\mathbf{w}}$ . To estimate coarse pose, the coordinate regression network
processes the query image to generate 3D coordinates and establishes a 2D-3D correspondence. This 2D-3D correspondence is then fed into PnP [16] to determine the coarse pose  $\hat{\mathbf{T}}$  of the query image. We empirically found that the lack of multi-view constraints in the boundary regions of the scene leads to a significant depth error in image edges. To mitigate the impact of the rendering quality error, we exclude the edge areas in the image of our pseudo-3D labels  $\mathbf{P}_{\mathbf{w}}$  when training our coordinate regression network.
4.2 Pose Optimization
To optimize the coarse poses, our method views pose optimization as the process of navigating the camera from the coarse pose towards the target pose using scenes prior provided by NeRF.
4.2.1 IBVS Module
IBVS initialization using NeRF. To start pose optimization, we first initialize IBVS using NeRF. This process mainly initializes the Jacobian matrix  $\mathbf{L}$  of IBVS using the image coordinates and depth of correspondences, which establishes the relationship between the velocity  $\mathbf{v}_c$  of the camera and the velocity  $\dot{\mathbf{s}}$  of image coordinates:
$$
\dot {\mathbf {s}} = \mathbf {L} \mathbf {v} _ {c}, \tag {3}
$$
where the  $\mathbf{L}$  corresponds to per correspondence can be formulated as:
$$
\mathbf {L} = \left[ \begin{array}{c c c c c c} - \frac {1}{Z _ {c}} & 0 & \frac {x}{Z _ {c}} & x y & - (1 + x ^ {2}) & y \\ 0 & - \frac {1}{Z _ {c}} & \frac {y}{Z _ {c}} & 1 + y ^ {2} & - x y & - x \end{array} \right], \tag {4}
$$
where  $x, y$  and  $Z_{c}$  denote image coordinates and depth of the correspondence in the currently rendered image. To initialize the Jacobian matrix  $\mathbf{L}$ , we first utilize NeRF to render an image  $C_r$  and depth map  $D_r$  that corresponds to the current coarse pose  $\hat{T}$ . Then the off-the-shelf correspondences establishment algorithm, such as SuperPoint [14] with SuperGlue [29], is employed to extract the pixel coordinate  $(\mathbf{m_q}, \mathbf{m_r})$  of correspondences between the query image  $C_q$  and the rendered image  $C_r$ . The depth  $Z_{c}$  of correspondences  $\mathbf{m_r}$  can be obtained by querying the rendered depth  $D_r$ . Image coordinates  $\mathbf{n_r} = (x, y)$  in the rendered image can be obtained using the pixel coordinates  $\mathbf{m_r}$  and the internal parameter  $\mathbf{K}$  follow [9]. Finally,  $\mathbf{n_r}$  with depth  $Z_{c}$  are used to initialize  $\mathbf{L}$ .
Optimization iteration. After the IBVS initialization, we started IBVS iteration to navigate the camera from the coarse pose toward the target pose. At each iteration, IBVS needs to calculate the velocity  $\mathbf{v}_c$  of the camera to update the pose. To control the 6 DOF velocity  $\mathbf{v}_c$  of the camera, four non-collinear correspondences are selected and the velocity  $\dot{\mathbf{s}}$  of the image coordinates are set to  $-\lambda (\mathbf{n_q} - \mathbf{n_r})$  follow [9]. According to Equation (3), we can obtain the camera velocity  $\mathbf{v}_c$  as:
$$
\mathbf {v} _ {c} = - \lambda \mathbf {L} ^ {+} \left(\mathbf {n} _ {\mathbf {q}} - \mathbf {n} _ {\mathbf {r}}\right), \tag {5}
$$
where  $\lambda, \mathbf{L}^{+}$ , and  $\mathbf{n_q}$  denote proportional factor, generalized inverse matrix, and image coordinates in the query image. Then, we update the current pose by integrating the camera velocity  $\mathbf{v}_c$  during unit iteration to obtain the variation of pose  $\Delta \mathbf{T}(\mathbf{v}_c(t))$ . The coarse poses are updated as follows:
$$
\hat {\mathbf {T}} (t + 1) = \hat {\mathbf {T}} (t) \Delta \mathbf {T} \left(\mathbf {v} _ {c} (t)\right). \tag {6}
$$
The IBVS iteration proceeds until either the pixel coordinate error of correspondences is less than a certain threshold or the maximum number of iterations is reached.
Theoretically, each iteration of IBVS requires NeRF rendering to update the parameters of the Jacobian matrix  $\mathbf{L}$ . However, utilizing NeRF rendering for each IBVS iteration is highly time-consuming and impractical. Therefore, we design a new fast iteration strategy that reduces the rendering frequency of NeRF. We first calculate the 3D coordinates of correspondences using the depth information provided by NeRF, while assuming the 3D coordinates of the correspondences are precise. At each iteration, We project the 3D coordinates of correspondences to the image plane by using the updated poses. This projection can be an approximate substitute for the correspondence in the rendered image. In this way, we skip the time-consuming NeRF rendering process, and NeRF rendering is only executed at the initialization of IBVS.
Due to rendering errors of NeRF, the assumption that the 3D coordinates of correspondences are accurate is violated, so accumulation errors will occur. Accumulated errors may lead to the optimization terminating with a large error. In order to eliminate accumulated errors, we render the image
again based on the most recently updated poses to relaunch IBVS iterations. Typically, the IBVS iterations are launched no more than four times to obtain a precise localization.
The first IBVS iterations launch aims to move the camera a significant distance toward the target pose, while the subsequent IBVS iterations launch only fine-tune the pose around the target pose. To improve the optimization efficiency, We dynamically set the number of iteration steps. Specifically, the number  $N(i)$  of iterations steps in the  $i$ th IBVS iterations launch is set dynamically based on coordinate error of correspondences:
$$
N (w (i)) = \left\{ \begin{array}{l} N _ {1}, w (i) > 1 \\ \frac {N _ {1}}{3}, w (i) <   \frac {1}{3} \\ w (i) N _ {1}, \text {o t h e r w i s e} \end{array} , \text {w h e r e} w (i) = \frac {\left\| \mathbf {n} _ {\mathbf {r}} ^ {i} - \mathbf {n} _ {\mathbf {q}} ^ {i} \right\| _ {2}}{\left\| \mathbf {n} _ {\mathbf {r}} ^ {1} - \mathbf {n} _ {\mathbf {q}} ^ {1} \right\| _ {2}}, \right. \tag {7}
$$
where  $N_{1}$  and  $\mathbf{n_r}^i$ ,  $\mathbf{n_q}^i$  denote the fixed number of iteration steps in the first IBVS iteration launch and image coordinates of correspondences in the  $i$ th IBVS iterations launch, respectively.
4.2.2 Correspondence Selection
The Jacobian matrix  $\mathbf{L}$  of IBVS is highly sensitive to the selection of the correspondences, and it requires accurate and non-collinear correspondences to update parameters during iteration. Due to rendering errors of NeRF, which may cause inaccurate correspondences. In order to increase the robustness of IBVS, we designed a correspondence selection algorithm to select appropriate correspondences for IBVS. We first query coarse 3D coordinates estimated by the coordinate regression network to obtain 3D coordinate  $\hat{\mathbf{P}}_{\mathbf{q}}$  of correspondences  $\mathbf{m_q}$  in the query image. Then, we use coarse pose  $\hat{\mathbf{T}}$  to project the 3D coordinate  $\hat{\mathbf{P}}_{\mathbf{q}}$  onto the rendered image to obtain image coordinates  $\hat{\mathbf{n}}_{\mathbf{r}}$ . Finally, we can obtain the coordinate distances of correspondences  $d = \| \hat{\mathbf{n}}_{\mathbf{r}} - \mathbf{n}_{\mathbf{r}}\| _2$ . Since the coarse pose  $\hat{\mathbf{T}}$  and 3D coordinates  $\hat{\mathbf{P}}_{\mathbf{q}}$  near the target, the accurate correspondences generally have a smaller distance. Therefore, we empirically set a threshold  $\tau$  for the coordinate distance  $d$  to filter out a large number of outliers. Finally, we use RANSAC [12] to further select accurate and non-collinear correspondences. Please see the supplementary material for more details.
4.3 Visual Navigation
NeRF-IBVS can provide navigation prior which enhances IBVS-based navigation. To enable IBVS navigation, it is necessary to use correspondences that are non-collinear and remain in the camera's field of view during navigation. However, acquiring such correspondences is challenging, current methods [18, 45, 34] usually rely on custom markers which limit the applications of IBVS. The navigation prior provided by our method can solve this problem. Specifically, We first use NeRF-IBVS to locate the initial pose of the navigation task. Then the initial pose is viewed as a coarse pose to start the NeRF-IBVS locating the target state of the navigation task. After that, the correspondence selection algorithm can obtain accurate and non-collinear correspondences and the IBVS module can obtain navigation trajectories in the image plane. The current correspondences are not yet guaranteed to remain in the camera's field of view during navigation. Benefiting from the navigation trajectory, the appropriate 2D correspondences that fall outside of the image plane are automatically filtered out. It ensures that the correspondences selected remain in the camera's field of view during navigation. We backproject 2D correspondences based on depth provided by NeRF to obtain coarse 3D coordinates as 3D correspondences. During the navigation iteration, the 3D correspondences are projected using the current pose to obtain the depth prior and the image coordinates prior. The image coordinates prior can assist in locating and associating the 2D correspondences in the current image, which enables IBVS-based navigation without custom markers. Moreover, the depth prior can be used to update the parameters of the Jacobian matrix in each iteration, which enables IBVS-based navigation without using a depth sensor. Although there is a small error in the depth obtained by projection, IBVS has the advantage of being robust to small depth errors[13].
In this section, we first compare the proposed NeRF-IBVS with state-of-the-art visual localization methods. Furthermore, our method extended to IBVS-based navigation without using custom markers and the depth sensor, and its effectiveness is verified in simulation experiments.
5.1 Visual Localization