LENS: Localization enhanced by NeRF synthesis
Authors: Arthur Moreau
Nathan Piasco
Dzmitry Tsishkou
Bogdan Stanciulescu
Arnaud de La Fortelle
Venue: CoRL2021 Poster (2021)
Keywords: visual localization
camera pose regression
novel view synthesis

Abstract: Neural Radiance Fields (NeRF) have recently demonstrated photorealistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.

LENS: Localization enhanced by NeRF synthesis
Arthur Moreau 12, Nathan Piasco', Dzmitry Tsishkou', Bogdan Stanciuleseu', Arnaud de La Fortelle 'ooV team, Paris Research Center, Huawei MINES ParisTech, PSL University, Center for robotics arthur moreaudminns--aristech.
Figure 1: Novel view synthesis with LENS: given 3D scene defined by set of images labeled with corresponding poses (green cameras), we train a NeRF and render novel views (depicted im- ages) on pose queries distributed across the scene (red cameras). Synthetic and real images are gathered to train camera pose regression model, which performs twice better when evaluated on test images (blue cameras) compared the same model trained only real training samples.
Abstract: Neural Radiance Fields [1] (NeRF) have recently demonstrated photo- realistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrel- evant places selected virtual camera locations from NeRF internal representa- tion of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images data augmentation during training. At the time of publication, our approach im- proved state of the art with 60% lower eтror on Cambridge Landmarks and 7- scenes datasets. Hence, the resulting accuracy becomes comparable to structure- based methods, without any architecture modification or domain adaptation con- straints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribu- tion of data used for training on public benchmarks. We concluded that pose re- gression accuracy mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.
Keywords: visual localization, camera pose regression, novel view synthesis
1 Introduction
In order to localize a robot or a vehicle in a known environment, one solution is to use camera images with visual based localization algorithms [2, 3]. These methods extract information from the image to predict camera position and orientation. Best performing methods in this field are structure- based methods: they solve the problem with pipelines that include feature extraction, image retrieval and/or 2D-3D feature matching [4, 5, 6]. However, their high computational cost and memory footprint make them difficult to deploy in real-time embedded systems for robotic and autonomous applications. In this setup, an alternative solution is to train a deep neural network to regress poses from images in end-to-end supervised way [7]. These methods, named camera pose regressors,
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

require an offline training step while being more efficient at test time [8]. However, despite many architectural improvements [9, 10, 8, 11], the accuracy provided by these networks limited and highly dependent on quantity and diversity of training data [12].
Our hypothesis is that camera pose regression formulated limited because training datasets are highly biased and lead to poor generalization. In contrast with many tasks solved with deep learning, camera pose regressors are overfitted to a single scene and we expect their output space to be the set of possible camera poses in this scene. But in practice, training datasets for camera pose regression are often built from a limited number of consecutive video frames. Consequently, they lack diversity compared to the set of well distributed camera positions and orientations in a given scene (see figure 1). Sattler et al. [12] have shown that these networks are not able to extrapolate to unseen camera positions, and then struggle to perform better than an image retrieval baseline. We suppose that a training dataset which well distributed on the entire scene should help to overcome this limitation.
machine learning problem is currently
Our proposal is to augment training datasets for pose regression using Neural Radiance Fields (NeRF) [1]. These networks perform novel view synthesis thanks to color and density predictions along camera rays and volume rendering techniques. Compared to standard generative models [13] used for view synthesis. NeRF rendering is tailored for our localization problem as it produces geometry-consiee consisten images for any pose query in the scene thanks to the ray tracing approach. To optimize the pose regressor localization performances and mitigate the cost of images rendering, we propose an algorithm, named LENS, which generates virtual camera locations distributed on a regular grid across the scene. In order to avoid usage of degenerate views due to occlusions or wrong orientations, LENS leverages the internal scene geometry learned by the NeRF model. It allows to discard poses close to occluders like buildings, and provides only meaningful images to the pose regressor without any scene-specific parameter tuning. We apply our method on both indoor and outdoor datasets to show the benefit of spatially balanced dataset for training a pose regressor.
In the next section, we review previous work related to novel view synthesis for visual localization. In section 3 we describe our method, specifically the choice of NeRF and the synthetic poses gener- ation algorithm. Section 4 is dedicated experiments conducted on Cambridge Landmarks and scenes datasets. Section 5 concludes the paper.
Related work
Novel view synthesis can be used at several steps of a visual localization method. Zhang et al. [13] refine reference poses iteratively by comparing rendered view of the pose estimate with the original image. This idea can also be exploited in the relocalization step of a structure-based method: InLoc [14] verifies the predicted pose by comparing local patch descriptors between original and rendered image. iNeRF [15] performs gradient- based optimization to recover a pose estimate thanks to NeRF differentiaablitty. Direct-PoseNer [16] adapts this idea to camera pose regression training by using an additional photometric loss between query image and NeRF synthesis on the predicted pose. PoseGan [17] learns jointly pose regression and view synthesis resulting in improved localization accuracy.
Another direction is to use synthetic images to enlarge the reference database with more densely sampled views. Localization algorithms perform better when an image with a similar viewpoint is available for matching (structure based methods) or training (learning based). This idea can also be applied to other vision tasks, such as 3D semantic segmentation [18]. These methods yield two important design choices: how to chose where virtual cameras are located, and how to perform novel view synthesis.
Virtual camera locations are usually sampled using ad-hoc methods: Torii et al. [19] expand a place recognition database by sampling on a regular grid, removing locations that lie in buildings thanks to coarse depth plans representing basic 3D structures or locations that are too far from the original trajectory. Aubry et al. [20] and Irschara et al. [21] use a similar approach and discard cameras that doesn't 't view any portion of the 3d model. Some alternatives propose to use probabilistic sampling strategy based on detectability of keypoints at specific pose [22], or try to find an optimal set of lo- cations with genetic algorithm [23]. However, ad-hoc methods have been observed to be sufficient in practice [21], so our method uses a regular grid combined with NeRF internal representation of the scene to ensure meaningful views.
2

---

One popular approach to render novel view is to use 3D textured mesh to represent the scene and create virtual cameras within this scene [20, 12]. However, obtaining such scene representation is costly and not easy to compute from crowd-sourced images acquired in dynamic environment. Meshs built from crowd sourced images usually fail describe low textured or crowded part of the scene, such as sky and ground. Synthetic images can also be rendered thanks to Generative Adver- sarial Networks [22, 24]. Finally, recent methods that learn a continuous volumetric representation of the scene such Neural Radiance Fields [1] outperform prior work and exhibit photorealistic sults. To the best of our knowledge, no study has been conducted to enlarge camera pose regression training dataset with NeRF rendered views. We refer readers to the work of Shavit and Ferens [25] for further details about camera pose regression.
Synthetic dataset rendering with LENS
We provide here description of our method, named LENS, which can be seen offline data aug- mentation pipeline to train pose regressors. Our goal to generate a large and uniformly distributed dataset of synthetic images, using a small set of images depicting the scene labeled with reference camera poses (usually provided by structure from motion methods such as COLMAP [26]). This dataset is then used train a camera pose regressor, resulting an improved accuracy without ad- ditional computation during localization inference. Our framework is described in figure 2 and is detailed below.
a. Train NeRF-W
Density volume
c. Virtual camera locations texining poss density volume number of images
MLPs3
RGB
camera locations
Sendey
XYZ
XYZ
MIRE
RGB
MLPe,
tronnenn
Novel view synthesis cls)
pose regressor
Real time localisation
RGB
real
XYZ
syntiatic
LENS pipeline: First, NeRF-W is trained with available real images (a.). Then, the trained model is used to detect high density points in the scene (b.). Poses from real images and high density locations used generate virtual camera locations (c.) which novel view synthesis is performed (d.). Combined real and synthetic datasets are used to train a pose regressor (e.).
Figure 2:
3.1 NeRF-W training
neural radiance field learns a continuous 31) representation of a scene from a collections of im- ages depicting this scene from known camera viewpoints. It uses 2 feed-forward neural networks: MLPp, connects 3D position to a density value a and MLP,, predicts a RGB color from a view- ing direction d and latent vector MLPs, (x, y, 2). The final RGB value of pixel is computed by approximating the volume rendering integral using N, coarse samples and N, fine samples along light ray with predicted colors (ci)n and densities (a.)..
With this formulation, NeRF models are able to render a static scene captured under controlled set- tings but fail in real world dynamic scenes with variable illuminations and dynamic objects. NeRF in the Wild [27] (NeRF-W W), overcomes this limitation by modelling these temporal modifications with 2 learned latent spaces c(()) (appearance embedding) and c ((T) (transient embedding) that pro- vide control on the appearance and dynamic content of each rendered view. An additional network
MLPo, is introduced to render scenes with transient objects during training thanks to c(²), whereas MLP2, only render the static part of scene with c!a) as an additional input, see figure 2-a. We train a NeRF-W model each scene using sparse set of registered images.
3

---

3.2 Density volume generation
In order to chose valid locations for the synthetic images, we first gather the volumetric representa- tion of the scene learned by the NeRF-W model.
We query MLPa, on a regular 3D grid that can be displayed as density volume (see figure 2-b and figure 3). Lets consider B the smallest 3D bounding box that contains all the poses of the training images. The 3D grid extend is defined by another 3D bounding box B4 obtained by extending B with an extrapolation distance parameter Emax- We sample m 3D points in B separated by a constant distance X that is obtained by dividing the smallest edge B by a fixed resolution parameter re. Using this method ensure that will be of the same order of magnitude for all scenes, avoiding the generation of intractable density volume.
To consider that given location is unreachable in the scene, we set threshold lo and only take in account 3D points with density higher than l--
3.3 Virfual camera locations
Next, virtual camera location generation algorithm takes poses of real training images input, but also the NeRF density volume and the desired number of virtual cameras n. Our method is two- step: defining the virtual camera positions and then determining their orientations. Our focus is to generate a training dataset for a pose regressor, therefore we want a set of locations to be uniformly distributed all over the area and viewpoints that could be visited by the agent to localize.
To generate our virtual camera positions candidates, we use similar strategy as the one described in the previous section: we sample n; 3D points in B4, using constant distance X between the 3D points obtain by dividing the smallest edge of B by a resolution parameter Tú, E N. Then proceed multiple criteria pruning step to remove irrelevant candidates: 3D points closer to da to a 3D point from the density volume are considered too close a structure and are discarded, 3D points further than dmaa real camera view position considered too far from the of interest and are also discarded. We implement the nearest neighbour search with KDTrees for efficiency.
After candidate pruning, we obtain a final number of virtual camera positions vig, If rig is smaller than desired number of cameras n, we update the resolution parameter and repeat the generation procedure: r++1 = T: + or, with o, the update step.
Finally, we need to define camera orientation (gx, Qy qa (w) attached to the camera center (z, 2) for each virtual camera. order avoid degenerated views, we copy the orientation of the nearest pose in training set and add small random perturbation where 0 is the maximum amplitude of the perturbation. As an output, we have a set of n poses (zr, y, 2, Oz Gy I= Qwd that are used as queries for novel view synthesis, see figure 2-c. More exam- ples of generated poses can be find in figure 3.
0 each axis drawn uniformly in [- 2 8
3.4 Novel view synthesis
Novel views are synthesised on each virtual camera location using MLP3, and MLPp, (figure 2-d). The appearance embedding ((a) is chosen by random interpolation of training set appearances.
As result, rendered images depict scene from chosen viewpoints where transient occluders ob- served in training images are removed. Appearance interpolation acts as a data augmentation tech- nique that increases robustness of the localization model under varying illuminations. Some example of resulting appearances are shown in figure 3 and detailed in supplementary materials.
3.5 Camera pose regressor training
The final step of our pipeline is to train the localization algorithm with our synthetic-augmented dataset. We simply combine real images with our synthesised images together; Then mini-batcher are sampled randomly in this combined dataset for training stage (figure 2-d). We use CoordiNet [8]
as a camera pose regressor. CoordiNet is a CNN that uses inductive biases in the network architec- ture that facilitate geometrical reasoning for the localization task, and also provides an uncertainty quantification for each pose that can be used as covariance matrix.

---

Shoo
Kings
Hospital
Church
Figure 3: Visualisation of density volumes, virtual camera queries (training poses, test poses, virtual cameras) and example of rendered images on Cambridge Landmarks.
4 Experiments
In this section evaluate LENS combined with CoordiNet standard localization benchmarks, conduct an ablation study to confirm our design choices and investigate on training poses distribution for pose regressor training.
4.1 Datasets and implementation details
Datasets. We evaluate LENS standard visual localization benchmarks:
Cambridge Landmarks [7] contains outdoor dynamic scenes captured by a smartphone. In each scene, common building is visible from each image. Camera poses recovered from StM and training sets contains between 200 and 2000 images extracted from videos. We downscale input images to 640x360 pixels.
7scenes [28] consists of 7 static indoor scenes, captured by Kinect RGB-D sensor. This dataset is challenging for camera pose regression algorithms because test sequences follow different paths than the ones of train sequences. We downscale input images to 320x240 pixels.
NeRF parameters. We use the code of [29] train NeRF-W [27] by scene, using the default parameters of the Pytorch implementation. We train each model for 20 epochs using the training images generate the training rays. We took maximum of 5 training sequences (5k images) for the 7 scenes dataset and we define one appearance embedding by image in the training set. Generating images of Cambridge scenes takes approximately 40s/GPU, with 256 coarse and 256 fine sampling by ray. It takes 7s/GPU for 7 scenes while sampling only 128 coarse and 128 fine values as the scenes are smaller and less complex than Cambridge scenes.
Virtual camera locations generation. outdoor scenes, virtual camera locations are sampled on 2D plane and LENS parameters are: 128, 1, 20, dmaa 8m, d = 1m, Emazx = 1m,

---

ro = 1, 11,0, = and 0 = 15". For indoor scenes, the regular grid is defined on 3 dimensions and we adapt the following LENS parameters: dmaa 50cm, d, 20cm, Emar 20cm and 20"
We set the desired number virtual views n. to 500% for outdoor scene and 1000% for indoor scene of the total number of real training images. We found that amount of images good trade-off between computational cost and localization accuracy. More details are provided in section 4.3.
Pose regressor. We use CoordiNet [8] as camera pose regressor with EfficientNet-b33 [30] as back- bone and optimize the network by maximizing heteroscedatic log- ligeliil (see [8]). We train our models for 250 epochs with a fixed learning rate of le 4 for both datasets. During training, we use a batch size of 10 for Cambridge Landmarks and 40 for scenes.
4.2 Comparison with related localization methods
Competitors. We compare our method CoordiNet + LENS with CoordiNet only trained on real images. SPP-Net [22] small CNN pose regressor that has been trained with and with- out additional synthetic data, providing a good baseline for LENS. We also report results from TransPoseNet[11] which is state of the art transformer approach DirectPoseNeet [16] uses NeRF model as a photometric supervisor during the training. Active search [3] a baseline for structure-based methods, where local image features are matched against a 3D point cloud obtained by SfM.
camera pose regression.
Dataset
Camera Pose Regression
CPR + synthesis
3D
Cambridge
SPPNet
CoordiNet
TransPN
DirectPN [16]
SPPNet +
CoordiNet
Active
[22]
[8]
[11]
synth [22]
LENS
scarch [31]
College
1.91/2.4
0.70/0.9
0.60/2.4
0.74
0.33 /0.5
0.42/0.6
OldHosp
2.51/3.7
0.97/2.1
1.45/3.1
2.18 f
0.44 /0,955
0,44/1.0
Shop
1.31/7.8
0.69/3.7
0.55/3.5
0.59
0.27% /1.6
0. 0.12/0.4
Church
3.21/7.0
1.32/3.6
1.09/5.0
/3.33
0.53 /1.6
0.19/0.5
Average
2.24/5.2
0.92/2.6
0.91/3.5
/2.77
0.39
0.29/0.6
7scenes
Chess
0.22/7.6
0.14/6.7
0.08/5.7
0.10/3.5
0.12
0.03
1.037"/1.3 /1.3
0.04/2.0
Fire
0.37/14.1
0.27/11.6
0,24/10.6
0.27/11.7
08.9""
0.10 /3,7
0.03/1.5
Hcads
0.22/14.1
0.13/13.6
0.13/12.7
0.17/13.1
08..3
0.07
0.02/1.5
Office
0.32/10.0
0.21/8.6
0.17/6.3
0.16/6.0
05.65
0.09/3.6
Pumpkin
0.47/10.2
0.25/7.2
0.17/5.6
0.19/3.9
0,21 042.9%
0.08 /2.2
0,08/3.1
Kitchen
0.34/11.3
0.26/7.5
0.19/6.8
0.22/5.1
0.21 /4,858
0.09 /2.2
0.07/3.4
Stairs
0.40/13.2
0.28/12.9
0.30/7.0
0.32/10.6
0.72
0.14 /3.67
0.03/2.2
Average
0.33/11.6
0.22/9.7
0.18/7.8
0.20/7.3
0.18
0.08 13.0
0.05/2.5
Table I: 6DOF localization errors of visual localization methods. We report median transla- tion/orientation error (meters/degrees). TransPN and DirectPN stand for TransPoseNet and Direct- PoseNet. Superscripts numbers refer to the relative improvement (green) or deterioration (red) in percentage brought by synthetic data.
Results. Median translation and orientation errors are reported in table CoordiNet + LENS achieves best reported results for camera pose regression methods on both Cambridge Landmarks and 7scenes with large margin. Moreover, our method is more accurate than Active Search [31] on 5 scenes out of 11, reducing the gap between camera pose regression and structure- based meth- ods. Then, we observe that LENS provides better relative localization improvement (approximately +60%) than the synthetic datasets generated by Purkait al. [22] (+45%). Visualizations and com- parison of resulting trajectories are provided supplementary materials.
4.3 Ablation study
Synthetic dataset size. We change the amount of generated sample from 100% 1000% (up to 5000% for fire scene) of the total number of images in the real training set three different scenes and we compare the relative improvement in term of localization accuracy. Results are shown in table 2.

---

Syuth,
ShopPacade
Church
Fire
emee /wwelioeeeenneee ciliio--rrrr (t2)
CITe booeitee STEAE (argg)
Real im
(231 im.)
(1487 im.)
(2000 im.)
0%
0.6174.2
1.06/3.1
0.27/11.6
100 %
0.61/2.4
0.75/2.4
0.18/5.7
200 %
0.41/2.0
0.58/1.9
0.16/5.7
500 %
0.26/1.2
0.51/1.5
0.12/4.2
Cheel
1000 %
0.25/1.2
0.45/1.6
0.10/3.2
5000 %
0.07/2.4
real
syath
Freal
Figure 4: Translation (tr., left) and angular (ang., right) error relative decrease vs synthetic dataset size.
Table 2: median translation (m) and orienta- tion (") errors depending on synthetic dataset size
As expected, we observe that using higher grid resolution containing more samples leads to a better localization. In addition that, we can in figure 4 that the relative improvement compared to the baseline (i.e. no synthetic images) is correlated to the ratio between synthetic and real images rather than the total number of images itself: ratio of 10 leads for ShopFacade, 58% for Church and 63% for Fire. Curves of error translation in figure 4 do not seems reach a plateau: an higher ratio would potentially bring better localization results.
59% translation improvement
Benefit of using volume and random appearances. In table 3, we observe that occluded views located too close or inside building (shown in supplementary materials) disturb the training of the pose regressor, decreasing the localization accuracy. We can see in figure 5 that the use of the density volume provided by the trained NeRF correctly remove distracting samples. Providing a random appearances embedding during the NeRF image generation also decrease slightly translation and rotation errors. Even the improvement is minor and not very significant on this scene, we expect this appearance augmentation to be very useful on experiments with more diversity in illuminations (day and night images for example).
Volume
Appearances emb.
Errors
with
without
random
constant
Translation
0.27m
0.36m
0.25m
0.26m
Rotation
1.6
1.7
1.2
1.4
Figure 5: virtual camera locations with (left) and without (right) NeRF volume pruning step.
Table 3: Localization errors comparison for density vol- ume and appearances embedding ablations on Shop Fa- cade scene.
4.4 Exploring camera pose regression on synthetic domain
In order to investigate the impact of training camera poses distribution the performance of pose regression without taking in account the domain gap between real and synthetic data, we perform the following experiment:
replace both training and testing real images by NeRn-rendered images the exact same location, then we trained and test a pose regressor on this synthetic dataset. Results reported for the indoor scene fire in the first columns of table 4 (seq3-4 real vs seq3-4 synth.). We get similar localization performances compared to the same model trained on real data, which allow perform deeper analysis the impact of distribution of camera poses used for the training.
Method
soq3-4
seq3-4
LENS
LENS
LENS
Act. Search [31]
Data type
real
synth.
synth.
synth.
synth.
real
Train size
2k
2k
10k
100k
Error (m/")
0.27/11.7
0.27/10.6
0.08/3.5
0.05/2.4
0.03/14 4
0.03/1.5
Table 4: Median translation (m) and orientation (") emror in Fire scene from 7scenes, seq3-4 refers to methods using images from sequences 3 & training data.
second experiment, we replace the original training camera poses by uniformly distributed poses generated by LENS. We observe important decrease in median localization enror from 27cm/11.77 8cm/3.5 whereas we use the same number of training data (table columns 2 vs 3). Increasing the number of synthetic data, up to 5000% of the number of training data in the original training set,
7

---

leads to even better localization results (table 4 columns to 5). We also show that pose regression can reach structure- based method accuracy (evaluated on real data) if the model is trained with the largest amount of data. A similar experiment had been made by Sattler et al. [12] with opposite conclusions: even with a big synthetic dataset depicting the scene, pose regressors were not able to reach an accuracy comparable to structure-baser methods, suggesting that camera pose regression algorithms are inherently limited by their approach without geometrical constraints.
From these experiments, we end up with different conclusion: datasets using only video sequences create an unbalanced regression problem where training labels does not cover the entire set of pos- sible poses and then lead to poor localization accuracy. Deep learning approaches are known to perform well in high data regimes, but the main finding of this research is that training datasets dis- tributed across the entire scene are crucial for camera pose regression performances. The different results we observe compared to [12] probably come from a higher quality of our synthesized views and pose regression architecture that performs better.
4.5 Limitations
The first limitation of our pipeline is the offline computation time: Nerf-W needs be trained several days with a GPU on single scene in order to reach optimal rendering results. The slow rendering of novel views forces us to generate a dataset before training the localization algorithm instead of an online data generation pipeline. Faster synthesis would enable to learn from an infinite quantity of data instead of a limited number of images. However, faster NeRF training [32] and rendering [33, 34] are active research fields that should enhance speed and performances of LENS in the future. Moreover, as reported in Table 5, our method offsets the costly offline computation by fast and light online localization, enabling real-time embedded applications for robotics.
We also observed that training with only synthetic images leads to poor performances real im- ages. This is due to the domain gap between real and synthetic images: no dynamic objects, different textures, more blur and some artifacts are observable the synthetic domain. Mixing real and syn- thetic images is the simpler way to mitigate this issue, however domain adaptation techniques [35] could be used to reduce domain discrepancy well as higher quality rendered samples.
Offline scene)
Online (1 image)
Task
SIM
Quantization
SIFT
Loc.
Structure based
Timing
Hours
Minutes"
50-500ms
~ 500ms
Memory
NR
NR
NR
GBs *
Task
SIM
Train NeRF
NVS
Train PR
Coordinet inference
LENS
Timing
Hours'
Hours
30s/im/gpu'
Hours
~ 50ms
CoordiNet
Memory
NR
NR
NR
NR
50MB
Table 5: Approximate time and memory requirements comparison between structure-based methods and ours. * denotes a scaling time and memory consumption according to the scene size, structure- based methods need load the features vocabulary and access the 3ID points cloud during local- ization.
5 Conclusion
In this paper, we address limitations of camera pose regressors at data-level: thanks to high qual- ity novel view synthesis provided by NeRF-W, we propose train a relocalization algorithm with synthetic images uniformly sampled on the entire scene. Our method enhance strongly localization performances, reducing the gap with structure based methods while keeping the advantages of pose regression: a fast inference with low memory footprint and the capability to scale to large environ- ments. Our experiments show that camera pose regression can perform well when trained with large and diverse datasets. LENS coupled with CoordiNet improves camera pose regression state of the art and can be used for accurate real-time robot relocalization system. As novel view synthesis active research field, we expect to improve LENS with faster rendering and higher quality samples as future direction.
8

---

References
[1] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the Euro- pean Conference on Computer Vision, 2020.
[2] N. Piasco, D. Sidibé, C. Demonceaux, and V. Gouet-Brunet. A survey on visual-based lo- calization: On the benefit of heterogeneous data. In Pattern Recognition, volume 74, pages 90-109. Elsevier, 2018.
[3] T. Sattler, W. Maddern, C. Toft, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Oku- tomi, M. Pollefeys, J. Sivic, F. Kahl, and T. Pajdla. Benchmarking 6DOF Outdoor Visual Localization in Changing Conditions, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8601-8610, 2018
[4] T. Sattler, B. Leibe, and L.. Kobbelt. Efficient effective prioritized matching for large-scale image-based localization. In JEEE Transactions on Puttern Analysis and Machine Intelligence, volume 39, pages 1744-1756, 2017. doi: 20000000000A.A0001166666111
[5] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4938-4947, 2020.
[6] P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, Kahl, al. Back to the feature: Learning robust camera localization from pixels to pose. In Proceedingy of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[7] A. Kendall, M. Grimes, and R. Cipolla. PoseNet: A convolutional network for real-time 6- DOF camera relocalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 2938-29989999 2946, 2015.
[8] A. Moreau, N. Piasco, D. Tsishkou, B. Stanciulescu, and A. de La Fortelle. Co- ordiNet: uncertainty-aware pose regressor for reliable vehicle localization. In arxiv preprint, arxiv:2103. 10796, 2021.
[9] S. Brahmbhatt, J. Gu, K. Kim, Hays, and J. Kautz. Geometry- aware learning of maps for camera localization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
[10] A. Kendall and R. Cipolla, Geometric loss functions for camera pose regression with deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6555-6564, 2017.
[11] Y. Shavit, R. Ferens, and Y. Keller. Paying attention to activation maps in camera pose regres- sion. In arxiv preprint, arxiv:2 11477 2021.
/477,
[12] T. Sattler, Q. Zhou, M. Pollefeys, and L. Leal-taix. Understanding the Limitations of CNN- based Absolute Camera Pose Regression Chalmers University of Technology. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3302-3312, 2019.
[13] Z. Zhang, T. Sattler, and D. Scaramuzza. Reference pose generation for Long-term visual localization via learned features and view synthesis. In International Journal of Computer Vision, volume 129, pages 821 821-844, 2020.
[14] H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii. InLoc: Indoor visual localization with dense matching and view synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7199-7209, 2018.
[15] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. iNeRF: Inverting neural radiance fields for pose estimation. In arxiv 2ttiiiviai2::00050//7777 2020.
[16] S. Chen, Z. Wang, and V. Prisacariu. Direct-PoseNet: Absolute pose regression with photo- metric consistency. In arXiv preprint arXhv:2104.04073, 2021.

---

[17] K. Liu, Q. Li, and G. Qiu. PoseGAN: A pose-to-image translation framework for camera localization. ISPRS Journal of Photogrammetry and Remote Sensing, volume 166, pages 308-315. Elsevier, 2020.
[18] A. Kundu, X. Yin, A. Fathi, D. Ross, B. Brewington, T. Funkhouser, and C. Pantofaru. Virtual multi-view fusion for 3D semantic segmentation. In European Conference on Computer Vision, pages 518-535. Springer, 2020.
[19] A. Torii, R. Arandjelović, J. Sivic, M. Okutomi, and T. Pajdla. 24/7 place recognition by view synthesis. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogni- tion, pages 1808-1817, 2015, doi: 10.1 1O..1///....01...22.2.
[20] M. Aubry, B. C. Russell, and J. Sivic. Painting-to-3D model alignment via discriminative visual elements. In ACM Transactions on Graphics (ToG), volume 33, pages 1-14. ACM New York, NY, USA, 2014.
[21] A. Irschara, C. Zach, J.-M. Frahm, and H. Bischof. From to fast location recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2599-2606, 2009, 10::10.1 1tt.1/////VPPR22000500000
point clouds
[22] P. Purkait, C. Zhao, and C. 7ach. Synthetic view generation for absolute pose regression and image synthesis. In Proceedings of the British Machine Vision Conference, page 69, 2018.
[23] S. Chen and Y. Li. Automatic sensor placement for model-based robot vision. In IEEE Trans- actions on Systems, Man, and Cybernetics, Part B (Cybernetics), volume 34, pages 393 493-408. IEEE, 2004.
[24] A. Toker, Q. Zhou, M. Maximov, and L. Leal-Taixé. Coming down to earth: Satellite-oo-stre view synthesis for geo-localization. In arXiv preprint arXiv:2/03.068/8, 2021.
[25] Y. Shavit and R. Ferens. Introduction to camera pose estimation with deep learning. In arXiv preprint arXiv: 1907005277 2019.
[26] J. L. Schönberger and J.-M. Frahm. cnucture-from-motie revisited. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2016.
[27] R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duck- worth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[28] J. Shotton, B. Glocker, C. Zach, S. Izadi, A. Criminisi, and A. Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2930-2937, 2013.
[29] C. Quei-An. Nerf pl: a pytorch-lightming implementation of NeRF. 2020. URL https: wewthu..com/teeeeeeeeeeee hwom///ea113//ee/...e // //githu.....
[30] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning, pages 6105-61 14. PMIR, 2019.
[31] T. Sattler, B. Leibe, and L. Kobbelt. Improving image-based localization by active correspon- dence search. In Proceedings of the European Conference on Computer Vision, 2012.
[32] M. Tancik, B. Mildenhall, T. Wang. D. Schmidt, P. P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2021.
[33] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. FastNeRF: High-fidelity neural rendering at 200FPS. In arXiv preprint arXiv:2103. 10380, 2021.
[34] C. Reiser, S. Peng, Y. Liao, and A. Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In arXiv preprint arXiv:2103.13744 2021.
[35] C. Cortes and M. Mohri, Domain adaptation in regression. In International Conference on Algorithmic Learning Theory, pages 308-323. Springer, 201
10


Reviews:
[{"review_id": "oZzzLiSt9Bh", "reviewer": "Reviewer_BBFW", "summary": "The paper presents an approach named LENS as an offline data augmentation technique for camera pose relocalization problems. A common mode of failure for camera pose relocalization approaches is the varying distribution between the training set and the test set which prevents the model from being able to reasonably extrapolate the pose information for a newly observed image. This work addresses this problem by proposing to leverage novel view synthesis as an offline intermediate step to generate synthetic views of the scene. A pose regressor method is then trained on a combination of the real and synthetic dataset demonstrating a substantial performance improvement. Several ablation experiments were conducted offering insights towards the underlying problem and the proposed solution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written, it follows a clear line of thought from problem to proposed solution. It is indeed enjoyable to read.\n- The proposed solution addresses an important problem and a weakness for a large number camera pose relocalization methods.\n- The results demonstrate that the use of synthetic images results achieving the same performance as  3D-based methods.\n\nWeaknesses / Questions:\n- The idea of generating virtual views for training is not entirely novel. See [1]. While the application area differs, the work should also be cited.\n- How are poses close to occluders discarded? \n- For the virtual camera location generation, how were the values of d_max and d_sigma selected? And how does one know if the pose is \"too close to a structure\" or \"too far away\"?\n- Current sampled virtual views are quite dense. Are they all required for the pose regressor to perform well? Instead of rendering on an evenly spaced grid can one generate virtual views in a more concise pattern to achieve the same results?\nWhat is your hypothesis behind the contradictory results in table 4 between your results and the results from Sattler et al? Is it due to the better distribution of the training images that the model can better capture the structure? Are the training images different here and in Sattler et al?\n- What are the failure cases for the localization? In the video at 01:26 there's a much larger gap between the model predictions and the ground-truth. What was the cause of that?\n\n\n[1] Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru. Virtual Multi-view Fusion for 3D Semantic Segmentation. In ECCV 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the proposed solution of generating synthetic views of the scene and using them for training may not be novel in its entirety, this is the first approach demonstrating that using views generated from NVS methods can be beneficial for the trained model while backing it up with ablation experiments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses / questions above.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "eA19xaB7JXJ", "reviewer": "Reviewer_CXrf", "summary": "This paper proposes a method for pose regression that makes use of neural\nradiance fields (NeRFs) as an intermediate representation. In the proposed\nmethod, first a NeRF model of a scene is constructed from a set of images with\nknown reference poses (e.g. as provided by a structure from motion pipeline).\nThe NeRF model can be queried at a variety of poses to produce synthetic images,\nwhich are subsequently used (in conjunction with the original training data) to\ntrain a pose regression network. Finally, at test time, the pose regressor\nnetwork can be queried online with new images to infer poses. The key insight\nbeing leveraged by the proposed approach is that NeRF models can be used to\nproduce (in principle) an arbitrarily large amount of good enough synthetic\nimages to offer clear improvements to the accuracy of the pose regression\nmodule. Pruning heuristics are provided to ensure the selected query viewpoints\nhave \"good\" geometric properties for pose regression. Experimental results\nsuggest not just that the approach improves over existing pose regression work,\nbut also that the pruning heuristics are important for mitigating degenerate\nimages obtained from the NeRF model.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Strengths\n\nThis is a well-written and organized paper, and implicit neural map\nrepresentations are definitely relevant.\n\nConceptually, I found this to be a really interesting approach. The approach of\ntraining a pose regression module offline using the rendered output of a NeRF\nmodel for subsequent use in an online fashion nicely circumvents the issue of\nslow rendering with present NeRF techniques (at the expensive of a\ntime-consuming offline training process).\n\nI hadn't thought about the issue of pruning degenerate images, but it is\ninteresting that this offered a measurable (albeit small?) improvement in the\naccuracy as in Section 4.3.\n\nAs a remark, there is a neat analogy between the overall approach here and the\nclassical approach to Monte Carlo localization (e.g. in Probabilistic Robotics\nby Thrun et al.). In practice, rather than computing an inverse sensor model at\nrun-time, one will pre-compute a cached representation of the model for lots of\npossible observations and look up / interpolate between cached observations\nonline (as in Chapter 8.2.3 of Probabilistic Robotics). It seems like this\napproach is essentially doing something similar: where a method like iNeRF (ref.\n[15]) would regress directly from the NeRF model to an SE(3) pose, the proposed\napproach densely queries the NeRF model to construct an implicit intermediate\n\"cached\" representation to approximate what would otherwise be the\nmaximum-likelihood estimate provided by an iNeRF style model, but which is\nfaster to query than the NeRF model itself.\n\nThe experimental results for the paper were very thorough. I particularly\nappreciated the ablation study and the supplementary results (+ video!). Very\nnice!\n\n## Weaknesses\n\nWhile the paper does a good job demonstrating improvements over prior work on\npose regression, it's not made explicit in the paper as presented what\nadvantages the proposed approach confers in comparison to a method like iNeRF\n(ref. [15]), where the NeRF model is used to directly infer the camera pose\nrather than training an intermediate pose regressor (let me know if I am missing\nsomething, though). One thought is that this approach should be faster\n(specifically at test time) than iNeRF, since only the pose regressor is used at\nruntime (thereby eliminating explicit runtime dependence on the NeRF model), and\nCoordiNet itself is capable of running at ~ 18 Hz. This is a distinction that I\nreally feel should be made more clear; to me, this would be the most blatant\nadvantage of this method with respect to those techniques!\n\nMore broadly, the approach of querying a NeRF model from lots and lots of poses\nseems a bit \"brute force.\" From my reading, the goal is essentially to translate\nfrom one implicit scene representation (the NeRF model) to another (the pose\nregression). I wonder if this could be done in a more direct way. For example,\nwhile the related work section suggests that this sort of \"ad hoc\" method for\nview selection is sufficient, maybe an alternative approach could help reduce\nthe number of views needed for training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending a \"weak accept\": I think this is a really interesting\ncontribution, and has some potential advantages over existing NeRF-based\nlocalization and current pose regression work. To improve the paper, I recommend\nclarifying the contribution with respect to strictly NeRF-based localization and\nadding a bit to the discussion of scalability (see Issues).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "\n## Clarifying Contribution\n\n- The contribution should be clarified a bit (see Strengths and Weaknesses) to\n  clarify the distinction between this approach and similar work and potential\n  advantages. It seems like the fact that this method can be faster (I am\n  inferring this based on the run time of CoordiNet) than strictly NeRF-based\n  pose inference is undersold relative to the result that the approach is more\n  accurate than pose regression without the NeRF model. Both seem (to me) to be\n  important.\n\n## Experimental Results\n\n- It would be helpful to additionally report test-time computation speed for the\nmethod (not necessarily for each dataset, etc, but even just a mention in the\ntext to give an idea); it seems this should be similar to CoordiNet, but good to\nreport this in any case, as it is a possible advantage relative to other\nNeRF-based localization methods.\n\n## Questions and Comments\n\n- Most of the examples considered contain scenes where the camera motion is\n  constrained to be more-or-less planar. I could see the data generation process\n  for this approach becoming impractical for large scale scenes with 3D camera\n  motions (e.g. for images from aerial vehicles); I'd be curious about the\n  authors' thoughts on the scalability of the method to these types of\n  environments (Section 4.5 mentions faster NeRF training and rendering, but the\n  number of query images for this approach still seems to scale with the\n  dimension of the environment - let me know if I'm misunderstanding something).\n\n- In Figure 1, it might help to mark which of the images shown are training\nimages and which are synthetic (it wasn't entirely clear to me from the arrows).\n\n- Some more detail on the training process (e.g. cost function used, etc.) would\nbe helpful.\n\n- Rotations in Table 3 use a decimal comma, whereas in Table 2,4 they use a\n  decimal period. Table 3 uses centimeters, other tables use meters (this is OK,\n  but I don't see a reason not to just keep the units the same?)\n\n- Some references are not formatted correctly (e.g. \"inerf\" -> \"iNeRF\") or\nmissing information e.g. [8], [11], [15], and others.\n"}}, {"review_id": "-hbxLZV-azr", "reviewer": "Reviewer_HedN", "summary": "This paper proposes to use images synthesized by NeRF as augmented data to improve camera pose regression. Specifically, authors find that deciding where to render images based on NeRF's density volume can further reduce the error since it can avoid views that are too close/far from the scene. Experiments show that training the same pose regression method with synthetic dataset generated by NeRF can dramatically reduce the error.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe idea is simple and sound, I like it!\nThe performance boost is huge.\nThe ablation study on the synthetic dataset size is a useful reference for the literature. The results clearly show that regression-based methods can work well when the data is abundant.\n\nWeaknesses: \nThe procedures of training NeRF and generating images are time-consuming. However, as mentioned by the authors, they may be addressed as the NeRF literature develops.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend accepting the paper because the idea of using NeRF for data augmentation is sound, the proposed strategy for placing virtual cameras is shown to be helpful, and the performance improvement is large. The paper can be more clearly written based on the issues I point out below but overall it is already complete.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Can you attach the memory footprint and runtime of both CoordiNet[8] and Active search [30]? Currently, the text only says they are slow but it would be nice to show the actual comparison.\n\n-  I think Figure 3 can be further improved by 1) making the camera queries' colors more obvious and 2) adding (left, middle, right) in the caption to emphasize the meaning of different columns.\n\n- I don't think equation 1 is really needed. The text description \"... random interpolation of training set\" is already clear.\n\n- Line 246, “is train with” -> “is trained with”.\n\n- Line 267, “artefacts” -> “artifacts”.\n\n- Line 113, “see figure 2-a” maybe can be moved to line 112 because it shows how the models are wired instead of the registered images.\n\n- Line 128 “the poses,” seems to be redundant.\n"}}]

Author rebuttal:

Meta-review: The reviews are generally positive. The AC endorses the reviewers' recommendation. The authors are encouraged to address questions and concerns raised in the reviews.
Decision: Accept (Poster)