SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The dataset is released under the CC-BY license. The accompanying code that shows the dataset use is released under the Apache License 2.0. The authors of the dataset bear all responsibility in case of violation of rights.
All contents of this submission (code, paper, data) can be accessed from our project page: https://navidataset.github.io. The data is hosted on Google Cloud by Google Research. The authors will maintain and update the dataset.
Figure 12: Common problems when aligning on Internet images. Small differences on the shapes introduce significant noise to the alignments. Additional parts (first, second, and last pairs), and slight differences in the manufacturing process (different colors on the first, and thicker wheels, texture mis-alignment on the third pair) are very common issues when aligning from internet images.
Extensive documentation of the dataset and how to use it for the various tasks can be found in the accompanying Github repo: https://github.com/google/navi. Users are invited to use the included Jupyter notebook NAVI Dataset Tutorial.ipynb for a quick start.
The released dataset consists of multiple folders with images (jpg), scans (obj, mtl, glb) and annotations (json) that connect them. Users can download the dataset at https://storage.googleapis.com/gresearch/navi-dataset/navi_v1.tar.gz (29GB).
The intended usage of the dataset is to enable benchmarking and systematic development of the 3D vision tasks presented in the main paper as well as this supplemental: 3D from multiview image collections, 3D from in-the-wild image collections, correspondence estimation, and 3D from a single image.
B 3D from multiview image collections
B.1 Comparison to existing multiview (MV) datasets
Table 7 compares NAVI (multiview part) to existing datasets of real images. While there exist datasets with much larger number of images, NAVI uniquely combines three different aspects: Very detailed 3D annotations, very high quality camera poses, and diversity of environments for a single object.
Regarding the type of 3D annotations, Objectron [1] is a large dataset that is annotated with 3D boxes. Some datasets [48, 72, 26] provide point clouds that cover parts of the objects, meshes that result from Multi-View Stereo (MVS) [70], or depth maps [9]. Closer to our work, [18] and [28] contain high quality scans resulting from a structured light scanner, and a laser scan, respectively. However, these works do not provide 2D-3D alignments of objects in real images. Regarding the quality of camera poses, most works use Structure-from-Motion (SfM) techniques like [52] for their camera poses [43, 1, 48, 26, 72, 70]. YCB [9] provides ground-truth quality poses from an RGB-D camera but was captured solely in a lab environment. Regarding diversity of environments, NAVI is the only dataset that includes scenes shot with different camera models (smartphones, consumer grade point-and-shoot, professional grade DSLR cameras, See Table 6) where the exact same object instance is shown in different illumination settings.
NAVI combines all three characteristics, and offers full  $360^{\circ}$  coverage of all objects in real environments together with high-quality 2D-3D alignments to detailed object scans. This makes it an ideal dataset for evaluating models trained on large datasets, and for training/evaluating scene-based optimization techniques such as methods of the NeRF [43] family.
Dataset	# Images	Type of Data	Camera
Poses	Multi
Camera	Multi
Environment	3D Annotations	3D←2D
Alignment
NeRF LLFF [43]	288	MV images	SfM [52]	X	X	-	X
Objectron [1]	4M	videos	AR session	✓	X	3D box	(✓)
CO3D [48]	1.5M	360° videos	SfM [52]	X	X	point cloud	-
DTU MVS [26]	40.1k	MV images	SfM	X	✓	point cloud	✓
MVIImgNet [72]	6.5M	180° videos	SfM [52]	(✓)	X	point cloud	✓
Blended MVS [70]	17.8k	MV images	SfM	X	✓	MVS mesh	✓
YCB Object Set [9]	46,2k	360° RGB-D	GT	X	X	depth	✓
Tanks&Temples [28]	153,4k	360° videos	-	X	laser scan	X
GoogleScan [18]	100k	-	-	-	-	scanned mesh	-
NAVI (Ours)	8217	360° images	Near-GT	✓	✓	scanned mesh	✓
Table 7: Comparison with real multiview (MV) datasets. We compare statistics for existing real multiview datasets. NAVI (multiview part) provides high quality 3D scans and alignment for objects captured in diverse environments using different camera models.
Figure 13: Novel view synthesis from multiview reconstructions. We compare test examples from runs on selected scenes initialized with COLMAP reconstructed camera poses to NAVI GT pose initialization. For each configuration an InstantNGP [45] instance is optimized.
B.2 COLMAP initialization details
Only 148 scenes of the 369 multiview scenes of NAVI could be completely reconstructed using COLMAP [52] with default parameters that are usually used in the context of NeRF [43] reconstructions. 155 scenes could be partially reconstructed with an average of  $69\%$  ( $\pm 34\%$ ) of the views registered. For the multiview collections included in the evaluation we randomly selected 5 scenes with more than 10 images for each of the objects. On those scenes COLMAP [52] successfully registered  $73\%$  ( $\pm 32\%$ ) of the views, on average. We only evaluate on the views of the validation set that were successfully registered by COLMAP, which gives the experiments initialized from COLMAP cameras an advantage.
B.3 Visual results
Figure 13 illustrates qualitative results for novel view synthesis from multiview reconstruction using InstantNGP [45]. We compare the camera poses obtained by COLMAP with the ground-truth camera poses of NAVI. We observe some artifacts like the oversmoothed and noisy contours, especially on the COLMAP variant. These artifacts can be attributed to slightly offset camera poses, as well as the relatively small number of images in certain scenes for NeRF-like methods. We add a small amount of distortion loss as proposed by [4] to reduce the risk of floater artifacts. Additional regularization might be beneficial for some scenes to further improve results.
C 3D from in-the-wild image collections
C.1 Details of evaluated techniques
In the following, we briefly summarize the different techniques we analyzed for 3D from in-the-wild image collections on NAVI. For more details, please refer to the respective paper.
- NeROIC [31] proposes a multi-stage approach to reconstruct geometry and material properties from online image collections of objects. Camera poses are initialized with a COLMAP-based pipeline and fine-tuned during the first reconstruction stage which is followed by a normal extraction stage to estimate high-quality surface normals. Finally, material properties and illumination are estimated to enable relighting in addition to novel view synthesis.
- NeRS [73] introduces Neural Reflectance Surfaces that constrain reconstructions using a surface-based representation. Starting from manually annotated rough initial poses and a template mesh the objects are decomposed into a surface mesh, illumination and surface reflectivity as albedo and shininess. We define the dimensions of an initial cuboid that approximates the object's bounding box for each scene as suggested by [73].
- SAMURAI [8] enables reconstruction of a NeRF representation and decomposes appearance into illumination and a physically based BRDF based on a differentiable renderer for pre-integrated lighting [7]. Camera poses are initialized as quadrants from manual annotation and then refined using a multiplex and a coarse-to-fine scheme. We obtain the initial directions from the GT poses and use them to initialize both NeRS and SAMURAI.
- GNeRF [41] employs an adversarial approach and a pre-trained inversion network for camera pose and shape optimization from completely unknown cameras. GNeRF is the only method presented here that does not account for the different lighting settings which is also reflected in worse performance on more challenging scenes.
Error distribution: Figure 14b and 14d visualize the combined mean scores of PSNR and SSIM for the view synthesis task from SAMURAI and NeRS for all in-the-wild scenes. We observe that the difficulty of the NAVI scenes varies, as indicated by the fluctuating scores. By analyzing the results, we notice that the most challenging scenes across methods are the ones featuring symmetric objects, such as water guns and a hand drill. Our intuition is that symmetric objects of complicated shapes pose an additional challenge for these methods.
Figure 14a and 14c show the normalized view error and camera pose error over the views in one example scene. It is a common artifact that some cameras still show a high error after the optimization, probably being trapped in a local minimum. Interestingly, both NeRS and SAMURAI have a similar camera pose error distribution despite the differences regarding their 3d representation and camera optimization strategy. It's also important to note that large errors of individual cameras do not necessarily result in a failed shape reconstruction as long as there are enough correctly aligned views.
View synthesis: Figure 15 presents additional view synthesis results from NeROIC [31], NeRS [73] and SAMURAI [8] for selected test views. NeROIC is tailored towards high quality view rendering results which can be achieved with good initial poses. When initialized further away from the GT poses the lack of additional camera regularization leads to failing reconstructions. NeRS is able to robustly reconstruct all objects in most settings but shows limited quality for more complex shapes. Poses are generally less precise and scaled differently compared to the ground truth. SAMURAI is tuned towards large camera updates in the beginning of the training and therefore is able to reconstruct even from large initial pose offsets. Rendering quality is limited by the used illumination model. Results often show some smoothness, also a result of small pose errors that can not be compensated by the neural representation due to the explicit rendering step.
Direct 3D Shape evaluation
NAVI enables directly evaluating 3D shape reconstruction by using the provided 3D scans that are aligned on the images.
(a) Camera and view rendering errors over the test views for NeRS [73]
(b) Mean view rendering score over all inthe-wild-scenes averaged from SAMURAI and NeRS results.
(c) Camera and view rendering errors over the test views for SAMURAI [8]
(d) Mean inverse camera error over all in-the-wild scenes averaged from SAMURAI and NeRS results.
Figure 14: Error distribution over scenes and views. Left we show the distribution of normalized view errors (inverse PSNR and SSIM) and camera pose errors (translation and rotation) over the test views of the "Keywest Showpiece" in-the-wild scene for two reconstruction methods, NeRS and SAMURAI. Lower values are better. On the right the normalized view score (SSIM and PSNR) and the normalized inverse camera error (translation and rotation) are given for all in-the-wild scenes in NAVI. Higher values are better. We report the average of SAMURAI and NeRS results.
Method	Initialization	Mean Distance↓	IoU↑
NeRS [73]	Quadrants	5.86	45.1%
SAMURAI [8]	Quadrants	3.65	63.7%
Table 8: Shape evaluation from in-the-wild images. The scanned 3D shapes provided by NAVI enable the evaluation of the reconstructed shape independently of the camera poses and rendering quality. We compare the point cloud extracted on the surface of the predicted mesh to points sampled on the ground truth mesh by using the Chamfer distance. We also measure Intersection-over-union (IoU) on fixed-resolution occupancy grid generated from the predicted and GT meshes.
For this experiment we use NeRS that generates a mesh as part of its optimization and SAMURAI that provides a mesh extraction pipeline. Generally, it is possible to generate point clouds from a NeRF representation which could also be compared to the GT mesh.
The reconstruction output of these methods is arbitrarily transformed by a rigid transformation due to the optimization setup. To align the predicted mesh with the GT mesh we first adjust its scale. We then perform fast global registration [75] on downsampled point clouds followed by refinement via point-plane ICP [50]. Figure 16 shows examples of the mesh alignment process.
For evaluation we use the mean Chamfer distance of the two set of vertices, and the 3D intersection over union (IoU) of the voxelized meshes. Results are presented in Table 8.
(b) NeROIC
(d) SAMURAI
Figure 15: Novel view synthesis with in-the-wild 3D reconstruction. Sample novel view synthesis results of different techniques for 3D reconstruction from in-the-wild image collections.
We observe that the relative reconstruction quality in terms of the shape metrics corresponds to the novel view synthesis results. Looking at the aligned meshes we can derive further insights. NeRS, which is initialized with a cuboid, fails to reconstruct objects with shapes that are very different from its initialization (see dinosaur in Figure 16a for NeRS vs. Figure 16c for SAMURAI). On the other hand, for objects with shapes closer to a cuboid NeRS tends to predict the bulk of the object more accurately (see less pink regions in Figure 16b), while SAMURAI is able to add finer details that are missing from the NeRS reconstruction (see Figure 16d).
D Correspondence estimation
D.1 Details of experimented techniques
In the following, we briefly summarize the different correspondence techniques we benchmarked with NAVI. See the respective papers for more details.
- SIFT + MNN/NN-Ratio [37]. We use SIFT local features with heuristics-based matchers to represent the traditional baseline used in many Structure-from-Motion pipelines. Specifically, we use two popular variants of nearest neighbor search: mutual nearest neighbor and Lowe's ratio test.
- SuperPoint + MNN/NN-Ratio [17]. We replace traditional SIFT feature extraction with a learned detect/describe method, SuperPoint. We use traditional matchers to predict correspondences and rely on the improved descriptiveness of SuperPoint's features.
- SuperPoint + SuperGlue [51]. We replace traditional matchers with the popular SuperGlue sparse learnable feature matcher, which relies on a graph neural network and attention modules to predict correspondences from the input keypoint set. SuperGlue is trained on multiview image pairs from outdoor scenes, and we perform no additional object-centric finetuning.
- LoFTR [57]. Dense learnable matchers are often used to overcome repeatability issues in sparse local feature detection and matching. LoFTR relies on a coarse-to-fine transformer to propose a wider set of correspondences across entire images. We use the Kornia [49] implementation of LoFTR, which is also pretrained on outdoor scene pairs.
(a) NeRS dinosaur mesh
(b) NeRS truck mesh
(c) SAMURAI dinosaur mesh
(d) SAMURAI truck mesh
Figure 16: Comparison of predicted and GT mesh after alignment. The reconstructed mesh (blue) is aligned to the GT mesh (pink) and overlaid with a  $50\%$  blend. We show two methods: NeRS (top) and SAMURAI (bottom).
Dense recall at varying thresholds. Fig. 17 presents Dense-Recall@N results for the same set of correspondence estimation techniques, but with varying pixel radius'. We sweep radius values between 5 and 50 pixels and show that methods vary in dense recall performance at different threshold values. LoFTR and SuperPoint+SuperGlue outperform most methods in low-radius scenarios, but traditional matchers (SuperPoint+MNN and SuperPoint+NN-Ratio) see stronger performance for higher pixel radius'.
Additional visual results. Fig. 18 presents additional qualitative results for three correspondence estimation techniques. We believe these visualizations show that popular correspondence estimation techniques still have significant headroom for the task of fine-grained, object-centric correspondence estimation.
E 3D from a Single Image
Problem setting. Given a single RGB image of an object, the aim is to reconstruct the 3D shape and optionally the 3D pose of the object depicted in it. Shapes are commonly represented as occupancy grids [47, 5, 12, 23, 63, 66, 67], point clouds [21, 38], or implicitly [46, 42, 11]. Poses are predicted relative to the camera, commonly as an explicit transformation [22, 32], or as part of the scene volume [47]. Single image 3D reconstruction is a highly ambiguous and under-constrained vision problem as the techniques have to reason about the complete 3D shape of the object from a single 2D projection of that object in a single environment.
Figure 17: Dense-Recall@N, where the pixel radius  $N$  ranges from 5-50 pixels. Plots are provided for both the multiview and in-the-wild sets.
Figure 18: Sample correspondence results of different techniques where the correct (within 3 pixels) and incorrect matches are shown in green and red respectively.
The distinctiveness of NAVI. Commonly used real-world datasets for single image 3D reconstruction such as Pix3D [58] and Pascal3D+ [65] are category-specific with objects of some common categories such as chairs, cars etc. As a result, simple recognition based 3D model retrieval techniques can already perform well on such class-specific datasets [59]. In contrast, NAVI objects are category-agnostic and provide a unique opportunity to evaluate the 3D geometric understanding capabilities of the techniques. Another key issue with the most existing real-world datasets is that the 3D shapes are only approximate (either nearest CAD models or reconstructed using SfM), whereas NAVI provides near-perfect 3D shape GT and alignments allowing for more accurate evaluations of 3D reconstructions.
NAVI dataset and metrics. In the experiments, we use all NAVI images, from both multiview and in-the-wild collection types. We split the images into train and test using three strategies: randomly  $(S_{i})$ , randomly along the object they depict  $(S_{o})$ , and along objects and environments  $(S_{b})$ . There are no images in common between the test and train splits in all three strategies. In addition, the set of object instances depicted on the images in  $S_{o}$  and  $S_{b}$  is disjoint between the train and test splits. Finally, the backgrounds against which objects are photographed are dissimilar between the train and test splits of  $S_{b}$ . We use the official NAVI splits for  $S_{i}$  and  $S_{o}$  and we rely on capture location to split the dataset for  $S_{b}$ .  $S_{i}$  and  $S_{o}$  follow the practice of [47, 22] for measuring performance in datasets with real images. Splitting along images in  $S_{i}$  allows the model to learn about the geometry of all objects in the dataset and to apply this knowledge to images it hasn't seen in the test split.  $S_{o}$  presents
Figure 19: Sample single image 3D reconstructions using CoreNet [47]. In all cases, the reconstructed geometry aligns well with the input image. When splitting along objects  $(S_o)$ , reconstructions contain errors in unobserved parts. In addition, CoReNet cannot resolve the depth/scale ambiguity for  $S_o + B_f$  and it reconstructs objects at a wrong depth. Both are evident when viewing reconstruction from a different view point.
images (Si)	objects (So)	backgrounds (Sb)
GT pose provided (Bg)	66.6%	46.3%	46.2%
Fixed grid (Bf)	49.7%	13.1%	-
Table 9: IoU performance of CoReNet on the NAVI dataset, under different settings. Columns indicate the way data is split into train and test, rows – whether the model has access to the ground-truth pose at test time.
a harder scenario, as the model has to reconstruct unseen geometry on unseen images.  $S_{b}$  poses an additional challenge, as the model can no longer rely on a familiar background on the test set.
Experiment setup. For our preliminary experiments, we use CoReNet [47] as a representative single-object reconstruction method. CoReNet predicts volumetric binary occupancy on a regular grid inside a given 3D box in front of the camera. The geometry and the pose of the object can be extracted from the grid using Marching Cubes [33]. We establish baselines by training and evaluating CoReNet on the NAVI dataset. We evaluate CoReNet in two scenarios: 1. We provide CoReNet with access to the object's GT pose at test time  $(B_{g})$ , using the mechanism described in the paper [47] for the Pix3D dataset and; 2. We ask CoReNet to reconstruct occupancy inside a fixed  $3m\times 3m\times 3m$  bounding box, placed  $1m$  in front of the camera  $(B_{f})$ . Scenario  $B_{g}$  is essentially equivalent to shape prediction. Scenario  $B_{f}$  combines shape and pose prediction. It is much harder than  $B_{g}$ , since the model has to resolve the depth/scale ambiguity from a single image only. It is more feasible in NAVI, as NAVI has a single prominent object in each image and the pose is given in metric space.
E.1 Analysis
We train CoReNet models on 5 combinations of split strategy and pose prediction. In all cases, we start from a model pre-trained on synthetic data  $(h_7$  from the CoReNet paper), and we train for 15 epochs. To evaluate, we measure intersection-over-union (IoU) between the predicted and ground truth occupancy grids. Table 9 summarizes the results, Figure 19 shows sample reconstructions.
Analysis across different splits. Models trained on  $S_{i}$  outperform those trained on  $S_{o}$  by a large margin (+20.3% for  $B_{g}$ , +36.6% for  $B_{f}$ ). As expected, splitting randomly along images allows the model to learn about object geometry and to apply this to the similar objects in the test set. This is also confirmed visually (Figure 19). Reconstructed objects re-project correctly over the input images in all scenarios, but they contain large errors in unobserved regions for  $S_{o}$ . The difference between models trained on  $S_{o}$  and  $S_{b}$  is negligible, indicating that learning about geometry is more important than background.
Analysis with and without pose prediction. Comparing models with access to the ground-truth pose  $(B_{g})$  to those without  $(B_{f})$ , shows that performance falls modestly for models trained on  $S_{i}$  (from  $66.6\%$  to  $49.7\%$ ) and significantly for models trained on  $S_{o}$  (from  $46.3\%$  to  $13.1\%$ ). Learning about model geometry and most importantly about object size becomes essential for  $B_{f}$ , as the model has no other means to resolve the depth/scale ambiguity. Visually, reconstructed objects re-project correctly over the input images, but looking from below reveals that they are smaller/larger than the ground truth and they are reconstructed at the wrong depth (Figure 19).