SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) NeROIC
Figure 6: Novel view synthesis with in-the-wild 3D reconstruction. Sample novel view synthesis results of different techniques for 3D reconstruction from in-the-wild image collections.
(c) NeRS
(d) SAMURAI
opportunity to systematically analyze different techniques using pose metrics. In addition, NAVI also enables thorough analysis of techniques with controlled noise levels in the camera parameters.
NAVI dataset and metrics. We divide each of the in-the-wild image sets of NAVI into  $80\% / 20\%$  splits for training and validation respectively, where the techniques optimize a 3D asset using the train images and are evaluated on validation sets. On average there are 65 images in each in-the-wild set with minimum of 46 and maximum of 93 images, respectively. We use 2 different setups for evaluation. First is the standard novel view synthesis metrics that measure PSNR, SSIM and LPIPS [74] scores on held-out validation images. Second is camera pose evaluation where we use Procrustes analysis [24] to compute the mean absolute rotation, translation and scale difference in camera pose estimations for all the images. The camera metrics are a unique feature of NAVI enabled by our near-GT poses, compared to existing real-world datasets with in-the-wild image collections.
Techniques. We analyze four recent reconstruction techniques that can jointly optimize camera poses and can also deal with varying illuminations to some extent: NeRS [73], SAMURAI [8], NeROIC [31], and GNeRF [41]. Different works use different camera initialization and also model the object appearance differently. NeROIC assumes roughly correct COLMAP poses. NeRS and SAMURAI assume rough quadrant pose initialization and GNeRF takes randomly initialized poses. See Appendix (Section C.1) for a brief introduction of these techniques and refer to their respective papers for more details. While these techniques use either pre-computed or GT objects masks, we use GT object masks in our experiments to ensure fair comparison.
4.1 Analysis
COLMAP vs. GT poses. Table 4 shows the view synthesis performance and camera pose errors for different techniques and camera initializations. We observe that COLMAP reconstruction only works for a subset of scenes  $S_{C}$  (19 out of 36 scenes) for which the camera pose estimation using COLMAP yields more than 10 cameras. For comparisons with NeROIC that rely on COLMAP initialization, we separately report the metrics on scenes  $S_{C}$  where COLMAP works and those where COLMAP fails  $(\sim S_{C})$ . We omit one scene (vitamins bottle) that shows some inconsistencies between views because of a moving cap. Compared to the results from Section 3, the increased complexity of the task is reflected in lower performance. Comparing the performance of NeROIC with COLMAP to the initialization with NAVI GT poses on the  $S_{C}$  subset, it is clear that the NAVI GT poses are also superior in this setting. In addition to any COLMAP inaccuracies, the 3D reconstruction task becomes harder as the number of images shrinks due to incomplete COLMAP pose recovery that recovers only a subset of views.
Method	Pose Init	PSNR↑	SSIM↑	LPIPS↓	Translation↓	Rotation °↓
SC	~ SC	SC	~ SC	SC	~ SC	SC	~ SC	SC	~ SC
NeROIC [31]	COLMAP	19.77	-	0.88	-	0.1498	-	0.09 ± 0.12	-	42.11 ± 17.19	-
NeRS [73]	Directions	18.67	18.66	0.92	0.93	0.1078	0.1067	0.49 ± 0.21	0.52 ± 0.19	122.41 ± 10.61	123.63 ± 8.80
SAMURAI [8]	Directions	25.34	24.61	0.92	0.91	0.0958	0.1054	0.24 ± 0.17	0.35 ± 0.24	26.16 ± 22.72	36.59 ± 29.98
GNeRF [41]	Random	8.30	6.25	0.64	0.63	0.52	0.57	1.02 ± 0.16	1.04 ± 0.09	93.15 ± 26.54	80.22 ± 27.64
NeROIC [31]	GT	22.75	21.31	0.91	0.90	0.0984	0.0845	0.07 ± 0.24	0.01 ± 0.01	33.17 ± 19.63	31.90 ± 11.11
NeRS [73]	GT	17.92	18.02	0.92	0.93	0.114	0.1098	0.62 ± 0.19	0.65 ± 0.20	86.96 ± 27.63	89.43 ± 22.60
SAMURAI [8]	GT	25.65	25.59	0.92	0.92	0.0949	0.0881	0.16 ± 0.14	0.25 ± 0.26	21.55 ± 21.72	28.25 ± 26.71
Table 4: Metrics for 3D shape and pose from image collections in the wild. View synthesis and pose metrics over two subsets from all wild-sets depending on the success of COLMAP ( $S_{C} / \sim S_{C}$ ). Rendering quality is evaluated on a holdout set of test views that are aligned as part of the optimization without contributing to the shape recovery. We include GNeRF as a separate baseline although this method is not designed for multi-illumination data. We report metrics with the methods' default camera initialization as well as initializing with the GT poses that come with NAVI.
Figure 7: Analysis with varying camera noise. For different techniques, we initialize cameras with different levels of noise added to the GT poses for in-the-wild sets. To limit the computation, we report the mean over a subset of four objects of medium complexity.
Optimizing with GT poses can give insights into the additional challenges of the in-the-wild task independent of any dependency like COLMAP. This enables us to observe the other limitations that have an impact on in-the-wild reconstruction quality like the illumination model in SAMURAI or material model in NeRS.
Comparing different methods. Table 4 shows that SAMURAI performs best although the camera reconstruction quality varies drastically from scene to scene as can be seen in the large uncertainty. This is partly by design as views with large reconstruction errors are discarded over the course of optimization in this approach. It should be noted that data similar to NAVI guided SAMURAI's design. The results indicate that there are aspects covered by this data not available in other datasets (predominantly synthetic) used for evaluations so far. Fig. 6 shows sample novel view synthesis results of different techniques on an example from the "Keywest showpiece" validation set. This is a challenging object with high frequency details (e.g. text), some symmetry, and glossy surface areas. We can observe different artifacts characteristic for the evaluated methods like the rotated view and the high specularity in NeRS, texture smoothness in SAMURAI, and floater artifacts in NeROIC. NAVI includes several challenging objects that are well suited to evaluate the methods' limits. Section C.2 (Appendix) provides further results and analysis of the distribution of the reconstruction errors over different objects in the dataset.
Camera metrics. Thanks to the GT camera pose annotations, both the novel view synthesis and camera evaluations can be done on the same data where multiple datasets, often including synthetic data had to be used in the past. Together with the GT masks from NAVI all the confounding varying assumptions on the input data across different techniques can be made uniform here. For all the techniques, camera errors are relatively high overall, still there is a correlation between pose error and view synthesis quality. NeRS shows a surprisingly large camera pose error. It can be visually confirmed that test views are not that well aligned, still 3D mesh generation based on the training views works relatively well. Camera pose not being a focus in the original work, techniques like NeRS can benefit from explicit pose evaluations for technical improvements.
Analysis with varying camera noise. Annotated camera parameters in NAVI allow for a controlled study of how different techniques work with increasing amount of camera noise in their camera initialization. Specifically, we add normal distributed noise with zero mean and varying standard deviation to the annotated poses before feeding it as input to different techniques. The rotational change is limited to  $+/- 90^{\circ}$  and the translation noise scales with the mean distance of the cameras to the object. A noise level of 1.0 translates to a standard deviation of  $10\%$  of the mean distance for translation and  $18^{\circ}$  standard deviation for the rotation noise on a linear scale. Fig. 7 shows the plots with novel view synthesis and camera metrics for SAMURAI, NeRS and NeROIC. While the pose error generally increases as the noise level increases, the camera rotation error is not strictly monotonically increasing, for example. This points to the shape of the loss landscape with local minima. Both SAMURAI and NeRS seems relatively robust with varying camera noises, while NeROIC performance degrades with increasing camera noise. SAMURAI seems to be robust to large noise levels but, except for GT poses, yields a high translation error. This might stem from the camera multiplex initialization and view weighting scheme. Translation can also be approximated by a focal length change to some extent which could also happen in SAMURAI where the global scene bound is part of a regularization that prefers cameras around the mean radius. NeROIC performs very well under small noise levels but cameras rotate too far away from the object bounding box for higher camera noise levels. It seems like small rotation errors can be compensated by the neural network (if conditioned on direction) to some extent here. In summary, different methodologies seem to be needed for different strengths of camera noise. NAVI can help systematically investigate how the camera optimization performs in a technique thereby informing on several useful design choices for technical improvements (e.g. larger vs. smaller pose updates, regularization weights, initialization and fine-tuning). In addition, investigations around the breaking point of a method can lead to valuable insights into the task of joint shape and camera optimization.
Method	Correspondence Metrics	Relative Pose Metrics
Precision@0.2↑	Dense-Recall@15px↑	AUC@5°↑	AUC@10°↑	AUC@20°↑
SIFT + MNN	2.8 / 1.2	6.3 / 2.2	7.3 / 5.9	13.6 / 11.9	23.5 / 23.0
SIFT + NN-Ratio	10.2 / 4.8	6.4 / 2.2	6.2 / 4.2	11.9 / 8.1	22.7 / 23.1
SuperPoint + MNN	6.1 / 3.3	9.7 / 5.6	10.0 / 8.2	19.2 / 16.0	31.6 / 28.0
SuperPoint + NN-Ratio	24.7 / 18.7	11.4 / 6.8	9.0 / 7.6	17.5 / 15.0	29.0 / 26.5
SuperPoint + SuperGlue	26.8 / 23.8	12.6 / 9.1	12.1 / 10.8	22.2 / 20.1	34.6 / 32.3
LoFTR	19.2 / 13.4	16.3 / 8.5	12.2 / 9.8	22.5 / 18.4	34.2 / 30.0
Table 5: Correspondence and relative pose estimation. For each metric, we present multiview (left) / in-the-wild (right) results. To calculate precision, we filter the matching confidence of correspondences at 0.2 [57]
and use a reprojection error threshold of 3 pixels. For all other metrics, we consider the entire prediction set.
5 Correspondence estimation
Problem setting. Given a pair of images of the same object, the goal of correspondence estimation is to match a set of object pixels from one image to the corresponding pixels in the second image. By definition, an image point can have at most one correspondence in the other image as some points may be unmatched due to occlusion. Image pair correspondences are fundamental for the downstream tasks of 3D reconstruction and pose estimation, where a robust estimator is often used to recover the underlying relative camera rotation and translation.
Existing datasets. Finding a suitable dataset for training and evaluating correspondence estimation methods can be a challenge. SPair-71k [44] and CUB [61] provide in-the-wild semantic correspondences, but these correspondences associate parts of different objects and have limited use in instance-level tasks. Manually labeling fine-grained, instance-level correspondences is a time-consuming and error-prone task, so datasets must rely on either known real [3] or synthetic [16] homographies, or complete scene information [14, 27, 34, 54]. However, synthetic homography pairs suffer from unrealistic image distortion, and many of the latter datasets focus only on indoor/outdoor scenes and not object-centric imagery. Alternatively, high-quality 3D models [18] can be used to render object-focused image pairs with known correspondences, but methods may suffer from a wide domain gap when transferring knowledge from synthetic renderings to real world scenes.
The distinctiveness of NAVI. In contrast, the NAVI dataset annotations allow us to generate real-world image pairs with dense per-pixel correspondences, due to the precise 2D-3D alignments. This provides a unique opportunity to have novel dense evaluation metrics for correspondence estimation techniques. Additionally, the NAVI in-the-wild collections allow correspondences to be annotated across images with different backgrounds, lighting conditions, and camera models. For example, Fig. 4 shows sample pixel correspondences on NAVI in-the-wild image pairs.
NAVI dataset and metrics. We sample two types of correspondence datasets in NAVI. The first dataset contains randomly sampled image pairs within the same multiview set to represent the scenario of a fixed scene and camera model. The second dataset contains randomly sampled pairs from the in-the-wild set to emulate the variety of backgrounds, illuminations, and cameras. For each image pair, we can use the complete camera-object knowledge to label ground truth correspondences between the two images while respecting self-occlusions. We sample up to 707 multiview pairs and 1035 in-the-wild pairs per object resulting in the validation sets with 24745 and 35931 pairs, respectively. We limit GT correspondence labels to object pixels, since the data annotation process limits the available depth information to object points only. Additionally, we resize each image before evaluation such that their largest dimension is 1200 pixels.
For benchmarking, we evaluate both correspondence and pose estimation metrics. We use precision (reprojection error less than 3 pixels) and recall to directly evaluate correspondences, but we define a recall metric that leverages the dense ground truth correspondences made available by the NAVI 2D-3D alignment. For each object pixel visible in the first image, we find the corresponding location in the second image, after filtering out instances of self-occlusion. Given a correspondence prediction set, we calculate the percentage of ground truth matches which have a corresponding prediction whose keypoints are within N pixels of error. We denote this metric dense recall, and it provides an understanding of how well-distributed the predicted correspondences are across the co-visible regions. In addition, we estimate relative camera poses from the predicted correspondences. We compute the essential matrix that relates a pair of cameras using their intrinsic parameters, and the correspondences (standard 5-points algorithm). Finally, we calculate the rotation error between the
SIFT+NN-Ratio
SuperPoint+SuperGlue
Figure 8: Sample correspondence results of different techniques where the correct (within 3 pixels) and incorrect matches are shown in green and red respectively.
LoFTR
predicted and ground truth rotation matrices using Rodrigues' formula, and report accuracy within  $5^{\circ}$ ,  $10^{\circ}$ , and  $20^{\circ}$  of error following [51].
Techniques. We evaluate the following 4 types of correspondence estimation methods: SIFT + MNN/NN-Ratio [37] that use traditional keypoint detection with heuristic traditional matching; SuperPoint + MNN/NN-Ratio [17] that use learned keypoint detection with traditional matching; SuperPoint + SuperGlue [51] that use both learned keypoint detection and learned matching and; LoFTR [57] that proposes dense learnable matching. We directly evaluate these off-the-shelf models trained on their respective datasets. See Section D.1 (Appendix) for a brief summary of these techniques and refer to the original papers for more details.
5.1 Analysis
Multiview vs. In-the-wild pairs. Table 5 presents the evaluation metrics on the multiview/in-the-wild image pair datasets in NAVI. Across all metrics, we observe a significant decrease in performance from multiview to in-the-wild pairs. Traditional methods (i.e. SIFT+MNN/Ratio) are insufficient to handle major changes in lighting conditions, such as ambient lighting and shadows produced by the environment. Learned methods (SuperPoint and SuperGlue) are more robust to changes across in-the-wild images with different backgrounds, lighting and cameras. We note that SuperGlue experiences a  $3\%$  decrease from multiview to in-the-wild in Prec@0.2 and a  $3.5\%$  decrease in Dense-Recall@15px, compared to  $6\%$  and  $4.6\%$  for the traditional matcher (SuperPoint + NN-Ratio). We also note that LoFTR proves to be less robust to changes in lighting conditions than the sparse feature-based SuperPoint+SuperGlue method. These results emphasize the importance of exposing learnable features and matchers to sufficient in-the-wild image pairs during training.
Dense coverage. Table 5 also shows dense recall metric enabled by dense GT correspondences in NAVI. This measures the coverage of pixel correspondences given a wide error tolerance (15 pixels). Local feature techniques are highly dependent on texture-rich regions and suffer from low coverage over smooth/textureless overlapping regions. LoFTR, a dense learnable matcher, performs well on the multiview split but is outperformed by SuperPoint+SuperGlue on the in-the-wild split. This dense recall metric highlights that existing matching techniques recover correspondence sets with low coverage of overlapping object regions, and that the NAVI dataset may serve as a benchmark for this important evaluation metric. Finetuning these methods on object-centric data is likely to yield better performance. Figure 8 shows some sample visual results of correspondences with different techniques.
Use of NAVI in other tasks. In addition to 3D from image collections and correspondence tasks, NAVI can be useful for single-image tasks such as single image 3D reconstruction, monocular depth or normal estimation and object segmentation. There exist several large-scale datasets for these tasks and NAVI can be used as an additional fine-tuning or evaluation dataset. We present some single image 3D reconstruction experiments in Section E.
Limitations. Scale is the main limitation of the NAVI dataset which consists of only 36 objects and  $\approx 10\mathrm{K}$  images. We prioritize annotation quality over quantity; and our current rigorous data capture and annotation pipeline is not easily scalable to collect large datasets. Since the techniques for 3D from image collections usually optimize the 3D models within an image collection, we do not find the small scale of NAVI to be a limiting factor. In the future, we also plan to extend the dataset to videos.
Concluding remarks. In summary, we propose NAVI dataset with multiview and in-the-wild image collections annotated with near-perfect 3D shapes and camera poses. We demonstrated the use of NAVI for better analysis on 3D from multiview image collections, 3D from in-the-wild image collections and pixel correspondence estimation problems. We believe NAVI is beneficial for a multitude of 3D reconstruction and correspondence tasks.
Acknowledgements. We thank Prabhanshu Tiwari, Gourav Jha, Ratandeep Singh, and Mohd Adil for coordinating the annotation process, and all annotators who contributed to NAVI. We also thank Mohamed El Banani and Amit Raj for their valuable feedback on the manuscript.
Figure 9: 3D-2D alignment tool. The user is able to rotate and translate the 3D object directly on the screen. We provide ways to improve the annotation experience such as a restricted 'easy' mode when the object appears upright, saving a backup pose that can be recovered, enabling/disabling texture, and others.
A Additional dataset details
A.1 3D  $\leftrightarrow$  2D alignment tool
Our interactive alignment tool was developed using the three.js library [62] that allows the user to interact with a 3D object directly on the browser. The objective is to directly produce alignments on images by rotating and translating the object.
For rotating the object, we used the intuitive mouse movements of Orbit Controls [36] that allow the user to rotate an object in 3D using 2D drag-and-drop movements. Orbit Controls constrains the 'up'-axis of the 3D shape ('easy' constrained mode), which makes the task much easier when the objects are in 'upright' position on the images. For adjustments, and for objects that do not appear in 'upright' position, the user has the option to remove the 'up-axis' constraint, and allow for all possible 3D poses ('difficult' unconstrained mode). Our tool has the option to switch between the two modes, which enables the user to first bring the object to a close-enough pose using 'easy' mode, and then switch to 'difficult' mode for the final adjustments. For translating the object, we used the panning functionality of [36].
Taking into account feedback from the annotators, we developed the option to save a backup pose, and revert back to it in case they need to restart the process (eg. when the backup pose is better than the current pose). We further developed simple keyboard shortcuts that improve the annotation experience, such as disabling/enabling texture on the 3D shape, and changing its opacity. Figure 9 illustrates our annotation interface. The camera parameters (focal length) are initialized from the ExIF metadata of the images, and can also be adjusted from within the tool.
We observed that our interactive tool gives the user full control over the alignments, and lead to higher-quality poses than alternative implicit alignment tools that optimize the pose from 3D  $\leftrightarrow$  2D point correspondences [58]. This allowed us to obtain annotations of higher quality than existing datasets that use general crowd-sourcing.
A.2 Quality of 3D scans
The 3D scans were obtained by using [20] and [19]. We used their fixed scan mode (accuracy of  $0.05\mathrm{mm}$  and  $0.04\mathrm{mm}$ , respectively). Additionally, we manually curated the mesh, removed noisy vertices, and posed them in a consistent pose ( $\mathbf{Y}+$  axis is up,  $\mathbf{Z}-$  axis is front, like in [10]). Figure 10 illustrates the quality of our scans. We provide accurate shapes with very fine-grained details.
A.3 Visualization: Manual vs. semi-automatic alignment process
We provide qualitative evidence of the analysis in Section 2.2 of the main paper (paragraph 'Manual vs. semi-automatic alignment process'). Figure 11 compares our pipeline to the output of [39],
Figure 10: Quality of 3D scans. We provide detailed 3D meshes as a result of the scanning process.
3D $\leftrightarrow$ 2D correspondences
Alignment from 3D  $\leftrightarrow$  2D correspondences
Alignment from the NAVI interactive tool
Figure 11: Comparison to aligning from  $3\mathbf{D}\leftrightarrow 2\mathbf{D}$  correspondences. The semi-automatic alignment process of [39] does not lead to the accuracy that we achieve in NAVI. See the mis-aligned paw and head of the lion from point correspondences (left). In our tool the annotators have full control of the final alignment and produce more accurate results (right).
a common semi-automatic alignment pipeline from  $3\mathrm{D}\leftrightarrow 2\mathrm{D}$  point correspondences. In general, we observed that [39] can not produce the near-perfect alignments that our dataset provides (eg. mis-aligned paw and head of lion from point correspondences as a result of the optimization process).
A.4 Visualization: On using self-captured vs. Internet images
Figure 12 illustrates common issues when aligning on internet images instead of our self-captured images. While our tool can yield accurate poses on internet images, discrepancies in the shapes introduce significant noise in the quality of the resulting alignments. Discrepancies include additional/missing parts on the shapes (first, and last alignment), moving parts (second alignment), and differences in the manufacturing process (different colors in the first, and different thickness on the wheels of the third alignment). Internet images would make sense for creating a lower-quality, noisier dataset than NAVI. Nearly none of the produced alignments would pass NAVI's strict quality threshold.
Camera type	# Images
pixel_5	2040
pixel_6pro	1688
pixel_4a	1674
canon_t4i	1237
ipad_5	1140
pixel_4x1	1024
pixel_7	869
sony_a7iv	228
iphone_7plus	224
sony_a6000	216
ipad_4	175
Table 6: Camera types used for the images of NAVI.
A.5 Data license, access details, and intended usage