SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Varun Jampani*	Kevis-Kokitsi Maninis*	Andreas Engelhardt	Arjun Karpur
Karen Truong	Kyle Sargent	Stefan Popov	Andre Araujo
Ricardo Martin-Brualla	Kaushal Patel	Daniel Vlasic	Vittorio Ferrari
Ameesh Makadia	Ce Liu†	Yuanzhen Li	Howard Zhou
Recent advances in neural reconstruction enable high-quality 3D object reconstruction from casually captured image collections. Current techniques mostly analyze their progress on relatively simple image collections where Structure-from-Motion (SfM) techniques can provide ground-truth (GT) camera poses. We note that SfM techniques tend to fail on in-the-wild image collections such as image search results with varying backgrounds and illuminations. To enable systematic research progress on 3D reconstruction from casual image captures, we propose 'NAVI': a new dataset of category-agnostic image collections of objects with high-quality 3D scans along with per-image 2D-3D alignments providing near-perfect GT camera parameters. These 2D-3D alignments allow us to extract accurate derivative annotations such as dense pixel correspondences, depth and segmentation maps. We demonstrate the use of NAVI image collections on different problem settings and show that NAVI enables more thorough evaluations that were not possible with existing datasets. We believe NAVI is beneficial for systematic research progress on 3D reconstruction and correspondence estimation. Project page: https://navidataset.github.io
The field of 3D object reconstruction from images or videos has been dramatically transformed in the recent years with the advent of techniques such as Neural Radiance Fields (NeRF) [43]. With recent techniques, we can reconstruct highly detailed and realistic 3D object models from multiview image captures, which can be used in several downstream applications such as gaming, AR/VR, movies, etc.
Despite such tremendous progress, current object reconstruction techniques make several inherent assumptions to obtain high-quality 3D models. A key assumption is that the near-perfect camera poses and intrinsics are given or readily available via traditional Structure-from-Motion (SfM) pipelines such as COLMAP [52]. This assumption imposes several restrictions on the input image collections. The input images have to be of sufficiently high-quality (e.g. non-blurry) and the number of input images should also be high (typically  $>30-50$ ) for SfM to estimate sufficient correspondences across images. In addition, SfM techniques typically fail on internet image sets that are captured with varying backgrounds, illuminations, and cameras. Such internet image collections do not require active capturing and are widely and readily available, such as product review photos or image search results (e.g., internet images of Statue-of-Liberty, Tesla Model-3 car, etc.). It is highly beneficial to develop 3D object reconstruction techniques that can automatically produce high-quality 3D models from such image collections in the wild.
$^{\dagger}$ C. Liu's current affiliation is Microsoft
In-the-wild Images
Multi-view Images
Sample Image
Figure 1: NAVI dataset overview. NAVI dataset consists of both multiview and in-the-wild image collections, where each image is aligned with the corresponding 3D scanned model resulting in high-quality 3D shape and pose annotations.
3D Scan
In this work, we propose a new dataset of image collections which we refer to as 'NAVI' (Not Average Image dataset). Specifically, our dataset contains two types of image collections with near-perfect camera poses and 3D shapes: 1. Standard multiview object captures and 2. In-the-wild object captures with varying backgrounds, illuminations and cameras. Fig. 1 shows examples of the in-the-wild and multiview images in NAVI along with the 2D aligned 3D scans. Next, we describe the key distinguishing properties of the NAVI dataset in relation to existing datasets.
Casual captures. Several existing multiview datasets are either synthetic or captured in lab settings [43]. We capture NAVI images in casual real settings using hand-held cameras.
In-the-wild image collections. In addition to typical multiview images, NAVI also provides in-the-wild image collections where objects are captured under varying backgrounds, illuminations, and cameras. SfM techniques usually fail on such image sets and NAVI provides a unique opportunity to systematically research joint shape and camera estimation from in-the-wild image collections.
Near-perfect 3D geometry and camera poses. We use high-quality 3D scanners to get 3D shape ground-truth and also obtain high-quality 3D camera pose annotations with manual 2D-3D alignment along with rigorous verification. This is in contrast to several recent datasets such as [64] that rely on SfM to provide GT, thereby limiting the image capture setups.
Accurate dense correspondences. We provide accurate per-pixel correspondences using the 3D shape alignments. While most real-world datasets for correspondence evaluation rely on known homographies [3] or sparse keypoint annotations recovered from estimated geometry [14, 27], NAVI's precise 2D-3D alignments lead to accurate and dense object correspondences.
Derivative annotations such as pixel-accurate object segmentation and monocular depth can be easily derived from high-quality 2D-3D alignments in NAVI.
Category-agnostic. Objects in the NAVI dataset are category-agnostic with image collections that do not have any category-specific shapes, which is in contrast to widely-used 2D-3D datasets [58, 65].
To demonstrate the utility of NAVI, we benchmark and analyze some representative techniques on three different problem settings: multiview object reconstruction, 3D shape and pose estimation from in-the-wild image collections, and dense pixel correspondence estimation from image pairs. In addition to these problem settings, one could also use NAVI images for other single-image vision problems such as single image 3D reconstruction, depth estimation, object segmentation, etc.
2 NAVI dataset
2.1 Dataset construction
Challenges. It is worth emphasizing the challenges in our data construction by taking a look at some existing 2D-3D aligned datasets. Several works [10, 13, 29, 15, 18] propose synthetic 3D assets which are used to render 3D-aligned images. Real-world datasets such as Scan2CAD [2] and Pascal3D+ [65] use nearest intra-category CAD models for alignment w.r.t 2D images, resulting in only coarse annotations. Similarly, IKEA Objects [35] and Pix3D [58] annotate retrieved images by aligning one 3D CAD model to images using point correspondences. Even for datasets with mostly
exactly-matching products [58], slight deformations and moving parts that appear different on images with respect to their 3D scan can lead to inaccurate alignments. Different instances of the same object can also have different shapes due to other factors (e.g. shoes of different sizes are not uniformly scaled versions etc.) Fig. 2 shows the sample alignments from the existing datasets showcasing the challenges in obtaining near-perfect 3D shapes and in the 2D-3D alignment.
Rationale. To avoid such issues in NAVI, we selected rigid objects without moving parts, manually scanned the object shape and took image captures of the same objects in diverse real-world settings. We then use our interactive alignment tool to obtain near-perfect 2D-3D alignments with precise pose control during the annotation. Most datasets, including our earlier attempts, use a multi-stage alignment process that involves annotating point correspondences and then optimizing the object pose. Even though this is a more scalable approach for dataset creation, the alignments are not as accurate as we want. The NAVI dataset construction consists of 4 steps: (1) Scanning the 3D objects, (2) Capturing image collections, (3) 2D-3D alignment, and (4) Alignment verification.
1. Scanning the 3D objects. We collect 36 rigid objects and use two professional 3D scanners, EinScan-SP [20] and EinScan Pro HD [19], to obtain high-quality 3D object scans. We center the scans at origin, but do not normalize the shapes to preserve their metric dimensions (in mm). Fig. 3 displays some NAVI images and their aligned 3D scans. Notice the diverse and category-agnostic nature of the objects.
2. Capturing image collections. For each object, we captured two types of image collections:
Pascal3D+
Figure 2: 2D-3D alignments from existing datasets have issues as 3D models do not exactly match the corresponding 2D image due to model discrepancies.
Scan2CAD
Pix3D
in-the-wild, and multiview. In-the-wild captures contain images with different backgrounds, illumination, and cameras. Multiview captures offer the standard multiview setup: same camera, object pose, and environment, but with different camera poses. For practical utility, we captured the images in casual settings with hand-held cameras ranging from mobile phones to DSLRs and mirrorless cameras. In total, we use 12 different cameras to capture around  $10.5\mathrm{K}$  images with  $2.3\mathrm{K}$  in-the-wild images and  $8.2\mathrm{K}$  multiview images. More dataset details are present in the Appendix (Section A).
3. 2D-3D alignment. The goal is to obtain near-perfect 2D-3D alignments; i.e., accurate 6DoF rigid object transformations along with accurate camera intrinsics. We developed an interactive tool on which the user can progressively align the 3D object by rotating and translating it in 3D, using the mouse. Since we know the cameras used to capture the images, we initialize the camera focal length, which can be further refined during the alignment process. Our interactive tool gives the user full control over the alignments, and we observe that this leads to higher-quality poses than alternative implicit alignment tools that optimize the pose from 3D  $\leftrightarrow$  2D point correspondences [58]. We trained 10 dedicated annotators for our alignment task allowing us to obtain higher quality annotations than several existing datasets that rely on generic non-expert annotators. Refer to the Appendix for more details on the alignment tool and the process (Section A.1).
4. Alignment verification. To ensure high-quality annotations, we further manually verify each 2D-3D alignment with 2 expert annotators. Specifically, we overlay the 3D shape onto the 2D image and ask trained annotators to label them as 'incorrect' if the alignments look even slightly wrong. For images labeled 'incorrect', we repeat the 2D-3D alignment and verification steps. After two stages of alignment and verification, we discard around  $7\%$  of the original captured images. We further annotate images with a binary occlusion label to indicate if the object is occluded by other objects. We exclude occluded object images from our validation sets for different tasks to avoid introducing artifacts in the metrics.
Derivative annotations. In addition to the full 3D alignments of scans to images, there are several derivative annotations that result from the accurate 2D-3D alignments: Relative camera poses, dense correspondences, metric depth maps, and binary masks. Relative camera poses are an implicit output of alignment, as all objects were posed with respect to their canonical pose. Since we have annotated multiple images of the same object, we obtain dense correspondences on the images by sampling the pixels in mutually visible parts of the 3D shape in image pairs. This enables dense correspondence evaluation both for the standard multiview setup, and for in-the-wild images captured in different environments. Fig. 4 visualizes sample GT pixel correspondences on NAVI image pairs. Furthermore,
Figure 3: NAVI samples. Sample object images and the corresponding 2D-3D alignments. NAVI consists of casually-captured and category-agnostic image collections with precise 2D-3D alignments.
Figure 4: Pixel correspondences. Sample image pairs and their corresponding GT pixel correspondences. For visualization purposes, we show sparsely sampled points and color-code the correspondences based on their 2D location from top to bottom.
metric depth maps are obtained by computing the depth of the 3D alignments from the camera viewpoint. The binary object masks are trivially obtained by binarizing the depth maps. Fig. 5 shows sample object depth and mask annotations in NAVI. For simplicity, we refer to our annotations as GT.
2.2 Dataset analysis
General statistics. Table 1 presents the general statistics of the NAVI dataset. It contains 10515 alignments in total, on 36 complicated object shapes, divided into in-the-wild images (2298) and multiview images (8217). Each object is aligned on 65 in-the-wild images on average. There are 267 unique multiview scenes, some of which were also captured by different cameras (324 multiview captures in total).
Annotation quality analysis. To analyze the quality of our 2D-3D alignments, we annotated 30 randomly selected images with two different annotators and measured the inter-annotator agreement [30] using two metrics: 3D translation distance (in milimeters), and 3D rotation distance (in degrees) between the two alignments. The average 3D rotation distance between two verified alignments is 1.7 degrees, and the 3D translation distance is 0.97
# Alignments (total)	10515
# Alignments (wild)	2298
# Alignments (multiview)	8217
# Objects	36
# Multiview Scenes	324
# Multiview Scenes (unique)	267
Table 1: General statistics of NAVI.
militimeters. The very small differences in the obtained alignments from two independent raters highlight the high quality of the alignments in the NAVI dataset. Similarly, we measured the quality of the alignments that we reject as "wrong", by comparing them to alignments of the same image that were verified as "correct". In this case, average annotator disagreement is 2.3 degrees of 3D rotation, and 2.01 millimeters of 3D translation. Practically, this means that even slight mis-alignments did not pass our strict verification process. To put these numbers into perspective, the rotation error of fully automatic methods are at least one order of magnitude larger (Table 4).
Manual vs. semi-automatic alignment process. Other works explore the use of  $3D \leftrightarrow 2D$  point correspondences for semi-automatically aligning a shape to an image [58, 39]. In this workflow the annotator needs to define points on the 3D shape and their corresponding points on the image. Alignments result from a pose optimization process. We explored such approach in the earlier stages of this work, and we argue that this process does not yield alignments as accurate as ours, mainly because the annotator does not have full control over the final alignment. For a quantitative comparison, we annotated the same 30 randomly sampled images with point correspondences, produced alignments using the software from [39], and measured inter-annotator agreement between two rounds of annotation. The average 3D rotation distance was 6.5 degrees, and the average 3D translation distance was 7.02 millimeters, significantly higher than our alignments (1.7 degrees, and
Image
Figure 5: Sample depths and masks. 2D-3D alignments on NAVI images allows to readily obtain high quality object depths and mask annotations.
0.97mm, respectively). For a qualitative comparison of the outputs of the two annotation processes, please refer to Appendix (Section A.3, Figure 11).
On using self-captured vs. Internet images. One possible way of scaling up the annotation pipeline could be to automate the acquisition process. Instead of self-captured images that we use in this work, we experimented with Google Image Search to scrape images from the internet, and align them. We used realistic renderings of the objects in [18] as reference. We noticed that even though the poses are in general of good quality, small differences in the shape introduce significant noise to the quality of the alignments. In this work, we strive for near-perfect alignments that enable tasks very sensitive to their quality, and we thus refrain from using internet images. For an illustration of common problems faced when using internet images, please refer to Appendix (Section A.4, Figure 12).
3 3D from multiview image collections
Problem setting. Given a set of images taken from different viewpoints, the task is to reconstruct the 3D shape and appearance of an object. The 3D representation can then be used for downstream tasks like scene editing, relighting, and rendering of novel views. Traditional multiview reconstruction pipelines such as Structure-from-Motion (SfM) first reconstruct camera poses together with a sparse object representation followed by a dense reconstruction and potential mesh generation step. After adding materials and textures, the resulting 3D asset can then be used to render new views. More recent techniques such as NeRF [43] optimize neural representations of objects directly on the RGB images with the camera poses obtained from an SfM reconstruction as a pre-processing.
Related datasets. Synthetic multiview scenes [56, 43, 25] are widely adopted for evaluations. In contrast to synthetic scenes that come with precise 3D scene and camera poses but only translate to real-world photography to a limited degree, real scenes usually require off-the-shelf SfM methods [52] for pose estimation. BlendedMVS [70], one of the first multi purpose datasets for stereo reconstruction comes with re-rendered images based on geometry and poses reconstructed via a SfM pipeline. CO3D [48] and Objectron [1] are large-scale datasets with object-centric videos, and provide either a rough point cloud reconstruction of the object [48] or a 3D bounding box [1]. The dataset of [28] offers a handful of 3D laser scans along with the corresponding real-world image collections. Recently, works of [55] and OmniObject3D [64] provide 3D object scans along with multiview image captures in constrained lab settings. These works rely on SfM for semi-automatic 2D-3D alignment. In summary, existing multiview datasets are synthetic [56, 43, 25] or based on reconstructed 3D models [70], with rough 3D shapes [48], provide only a limited number of scenes [28] or they consist of image captures in constrained settings [55, 64].
The distinctiveness of NAVI. In contrast, NAVI satisfies multiple requirements by offering highly-accurate 3D shapes and alignments for multiple objects from different categories in different real-world environments and illumination. This allows for more precise evaluation of 3D reconstruction techniques on real-world object image collections.
NAVI dataset and metrics. We split each of the multiview image sets into  $80\% / 20\%$  train/validation sets. The multiview sets are object-centric with an average of 25 images per set (minimum 3 to maximum 180). For even evaluation across the objects, we randomly sample 5 multiview scenes for each object from the subsets that include more than 6 images, resulting in 180 multiview sets for our experiments. We use the standard novel view synthesis metrics, PSNR, SSIM and, LPIPS [74], on validation images and report average metrics across all sets.
Experiment. A key assumption in most existing works is that SfM provided camera poses are good enough for 3D reconstruction. We want to test this hypothesis by evaluating how our annotated camera poses compare against COLMAP [52] poses for off-the-shelf 3D reconstruction techniques. For this, we use the widely-used InstantNGP [45] to reconstruct Radiance Fields from multiview images. For optimization we use the GT masks to limit the reconstruction to the object area.
Dataset	# Objects	# Scenes	# Images	Camera Positions	3D Annotations	3D←2D Alignment
LASSIE [68]	6	6	180	-	Keypoints	X
E-LASSIE [69]	6	6	270	-	Keypoints	X
NeRD [6]	8	8	396	Synthetic-GT	-	X
NeRF-W [40]	-	6	7658	-	-	X
SAMURAI [8]	8	8	560	-	-	X
NeROIC [31]	3	3	132	COLMAP	-	X
NAVI (Ours)	36	36	2298	Near-GT	Scanned mesh	✓
Results: COLMAP vs. GT poses. Table 2 shows the novel view synthesis metrics on validation images. Results on all the metrics demonstrate considerably better reconstruction with our GT poses compared to using COLMAP poses. COLMAP only registers partial set of views for several cases. This shows that the our GT poses are accurate and are still valuable in the multiview reconstruction setting to analyze reconstruction techniques independent of inaccuracies from the camera registration. While COLMAP poses are arbitrarily rotated and scaled, all NAVI scenes are centered at the origin and in a common coordinate frame. This facilitates evaluation across different objects, especially in the context of grid-based methods like InstantNGP where the scene bounds have some impact on performance.
4 3D from in-the-wild image collections
Problem setting. The aim is to estimate 3D shape and appearance of an object given an unconstrained image collection; where the object is captured with different backgrounds, cameras and illuminations. Such image collections are readily available on the internet; e.g., image search results, product review photos, etc. The high variability in the appearance across images makes pose estimation and reconstruction highly challenging compared to the more controlled multiview captures. Techniques
Table 3: Comparing NAVI with existing in-the-wild image collections, where the task is to 3D reconstruct an object given its images captured in different environments and lighting settings.
Camera Poses	PSNR↑	SSIM↑	LPIPS↓
COLMAP	24.04	0.93	0.079
NAVI Poses (GT)	27.54	0.94	0.045
Table 2: View synthesis metrics using COLMAP and GT poses. Our GT poses lead to significantly better performance, because COLMAP [52] does not always accurately pose our multiview scenes.
need to jointly reason camera poses and illuminations in addition to 3D geometry and appearance. Standard SfM techniques [53, 52] fail to recover camera poses on such in-the-wild image sets.
Existing datasets. Curated object centric image collections from in-the-wild data are scarce. While one could search online image databases for multiple occurrences of the same object or class [68], additional data like camera parameters or object shape as well as the certainty that all images actually depict the same object instance is critical for faithful evaluation. Table 3 compares published image collections used for 3D reconstruction from in-the-wild data to NAVI. Most existing collections include only a few number of scenes and objects and, if at all, sparse 3D annotations. NAVI is the only dataset of that kind with real-world image collections of objects in the wild (with varying environments and cameras), and with near-perfect 2D-3D alignments of 3D meshes. Other datasets such as Pix3D [58] contain many images (10k) with aligned 3D objects, which are however too inaccurate to be used for the task. DTU MVS dataset [26] is widely used as a proxy for in-the-wild data [60, 41, 71] as it comes with different lighting conditions for each of the 124 scenes. However, the controlled acquisition environment does not fully reflect in-the-wild conditions. Additionally, 3D scans and depths are of limited quality and coverage since the structured-light scan is only acquired at the given view positions. NeROIC [31] and NeRD [6] provide small collections of scenes for  $360^{\circ}$  object reconstruction featuring lighting changes and poses reconstructed via SfM. However, no GT object shapes are included. SAMURAI [8] adds eight image sets to the NeRD dataset with different cameras, backgrounds and illuminations; but it only provides RGB images without any GT camera poses or shapes. NAVI dataset subsumes these 8 SAMURAI in-the-wild image collections where we provide near-GT poses and 3D shapes.
The distinctiveness of NAVI. NAVI provides the first real-world in-the-wild image collections with GT 3D shapes and camera poses. For evaluation, existing techniques such as [8] rely on novel view synthesis metrics on held-out images which entangle the role of estimated camera poses and shapes. It is not possible to assess whether the view synthesis is poor due to a wrongly estimated camera pose or a wrongly estimated 3D object. GT poses and shapes in NAVI wild-sets provide a unique
(a) GT Novel View Image