SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhengfei Kuang* Yunzhi Zhang* Hong-Xing Yu
Samir Agarwala Shangzhe Wu Jiajun Wu
We introduce Stanford-ORB, a new real-world 3D Object inverse Rendering Benchmark. Recent advances in inverse rendering have enabled a wide range of real-world applications in 3D content generation, moving rapidly from research and commercial use cases to consumer devices. While the results continue to improve, there is no real-world benchmark that can quantitatively assess and compare the performance of various inverse rendering methods. Existing real-world datasets typically consist only of the shape and multi-view images of objects, which are not sufficient for evaluating the quality of material recovery and object relighting. Methods capable of recovering material and lighting often resort to synthetic data for quantitative evaluation, which on the other hand does not guarantee generalization to complex real-world environments. We introduce a new dataset of real-world objects captured under a variety of natural scenes with ground-truth 3D scans, multi-view images, and environment lighting. Using this dataset, we establish the first comprehensive real-world evaluation benchmark for object inverse rendering tasks from in-the-wild scenes and compare the performance of various existing methods. All data, code, and models can be accessed at https://stanfordorb.github.io/.
When we take a photograph of an object, light travels through space, bounces off surfaces, and eventually reaches the camera sensors to create an image. Inverting this image formation process is a fundamental problem in computer vision—from just the resulting single 2D image, we aim to reconstruct the physical scene in 3D, inferring 3D object shape, reasoning about its reflectance, and disentangling its intrinsic appearance from illumination effects. This process is often referred to as inverse rendering. Enabling machines to do so is not only crucial for general image understanding and object analysis, but also facilitates numerous downstream applications, such as controllable 3D content generation [65, 50, 12, 91, 52] and robotic manipulation [77, 51, 20, 14, 66, 61].
Recent years have witnessed significant progress in inverse rendering, with the emergence of various differentiable and neural rendering techniques. Neural Radiance Fields (NeRFs) [43], for instance, have emerged as a powerful volumetric representation that effectively encodes the geometry and view-dependent appearance of a scene into a neural network, allowing for high-fidelity novel view synthesis of complex real-world scenes given raw multi-view images as input. This framework has been successfully extended to further decompose the appearance into explicit material representations (e.g., BRDFs) and environment lighting [63, 88, 9, 7]. As the visual granularity of these inverse rendering results continues to improve, it becomes an increasingly pressing challenge to quantify and compare the performance of different methods.
Quantitative evaluation is particularly challenging for inverse rendering tasks because obtaining ground-truth data for material and lighting is prohibitively difficult in practice. Many existing works, therefore, resort to synthetic data for quantitative evaluation [30, 63, 7, 85, 88, 44, 74]. However, evaluation using synthetic data has several critical drawbacks. First, despite the constant evolution of 3D modeling and graphics rendering engines, it remains challenging to date to simulate intricate lighting effects that simply appear in the natural world, e.g., light bouncing inside a pitcher made of shiny metal. Second, synthetic rendering engines typically assume a perfect image formation model, omitting all the noise in a real-world imaging process, which abolishes the generalization guarantee to real-world scenarios. More fundamentally, such synthetic evaluation protocols inevitably give an advantage to methods that exploit similar assumptions that were used to generate the evaluation data.
Existing real-world evaluation benchmarks, on the other hand, are limited on two fronts. Many of them are designed primarily to evaluate geometry and novel view synthesis and only include multiview images and 3D captures of the objects, without access to the underlying material or lighting properties [26, 28, 7]. Datasets that provide ground-truth lighting and relit images for material and relighting evaluation [29, 67], however, are typically captured in constrained environments and do not represent the rich lighting variations in the natural world.
In this work, we aim to bridge this gap in real-world inverse rendering evaluation by introducing Stanford-ORB, a new Real-World 3D Object Inverse Rendering Benchmark. To do this, we collect a new real-world dataset of 14 objects captured under 7 different in-the-wild scenes, with ground-truth 3D scans, dense multi-view images, and environment lighting. This allows us to evaluate different aspects of inverse rendering methods through a number of metrics, including shape estimation, in-the-wild relighting with ground-truth lighting, and novel view synthesis. We compare a wide range of inverse rendering methods using this benchmark, including classical optimization-based methods and learning-based methods with neural representations. All data, trained models, and the evaluation protocol will be released to facilitate future work on this topic.
State of the Art on Inverse Rendering. Early works study the interaction of object shape, material, and lighting to estimate individual components, such as shape from shading [6, 87, 2], material acquisition [31, 32, 48, 47, 78], and lighting estimation [73, 80], or to recover reflectance and illumination assuming known shapes [36, 37]. Full-fledged inverse rendering methods aim at estimating all components simultaneously [40], which can be achieved with differentiable renderers [39, 13, 11]. Neural representations have recently emerged as a powerful approach in many state-of-the-art inverse rendering methods, due to their continuous nature and capacity to model complex geometry and appearance. Volumetric representations such as Neural Radiance Fields (NeRFs) [43] and the like [4, 7, 9, 8, 88, 54, 81] encode geometry and appearance as volumetric densities and radiance with a Multi-Layer Perceptron (MLP) network, and render images using the volume rendering equation [42]. While permitting flexible geometry, volumetric representations often suffer from over-parameterization, resulting in artifacts like floaters.
Other surface-based representations [69, 85, 84, 44, 89, 24, 75, 64] extract surfaces as the zero level set, for instance, of a signed distance function (SDF) or an occupancy field [46], allowing them to efficiently model the appearance on the surface with an explicit material model, such as using bidirectional reflectance distribution functions (BRDFs). This also enables modeling more complex global illumination effects, such as self-shadows. Most of these methods focus on per-scene optimization and require dense multiple views as input. Recently, researchers have incorporated learning-based models, distilling priors from large training datasets for fast inference on limited test views [59, 25, 32, 5, 10, 76, 74, 90].
Inverse rendering tasks have also been explored at a scene level beyond single objects. For indoor scenes, existing work has created large synthetic datasets, such as OpenRooms [33], and trained supervised models to predict geometry, material, and spatially-varying lighting with ground-truth data [30, 71, 92]. For outdoor scenes, others have used time-lapse images or Internet photos and trained models with weak supervision from optimization [83, 82, 34]. In this work, we focus on evaluating object-centric inverse rendering tasks.
Existing Inverse Rendering Benchmarks. Evaluating inverse rendering results is challenging because collecting ground-truth data is difficult. Table 1 summarizes existing datasets for object
Table 1: Comparison with Existing Object-centric Inverse Rendering Datasets. *Objaverse [19] consists of both synthetic objects and real scans.
Dataset	# Scenes	# Objects	Real	Scene Type	Multi-view	Shape	Relit Image	Lighting
ShapeNet-Intrinsics [59]	98	31K	X	synthetic	✓	✓	✓	✓
NeRD Synthetic [7]	30	3	X	synthetic	✓	✓	✓	✓
ABO [16]	40K	8K	X	synthetic	✓	✓	✓	✓
MIT Intrascripts [23]	1	20	✓	studio	✓	X	X	X
DTU-MVS [26]	1	80	✓	studio	✓	✓	X	X
Objaverse [19]	818K	818K	(✓)*	studio	✓	✓	X	X
DiLiGenT-MV [29]	1	5	✓	studio	✓	✓	✓	✓
ReNe [67]	1	20	✓	studio	✓	X	✓	✓
OpenIllumination [35]	155	64	✓	studio	✓	X	✓	✓
Lombardi et al. [36]	5	6	✓	in-the-wild	X	✓	✓	✓
NeRD Real [7]	4	4	✓	in-the-wild	✓	X	✓	X
NeROIC [28]	10	3	✓	in-the-wild	✓	X	✓	X
Oxholm et al. [47]	3	4	✓	in-the-wild	✓	✓	✓	✓
Stanford-ORB (ours)	7	14	✓	in-the-wild	✓	✓	✓	✓
inverse rendering evaluation. The early MIT Intrinsicss dataset [23] provides a small dataset of intrinsic images of real objects, including albedo and shading maps obtained through polarization techniques [45]. Bell et al. [3] instead proposes to leverage human judgments on relative reflectance for evaluation. However, these datasets do not provide shape or physically-based material ground truth. Synthetic datasets [59, 32, 7, 76, 74, 16] are widely used for evaluation as ground truth, as all components can be directly exported. However, these scenes are relatively simple and do not warrant generalization to complex real-world environments. Existing real object datasets [26, 19, 29, 67] are typically captured in studio setups with constrained lighting. In-the-wild object datasets, on the other hand, typically do not capture shape and ground-truth lighting, which are crucial for disentangled material evaluation through relighting. To the best of our knowledge, Lombard et al. [36, 47] is the only in-the-wild dataset providing ground-truth lighting, but is of a small scale compared to ours. In this work, we propose the first comprehensive real-world object inverse rendering benchmark with ground-truth scans and in-the-wild environment lighting.
3 Stanford-ORB: A Real-World Object Inverse Rendering Benchmark
Our goal is to create a quantitative evaluation benchmark for real-world 3D object inverse rendering tasks. The primary objective of object inverse rendering is to recover the 3D shape and surface material of the underlying object from images, where the input to the systems can be either a dense coverage of views of the object or as few as a single image, and optionally with additional assumptions, such as actively lighting. This task can be framed either as a per-scene optimization problem or as a learning-based one. In general, the fewer views available as input, the more challenging it is to recover accurate geometry and material, and hence the more the method relies on learned priors. In this work, we focus on evaluating the quality of the shape and material recovered by various inverse rendering methods with a comprehensive benchmark. To do this, we collected a new dataset of 14 common real-world objects with different materials, captured from multiple viewpoints in a diverse set of real-world environments, paired with their ground-truth 3D scans and environment lighting.
3.1 Evaluation Benchmarks
We design a number of metrics to evaluate the recovered shape and material from three perspectives. We give an outline of these benchmarks in this section and lay out the metric definitions in Section 4.
Geometry Estimation Benchmark. To assess the quality of the 3D shape reconstructions, we provide ground-truth 3D scans of each object. These 3D scans are obtained using a high-definition 3D scanner and carefully processed into watertight meshes. We measure the quality of the geometry estimated from different methods by comparing the predicted depth maps and normal maps to those generated from the ground-truth scans, as well as directly computing the bidirectional Chamfer Distance between the predicted meshes and the scans.
Novel Scene Relighting Benchmark. Evaluating surface materials is challenging as obtaining ground-truth data is generally difficult in practice, and different methods might adopt different
Figure 1: Data Capture Pipeline Overview. For each object, Left: we obtain its 3D shape using a 3D scanner and Physics-Based Rendering (PBR) materials using high-quality light box images. Middle: we also capture multi-view masked images in 3 different in-the-wild scenes, together with the ground-truth environment maps. Right: we carefully register the camera poses for all images using the scanned mesh and recovered materials, and prepare the data for the evaluation benchmarks. Credit to Maurice Svay [41] for the low-poly camera mesh model.
material representations. Here, we propose to evaluate the quality of material recovery through the task of relighting in a real-world environment unseen during training, given the ground-truth environment lighting, regardless of the underlying material representations. To do this, for each object, we capture multi-view image sets under multiple in-the-wild scenes, together with the ground-truth environment map without the object for each scene. To evaluate the performance of a model, we render the object using the predicted geometry and materials and the ground-truth lighting of a new scene, and compare the rendered image with the captured ground-truth image from a given viewpoint.
Novel View Synthesis Benchmark. In addition to the novel-scene relighting evaluation, we also perform an evaluation on the typical novel view synthesis task, where novel views of the object are rendered in the same scene as the input but from novel viewpoints, and compared to the ground-truth images. Note that this metric measures the accuracy of appearance modeling within one scene, but not necessarily the accuracy of material modeling that ensures accurate appearance in arbitrary scenes.
3.2 Data Capture
To support these evaluations, we capture a new real-world dataset comprised of ground-truth 3D scans, multi-view images taken in the wild, and the corresponding environment lighting. The dataset contains 14 common objects captured in 7 natural scenes. For each object, we take 60 training views and 10 testing views in high dynamic range (HDR) under 3 different scenes, resulting in a total of 42 scenes. For each testing view, we also capture an HDR panorama environment map. The ground-truth shape for each object is obtained using a professional 3D scanner. Fig. 1 gives an overview of our data capture pipeline.
Figure 2: Selection of Objects. From top to bottom: Block, Gnome, Ball, Car; Curry, Pepsi, Salt, Baking, Chips; Cactus, Pitcher, Grogu, Cup, Teapot.
3.2.1 Object Selection
Existing work often curates a small selection of objects for evaluation specific to the focus of their proposed methods. For instance, PhySG [85] evaluates on glossy objects, and NeRD [7] tests on objects with rougher surfaces. To establish a comprehensive benchmark, we carefully select 14 objects that cover a wide variety of both geometry and material properties. Fig. 2 gives a glimpse into the list of objects. The geometry ranges from simple cylindrical shapes to complex shapes,
(a) Geometry scanning
Figure 3: Studio Capture Setup. (a) 3D shape scanning. A: object, B: hand-held EinScan Pro HD 3D Scanner, C: printed patterns for camera registration, D: spray for high-quality scanning, E: desktop for processing. The scanned mesh is visualized in ExScan Pro [60] and MeshLab [15] on the right. (b) Light box capture setup viewed from outside and inside. F: DSLR camera, G: cloth cover to block light from outside, H: object, I: printed patterns for camera registration, J: (optional) dark background for object segmentation, K: remote-controlled turntable. The captured image and the chrome ball are visualized on the right.
(b) Light box scanning
and the materials cover diffuse, glossy as well as mirror-like metal surfaces. We focus primarily on non-translucent objects in this work.
3.2.2 3D Shape and Appearance Capture in Studio
For each object, we obtain its ground-truth 3D shape using an EinScan Pro HD 3D Scanner. Fig. 3 illustrates the capture setup. Despite the high scanning quality in general, the scanner can struggle to reconstruct shiny and dark surfaces for some objects, and we paint these areas with a special spray designed to improve the robustness. To help with camera registration, we print geometric patterns from the HPatches dataset [1] and place them underneath the objects. Each object is scanned 2-3 times in different poses to cover the entire shape. All scans are aligned and processed in the ExScan Pro software [60] to obtain a watertight textured mesh. For computation efficiency, we reduce the number of faces in each mesh to 200,000 using the method proposed by Garland et al. [22] and further smooth the surfaces using the HC Laplacian smoothing algorithm from Vollmer et al. [68].
In addition to the 3D shapes, we also obtain pseudo material decomposition for each object by capturing them in a light box. These pseudo materials are used to refine relative camera poses for in-the-wild scenes as detailed in Section 3.3, and to build a set of high-performing baseline results on the relighting benchmark as shown in Table 2. As illustrated in Fig. 3, we place each object on a remote-controlled turntable inside a simple light box. The turntable rotates  $6^{\circ}$  at a time; a bracket of three images are taken from each view at different exposure, which are fused into a high dynamic range (HDR) image for high-quality recovery using a simplified version of the algorithm in [18]. To solve for materials, we also capture the environment map inside the light box using a 3-inch steel chrome ball, following [80]. The chrome ball is placed in the same place as the object and reflects the environment light, which allows us to recover the environment map as detailed in Section 3.3.
3.2.3 Image and Lighting Capture in the Wild
To build the relighting and novel view synthesis benchmarks, we capture dense multi-view images of each object in different real-world in-the-wild environments, together with the corresponding ground-truth environment lighting. Each object is captured in 3 different scenes (7 unique scenes in total), including both indoor and outdoor environments, resulting in a total of 36 captures.
Fig. 4 illustrates the capture setup. In each scene, we fixate the object to a small platform and move the camera around the object in a circle from various heights. We take images from approximately 70 viewpoints roughly uniformly covering  $360^{\circ}$  views of the objects, including 10 test views and 60 training views, which are suitable for training most existing inverse rendering methods.
For each scene, we also capture the ground-truth environment lighting for relighting evaluation. Similar to Section 3.2.2, we reuse the chrome ball and solve for the environment maps following [80]. Unlike in the studio setup, where lighting is highly constrained and static, real-world (e.g., outdoor) environments are constantly changing while we capture the data. To minimize such errors, we capture one environment map per test view, forming an 'image-envmap' pair as illustrated in Fig. 4 (b). To
(a) Hardware
Figure 4: In-the-Wild Capture Setup. (a) Hardware for capturing. A: chrome ball, B: magenta platform for object segmentation, C: object, D: printed patterns for camera registration, E: DSLR camera, F: cloth for hiding photographer, G: mobile cart. (b) An example of the image-envmap pair. The environment map is solved from the reflection image on the chrome ball.
(b) Image-Envmap Pair
Figure 5: Data Processing Pipeline. Left: Overview of the data processing pipelines for both studio and in-the-wild captures. Three individual modules painted blue are expanded on the right. Middle-Top: The semi-automatically segmentation module produces object masks for all images. Middle-Bottom: Environment maps are solved from the chrome ball images. Right: Accurate camera poses are obtained from COLMAP and refined using NVDiffRec [44], given the scanned mesh and (for in-the-wild images) the pseudo materials optimized from light box captures.
avoid swapping the object for the chrome ball every time (which leads to inconsistent poses), we affix the object and only move the chrome ball in front of the object in each view. The photographer will hide themselves inside a white cloth underneath the camera to avoid being captured in the images.
For better inverse rendering quality, all images are taken in  $2048 \times 2048$  resolution and in HDR by fusing multiple exposure shots. We carefully tune the exposure values for each scene, taking into account the lighting condition and the reflections on the object surface. All cameras are calibrated with a chessboard and undistorted using standard OpenCV libraries.
3.3 Data Processing
Next, we describe the detailed processing steps for preparing the evaluation data. In essence, we need to obtain: (1) object masks for all images, (2) environment maps solved from the chrome ball images, (3) pseudo materials for each object using the studio captures, and (4) relative camera poses across multiple views and across multiple scenes. Fig. 5 shows the overview of the pipeline, and the
details of each step are laid out below. Overall, for each object, it takes roughly 3 hours to capture the raw data (2.5 hours for the in-the-wild capture and 0.5 hour for studio capture). The data processing pipeline takes roughly 10 hours on one machine for each object, which can be paralleled, and an additional 0.5 hour of human labor to manually adjust the annotations and alignments.
Object Masks. Getting high-quality inverse rendering results requires highly accurate object masks. We make use of the state-of-the-art object segmentation model, SAM [27], which is trained on over a billion curated masks. Given a sparse set of around 5 query points, the model produces reasonable object masks on our images. However, we observe that the resulting masks often appear smaller than the actual object. Hence, we further refine the masks using the classic GrabCut algorithm [53].
Environment Maps. Given the chrome ball images, we first apply the same object masking procedure described above to segment the chrome ball. We then fit a synthetic 3D sphere to the chrome ball by optimizing its 3D location w.r.t. the camera. With a differentiable Monte-Carlo-based renderer, we optimize the environment map to fit the reflection image on the chrome ball using the single-view light estimation method proposed in [80].
Pseudo Material Decomposition. For the purpose of refining the in-the-wild camera poses as detailed below, we optimize a set of pseudo surface materials for each object through an inverse rendering method NVDiffRec [44], making use of the 3D scan and the images captured in the light box. Specifically, we first register all light box images using COLMAP [56, 57] and SuperGlue [55] features to obtain a set of initial camera poses. In order to fit our 3D scanned mesh to the images, we optimize a coarse mesh with NVDiffRec, and roughly align the scanned mesh to the optimized mesh manually. The ground-truth environment map is also computed from the chrome ball images. Finally, we optimize the materials with a microfacet BRDF model [17] using NVDiffRec, jointly refining the COLMAP poses like [72], given the scanned mesh and ground-truth environment map.
Camera Pose Registration. For both training and evaluation, we need accurate relative camera poses both between different viewpoints of a scene and across different scenes for relighting. Similarly to the registration of light box images above, we make use of the 3D scans and the pseudo materials previously solved from the studio captures. For each scene, we solve for initial poses using COLMAP and SuperGlue, optimize a coarse mesh with NVDiffRec, and align the scanned mesh to it. The camera poses are further refined using NVDiffRec given the scanned mesh, pseudo materials, and ground-truth lighting. The relative pose between each source and target scenes for relighting is thus obtained through the same canonical scanned mesh.
4 Benchmarking State-of-the-Art Methods
This section provides an overview of baseline methods<sup>2</sup>, followed by evaluation details for the three benchmarks from Section 3.1 and discussion of the results.