Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark
Authors: Zhengfei Kuang
Yunzhi Zhang
Hong-Xing Yu
Samir Agarwala
Shangzhe Wu
Jiajun Wu
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Inverse Rendering
3D Objects
BRDF
TL;DR: We introduce a new real-world 3D object inverse rendering benchmark, with ground-truth 3D scans, multi-view images, and environment lighting of in-the-wild scenes.

Abstract: We introduce Stanford-ORB, a new real-world 3D Object inverse Rendering Benchmark. Recent advances in inverse rendering have enabled a wide range of real-world applications in 3D content generation, moving rapidly from research and commercial use cases to consumer devices. While the results continue to improve, there is no real-world benchmark that can quantitatively assess and compare the performance of various inverse rendering methods. Existing real-world datasets typically only consist of the shape and multi-view images of objects, which are not sufficient for evaluating the quality of material recovery and object relighting. Methods capable of recovering material and lighting often resort to synthetic data for quantitative evaluation, which on the other hand does not guarantee generalization to complex real-world environments. We introduce a new dataset of real-world objects captured under a variety of natural scenes with ground-truth 3D scans, multi-view images, and environment lighting. Using this dataset, we establish the first comprehensive real-world evaluation benchmark for object inverse rendering tasks from in-the-wild scenes, and compare the performance of various existing methods. All data, code, and models can be accessed at https://stanfordorb.github.io/

Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark
Zhengfei Kuang"
Yunzhi Zhang"
Hong-Xing Yu
Samir Agarwala Shangzhe Wu
Jiajun Wu
Stanford University
Abstract
We introduce StanfordORRB, a new real-world 3D Object inverse Rendering Bench- mark. Recent advances in inverse rendering have enabled wide range of real-world applications 3D content generation, moving rapidly from research and commer- cial use cases consumer devices. While the results continue improve, there is no real-world benchmark that can quantitatively assess and compare the perfor- mance various inverse rendering methods. Existing real-world datasets typically consist only of the shape and multi-view images of objects, which are not sufficient for evaluating the quality of material recovery and object relighting. Methods capa- ble of recovering material and lighting often resort to synthetic data for quantitative evaluation. which on the other hand does not guarantee generalization to complex real-worid environments. We introduce new dataset of real-world objects captured under a variety of natural scenes with ground-trute 3D seans, multi-view images, and environment lighting. Using this dataset, we establish the first comprehensive real-world evaluation benchmark for object inverse rendering tasks from in-the- wild scenes and compare the performance of various existing methods. All data, code, and models can be accessed hhtps::/. http:::://saafffooooo wwwp.iiihhu.......... io/
Introduction
When we take a photograph an object, light travels through space, bounces off surfaces, and eventually reaches the camera sensors to create an image. Inverting this image formation process a fundamental problem in computer vision-froom just the resulting single 2D image, we aim to reconstruct the physical scene in 3D, inferring 3D object shape, reasoning about its reflectance, and disentangling its intrinsic appearance from illumination effects. This process is often referred to as inverse rendering. Enabling machines to do is not only crucial for general image understanding and object analysis, but also facilitates numerous downstream applications, such as controllable 31)
content generation 665 50
12,
91 52 and robotic manipulation 77 51 20
66 61].
Recent years have witnessed significant progress inverse rendering. with the emergence of various differentiable and neural rendering techniques. Neural Radiance Fields (NeRFs) [43, for instance, have emerged as a powerful volumetric representation that effectively encodes the geometry and view-dependent appearance of scene into neural network, allowing for high-fidelity novel view synthesis of complex real-world scenes given raw multi-view images as input. This framework has been successfully extended to further decompose the appearance into explicit material representations (e.g., BRDFs) and enviromment lighting 63 8g 7]. As the visual granularity of these inverse rendering results continues improve, becomes increasingly pressing challenge quantify and compare the performance different methods.
"Equal contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Quantitative evaluation is particularly challenging for inverse rendering tasks because obtaining ground -ruth data for material and lighting is prohibitively difficult in practice. Many existing works, therefore, resort to synthetic data for quantitative evaluation 30 63
74. However, evaluation using synthetic data has several critical drawbacks, First, despite the constant evolution of 3D modeling and graphics rendering engines, it remains challenging to date to simulate intricate lighting effects that simply appear in the natural world, e.g., light bouncing inside a pitcher made of shiny metal. Second, synthetic rendering engines typically assume a perfect image formation model, omitting all the noise real-world imaging process, which abolishes the generalization guarantee to real-world scenarios. More fundamentally, such synthetic evaluation protocols inevitably give an advantage to methods that exploit similar assumptions that were used to generate the evaluation data.
Existing real-world evaluation benchmarks, on the other hand, are limited on two fronts. Many of them are designed primarily to evaluate geometry and novel view synthesis and only include multi- view images and 3D captures of the objects, without access to the underlying material or lighting properties [26 28 7 Datasets that provide ground-truth lighting and relit images for material and
relighting evaluation 29 677. however, are typically captured in constrained environments and do not represent the rich lighting variations in the natural world.
In this work, we aim to bridge this gap in real-world inverse rendering evaluation by introducing Stanford-ORB, new Real-Worle 3D Object Inverse Rendering Benchmark. To do this, we collect a new real-world dataset of 14 objects captured under 7 different in- in-the-wild scenes, with ground-trum 3D scans, dense multi-view images, and environment lighting. This allows us evaluate different aspects of inverse rendering methods through number of metrics, including shape estimation, in-the-wild relighting with ground-truth lighting, and novel view synthesis. We compare a wide range of inverse rendering methods using this benchmark, including classical optimization-be based methods and learning-base -based methods with neural representations. All data, trained models, and the evaluation protocol will be released to facilitate future work on this topic.
2 Related Work
State of the Art on Inverse Rendering. Early works study the interaction of object shape, material, and lighting to estimate individual components, such shape from shading [6] 87 2]. material
acquisition [I 32 48
[4 (8). and lighting estimation 173 80], or to recover reflectance and illumi-
nation assuming known shapes 36 37]. Full-fledged inverse rendering methods aim at estimating all components simultaneously 40). which can be achieved with differentiable renderers [39 13
Neural representations have recently emerged as a powerful approach in many state-of-the-aae in- verse rendering methods, due to their continuous nature and capacity to model complex geometry and appearance. Volumetric representations such as Neural Radiance Fields (NeRFs) [K] and the like [4] [ 9 8 88 [4, 81 encode geometry and appearance as volumetric densities and radiance with a Multi-Layer Perceptron (MIP) network, and render images using the volume rendering equation 42). While permitting flexible geometry, volumetric representations often suffer from pver-parameterization, resulting artifacts like floaters.
Other surface-based representations 69 85
64 extract surfaces as the zero occupancy field (46), allowing them to efficiently model the appearance on the surface with an explicit material model, such as using bidirectional reflectance distribution functions (BRDFs). This also enables modeling more complex global illumination effects, such as self-shadows. Most of these methods focus on per-scene optimization and require dense multiple views input. Recently, researchers have incorporated barning-based models, distilling priors from large training datasets for fast inference on limited test
24
75
level set, for instance, of signed distance function (SDF)
views 59 25
32
10
76 74 90].
Inverse rendering tasks have also been explored at a scene level beyond single objects. For indoor scenes, existing work has created large synthetic datasets, such OpenRooms [3],, and trained supervised models to predict geometry, material, and spatially-varying lighting with ground-truth data [30 7 [2]. For outdoor scenes, others have used time-lapse images Internet photos and trained models with weak supervision from optimization 83 82 34]. In this work, we focus on evaluating object-centric inverse rendering tasks.
Existing Inverse Rendering Benchmarks. Evaluating inverse rendering results is challenging because collecting ground-truth data is difficult. Table summarizes existing datasets for object

---

Table I: Comparison with Existing Object-centric Inverse Rendering Datasets.
*Obaverse
19
consists of both synthetic objects and real scans.
Dataset
# Scenes
Objects
Real
Scone Type
Multi-view
Shape
Relit Image
Lighting
ShapeNet-
59
98
31K
X
synthetic
NeRD Synthetic
7
30 40K
X
synthetic
ABO [16
X
synthetic
MIT Intrinsics
23
1
20
studio studio
/
X
X
X
DTU-MVS 26
/
Objaverse 19
818K
818K
studio
/
DiL.GGenT-Mi
29
studio
5
V
ReNe 67
1
20
studio studio
X
OpenIllumination
35
155
X
Lombardi
36
5
in-the-wild
NeRD Real
7
4
in-the-wild
X
X
NeROIC 28
10
in-the-wild
Oxholm et ol.
+
3
in-the-wild
Stanford-ORB (ours)
7
in-the-wild
provides small dataset of intrinsic images of real objects, including albedo and shading maps obtained through polarization techniques [45]. Bell et al. [3] instead proposes to leverage human judgments on relative reflectance for evaluation. However, these datasets do not provide shape or physically-based material ground 74 [6] are widely used for evaluation as ground truth, as all components can be directly exported. However, these scenes are relatively simple and not warrant generalization complex real-world environments. Existing real object datasets [26 19 29 67
inverse rendering evaluation. The early MIT Intrinsics dataset 23
truth. Synthetic datasets (59 32 7
76
typically captured in studio setups with constrained lighting. In-the-will wild object datasets, on the other hand, typically do not capture shape and ground-truth lighting, which crucial for disentangled material evaluation through relighting. To the best our knowledge, Lombard et al. [36] 47 the only in-the-wild dataset providing ground-truth lighting, but is of small scale compared ours. In this work, we propose the first comprehensive real-world object inverse rendering benchmark with ground-truth scans and in-the-wild environment lighting.
3 Stanford-ORB. A World Object Inverse Rendering Benchmark
Our goal is to create quantitative evaluation benchmark for real-world 3D object inverse rendering tasks. The primary objective object inverse rendering is to recover the 3D shape and surface material of the underlying object from images, where the input to the systems can be either dense coverage of views of the object or as few as single image, and optionally with additional assumptions, such actively lighting. This task can be framed either per-scene optimization problem as a learning- based one. In general, the fewer views available as input, the more challenging it is to recover accurate geometry and material, and hence the more the method relies learned priors. In this work, we focus on evaluating the quality of the shape and material recovered by various inverse rendering methods with comprehensive benchmark. To do this, we collected new dataset of 14 common real-world objects with different materials, captured from multiple viewpoints in a diverse of real-world environments, paired with their ground-truth 3D scans and environment lighting.
3.1 Evaluation Benchmarks
We design a number of metrics to evaluate the recovered shape and material from three perspectives. We give an outline of these benchmarks in this section and lay out the metric definitions in Section
Geometry Estimation Benchmark. To assess the quality of the 3D shape reconstructions, we provide ground-truite 3D scans of each object. These 3D scans are obtained using high-definition 3D scanner and carefully processed into watertight meshes. We measure the quality the geometry estimated from different methods by comparing the predicted depth maps and normal maps those generated from the ground-trute scans, as well as directly computing the bidirectional Chamfer Distance between the predicted meshes and the scans.
Novel Scene Relighting Benchmark. Evaluating surface materials is challenging as obtaining ground-truie data is generally difficult in practice, and different methods might adopt different
3

---

Scanned Mesh
In-the-Wild
w/ Texture
Data
Posp Processing
X3
Geomeny Sean
Light Box Captuee
In Whe-
Stadio Capture
Multi-Soune the Wild Capture
Image Poses
Studio Dara Processing
To-the-Wild Data Processing
Poses
3D Mesh
MER M Maturtal
Kendered
Maskedl Image Maskedl Image
Env. Maps
Imagte
Dugth
Maps
Scanned Mesh with Pseudo Matecial
In-the- Data
Benchmark Data
Data Capture Pipeline Overview. For each object, Left: we obtain its 3D shape using 3D scanner and Physics-Based Rendering (PBR) materials using high-quality light box images. Middle: we also capture multi-view masked images in 3 different tn-th scenes, together with the ground-truth environment maps. Right: carefully register the camera poses for all images using the scanned mesh and recovered materials, and prepare the data for the evaluation benchmarks. Credit Maurice Svay
Figure I:
4 for the low-poly camera mesh model.
material representations. Here, propose to evaluate the quality of material recovery through the task of relighting in a real-world environment unseen during training, given the ground environment lighting, regardless of the underlying material representations. To do this, for each object, we capture multi-view image sets under multiple in-the-wild scenes, together with the ground-truth environment map without the object for each scene. To evaluate the performance of model, render the object using the predicted geometry and materials and the ground-truth lighting of new scene, and compare the rendered image with the captured ground-truth image from given viewpoint.
Novel View Synthesis Benchmark. In addition to the novel-scene relighting evaluation, we also perform evaluation the typical novel view synthesis task, where novel views of the object rendered the same scene as the input but from novel viewpoints, and compared to the ground -ruth images. Note that this metric measures the accuracy of appearance modeling within one scene, but not necessarily the accuracy of material modeling that ensures accurate appearance arbitrary scenes.
3.2
Data Capture
To support these evaluations, we capture a new real-world dataset comprised of ground-truth 3D scans, multi-view images taken the wild, and the corresponding environ- ment lighting. The dataset contains 14 common objects captured in natural scenes. For each object, we take 60 training views and 10 testing views in high dynamic range (HDR) under 3 different scenes, resulting in a total of 42 scenes. For each testing view, also capture HDR panorama enviromment map. The ground-truth shape for each object obtained using professional 3D scanner. Fig. gives an overview of our data capture pipeline.
Figure 2: Selection of Objects. From top to bottom: Block, Gnome, Ball, Car; Curry, Pepsi, Salt, Baking, Chips; Cac- tus, Pitcher, Grogu, Cup, Teapot.
3.2.1 Object Selection
Existing work often curates a small selection of objects for evaluation specific to the focus of their proposed methods. For instance, PhySG 85 evaluates on glossy objects, and NeRD tests on objects with rougher surfaces. To establish a comprehensive benchmark, we carefully select 14 objects that cover a wide variety of both geometry and material properties. Fig. givess glimpse into the list of objects. The geometry ranges from simple cylindrical shapes complex shapes,

---

(a) Geometry scanning
(b) Light box scanning
Figure 3: Studio Capture Setup. (a) 3D shape scanning. A: object, B: hand-held EinScan Pro HD 3D Scanner, C: printed patterns for camera registration, D: spray for high-quality scanning. E: desktop for processing. The scanned mesh is visualized ExScan Pro [60] and MeshLab [3] on the right. (b) Light box capture setup viewed from outside and inside. F: DSLR camera, G: cloth cover to block light from outside, H: object, printed patterns for camera registration, J: (optional) dark background for object segmentation, K: remote-controlled turntable. The captured image and the chrome ball are visualized on the right.
and the materials cover diffuse, glossy as well as mirror-like metal surfaces. We focus primarily on non-translucent objects in this work.
3.2.2 3D Shape and Appearance Capture in Studio
For each object, we obtain its ground 3D shape using an EinScan Pro HD 3D Scanner. Fig. illustrates the capture setup. Despite the high scanning quality in general, the scanner struggle to reconstruct shiny and dark surfaces for some objects, and we paint these areas with a special spray designed to improve the robustness. To help with camera registration, from the HPatches dataset I and place them underneath the objects. Each object scanned 2-3 times in different poses cover the entire shape. All scans are aligned and processed in the ExScan Pro software [60] to obtain a watertight textured mesh. For computation efficiency, we reduce the number of faces in each mesh to 200,000 using the method proposed by Garland al. [22] and further smooth the surfaces using the HC Laplacian smoothing algorithm from Vollmer et al. [68]
print geometric patterns
In addition to the 3D shapes, also obtain pseudo material decomposition for each object by capturing them in a light box. These pseudo materials are used to refine relative camera poses for in-the-wild scenes detailed in Section 3.3 and to build set of high-performing baseline results on the relighting benchmark as shown Table 2 As illustrated in Fig. 3 we place each object on a contrllcontoolled turntable inside simple light box. The turntable rotates 6 at time; bracket of three images are taken from each view at different exposure, which are fused into a high dynamic range (HDR) image for high-quality recovery using a simplified version of the algorithm in [18]. To solve for materials, we also capture the environment map inside the light box using a 3-inch steel chrome ball, following [80]. The chrome ball is placed in the same place the object and reflects the environment light, which allows us to recover the environment map as detailed in Section 3.3
3.2.3 Image and Lighting Capture in the Wild
To build the relighting and novel view synthesis benchmarks, capture dense multi-view images of each object different real-world in-the-wild environments. together with the corresponding ground-truth environment lighting. Each object is captured in 3 different scenes (7 unique scenes in total), including both indoor and outdoor environments, resulting in a total of 36 captures.
Fig. illustrates the capture setup. In each scene, fixate the object to small platform and move the camera around the object in circle from various heights. We take images from approximately 70 viewpoints roughly uniformly covering 360" views of the objects, including 10 test views and 60 training views, which are suitable for training most existing inverse rendering methods.
For each scene, we also capture the ground-truie truth environment lighting for relighting evaluation. Similar to Section we reuse the chrome ball and solve for the environment maps following {0) Unlike in the studio setup, where lighting is highly constrained and static, real-world (e.g.,, outdoor) environments are constantly changing while we capture the data. To minimize such errors, we capture one environment map per test view, forming an 'imaac-envmap' pair illustrated in Fig. 4 (b). To
5

---

Oh Only
Gbe ay Chrome Ball
Solved Eny. Map
(a) Hardware
(b) Image- Soage-Envmap Pair
Figure 4: In-the Wild Capture Setup. (a) Hardware for capturing. A: chrome ball, B: magenta platform for object segmentation, C: object, D: printed patterns for camera registration, E: DSLR camera, F: cloth for hiding photographer, G: mobile cart. (b) An example of the image-envmap pair. The environment map solved from the reflection image the chrome ball.
Scuare Mus
DOLMAP
Mirk
Tritdl Roses
Kookle
Jayn Images
Light Oap
Joper Image
Ourpor Mask
Emarce
1t.1
Silver
Mask Antonsion
Studie Capture Prnocssingg
Alieed
Scamadd Mask
Misk
Rekrenes Chunce Bull
Masuc Dijua
Nettrftashoaa budib
Zmy:
Makel map
Ean. Map
Refred Poes Optimizaion
FBR Soste
Lxuge
Tex. Map
Procssing
NYDifike: Opismizziio Modurie
Figure 5: Data Processing Pipeline. Left: Overview of the data processing pipelines for both studio and in-the-wild captures. Three individual modules painted blue are expanded on the right. Middle-Top The semi- aonii-utommaicallyy segmentation module produces object masks for all images. Middle-Bottom: Environment maps are solved from the chrome ball images. Right: Accurate camera poses are obtained from COLMAP and refined using NVDiffRec (44), given the scanned mesh and (for in-the-wild images) the pseudo materials optimized from light box captures.
avoid swapping the object for the chrome ball every time (which leads to inconsistent poses), we affix the object and only move the chrome ball in front of the object in each view. The photographer will hide themselves inside a white cloth underneath the camera to avoid being captured in the images.
For better inverse rendering quality, all images taken in 2048 2048 resolution and in HDR by fusing multiple exposure shots. We carefully tune the exposure values for each scene, taking into account the lighting condition and the reflections on the object surface. All cameras are calibrated with chessboard and undistorted using standard OpenCV libraries.
3.3 Data Processing
Next, we describe the detailed processing steps for preparing the evaluation data. essence, we need to obtain: (1) object masks for all images, (2) environment maps solved from the chrome ball images, (3) pseudo materials for each object using the studio captures, and (4) relative camera poses across multiple views and across multiple scenes. Fig, 5 shows the overview of the pipeline, and the

---

details of each step are laid out below. Overall, for each object, it takes roughly 3 hours to capture the raw data (2.5 hours for the in-the-wild capture and 0.5 hour for studio capture). The data processing pipeline takes roughly 10 hours on one machine for each object, which can be paralleled, and an additional 0.5 hour of human labor to manually adjust the annotations and alignments.
Object Masks. Getting high-quality inverse rendering results requires highly accurate object masks. We make use of the state-of-the-art object segmentation model, SAM [27). which is trained on over billion curated masks. Given sparse set of around 5 query points, the model produces reasonable object masks on our images. However, we observe that the resulting masks often appear smaller than the actual object. Hence, we further refine the masks using the classic GrabCut algorithm [53].
Environment Maps. Given the chrome ball images, we first apply the same object masking procedure described above to segment the chrome ball. We then fit a synthetic 3D sphere to the chrome ball by optimizing its 3ID location w.r.t the camera. With differentiable Monte-Carlo-ben renderer, we optimize the environment map to fit the reflection image on the chrome ball using the single-view light estimation method proposed in [80].
Pseudo Material Decomposition. For the purpose of refining the in-the-wild camera poses as detailed below, we optimize set of pseudo surface materials for each object through inverse rendering method NVDiffRec [44], making use of the 3D scan and the images captured in the light box. Specifically,
first register all light box images using COLMAP [56 (57] and SuperGlue [55] features to obtain a set of initial camera poses. In order to fit our 3D scanned mesh to the images, we optimize coarse mesh with NVDifiRec, and roughly align the scanned mesh the optimized mesh manually. The ground-truth environment map is also computed from the chrome ball images. Finally, optimize the materials with a microfacet BRDF model 17 using NVDiffRec, jointly refining the COLMAP poses like (72), given the scanned mesh and ground-truth environment map.
Camera Pose Registration. For both training and evaluation, we need accurate relative camera poses both between different viewpoints of scene and across different scenes for relighting. Similarly to the registration of light box images above, make use of the 3D scans and the pseudo materials previously solved from the studio captures. For each scene, we solve for initial poses using COLMAP and SuperGlue, optimize coarse mesh with NVDifiRee, and align the scanned mesh to it. The camera poses are further refined using NVDifiRec given the seanned mesh, pseudo materials, and ground-truth lighting. The relative pose between each source and target scenes for relighting is thus obtained through the same canonical scanned mesh.
Benchmarking State-of-the-AAt Methods
This section provides an overview of baseline method followed by evaluation details for the three benchmarks from Section 3.1 and discussion of the results.
4.1 Baseline Methods
Existing inverse rendering methods can be roughly categorized into
Material Decomposition methods from multi-view images, such NeRD 7 Neural- PIL [9]. PhySG [85), InvRender (75), NVDifiRec 44] and NVDiffRecMC 24 which
typically recovers surface materials as BRDFs and environment lighting of scene:
Novel View Synthesis and 3D Reconstruction methods from multi-view images, such as NeRF 43 and IDR (7)., which focus primarily on reconstructing 3D geometry and free-viewpoint appearance synthesis of a scene, without explicit material decomposition;
Single-View Prediction methods, including SIRFS 2 and SI-SVBRDF [30), which focus primarily on predicting intrinsic images, e.g., depth, normal, albedo, and shading maps, and potentially full 3D reconstructions, from a single image.
Note that these methods can be either per-scene optimization-bl basieatttio-baaddd...o. learning based, or hybrid, where priors learned from large training datasets can be exploited for test time inference with limited views.
"We include several representative methods in this paper. While this is not an exhaustive list, we provide instructions for future models to submit results to the benchmark and host the most up-to-date results in our public repository: https: http::///gttt //tgtttt.t.tt//////////.ge...e conm/StanfordO0///taatttt wStp://aaaaaadaaeeeeeea
7

---

NVD:fRRecMC w/ scanned med:
IDR
Neural PIL
NVDidRec
InvReeder
SI SVORDT
NVDitRee scarned
Physe
NVDDIRRaMM
Geometry
Relighting
0.8
35
0.15
1,00
0.6
30
0.10
0.95
25
0.05
0.90
15
Normal Cos Dist. $
10
PSNR-HI
0.00
LPIPS
0.85
SSIM
61.48
Novel View Synthesis
35
0.15
1,00
30
0.30
0.95
25
20
15
Depth SI MSE x10004
PSNR-EIt
LPIPS
SSIM
Figure 6: Visualization of the Benchmark Comparisons. Metrics on geometry are visualized on the left, and metrics on relighting and novel view synthesis are on the top right and the bottom right. The black line on each bar indicates the standard deviation of the scores across all scenes.
All methods assume access to images and camera poses during optimization or training. Inverse rendering methods that additionally assume access to the ground truth shape [36 37 or ground truth are not included in this benchmark.
lighting 48 47
78
Furthermore, we design two strong baselines, using NVDifiRec 44 and NVDifRecMC [24] with ground-truth 3D scans and pseudo materials optimized from light-box captures. As shown in Table 2 these two baselines achieve the best results in relighting as expected. For all methods, we use their public codebase and default configurations in all experiments, and feed in HDR images if possible by default. All models are trained and tested with 512 x 512 images, except SI-SVBRDF (30) which trained and tested with 256 x 256 images (results are upsampled to 512 for consistency).
4.2 Tasks
We consider three inverse rendering tasks: geometry estimation, novel scene relighting, and novel view synthesis.
Geometry Estimation. We evaluate geometry reconstruction quality by comparing the rendered depth maps, normal maps, and predicted 3D meshes to the ground truth obtained from 3D scans. For per-scene multi-view optimization methods, we evaluate the predictions on held-out test views of the same scene. For single-imag prediction methods, we simply run the pre- pre-trained model (30] or optimization [2] on all test images and evaluate the results. Since neither of [30 predicts novel views or full 3D shapes, we only evaluate their depth and normal predictions and omit the other two benchmarks.
To evaluate depth maps, we use the Scale-Invariant Mean Squared Error (SI-MSE) 21 between pre- dicted and trruh images masked with ground-truth object masks: min Σx= K [1(T a aTpeed.)) @ M1..s1.. where @ denotes the Hadamard product, and is the scale optimized for all K testing views each scene. For normal evaluation, we compute the Cosine Distance be- tween the predicted and ground-truth -ruth masked normal maps. For direct shape evaluation, we follow DeepSDF [49] to compute the bi-directional Chamfer Distance between 30, 000 points sampled from the predicted mesh surface and ground-truth mesh vertices. For methods that do not produce explicit meshes 77
88 75), we run the marching cube algorithm (38 for mesh extraction. Following IDR 79 for all methods, take the largest connected component the final mesh prediction. rare cases where a method fails to converge during the training, we simply sample 30k dummy points at the origin.
Novel Scene Relighting. We evaluate the quality of material decomposition through the task of relighting under novel scenes with the ground truth environment lighting. To do this, we take

---

Table 2: Benchmark Comparison of Existing Methods. tdenotes models trained with the ground- truth 3D scans and pseudo materials optimized from light-box captures. Depth SI-MSE x10 Shape Chamfer distance x x10-3.
Geometry
Novel Scene Relighting
Novel Synthesis
Depth Normal,
Shape
PSNR-HG
PSNR-L1
SSIMf
LPIPSL
PSNR-HA PSNR-L1
SSIMF LPIPSL
MCDiffRe:MC 24
N/A
25. 08
32.28
0.974
0.027
NIA
NVDifRec
N/A
24.93
32 42
0.975
0.027
N/A
IDR 29
30.11
NeRF 48
2.19
0.02
62.05
N/A
26.31
33.59
0,968
0.044
Neural PII.
0
0.86
0.29
4.14
N/A
25.79
33.35
0.963
0.051
Ph 31
1.90
0.17
9.28
21.81
28. 11
0.960
0.055
24.24
32:15
0.974
0.047
NVDifRee
44
0.31
0.06
0.62
22.91
29.72
0.963
0.039
21.94
28.44
0.969
0.030
NeRD 7
1.39
0.28
13.70
23. 29
29.66
0.957
0.059
25.83
32.61
0.963
0.054
NeRFactor
88
9.53
0.969
26.06
33.41
0.046
InvRender
75
0.59
0.06
0.44
23.76
30.83
0.970
0.046
25.91
34.01
0.977
0.042
NVDifTRecMC
0.51
24.43
31.60
0.972
0.036
28.03
36.40
0.962
0.028
SI-SVBRDE
30
81.48
0.29
NA
N/A
N/A
SIRES
2
N/A
0.59
N/A
N/A
N/A
decomposed results one training scene and render them in the other 2 scenes with the respective ground-truth lighting. Note that IDR, NeRF, and Neural-PIL use implicit lighting or appearance representations and cannot directly support relighting. Hence, we do not evaluate relighting for these methods.
We adopt three widely-used metrics to compare the rendered relit images against the ground-truth images: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [70], and Learned Perceptual Image Patch Similarity (LPIPS) [86). Since there is an inherent scale ambiguity in material and lighting decomposition, we follow PhySG [85) and adapt all metrics to be scale-invariant by rescaling the predictions with an optimized scale for each RGB channel. Specifically, given the ground-truth image Ig and the predicted image Iprdd we rescale the prediction 1' pred (argmin, (Il /
gt &yymell (2 / before computing the metrics.
We report PSNR in both HDR and LDR (after the standard sRGB tone mapping), denoted as PSNR-H and PSNR-L, respectively. The HDR values are clamped to a maximum of to suppress the effects of over-saturated pixels (e.g., mirror reflection of the sun). SSIM and LPIPS computed with LDR values. In cases where the baselines fail to converge during training, we use a constant gray image to represent the prediction.
Novel View Synthesis. We also report the typical novel view synthesis metrics on the same scene training. Specifically, for each trained model, we render the object from test views under the same scene and compare the rendered images with ground-truth test images. The two single-view inverse rendering methods 30 ( do not explicitly perform view synthesis, and therefore not evaluated on this task. We use the same metrics as in relighting (without rescaling), namely PSNR-H, PSNR-L, SSIM, and LPIPS.
4.3 Results
We briefly summarize the key findings of the results in Table 2 and Fig. 6. For the tasks of geometry reconstruction and novel view synthesis (NVS), IDR [79] outperforms all other baselines, including NeRF (3). potential reason is that IDR focuses on surface reconstruction and represents the geometry using a signed distance function (SDF) [49], hence particularly advantageous in modeling these object-centric scenes. While NeRF is powerfull in modeling large-scale scenes, it fails to reconstruct consistent geometry and texture details for certain scenes (see Fig. 3 in the Appendix).
7 [8) for further decomposition of lighting and surface materials tend to result in degradation of performance in recover better geometry. suggesting the advantages of the underlying DMTet (58) -based shape representation over SDF or NeRF. NVDiffRecMC also significantly outperforms NVDiffRec and other material decomposition baselines NVS thanks the differentiable Monte Carlo-bascd (MC) renderer.
Inverse rendering methods that extend from IDR ( 885 (55) or NeRF
geometry reconstruction and NVS. In comparison, NVDiffRec (24] and NVDiffRecMC

---

For the task of relighting, the two most recent inverse rendering methods [75 24 performance than the earlier ones. In particular, NVDiffRecMC again improves upon its preeursor NVDiffRec and consistently outperforms all other inverse rendering baselines across all metrics, further confirming the effectiveness of the MC-based renderer. Finally, as expected, the two baselines that have access to ground truth 3D scans and pseudo-materials (marked with 1) perform significantly better than methods without this information, and this gap suggests room for improvement for all
achieve better existing methods. Here, the performance gain from using the MC renderer seems negligible.
Please refer to the Appendix for more results. particular, we additionally evaluate and compare the albedo prediction quality of various methods using the albedo maps optimized from the studio capture data as pseudo ground-truth. The complete comparison results are summarized in Table I of the Appendix, and few visual examples are provided in Fig. 3 of the Appendix.
5 Conclusion
We present Stanford-ORB, the first real-world evaluation benchmark for object inverse rendering tasks including shape reconstruction, object relighting, and novel view synthesis under in-the- wild environments. To build this benchmark, we capture a new dataset of 14 real-world objects under various in-the-wild scenes with ground-truth 3D scans, multi-view images, and ground-truth environment lighting. Using the dataset, we evaluate and compare a wide range of sfatc-of-the-at inverse rendering methods. All data collected in this work are focused on daily objects and do not contain any personal information. Please refer to the website for more information on the dataset: https://stanfordoon github.ion
This work was part supported by NSF CCRI #2120095, RI #221 ONR MURI N00014-22-1-27400 the Stanford Institute for Human-Centered Al (HAI), Adobe, Amazon, Ford, and Google.
Acknowledgments.
References
[1] Vassilcios Baintas, Karel Lene, Andrea Vodaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In CVPR, 2017.
[2] Jonathan Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE TPAMI. 37(8):1670-1687, 2014.
[3] Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in wild. ACM TOG, 33 2014.
[4] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold food-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In ECCV, 2020.
[5] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. Deep 3D capture: Geometry and reflectance from sparse multi-view images. In CVPR, 2020.
[6] Martin Bichsel and Alex P Pentland, simple algorithm for shape from shading. In CVPR, 1992.
[7] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD; Neural reflectance decomposition from image collections. In ICCV, 2021.
[8] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. SAMURAL: Shape and material from unconstrained real-world arbitrary image collections. In NeurlPS, 2022.
[9] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-PII.: Neural pre-integrated lighting for reflectance decomposition. In NeurlPS, 2021.
[10] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot Spatially-wayying BRDF and Shape Estimation. In CVPR, 2020.
[11] Guangyan Cai, Kai Yan, Zhao Dong, loannis Gkioulekas. and Shuang Zhao. Physics-based inverse rendering using combined implicit and explicit geometries. CGF, (1(4): 2022.
[12] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation. In ICCV, 2023.
10

---

[13] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang. Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. DIB-R++: learning to predict lighting and material with hybrid differentiable renderer. In NeurIPS, 2021.
[14] Cheng Chi, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Iterative residual policy for goal-conditioned dynamic manipulation of deformable objects. In RSS, 2022.
[15] Paolo Cignoni, Marco Callieri, Massimiliano Corsini, Matteo Dellepiane, Fabio Ganovelli, and Guido Ranzuglia. MeshLab: open-source mesh processing tool. In Eurographics Italian Chapter Conference. Salerno, Italy, 2008.
[16] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. ABO: Dataset and benchmarks for real-world 3D object understanding. In CVPR, 2022.
[17] Robert L Cook and Kenneth E. Torrance, A reflectance model for computer graphics. ACM TOG, 1(1):7224, 1982.
[18] Paul E Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH Classes, pages 1-10, 2008.
[19] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In CVPR, 2023.
[20] Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, and Mare Toussaint. Reinforcement learning with neural radiance fields. In NeurIPS, 2022.
[21] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NeurlPS, 2014.
[22] Michael Garland and Paul Heckhert. Simplifying surfaces with color and texture using quadric error metrics. In Proceedings Visualization, pages 263-269. IEFF, 1998.
[23] Roger Grosse, Micah Johnson, Edward H Adelson, and William T Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, 2009.
[24] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light & material decomposition from images using Monte Carlo rendering and denoising. In NeurlPS, 2022.
[25] Michael Janner, Jiajun Wu, Tejas Kulkarni, Ilker Yildirim, and Joshua B Tenenbaum. Sele-supervised intrinsic image decomposition. In NeurlPS. 2017.
[26] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, 2014.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023.
[28] Zhengfei Kuang, Kyle Olszewski, Menglei Chai, Zeng Huang, Panos Achlioptas, and Sergey Tulyakov. NeROIC: neural rendering of objects from online image collections. ACM TOG, 41(4):1-12, 2022.
[29] Min Li, Zhenglong Zhou, Zhe Wu, Boxin Shi, Changyu Diao, and Ping Tan. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEFE TIP.
29:4159-4173, 2020.
[30] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varyinn lighting and SVBRDF from single image. In CVPR, 2020.
[31] Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. Materials for masses: SVBRDF acquisition with single mobile phone image. In ECCV, 2018
[32] Zhenggin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from single image. ACM TOG, 1-11, 2018.
[33] Zhengqin Li, Ting- Wei Yu, Shen Sang. Sarah Wang. Meng Song, Yuhan Liu, Yu-Ying Ying Yeh, Rui Zhu, Nitesh Gundavarapu, Jia Shi, et al. OpenRooms: An open framework for photorealistic indoor scene datasets. In CVPR, 2021.
11

---

[34] Andrew Liu, Shiry Ginosar. Tinghui Zhou, Alexei A. Efros, and Noah Snavely. Learning to factorize and relight city. In ECCV, 2020.
[35] Isabella Liu, Linghao Chen, Ziyang Fu, Liwen Wu, Haian Jin, Zhong Li, Chin Ming Ryan Wong, Yi Xu, Ravi Ramamoorthi, Zexiang Xu, and Hao Su. Openillumination: A multi-llumination dataset for inverse rendering evaluation on real ohjects. NeurlPS, 2023.
[36] Stephen Lombardi and Ko Nishino. Reflectance and natural illumination from single image. ECCV, 2012.
[37] Stephen Lombardi and Ko Nishino. Reflectance and illumination recovery in the wild. IEFE TPAMI, 2015.
[38] William Lorensen and Harvey Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM SIGGRAPH Computer Gruphics, 21(4):163-169, 1987.
[39] Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. Unified shape and SVBRDF recovery using diffcrentiable Monte Carlo rendering. CGF, 4014)1101111,, 2021.
[40] Stephen Robert Marschner. Inverse Rendering for Computer Gruphics. Comell University, 1998.
[41] Maurice Svay. https://skecchfab.comm htwp://eeecaaa..ommmmmmmmeoooooo wwwp://wwwyy..ooooaaaaaaaaaaaaaa
[42] Nelson Max. Optical models for direct volume rendering. IEEE TVCG, (2):99-108, 1995.
[43] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[44] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3D models, materials, and lighting from images. In CVPR, 2022.
[45] Shree K. Nayar. Xi-Sheng Fang, and Terrance Boult. Separation of reflection components using color and polarization. NCV, 1997.
[46] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR, 2020.
[47] Geoffrey Oxholm and Ko Nishino. Multiview shape and reflectance from natural illumination. In CVPR, 2014.
[48] Geoffrey Oxholm and Ko Nishino. Shape and reflectance cstimation in the wild. IEEE TPAMI, 2015.
[49] Jeong Joon Park, Peter Florence. Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[50] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3D using 2D diffusion. In ICLR, 2023.
[51] Yuzhe Qin. Binghao Huang, Zhao-Heng Yin, Hao Su, and Xiaolong Wang. DexPoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation. In CoRL, 2022.
[52] Anurag Ranjan, Kwang Moo Yi, Rick Chang, and Oncel Tuzel. FaceLit: Neural 3D relightable faces. In CVPR, 2023.
[53] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. GrabCut: Interactive foreground extraction using iterated graph cuts. ACM TOG, 23(3):309-314, 2004.
[54] Viktor Rudney, Mohamed Elgharib. William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theohalt. Nerf for outdoor scene relighting. In ECCV, pages 61556611 Springer, 2022.
[55] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020.
[56] Johannes Lutz Schönberger and Jan Jan-Michael Frahm. Structure-front motion revisited. In CVPR, 2016.
[57] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
[58] Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In NeurIPS, 2021. 2021.
12

---

[59]
Jian Shi, Yue Dong. Hao Su, and Stella X Yu. Learning non-Lambertian object intrinsics across ShapeNet categories. In CVPR, 2017.
[60] Shining
3D.
2023.
exscan- Mascan-pro--oe 1rr00soottare-eo download/
[61] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL., 2023.
[62] sRGB Tonc-mapping Formula, 2012.
[63] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
[64] Cheng Sun, Guangyan Cai, Zhengqin Li, Kai Yan, Cheng Zhang, Carl Marshall, Jia- Bin Huang, Shuang Zhao, and Zhao Dong. Neural- PBrRal-PIIRRaae reconstruction of shape, material, and illumination. In ICCV, 2023,
[65] Feitong Tan, Sean Fanello, Abhimitra Meka, Sergio Orts-Escolano, Danhang Tang, Rohit Pandey, Jonathan Taylor, Ping Tan, and Yinda Zhang. Volux-gan: A generative model for 3d face synthesis with hdri relighting. In SIGGRAPH, 2022.
[66] Stephen Tian, Yancheng Cai, Hong-Xing Yu, Sergey Zakharov, Katherine Liu, Adrien Gaidon, Yunzhu Li, and Jiajun Wu. Multi-object manipulation via object-centric neural scattering functions. In CVPR, 2023.
[67] Marco Toschi, Riccardo De Mattco, Riccardo Spezialetti, Daniele De Gregorio, Luigi Di Stefano, and Samuele Salti. Relight my NeRF: A dataset for novel view synthesis and relighting of real world objects. In CVPR, 2023.
[68] Jörg Vollmer, Robert Menel, and Heinrich Mueller. Improved laplacian smoothing of noisy surface meshes. CGF, 18(3):311-38, 1999.
[69] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theohalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NewrlPS, 2021.
[70] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4);600- 612, 2004.
[71] Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3D vpatially-varyinng lighting. In ICCV, 2021.
[72] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF--; Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.0706/ 2021.
[73] Henrique Weber, Donald Prévost, and Jean-François Lalonde. Learning estimate indoor lighting from 3D objects. In 3DV, 2018
[74] Felix Wimbauer. Shangzhe Wu, and Christian Rupprecht. De-rendering 3D objects in the wild. In CVPR, 2022.
[75] Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, and Xin Yu. NeFII: Inverse rendering for reflectance decomposition with near-field indirect illumination. In CVPR, 2023.
[76] Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa. De-rendering the world's revolutionary artefacts. In CVPR, 2021.
[77] Zhenjia Xu, Zhanpeng He, Jiajun Wu. and Shuran Song. Learning 3D dynamic scene representations for robot manipulation. In CoRL, 2020.
[78] Kohei Yamashita, Yuto Enyo, Shohei Nobuhara, and Ko Nishino. nemvs-net: Deep non- lon-eambertian multi-view stereo. In WACV, 2023.
[79] Lior Yariv, Yoni Kasten, Dror Moran, Mcirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurlPS, 2020.
[80] Hong-Xing Yu, Samir Agarwala, Charles Herrmann, Richard Szeliski, Noah Snavely, Jiajun Wu, and Deqing Sun. Accidental light probes. In CVPR, 2023.
[81] Hong-Xing Yu, Michelle Guo, Alireza Fathi, Yen- Yu Chang, Eric Ryan Chan, Ruohan Gao, Thomas Funkhouser, and Jiajun Wu. Learning object-centrie neural scattering functions for free-vine viewpoint relighting and seene composition. TMLR, 2023.
13

---

[82] Ye Yu, Abhimetra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and Will Smith. Self-supervieed outdoor relighting, In ECCV, 2020.
[83] Ye Yu and William AP Smith. InverseRenderNet: Learning single image inverse rendering. In CVPR, 2019.
[84] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. IRON: Inverse rendering by optimizing neural SDFs and materials from photometric images. CVPR, 2022.
[85] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021.
[86] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
[87] Ruo Zhang, Ping- Tsai, James Edwin Cryer, and Mubarak Shah. Shape-foom-shadng survey, IEEE TPAMI, 21 2168):690-706, 1999.
[88] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. NeRFactor: Neural factorization of shape and reflectance under unknown illumination. ACM TOG, 40(6):1-18, 2021.
[89] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination inverse rendering. In CVPR, 2022.
[90] Yunzhi Zhang, Shangzhe Wu, Noah Snavely, and Jiajun Wu. Sceing rose in five thousand ways. In CVPR, 2023.
[91] Taotao Zhou, Kai He, Di Wu, Teng Xu, Qixuan Zhang, Kuixiang Shao, Wenzheng Chen, Lan Xu, and Jingyi Relightable neural human assets from multi-view gradient illuminations. CVPR, 2023.
[92] Rui Zhu, Zhengqin Li, Janarbek Matai, Fatih Porikli, and Manmohan Chandraker. IRISformer: Dense vision transformers single-image inverse rendering indoor scenes. CVPR, 2022.
14

---

F
Additional Dataset Details
We list all the in-the-wild scenes and captured objects in Fig. 7 Our dataset is captured in 5 indoor scenes and 2 outdoor scenes (one in the daytime, one at night), and each object is captured in two indoor scenes and one outdoor scene. For each object-scene pair, capture around 60 training images and 10 testing image-envmap pairs. The camera positions are evenly spaced in a 360-degree circle around the object, and at three different elevation levels (roughly 0", 20", and 40%). As result, the in-the-wild dataset consists of 2,975 object image brackets and 418 chrome ball image brackets.
For the light box capture, since the camera and the environment are fixed during the capture, we use universal chrome ball image bracket for multiple objects captured in the same setting. There are in total two settings: For objects with white surfaces, we attach several black papers in the light box as background. For other objects, we do not add anything in the light box. In total, 872 object image brackets are captured.
All data, scripts. documents, and metadata can be accessed via hhttp::///dde github io/ We ve made our dataset available under the MIT license. For long-teri sustainability, we' ve housed our code on Github and our data sets on Google Drive, as they both ensure prolonged availability.
F.1 Image Processing
We fuse the exposre RAW image brackets into HDR images. In our setting, captures in each bracket share the same f-number and ISO, and only differ in the exposure times. For a bracket of raw images Ri..., captured with exposure times of fi....s let M' represent the mask of non-saturated (i.e., not overly exposed) pixels of each image, the output HDR image is generated by:
a R M,
Li=1 M44
where @ represents the Hadamard product.
We also provide LDR images for the following processing steps and the final benchmark. As some of the HDR images are too bright, we rescale them to ensure that 99% of the rescaled pixel values are less than 1. The scale factor is calculated globally across all images for each object in each scene to keep the brightness of the images consistent. The scaled images are then tone-mapped with the commonly used sRGB formula 62]
For the camera calibration, we directly extract the focal length (fz, fy) from the EXIF metadata, and optimize the principle point (Ca, Cy) and dis- tortion parameters (2 parameters for radial distor- tion kn, li and 2 for tangential distortion Pl, P2). To do that, we print the chessboard pattern on a paper sheet, as shown in Fig. on the right. We then take pictures from multiple viewpoints and apply the calibration algorithm in the OpenCV library solve for the camera intrinsics. All images in the dataset are then undistorted using the estimated camera intrinsics.
Figure 8: Example images of the printed chessboard patterns for calibration.
F.2 Environment Map Solver
As discussed in the paper, we fit synthetic 3D ball recover the environment map from the chrome ball image. This process consists of two steps. The first step optimizes the pose of the synthetic ball w.r.t. the camera, by minimizing two losses given the masks of the chrome ball and the rendered synthetic ball: L2 loss between the masks of two balls, and the L2 loss between the center pixels of two balls. The second step optimizes the environment map from the synthetic ball. Here each iteration, we randomly sample batch of rays from the camera and render them with a differentiable path tracer. The environment map is optimized from the L2 loss between the rendered color and the ground truth color from the image.
15

---

Environment Map
Selected Object Images
Object List
Scene001
Baking, Grogu,
Pitcher, Teapot
Scene002
Baking, Grogu, Pepsi, Car, Ball, Teapot, Blocks, Chips
Soanc003
Baking, Gnome, Grogu, Pepsi, Cup, Ball, Chips
Scen:004
Salt, Pepsi, Car, Ball, Chips
Scene005
Salt, Gnome, Pitcher, Blocks
Scenc006
Car, Cup, Teapot, Blocks
Scene007
Salt, Gnome, Cup, Pitcher
Figure 7: Overview of the dataset. We capture the objects in 7 different in-the-wild scenes. For each scene, we showcase the environment maps on the left, three examples of the objects captured in the middle, and the full list of objects in the last column.
16

---

F33 Material Optimization for Studio Data
To process the studio data, the pseudo surface materials of the captured object are generated from an NVDiffRec-based 4 optimization pipeline. As explained in the paper, it is done with two optimization steps, one to obtain a coarse mesh from scratch and align it with the scanned mesh, and the other to optimize the materials from the aligned scanned mesh. Here we provide additional details of the two passes. In the first pass, we train the DMTet module of the NVDiffRec from scratch for 400 iterations with batch size of and learning rate of 3 x 10-2 The coarse mesh is then extracted from the resulting SDF using Deep Marching Tetrahedra 588. After aligning the optimized mesh to the scanned mesh, the second pass optimizes the DLMesh module of NVDiffRec for 2000 iterations, given the scanned mesh and the environment map from the chrome ball. Here, all trainable components are frozen except the material textures and camera poses. The learning rate is set to 3 x 10-3 and the batch size is set to 16. We use the same losses proposed in NVDiffRec.
E.4 Pose Optimization for In-the-wild Data
To process the in-the-wild data, the same NVDiffRec-based optimization module is deployed to obtain accurate camera poses of in-the-wild images. The setting of this module resembles the previous one, with only a few differences in the second pass. Here, the material textures are frozen, and the environment map and the camera poses are optimized. The training consists of 2000 iterations with a batch size of 16. In addition to the losses in NVDifiRec, we also apply the regularization loss from NeRF-- 72 to regularize the camera transformations. For each data point, we try multiple runs with different loss weights and learning rates, and select the set of parameters for the best performance in terms of PSNR on the test images.
G More Results
As supplement to Table 2 in the main paper, 3 presents the full benchmark results with all metries and baselines, including the standard deviations.
Albedo Benchmark. In addition to the three benchmarks proposed in the main paper, we also compare albedo maps predicted by basclines with the pseudo ground truth albedo maps optimized from the studio capture. We compute scale-invariant metrics, PSNR-L, SSIM, and LPIPS, as introduced in Section 4.2 of the main paper to evaluate the results. The results reported in Table 3 We do not evaluate other material properties (such as roughness) as different methods adopt different material representations (e.g., Lambertian, Phong, microfacet BRDFs).
More Qualitative Results. Finally, we show qualitative results of all baselines on two example scenes in Fig. 9 including the normal maps, depth maps. view synthesis images, relighting images, and albedo maps. In particular, to examine why IDR [79] achieves a higher performance in the geometry reconstruction and novel view synthesis benchmark, Fig. 10 provides visual comparison of the two methods on two example scenes. IDR reconstructs sharper texture details compared to NeRF, yielding higher novel view synthesis score.
We also observed that despite having similar geometry. objects with different appearances can lead to drastically different inverse rendering results. For example, although the Baking can and the Salt can have similar cylindrical shapes, NVDiffRecMC (24) achieves a PSNR-HDR score of 31.0 (view synthesis) / 24.4 (relighting) on Baking and 17.9 (view synthesis) / 7.5 (relighting) on Salt. Visual comparisons are presented in Fig. [T] In particular, the black texture of the Salt can poses significant challenges to the inverse rendering task due to the inherent ambiguity between lighting and texture.
H Limitations
Although the current version of the dataset already contains a variety of objects, it 300 does not cover translucent objects, like glass. It also remains challenging to capture thin, deformable objects like leaves as their shape will change while being transported to different scenes, breaking the consistency for novel scene relighting, and the scanner also tends to fail in capturing extremely intricate geometry details. We plan to continue expanding the collection of objects. Currently, we only capture one object in scene at a time. Extending it to multi-object scenes will allow us to further analyze the
17

---

Novel View
Novel View
Normal
Depth
Relight
Albedo
Normal
Depth
Relight
Albedo
Ground Truth
NVDiffRecMC*
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
NVDiffRee*
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
IDR
N/A
N/A
N/A
NeRF
N/A
N/A
N/A
Neural-PII.
PhySG
NVDifRRee
NeRD
NeRFactor
InvRender
NVDifReeMC
SI-SVBRDF
N/A
N/A
N/A
N'A
SIRFS
N/A
N/A
NAA
N'A
Figure 9: Qualitative Comparisons of Baseline Methods. We show qualitative results on two example scenes. denotes models trained with the ground-truth 3D scans and pscudo materials optimized from light-box captures.
18

---

Curry
Chips
Ground Truth
IDR
NeRF
Figure 10: Comparisons of IDR and NeRF on Novel View Synthesis. We compare the novel view synthesis results of IDR and NeRF on two example scenes, Chips and Curry. IDR recovers more texture details and hence achieves higher scores.
Baking
Salt
MORTON
VICHTTOR
Ground Truth
Prediction (Novel View)
Ground Truth (Relighting)
Prediction (Relighting)
(Novel View)
Figure 1I: Result Comparisons on Two Can-shaped Objects with Different Appearances. We compare the novel view synthesis and relighting results of NVDiffRecMC [24] on two objects: Baking and Salt. Despite their similar geometry, these two objects lead to drastically different results due to the different in their textures.
effects of global illumination. The current environment maps extracted from individual chrome ball images do capture the environment behind the ball, and we plan to align and stitch the maps from all views, which will enable lighting evaluation.
I Societal Impact
The purpose of this work is to facilitate the development of general object inverse rendering methods by contributing real-world evaluation dataset and benchmark. We believe there is direct negative societal impact resulting from this benchmark. The technology of inverse rendering has numerous applications that will have positive societal impact, such as improving the realism of computer graphics for virtual and augmented reality, or aiding in scene understanding for robotics applications. However, if leveraged for malicious intentions, it could create significant negative impact society. For instance, extremists might exploit an inverse rendering methods to generate fake imagery, commonly known 'deepfakes', which may lead to the spread of misinformation even intimidate the public. Nevertheless, we believe these possibilities can be eliminated with sufficient supervision from the governments and the research communities. It is important note that our work focuses solely on common everyday objects, steering clear of any personal information or hazardous items.
19

---

Table 3: Full Benchmark Com pampssiion of Existing Methods. tdenotes models trained with the ground-truth 3D scans and pseudo materials optimized from light-box captures. Depth SI-MSE x10-3 Shape Chamfer distance x10- x10-3
0. 0.058 =5 0223
0.043 021
11.08800 1126
1.051=0.020
0,040=0 0200
0.038 0.038.00.0 0..018
0.064 1. 023
LPPSS
0,051
010007
0. 0.870-s: 014
0..80022: 00000000
011
1. 11.001 20.012
0..0220.000 .010
0.986 0.980.1...0 000
0 0.987 021
1.977400002
Alhado
NA NW
NA
N/A
01.85523
PSNR-L1
36 8023377
40 4.
401 10..5133.51
40 00.3233.88
41 11.0113.57
3 $1.111351
36
40,
0.017 D09
0.04420.000
086
0.05 150 022
1.04720.000 02002
0.000000111
0.08420.022
0.D4250011
0.02 0.0281..00 1.0001011
0.012
LPDSS
1.968=0.08 0.08
0.963=0000 DET
1074-00014
0.977-6.011
0.982 0.008
Novel Synntesss
SSIMI
0. D
0.969
N/A
N/A
N/
PSNR-LI
30 08-66.2222 ag
33.50 00.50477.22 28
33..55533.00 0M
222 01 153.00
34.01
36 40.43.3.47
33.35
32.
25
11:-1 34
26. 20..1125.60 08
25. 70.7008... 75
00
18
2n. 27.N223...
20 01.1123..6
28 08.0334440
24
21
0.97120.013 0.027.00.022
0.027:0.0012 0012
0.080200022
0.03910001
0.005922.022
0.048500.013
0,0422001
0.0 1006600.01
LPIPS
0. 6.97540.012
0. 0.960200002
0. 9696440011
0.057
1.989-00-033
1.97020011
0. 0.97240.014
Novel Relighting
SSIM
N/A
X/A
NA
N/A
21-22822-000 32. 02.42222.55
35
28.1 15.1123.15
29.65 6065522.16
30 833233255 57
31.60
PSNR
29
24 00..9822.44
21.8 21.8122.00
22 11.01:2330
23 24:244444444
21.7632272
24 03.4331..00
23
18.701130 28
0 0.30440.30 35
62 22.064178..1
4.44110.05
0.6240.74
0-443-055
0.5L
Stape:
NA
NAA
Geounery
Narmal
NA
AAA
0.05=0:00
0.62-6.34 34
0.29-0.20 26
0.17=0.22
0.06-0:07
66.8
0.06-0066
0.0466.0
Depth
0.35220000 06
2. 19119.44222 27
0. 0.8611100 96
1.0025-07
1.3110000
1.3022-42 4-42
0.8920.00
0.32
81 31.48117.88
NVDiMRRRMM 14it
RI
NVDaiRe: 64)
411
NVDIIRRM
BESVBRDFF K0
3
Neural PIL
PhySC [S]
NeRD +
Inv Kenden
NeR1
IDR
20


Reviews:
[{"review_id": "SAv4LE2Q6e", "reviewer": "Reviewer_uqj7", "summary": "", "questions": "", "limitations": "Limitations are explicitly discussed the paper.\nSocietal impact is not discussed, but I do not expect any negative impact from this work.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper makes a good effort of creating a realistic dataset while ensuring the availability of GT data.\nI particularly appreciate the effort to go beyond a fully controlled studio setup by additionally capturing objects \n\n- The paper presents an elaborate evaluation of various baseline methods on the proposed dataset.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "4lCafBbWlS", "reviewer": "Reviewer_gijK", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "> + It seems to be the first real-world dataset that supports \"shape\", \"relit image\", and \"lighting\" by providing ground-truth scans, multi-view images, and environment lighting, as summarized in Table 1\n> + Clear figures: The figures are easy-to-follow and clearly show the data processing pipeline and benchmark comparisons.\n> + Comprehensive experiments: Representative inverse rendering methods are included in the experiments.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vXLxt3XDYT", "reviewer": "Reviewer_DSB3", "summary": "", "questions": "", "limitations": "I could not find the discussion about negative societal impacts in the supplement but the datasheet shows that a negative impact is unlikely.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The dataset is comprehensive and includes ground truth for geometry, and images for relighting and novel view synthesis.\n\n- Data is captured in real environments. This is in contrast to artificial light stage setups which simplify acquisition but produce data captured in an unrealistic environment.\n\n- The paper evaluates multiple state-of-the-art methods.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FWqItGaxuD", "reviewer": "Reviewer_8Dpe", "summary": "", "questions": "", "limitations": "Though the paper contains a coverage of limitations, it lacks a discussion of the potential negative social impact of their work. \n\nWithin the limitations, the claim that the dataset contains a variety of objects is not well-substantiated, as indicated above. It would be more beneficial to frame the discussion of limitations from the standpoint of the current narrow range of objects, and how should future collections be structured to diversify the range. \n\nThere are concrete negative social impacts of 'unchecked' inverse rendering that should be discussed in the paper. High-quality inverse rendering methods pave the way for high-fidelity 3D image manipulation, where traditional techniques to detect image forgery (e.g., Kee et al.) are likely to fail. 3D image manipulation has been an area of significant concern when it has come to the manipulation of faces, however, manipulation of objects can have negative consequences ranging from socially concerning to devastating, with legal and national security implications. E.g., imagine being able to insert a nuclear missile into an image with high fidelity (even though at least one prior attempt was successfully detected: https://www.npr.org/2008/07/11/92442928/photo-of-irans-missile-launch-was-manipulated), or a firearm in the hands of an innocent person in a photograph. \n\n[It is understandable that the paper is a benchmark for other algorithms rather than an algorithm itself. However, the very contribution of a benchmark serves as the channel for future researchers to contribute more improved algorithms, making it imperative for benchmark methods to share in the burden of negative consequences.]\n\nKee, E., O'Brien, J. F., & Farid, H. (2014). Exposing Photo Manipulation from Shading and Shadows. ACM Trans. Graph., 33(5), 165-1.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The work has significance in overcoming the limitations of existing inverse rendering datasets, that, if real-world, lack detailed object-centric geometry focusing solely on lighting and appearance disambiguation, or when providing richer detail, are synthetically generated. \n\nThe authors have invested a significant amount of effort in the incredibly cumbersome data acquisition process that goes into full 360 scene understanding of a single scene, having acquired dense full-range 3D scans, high-dynamic range environment maps, and multi-viewpoint images. The data acquisition process is well detailed, both in the main paper and in the supplementary material. \n\nThe challenge addressed of acquiring full-scale real-world 3D scene data for tasks in inverse rendering has the benefit of enabling the research community to stress test multiple aspects of inverse rendering algorithms, such as geometry, illumination, and appearance, with respect to real-world ground truth.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All four reviewers recommend acceptance, with three of them stating it is a good paper (clear accept). The reviewers appreciated the realism of the proposed dataset, the quality of the acquisition, the extensive baseline experiments, and the clarity of the writing. The AC additionally likes the timeliness of the dataset, which seems covering a real need of the community these days.
Decision: Accept (Poster)