SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Existing inverse rendering methods can be roughly categorized into
- Material Decomposition methods from multi-view images, such as NeRD [7], NeuralPIL [9], PhySG [85], InvRender [75], NVDiffRec [44] and NVDiffRecMC [24], which typically recovers surface materials as BRDFs and environment lighting of a scene;
- Novel View Synthesis and 3D Reconstruction methods from multi-view images, such as NeRF [43] and IDR [79], which focus primarily on reconstructing 3D geometry and free-viewpoint appearance synthesis of a scene, without explicit material decomposition;
- Single-View Prediction methods, including SIRFS [2] and SI-SVBRDF [30], which focus primarily on predicting intrinsic images, e.g., depth, normal, albedo, and shading maps, and potentially full 3D reconstructions, from a single image.
Note that these methods can be either per-scene optimization-based, learning-based, or hybrid, where priors learned from large training datasets can be exploited for test time inference with limited views.
2We include several representative methods in this paper. While this is not an exhaustive list, we provide instructions for future models to submit results to the benchmark and host the most up-to-date results in our public repository: https://github.com/StanfordORB/Stanford-ORB.
Figure 6: Visualization of the Benchmark Comparisons. Metrics on geometry are visualized on the left, and metrics on relighting and novel view synthesis are on the top right and the bottom right. The black line on each bar indicates the standard deviation of the scores across all scenes.
All methods assume access to images and camera poses during optimization or training. Inverse rendering methods that additionally assume access to the ground truth shape [36, 37] or ground truth lighting [48, 47, 78] are not included in this benchmark.
Furthermore, we design two strong baselines, using NVDiffRec [44] and NVDiffRecMC [24] with ground-truth 3D scans and pseudo materials optimized from light-box captures. As shown in Table 2, these two baselines achieve the best results in relighting as expected. For all methods, we use their public codebase and default configurations in all experiments, and feed in HDR images if possible by default. All models are trained and tested with  $512 \times 512$  images, except SI-SVBRDF [30] which is trained and tested with  $256 \times 256$  images (results are upsampled to 512 for consistency).
We consider three inverse rendering tasks: geometry estimation, novel scene relighting, and novel view synthesis.
Geometry Estimation. We evaluate geometry reconstruction quality by comparing the rendered depth maps, normal maps, and predicted 3D meshes to the ground truth obtained from 3D scans. For per-scene multi-view optimization methods, we evaluate the predictions on held-out test views of the same scene. For single-image prediction methods, we simply run the pre-trained model [30] or optimization [2] on all test images and evaluate the results. Since neither of [30, 2] predicts novel views or full 3D shapes, we only evaluate their depth and normal predictions and omit the other two benchmarks.
To evaluate depth maps, we use the Scale-Invariant Mean Squared Error (SI-MSE) [21] between predicted and ground-truth images masked with ground-truth object masks:  $\min_{\alpha \in \mathbb{R}}\sum_{k = 1}^{K}\frac{1}{K}\| (I_{\mathrm{gt},k} - \alpha I_{\mathrm{pred},k})\odot M_{\mathrm{gt},k}\| _2^2$ , where  $\odot$  denotes the Hadamard product, and  $\alpha$  is the scale optimized for all  $K$  testing views of each scene. For normal evaluation, we compute the Cosine Distance between the predicted and ground-truth masked normal maps. For direct shape evaluation, we follow DeepSDF [49] to compute the bi-directional Chamfer Distance between 30,000 points sampled from the predicted mesh surface and ground-truth mesh vertices. For methods that do not produce explicit meshes [79, 43, 9, 85, 7, 88, 75], we run the marching cube algorithm [38] for mesh extraction. Following IDR [79], for all methods, we take the largest connected component as the final mesh prediction. In rare cases where a method fails to converge during the training, we simply sample 30k dummy points at the origin.
Novel Scene Relighting. We evaluate the quality of material decomposition through the task of relighting under novel scenes with the ground-truth environment lighting. To do this, we take
Table 2: Benchmark Comparison of Existing Methods. †denotes models trained with the ground-truth 3D scans and pseudo materials optimized from light-box captures. Depth SI-MSE ×10-3. Shape Chamfer distance ×10-3.
Geometry	Novel Scene Relighting	Novel View Synthesis
Depth↓	Normal↓	Shape↓	PSNR-H↑	PSNR-L↑	SSIM↑	LPIPS↓	PSNR-H↑	PSNR-L↑	SSIM↑	LPIPS↓
NVDiffRecMC [24]†	N/A	25.08	32.28	0.974	0.027	N/A
NVDiffRec [44]†	N/A	24.93	32.42	0.975	0.027	N/A
IDR [79]	0.35	0.05	0.30	N/A	30.11	39.66	0.990	0.017
NeRF [43]	2.19	0.62	62.05	N/A	26.31	33.59	0.968	0.044
Neural-PIL [9]	0.86	0.29	4.14	N/A	25.79	33.35	0.963	0.051
PhySG [85]	1.90	0.17	9.28	21.81	28.11	0.960	0.055	24.24	32.15	0.974	0.047
NVDiffRec [44]	0.31	0.06	0.62	22.91	29.72	0.963	0.039	21.94	28.44	0.969	0.030
NeRD [7]	1.39	0.28	13.70	23.29	29.65	0.957	0.059	25.83	32.61	0.963	0.054
NeRFactor [88]	0.87	0.29	9.53	23.54	30.38	0.969	0.048	26.06	33.47	0.973	0.046
InvRender [75]	0.59	0.06	0.44	23.76	30.83	0.970	0.046	25.91	34.01	0.977	0.042
NVDiffRecMC [24]	0.32	0.04	0.51	24.43	31.60	0.972	0.036	28.03	36.40	0.982	0.028
SI-SVBRDF [30]	81.48	0.29	N/A	N/A	N/A
SIRFS [2]	N/A	0.59	N/A	N/A	N/A
decomposed results on one training scene and render them in the other 2 scenes with the respective ground-truth lighting. Note that IDR, NeRF, and Neural-PIL use implicit lighting or appearance representations and cannot directly support relighting. Hence, we do not evaluate relighting for these methods.
We adopt three widely-used metrics to compare the rendered relit images against the ground-truth images: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [70], and Learned Perceptual Image Patch Similarity (LPIPS) [86]. Since there is an inherent scale ambiguity in material and lighting decomposition, we follow PhySG [85] and adapt all metrics to be scale-invariant by rescaling the predictions with an optimized scale for each RGB channel. Specifically, given the ground-truth image  $I_{\mathrm{gt}}$  and the predicted image  $I_{\mathrm{pred}}$ , we rescale the prediction to  $I_{\mathrm{pred}}' = \{\arg \min_s(\|I_{\mathrm{gt}}^i - sI_{\mathrm{pred}}^i\|^2) \cdot I_{\mathrm{pred}}^i\}_{i = \mathrm{R}, \mathrm{G}, \mathrm{B}}$  before computing the metrics.
We report PSNR in both HDR and LDR (after the standard sRGB tone mapping), denoted as PSNR-H and PSNR-L, respectively. The HDR values are clamped to a maximum of 4 to suppress the effects of over-saturated pixels (e.g., mirror reflection of the sun). SSIM and LPIPS are computed with LDR values. In cases where the baselines fail to converge during training, we use a constant gray image to represent the prediction.
Novel View Synthesis. We also report the typical novel view synthesis metrics on the same scene as training. Specifically, for each trained model, we render the object from test views under the same scene and compare the rendered images with ground-truth test images. The two single-view inverse rendering methods [30, 2] do not explicitly perform view synthesis, and therefore are not evaluated on this task. We use the same metrics as in relighting (without rescaling), namely PSNR-H, PSNR-L, SSIM, and LPIPS.
We briefly summarize the key findings of the results in Table 2 and Fig. 6. For the tasks of geometry reconstruction and novel view synthesis (NVS), IDR [79] outperforms all other baselines, including NeRF [43]. A potential reason is that IDR focuses on surface reconstruction and represents the geometry using a signed distance function (SDF) [49], hence particularly advantageous in modeling these object-centric scenes. While NeRF is powerful in modeling large-scale scenes, it fails to reconstruct consistent geometry and texture details for certain scenes (see Fig. 3 in the Appendix). Inverse rendering methods that extend from IDR ([85, 75]) or NeRF ([9, 7, 88]) for further decomposition of lighting and surface materials tend to result in a degradation of performance in geometry reconstruction and NVS. In comparison, NVDiffRec [24] and NVDiffRecMC [24] recover better geometry, suggesting the advantages of the underlying DMTet [58]-based shape representation over SDF or NeRF. NVDiffRecMC also significantly outperforms NVDiffRec and other material decomposition baselines on NVS thanks to the differentiable Monte Carlo-based (MC) renderer.
For the task of relighting, the two most recent inverse rendering methods [75, 24] achieve better performance than the earlier ones. In particular, NVDiffRecMC again improves upon its precursor NVDiffRec and consistently outperforms all other inverse rendering baselines across all metrics, further confirming the effectiveness of the MC-based renderer. Finally, as expected, the two baselines that have access to ground truth 3D scans and pseudo-materials (marked with  $\dagger$ ) perform significantly better than methods without this information, and this gap suggests room for improvement for all existing methods. Here, the performance gain from using the MC renderer seems negligible.
Please refer to the Appendix for more results. In particular, we additionally evaluate and compare the albedo prediction quality of various methods using the albedo maps optimized from the studio capture data as pseudo ground-truth. The complete comparison results are summarized in Table 1 of the Appendix, and a few visual examples are provided in Fig. 3 of the Appendix.
We present Stanford-ORB, the first real-world evaluation benchmark for object inverse rendering tasks including shape reconstruction, object relighting, and novel view synthesis under in-the-wild environments. To build this benchmark, we capture a new dataset of 14 real-world objects under various in-the-wild scenes with ground-truth 3D scans, multi-view images, and ground-truth environment lighting. Using the dataset, we evaluate and compare a wide range of state-of-the-art inverse rendering methods. All data collected in this work are focused on daily objects and do not contain any personal information. Please refer to the website for more information on the dataset: https://stanfordorb.github.io/.
Acknowledgments. This work was in part supported by NSF CCRI #2120095, RI #2211258, ONR MURI N00014-22-1-2740, the Stanford Institute for Human-Centered AI (HAI), Adobe, Amazon, Ford, and Google.
[1] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk. Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In CVPR, 2017.
[2] Jonathan T Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE TPAMI, 37(8):1670-1687, 2014.
[3] Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM TOG, 33(4):1-12, 2014.
[4] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hasan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In ECCV, 2020.
[5] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. Deep 3D capture: Geometry and reflectance from sparse multi-view images. In CVPR, 2020.
[6] Martin Bichsel and Alex P Pentland. A simple algorithm for shape from shading. In CVPR, 1992.
[7] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural reflectance decomposition from image collections. In ICCV, 2021.
[8] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. SAMURAI: Shape and material from unconstrained real-world arbitrary image collections. In NeurIPS, 2022.
[9] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-PIL: Neural pre-integrated lighting for reflectance decomposition. In NeurIPS, 2021.
[10] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot Spatially-varying BRDF and Shape Estimation. In CVPR, 2020.
[11] Guangyan Cai, Kai Yan, Zhao Dong, Ioannis Gkioulekas, and Shuang Zhao. Physics-based inverse rendering using combined implicit and explicit geometries. CGF, 41(4):129-138, 2022.
[12] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. *Fantasia3D: Disentangling geometry and appearance for high-quality text-to-3D content creation*. In ICCV, 2023.
[82] Ye Yu, Abhimetra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and Will Smith. Self-supervised outdoor scene relighting. In ECCV, 2020.
[83] Ye Yu and William AP Smith. InverseRenderNet: Learning single image inverse rendering. In CVPR, 2019.
[84] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. IRON: Inverse rendering by optimizing neural SDFs and materials from photometric images. In CVPR, 2022.
[85] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snively. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021.
[86] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[87] Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah. Shape-from-shading: a survey. IEEE TPAMI, 21(8):690-706, 1999.
[88] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. NeRFactor: Neural factorization of shape and reflectance under an unknown illumination. ACM TOG, 40(6):1-18, 2021.
[89] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In CVPR, 2022.
[90] Yunzhi Zhang, Shangzhe Wu, Noah Snavely, and Jiajun Wu. Seeing a rose in five thousand ways. In CVPR, 2023.
[91] Taotao Zhou, Kai He, Di Wu, Teng Xu, Qixuan Zhang, Kuixiang Shao, Wenzheng Chen, Lan Xu, and Jingyi Yi. Relightable neural human assets from multi-view gradient illuminations. In CVPR, 2023.
[92] Rui Zhu, Zhengqin Li, Janarbek Matai, Fatih Porikli, and Manmohan Chandraker. IRISformer: Dense vision transformers for single-image inverse rendering in indoor scenes. In CVPR, 2022.
F Additional Dataset Details
We list all the in-the-wild scenes and captured objects in Fig. 7. Our dataset is captured in 5 indoor scenes and 2 outdoor scenes (one in the daytime, one at night), and each object is captured in two indoor scenes and one outdoor scene. For each object-scene pair, we capture around 60 training images and 10 testing image-envmap pairs. The camera positions are evenly spaced in a 360-degree circle around the object, and at three different elevation levels (roughly  $0^{\circ}$ ,  $20^{\circ}$ , and  $40^{\circ}$ ). As a result, the in-the-wild dataset consists of 2,975 object image brackets and 418 chrome ball image brackets.
For the light box capture, since the camera and the environment are fixed during the capture, we use a universal chrome ball image bracket for multiple objects captured in the same setting. There are in total two settings: For objects with white surfaces, we attach several black papers in the light box as background. For other objects, we do not add anything in the light box. In total, 872 object image brackets are captured.
All data, scripts, documents, and metadata can be accessed via https://3dorb.github.io/. We've made our dataset available under the MIT license. For long-term sustainability, we've housed our code on Github and our data sets on Google Drive, as they both ensure prolonged availability.
F.1 Image Processing
We fuse the multi-exposure RAW image brackets into HDR images. In our setting, captures in each bracket share the same f-number and ISO, and only differ in the exposure times. For a bracket of raw images  $R_{1,\dots,n}$  captured with exposure times of  $t_{1,\dots,n}$ , let  $M_{k}$  represent the mask of non-saturated (i.e., not overly exposed) pixels of each image, the output HDR image is generated by:
$$
I = \frac {\sum_ {i = 1} ^ {n} R _ {i} \odot M _ {i} t _ {i}}{\sum_ {i = 1} ^ {n} M _ {i} t _ {i}}, \tag {1}
$$
where  $\odot$  represents the Hadamard product.
We also provide LDR images for the following processing steps and the final benchmark. As some of the HDR images are too bright, we rescale them to ensure that  $99\%$  of the rescaled pixel values are less than 1. The scale factor is calculated globally across all images for each object in each scene to keep the brightness of the images consistent. The scaled images are then tone-mapped with the commonly used sRGB formula [62].
For the camera calibration, we directly extract the focal length  $(f_x,f_y)$  from the EXIF metadata, and optimize the principle point  $(c_x,c_y)$  and distortion parameters (2 parameters for radial distortion  $k_{1},k_{2}$  and 2 for tangential distortion  $p_1,p_2$ ). To do that, we print the chessboard pattern on a paper sheet, as shown in Fig. 8 on the right. We then take pictures from multiple viewpoints and apply the calibration algorithm in the OpenCV library to solve for the camera intrinsics. All images in the dataset are then undistorted using the estimated camera intrinsics.
Figure 8: Example images of the printed chessboard patterns for calibration.
F.2 Environment Map Solver
As discussed in the paper, we fit a synthetic 3D ball to recover the environment map from the chrome ball image. This process consists of two steps. The first step optimizes the pose of the synthetic ball w.r.t. the camera, by minimizing two losses given the masks of the chrome ball and the rendered synthetic ball:  $\mathcal{L}_2$  loss between the masks of two balls, and the  $\mathcal{L}_2$  loss between the center pixels of two balls. The second step optimizes the environment map from the synthetic ball. Here in each iteration, we randomly sample a batch of rays from the camera and render them with a differentiable path tracer. The environment map is optimized from the  $\mathcal{L}_2$  loss between the rendered color and the ground truth color from the image.
Figure 7: Overview of the dataset. We capture the objects in 7 different in-the-wild scenes. For each scene, we showcase the environment maps on the left, three examples of the objects captured in the middle, and the full list of objects in the last column.
F.3 Material Optimization for Studio Data
To process the studio data, the pseudo surface materials of the captured object are generated from an NVDiffRec-based [44] optimization pipeline. As explained in the paper, it is done with two optimization steps, one to obtain a coarse mesh from scratch and align it with the scanned mesh, and the other to optimize the materials from the aligned scanned mesh. Here we provide additional details of the two passes. In the first pass, we train the DMTet module of the NVDiffRec from scratch for 400 iterations with a batch size of 4 and a learning rate of  $3 \times 10^{-2}$ . The coarse mesh is then extracted from the resulting SDF using Deep Marching Tetrahedra [58]. After aligning the optimized mesh to the scanned mesh, the second pass optimizes the DLMesh module of NVDiffRec for 2000 iterations, given the scanned mesh and the environment map from the chrome ball. Here, all trainable components are frozen except the material textures and camera poses. The learning rate is set to  $3 \times 10^{-3}$  and the batch size is set to 16. We use the same losses proposed in NVDiffRec.
F.4 Pose Optimization for In-the-wild Data
To process the in-the-wild data, the same NVDiffRec-based optimization module is deployed to obtain accurate camera poses of in-the-wild images. The setting of this module resembles the previous one, with only a few differences in the second pass. Here, the material textures are frozen, and the environment map and the camera poses are optimized. The training consists of 2000 iterations with a batch size of 16. In addition to the losses in NVDiffRec, we also apply the regularization loss from NeRF--[72] to regularize the camera transformations. For each data point, we try multiple runs with different loss weights and learning rates, and select the set of parameters for the best performance in terms of PSNR on the test images.
G More Results
As a supplement to Table 2 in the main paper, Table 3 presents the full benchmark results with all metrics and baselines, including the standard deviations.
Albedo Benchmark. In addition to the three benchmarks proposed in the main paper, we also compare albedo maps predicted by baselines with the pseudo ground truth albedo maps optimized from the studio capture. We compute scale-invariant metrics, PSNR-L, SSIM, and LPIPS, as introduced in Section 4.2 of the main paper to evaluate the results. The results are reported in Table 3. We do not evaluate other material properties (such as roughness) as different methods adopt different material representations (e.g., Lambertian, Phong, microfacet BRDFs).
More Qualitative Results. Finally, we show qualitative results of all baselines on two example scenes in Fig. 9, including the normal maps, depth maps, view synthesis images, relighting images, and albedo maps. In particular, to examine why IDR [79] achieves a higher performance in the geometry reconstruction and novel view synthesis benchmark, Fig. 10 provides a visual comparison of the two methods on two example scenes. IDR reconstructs sharper texture details compared to NeRF, yielding a higher novel view synthesis score.
We also observed that despite having similar geometry, objects with different appearances can lead to drastically different inverse rendering results. For example, although the Baking can and the Salt can have similar cylindrical shapes, NVDiffRecMC [24] achieves a PSNR-HDR score of 31.0 (view synthesis) / 24.4 (relighting) on Baking and 17.9 (view synthesis) / 7.5 (relighting) on Salt. Visual comparisons are presented in Fig. 11. In particular, the black texture of the Salt can poses significant challenges to the inverse rendering task due to the inherent ambiguity between lighting and texture.
Although the current version of the dataset already contains a variety of objects, it 300 does not cover translucent objects, like glass. It also remains challenging to capture thin, deformable objects like leaves as their shape will change while being transported to different scenes, breaking the consistency for novel scene relighting, and the scanner also tends to fail in capturing extremely intricate geometry details. We plan to continue expanding the collection of objects. Currently, we only capture one object in a scene at a time. Extending it to multi-object scenes will allow us to further analyze the
Figure 9: Qualitative Comparisons of Baseline Methods. We show qualitative results on two example scenes. * denotes models trained with the ground-truth 3D scans and pseudo materials optimized from light-box captures.
Chips
Ground Truth
Figure 10: Comparisons of IDR and NeRF on Novel View Synthesis. We compare the novel view synthesis results of IDR and NeRF on two example scenes, Chips and Curry. IDR recovers more texture details and hence achieves higher scores.
Baking
Salt
Ground Truth (Novel View)
Figure 11: Result Comparisons on Two Can-shaped Objects with Different Appearances. We compare the novel view synthesis and relighting results of NVDiffRecMC [24] on two objects: Baking and Salt. Despite their similar geometry, these two objects lead to drastically different results due to the different in their textures.
Prediction (Novel View)
Ground Truth (Relighting)
Prediction (Relighting)
effects of global illumination. The current environment maps extracted from individual chrome ball images do not capture the environment behind the ball, and we plan to align and stitch the maps from all views, which will enable lighting evaluation.
I Societal Impact