SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Geometry optimization: We optimize the geometry model by minimizing the difference between rendered cues and observed cues.
$$
\min  _ {S, C} E _ {\text {c o l o r}} + \lambda_ {1} E _ {\text {r e g}} + \lambda_ {2} E _ {\text {m a s k}} + \lambda_ {3} E _ {\text {n o r m a l}}, \tag {4}
$$
where  $\lambda_1 = 0.1$ ,  $\lambda_2 = \lambda_3 = 0.5$ . And each term is defined as follows:
Multi-view	Albedo	Roughness	Relighting	Env Light	Geometry
PSNR ↑	MSE ↓	PSNR ↑	MSE ↓	CD ↓
PhySG	16.233	0.087	21.323	0.054	0.024
Nv-DiffRec	16.123	0.116	17.418	0.168	0.268
InvRender	16.984	0.084	22.224	0.067	0.024
Ours	21.961	0.026	25.486	0.029	0.011
Single-view	Albedo	Roughness	Relighting	Env Light	Geometry
PSNR ↑	MSE ↓	PSNR ↑	MSE ↓	CD ↓
PhySG*	14.977	0.255	18.504	0.082	0.033
Nv-DiffRec*	14.021	0.165	17.214	0.067	0.050
InvRender*	14.724	0.247	17.998	0.082	0.033
Ours	17.629	0.062	21.374	0.052	0.034
Table 1: (Left) Multi-view inverse rendering on synthetic data. Both our model and the baseline are trained on multi-view images. Our model is significantly better than baseline in terms of geometry and PBR texture. (Right) Single-view inverse rendering on synthetic data. While our model is trained on a single-view image, the baselines * are trained on 10 multi-view images of the same scene.
- The color consistency term  $E_{\mathrm{color}}$  is a L2 color consistency loss between the rendered color  $\mathbf{c}$  and the observed color  $\hat{\mathbf{c}}$  for all pixel rays:  $E_{\mathrm{color}} = \sum_{\mathbf{r}} \| \mathbf{c}_{\mathbf{r}} - \mathbf{c}_{\mathbf{r}} \|_2$ .
- The normal consistency term  $E_{\mathrm{normal}}$  measures the rendered normal  $\hat{\mathbf{n}}$  and a predicted normal  $\hat{\mathbf{n}}$ :  $E_{\mathrm{normal}} = \sum_{\mathbf{r}} \| 1 - \hat{\mathbf{n}}_{\mathbf{r}}^{\mathrm{T}} \mathbf{n}_{\mathbf{r}} \|_1 + \| \hat{\mathbf{n}}_{\mathbf{r}} - \mathbf{n}_{\mathbf{r}} \|_1$ . Our monocular predicted normal  $\hat{\mathbf{n}}$  is obtained using a pretrained Omnidata model [13].
- The mask consistency term  $E_{\mathrm{mask}}$  measures the discrepancy between the rendered mask  $\mathbf{m}_{\mathbf{r}}$  and the observed mask  $\hat{\mathbf{m}}_{\mathbf{r}}$ , in terms of binary cross-entropy (BCE):  $L_{\mathrm{mask}} = \sum_{\mathbf{r}} \mathrm{BCE}(\mathbf{m}_{\mathbf{r}}, \hat{\mathbf{m}}_{\mathbf{r}})$ .
- Finally, inspired by NeuS [60], we incorporate an Eikonal regularization to ensure the neural field is a valid signed distance field:  $L_{\mathrm{eikonal}} = \sum_{\mathbf{x}} (\|\nabla_{\mathbf{x}} S\|_2 - 1)^2$ ,
Visibility optimization: Ambient occlusion and self-cast shadows pose challenges to the accuracy of inverse rendering, as it's difficult to separate them from albedo when optimizing photometric loss. However, with an estimated geometry, we can already obtain a strong visibility cue. Consequently, we utilize ambient occlusion mapping to prebake the visibility map onto the object surfaces obtained from the previous stage. We then minimize the visibility consistency term to ensure the rendered visibility  $v_{\mathbf{r}}$  from MLP  $V$  aligns with the derived visibility  $\hat{v}_r$ :  $\min_V \sum_{\mathbf{r}} \mathrm{BCE}(v_{\mathbf{r}}, \hat{v}_{\mathbf{r}})$ .
Material and illumination optimization: In the final stage, given the obtained surface geometry and the visibility network, we jointly optimize the environmental light and the PBR material network. The overall objective is as follows:
$$
\min  _ {M, \omega , \phi} E _ {\text {c o l o r}} + \lambda_ {4} E _ {\text {s p a r s e}} + \lambda_ {5} E _ {\text {s m o o t h}} + \lambda_ {6} E _ {\text {m e t a l}}, \tag {5}
$$
where  $\lambda_4 = 0.01$ ,  $\lambda_5 = 0.1$ ,  $\lambda_6 = 0.01$ . The four terms are defined as follows:
- The color consistency term  $E_{\mathrm{color}}$  minimizes the discrepancy between the rendered color and observed color, akin to the geometry optimization stage. However, we use PBR-shaded color in place of the color queried from the auxiliary radiance field.
- The sparse regularization  $E_{\mathrm{sparse}}$  constrains the latent code  $\pmb{\rho}$  of the material network to closely align with a constant target vector  $\pmb{\rho}'$ :  $E_{\mathrm{latent}} = \mathrm{KL}(\pmb{\rho}||\pmb{\rho}')$ . We set  $\pmb{\rho}' = 0.05$ .
- The smooth regularization  $E_{\mathrm{smooth}}$  force the BRDF decoder to yield similar values for close latent code  $z$ :  $E_{\mathrm{smooth}} = \| D(z) - D(z + dz)\|_1$ , where  $dz$  is randomly sampled from normal distribution  $N(0; 0.01)$ .
- Lastly, inspired by the fact that most common objects are either metallic or not, we introduce a metallic regularization  $L_{\mathrm{metal}}$  to encourage the predicted metallic value to be close to either 0 or 1:  $L_{\mathrm{metal}} = \sum_{\mathbf{r}} m_{\mathbf{r}} (1 - m_{\mathbf{r}})$ .
In this section, we evaluate the effectiveness of our model on synthetic and real-world datasets, analyze its characteristics, and showcase its applications.
Data: Since existing multi-view datasets do not contain duplicate objects, we collect  $Dup$ , a novel inverse rendering dataset featuring various duplicate objects.  $Dup$  consists of 13 synthetic and 6
Albedo	Roughness	Env Light	Mesh	Rotation	Translation
Num	PSNR ↑	MSE ↓	MSE ↓	CD ↓	degree ↓	length ↓
6	18.48	0.13	0.116	0.025	2.515	0.070
8	18.26	0.09	0.172	0.017	0.600	0.003
10	18.99	0.09	0.204	0.029	0.186	0.003
20	19.22	0.09	0.063	0.021	0.299	0.005
30	19.69	0.07	0.119	0.020	0.448	0.006
50	18.56	0.08	0.153	0.038	0.464	0.008
60	16.45	0.14	0.150	0.091	51.512	0.644
Albedo	Roughness	Metallic	Relighting
Model	PSNR ↑	MSE ↓	MSE ↓	PSNR ↑
full model	17.27	0.30	0.02	19.95
w/o clean seg	16.68	0.23	0.13	19.12
w/o metal loss	17.09	0.19	0.08	19.02
w/o latent smooth	17.00	0.23	0.02	19.71
w/o normal	16.51	0.42	0.97	18.81
w/o eik loss	16.79	0.51	0.97	19.31
w/o mask	15.14	0.29	0.02	18.11
Table 2: (Left) Performance vs. number of duplicates for "color box" dataset. We highlight the best, second and third values. (Right) Ablation study for the contribution of each loss term.
Rendering
PSNR ↑	SSIM ↑	LPIPS ↓	Albedo	Roughness	Relighting	Env Light Geometry
PhySG*	20.624	0.641	0.263	PSNR ↑	MSE ↓	PSNR ↑	MSE ↓ CD ↓
Nv-DiffRec*	18.818	0.569	0.282	M-S	20.229	0.096	21.328	0.045 0.010
InvRender*	20.665	0.639	0.262	S-M	23.448	0.050	24.254	0.052 0.007
Ours	20.326	0.660	0.192
Table 3: (Left) Single-view inverse rendering on real-world data. * indicates that the baselines are trained on multi-view observations. (Right) Multi-view single-object (M-S) vs. single-view multi-object (S-M).
real-world scenes, each comprising 5-10 duplicate objects such as cans, bottles, fire hydrants, etc. For synthetic data, we acquire 3D assets from PolyHaven† and utilize Blender Cycles for physics-based rendering. We generate 10-300 images per scene. As for the real-world data, we place the objects in different environments and capture 10-15 images using a mobile phone. The data allows for a comprehensive evaluation of the benefits of including duplicate objects in the scene for inverse rendering. We refer the readers to the supp. materials for more details.
Metrics: Following prior art [74; 69], we employ Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and LPIPS [71] to assess the quality of rendered and relit images. For materials, we utilize PSNR to evaluate albedo, and mean-squared error (MSE) to quantify roughness and environmental lighting. And following[60; 65; 47], we leverage the Chamfer Distance to measure the distance between our estimated geometry and the ground truth.
Baselines: We compare against three state-of-the-art multi-view inverse rendering approaches: Physg [69], InvRender [74], and NVdiffrec [47]. Physg and InvRender employ implicit representations to describe geometric and material properties, while NVdiffrec utilizes a differentiable mesh representation along with UV textures. Additionally, we enhance Physg by equipping it with a spatially-varying roughness.
Implementation details: We use the Adam optimizer with an initial learning rate 1e-4. All experiments are conducted on a single Nvidia A40 GPU. The first stage takes 20 hours, and the 2nd and 3rd stage takes about 2 hours. Please refer to the supp. material for more details.
Single-view inverse rendering: We first evaluate our approach on the single-image multi-object setup. Since the baselines are not designed for this particular setup, we randomly select another 9 views, resulting in a total of 10 multi-view images, to train them. As shown in Tab. 1(right), we are able to leverage duplicate objects to constrain the underlying geometry, achieving comparable performance to the multi-view baselines. The variations in lighting conditions across instances also aid us in better disentangling the effects of lighting from the materials.
Multi-view inverse rendering: Our method can be naturally extended to the multi-view setup, allowing us to validate its effectiveness in traditional inverse rendering scenarios. We utilize synthetic data to verify its performance. For each scene, we train our model and the baselines using 100 different views and evaluate the quality of the reconstruction results. Similar to previous work, we
$\dagger$  https://polyhaven.com/models
Figure 6: The surface normal and rendering result on real-world cola image. Our model has the smoothest surface normal compared with other baselines.
assume the ground truth poses are provided. As shown in Tab. 1(left), we outperform the baselines on all aspects. We conjecture that this improvement stems from our approach explicitly incorporating a material- and geometry-sharing mechanism during the modeling process. As a result, we have access to a significantly larger number of "effective views" during training compared to the baselines. We show some qualitative results in Fig. 4(left).
Real-world single-view inverse rendering: Up to this point, we have showcased the effectiveness of our approach in various setups using synthetic data. Next, we evaluate our model on real-world data. Due to the challenge of obtaining highly accurate ground truth for materials and geometry, we focus our comparison solely on the rendering results. As indicated in Tab. 3(Left), our method achieves comparable performance to the multi-view baselines, even when trained using only a single view. We visualize some results in Fig. 1, Fig. 6 and Fig. 7.
Ablation study for each loss term: As shown in Tab. 2(right), since the metallicity of natural materials are usually binary, incorporating the metallic loss can properly regularize the underlying metallic component and prevent overfitting; Eikinol loss and mask loss help us better constrain the surface and the boundary of the objects, making them more accurate and smooth. Removing either term will significantly affect the reconstructed geometry and hence affect the relighting performance; The pre-trained surface normal provides a strong geomeotry prior, allowing us to reduce the ambiguity of sparse view inverse rendering. Removing it degrades the performance on all aspects.
Importance of the number of duplicate objects: Since our model utilize duplicate objects as a prior for 3D reasoning, one natural question to ask is: how many duplicate objects do we need? To investigate this, we render 9 synthetic scenes with  $4\sim 60$  duplicates with the same object. We train our model under each setup and report the performance in Tab. 2. As expected, increasing the number of duplicates from 6 to 30 improves the accuracy of both material and geometry, since it provides more constraints on the shared object intrinsics. However, the performance decrease after 30 instances due to the accumulated 6 DoF pose error brought by heavier occlusion, the limited capacity of the visibility field, and limited image resolution.
The influence of different geometry representation: To demonstrate that our method does not rely on a specific neural network architecture, we replace the vanilla MLP using Fourier position encoding to triplane representation from PET-NeuS[61] and hash position encoding from neuralangelo[35] respectively. The result (please refer to supplementary material) shows that the triplane (28.0MB) can recover better geometry and texture than our original model (15.8MB) because the explicit planes provide higher resolution features and can better capture local detail[4]. However, the model of hash positional encoding (1.37GB) and produces high frequency noisy in geometry, indicating that it overfits the training view.
Figure 7: Qualitative comparison on real-world data. InvRender [74] takes as input 10 images, while we only consider one single view. Yet our approach is able to recover the underlying geometry and materials more effectively.
Multi-view single object (M-S) vs. single-view multi-objects (S-M): Is observing an object from multiple views equivalent to observing multiple objects from a single view? Which scenario provides more informative data? To address this question, we first construct a scene containing 10 duplicate objects. Then, we place the same object into the same scene and capture 10 multi-view images. We train our model under both setups. Remarkably, the single-view setting outperforms the multi-view setting in all aspects (see Tab. 3(right)). We conjecture this discrepancy arises from the fact that different instances of the object experience environmental lighting from various angles. Consequently, we are better able to disentangle the lighting effects from the material properties in the single-view setup. Fig. 5(right) shows some qualitative results.
**Applications:** Our approach supports various scene edits. Once we recover the material and geometry of the objects, as well as the illumination of the scene, we can faithfully relight existing objects, edit their materials, and seamlessly insert new objects into the environment as if they were originally present during the image capturing process (see Fig. 1).
Limitations: One major limitation of our approach is that we require the instances in each image to be nearly identical. Our method struggles when there are substantial deformations between different objects, as we adopt a geometry/material-sharing strategy. One potential way to address this is to loosen the sharing constraints and model instance-wise variations. We explore this possibility in the supplementary material. Furthermore, our model requires accurate instance segmentation masks. Additionally, our approach currently requires decent 6 DoF poses as input and keeps the poses fixed. We found that jointly optimizing the 6 DoF pose and SDF field with BARF [37] will cause the pose error to increase. This is because recovering camera pose from sparse views and changing light is a difficult non-convex problem and prone to local minima. We believe the more recent works, like CamP[48] and SPARF[58] could potentially further refine our estimations. Moreover, our model cannot effectively model/constrain the geometry of unseen regions similar to existing neural fields methods.
We introduce a novel inverse rendering approach for single images with duplicate objects. We exploit the repetitive structure to estimate the 6 DoF poses of the objects, and incorporate a geometry and material-sharing mechanism to enhance the performance of inverse rendering. Experiments show that our method outperforms baselines, achieving highly detailed and precise reconstructions.
[64] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. CVPR, 2023.
[65] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurIPS, 2020.
[66] Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, 1999.
[67] Albert J Zhai and Shenlong Wang. Peanut: Predicting and navigating to unseen targets. arXiv, 2022.
[68] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In CVPR, 2022.
[69] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5453-5462, 2021.
[70] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[71] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. IEEE, 2018.
[72] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1-18, 2021.
[73] Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv, 2020.
[74] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18643-18652, 2022.
[75] Yunzhi Zhang, Shangzhe Wu, Noah Snavely, and Jiajun Wu. Seeing a rose in five thousand ways. arXiv, 2022.
[76] Tinghui Zhou, Philipp Krahenbuhl, and Alexei A Efros. Learning data-driven reflectance priors for intrinsic image decomposition. In ICCV, 2015.
[77] Maria Zontak and Michal Irani. Internal statistics of a single natural image. In CVPR. IEEE, 2011.