SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given the augmented reality representation  $\{\mathcal{M}',\mathbf{E}^{\mathrm{src}},\mathbf{I}_{\mathrm{src}}'\}$ , we can perform physically-based rendering under a novel lighting condition  $\mathbf{E}^{\mathrm{tgt}}$  to generate a relit rendered video. The rendered images faithfully capture scene relighting effects, such as changes in shadows or overall scene illumination. However, due to imperfect geometry and noise in material/lighting decomposition, the rendering results lack realism (e.g., they may contain blurriness, unrealistic surface reflections and boundary artifacts). To mitigate this, we propose a photo-realism enhanced neural deferred rendering paradigm. Deferred rendering [16, 65] splits the rendering process into multiple stages (i.e., rendering geometry before lighting, then composing the two). Inspired by recent work [58, 61, 98], we use an image synthesis network that takes the source image and pre-computed buffers of lighting-relevant data generated by the rendering engine to produce the final relit image. We also provide the network the environment maps for enhanced lighting context and formulate a novel paired-data training scheme by leveraging the digital twins to generate synthetic paired images.
Generate lighting-relevant data with physically-based rendering: To perform neural deferred rendering, we place the static background and dynamic actor textured meshes  $\mathcal{M}$  in a physically-based renderer [7] and pre-compute the rendering buffers  $\mathbf{I}_{\mathrm{buffer}} \in \mathbb{R}^{h \times w \times 8}$ , including position, depth, normal and ambient occlusion for each frame. Additionally, given an environment map  $\mathbf{E}$  and material maps, the physically-based renderer performs ray-tracing to generate the rendered image  $\mathbf{I}_{\mathrm{render|E}}$ . We omit  $\mathbf{E}$  in the following for simplicity. To model shadow removal and insertion, we also generate a shadow ratio map  $\mathbf{S} = \mathbf{I}_{\mathrm{render}} / \widetilde{\mathbf{I}}_{\mathrm{render}}$ , where  $\widetilde{\mathbf{I}}_{\mathrm{render}}$  is the rendered image without rendering shadow visibility rays, for both the source and target environment light maps  $\mathbf{E}^{\mathrm{src}}$ ,  $\mathbf{E}^{\mathrm{tgt}}$ .
We then use a 2D U-Net [63] that takes the source image  $\mathbf{I}^{\mathrm{src}}$ , render buffers  $\mathbf{I}_{\mathrm{buffer}}$ , and shadow ratio maps  $\{\mathbf{S}^{\mathrm{src}}, \mathbf{S}^{\mathrm{tgt}}\}$ , conditioned on the source and target HDR sky domes  $\{\mathbf{E}^{\mathrm{src}}, \mathbf{E}^{\mathrm{tgt}}\}$ . This network outputs the rendered image  $\mathbf{I}^{\mathrm{tgt}}$  under the target lighting conditions as follows:
$$
\mathbf {I} ^ {\mathrm {t g t}} = \operatorname {R e l i t N e t} \left(\left[ \mathbf {I} ^ {\mathrm {s r c}}, \mathbf {I} _ {\text {b u f f e r}}, \mathbf {S} ^ {\mathrm {s r c}}, \mathbf {S} ^ {\mathrm {t g t}} \right], \left[ \mathbf {E} ^ {\mathrm {s r c}}, \mathbf {E} ^ {\mathrm {t g t}} \right]\right). \tag {4}
$$
Figure 4: Qualitative comparison of scene relighting. For the first and third rows, the real images (other PandaSet snippets) under target lighting conditions are provided for better reference.
This enables us to edit the scene, perform scene relighting, and generate a sequence of images under target lighting as the scene evolves to produce simulated camera videos. The simulation is spatially and temporally consistent since our method is physically-based and grounded by 3D digital twins.
Learning: To ensure that our rendering network maintains controllable lighting and is realistic, we train it with a combination of synthetic and real-world data. We take advantage of the fact that our digital twin reconstructions are derived from real-world data, and that our physically-based renderer can generate paired data of different source and target lightings of the same scene. This enables two main data pairs for training the network to learn the relighting task with enhanced realism. For the first data pair, we train our network to map  $\mathbf{I}_{\mathrm{render}|\mathbf{E}^{\mathrm{src}}}\rightarrow \mathbf{I}_{\mathrm{render}|\mathbf{E}^{\mathrm{tgt}}}$ , the physically-based rendered images under the source and target lighting. With the second data pair, we improve realism by training the network to map  $\mathbf{I}_{\mathrm{render}|\mathbf{E}^{\mathrm{src}}}\rightarrow \mathbf{I}_{\mathrm{real}}$ , mapping any relit synthetic scene to its original real world image given its estimated environment map as the target lighting. During training, we also encourage self-consistency by ensuring that, given an input image with identical source and target lighting, the model recovers the original image. The training objective consists of a photometric loss  $(\mathcal{L}_{\mathrm{color}})$ , a perceptual loss  $(\mathcal{L}_{\mathrm{lpips}})$ , and an edge-based content-preserving loss  $(\mathcal{L}_{\mathrm{edge}})$ :
$$
\mathcal {L} _ {\text {r e l i g h t}} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\underbrace {\left\| \mathbf {I} _ {i} ^ {\mathrm {t g t}} - \hat {\mathbf {I}} _ {i} ^ {\mathrm {t g t}} \right\| _ {2}} _ {\mathcal {L} _ {\text {c o l o r}}} + \lambda_ {\text {l p i p s}} \underbrace {\sum_ {j = 1} ^ {M} \left\| V ^ {j} \left(\mathbf {I} _ {i} ^ {\mathrm {t g t}}\right) - V ^ {j} \left(\hat {\mathbf {I}} _ {i} ^ {\mathrm {t g t}}\right) \right\| _ {2}} _ {\mathcal {L} _ {\text {l p i p s}}} + \lambda_ {\text {e d g e}} \underbrace {\left\| \nabla \mathbf {I} _ {i} ^ {\mathrm {t g t}} - \nabla \hat {\mathbf {I}} _ {i} ^ {\mathrm {t g t}} \right\| _ {2}} _ {\mathcal {L} _ {\text {r e g}}}\right), \tag {5}
$$
where  $N$  is the number of training images and  $\mathbf{I}^{\mathrm{tgt}} / \hat{\mathbf{I}}^{\mathrm{tgt}}$  are the observed/synthesized label image and predicted image under the target lighting, respectively.  $V^{j}$  denotes the  $j$ -th layer of a pre-trained VGG network [97], and  $\nabla \mathbf{I}$  is the image gradient approximated by Sobel-Feldman operator [70].
We showcase LightSim's capabilities on public self-driving data, which contains a rich collection of sensor data of dynamic urban scenes. We first introduce our experiment setting, then compare LightSim against state-of-the-art (SoTA) scene-relighting methods and ablate our design choices. We then show that our method can generate realistic driving videos with added actors and modified trajectories under diverse lighting conditions. Finally, we show that using LightSim to augment training data can significantly improve 3D object detection.
Datasets: We evaluate our method primarily on the public real-world driving dataset PandaSet [87], which contains 103 urban scenes captured in San Francisco, each with a duration of 8 seconds
Method	FID ↓	KID (×103) ↓
Self-OSR [94]	124.8	107.1 ± 4.3
NeRF-OSR [64]	143.9	94.0 ± 7.5
Color Transfer [60]	85.4	29.5 ± 4.3
EPE [61]	93.0	56.0 ± 5.0
Ours	87.1	30.4 ± 4.0
Table 1: Perceptual quality evaluation.
Model	mAP (%)
Real	32.1
Real + Color aug. [44]	33.8 (+1.7)
Real + Sim (Self-OSR)	30.3 (-1.8)
Real + Sim (EPE)	32.5 (+0.4)
Real + Sim (Color Transfer)	35.1 (+3.0)
Real + Sim (Ours)	36.6 (+4.5)
Table 2: Data augmentation with simulated lighting variations.
Figure 5: Ablation study on neural deferred rendering. Artifacts highlighted with different colors.
(80 frames, sampled at 10hz) acquired by six cameras (1920 × 1080) and a  $360^{\circ}$  64-beam LiDAR. To showcase generalizability, we also demonstrate our approach on ten dynamic scenes from the nuScenes [11] dataset. These driving datasets are challenging as the urban street scenes are unbounded; large-scale ( $>300\mathrm{m} \times 80\mathrm{m}$ ); have complex geometry, materials, lighting, and occlusion; and are captured in a single drive-by pass (forward camera motion).
Baselines: We compare our model with several SoTA scene-relighting methods. We consider several inverse-rendering approaches [94, 64], an image-based color-transfer approach [60], and a physics-informed image-synthesis approach [61]. Self-OSR [94] is an image-based inverse-rendering approach that uses generative adversarial networks (GANs) to decompose the image into albedo, normal, shadow, and lighting. NeRF-OSR [64] performs physically-based inverse rendering using neural radiance fields. Color Transfer [60] utilizes histogram-based color matching to harmonize color appearance between images. Enhancing Photorealism Enhancement (EPE) [61] enhances the realism of synthetic images using intermediate rendering buffers and GANs. EPE uses the rendered image  $\mathbf{I}_{\text {render }}| \mathbf{E}^{\text {tgt }}$  and G-buffer data generated by our digital twins to predict the relit image.
5.2 Neural Lighting Simulation
Comparison to SoTA: We report scene relighting results on PandaSet in Table 1. Since the ground truth is unavailable, we use FID [25] and KID [6] to measure the realism and diversity of relit images. For each approach, we evaluate on 1,380 images with 23 lighting variations and report FID/KID scores. 11 of the target lighting variations are estimated from real PandaSet data, while the remaining twelve are outdoor HDRs sourced from HDRMaps [24]. See Appendix C.1 for more details.
Compared to Self-OSR, NeRF-OSR and EPE, LightSim achieves better performance on FID, which indicates that our relit images are more realistic and contain fewer visual artifacts when employed as inputs by ML models. We also show qualitative results in Fig. 4 together with source and target lighting HDRs (Row 1 and 3: relighting with estimated lighting conditions of other PandaSet snippets, Row 2 and 4: third-party HDRs). While Color Transfer achieves the best FID, visually we can see that it only adjusts the global color histogram and does not perform physically-accurate directional lighting (e.g., no newly cast shadows). Self-OSR estimates the source and target lighting as spherical harmonics, but since it must reason about 3D geometry and shadows using only a single image, it produces noticeable artifacts. NeRF-OSR has difficulty with conducting reasonable intrinsic decomposition (e.g., geometry, shadows) and thus cannot perform realistic and accurate scene relighting. EPE incorporates the simulated lighting effects from our digital twins and further enhances realism, but there are obvious artifacts due to blurry texture, broken geometry, and unrealistic hallucinations. In contrast, LightSim produces more reliable and higher-fidelity relighting results
Figure 6: Lighting-aware multi-camera simulation. Inserted actors highlighted with boxes.
Figure 7: Lighting-aware camera simulation of novel scenarios.
under diverse lighting conditions. See Appendix E.1 for more results. In Appendix E.4, we also evaluate LightSim's lighting estimation compared to SoTA and demonstrate improved performance.
Downstream perception training: We now investigate if realistic lighting simulation can help improve the performance of downstream perception tasks under unseen lighting conditions. We consider a SoTA camera-based birds-eye-view (BEV) detection model BEVFormer [44]. Specifically, we train on 68 snippets collected in the city and evaluate on 35 snippets in a suburban area, since these two collections are exposed to different lighting conditions. We generate three lighting variations for data augmentation. One lighting condition comes from the estimated sky dome for log-084 (captured along the El Camino Real in California), and the other two are real-world cloudy and sunny HDRs. We omit comparison to NeRF-OSR as its computational cost makes it challenging to render at scale. Table 2 demonstrates that LightSim augmentation yields a significant performance improvement (+4.5 AP) compared to baseline augmentations, which either provide smaller benefits or harm the detection performance.
Ablation Study: Fig. 5 showcases the importance of several key components in training our neural deferred rendering module. Pre-computed rendering buffers help the network predict more accurate lighting effects. The edge-based content-preserving loss results in a higher-fidelity rendering that retains fine-grained details from the source image. Training the network to relight synthetic rendered images to the original real image with its estimated lighting enhances the photorealism of the simulated results. Please see more ablations in Appendix E.2.
Realistic and controllable camera simulation: LightSim recovers more accurate HDR sky domes compared to prior SoTA works, resulting in more realistic actor insertion (Fig. 6). LightSim inserts the new actors seamlessly and can model lighting effects such as cast shadows for the actors and
Figure 8: Scene relighting on nuScenes. The real images are displayed in the leftmost column.
Figure 9: Physically based rendered images  $(\mathbf{I}_{\text {render }}|\mathbf{E}^{\text {src }}$ ,  $\mathbf{I}_{\text {render }}|\mathbf{E}^{\text {tgt}})$  and relighting results.
static scene, all in a 3D-aware manner for consistency across cameras. Our simulation system also performs realistic, temporally-consistent and lighting-aware scene editing to generate immersive experiences for evaluation. In Fig. 7, we start from the original scenario and perform scene editing by removing all dynamic actors and inserting traffic cones, barriers, and three vehicle actors in the crossroads. Then, we apply scene relighting to change the scene illumination to sunny, cloudy, etc. In Fig. 1, we show another example where we modify the existing real-world data to generate a challenging scenario with two cut-in vehicles.
Generalization study on nuScenes: We now showcase LightSim's ability to generalize to driving scenes in nuScenes [11]. We build lighting-aware digital twins for each scene, then apply a neural deferred rendering model pre-trained on PandaSet. LightSim transfers well and performs scene relighting robustly (see Fig. 8). See Appendix E.6 for more examples.
LightSim assumes several simplifications when building lighting-aware digital twins, including approximate diffuse-only reconstruction, separate lighting prediction, and fixed base materials. This results in imperfect intrinsic decomposition and sim-to-real discrepancies (see Fig. 9). One major failure case we notice is that LightSim cannot seamlessly remove shadows, particularly in bright, sunny conditions where the original images exhibit distinct cast shadows (see Fig. A26). This is because the shadows are mostly baked during neural scene reconstruction, thus producing flawed synthetic data that confuses the neural deferred rendering module. We believe those problems can be addressed through better intrinsic decomposition with priors and joint material/lighting learning [83, 64]. Moreover, LightSim cannot handle nighttime local lighting sources such as street lights, traffic lights and vehicle lights. Finally, faster rendering techniques can be incorporated to enhance LightSim's efficiency [69, 56].
In this paper, we aimed to build a lighting-aware camera simulation system to improve robot perception. Towards this goal, we presented LightSim, which builds lighting-aware digital twins from real-world data; modifies them to create new scenes with different actor layouts, SDV viewpoints, and lighting conditions; and performs scene relighting to enable diverse, realistic, and controllable camera simulation that produces spatially- and temporally-consistent videos. We demonstrated LightSim's capabilities to generate new scenarios with camera video and leveraged LightSim to significantly improve object detection performance. We plan to further enhance our simulator by incorporating material model decomposition, local light source estimation, and weather simulation.
We sincerely thank the anonymous reviewers for their insightful suggestions. We would like to thank Andrei Bârsan and Joyce Yang for their feedback on the early draft. We also thank the Waabi team for their valuable assistance and support.
[1] David Acuna, Jonah Philion, and Sanja Fidler. Towards optimal strategies for training self-driving perception models in simulation. NIPS, 2021.
[2] Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, and Daniela Rus. Learning robust control policies for end-to-end autonomous driving from data-driven simulation. IEEE Robotics and Automation Letters, 2020.
[3] Alexander Amini, Tsun-Hsuan Wang, Igor Gilitschenski, Wilko Schwarting, Zhijian Liu, Song Han, Sertac Karaman, and Daniela Rus. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. In ICRA, 2022.
[4] Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, and Chunxia Xiao. Deep image-based illumination harmonization. In CVPR, 2022.
[5] Anand Bhattachad and David A Forsyth. Enriching stylegan with illumination physics. arXiv, 2022.
[6] Mikołaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
[7] Blender Foundation. Blender, 2021.
[8] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially-varying brdf and shape estimation. In CVPR, 2020.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv, 2016.
[10] Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In ACM SIGGRAPH, 2012.
[11] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. CVPR, 2020.
[12] Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Manivasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self-driving. CVPR, 2021.
[13] Yuxiao Chen and Boris Ivanovic. Traffic behavior simulation. https://github.com/NVlabs/traffic-behavior-simulation, 2021.
[14] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CVPR, 2016.
[15] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
[16] Michael Deering, Stephanie Winner, Bic Schediwy, Chris Duffy, and Neil Hunt. The triangle processor and normal vector Shader: a vlsi system for high performance graphics. ACM SIGGRAPH computer graphics, 1988.
[17] Rosen Diankov and James Kuffner. Openrave: A planning architecture for autonomous robotics. Robotics Institute, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-08-34, 2008.
[18] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. Conference on robot learning, 2017.
[19] Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean-François Lalonde. Deep parametric indoor lighting estimation. In ICCV, 2019.
[20] Marc-Andre Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-François Lalonde. Learning to predict indoor illumination from a single image. arXiv, 2017.
[21] Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, 1997.
[22] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. ICML, 2020.
[23] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. NeurIPS, 2022.
[24] HDRMaps. Hdrmaps. https://hdrmaps.com/, Access date: 2023-05-17.
[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
[26] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
[27] Yannick Hold-Geoffroy, Akshaya Athawale, and Jean-François Lalonde. Deep sky modeling for single image outdoor lighting estimation. In CVPR, 2019.
[28] Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean-François Lalonde. Deep outdoor illumination estimation. In CVPR, 2017.
[29] Louis Hugues and Nicolas Bredeche. Simbad: an autonomous robot simulation package for education and research. In International Conference on Simulation of Adaptive Behavior, 2006.
[30] Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 2013.
[31] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7, page 0, 2006.
[32] Ingo Keller and Katrin S Lohan. On the illumination influence for object learning on robot companions. Frontiers in Robotics and AI, 2020.
[33] Pradeep K Khosla and Takeo Kanade. Parameter identification of robot dynamics. In 1985, 24th IEEE conference on decision and control, 1985.
[34] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. CVPR, 2021.
[35] Nathan Koenig and Andrew Howard. Design and use paradigms for gazebo, an open-source multi-robot simulator. In IROS, 2004.
[36] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In CVPR, 2022.
[93] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In ICCV, 2019.
[94] Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and William AP Smith. Self-supervised outdoor scene relighting. In ECCV, 2020.
[95] Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3155-3164, 2019.
[96] Jinsong Zhang, Kalyan Sunkavalli, Yannick Hold-Geoffroy, Sunil Hadap, Jonathan Eisenman, and Jean-François Lalonde. All-weather deep outdoor lighting estimation. In CVPR, 2019.
[97] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018.
[98] Xianling Zhang, Nathan Tseng, Ameerah Syed, Rohan Bhasin, and Nikita Jaipuria. Simbar: Single image-based scene relighting for effective data augmentation for automated driving vision tasks. In CVPR, 2022.
[99] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[100] Yichao Zhou, Jingwei Huang, Xili Dai, Shichen Liu, Linjie Luo, Zhili Chen, and Yi Ma. Holicity: A city-scale data platform for learning holistic 3d structures. arXiv, 2020.
[101] Yongjie Zhu, Yinda Zhang, Si Li, and Boxin Shi. Spatially-varying outdoor lighting estimation from intrinsics. In CVPR, 2021.