Neural Lighting Simulation for Urban Scenes
Authors: Ava Pun
Gary Sun
Jingkang Wang
Yun Chen
Ze Yang
Sivabalan Manivasagam
Wei-Chiu Ma
Raquel Urtasun
Venue: NeurIPS 2023 poster (2023)
Keywords: Scene Relighting
Lighting Estimation
Camera Simulation
Self-Driving
Lighting Simulation
Scene Editing
TL;DR: We introduce a neural lighting simulation approach to simulate realistic videos under different lighting conditions

Abstract: Different outdoor illumination conditions drastically alter the appearance of urban scenes, and they can harm the performance of image-based robot perception systems if not seen during training. Camera simulation provides a cost-effective solution to create a large dataset of images captured under different lighting conditions. Towards this goal, we propose LightSim, a neural lighting camera simulation system that enables diverse, realistic, and controllable data generation. LightSim automatically builds lighting-aware digital twins at scale from collected raw sensor data and decomposes the scene into dynamic actors and static background with accurate geometry, appearance, and estimated scene lighting. These digital twins enable actor insertion, modification, removal, and rendering from a new viewpoint, all in a lighting-aware manner. LightSim then combines physically-based and learnable deferred rendering to perform realistic relighting of modified scenes, such as altering the sun location and modifying the shadows or changing the sun brightness, producing spatially- and temporally-consistent camera videos. Our experiments show that LightSim generates more realistic relighting results  than prior work. Importantly,  training perception models on data generated by LightSim can significantly improve their performance. Our project page is available at https://waabi.ai/lightsim/.

Neural Lighting Simulation for Urban Scenes
Ava Pun 1,31 l.n
Gary Sun 1,3+t Jingkang Wang 1.2+ Yun Chen Ze Yang
Sivabalan Manivasagam 1.2
Wei-Chiu Ma
Raquel Urtasun 1.2
Waabi²
University Toronto²
University of Waterloo
MIT
Abstract
Different outdoor illumination conditions drastically alter the appearance of ur- ban scenes, and they harm the performance of image-based robot perception systems not seen during training. Camera simulation provides a cost-effective solution create large dataset of images captured under different lighting condi- tions. Towards this goal, we propose LightSim, a neural lighting camera simulation system that enables diverse, realistic, and controllable data generation. LightSim automatically builds lighting- aaaeiing--wareeeeee aware digital twins at scale from collected raw sensor data and decomposes the scene into dynamic actors and static background with accurate geometry, appearance, and estimated scene lighting. These digital twins enable actor insertion, modification, removal, and rendering from new viewpoint, all in lighting-awel aware manner. LightSim then combines physically-base learnable deferred rendering to perform realistic relighting of modified scenes, such as altering the sun location and modifying the shadows or changing the sun brightness, producing spatially- and temporally-consisten camera videos. Our experiments show that LightSim generates more realistic relighting results than prior work. Importantly, training perception models on data generated by Light- Sim significantly improve their performance. Our project page is available at https: / wwwbi..aaaaa/aaaa//.aa//aaaa..aa ai/ ht///ightsimm.
and
Introduction
Humans can perceive their surroundings under different lighting conditions, such identifying traffic participants while driving on a dimly lit road or under mild sun glare. Unfortunately, modern camera- based perception systems, such those in self-driving vehicles (SDVs), not robust [52, 32]. They typically only perform well in the canonical setting they were trained in, and their performance drops significantly under different reduce distribution shift, we could collect data under various lighting conditions for each area in which we want deploy the SDVs, generating diverse dataset on which train the perception system. Unfortunately, this is not scalable as it is too expensive and time-consuming.
scenarios, such low-light conditions [52, 59,32], To
Simulation is cost-effective alternative for generating large-scale data with diverse lighting. To be effective, simulator should be realistic, controllable, and diverse. Realistic simulation of camera data enables generalization the real world. Controllable actor placement and lighting allow the simulator to generate the desired training scenarios. Diverse backgrounds, actor assets, and lighting conditions allow simulation cover the full real-world distribution. While existing game-engine-basee self- driving simulators such as CARLA [18, 68] are controllable, they provide a limited number of manually designed assets and lighting conditions. Perception systems trained this data generalize poorly to the real world [26, 1, 61].
"Equal contributions. Work done while research intern at Waahi.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Scone relighting (Top) and shadow editing (Bottom)
Orginal
Editing
Scane Relighting
Edtting
Original
Relighting
Realistic and controllable camera simulation for safety-critical sconario
Figure 1: LightSim builds digital twins from large-scale data with lighting variations and gener- ates high-fidelity simulation videos. Top: LightSim produces realistic scene relighting and shadow editing videos. Bottom: We generate a safety-critical scenario with two vehicles cutting in and perform lighting-aware camera simulation. See Appendix and project page for more examples.
To build more realistic, scalable, and diverse simulator, we instead propose reconstructing "digital twins" of the real world scale from moving platform equipped with LiDAR and cameras. By reconstructing the geometry and appearance of the actors and static backgrounds to create composable neural assets, as well as estimating the scene lighting, the digital twins can provide a rich asset library for simulation. Existing methods for building digital twins and data-driven simulators [2, 40, 89, 73, 57] bake the lighting into the scene, making simulation under new lighting conditions impossible. In contrast, we create lighting-awan aware digital twins, which enable actor insertion, removal, modification, and rendering from new viewpoints with accurate illumination, shadows, occlusion, and temporal consistency. Moreover, by estimating the scene lighting, we can use the digital twins for relighting.
Given digital twin, we must relight the scene to target novel lighting condition. This is, however, challenging task. Prior image-based synthesis methods [4, perform relighting via 2D style transfer techniques, but they typically lack consistent, and can have artifacts. Inverse neural rendering methods [82, 49, 39, 83] aim decompose the scene into geometry, materials and lighting, which allows for relighting through physically-basee rendering [7, 30]. However, the ill- ill-po nature of lighting estimation and intrinsic decomposition makes it challenging to fully disentangle each aspect accurately, resulting in unrealistic relit images. Both approaches, while successful on synthetic scenes or outdoor landmarks [51] with dense data captured under many lighting conditions, have difficulty performing well on large outdoor scenes.
controllability, are not temporally or spatially
This is primarily due to the scarcity of real-world scenes captured under different lighting conditions. Moreover, most prior works perform relighting static scenes and have not demonstrated realistic relighting for dynamic scenes where both the actors and the camera viewpoint are changing, resulting in inter-object lighting effects that are challenging to simulate.
In this paper, we present LightSim, a novel lighting simulation system for urban driving scenes that generates diverse, controllable, and realistic camera data. To achieve diversity, LightSim reconstructs lighting-aware digital twins from real-world sensor data, creating a large library of assets and lighting environment maps for simulation. LightSim then leverages physically-basee rendering to enable controllable simulation of the dynamic scene, allowing for arbitrary actor placement, SDV location, and novel lighting conditions. To improve realism, further enhance the physically- based renderer

---

with an image- image-based neural deferred renderer to perform relighting, enabling data-driven learning to overcome geometry artifacts and ambiguity in the decomposition due to not having perfect knowledge of the scene and sensor configuration. To overcome the lack of real-world scenes captured under different lighting conditions, we train our neural deferred renderer on a mixture of real and synthetic data generated with physically-basee rendering on our reconstructed digital twins.
We demonstrate the effectiveness of our approach on PandaSet [87]. which contains more than 100 real-world self-driving scenes covering 20 unique kilometers. LightSim significantly outperforms the state of the art, producing high- high-quality photorealistic driving videos under a wide range of lighting conditions. We then showcase several capabilties of LightSim. such as its ability to create high-fidelity and Highting-awars actor insertion, scene relighting, and shadow editing (Fig. 1). We demonstrate that by training camera perception models with LightSim- SighSii--enerated generated data, we can achieve significant performance improvements, making the models more robust to lighting variation. We believe LightSim is an important step towards enhancing the safety of self-driving development and deployment.
2 Related Work
Outdoor lighting estimation: As first step for lighting simulation, lighting estimation aims to recover the 360 HDR light field from observed images for photo-realistic virtual object insertion [20, neural networks to
28, 19, 27, 37, 96, 67, 71, 101, 82, 78, 74, 83). Existing works generally
predict various lighting representations, such as environment maps [71, 101, 78, 82, 74, 83), spherical lobes [8, 43], light probes [37], and sky models [28, 27, 96], from single image. For outdoor scenes, handling the high dynamic range caused by the presence of the sun is a significant challenge. Moreover, due to the scarcity of real-world datasets and the ill- ill-posed nature of lighting estimation [82, 74], it is challenging to precisely predict peak intensity and sun location from a single limited field- of-view low-dynamic range (LDR) image. To mitigate these issues, SOLDNet [101, 74] enhances the diversity of material and lighting conditions with synthetic data and introduces a disentangled global and local lighting latent representation to handle spatially-varying effects. NLFE [82] uses hybrid sky dome/ / light volume and introduces adversarial training to improve realism with differentiable actor insertion. In contrast, our work fully leverages the sensory data available in a self-driving setting (i.e., multi-camera images, LiDAR, and GPS) to accurately recover spatially-varying environment maps.
Inverse rendering with lighting: Another way to obtain lighting representations is through joint geometry, material, and lighting optimization with inverse rendering [67, 42, 84, 55, 79, 23, 49, 91, 64, 39, 41, 92, 83]. However, since optimization is conducted on a single scene and material decomposition is the primary goal, the optimized lighting representations are usually not generalizable and do not work well for relighting [67, 55, 233. Inspired by the success of NeRF in high-fidelity 3D reconstruction, some works use volume rendering to recover material and lighting given image collections under different lighting conditions [64, 49, 391, but they are limited in realism and cannot generalize unseen scenes. Most recently, FEGR [83], independent and concurrent to work, proposes a hybrid framework to recover scene geometry, material and HDR lighting of urban scenes from posed camera images. It demonstrates realistic lighting simulation (actor insertion, shadow editing, and scene relighting) for static scenes. However, due to imperfect geometry and material/lightnng decomposition, relighting in driving scenes introduces several noticeable artifacts, including unrealistic scene color, blurry rendering results that miss high-frequency details, obvious mesh boundaries on trees and buildings, and unnatural sky regions when using other HDR maps. In contrast, LightSim learns on many driving scenes and performs photorealistic enhanced deferred shading to produce more realistic relighting videos for dynamic scenes.
Camera simulation for robotics: There is extensive work on developing simulated environments
for safer and faster robot development [33, 86, 53, 17, 75, 38, 15, 29, 17, 35, 9, 13]. Two major lines of work in sensor simulation for self-driving include graphics-basee based [18, 68] and data-driven simulation [80, 66, 50, 88]. Graphics-bart based simulators like CARLA [18] and AirSim [68] are fast and controllable, but they face limitations in scaling and diversity due to costly manual efforts in asset building and scenario creation. Moreover, these approaches can generate unrealistic sensor data that have a large domain gap for autonomy [26, 85). Data-driven approaches leverage computer vision techniques and real-world data to build simulators for self driving [34, 3, 2, 81, 40]. Unfortunately, existing works tend to fall short of realism, struggle with visual artifacts and domain gap [34, 3, 2, 81,
3

---

Digial Twis
Engine
Recon.
Physisaly-hased randering
Lighting Estimation
Neural Deferred Rondering
Seneor data
Dig tal Twins (Geometry, Material, Lighting)
Source
Target Lightng
Simulated Images
Figure 2: Overview of LightSim. Given sensor observations of the scene, first perform neural scene reconstruction and lighting estimation to build lighting- aware digital twins (left). Given a target lighting, we then perform both chysically-based and neural deferred rendering simulate realistic driving videos under diverse lighting conditions (right).
40], and lack comprehensive control in synthesizing novel views [12, 82, 79, 90, 72]. Most recent works [57, 36, 77, 89, 46] use neural radiance fields to build digital twins and represent background scenes and agents as MLPs, enabling photorealistic rendering and controllable simulation across a single snippet. However, these works bake lighting and shadows into the radiance field and therefore cannot conduct actor insertion under various lighting conditions or scene-level lighting simulation. In contrast, LightSim builds lighting- digital twins for more controllable camera simulation.
3 Building Lighting-Aware Digital Twins of the Real World
The goal of this paper is to create a diverse, controllable, and realistic simulator that can generate camera data of scenes at scale under diverse lighting conditions. Towards this goal. LightSim first reconstructs lighting-aware digital twins from camera and LiDAR data collected by moving platform. The digital twin comprises the geometry and appearance of the static background and dynamic actors obtained through neural rendering (See. 3.1), as well as the estimated scene lighting (Sec. 3.2). We carefully build this representation to allow full controllability of the scene, including modifying actor placement SDV position, adding and removing actors in lighting-ave aware manner for accurate shadows and occlusion, and modifying lighting conditions, such as changing the sun's location or intensity. In Sec. 3.3, we then describe how we perform learning sensor data to build the digital twin and estimate lighting. In Sec. 4, we describe how we perform realistic scene relighting with the digital twin generate the final temporally consistent video.
3.1 Neural Scene Reconstruction
Inspired by [89, 57). we learn the scene geometry and base texture via neural fields. We design our neural field F > (s, kz) map 3D location to signed distance IK. and view-independent
diffuse color Kg E R33 We decompose the driving scene into a static background B and a set of dynamic actors {-A, J.M. and map multi-resolution spatial feature grids [54] using two MLP networks: one for the static scene and one for the dynamic actors. This compositional representation allows for 3D-aware actor insertion, removal, or manipulation within the background. From our learned neural field, we use marching cubes [47] and quadric mesh decimation [21] to extract simplified textured meshes M for the scene. For simplicity, we specify base materials [10] for all the assets and defer material learning to future work. Please see Appendix A.1 for details. Given the desired lighting conditions, we can render our reconstructed scene a physically- based renderer to model object-light interactions.
3.2 Neural Lighting Estimation
In addition extracting geometry and appearance, we estimate the scene lighting (Fig. 3, left). We use feth--dnamic--ange (HDR) panoramic sky dome E to represent the light from the sun and the sky. This representation well models the major light sources of outdoor daytime scenes and is compatible with rendering engines [7, 30). Unfortunately, estimating the HDR sky dome from sensor data challenging, as most cameras on SDVs have limited field-of-view (FoV) and not capture the full sky. Additionally, camera data are typically stored with low dynamic range (LDR) in self-driving datasets, i.e., intensities are represented with 8-bits. To overcome these challenges, we first leverage multi-camera data and our extracted geometry to estimate an incomplete panorama LDR image that

---

D. Sap
L
E
Sours ducov
pantl pancerta 1...
1:n pansrama L
FER dayr
5
Reri 1
Figure 3: LightSim modules. Left: neural lighting estimation. Right: neural deferred rendering.
captures scene context and available sky observations. We then apply an inpainting network fill in missing sky regions. Then, we utilize sky dome estimator network that lifts the LDR panorama image to an HDR sky dome and fuses it with GPS data obtain accurate sun direction and intensity, Unlike prior works that estimate scene lighting from a single limited-Fo V LDR image [82, 94, 74], our work leverages multi-sensor data for more accurate estimation. We now describe these steps (as shown in Fig. 3, left) in detail.
Panorama reconstruction: Given K images {I captured by multiple cameras triggered close in time and their corresponding camera poses P [P J we first render the corresponding depth maps D = [D from extracted geometry M: D; = (MM, where & is the depth rendering function and P, E R3x4 is the camera projection matrix (a composition of camera intrinsics and extrinsics). We set the depth values for the sky region to infinity. For each camera pixel (u/, e'), the rendered depth and projection matrix estimate 3D world coordinates, then apply an cquirectangular projection to determine its intensity contribution to panorama pixel (u, v), resulting in Iann:
pano O (I, F (.
where @ is the pixel-wise transformation that maps the RGB of limited field-of-view (FoV) images I at coordinate (u', r') to the (u, v) pixel of the panorama. For areas with overlap, we average all source pixels that projected to the same panorama (u, v). In the self-driving domain, the stitched panorama Ipano usually covers a 360 horizontal VoV, but the vertical FoV limited and cannot fully cover the sky region. Therefore, leverage an inpainting network [93] to complete Ipano- creating full-coverage (360 x 180%) panorama image.
Generating HDR sky domes: For realistic rendering, an HDR sky dome should have accurate sun placement and intensity, as well as sky appearance. Following [96, 82], we learn an encoder-decoder sky dome estimator network that lifts the incomplete LDR panorama to HDR, while also leveraging GPS and time of day for more accurate sun direction. The encoder first maps the LDR panorama image low-dimensional representation to capture the key attributes of the sky dome, including a sky appearance latent Zaky E R2, peak sun intensity fine. and sun direction faie. By explicitly encoding intensity and direction, we enable more numman-nneerpreabbee control of the lighting conditions and more accurate lighting estimation. The decoder network processes this representation and outputs the HDR sky dome E as follows:
E = HDRdecoder (Z.ky: futt faie)). where Z,ky, fint far = LDRencoder(L),
(2)
When GPS and time of day are available, we replace the encoder estimated direction with the GPS-derived sun direction for more precise placement. Please see Appendix A.1 for details.
3.3 Learning
We now describe the learning process extract static scenes and dynamic actor textured meshes, as well as training the inpainting network and sky dome estimator.
Optimizing neural urban scenes: We jointly optimize feature grids and MLP headers (Sa, Jka
to reconstruct the observed sensor data via volume rendering. This includes a photometric loss on the rendered image, a depth loss on the rendered LiDAR point cloud, and regularizer, as follows: Lecene = Ligh + Llidar + Aveg Eyeg. Specifically, we have
Ligb
Σ reRing
|((1)) -((r)
Liidar
1
Σ Ridda
|[(r)) D(r) (3)
Ringg
R
5

---

Here, R represents the set of camera or LiDAR rays. C(r) is the observed color for ray and ((r) is the predicted color. (r)) is the observed depth for ray r, and D(r) is the predicted depth in the range view. To encourage smooth geometry, we also regularize the SDF to satisfy the Eikonal equation and have free space away from the LiDAR observations [89].
Training panorama inpainting: We train panorama inpainting network to fill the unobserved regions for stitched panorama I pano- We adopt the DeepFill-v2 [93] network and train on the Holic- ity [100] dataset, which contains 6k panorama images. During training, we first generate camera visibility mask using limited-Fo' camera intrinsics to generate an incomplete panorama image. The masked panorama is then fed into the network and supervised with the full panorama. Following [93], we use the hinge GAN loss as the objective function for the generator and discriminator.
Training sky dome estimator: We train sky dome estimator network collected HDR sky images from HDRMaps [24]. The HDRs are randomly distorted (including random exposure scaling, horizontal rotation, and flipping) and then tone-mapped Following [82], we apply teacher forcing randomly and employ the L angular loss, L1 peak intensity, and Lg HDR reconstruction loss in the log space during training.
form LDR-HDR pairs (L, E) pairs.
4 Neural Lighting Simulation of Dynamic Urban Scenes
As is, our lighting-aware digital twin reconstructs the original scenario. Our goal now is to enable controllable camera simulation. To be controllable, the scene representation should not only replicate the original scene but also handle changes in dynamic actor behavior and allow for insertion of synthetic rare objects, such as construction cones, that are challenging find in real data alone. This enables diverse creation of unseen scenes. As our representation is compositional, we can add and remove actors, modify the locations and trajectories of existing actors, change the SDV position. and perform neural rendering on the modified scene to generate new camera video in a spatially- and consisely manner. Using our estimated lighting. we can also use a physically- -based renderer to seamlessly composite synthetic assets, such as CAD models [76], into the scene in 3D- and lighting- aware manner. These scene edits result in an "augmeeted reality" representation M', E and source image I We now describe how we perform realistic scene relighting (Fig. 3 right) to generate new relit videos for improving camera-bant based perception systems.
Given the augmented reality representation [M', Exc, J. we can perform physically-ban -based ren- dering under a novel lighting condition E*st to generate relit rendered video. The rendered images faithfully capture scene relighting effects, such as changes in shadows or overall scene illumination. However, due to imperfect geometry and noise in material/fighting dccomposition, the rendering results lack realism (e.g., they may contain blurriness, unrealistic surface reflections and boundary artifacts). To mitigate this, we propose a photo-realism enhanced neural deferred rendering paradigm. Deferred rendering [16, 65] splits the rendering process into multiple stages (i.e., rendering geometry before lighting, then composing the two). Inspired by recent work [58, 61, 98], we use an image synthesis network that takes the source image and pre-computed buffers of lighting-relevant data generated by the rendering engine to produce the final relit image. We also provide the network the environment maps for enhanced lighting context and formulate novel paired-data training scheme by leveraging the digital twins to generate synthetic paired images.
Generate lighting-relevan data with physically-based rendering: To perform neural deferred rendering, we place the static background and dynamic actor textured meshes M in a physically- based renderer [7] and pre-compute the rendering buffers butter E IR hgx including position, depth, normal and ambient occlusion for each frame. Additionally, given an environment map E and material maps. the physically- bassed renderer performs ray- ray-tracing to generate the rendered image Irender/E- We omit E in the following for simplicity. To model shadow removal and insertion, we
also generate shadow ratio map S = Ieender/T,eaaa where Trender is the rendered image without rendering shadow visibility rays, for both the source and target environment light maps Eare Elst,
We then use a 2D U-Net [63] that takes the source image Ist render buffers buffers and shadow ratio maps (S#Fc St&t], conditioned on the source and target HDR sky domes (E*t, E'gt), J. This network outputs the rendered image g under the target lighting conditions as follows:
tg RelitNet (frare, Tbuffer, Saro Stgt
E aae E Etgt))
(4)

---

Source image Target
Neft
Color
Figure 4: Qualitative comparison of scene relighting, For the first and third rows, the real images (other PandaSet snippets) under target lighting conditions are provided for better reference.
This enables us to edit the scene, perform scene relighting, and generate a sequence of images under target lighting the evolves produce simulated camera videos. The simulation is spatially and temporally consistent since our method is physically-be bassed and grounded by 3D digital twins.
Learning: To ensure that our rendering network maintains controllable lighting and is realistic, we train it with a combination of synthetic and real-world data. We take advantage of the fact that our digital twin reconstructions are derived from real-world data, and that our physically-base renderer can generate paired data of different source and target lightings the same scene. This enables two main data pairs for training the network to learn the relighting task with enhanced realism. For the first data pair, we train our network to map Lender] [rre + [ender the physically- -basedd rendered images under the source and target lighting. With the second data pair, we improve realism by training the network to map Lender Ber - Teall mapping any relit synthetic scene to its original real world image given its estimated environment map the target lighting. During training, self-consistency by ensuring that, given an input image with identical source and target lighting, the model recovers the original image. The training objective consists of photometric loss (Loolor), perceptual loss (Lipips), and an edge-based content-preservivg loss (Gedge):
also encourage
Crolight
1 N
Σ
I
ggt
Alpips
Σ
v'
V (i:") ||,
+Amlee
VI
vi;a0 vicu!l))
ll.) (5)
Ecolor
Creg
Cipips
where N the number of training images and / are the observed/synthesized label image and predicted image under the target lighting, respectively. V denotes the j-th layer of pre-trained VGG network [97], and VI is the image gradient approximated by Sobel-Feldman operator [70].
Experiments
We showease LightSim's capabilities on public self-driving data, which contains a rich collection of sensor data of dynamic urban scenes. We first introduce experiment setting, then compare (SoTA) scene-relighting methods and ablate our design choices.
LightSim against state-of-the We then show that our method can generate realistic driving videos with added actors and modified trajectories under diverse lighting conditions. Finally, we show that using LightSim to augment training data can significantly improve 3D object detection.
5.1 Experimental Setup
Datasets: We evaluate our method primarily on the public real-world driving dataset PandaSet [87], which contains 103 urban scenes captured in San Francisco, each with duration of 8 seconds
7

---

Model
mAP (9)
Method
FID 4
KID (x10))
Real
32.1
Real + Color aug. [44]
33.8 (+-1.7)
Self-OSR [94]
124.8
107.1
4.3
NeRF-OSR [64]
143.9
94.0
Real + Sim (SelF-OSR)
30.3 (-18)
Color Transfer [60]
29.5 -
Real + Sim (EPE)
32.5 (+0.4)
EPE [6]]
Real + Sim (Color Transfer)
35.1 (+3.0)
Ours
87.1
30.4
Real + Sim (Ours)
36.6 (+4.55)
Table I: Perceptual quality evaluation.
Table 2: Data augmentation with simulated lighting variations.
Figure 5: Ablation study on neural deferred rendering. Artifacts highlighted with different colors.
(80 frames, sampled at 10hz) acquired by six cameras (1920 x 1080) and 360 64-beam LiDAR. To showcase generalizability, we also demonstrate our approach on ten dynamic scenes from the nuScenes [11] dataset. These driving datasets are challenging as the urban street scenes are un- bounded; large-scale (> 300 m x 80 m); have complex geometry, materials, lighting, and occlusion; and are captured single drive-by pass (forward camera motion).
Baselines: We compare our model with several SoTA seene-relighting methods. We consider several inverse-rendering approaches [94, 64), an image-based color-transfer approach [60], and physics-informed image-synthesis approach [61]. Self-OSR [94] is an image-based inverse-rendering approach that uses generative adversarial networks (GANs) decompose the image into albedo, normal, shadow, and lighting. NeRF-OSR [64] performs physically-based inverse rendering using neural radiance fields. Color Transfer [60] utilizes histogram-based color matching harmonize color appearance between images. Enhancing Photorealism Enhancement (EPE) [61] enhances the realism of synthetic images using intermediate rendering buffers and GANs. EPE uses the rendered image Fender Etat and G-buffer data generated by our digital twins to predict the relit image.
5.2 Neural Lighting Simulation
We report scene relighting results on PandaSet in Table 1. Since the ground truth is unavailable, we use FID [25] and KID [6] to measure the realism and diversity of relit images. For each approach, we evaluate on 1,380 images with 23 lighting variations and report FID/KID scores. 11 of the target lighting variations are estimated from real PandaSet data, while the remaining twelve are outdoor HDRs sourced from HDRMaps [24]. See Appendix C.1 for more details.
Comparison SoTA:
Compared to Self-OSR, NeRF-OSR and EPE, LightSim achieves better performance on FID, which indicates that our relit images are as inputs by MI models. We also show qualitative results in Fig. 4 together with source and target lighting HDRs (Row 1 and 3: relighting with estimated lighting conditions of other PandaSet snippets, Row 2 and 4: third-party HDRs). While Color Transfer achieves the best FID, visually we can see that it only adjusts the global color histogram and does not perform physically-accurre directional lighting (e.g., no newly cast shadows). Self-OSR estimates the source and target lighting as spherical harmonics, but since it must reason about 3D geometry and shadows using only single image, it produces noticeable artifacts. NeRF-OSR has difficulty with conducting reasonable intrinsic decomposition (e.g., geometry, shadows) and thus cannot perform realistic and accurate scene relighting. EPE incorporates the simulated lighting effects from our digital twins and further enhances realism, but there are obvious artifacts due to blurry texture, broken geometry, and unrealistic hallucinations. In contrast, LightSim produces more reliable and higher- fidelity relighting results
realistic and contain fewer visual artifacts when employed

---

Figure 6: Lighting multi-camera simulation. Inserted actors highlighted with boxes.
Orginal
Relighing Edilnng
Scare Relighting Shadow
Scena Rellghting Intensity Editing
Figure Lighting-aware camera simulation of novel scenarios.
under diverse lighting conditions. See Appendix E. for more results. In Appendix E.4, we also evaluate LightSim's 's lighting estimation compared to SoTA and demonstrate improved performance.
Downstream perception training: We now investigate if realistic lighting simulation can help improve the performance of downstream perception tasks under unseen lighting conditions. We consider SoTA camera-based birds-eye-view (BEV) detection model BEVFormer [44]. Specifically, we train on 68 snippets collected in the city and evaluate on 35 snippets in suburban area, since these two collections are exposed to different lighting conditions. We generate three lighting variations for data augmentation. One lighting condition comes from the estimated sky dome for Log-084 (captured along the El Camino Real in California), and the other are real-world cloudy and sunny HDRs. We omit comparison to NeRF- ORRR-SSR as its computational cost makes it challenging to render at scale. Table 2 demonstrates that LightSim augmentation yields a significant performance improvement (+4.5 AP) compared to baseline augmentations, which either provide smaller benefits or harm the detection performance.
Ablation Study: Fig. 5 showcases the importance of several key components in training our neural deferred rendering module. Pre-computed rendering buffers help the network predict more accurate lighting effects. The edge-based content-preservin loss results in a higher-fidelity rendering that retains Ene-grained details from the source image. Training the network to relight synthetic rendered images to the original real image with its estimated lighting enhances the photorealism of the simulated results. Please see more ablations in Appendix E.2.
Realistic and controllable camera simulation: LightSim recovers more accurate HDR sky domes
compared to prior SoTA works, resulting more realistic actor insertion (Fig. 6). LightSim inserts the new actors seamlessly and can model lighting effects such as cast shadows for the actors and

---

Figure 8: Scene relighting nuScenes. The real images displayed in the leftmost column.
Orginal RCB
Relk Image
Reforence
Figure 9: Physically based rendered images (Trender Earc render E and relighting results.
static scene, all in 3D-aware manner for consistency performs realistic, temporally-consisten and lighting-aware scene editing to generate immersive experiences for evaluation. In Fig. 7, we start from the original scenario and perform scene editing by removing all dynamic actors and inserting traffic cones, barriers, and three vehicle actors in the crossroads, Then, we apply scene relighting to change the scene illumination sunny, cloudy, etc. In Fig. 1, we show another example where we modify the existing real-world data to generate challenging scenario with two cut-in vehicles.
cameras. Our simulation system also
Generalization study on nuScenes: We now showcase LightSim's ability to generalize to driving scenes in nuScenes [11]. We build lighting- digital twins for each scene, then apply neural deferred rendering model pre-trained on PandaSet. LightSim transfers well and performs scene relighting robustly (see Fig. 8). See Appendix E.6 for more examples.
6 Limitations
LightSim assumes several simplifications when building lighting-aware digital twins, including approximate diffuse-only reconstruction, separate lighting prediction, and fixed base materials. This results in imperfect intrinsic decomposition and sim-to-real discrepancies (see Fig. 9). One major failure
notice is that LightSim cannot seamlessly remove shadows, particularly in bright, sunny conditions where the original images exhibit distinct cast shadows (see Fig. A26). This is because the shadows mostly baked during neural scene reconstruction. thus producing flawed synthetic data that confuses the neural deferred rendering module. We believe those problems be addressed through better intrinsic decomposition with priors and joint material/lighting learning [83, 64]. Moreover, LightSim cannot handle nighttime local lighting sources such as street lights, traffic lights and vehicle lights. Finally, faster rendering techniques can be incorporated to enhance LightSim's 's efficiency [69,56].
7 Conclusion
In this paper, we aimed build a lighting-aware camera simulation system to improve robot perception. Towards this goal, we presented LightSim, which builds lighting-aware digital twins from real-world data; modifies them to create new scenes with different actor layouts, SDV viewpoints, and lighting conditions; and performs scene relighting to enable diverse, realistic, and controllable camera simulation that produces spatially- and temporally-consisteen videos. We demonstrated LightSim's 'ss capabilities to generate new scenarios with camera video and leveraged LightSim to significantly improve object detection performance. We plan to further enhance our simulator by incorporating material model decomposition, local light source estimation, and weather simulation.
10

---

Acknowledgement
We sincerely thank the anonymous reviewers for their insightful suggestions. We would like thank Andrei Bársan and Joyce Yang for their feedback on the early draft. We also thank the Waabi team for their valuable assistance and support.
References
[1] David Acuna, Jonah Philion, and Sanja Fidler. Towards optimal strategies for training self- driving perception models in simulation. NIPS, 2021.
[2] Alexander Amini, Igor Gilitschenski, Jacob Phillips, Julia Moseyko, Rohan Banerjee, Sertac Karaman, and Daniela Rus. Learning robust control policies for end-to-end autonomous driving from data-driven simulation. IEEE Robotics and Automation Letters, 2020.
[3] Alexander Amini, Tsun-Hsuan Wang, Igor Gilitschenski, Wilko Schwarting, Zhijian Liu, Song Han, Sertac Karaman, and Daniela Rus. Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. ICRA, 2022.
[4] Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, and Chunxia Xiao. Deep image-based illumination harmonization. In CVPR, 2022.
[5] Anand Bhattad and David Forsyth. Enriching stylegan with illumination physics. arXiv, 2022.
[6] Mikolaj Binkkwski,, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying
MMD GANs. In International Conference on Learning Representations, 2018
[7] Blender Foundation. Blender, 2021.
[8] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially- varying brdf and shape estimation. In CVPR, 2020.
[9] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv, 2016.
[10] Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In ACM SIGGRAPH, 2012.
[11] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. CVPR, 2020.
[12] Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Mani- vasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Realistic video simulation via geometry-awarr composition for -elv-driving. CVPR, 2021.
[13] Yuxiao Chen and Boris Ivanovic. Traffic behavior simulation. http::/ggihub.com//Vaabbrro behavior-simullation, 2021.
[14] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. CVPR, 2016.
[15] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[16] Michael Deering, Stephanie Winner, Bic Schediwy, Chris Duffy, and Neil Hunt. The triangle processor and normal vector shader: a vlsi system for high performance graphics. ACM SIGGRAPH computer graphics, 1988.
[17] Rosen Diankov and James Kuffner. Openrave: planning architecture for autonomous robotics. Robotics Institute, Pittsburgh. PA. Tech. Rep. CMU-RI-TR-08-34 2008.
11

---

[18] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. Conference on robot learning, 2017.
[19] Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean-François Lalonde. Deep parametric indoor lighting estimation. In ICCV, 2019.
[20] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and Jean-Frand cannFrraçoii Lalonde. Learning to predict indoor illumination from a single image. arXiv, 2017.
[21] Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Proceedings of the 24th annual conference Computer graphics and interactive techniques, 1997.
[22] Amos Gropp, Lior Yariy, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. ICML, 2020.
[23] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, Light, and Material Decom- position from Images using Monte Carlo Rendering and Denoising, NeurIPS, 2022.
[24] HDRMaps. Hdrmaps. htpss///hdrmaps.com// Access date: 2023-05-17.
[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 2017.
[26] Judy Hoffman, Eric Tzeng, Tacsung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In ICML, 2018.
[27] Yannick Hold-Geoffroy, Akshaya Athawale, and Jean-François Lalonde. Deep sky modeling for single image outdoor lighting estimation. In CVPR, 2019.
[28] Yannick Hold-Geoffroy, Kalyan Sunkavalli, Sunil Hadap, Emiliano Gambaretto, and Jean- François Lalonde. Deep outdoor illumination estimation. In CVPR, 2017.
[29] Louis Hugues and Nicolas Bredeche. Simbad: an autonomous robot simulation package for education and research. In International Conference on Simulation of Adaptive Behavior, 2006.
[30] Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 2013.
[31] Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7, page O, 2006.
[32] Ingo Keller and Katrin Lohan. On the illumination influence for object learning on robot companions. Frontiers in Robotics and AI, 2020.
[33] Pradeep K Khosla and Takeo Kanade. Parameter identification of robot dynamics. In 1985 24th IEEE conference on decision and control, 1985.
[34] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards controllable high-quality neural simulation. CVPR, 2021.
[35] Nathan Koenig and Andrew Howard. Design and use paradigms for gazebo, an open- open-source multi-robot simulator. In IROS, 2004.
[36] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation. In CVPR, 2022.
12

---

[37] Chloe LeGendre, Wan- Wan-Chun Ma, Graham Fyffe, John Flynn, Laurent Charbonnel, Jay Busch, and Paul Debevec. Deeplight: Learning illumination for unconstrained mobile mixed reality. In CVPR, 2019.
[38] Chengshu Li, Fei Xia, Roberto Martin-Marttn, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv, 2021.
[39] Quewei Li, Jie Guo, Yang Fei, Feichao Li, and Yanwen Guo. Neulighting: Neural lighting for free viewpoint outdoor scene relighting with unconstrained photo collections. In SIGGRAPH Asia 2022 Conference Papers, 2022.
[40] Wei Li, CW Pan, Rong Zhang, JP Ren, YX Ma, Jin Fang, FL. Yan, QC Geng, XY Huang, HJ Gong, et al. Aads: Augmented autonomous driving simulation using data-driven algorithms. Science robotics, 2019.
[41] Zhen Li, Lingli Wang, Mofang Cheng, Cihui Pan, and Jiaqi Yang. Multi-view inverse rendering for large-scale real-world indoor scenes. arXiv, 2022.
[42] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from single image. In CVPR, 2020.
[43] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and apatially-varyinnn reflectance from a single image. TOG, 2018.
[44] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird's sedd's-yye-vvee representation from multi-camera images via spatiotemporal transformers. arXiv, 2022.
[45] Zhi-Hao Lin, Bohan Liu, Yi-Ting Chen, David A. Forsyth, Jia-Bin Huang, Anand Bhattad, and Shenlong Wang. Urbanir: Large-scale urban scene inverse rendering from a single video. CoRR, abs/2306.09334, 2023.
[46] Jeffrey Yunfan Liu, Yun Chen, Ze Yang, Jingkang Wang, Sivabalan Manivasagam, and Raquel Urtasun. Neural scene rasterization for large scene rendering in real time. In The IEEE International Conference on Computer Vision (ICCV), 2023.
[47] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM SIGGRAPH computer graphics, 1987.
[48] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv, 2017.
[49]
Linjie Lyu, Ayush Tewari, Thomas Leimkühler, Marc Habermann, and Christian Theobalt. Neural radiance transfer fields for relightable novel-view synthesis with global illumination. In ECCV, 2022.
[50] Sivabalan Manivasagam, Ioan Andrei Bårsan, Jingkang Wang, Ze Yang, and Raquel Urtasun. Towards zero domain gap: A comprehensive study of realistic lidar simulation for autonomy testing. In ICCV, 2023.
[51] Ricardo Martin Bruulla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Doso- vitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. CVPR, 2021.
[52] Julieta Martinez, Sasha Doubov, Jack Fan, Shenlong Wang, Gellért Máttyus, Raquel Urtasun, et al. Pit30m: A benchmark for global localization in the age of self-driving cars. In IROS, 2020.
[53] Olivier Michel. Cyberbotics Itd. TM, professional mobile robot simulation. International Journal of Advanced Robotic Systems, 2004.
[54] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. 2022.
13

---

[55] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In CVPR, 2022.
[56] Merlin Nimier-Davias Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retar- getable forward and inverse renderer. ACM Transactions on Graphics (TOG), 38(6):1-17, 2019.
[57] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. CVPR, 2021.
[58] Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A Efros, and George Drettakis. Multi-view relighting using a geometry-aware network. ACM Trans. Graph., 2019.
[59] Hazem Rashed, Mohamed Ramzy, Victor Vaquero, Ahmad EI Sallab, Ganesh Sistu, and Senthil Yogamani. Fusemodnet: Real-time camera and lidar based moving object detection for robust low-light autonomous driving, In ICCV, 2019.
[60] Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer graphics and applications, 2001.
[61] Stephan R Richter, Hassan Abu AlHaija, and Vladlen Koltun. Enhancing photorealism enhancement. PAMI, 2021.
[62] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. ECCV, 2016.
[63] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part 18, 2015.
[64] Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. Nerf for outdoor scene relighting. In ECCV, 2022.
[65] Takafumi Saito and Tokiichiro Takahashi. Comprehensible rendering of 3-d shapes. In Proceedings of the 17th annual conference on Computer graphics and interactive techniques, 1990.
[66] Jay Sarva, Jingkang Wang, James Tu, Yuwen Xiong, Sivabalan Manivasagam, and Raquel Urtasun. Adv3d: Generating safety- cafety-critical 3d objects through closed-loop simulation. In 7th Annual Conference on Robot Learning, 2023.
[67] Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In ICCV, 2019.
[68] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics, 2018.
[69] Dave Shreiner, Bill The Khronos OpenGL ARB Working Group, al. OpenGL programming guide: the official guide to learning OpenGL, versions 3.0 and 3.1. 2009.
[70] Irwin Sobel and Gary Feldman. A 3x3 isotropic gradient operator for image processing. a talk at the Stanford Artificial Project, 1968.
[71] Gowri Somanath and Daniel Kurz. Hdr environment map estimation for real-time augmented reality. In CVPR, 2021.
[72] Abhijeet Tallavajhula, Çetin Meriçli, and Alonzo Kelly. Off-road lidar simulation with data- driven terrain primitives. In ICRA, 2018.
[73] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
14

---

[74] Jiajun Tang, Yongjie Zhu, Haoyu Wang. Jun Hoong Chan, Si Li, and Boxin Shi. Estimating spatial lighting in urban scenes with disentangled representation. In ECCV, 2022.
[75] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[76] TurboSquid. https:/www..tuubosppi..commm Access date: 2023-05-17.
[77] Haithem Turki, Jason Zhang. Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. CVPR, 2023.
[78] Guangcong Wang, Yinuo Yang. Chen Change Loy, and Ziwei Liu. Stylelight: Hdr panorama generation for lighting estimation and editing. In ECCV, 2022.
[79] Jingkang Wang, Sivabalan Manivasagam, Yun Chen, Ze Yang. loan Andrei Bårsan, Anqi Joyce Yang, Wei-Chiu Ma, and Raquel Urtasun. Cadsim: Robust and scalable in-the-wild 3d reconstruction for controllable sensor simulation. In 6th Annual Conference on Robot Learning. 2022.
[80] Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self- driving vehicles. In CVPR, 2021.
[81] Tsun-Hsuan Wang, Alexander Amini, Wilko Schwarting, Igor Gilitschenski, Sertac Karaman, and Daniela Rus. Learning interactive driving policies via data-driven simulation. In ICRA, 2022.
[82] Zian Wang. Wenzheng Chen, David Acuna, Jan Kautz, and Sanja Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In ECCV, 2022.
[83] Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeie, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In CVPR, 2023.
[84] Xin Wei, Guojun Chen, Yue Dong, Stephen Lin, and Xin Tong. Object-based illumination estimation with rendering-awand aware neural networks. ECCV, 2020.
[85] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud. In ICRA, 2019.
[86] Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner. Tores, the open racing car simulator. http://torrs.ssuurcfforge.e. 2000.
[87] Pengehuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun, Kun Jiang, et al. Pandaset: Advanced sensor suite dataset for autonomous driving. /TSC, 2021.
[88] Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. Learning compact repre- sentations for lidar completion and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1074-1083, 2023.
[89] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In CVPR, 2023.
[90] Ze Yang, Sivabalan Manivasagam, Yun Chen, Jingkang Wang, Rui Hu, and Raquel Urtasun. Reconstructing objects in-the-wild for realistic sensor simulation. In ICRA, 2023.
[91] Weicai Ye. Shuo Chen, Chong Bao, Hujun Bao, Mare Pollefeys, Zhaopeng Cui, and Guofeng Zhang. Intrinsicnerf: Learning intrinsic neural radiance fields for editable novel view synthesis. arXiv, 2022.
[92] Hong-Xing Yu, Samir Agarwala, Charles Herrmann, Richard Szeliski, Noah Snavely, Jiajun Wu, and Deqing Sun. Accidental light probes. arXiv, 2023.
15

---

[93] Jiahui Yu, Zhe Lin. Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In ICCV, 2019.
[94] Ye Yu, Abhimitra Meka, Mohamed Elgharib, Hans-Peter Seidel, Christian Theobalt, and William AP Smith. Self-supervised outdoor scene relighting. In ECCV, 2020.
[95] Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3155-3164, 2019.
[96] Jinsong Zhang, Kalyan Sunkavalli, Yannick Hold-Geoffroy, Sunil Hadap, Jonathan Eisenman. and Jean-François Lalonde. All-weather deep outdoor lighting estimation. In CVPR, 2019.
[97] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. CVPR, 2018.
[98] Xianling Zhang, Nathan Tseng, Ameerah Syed, Rohan Bhasin, and Nikita Jaipuria. Simbar: Single image- based scene relighting for effective data augmentation for automated driving vision tasks. In CVPR, 2022.
[99] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881 2881-2890, 2017.
[100] Yichao Zhou, Jingwei Huang, Xili Dai, Shichen Liu, Linjie Luo, Zhili Chen, and Yi Ma. Holicity: cctye data platform for learning holistic 3d structures. arXiv, 2020.
[101] Yongjie Zhu, Yinda Zhang. Li, and Boxin Shi, Spatially-varying outdoor lighting estimation from intrinsics. In CVPR, 2021.
16

---

Appendix
This appendix details our method, implementation, experimental designs, additional quantitative and qualitative results, limitations, utilized resources, and broader implications. We first detail how we build the lighting-aware digital twins from real-world data (Sec. A.I), then show the network architecture and training details for neural lighting simulation (Sec. A.2). In Sec. B, provide details on baseline implementations and how we adapt them to our scene- scene-relighting setting. Next, provide the experimental details for perceptual quality evaluation and downstream detection training in Sec. C. We then report perception quality evaluation at a larger scale with more qualitative comparison with baselines (Sec. F.1) and detailed detection metrics for detection training (Sec. E.3). We demonstrate that our lighting estimation module yields more accurate sun prediction (Sec. E.4) and show additional scene relighting, shadow editing and controllable camera simulation results (Sec. E.S). Finally, we discuss limitations and future work (Sec. F), licenses of assets (Sec. H), computational resources used in this work (Sec. G), and broader impact (Sec. I).
A LightSim Implementation Details
A.1 Building Lighting-Aware Digital Twins of the Real-World
Neural Scene Reconstruction: We first perform neural rendering to reconstruct the driving scene using both front-facing camera images and 360 spinning LiDAR point clouds. We use modified version of UniSim [89], a neural sensor simulator, to perform neural rendering to learn the asset representations. Unisim [89] incorporates the LiDAR information to perform efficient ray-marehing and adopts a signed distance function (SDF) representation to accurately model the scene geometry. To ensure a smooth zero level set, the SDF representation is regularized using Eikonal loss [22] and occupancy loss. To capture the view ineeeindeeendent base color ka. we make adjustments to the appearance MLP by taking only grid features as input and omitting the view direction term; all other configurations remain consistent with [89].
After learning the neural representation, we employ Marching Cubes [47] to extract the mesh from the learned SDF volume. Finally, we obtain the vertex base color ka by querying the appearance head the vertex location. We adopt base Blender PBR materials [10] for simplicity. Specifically, we use a Principled BSDF with vertex color as the base color. We set materials for dynamic assets and background separately.
Dowe
sun-pos-enc
Peak Direction
Peak Intersily
Sky
sun-int-enc
Sky Dome
sky-enc
Sky Labent
Sky decoder
Encocing
Sky Encoders
Figure A10: Network architecture of the neural lighting estimation module. Red and blue boxes denote the sky encoders and HDR sky dome decoder respectively.
Neural Lighting Estimation: We leverage multi-camers data and our extracted geometry to esti- mate an incomplete panorama LDR image that captures scene context and the available observations of the sky. We then apply an inpainting network to fill in missing sky regions. Finally, we leverage sky dome estimator network that lifts the LDR panorama image to an HDR sky dome and fuses with GPS data to obtain accurate sun direction and intensity. To inpaint the panorama image from the stitched multi-camera image, we use the code of the inpainting network DeepFilly2. The network
patttp://giiee http:pp//gggaa....oo/hooooo///// http://///maaaaa///e/tlllgg
17

---

unchanged from its implementation on Github, and it is trained using the hinge loss. Holicity's LDR panoramas are used for training. Each panorama in the training set is first masked using an intrinsics mask - random mask of observed pixels from a PandaSet log. Distortions are then applied to the mask, including random scaling and the addition of noise. The masked panorama is then fed into the network and supervised with the unedited Holicity panorama.
Sky dome estimator architecture: After receiving the full LDR output from the inpainting net- work, the sky dome estimator network converts the incomplete L,DR panorama to HDR. This is achieved by employing GPS and time of day to improve the accuracy of sun direction estimation. To produce an LDR sky dome, only the top half of the panorama is used, as the bottom half does not contribute to global lighting from the sky. The network uses encoder-decoder architecture, as shown in Fig. A10. The input consists of the LR panorama and the positional encoding map, which is an equirectangular projection that stores a unit vector pointing towards each pixel direction. The positional encoding is concatenated channel-win with the LDR panorama. Three separate encoders with the same architecture utilized to estimate the peak direction fair and intensity of the sun fint. as well as the latent of the sky Zakyy E R61, To encode the predicted sun peak intensity fint and peak direction fdir, five intermediate representations with the same size as the input LDR are explicitly encoded. The sky decoder is an U-Net which takes the encoded predicted sun peak intensity fint and peak direction fdir and fuses them with the sky latent vector to produce the HDR sky dome.
Training details: We train the sky dome estimator using 400 HDRs sourced from HDRMMpps [24].
Accurately predicting the peak intensity fint and direction fair is of utmost importance for achieving realistic rendering. Nevertheless, due to the inherently ill-posed nature of this problem, is chal- lenging to predict these parameters precisely, especially for cloudy skies. Therefore, we propose a dual-encoder architecture, with one encoder trained on HDRs with a clearly visible sun and another trained on all HDR images to capture peak intensity and sky latent more robustly. We empirically find that this works better than using single encoder trained on all HDR images. The model takes around 12 hours to train with single RTX A5000.
A.2 Neural Lighting Simulation of Dynamic Urban Scenes
Building augmented reality representation: Given our compositional neural radiance fields, we can remove actors (Fig. 7), add actors (Fig. A23), modify actor trajectories (Fig. 1 and Fig. A23), and perform neural rendering on the modified scene to generate new camera video in a spatially- manner. Using our estimated lighting representation, we can also insert synthetic assets such as CAD models [76] into the scenes and composite them with real images in a 3D- and lighting-awerr righting-aware manner (Fig. 1, 7, A23). These scene edits lead to an "augmented reality representation" M'. E and source image I We further explain the details of creating augmented reality representations for all the examples we have created in this paper. In Fig. 1, we insert a new CAD vehicle, barriers, and traffic cones and modify the trajectory of the white car. In Fig. 7, we insert three new CAD vehicles (black, white, gray) and construction items and remove all existing dynamic vehicles. In Fig. A23 (top example), we modify the trajectory of the original white vehicle and insert construction items. In Fig. A23 (bottom example), we remove all existing dynamic actors (pedestrians and vehicles), insert three dynamic actors from another sequence 016 with new trajectories, and insert construction items.
and cemporally-consistee
Neural deferred rendering architecture: The neural deferred rendering architecture is depicted in Fig. A11. It image encoder, lighting encoder, latent fitser, and rendering decoder. The inputs to the image encoder comprise a source RGB image Lare. rendering buffers Lbuffe (ambient occlusion, normal, depth, and position), and source/target shadow ratio maps (Sare, Sig: J: each of these components has 3 channels. In the latent fuser, the output of the image encoder is run through a 2D convolution layer, then linear layer that compresses the input into a latent vector. The image latent and two lighting latent vectors (source and target) are concatenated and upsampled. Finally, the rendering decoder upsamples the fused latent vectors and produces the final relit image i E Hx
an image-to-image network adapted from U-Net [63] that consists of four components:
Learning details: We train the relighting network using a weighted sum of the photometric loss (Coolor), perceptual loss (Cipips), and edge-bases based content preserving loss (Cedge), as described in
18

---

Latent Fuser
Donv
Uhear
Image Encoder
Decoder
MaxPool
Conv
Unear
Linear
Up
Concat
Conv
Comv
Linor
Lifting Noll
Source shadow g
Conv20
Down
Down
Down
Down
Up
ComvdD
Lighting Encoder
Render bulfera
Target shadoww 8
ger
Down
Down
Down
Figure A11: Network architecture of the neural deferred rendering module. Green, red, orange, and blue boxes denote the image encoder, lighting encoder, latent fuser and rendering decoder, respectively.
section 4 of the main paper: Lrelight N Ex= (Coolor + AppipaCipips + Aedge Codge) where Appips set and Aedge to 400. Our model is trained pairs of PandaSet [87] scenes, lit with NHDR = 20 HDRs: ten estimated from PandaSet scenes using our neural lighting estimation module, and ten obtained from the HDRMaps dataset [24]. We apply random intensity and offset changes to the HDRs as data augmentation.
B Implementation Details for Baselines
For each approach, we evaluate on 1,380 images, applying 23 lighting variations to 15 PandaSet scenes with 4 frames per scene, and report FID/KID scores. To test the diversity and controllability of each method, Il of the target lighting variations are estimated from real PandaSet data, while the remaining 12 outdoor HDRs sourced from HDRMaps [24].
B.1 Self-supervised Outdoor Scene Relighting (Self-OSR)
Self-OSR [94] is an image-based inverse-rendering method that utilizes InverseRenderNet [95] to decompose an image into albedo, normal, shadow, and lighting. Subsequently, it employs two generative adversarial networks (GANs) for neural rendering and sky hallucination, based on the predicted lighting. We the official pre-trained models² and perform inference under novel lighting conditions. If the target lighting originates from PandaSet logs, the model is first applied to single front-camera image (first frame) recover the lighting parameters 9 using an order-2 spherical harmonics model. For outdoor HDRs sourced from HDRMaps, the HDR environment maps are converted into spherical harmonics. The sky masks for PandaSet images are estimated using PSPNet [99].
B.2 NeRF for Outdoor Scene Relighting (NeRF-OSR)
We use the official code from NeRF-OSR and make several modifications to improve its performance on the PandaSet dataset. Since the self-driving scenes in the PandaSet dataset are usually larger than the front-facing scenarios on the NeRF-OSR dataset, more space needs to sampled, which presents challenge. To overcome this, use LiDAR points guide to sample the points
Phttps:/ggithu wwtp:/o0tmma//eeR-OO0000 http:://ggghiiaooooooaa/aaaattt
19

---

for camera rays by aggregating LiDaR points and creating surfel mesh. Then, we sample only the points close the surface =50em for each camera ray. By doing so, we significantly reduce the number of sampled points to eight in the coarse and fine stages. Without these modifications, NeRF-OSR demonstrates slower convergence rates and struggles to learn reasonable geometry for relighting. Since our inference HDRs (from PandaSet or HDRMaps) differ greatly from NeRF-OSR environment maps (indicating inaccurate scene decomposition by the model), directly applying the environment maps to the scene leads to significant artifacts. Therefore, we retrieve the nearest HDR in the NeRF-OSR dataset during inference for better perceptual quality, For training, we follow the original code base for all other settings. Training one log NeRF-OSR usually takes 15 hours 4 T4 GPUs.
B.3 Color Transfer
Color Transfer [60] uses image statistics in """,a",b, space to adjust the color appearance between images. We adopt the public Python implementation" for our experiments. If the target lighting originates from PandaSet logs, we use the single front-camera image as the target image for color transfer. For outdoor HDRs sourced from HDRMaps, we use the rendered synthetic images (using LightSim digital twins) render EEts as the target image, as it produces much better performance compared transferring with the LDR environment map. The latter results in worse performance since the source environment map and target limited-FoV images are in different content spaces.
B.4 Enhancing Photorealism Enhancement (EPE)
EPE [61] was designed to enhance the realism of synthetic images (e.g., GTA-5 [62] to CitySeapes [14]) using intermediate rendering buffers and GANs. We adapt EPE to handle lighting simulation with our established lighting-awar digital twins. Specifically, EPE uses the rendered image Esss and rendering buffers buffer generated by our digital twins to predict the relit image. Note that EPE uses the same training data as LightSim, with 20 HDR variations. It also takes all PandaSet front-camera images (103 logs) as the referenced real data. We adopt the official implementation" and follow the instructions to compute robust label maps, crop the images, match the crops (5 nearest neighbours) and obtain 459k sim-real pairs. We train the EPE model until convergence for 60M iterations on one single RTX A5000 for around six days.
C LightSim Experiment Details
C.1 Perceptual Quality Evaluation
Following [61, 12, 89], we report Fréchet Inception Distance [25] (FID) and Kernel Inception Distance [25] (KID) to measure perceptual quality since ground truth data are not available. Due to NeRF-OSR's 's large computational cost, we select 15 sequences {001, 002, 011, 021, 023, 024, 027, 028, 029, 030, 032, 033, 035, 040, 053) for quantitative evaluation in the main paper. We also provide Table A3 for larger-scale evaluation (NeRF-OSR excluded), in which 47 sequences (all city logs in PandaSet excluding the night logs and 004 where the SDV is stationary) are used for evaluation. The 47 sequences are (001, 002, 003, 004, 005, 006, 008, 011, 012, 013, 014, 015, 016, 017, 018, 019, 020, 021, 023, 024, 027, 028, 029, 030, 032, 033, 034, 035, 037, 038, 039, 040, 041, 042, 043, 044, 045, 046, 047, 048, 050, 051, 052, 053, 054, 055, 056, 139).
For each sequence, we select four frames (6, 12, 18, 24) and simulate 23 lighting variations (see Fig. A12). Note that 23 lighting variations, 20 HDRs (10 estimated HDRs from PandaSet, 10 real HDRs sourced from HDRMaps) are used for data generation to train the LightSim models, while 3 HDRs are unseen and only used during inference. Unless stated otherwise, we use all 8240 real
PandaSet images the reference dataset.
20

---

B
10 training PandaSet HDRs
10 training HDRMaps HDRs
C: 3 unseen inference HDRs
Figure A12: 23 HDR sky domes used for perceptual quality evaluation.
C.2 Downstream Perception Training
To investigate if realistic lighting simulation can improve the performance of downstream perception tasks under unseen lighting conditions, we conduct experiments on PandaSet using the SoTA camera-based 3D vehicle detection model BEVFormer [44]. We use train on 68 snippets collected in the city and evaluate on 35 snippets a suburban area, since these two collections are independent and exposed to different lighting conditions. Specifically, the sequences {080, , 084, 085, 086, 088, 089, 090, 091, 092, 093, 094, 095, 097, 098, 099, 100, 101, 102, 103,
104, 105, 106, 109, 110, 112, 113, 115, 116, 117, 119, 120, 122, 123, 124,
158) are selected for the validation set, and the remaining sequences are used for training. For the experiments, at different IoU thresholds: 0.1, 0.3, and 0.5. The mean average precision (mAP) is calculated as mAP (AP@0.1 AP@0.3 AP@0.5)/3.0.
all 80 frames for training and evaluation. We report the average precision (AP)
As shown Table 2, integration of LightSim synthetic simulation significantly enhances the performance of monocular detection compared to training with other basic augmentation methods. Further exploration on sufficiently utilizing the simulated data, such as actor behavior simulation or actor insertion, is left to future work.
BEVFormer Implementation Details: We use the official repository training and evaluating our model on PandaSet. We focus single-frame monocular vehicle detection using the front camera, disregarding actors outside the camera's field of view. The models are trained within vehicle frames using the FLU convention (x: forward, y: left, up), with the region of interest defined as x E [0, 80 m), y e [-40 m, 40 m], E [-2 m, 6 m]. Given memory constraints, we adopt the sEll architecture' with batch size of per GPU. Models
trained for five epochs using the Adam optimizer [48], coupled with the cosine learning rate schedule." Training each model took approximately six hours on 2x RTX A5000 GPUs. We report the best validation performance across all data augmentation approaches, as models can begin to overfit in the final training stage.
C.3 Generalization nuScenes
To evaluate the generalizability of our model, we train the model on PandaSet [87] and evaluate the pre-trained model nuScenes [1]]. The nuScenes [11] dataset contains 1000 driving scenes collected in Boston and Singapore, each with duration of 20 seconds (+s 40 frames, sampled at
'nttps://pytorch.co http:///ccc/.tlll/////aaaa/trraa lr_scheduler.
CosineAnnealingLR..Rt
21

---

2 Hz) acquired by six cameras (Basler and five long-range RADAR (Continental ARS 408-21) sensors. We curate 10 urban scenes from nuScenes [11] characterized by dense traffic and interesting scenarios.
one spinning LiDAR (Velodyne HDL32E),
We incorporate front-facing camera and spinning LiDAR and the neural scene reconstruction module (Section 3.1) to extract the manipulable digital twins for each scene. Then, we utilize the neural lighting estimation module (Section 3.2) recover the HDR sky domes. This enables to generate new scenarios and produce the rendering buffers huffer (Eqn. 4 in the main paper) for scene relighting.
D Additional Discussions
Challenges of inverse rendering on urban scenes: LightSim assumes several simplifications when building lighting-aware digital twins, including approximate diffuse-only reconstruction, sepa- rate lighting prediction, and fixed base materials. Those result in imperfect intrinsic decomposition and sim/real discrepancies. Recent concurrent works such as FEGR [83] and UrbanIR [45] make steps towards better decomposition, but it is still a challenging open problem to recover perfect decomposition of materials and light sources for large urban scenes. As shown in Fig. A13 (top right), the recovered materials bear little semblance to semantic regions in the original scene. These recent relighting works [61, 64, 83, 45] also have shadows baked into the recovered albedo (Fig. A13 left).
We remark that our novelty lies in leveraging neural deferred rendering to overcome the limitations of generate
purely physically-bain bbased rendering when the decomposition is imperfect. This allows better relighting results than prior works that have imperfect decompositions. It is an exciting future direction incorporate better intrinsic decomposition along with neural deferred rendering for improved relighting.
Prediction-basee vs. optimization- based lighting: We explain design choices in the follow- ing two aspects. (a) We use feed-forwan network for lighting estimation, which more efficient and can benefit from learning on larger dataset. In contrast, the optimization paradigm is more expensive, requiring per-scene optimization, but has the potential to recover more accurate scene light- ing from partial observations. (b) The ill-posed nature of lighting estimation and extreme intensity range make inverse rendering challenging for outdoor scenes [83]. Optimization the environment map requires a differentiable renderer and high-quality geometry/material achieve good results. The existing/concurrent -tate-of-the-art works [64, 83] cannot solve the problem accurately, as shown in Fig. A13 bottom right.
Desopoodd
shadon
FIGR Abedo
FEOR
Sakod streow
UntariR Aloedo
N=RF-OBR (up) Swi-DBR (bolion) Abbedo
Figure A13: Challenges of inverse rendering on urban scenes for existing works.
Temporal consistency for neural deferred shading: While we do not guarantee temporal consis- tency, LightSim can produce temporally-consistenn lighting simulation videos most cases. We believe this temporal consistency comes from temporally- and multi-view-consisee inference (real image, G-buffers), as well as our combination of simulation and real paired relighting data during training. Explicitly enforcing temporal consistency is an interesting direction for future work.
inputs during
We noticed that random shadowed triangles are common in the [ Euee due to non-smooth extracted meshes. This is more obvious for the nuScenes dataset where the LiDAR is sparser (32-beam) and the capture frequency is lower (2Hz); for this dataset, we notice many holes and random shadowed triangles. However, thanks to our image-based neural deferred rendering pass trained with mixed sim-real data, our relighting network takes the original image and modifies the lighting, which removes many of those artifacts in the final relit images. We show two nuScenes examples in Fig, A14.
Random shadowed triangles when relighting due to mesh artifacts:
22

---

Original RGB
Relit Image1 Lege
Relit Image2 Iuge
Rerendered RGB
Normal
Ture
D BDR
Original Image1 1egs
Image2 Aug
Rerendered RGB
Normal
Lee E
Figure A14: Random shadowed triangles are removed after neural deferred rendering.
E
Additional Experiments and Analysis
We provide additional results and analysis for scene relighting, ablation studies, downstream training, and lighting estimation. We then showcase more simulation examples using LightSim.
E.1 Additional Perception Quality Evaluation
Due the large computational cost of NeRF-OSR [64], select 15 PandaSet sequences for perceptual quality evaluation in Table 1. Here, we supplement the evaluation at larger scale (47 sequences in total) in Table A3. LightSim achieves perceptual quality (FID and KID) on par with Color Transfer, while the latter approach only adjusts the color histogram and cannot simulate intricate lighting effects properly. Self-OSI and EPE suffer from noticeable artifacts, resulting in significantly worse perception quality and a larger sim-real domain gap.
Method
FID |
KID (x10%) 4
Self-OSR [94]
97.3
89.7 + 11.5
Color Transfer [60]
50.1
18.0
EPE [61]
79.6
50.0 + 9.1
Ours
52.3
16.0 4.6
Table A3: Additional perceptual quality evaluation on 47 sequences.
23

---

Source Image & Target HDR
Seff-OSR
NeRF NeRF-OSR
Color Transfer
EPE
Ours
Figure A15: Qualitative comparison against SoTA approaches in scene relighting.
We also provide more qualitative comparisons against SoTA scene relighting approaches in Fig. A15. For each scene, we showcase two different lighting conditions with the inset target HDRs in the leftmost column. We show more scene relighting results of LightSim in Fig. A16 and Fig. A17. We also show results in Fig. A A18, where we rotate the HDR skydome and render the shadows at different sun locations, demonstrating controllable outdoor illumination and realistic simulated results. Please refer to the project page for video examples.
24

---

Figure A16: Qualitative examples of scene relighting for LightSim (Part 1).
25

---

OIS
0 LS
2 LSS
OIS
2 S
OSS
2 O
O
LS
ES
P/S
LLS
LiS
2LS
Figure A17: Qualitative examples of scene relighting for LightSim (Part 2).
26

---

Figure A18: Qualitative examples of scene relighting with shadows edited.
27

---

E.2
Additional Ablation Study
We provide more thorough ablation study on the important components of the neural deferred rendering module. The perception quality metrics and additional qualitative examples are provided in Table A4 and Fig. A19.
Data Pairs
Rendering Buffers
Edge loss Ledge
#ID
FID
KID (x109) L
sim-real
identity
buffer
{ }
0
x
60.9
31.1 + 4.0
1
X
62.5
32.7 + 4.0
2
X
50.5
21.4 +
3
X
49.8
23.1 5.0
4
X
109.8
88.7 7.3
5
200
67.1
40.9 + 4.9
6
800
57.3
31.8 + 4.3
Ours
400)
55.4
27.6 + 3.7
Table A4: Ablation studies LightSim components. For clarity, only mark the differences
between our final model and other configurations. Blank components indicate that the setting is identical our final model.
Aede
Aedee 800
Ours D
400)
w/o sim-real pairs
wio identity pairs
Ours
wio rendering buffers
wio shadow maps
Ours
Figure A19: Additional ablation study neural deferred rendering. LightSim can simulate intricate lighting effects (highlights, shadows) while maintaining realism.
28

---

We choose sequence 001 for the quantitative evaluation (FID/KID scores), and all 80 front-camera images are used as the reference dataset. To sufficiently measure rendering quality, we generate 3 x 80 relit images using the three unseen HDRs in Fig. A12. We also generate 72 relit images (first frame) with shadows edited using the first unseen HDR. Then, to test generalization to unknown lighting conditions, we pick the first frame and generate 92 relit images using other real HDR maps. Therefore, for each model configuration, there are 404 simulated images used in total for perceptual quality evaluation. We set add_subeetssizeeee for this experiment.
As shown in Table A4 and Fig. A19, sim-real and identity data pairs provide useful regularization for the neural deferred rendering module by reducing visual artifacts caused by imperfect geometry. Removing those data pairs leads to less realistic simulation results and worse FID/KID scores. On the other hand, the rendering buffers and shadow maps play an important role in realistically simulating intricate lighting effects such as highlights and shadows. We observe unrealistic color and missing cast shadows if pre-computed buffers buffer and shadow maps {Sere, S ] are removed. Note that the KID/FID metries are lower when removing rendering buffers since the reference dataset does not include the real data under new lighting conditions; this cannot be interpreted as better visual quality. Finally, we ablate the content-pir -preeeriinrs loss Ledge and find that a proper loss weight helps the model reduce synthetic mesh-like artifacts (compared to Xedge = 0) while properly simulating new lighting effects (compared to Aedee 800).
E.3 Additional Object Detection Metrics
We report detailed detection metrics for perception training with different data augmentation ap- proaches for better reference. Specifically, we report the average precision (AP) at different IoU thresholds: 0.1, 0.3, and 0.5. As shown in Table A5, using LightSim-simulte simulated data yields the best performance improvements. Color Transfer and standard color augmentation [44] are also effective ways to promote the performance of autonomy model under novel lighting conditions. In contrast, Self-OSR and EPE either harm the detection performance or bring marginal gains due to noticeable visual artifacts that cause sim-real domain gap between training and validation.
Model
mAP (%)
AP@0.1
AP@0.3
AP@0.5
Real
32.1
51.2
29.5
15.7
Real Color aug. [44]
33.8 (+1.7)
53.9
31.0
16.4
Real Sim (Self-OSR)
30.3 (-1.8)
45.6
29.4
16.0
Real + Sim (EPE)
32.5 (+0.4)
50.2
30.6
16.7
Real Sim (Color Transfer)
35.1 (+3.0)
55.3
32.3
17.6
Real + Sim (Ours)
36.6 (+4.5)
57.1
33.8
19.0
Table A5: Data augmentation with simulated lighting variations.
E,4 Comparison with SoTA Lighting Estimation works
We further compare our neural lighting estimation module with the SoTA lighting estimation ap- proaches SOLDNet [74] and NLFE [82]. Table A6 shows the lighting estimation results on PandaSet, where the GPS GPS-calculated sun position is used as reference in eтror computation. For SOLDNet, we use the official pre-trained model run inference on limited field-of-view (FoV) Pront-camera images. For NLFE, we re-implement the sky dome estimation branch without differentiable actor insertion and local volume lighting since the public implementation unavailable. We also compare variation of NLFE (named NI.FE*) that takes our completed LDR panorama image L as input. For a fair comparison, LightSim uses the predicted sun position during the encoding procedure. For NLFE and LightSim, the sky dome estimators take the sun intensity and direction explicitly to enable more human-inteerne cuutannnterpretable lighting control. Therefore, we also evaluate the decoding consistency error flg-scale for sun intensity and degree for sun direction). The average metrics are reported on all PandaSet sequences with night logs excluded.
As shown in Table A6, LightSim recovers more accurate HDR sky domes compared to prior SoTA works, with the lowest angular error. also produces lower decoding error compared to NLFE. Interestingly, we also find that using more camera data (panorama vs limited-FoV image) significantly enhances NLFE''s estimation performance and reduces decoding errors. This verifies our idea of leveraging real- real-world data sufficiently to build the lighting-awane digital twins.
29

---

Decoding Error 1
Method
Input
Angular Error
Intensity
Angle
SOLDNet [74]
Limited-FoY
69.98-
NLFE [82]
Limited-FoV
78.29º
2.68
12.15
NLFE [82]
Panorama
47.39'
2.27
8.53
Ours (no GPS)
Panorama
20.01
1.25
1.78
Table A6: Comparisons of sky dome estimation PandaSet. As reference, model with GPS
leads to 3.78 angular error in sun direction prediction and 1.64 decoding error.
Fig. A20 shows more lighting estimation examples on PandaSet, including stitched partial panorama images Ipano completed LDR panorama L, and HDR sky domes E. LightSim leverages GPS and time data to get the approximate sun location, enabling recovery of the sun in predicted HDRs even observed in partial panorama images pano, and the surrounding observed sky and scene context can still be used to approximately estimate the sun intensity.
parfolparoraro Leuee
LoR
pazz
moonday
Figure A20: More lighting estimation results on PandaSet.
We further compare virtual actor insertion against SOLDNet and NLFE in Figure A21 on PandaSet sequence 001. We highlight two regions {A, B} for comparison to showcase the importance of accurate sun intensity and location prediction, as well as the capability model inter-object lighting effects. For SOLD-Net and NLFE, we use Poisson surface reconstruction (PSR) [31] to obtain the ground mesh as the plane for virtual actor insertion. Specifically, using semantic segmentation labels, estimate per-point normals from the 200 nearest neighbors within 20em, and orientate the normals upwards. Then, conduct PSR with octree depth set 12 and remove the 2% lowest density vertices.
first only keep the ground points
Original
SOLD-Net
NLFE' (Panorama)
Ours
Figure A21: Qualitative comparison of lighting- virtual object insertion.
For SOLD-Net, the inserted vehicle looks too bright, with hard shadows cast differently from the other actors since the predicted sun intensity is too strong and the sun direction is not correctly inferred. NLFE estimates the sun intensity and direction more reasonably by consuming our completed LDR panorama image. However, it cannot simulate the shadow cast by the original actor onto the
30

---

inserted green vehicle due to the lack of 3D digital twins. In contrast, LightSim can perform virtual actor insertion with inter-object lighting effects simulated accurately thanks to the accurate lighting estimation and 3D world modelling.
In Fig. A22, show additional lighting-awarr actor insertion examples another PandaSet sequence 024, where the sun is visible in the front camera. LightSim inserts the new actors seamlessly and can model lighting effects such as inter-object shadow effects (between real and virtual objects).
Real
Camera Simulation with LightSim
Real vs Sim
Figure A22: Additional lighting-awann actor insertion examples with LightSim.
E.5
Additional Camera Simulation Examples
Combining all these capabilities results in a controllable, diverse, and realistic camera simulation with LightSim. Fig. A23, we show additional camera simulation examples similar to Fig. I and Fig. 7 in the main paper. We show the original scenario in the first block. In the second block, we show simulated scene variations with an actor cutting into the SDV's lane, along with inserted traffic barriers. resulting in completely new scenario with generated video data under multiple lighting conditions. In the third block, we show another example where we add inserted barriers and replace all the scene actors with completely new set of actors reconstructed from another scene. The actors are seamlessly inserted into the scenario with the new target lighting. Please refer to the project page for video examples.
31

---

Figure A23: Additional controllable camera simulation examples.
E.6 Additional Qualitative Results on nuScenes
We further showcase LightSim's ability to generalize to driving scenes in nuScenes. We provide more qualitative scene relighting results in Fig. A24 and Fig. A25. Specifically, we select ten diverse scenarios that involve traffic participants such as vehicles, pedestrians and construction items. The sequence IDs are 011, 135, 154, 158, 159, 273, 274, 355, 544, 763. As described in Sec. C.3, we conduct neural scene reconstruction and lighting estimation (pre-trained on PandaSet) to build the lighting-aware digital twins. Then, apply the neural deferred rendering model pre-trained on PandaSet to obtain the relit images. Although the nuScenes sensor data are much more sparse compared PandaSet (32-beam LiDAR, 2Hz sampling rate), LightSim still produces reasonable scene relighting results, indicating good generalization and robustness. Please refer to the project page for video examples.
Occasionally, we observe noticeable black holes (e.g. on log 355 and 763) in the relit images. This is because the reconstructed meshes are low-quality (nte-watertigh ground, broken geometry) due to sparse LiDAR supervision and mis-calibration. While the neural deferred rendering module designed to mitigate this issue, it cannot handle large geometry error perfectly. Stronger smoothness regularization during the neural scene reconstruction step can potentially improve the model's performance.
32

---

Original
Sim
Sim 2
Sim 3
Log 011
Log 135
AHEAS
Log 154
Log 158
Log 159
Figure A24: Generalization to nuScenes (Part 1).
33

---

Original
Sim
Sim 2
Sim 3
Log 273
Log 274
Log 355
Log 544
Log 763
Figure A25: Generalization to nuScenes (Part 2).
34

---

F Limitations and Future Works
While LightSim can simulate diverse outdoor lighting conditions, there are several areas where could benefit from further improvements. First, LightSim cannot seamlessly remove shadows, as shown in Fig. A26, particularly in bright, sunny conditions where the original images exhibit distinct cast shadows. This is because the shadows are mostly baked during neural scene reconstruction (see view-independent reconstruction Fig. A27), producing flawed synthetic data that confuses the neural deferred rendering module. Moreover, we specify fixed materials [10] and predict sky domes that are not ideal for different urban scenes and may cause real-sim discrepancies as shown in Fig. 9. Those issues can potentially be addressed by better intrinsic decomposition with priors and joint material/lighting learning [83, 64]. More discussions for inverse rendering on urban scenes are provided in Sec D)
Second, LightSim uses an HDR sky dome model the major light sources for outdoor daytime scenes. Therefore, LightSim cannot handle nighttime local lighting sources such as street lights, traffic lights, and vehicle lights. A potential solution is to leverage semantic knowledge and create local lighting sources (e.g., point/area lights [56] or volumetric local lighting [82]). Moreover, our experiments also focus on camera simulation for perception models, and we may investigate the performance of downstream planning tasks in future works. Lastly, our current system implementation relies on the Blender Cycles rendering engine [7], which is slow to render complex lighting effects. Faster rendering techniques can be incorporated to further enhance the efficiency of LightSim [69, 56].
Apart from the further method improvements mentioned above, it is important collect extensive data from real-world urban scenes under diverse lighting conditions (e.g., repeating the same driving route under varying lighting conditions). Such data collection aids in minimizing the ambiguity inherent in intrinsic decomposition. Moreover, it paves the way for multi-log training with authentic data by providing larger set of real-real pairs for lighting training.
270
Saring Divector Light
bakad deazon
Target
balod chabo
Figure A26: Failure cases with strong directional lighting. The neural deferred rendering network cannot fully remove the baked shadows (top row) and other sensor effects (e.g., lens flare bottom row). We use green and red arrows to highlight areas where LightSim performs well and not well.
Oriiina
RGB
Defluo Rocon.
Figure A27: View-independent reconstruction results for LightSim. Shadows are baked this stage, which are then mitigated by neural deferred rendering. The relighting failure cases for the last example are shown in Fig. A26.
In Fig. A26, we highlight two examples of LightSim applied to scenes with strong directional lighting and high sun intensity. Each row shows the shadow editing/relighting results under four different sun angles of the target environment map. In the top row, LightSim cannot fully remove source shadows
35

---

in bright and sunny conditions due to the baked shadows in the view-independent reconstruction. Moreover, due inaccurate HDR peak intensity estimation, the brightness of cast shadows cannot match the original images well. In the bottom row, we depict a source image with high sun intensity challenging to remove the sun glare and alter the over-
and glare relit to new target lighting. exposed regions in this setting, but we can still apply some relighting effects to the cars and buildings in the scene (see arrows).
G Computation Resources
In this project, ran the experiments primarily on NVIDIA Tesla T4s provided by Amazon Web Services (AWS). For prototype development and small-scale experiments, we used local workstations with RTX A5000s. Overall, this work used approximately 8,000 GPU hours (a rough estimation based on internal GPU usage reports), of which 3,000 were used for the final experiments and the rest for exploration and concept verification during the early stages of the research project. We provide rough estimation of GPU hours used for the final experiments in Table A7, where we convert one A5000 hour to two T4 hours approximately.
Experiment
T4 Hours
Comments
Table 1 (perceptual quality validation)
1850
LightSim (100), NeRF-OSR (1500), EPE (150)
Table 2 (downstream training)
150
6x models, each takes 25 GPU hours
Table A4 (lighting estimation)
40
15h NLFE, 25h LightSim
Fig. 5 & Table A2 (ablations)
400
50h cach model
Fig. (nuScenes)
40
20h for digital twins, 20h for relighting
Others (data generation & demos)
510
500h lighting data generation + 10h demos
Table A7: Summary of GPU hours used for the final experiments.
H Licenses of Assets
We summarize the licenses and terms of use for all assets (datasets, software, code, pre-trained models) in Table A8.
Blender [7]
GNU General Public License (GPL.) Royalty-Free"
rttps:///us: http://///wbbbbdrr.ooo
HDRMaps [24]
https httpma///ddmapp.coom con/
HoliCity [100]
Non commercial purpase!"
https://holicity... 1o/
TurboSquid 3D Models
Royalty-Free" BC 4.0 Non-commercial (CC BY-NC-SA 4.0)
hstps://uve wwtp:w/wbbbruudd..mmm con!
PandaSer [87]
httpa attpa://Ai wttppppll.co/oopoaaaaaaaaaaaa wtta-a-aa-aaattteaaaaaaaaa
nuScenes [1]]
hetps : http:a:///oe..mmccee org/
SOLDNet [74]
Apache License 2.0 MIT' License
https: htttp:://ggthbb.. wwtpe/cccnccccc.of//ee..
Ket./
EPE [6]]
hstps: stttp://githaen http://gtbbbcccceeo wtt.aal..co//lll/aee
SelF-OSR [6]]
Apache License 2.0
https: htttss///gttubb.c. wwtp///eee/eeelggeeeeeeeeee
NeRI-OSR [64]]
Non-commercial purpose!!
https: // /tth/b.ttt.btt///// coa/ con/r00tman//eet /Wefr oSa
Color Transfer [60]
MIT License
sstpa://githus con/ http///rrr//ooooee tooas//taatorreeterr transfer
Table A8: Summary of the licenses of assets.
I Broader Impact
LightSim offers enhancements in camera-based robotic perception, applicable to various domains such self-driving vehicles. Its ability to generate controllable camera simulation videos (e.g., actor insertion, removal, modification, and rendering from new viewpoints) and adapt to varying outdoor lighting conditions can potentially improve the reliability and safety of intelligent robots for broad range of environmental conditions. Additionally, LightSim's ''s capacity to create lighting- twins can improve realism digital entertainment applications such as augmented reality or virtual reality. However, as with any technology, the responsible use of LightSim is important. Privacy concerns may arise when creating digital twins of real-worke locations. We also caution that our system might produce unstable performance or unintended consequences under different datasets, especially when the sensory data are very sparse and noisy.
digital
36


Reviews:
[{"review_id": "iJcMW2Zu4K", "reviewer": "Reviewer_5Bwf", "summary": "This work extends a recent novel view synthesis approach for autonomous driving scenes with relighting and virtual object insertion with properly cast shadows. The proposed method is a system that consists of the following steps: i) geometry and albedo reconstruction from sensor data, ii) estimation of the environment map (inpainting the sensor data and lifting LDR to HDR), iii) neural deferred rendering that takes in the source image and relights it based on a new environment map. The proposed system enables relighting the original source images or novel views rendered from the step (i). Apart from this, it also allows virtual object insertion of either reconstructed or synthetic assets. The proposed approach is thoroughly evaluated on the tasks of relighting, and virtual object insertion on Pandas and Kitti datasets. Furthermore, it is used to simulate data for a downstream task (3D object detection) where it boosts the performance when compared to using only real world data (a subset of all scenes) . In most experiments/evaluation metrics, the proposed outperforms the selected baselines.", "questions": "**Comments**:\nL7: I would suggest softening the claim that the reconstructed assets are *\"relightable digital twins*\". The two simplifications mentioned above are in contradiction with this statement.\n\nI think that tit would be very beneficial to show some results of the UniSIM albedo reconstruction. Do the shadows get baked into albedo and can the network then neural renderer recover from this?\n\nOn a similar note, it would be good to see some results of the physics based rendered images with target env maps that are used to supervise the neural renderer (Eq. 5). Also, the I_rendered|E_src would be interesting to see.\n\nMost of the results that are shown are based on the source images captured on cloudy days or without strong directional light. While I realize that the results on very sunny days or even with sun glare will be worse, I would like to see how gracefully the proposed method degrades. Including some failure cases is always a plus in my mind.\n\n\n**Questions**\n- L137: relating to the comment in the *weaknesses*, how is the diffuse color supervised? Simply using the full RGB of the source images?\n- In the environment modeling, why is the sky intensity a vector quantity? Is it a full image representation?\n- In the physical rendering the buffers are mentioned to be 12 dim, but there is only position (3), normal (3), depth (1), ambient occlusion (1?). What do other dimensions represent?", "limitations": "The authors have extensively described the limitation and societal impacts of the proposed work in the supplementary material. I also appreciated the information about the GPU hours and proper acknowledgment of the data sources.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- This work is a well-designed system that efficiently builds upon prior work (UniSIM). While individual components are not very special, their combination is technically sound and leads to good results.\n- The experimental evaluation is really thorough with several ablation studies and qualitative/quantitative results including the evaluations on the downstream tasks.\n- I agree with the authors that trying to recover perfect materials and geometry in AV scenes is a very challenging task that requires strong priors. Therefore, I really like the use of the neural deferred renderer. \n- The paper is well written and easy to follow, the motivation for the approach is very clear, and the design choices are well-supported by the results/ablation studies.", "weaknesses": "From my perception, the following are the most important weaknesses of this work:\n\n- **Simplification of the reconstruction process**: During the reconstruction stage ((i) above) two strong simplification are made which in my opinion prevent obtaining good results on challenging illuminations (strong directional light). First, (If I understood this part correctly) the method aims to reconstruct the albedo of the scene, by simply removing the directional dependence of the color MLP, but the supervision still comes from the full RGB images. Second, a single material (no information which?) is used for all the assets after the reconstruction. However, material differences in AV scenes are quite large (e.g. asphalt compared to metallic cars). While the reconstructed albedo images are not shown in the submission (if they are, and I missed them, I am sorry), but I assume that in case of strong directional light the shadows simply get baked into the \"albedo\" representation.\n\n- **Temporal aspect**: The prediction of the relighted frames is done \"independently\" for each frame using a deferred neural renderer. This means that the temporal aspect is not considered and there is no guaranty for mutliview consistency. \n\n- **Recovering the env map**: The env map is recovered in two steps, i) the sensor data is projected onto a panorama and inpainted, ii) the LDR image is lifted to HDR. However, if the sun is not observed in the sensor data, I assume that it is very difficult for the inpainting network to predict its location (this is actually very ill-posed as set up). While GPS information can help with the sun location, it cannot predict the occlusion, by clouds. I think that integrating the optimization of the ENV map in the first step and guide the sun location/intensity with the shadows in the scene would be more principled. \n\n- **Somewhat limited novelty**:  This is actually not a major weakness in my mind, but still wanted to bring it up. The individual components of the proposed system are in my mind not very novel (e.g. reconstruction is from UniSim, env in-painting and lifting to HDR has often been done before).  This being said, I do think that the whole system and the combination of these modules are sufficiently novel, but probably  not be too far away from the border of acceptable/expected novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are excited that LightSim is recognized as a “well-designed system” which is “technically sound” with “really thorough experimental evaluation” and “the design choices are well supported”. We are also glad that the reviewer believes the motivation is “very clear” and likes the idea of “neural deferred renderer”. We now address comments.\n\n**Q1: Simplification of reconstruction (i.e., diffuse color, material)** \\\n**A1:** As mentioned in our limitations (supp. L343-345) we agree with the reviewer that LightSim makes simplifications in the reconstruction process and that improvements in intrinsic decomposition can further enhance performance. Recent concurrent works make steps towards better decomposition (FEGR [1], UrbanIR [2]), but it is still a challenging open problem to recover perfect decomposition of materials and light sources for large urban scenes (see **Figure R1-Right** of intrinsic decompositions of recent works - the recovered materials lack little semblance to semantic parts in the original scene). Additionally, these recent relighting works [1-4] also have shadows baked into the recovered albedo (**Figure R1-Left**).  **Figure R2** includes examples of the recovered albedo for four scenes from LightSim, and we also have baked shadows (mentioned in limitations, supp. L339-342). As discussed in L221-226, our novelty is in leveraging neural deferred rendering to overcome the limitations of purely physics-based rendering when the decomposition is not perfect. This allows us to generate better relighting results than prior works that have imperfect decompositions. It is an exciting future direction to incorporate better material decomposition along with neural deferred rendering for improved relighting.  \n\n**Q2: Temporal consistency** \\\n**A2:** We refer the reviewer to the relighting / shadow editing videos (3min 08s to 3min 51s) in our supplementary video. While we do not guarantee temporal consistency, LightSim can produce temporally consistent lighting simulation videos in most cases, therefore we did not explore other techniques to enforce temporal consistency explicitly. We believe the temporal consistency in neural deferred rendering comes from temporally and multi-view consistent inputs during inference (real image, G-buffers) as well as our combination of simulation and real paired relighting data during training.\nWe believe further improving the temporal consistency is an interesting direction for future work. \n\n**Q3: Recovering the env map** \\\n**A3:** Thanks for the suggestion. We believe optimization of the environment map would be an exciting direction to improve relighting. We explain our design choice in the following two aspects. (a) We use a feed-forward network for lighting estimation which is more efficient and can benefit from learning from a larger set of data. In contrast, the optimization paradigm is more expensive which requires per scene optimization but might be more accurate as mentioned by the reviewer. (b) The ill-posed nature of lighting estimation and extreme intensity range make inverse rendering challenging for outdoor scenes. Optimization of the environment map requires a differentiable renderer and high-quality geometry/material to achieve good results. The existing / concurrent state-of-the-art works [1, 3] still cannot solve the problem accurately (e.g., cloud occlusion example mentioned by the reviewer) as shown in Rebuttal Figure R1 (right bottom).\n\n**Q4: Novelty** \\\n**A4:** See **General Response**.\n\n**Q5: Examples of view-independent reconstruction, I_rendered | E_src, strong directional light - failure cases** \\\n**A5:** Thanks for the suggestions. We included those examples in Rebuttal Figure R2 (view-independent reconstruction), R3 (blender renderings) and R4 (figure cases). We will include those results and more analysis in the next version.\n\n**Figure R2:** \\\nWe provide examples of view-independent reconstruction for LightSim. Since we adopt the original RGB as the supervision to train the neural fields that map 3D location to the diffuse $\\mathbf{k}_d$, the diffuse reconstruction results have the shadow baked. \n\n**Figure R3:** \\\nThe original RGB, diffuse reconstruction,  $\\mathbf{I}_\\mathrm{render} | \\mathbf{E}^{\\mathrm{src, tgt}}$, relit image and lighting reference are provided. We generate paired synthetic data under different lighting conditions and design a mixed sim-real training scheme.\n\n**Figure R4:** \\\nWe highlight two examples of LightSim applied to scenes with strong directional lighting and high sun intensity.  Each row shows the shadow editing / relighting results under 4 different sun angles of the target environment map. In Row 1, LightSim cannot fully remove source shadows in bright and sunny conditions (as mentioned in limitations) due to the baked shadows in the view-independent reconstruction, but with neural deferred rendering, LightSim still generates reasonable results. In Row 2, we depict a source image with high sun intensity and glare relit to a new target lighting. It is challenging to remove the sun glare and alter the over-exposed regions in this setting, but we can still apply some relighting effects to the cars and buildings in the scene (see arrows).\n\n**Q6: Why is the sky intensity a vector quantity?** \\\n**A6:** Sky intensity contains 3 channels for R, G, B.\n\n**Q7: G-buffer dimension: position (3), normal (3), depth (1), ambient occlusion (1)** \\\n**A7:** Yes, the buffers are 8 dimensions instead of 12 (in implementation, we loaded depth and ambient occlusion as 3-dim. images for simplicity). We will revise it in the next version. \n\n**Q8: \"*relightable digital twins*\" claim** \\\n**A8:**  We will revise the term to be “*lighting-aware digital twins*” instead to indicate that the reconstructions also include scene lighting estimations.  \n\n[1] FEGR. Wang et al., 2023. \\\n[2] UrbanIR. Lin et al., 2023. \\\n[3] NeRF-OSR. Rudnev et al.., 2022. \\\n[4] Self-OSR. Yu et al., 2020."}, {"review_id": "i0uIn2eovu", "reviewer": "Reviewer_25CV", "summary": "The paper proposes a method, LightSim, for recovering geometry, appearance and scene lighting for driving scenes, which enables downstream applications of scene editing and lighting editing. The method incorporates a learned sky dome estimator for hallucinating the original lighting from limited observations, as well as a image-based rendering module for rendering with novel lighting, using rendering proxies as input. The paper demonstrates more realistic light editing results compared to baseline methods in qualitative evaluations, as well as quantitative improvement in scores of perceptual quality and downstream perception tasks where the proposed method is used for training data augmentation.", "questions": "Please see the above section for questions to be addressed. Without understanding those questions, it is difficult to fully evaluate the soundness of the method and results.", "limitations": "No potential negative societal impact of the work is discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The method is generally novel, and presents reasonable improvements in results. The method takes advantage of full array of sensor data including lidar, RGB, as well as GPS to facilitate reconstruction of scene geometry, lighting including the sun. The paper also introduces a learned rendering model with rendering proxies and lighting cues as input, which is able to alleviate artifacts in rendering.\n\n[2] The paper demonstrates noticeable qualitative improvements when compared to baseline methods. More importantly, given the lack of ground truth images under novel lighting, the paper is able to include indirect quantitative evaluation with perceptual scores and downstream tasks, to demonstrate the method not only produces visually convincing results, but also benefits downstream tasks.", "weaknesses": "[1] Clarity. The paper is in general well-written and easy to follow. However, clarify and additional details have to be enhanced for a polished version. For example, \n\n- (L143) details on the representation of base materials for all assets;\n- (Sec. 3.2) details on how to acquire geometry for dynamic scenes? Is geometry estimated per-frame? If so how to deal with temporal consistency? If not, how dynamic scene is models in a NeRF-like framework for geometry reconstruction?\n- (L246-) What synthetic data is used? What are the specs of the training datasets? What is the training scheme?\n- In learning the image-based renderer on real scenes, the estimated sky dome lighting is needed from the previous stage in the pipeline. What if the estimated sky dome lighting is not perfect? Does that affect the learning of the renderer?\n\n[2] Evaluation. The paper is unique in that it leverages a collection of sensor data besides RGB for the task. In this sense, comparing the method to baselines which leverage RGB data only may not be fair. Moreover, is there any reason why the method is not compared against the SOTAs of [69] and [70]? Is it because of the lack of source code and difficulty to reproduce?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and comments. We are glad that our work is recognized as “novel” with “noticeable qualitative improvements”. We address the concerns as follows.\n\n**Q1: Clarifications and additional details** \\\n**A1:**  We thank the reviewer for the feedback to improve the presentation quality. We will revise the statements clearly and move more details to the main paper for clarity.\n\n**(1) Base materials for all assets:** \\\nWe use Blender PBR materials. Specifically, we use principled BSDF with vertex color as the base color. For all dynamic assets, we set metal=0.5, roughness=0.2, sepcular=0.5, clearcoat = 1.0, clearcoat roughness = 1.0. For the background asset, we set metal=0.0, roughness=0.7, clearcoat = 0.0, specular=0.5. The other material parameters are initialized with Blender default. We will include the details in the revision.\n\n**(2) Details on how to acquire geometry for dynamic scenes:** \\\nWe decompose the dynamic scenes into static background and dynamic objects (assuming to be rigid). Specifically, we use 3D bounding boxes annotation and separate feature grids to model the background and foreground. The neural scene reconstruction is a view-independent version of [75]. Please refer to Sec A.1 (supp. material) for more details. \n\n**(3) Synthetic data used (Line246). What are the specs of the training datasets? What is the training scheme?** \\\nWe use our reconstructed digital twins to generate synthetic data under different lighting conditions as the paired supervision for neural relighting network training. The training dataset configuration and training scheme are described in detail in the supplementary pdf (Line 110 to 121).\n\n**(4) What if the estimated sky dome lighting is not perfect? Does that affect the learning of the renderer?** \\\nIf the estimated sky dome is not perfect, the inference and training of the neural deferred rendering network will be affected since the estimated sky dome is taken as the input during inference and we also use estimated lighting to generate synthetic data. \n \n\n**Q2: Comparison to baselines that only leverage RGB information?** \\\n**A2:** We tried our best to compare existing works for outdoor relighting and lighting estimation works.  Existing works [1, 2] however. are not designed for self-driving  and do not fully leverage the available data (e.g., LiDAR, time and GPS data). We noticed that [1, 2] are also important baselines compared in concurrent works FEGR [3] and UrbanIR [4].\nFurthermore, we enhance the baseline [5] to leverage the digital twins built by LightSim which leverages all available data as LightSim.\n\n[1] Neural Radiance Fields for Outdoor Scene Relighting. Rudnev et al.., ECCV 2022. \\\n[2] Self-supervised Outdoor Scene Relighting. Yu et al., ECCV 2020. \\\n[3] Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes. Wang et al., CVPR 2023. \\\n[4] UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video. Lin et al., Arxiv 2023. \\\n[5] Enhancing Photorealism Enhancement. Richter PAMI, 2021.\n\n**Q3: Comparison with NLFE [69] and FEGR [70]** \\\n**A3:** Thanks for the suggestion. We compared with NLFE on the lighting estimation task and provided results in the supplementary material (Table A4 and Figure A9 in Sec D.4 ).  In summary, LightSim achieves more accurate lighting estimation compared to NLFE. LightSim can also model the inter-object lighting effects compared to NLFE in actor insertion application.\n\nFor FEGR, we notice it is a *concurrent and independent* work (Sec 2 Line 98 to Line 106). It was first made publicly available on arXiv on April 6., **less than two month from the  NeurIPS submission deadline May 11**. According to NeurIPS policy, “Authors are not expected to compare to work that appeared only a month or two before the deadline.” Additionally, since FEGR does not offer its source code, reproducing it within a tight deadline becomes challenging.\n\nWe hope the following response addresses the reviewer’s concern. We look forward to any follow up discussions."}, {"review_id": "9Tw6rakZgF", "reviewer": "Reviewer_ENz6", "summary": "This paper presents a system for decomposing urban outdoor driving scenes into estimated geometry and lighting components, which are then used as inputs to their deferred neural rendering workflow.  The geometry is represented as a mesh that is extracted using Marching Cubes, from an optimized SDF volume, while the lighting is represented as an inferred HDR sky dome.  A physically based renderer (Blender) is used to render deferred shading passes, which are then provided to a U-Net based \"neural renderer\" to produce the final output images.  The target application is realistic relighting, to improve diversity of training data for vision-based perception systems in the driving domain.", "questions": "In Section 3.1, for the base material that is assigned to reconstructed objects, what are the specific material parameters used?\nIn Section 3.1, how exactly is the separation between static background and dynamic actors achieved?\n\nOn L240, why is it beneficial to use a U-Net to generate the final relit image from the deferred rendering passes?  Given that the deferred rendering passes contain imperfections, does the U-Net \n\nCould you provide more details about the feature grids?  These are mentioned, but not elaborated upon as to their importance, or what purpose they serve.  Do these bear some relation to the multi-resolution hash grids from [Instant-NGP, Müller et al, 2022]?  If so, I suggest including the proper citation, and if not, further elaboration would be ideal.\n\nCould you elaborate on why BEVFormer was chosen for the downstream perception training analysis?\n\nCould you clarify which components of the system are optimized per-scene, and which parts are optimized from a larger dataset?  It seems that the geometry and initial LDR panorama are optimized per-scene, while the other modules are learned from large-scale data -- is this accurate?", "limitations": "The relighting results seem to be limited in which aspects of the appearance they affect.  For example, re-rendered shadows look convincing, but the reflections on the cars themselves appear to largely be unaffected in terms of lighting direction / appearance of specular highlights.\n\nEstimation of materials left as future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The system presented in this paper involves significant engineering effort, successfully combining multiple learning-based modules.  Their design of the neural deferred renderer module presents some novel extensions beyond prior work, such as the choice of conditioning the U-Net upon the environment map, as well as their loss formulation that includes terms for perceptual loss and edge loss.  Qualitatively, results generally appear to be of a high quality, on par with or marginally better than existing and concurrent works.", "weaknesses": "The paper presents a system that draws inspiration from existing works [i.e. FEGR], which makes its own novelty/contribution hard to discern.  It would be helpful for the authors to clarify what the unique contributions of this work are that distinguish it from other related works.\n\nSome terminology is inconsistent and/or using non-standard terms, e.g. use of all of the following terms \"physically based rendering\", \"physics based rendering\", and \"physics rendering\" which all share the same meaning, the latter two of which are not commonly used in existing literature, and therefore may either cause confusion, and in the worst case, be inaccurate.  Another such example is \"camera simulation\".  I recommend proofreading to improve this particular aspect of the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and comments. We also appreciate the detailed suggestions to improve the presentation quality. We address the concerns as follows.\n\n**Q1: Discern novelty/contribution with FEGR** \\\n**A1**: As discussed in the related work section, FEGR is a **concurrent and independent** work (Sec 2 Line 98 to Line 106). It was first made publicly available on arXiv on April 6., **less than two month of NeurIPS submission deadline May 11**. According to NeurIPS policy, “Authors are not expected to compare to work that appeared only a month or two before the deadline.” Therefore, the existence of FEGR should not weaken the novelty of LightSim. Moreover, as stated in related work, while FEGR and LightSim are dealing with a similar task, the methods are different. FEGR aims to conduct inverse rendering (intrinsic decomposition) from a single scene with strong regularization and priors. In contrast, LightSim proposes a “novel data pair training scheme” [Reviewer 6GXP]  and learns on many driven scenes to bypass the challenges of inverse rendering. We believe these two approaches are complementary and the combination of both can lead to a better system. Unlike FEGR, LightSim also produces more realistic relighting videos for **dynamic scenes** (reconstruction and relighting of dynamic actors in the original urban scenes, with inter-object lighting effects) and we also demonstrate the effectiveness of our approach for downstream detection tasks. \n\nWe plan to move the discussion in related work into a separate paragraph to make it clearer. Any follow up discussions are highly appreciated.\n\n**Q2: Unify the terminology** \\\n**A2:** Thanks for the suggestion. We will unify the terminology (“physically based rendering” and “camera simulation”) to avoid confusion.\n\n**Q3: Base material parameters used for reconstructed objects?** \\\n**A3:** Thanks for pointing it out. We use Blender PBR materials. Specifically, we use principled BSDF with vertex color as the base color. For all dynamic assets, we set metal=0.5, roughness=0.2, sepcular=0.5, clearcoat = 1.0, clearcoat roughness = 1.0. For the background asset, we set metal=0.0, roughness=0.7, clearcoat = 0.0, specular=0.5. The other material parameters are initialized with Blender default. We will include the details in the revision.\n\n**Q4: Separation between static background and dynamic actors?** \\\n**A4:** We decompose the dynamic scenes into static background and dynamic objects (assuming to be rigid). Specifically, we use 3D bounding box annotations and separate feature grids to model the background and foreground. The neural scene reconstruction is a modified version of [75]. Please refer to Sec A.1 (supp. material) for more details. \n\n**Q5: Why use a U-Net to generate the final relit image from the deferred rendering passes?** \\\n**A5:** The U-Net learns to relight from large-scale generated synthetic data under different lighting conditions (low resolution and contain imperfections), meanwhile maintaining the context and quality of original RGB image. By carefully designing the training scheme (sim-to-sim, sim-to-real pair) and learning from large-scale data, our neural deferred rendering pass takes the original RGB image / rendered buffers as inputs and produces high-quality rendering results in a lighting aware manner.\n\n**Q6: Details about feature grids and relation to Instant-NGP, details, citation?** \\\n**A6:** We adopt the instant-NGP feature grids with hash encoding. Specifically, we set 16 levels of multi-level feature grids with a hash table size of 2**19. The dimensionality of the feature vector stored in each level's entries is set as 2 and the resolution of the coarsest level is set 16**3. We apologize for missing the citation. We will add the citation and details in the revision.\n\n**Q7: Why was BEVFormer chosen for the downstream perception training analysis?** \\\n**A7:** BEVFormer is a state-of-the-art camera only 3D detection model for self-driving scenes, which has been used for comparison and autonomy evaluation in prior works [1, 2, 3, 4].\n\n[1] BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation. Liu et al., ICRA 2022. \\\n[2] RoboBEV: Towards Robust Bird’s Eye View Perception under Corruptions. Xie et al., ICCV 2023. \\\n[3] Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving. Dong et al., 2023. \\\n[4] On the Adversarial Robustness of Camera-based 3D Object Detection. Xie et al., 2023.\n\n**Q8: Which components of the system are optimized per-scene, and which parts are optimized from a larger dataset? geometry and initial LDR panorama are optimized per-scene, while the other modules are learned from large-scale data?** \\\n**A8:** Yes. The geometry (baked with view-dependent color) and LDR/HDR panorama are optimized / predicted per-scene. The other modules including panorama completion network, LDR2HDR lifting network, neural deferred rendering network are learned from large-scale data. \n\n**Q9: Reflections on the cars themselves appear to largely be unaffected in terms of lighting direction / appearance of specular highlights. Estimation of materials left as future work.** \\\n**A9:** We agree that better handling of specular highlights is an exciting research direction that can enhance LightSim. As depicted in Figure R1, this is an open-problem for large-scale scene relighting.\n\nWe hope the following response addresses the reviewer’s concern. We look forward to any follow up discussions."}, {"review_id": "c13LiJTVfI", "reviewer": "Reviewer_6GXP", "summary": "Generating training data for self-driving cars is a challenging task due to the difficulty of capturing real-world scenarios. While video games have been used to generate training data, there exists a domain gap between virtual and real-world environments. \nTo address these challenges, the authors of this paper propose an approach that generates composable and relightable scenes. The method involves a multi-stage process aimed at training a dynamic Neural Radiance Fields (NeRF) model, which decomposes a scene into static and dynamic components. \n\nFurthermore, the illumination in the scene is learned as a high dynamic range (HDR) sky dome. The proposed approach tackles several sub-problems to achieve its goal. Firstly, panorama reconstruction is performed from the input data. Additionally, in-painting techniques are employed to fill unobserved areas caused by occlusions. An LDR-to-HDR estimation is carried out to enhance the illumination information. Supervision signals are provided by leveraging known sun angles and intensities from GPS data and timestamps. These signals aid in training the model to accurately estimate the scene illumination. \n\nTo generate lighting-relevant data, a non-differentiable rendering step is performed, producing essential information such as surface normals, depth, position, and ambient occlusion, along with a render using a single base material.\n\nA 2D U-Net utilizes the render buffer and a target illumination to guide the relighting or editing of a source image. This process allows for consistent relighting of the scene and offers flexibility to insert new objects, as well as move or remove dynamic objects. To train the system, the authors introduce a novel data pair training scheme.", "questions": "- The geometry from marching cubes often has obvious artifacts, which are especially apparent when lighting the mesh (random shadowed triangles). Have the authors noticed these effects? Is the neural rendering removing them? Is this due to the I_render|E_src -> I_real step?\n- The authors propose to only model view-independent diffuse colors instead of the view-dependent typical NeRF one. Did the authors notice any artifacts around reflective surfaces, which degrade the meshes severely?\n", "limitations": "The authors discuss the limitations of their method.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- I like the fusing of all available data, especially the supervision from sun angles based on time and GPS data. This is a really strong regularization loss in outside scenes, as the illumination is dominated by sunlight.\n- The paired learning scheme for the neural renderer is a good idea and seems to provide good results.  \n- The editing produces plausible relighting and edits that surpass the quality of purely synthetic data from common game engines.\n", "weaknesses": "- Judging illumination without any specific reference is hard for humans. This is apparent in Fig. 4, where the relighting is hard to judge for plausibility. I suggest taking two images: Source 1 under illumination 1 and Source 2 under illumination 2. Then generate Source 1 under Illumination 2 and vice versa. This way, there exists a reference point for each illumination, and due to the similarity in scenes, it is easier to judge if the illumination is plausible.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and comments. We are glad that the reviewer appreciates the novelty and contribution of our work. We address the concerns as follows.\n\n**Q1: Judging illumination is hard for human without reference** \\\n**A1:** Thanks for the suggestion. We agree that it is challenging to judge illumination for humans (even experts) without ground truth data or other reference.  In Figure 4, the first and third row are relighting with estimated lighting conditions of other PandaSet log snippets. We added the reference real image in **Rebuttal Figure R5**. We note that the lighting variations of public driving datasets (e.g., PandaSet and NuScenes) are limited, therefore we also use third party HDRs for evaluation (without limited FoV real-image reference). As discussed in Appendix E, it is an important future direction to collect extensive data from real-world urban scenes under diverse lighting conditions (e.g., repeating the same driving route under varying lighting conditions), which would be beneficial to the community (reducing the ambiguity and more helpful for lighting evaluation).\n\n\n**Q2: Random shadowed triangles when relighting due to artifacts in marching cube extracted geometry? (Have the authors noticed these effects? Is neural rendering removing them? Is this due to the $\\mathbf{I}_\\mathrm{render} | \\mathbf{E}^\\mathrm{src}$ -> I_real step?)** \\\n**A2:** Yes, we noticed that random shadowed triangles are very common in the $\\mathbf{I}_\\mathrm{render} | \\mathbf{E}^\\mathrm{src}$ due to non-smooth extracted meshes. This is more obvious for the nuScenes dataset where the LiDAR is sparser (32-beam) and the capture frequency is lower (2Hz). We can notice there are a lot of holes and random shadowed triangles. However, thanks to our image-based neural deferred rendering pass which are trained with mixed sim-real data, our relighting network takes the original image and modifies the lighting, which removes a lot of those artifacts in the final relit images. We show two nuScenes examples in **Rebuttal Figure R6**.\n\n**Q3: Any artifact in the meshes around reflective surfaces due to simple modelling of view-independent diffuse colors.** \\\n**A3:** Yes, due to the simplification of view-independent diffuse color and base materials, we cannot accurately simulate the lighting effects around reflective surfaces (e.g., car windows).  Because we leverage both LiDAR and camera data to build the meshes, we did not notice large degradation in the mesh quality for these reflective surfaces.\nWe hope LightSim can leverage better intrinsic decomposition in the future to further enhance the performance.\n"}, {"review_id": "jw1XGsKzXr", "reviewer": "Reviewer_G49E", "summary": "This paper proposed LightSim, a lighting-aware camera simulation system for improving robot perception. This system built relightable digital twins from real-world raw sensor data and enabled applications, e.g., actor insertion, modification, removal, and re-rendering.", "questions": "see the Weakness", "limitations": "see the Weakness", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Propose a complete system for outdoor illumination estimation of urban driving scenes and its application.\n2. Leverage physics-based rendering to enable controllable simulation of the dynamic scene.\n3. Better results than the baseline.", "weaknesses": "1. The workload of the paper is large, but it seems an integration of previous works and lacks innovation. e.g., [25] for scene reconstruction, DeepFillv2 for panorama image inpainting, [17] for neural deferred rendering.\n2. The motivation for neural deferred rendering needs to be clarified. The motivation for neural deferred rendering needs to be clarified. In the right of Fig. 3, I^src,  E^src, and E^tgt are known, I_buffer, S^src, and S^tgt are generalized by Blender, and digital twins are estimated, then I^tgt can be directly rendered by Blender. Why train an extra network for rendering?\n3. What is the material map in L235? Is Blender's default Principle BSDF? and link the vertex color as the base color?\n4. Digital Twins is described differently in Fig. 2 and L126-127.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and comments. We address the concerns as follows\n\n**Q1: Novelty - Integration of previous works** \\\n**A1:** Our paper’s novelty lies in developing a neural lighting simulation system for self driving, which is critical for thorough evaluation and more robust training of robot autonomy  before safe deployment. To our knowledge, we are one of the first to perform relighting for dynamic urban scenes (SDV and other actors moving in the scenario). LightSim significantly outperforms the state of the art, producing high-quality photorealistic driving videos under a wide range of lighting conditions.\n\nWe strongly believe LightSim is a critical and innovative step towards realistic and scalable lighting simulation for robotics. Through this paper, we also hope to convey the importance of leveraging available real data (digital twins) and propose a new regime of exploiting digital twins for lighting simulation.\n\nWhile some individual components (reconstruction for dynamic scenes [75], environment map in-painting [59] and LDR-HDR lifting [69], G-buffers to aid learning [52]) are studied before, why they are used and how they are used are all carefully designed in the context of self-driving. Specifically, different from existing works that conduct lighting estimation from single image (limited FoV) which is ill-posed and challenging, we fuse all available data (i.e., all six cameras and sun angles based on time and GPS) to reduce ambiguity (see Table A4). To overcome the lack of real world driving scenes captured under different lightings, we propose a “novel data pair training scheme” [Reviewer 6GXP] that leverages the digital twins that are built from real world to generate synthetic paired images under different lighting conditions. Those generated diverse synthetic lighting variation data are then combined with real data to train the neural deferred rendering network. The resulting framework is generic, interpretable and has various capacities including actor insertion, removal, modification, and rendering from new viewpoints, all in lighting-aware manner. We believe LightSim is not just a simple extension or integration of previous works. \nAlso, exploiting existing algorithms to realize a novel idea does not mean there is no technical contribution [Reviewer 5Bwf and 6GXP]. We hope the reviewer can acknowledge this. We look forward to follow-up discussions.\n\n**Q2: Clarification of motivation for neural deferred rendering. Why not directly render using Blender?** \\\n**A2:** Recovery of perfect geometry, material and lighting in urban driving scenes is a very challenging task which requires strong priors and real-world data under different lighting conditions. To mitigate this issue, we propose to generate synthetic paired lighting variations from our imperfect digital twins and use neural deferred renderer to generate relighting videos in a more realistic manner. Specifically, the neural deferred rendering network learns to relight (guided by coarse blender renderings) while maintaining the realism by taking the original real image as input. The direct blender rendering results have noticeable artifacts as shown in **Rebuttal Figure R3 and R6**.\n\n**Q3: Material map in L234 (default Principle BSDF and link vertex color as base color?)** \\\n**A3:** Yes, We use default Principled BSDF in blender and link vertex color as the base color. For all dynamic assets, we set metal=0.5, roughness=0.2, sepcular=0.5, clearcoat = 1.0, clearcoat roughness = 1.0. For the background asset, we set metal=0.0, roughness=0.7, clearcoat = 0.0, specular=0.5. The other material parameters are initialized with Blender default. We will include the details in the revision.\n\n**Q4: “Digital Twins” is described differently in Fig. 2 and L126-127.** \\\n**A4:** Both Fig.2 and L126-127 describe lighting-aware digital twins as containing geometry, material and lighting. We look forward to further discussions if additional clarification is required.\n\nWe hope the following response addresses the reviewer’s concerns. We look forward to follow-up discussions."}]

Author rebuttal:
We thank the reviewers for their thoughtful reviews and valuable comments. We are excited that the reviewers found our approach “novel” [**Reviewer 5Bwf, Reviewer 25CV, Reviewer ENZ6, Reviewer 6GXP**], and acknowledged our evaluation is “thorough” [**Reviewer 5Bwf**], the results are “good/high-quality” or have “noticeable improvements” over prior arts [**Reviewer 25CV, Reviewer ENZ6, Reviewer 6GXP, Reviewer G49E**].

In the following, we briefly summarize a few points. Please see individual responses for more details.

**Novelty [Reviewer G49E, Reviewer ENz6]**

Our paper’s novelty lies in developing a neural lighting simulation system for self driving, which is critical for thorough evaluation and more robust training of robot autonomy  before safe deployment. To our knowledge, we are one of the first to perform relighting for dynamic urban scenes (SDV and other actors moving in the scenario). LightSim significantly outperforms the state of the art, producing high-quality photorealistic driving videos under a wide range of lighting conditions.

We strongly believe LightSim is a critical and innovative step towards realistic and scalable lighting simulation for robotics. Through this paper, we also hope to convey the importance of leveraging available real data (digital twins) and propose a new regime of exploiting digital twins for lighting simulation.

While some individual components (reconstruction for dynamic scenes [75], environment map in-painting [59] and LDR-HDR lifting [69], G-buffers to aid learning [52]) are studied before, why they are used and how they are used are carefully designed for self-driving simulation. Specifically, different from existing works that conduct lighting estimation from single image (limited FoV) which is ill-posed and challenging, we fuse all available data (i.e., all six cameras and sun angles based on time and GPS) to reduce ambiguity (see Table A4). To overcome the lack of real world driving scenes captured under different lightings, we propose a “novel data pair training scheme” [**Reviewer 6GXP**] that leverages the digital twins that are built from real world to generate synthetic paired images under different lighting conditions. Those generated diverse synthetic lighting variation data are then combined with real data to train the neural deferred rendering network. The resulting framework is generic, interpretable and has various capacities including actor insertion, removal, modification, and rendering from new viewpoints, all in lighting-aware manner. We believe LightSim is not just a simple extension or integration of previous works. 
Also, exploiting existing algorithms to realize a novel idea does not mean there is no technical contribution [**Reviewer 5Bwf and 6GXP**]. We hope the reviewers, in particular **Reviewer G49E**, can acknowledge this. 

**Comparison to FEGR [Reviewer ENZ6, Reviewer 25CV]**

FEGR is a *concurrent and independent* work (Sec 2 Line 98 to Line 106). It was first made publicly available on arXiv on April 6., **less than two month of NeurIPS submission deadline May 11**. According to NeurIPS policy, “Authors are not expected to compare to work that appeared only a month or two before the deadline.” Therefore, the existence of FEGR should not weaken the novelty of LightSim. Moreover, as clearly stated in related work, while FEGR and LightSim are dealing with a similar task, the methods are very different. FEGR aims to conduct inverse rendering (intrinsic decomposition) from a single scene with strong regularization and priors. In contrast, LightSim proposes a “novel data pair training scheme” [Reviewer 6GXP]  and learns on many driven scenes to bypass the challenges of inverse rendering. We believe these two approaches are complementary and the combination of both can lead to a better system. LightSim also produces more realistic relighting videos for **dynamic scenes** and demonstrates the effectiveness in downstream detection.

**Please see the attached pdf for all rebuttal figures (Figure R1-6)**. We hope our responses and additional results can address the concerns. We look forward to follow-up discussions.


Meta-review: This paper proposes a neural lighting simulation system for self-driving. Although the initial reviews are mixed, the authors provide a strong rebuttal and address most of the concerns raised by reviewers. The final assessments are generally positive, with four reviewers recommending acceptance and one reviewer with "borderline reject" indicating his/her willingness to raise the score to "borderline accept" in the comment. Aligned with the consensus of the reviewers, the AC thinks this paper makes good technical progress and thus recommends acceptance.
Decision: Accept (poster)