SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure A17: Qualitative examples of scene relighting for LightSim (Part 2).
Figure A18: Qualitative examples of scene relighting with shadows edited.
E.2 Additional Ablation Study
We provide a more thorough ablation study on the important components of the neural deferred rendering module. The perception quality metrics and additional qualitative examples are provided in Table A4 and Fig. A19.
#ID	Data Pairs sim-real identity	Rendering Buffers I_buffer \(\{\mathbf{S}^{\mathrm{src}},\mathbf{S}^{\mathrm{tgt}}\}\)	Edge loss \(\mathcal{L}_{\text {edge}}\)	FID ↓	KID (×103) ↓
0	X	60.9	31.1 ± 4.0
1	X	62.5	32.7 ± 4.0
2	X	50.5	21.4 ± 4.1
3	X	49.8	23.1 ± 5.0
4	X	109.8	88.7 ± 7.3
5	200	67.1	40.9 ± 4.9
6	800	57.3	31.8 ± 4.3
Ours	✓	✓	✓	✓	400	55.4	27.6 ± 3.7
Table A4: Ablation studies on LightSim components. For clarity, we only mark the differences between our final model and other configurations. Blank components indicate that the setting is identical to our final model.
Figure A19: Additional ablation study on neural deferred rendering. LightSim can simulate intricate lighting effects (highlights, shadows) while maintaining realism.
We choose sequence 001 for the quantitative evaluation (FID/KID scores), and all 80 front-camera images are used as the reference dataset. To sufficiently measure rendering quality, we generate  $3 \times 80$  relit images using the three unseen HDRs in Fig. A12. We also generate 72 relit images (first frame) with shadows edited using the first unseen HDR. Then, to test generalization to unknown lighting conditions, we pick the first frame and generate 92 relit images using other real HDR maps. Therefore, for each model configuration, there are 404 simulated images used in total for perceptual quality evaluation. We set kid Subset_size=10 for this experiment.
As shown in Table A4 and Fig. A19, sim-real and identity data pairs provide useful regularization for the neural deferred rendering module by reducing visual artifacts caused by imperfect geometry. Removing those data pairs leads to less realistic simulation results and worse FID/KID scores. On the other hand, the rendering buffers and shadow maps play an important role in realistically simulating intricate lighting effects such as highlights and shadows. We observe unrealistic color and missing cast shadows if pre-computed buffers  $\mathbf{I}_{\mathrm{buffer}}$  and shadow maps  $\{\mathbf{S}^{\mathrm{src}}, \mathbf{S}^{\mathrm{tgt}}\}$  are removed. Note that the KID/FID metrics are lower when removing rendering buffers since the reference dataset does not include the real data under new lighting conditions; this cannot be interpreted as better visual quality. Finally, we ablate the content-preserving loss  $\mathcal{L}_{\mathrm{edge}}$  and find that a proper loss weight helps the model reduce synthetic mesh-like artifacts (compared to  $\lambda_{\mathrm{edge}} = 0$ ) while properly simulating new lighting effects (compared to  $\lambda_{\mathrm{edge}} = 800$ ).
E.3 Additional Object Detection Metrics
We report detailed detection metrics for perception training with different data augmentation approaches for better reference. Specifically, we report the average precision (AP) at different IoU thresholds: 0.1, 0.3, and 0.5. As shown in Table A5, using LightSim-simulated data yields the best performance improvements. Color Transfer and standard color augmentation [44] are also effective ways to promote the performance of autonomy model under novel lighting conditions. In contrast, Self-OSR and EPE either harm the detection performance or bring marginal gains due to noticeable visual artifacts that cause sim-real domain gap between training and validation.
Model	mAP (%)	AP@0.1	AP@0.3	AP@0.5
Real	32.1	51.2	29.5	15.7
Real + Color aug. [44]	33.8 (+1.7)	53.9	31.0	16.4
Real + Sim (Self-OSR)	30.3 (-1.8)	45.6	29.4	16.0
Real + Sim (EPE)	32.5 (+0.4)	50.2	30.6	16.7
Real + Sim (Color Transfer)	35.1 (+3.0)	55.3	32.3	17.6
Real + Sim (Ours)	36.6 (+4.5)	57.1	33.8	19.0
Table A5: Data augmentation with simulated lighting variations.
E.4 Comparison with SoTA Lighting Estimation works
We further compare our neural lighting estimation module with the SoTA lighting estimation approaches SOLDNet [74] and NLFE [82]. Table A6 shows the lighting estimation results on PandaSet, where the GPS-calculated sun position is used as reference in error computation. For SOLDNet, we use the official pre-trained model to run inference on limited field-of-view (FoV) front-camera images. For NLFE, we re-implement the sky dome estimation branch without differentiable actor insertion and local volume lighting since the public implementation is unavailable. We also compare a variation of NLFE (named NLFE*) that takes our completed LDR panorama image  $\mathbf{L}$  as input. For a fair comparison, LightSim uses the predicted sun position during the encoding procedure. For NLFE and LightSim, the sky dome estimators take the sun intensity and direction explicitly to enable more human-interpretable lighting control. Therefore, we also evaluate the decoding consistency error (log-scale for sun intensity and degree for sun direction). The average metrics are reported on all PandaSet sequences with night logs excluded.
As shown in Table A6, LightSim recovers more accurate HDR sky domes compared to prior SoTA works, with the lowest angular error. It also produces lower decoding error compared to NLFE. Interestingly, we also find that using more camera data (panorama vs limited-FoV image) significantly enhances NLFE's estimation performance and reduces decoding errors. This verifies our idea of leveraging real-world data sufficiently to build the lighting-aware digital twins.
Method	Input	Angular Error ↓	Decoding Error ↓
Intensity	Angle
SOLDNet [74]	Limited-FoV	69.98°	-	-
NLFE [82]	Limited-FoV	78.29°	2.68	12.15°
NLFE* [82]	Panorama	47.39°	2.27	8.53°
Ours (no GPS)	Panorama	20.01°	1.25	1.78°
Table A6: Comparisons of sky dome estimation on PandaSet. As reference, our model with GPS leads to  $3.78^{\circ}$  angular error in sun direction prediction and  $1.64^{\circ}$  decoding error.
Fig. A20 shows more lighting estimation examples on PandaSet, including stitched partial panorama images  $\mathbf{I}_{\mathrm{pano}}$ , completed LDR panorama  $\mathbf{L}$ , and HDR sky domes  $\mathbf{E}$ . LightSim leverages GPS and time data to get the approximate sun location, enabling recovery of the sun in predicted HDRs even if not observed in partial panorama images  $\mathbf{I}_{\mathrm{pano}}$ , and the surrounding observed sky and scene context can still be used to approximately estimate the sun intensity.
Figure A20: More lighting estimation results on PandaSet.
We further compare virtual actor insertion against SOLDNet and NLFE in Figure A21 on PandaSet sequence 001. We highlight two regions  $\{A, B\}$  for comparison to showcase the importance of accurate sun intensity and location prediction, as well as the capability to model inter-object lighting effects. For SOLD-Net and NLFE, we use Poisson surface reconstruction (PSR) [31] to obtain the ground mesh as the plane for virtual actor insertion. Specifically, we first only keep the ground points using semantic segmentation labels, estimate per-point normals from the 200 nearest neighbors within  $20\mathrm{cm}$ , and orientate the normals upwards. Then, we conduct PSR with octree depth set as 12 and remove the  $2\%$  lowest density vertices.
Figure A21: Qualitative comparison of lighting-aware virtual object insertion.
For SOLD-Net, the inserted vehicle looks too bright, with hard shadows cast differently from the other actors since the predicted sun intensity is too strong and the sun direction is not correctly inferred. NLFE estimates the sun intensity and direction more reasonably by consuming our completed LDR panorama image. However, it cannot simulate the shadow cast by the original actor onto the
inserted green vehicle due to the lack of 3D digital twins. In contrast, LightSim can perform virtual actor insertion with inter-object lighting effects simulated accurately thanks to the accurate lighting estimation and 3D world modelling.
In Fig. A22, we show additional lighting-aware actor insertion examples on another PandaSet sequence 024, where the sun is visible in the front camera. LightSim inserts the new actors seamlessly and can model lighting effects such as inter-object shadow effects (between real and virtual objects).
Figure A22: Additional lighting-aware actor insertion examples with LightSim.
E.5 Additional Camera Simulation Examples
Combining all these capabilities results in a controllable, diverse, and realistic camera simulation with LightSim. In Fig. A23, we show additional camera simulation examples similar to Fig. 1 and Fig. 7 in the main paper. We show the original scenario in the first block. In the second block, we show simulated scene variations with an actor cutting into the SDV's lane, along with inserted traffic barriers, resulting in a completely new scenario with generated video data under multiple lighting conditions. In the third block, we show another example where we add inserted barriers and replace all the scene actors with a completely new set of actors reconstructed from another scene. The actors are seamlessly inserted into the scenario with the new target lighting. Please refer to the project page for video examples.
Figure A23: Additional controllable camera simulation examples.
E.6 Additional Qualitative Results on nuScenes
We further showcase LightSim's ability to generalize to driving scenes in nuScenes. We provide more qualitative scene relighting results in Fig. A24 and Fig. A25. Specifically, we select ten diverse scenarios that involve traffic participants such as vehicles, pedestrians and construction items. The sequence IDs are 011, 135, 154, 158, 159, 273, 274, 355, 544, 763. As described in Sec. C.3, we conduct neural scene reconstruction and lighting estimation (pre-trained on PandaSet) to build the lighting-aware digital twins. Then, we apply the neural deferred rendering model pre-trained on PandaSet to obtain the relit images. Although the nuScenes sensor data are much more sparse compared to PandaSet (32-beam LiDAR, 2Hz sampling rate), LightSim still produces reasonable scene relighting results, indicating good generalization and robustness. Please refer to the project page for video examples.
Occasionally, we observe noticeable black holes (e.g., on log 355 and 763) in the relit images. This is because the reconstructed meshes are low-quality (non-watertight ground, broken geometry) due to sparse LiDAR supervision and mis-calibration. While the neural deferred rendering module is designed to mitigate this issue, it cannot handle large geometry errors perfectly. Stronger smoothness regularization during the neural scene reconstruction step can potentially improve the model's performance.
Figure A24: Generalization to nuScenes (Part 1).
Figure A25: Generalization to nuScenes (Part 2).
F Limitations and Future Works
While LightSim can simulate diverse outdoor lighting conditions, there are several areas where it could benefit from further improvements. First, LightSim cannot seamlessly remove shadows, as shown in Fig. A26, particularly in bright, sunny conditions where the original images exhibit distinct cast shadows. This is because the shadows are mostly baked during neural scene reconstruction (see view-independent reconstruction in Fig. A27), producing flawed synthetic data that confuses the neural deferred rendering module. Moreover, we specify fixed materials [10] and predict sky domes that are not ideal for different urban scenes and may cause real-sim discrepancies as shown in Fig. 9. Those issues can potentially be addressed by better intrinsic decomposition with priors and joint material/lighting learning [83, 64]. More discussions for inverse rendering on urban scenes are provided in Sec D.
Second, LightSim uses an HDR sky dome to model the major light sources for outdoor daytime scenes. Therefore, LightSim cannot handle nighttime local lighting sources such as street lights, traffic lights, and vehicle lights. A potential solution is to leverage semantic knowledge and create local lighting sources (e.g., point/area lights [56] or volumetric local lighting [82]). Moreover, our experiments also focus on camera simulation for perception models, and we may investigate the performance of downstream planning tasks in future works. Lastly, our current system implementation relies on the Blender Cycles rendering engine [7], which is slow to render complex lighting effects. Faster rendering techniques can be incorporated to further enhance the efficiency of LightSim [69, 56].
Apart from the further method improvements mentioned above, it is important to collect extensive data from real-world urban scenes under diverse lighting conditions (e.g., repeating the same driving route under varying lighting conditions). Such data collection aids in minimizing the ambiguity inherent in intrinsic decomposition. Moreover, it paves the way for multi-log training with authentic data by providing a larger set of real-real pairs for lighting training.
Figure A26: Failure cases with strong directional lighting. The neural deferred rendering network cannot fully remove the baked shadows (top row) and other sensor effects (e.g., lens flare - bottom row). We use green and red arrows to highlight areas where LightSim performs well and not well.
Figure A27: View-independent reconstruction results for LightSim. Shadows are baked at this stage, which are then mitigated by neural deferred rendering. The relighting failure cases for the last example are shown in Fig. A26.
In Fig. A26, we highlight two examples of LightSim applied to scenes with strong directional lighting and high sun intensity. Each row shows the shadow editing/relighting results under four different sun angles of the target environment map. In the top row, LightSim cannot fully remove source shadows
in bright and sunny conditions due to the baked shadows in the view-independent reconstruction. Moreover, due to inaccurate HDR peak intensity estimation, the brightness of cast shadows cannot match the original images well. In the bottom row, we depict a source image with high sun intensity and glare relit to a new target lighting. It is challenging to remove the sun glare and alter the overexposed regions in this setting, but we can still apply some relighting effects to the cars and buildings in the scene (see arrows).
In this project, we ran the experiments primarily on NVIDIA Tesla T4s provided by Amazon Web Services (AWS). For prototype development and small-scale experiments, we used local workstations with RTX A5000s. Overall, this work used approximately 8,000 GPU hours (a rough estimation based on internal GPU usage reports), of which 3,000 were used for the final experiments and the rest for exploration and concept verification during the early stages of the research project. We provide a rough estimation of GPU hours used for the final experiments in Table A7, where we convert one A5000 hour to two T4 hours approximately.
Experiment	T4 Hours	Comments
Table 1 (perceptual quality validation)	1850	LightSim (100), NeRF-OSR (1500), EPE (150)
Table 2 (downstream training)	150	6× models, each takes 25 GPU hours
Table A4 (lighting estimation)	40	15h NLFE, 25h LightSim
Fig. 5 & Table A2 (ablations)	400	50h each model
Fig. 8 (nuScenes)	40	20h for digital twins, 20h for relighting
Others (data generation & demos)	510	500h lighting data generation + 10h demos
H Licenses of Assets
We summarize the licenses and terms of use for all assets (datasets, software, code, pre-trained models) in Table A8.
Table A7: Summary of GPU hours used for the final experiments.
Assets	License	URL
Blender 3.5.0 [7]	GNU General Public License (GPL)	https://www.blender.org/
HDRMaps [24]	Royalty-Free9	https://hdrmaps.com/
HoliCity [100]	Non-commercial purpose10	https://holicity.io/
TurboSquid 3D Models	Royalty-Free11	https://www.turbosquid.com12
PandaSet [87]	CC BY 4.013	https://scale.com/open-av-datasets/pandaset
nuScenes [11]	Non-commercial (CC BY-NC-SA 4.0)14	https://www.nuscenes.org/
SOLDNet [74]	Apache License 2.0	https://github.com/ChemJeff/SOLD-Net/
EPE [61]	MIT License	https://github.com/isl-org/PhotorealismEnhancement
Self-OSR [61]	Apache License 2.0	https://github.com/YeeU/relightingNet
NeRF-OSR [64]	Non-commercial purpose15	https://github.com/r00tman/NeRF-OSR
Color Transfer [60]	MIT License	https://github.com/jrosebr1/color_transfer
Table A8: Summary of the licenses of assets.
LightSim offers enhancements in camera-based robotic perception, applicable to various domains such as self-driving vehicles. Its ability to generate controllable camera simulation videos (e.g., actor insertion, removal, modification, and rendering from new viewpoints) and adapt to varying outdoor lighting conditions can potentially improve the reliability and safety of intelligent robots for a broad range of environmental conditions. Additionally, LightSim's capacity to create lighting-aware digital twins can improve realism in digital entertainment applications such as augmented reality or virtual reality. However, as with any technology, the responsible use of LightSim is important. Privacy concerns may arise when creating digital twins of real-world locations. We also caution that our system might produce unstable performance or unintended consequences under different datasets, especially when the sensory data are very sparse and noisy.