SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This appendix details our method, implementation, experimental designs, additional quantitative and qualitative results, limitations, utilized resources, and broader implications. We first detail how we build the lighting-aware digital twins from real-world data (Sec. A.1), then show the network architecture and training details for neural lighting simulation (Sec. A.2). In Sec. B, we provide details on baseline implementations and how we adapt them to our scene-relighting setting. Next, we provide the experimental details for perceptual quality evaluation and downstream detection training in Sec. C. We then report perception quality evaluation at a larger scale with more qualitative comparison with baselines (Sec. E.1) and detailed detection metrics for detection training (Sec. E.3). We demonstrate that our lighting estimation module yields more accurate sun prediction (Sec. E.4) and show additional scene relighting, shadow editing and controllable camera simulation results (Sec. E.5). Finally, we discuss limitations and future work (Sec. F), licenses of assets (Sec. H), computational resources used in this work (Sec. G), and broader impact (Sec. I).
A LightSim Implementation Details
A.1 Building Lighting-Aware Digital Twins of the Real-World
Neural Scene Reconstruction: We first perform neural rendering to reconstruct the driving scene using both front-facing camera images and  $360^{\circ}$  spinning LiDAR point clouds. We use a modified version of UniSim [89], a neural sensor simulator, to perform neural rendering to learn the asset representations. Unisim [89] incorporates the LiDAR information to perform efficient ray-marching and adopts a signed distance function (SDF) representation to accurately model the scene geometry. To ensure a smooth zero level set, the SDF representation is regularized using Eikonal loss [22] and occupancy loss. To capture the view-independent base color  $\mathbf{k_d}$ , we make adjustments to the appearance MLP by taking only grid features as input and omitting the view direction term; all other configurations remain consistent with [89].
After learning the neural representation, we employ Marching Cubes [47] to extract the mesh from the learned SDF volume. Finally, we obtain the vertex base color  $\mathbf{k_d}$  by querying the appearance head at the vertex location. We adopt base Blender PBR materials [10] for simplicity. Specifically, we use a Principled BSDF with vertex color as the base color. We set materials for dynamic assets and background separately.
Figure A10: Network architecture of the neural lighting estimation module. Red and blue boxes denote the sky encoders and HDR sky dome decoder respectively.
Neural Lighting Estimation: We leverage multi-camera data and our extracted geometry to estimate an incomplete panorama LDR image that captures scene context and the available observations of the sky. We then apply an inpainting network to fill in missing sky regions. Finally, we leverage a sky dome estimator network that lifts the LDR panorama image to an HDR sky dome and fuses it with GPS data to obtain accurate sun direction and intensity. To inpaint the panorama image from the stitched multi-camera image, we use the code of the inpainting network DeepFillv2. The network is
<https://github.com/zhaoyuzhi/deepfillv2>
unchanged from its implementation on Github, and it is trained using the hinge loss. Holicity's LDR panoramas are used for training. Each panorama in the training set is first masked using an intrinsic mask - a random mask of observed pixels from a PandaSet log. Distortions are then applied to the mask, including random scaling and the addition of noise. The masked panorama is then fed into the network and supervised with the unedited Holicity panorama.
Sky dome estimator architecture: After receiving the full LDR output from the inpainting network, the sky dome estimator network converts the incomplete LDR panorama to HDR. This is achieved by employing GPS and time of day to improve the accuracy of sun direction estimation. To produce an LDR sky dome, only the top half of the panorama is used, as the bottom half does not contribute to global lighting from the sky. The network uses an encoder-decoder architecture, as shown in Fig. A10. The input consists of the LR panorama and the positional encoding map, which is an equirectangular projection that stores a unit vector pointing towards each pixel direction. The positional encoding is concatenated channel-wise with the LDR panorama. Three separate encoders with the same architecture are utilized to estimate the peak direction  $\mathbf{f}_{\mathrm{dir}}$  and intensity of the sun  $\mathbf{f}_{\mathrm{int}}$  as well as the latent of the sky  $\mathbf{z}_{\mathrm{sky}} \in \mathbb{R}^{64}$ . To encode the predicted sun peak intensity  $\mathbf{f}_{\mathrm{int}}$  and peak direction  $\mathbf{f}_{\mathrm{dir}}$ , five intermediate representations with the same size as the input LDR are explicitly encoded. The sky decoder is an U-Net which takes the encoded predicted sun peak intensity  $\mathbf{f}_{\mathrm{int}}$  and peak direction  $\mathbf{f}_{\mathrm{dir}}$  and fuses them with the sky latent vector to produce the HDR sky dome.
Training details: We train the sky dome estimator using 400 HDRs sourced from HDRMaps [24]. Accurately predicting the peak intensity  $\mathbf{f}_{\mathrm{int}}$  and direction  $\mathbf{f}_{\mathrm{dir}}$  is of utmost importance for achieving realistic rendering. Nevertheless, due to the inherently ill-posed nature of this problem, it is challenging to predict these parameters precisely, especially for cloudy skies. Therefore, we propose a dual-encoder architecture, with one encoder trained on HDRs with a clearly visible sun and another trained on all HDR images to capture peak intensity and sky latent more robustly. We empirically find that this works better than using a single encoder trained on all HDR images. The model takes around 12 hours to train with a single RTX A5000.
A.2 Neural Lighting Simulation of Dynamic Urban Scenes
Building augmented reality representation: Given our compositional neural radiance fields, we can remove actors (Fig. 7), add actors (Fig. A23), modify actor trajectories (Fig. 1 and Fig. A23), and perform neural rendering on the modified scene to generate new camera video in a spatially- and temporally-consistent manner. Using our estimated lighting representation, we can also insert synthetic assets such as CAD models [76] into the scenes and composite them with real images in a 3D- and lighting-aware manner (Fig. 1, 7, A23). These scene edits lead to an "augmented reality representation"  $\mathcal{M}'$ ,  $\mathbf{E}^{\mathrm{src}}$  and source image  $\mathbf{I}_{\mathrm{src}}'$ . We further explain the details of creating augmented reality representations for all the examples we have created in this paper. In Fig. 1, we insert a new CAD vehicle, barriers, and traffic cones and modify the trajectory of the white car. In Fig. 7, we insert three new CAD vehicles (black, white, gray) and construction items and remove all existing dynamic vehicles. In Fig. A23 (top example), we modify the trajectory of the original white vehicle and insert construction items. In Fig. A23 (bottom example), we remove all existing dynamic actors (pedestrians and vehicles), insert three dynamic actors from another sequence 016 with new trajectories, and insert construction items.
Neural deferred rendering architecture: The neural deferred rendering architecture is depicted in Fig. A11. It is an image-to-image network adapted from U-Net [63] that consists of four components: image encoder, lighting encoder, latent fuser, and rendering decoder. The inputs to the image encoder comprise a source RGB image  $\mathbf{I}_{\mathrm{src}}$ , rendering buffers  $\mathbf{I}_{\mathrm{buffer}}$  (ambient occlusion, normal, depth, and position), and source/target shadow ratio maps  $\{\mathbf{S}_{\mathrm{src}}, \mathbf{S}_{\mathrm{tgt}}\}$ ; each of these components has 3 channels. In the latent fuser, the output of the image encoder is run through a 2D convolution layer, then a linear layer that compresses the input into a latent vector. The image latent and two lighting latent vectors (source and target) are concatenated and upsampled. Finally, the rendering decoder upsamples the fused latent vectors and produces the final relit image  $\hat{\mathbf{I}}^{\mathrm{tgt}} \in \mathbb{R}^{H \times W \times 3}$ .
Learning details: We train the relighting network using a weighted sum of the photometric loss  $(\mathcal{L}_{\mathrm{color}})$ , perceptual loss  $(\mathcal{L}_{\mathrm{lpips}})$ , and edge-based content preserving loss  $(\mathcal{L}_{\mathrm{edge}})$ , as described in
Figure A11: Network architecture of the neural deferred rendering module. Green, red, orange, and blue boxes denote the image encoder, lighting encoder, latent fuser and rendering decoder, respectively.
section 4 of the main paper:  $\mathcal{L}_{\mathrm{relight}} = \frac{1}{N}\sum_{i = 1}^{N}\left(\mathcal{L}_{\mathrm{color}} + \lambda_{\mathrm{lpips}}\mathcal{L}_{\mathrm{lpips}} + \lambda_{\mathrm{edge}}\mathcal{L}_{\mathrm{edge}}\right)$ , where  $\lambda_{\mathrm{lpips}}$  is set to 1 and  $\lambda_{\mathrm{edge}}$  to 400. Our model is trained on pairs of PandaSet [87] scenes, lit with  $N_{\mathrm{HDR}} = 20$  HDRs: ten estimated from PandaSet scenes using our neural lighting estimation module, and ten obtained from the HDRMaps dataset [24]. We apply random intensity and offset changes to the HDRs as data augmentation.
B Implementation Details for Baselines
For each approach, we evaluate on 1,380 images, applying 23 lighting variations to 15 PandaSet scenes with 4 frames per scene, and report FID/KID scores. To test the diversity and controllability of each method, 11 of the target lighting variations are estimated from real PandaSet data, while the remaining 12 are outdoor HDRs sourced from HDRMaps [24].
B.1 Self-supervised Outdoor Scene Relighting (Self-OSR)
Self-OSR [94] is an image-based inverse-rendering method that utilizes InverseRenderNet [95] to decompose an image into albedo, normal, shadow, and lighting. Subsequently, it employs two generative adversarial networks (GANs) for neural rendering and sky hallucination, based on the predicted lighting. We use the official pre-trained models² and perform inference under novel lighting conditions. If the target lighting originates from PandaSet logs, the model is first applied to a single front-camera image (first frame) to recover the lighting parameters  $\in \mathbb{R}^{9\times 3}$ , using an order-2 spherical harmonics model. For outdoor HDRs sourced from HDRMaps, the HDR environment maps are converted into spherical harmonics. The sky masks for PandaSet images are estimated using PSPNet [99].
B.2 NeRF for Outdoor Scene Relighting (NeRF-OSR)
We use the official code from NeRF-OSR<sup>3</sup> and make several modifications to improve its performance on the PandaSet dataset. Since the self-driving scenes in the PandaSet dataset are usually larger than the front-facing scenarios on the NeRF-OSR dataset, more space needs to be sampled, which presents a challenge. To overcome this, we use LiDAR points as a guide to sample the points
$^{2}$ https://github.com/YeeU/relightingNet
<sup>3</sup>https://github.com/rO0tman/NeRF-OSR
for camera rays by aggregating LiDaR points and creating a surfel mesh. Then, we sample only the points close to the surface  $\pm 50cm$  for each camera ray. By doing so, we significantly reduce the number of sampled points to eight in the coarse and fine stages. Without these modifications, NeRF-OSR demonstrates slower convergence rates and struggles to learn reasonable geometry for relighting. Since our inference HDRs (from PandaSet or HDRMaps) differ greatly from NeRF-OSR environment maps (indicating inaccurate scene decomposition by the model), directly applying the environment maps to the scene leads to significant artifacts. Therefore, we retrieve the nearest HDR in the NeRF-OSR dataset during inference for better perceptual quality. For training, we follow the original code base for all other settings. Training one log on NeRF-OSR usually takes 15 hours on 4 T4 GPUs.
B.3 Color Transfer
Color Transfer [60] uses image statistics in  $(L^{*},a^{*},b^{*})$  space to adjust the color appearance between images. We adopt the public Python implementation<sup>4</sup> for our experiments. If the target lighting originates from PandaSet logs, we use the single front-camera image as the target image for color transfer. For outdoor HDRs sourced from HDRMaps, we use the rendered synthetic images (using LightSim digital twins)  $\mathbf{I}_{\mathrm{render}|\mathbf{E}^{\mathrm{tgt}}}$  as the target image, as it produces much better performance compared to transferring with the LDR environment map. The latter results in worse performance since the source environment map and target limited-FoV images are in different content spaces.
B.4 Enhancing Photorealism Enhancement (EPE)
EPE [61] was designed to enhance the realism of synthetic images (e.g., GTA-5 [62] to CityScapes [14]) using intermediate rendering buffers and GANs. We adapt EPE to handle lighting simulation with our established lighting-aware digital twins. Specifically, EPE uses the rendered image  $\mathbf{I}_{\mathrm{render}|\mathbf{E}^{\mathrm{tgt}}}$  and rendering buffers  $\mathbf{I}_{\mathrm{buffer}}$  generated by our digital twins to predict the relit image. Note that EPE uses the same training data as LightSim, with 20 HDR variations. It also takes all PandaSet front-camera images (103 logs) as the referenced real data. We adopt the official implementation<sup>5</sup> and follow the instructions to compute robust label maps, crop the images, match the crops (5 nearest neighbours) and obtain 459k sim-real pairs. We train the EPE model until convergence for 60M iterations on one single RTX A5000 for around six days.
C LightSim Experiment Details
C.1 Perceptual Quality Evaluation
Following [61, 12, 89], we report Fréchet Inception Distance [25] (FID) and Kernel Inception Distance [25] (KID) to measure perceptual quality since ground truth data are not available. Due to NeRF-OSR's large computational cost, we select 15 sequences {001, 002, 011, 021, 023, 024, 027, 028, 029, 030, 032, 033, 035, 040, 053} for quantitative evaluation in the main paper. We also provide Table A3 for larger-scale evaluation (NeRF-OSR excluded), in which 47 sequences (all city logs in PandaSet excluding the night logs and 004 where the SDV is stationary) are used for evaluation. The 47 sequences are {001, 002, 003, 004, 005, 006, 008, 011, 012, 013, 014, 015, 016, 017, 018, 019, 020, 021, 023, 024, 027, 028, 029, 030, 032, 033, 034, 035, 037, 038, 039, 040, 041, 042, 043, 044, 045, 046, 047, 048, 050, 051, 052, 053, 054, 055, 056, 139}.
For each sequence, we select four frames  $\{6, 12, 18, 24\}$  and simulate 23 lighting variations (see Fig. A12). Note that in 23 lighting variations, 20 HDRs (10 estimated HDRs from PandaSet, 10 real HDRs sourced from HDRMaps) are used for data generation to train the LightSim models, while 3 HDRs are unseen and only used during inference. Unless stated otherwise, we use all 8240 real PandaSet images as the reference dataset.
4https://github.com/jrosebr1/color_transfer
<sup>5</sup>https://github.com/is1-rg/PhotorealismEnhancement
Figure A12: 23 HDR sky domes used for perceptual quality evaluation.
C.2 Downstream Perception Training
To investigate if realistic lighting simulation can improve the performance of downstream perception tasks under unseen lighting conditions, we conduct experiments on PandaSet using the SoTA camera-based 3D vehicle detection model BEVFormer [44]. We use train on 68 snippets collected in the city and evaluate on 35 snippets in a suburban area, since these two collections are independent and exposed to different lighting conditions. Specifically, the sequences {080, 084, 085, 086, 088, 089, 090, 091, 092, 093, 094, 095, 097, 098, 099, 100, 101, 102, 103, 104, 105, 106, 109, 110, 112, 113, 115, 116, 117, 119, 120, 122, 123, 124, 158} are selected for the validation set, and the remaining sequences are used for training. For the experiments, we use all 80 frames for training and evaluation. We report the average precision (AP) at different IoU thresholds: 0.1, 0.3, and 0.5. The mean average precision (mAP) is calculated as  $\mathrm{mAP} = (\mathrm{AP}@0.1 + \mathrm{AP}@0.3 + \mathrm{AP}@0.5) / 3.0$ .
As shown in Table 2, the integration of LightSim synthetic simulation significantly enhances the performance of monocular detection compared to training with other basic augmentation methods. Further exploration on sufficiently utilizing the simulated data, such as actor behavior simulation or actor insertion, is left to future work.
BEVFormer Implementation Details: We use the official repository<sup>6</sup> for training and evaluating our model on PandaSet. We focus on single-frame monocular vehicle detection using the front camera, disregarding actors outside the camera's field of view. The models are trained within vehicle frames using the FLU convention ( $x$ : forward,  $y$ : left,  $z$ : up), with the region of interest defined as  $x \in [0,80\mathrm{m}], y \in [-40\mathrm{m},40\mathrm{m}], z \in [-2\mathrm{m},6\mathrm{m}]$ . Given memory constraints, we adopt the BEVFormer-small architecture<sup>7</sup> with a batch size of two per GPU. Models were trained for five epochs using the AdamW optimizer [48], coupled with the cosine learning rate schedule.<sup>8</sup> Training each model took approximately six hours on  $2 \times$  RTX A5000 GPUs. We report the best validation performance across all data augmentation approaches, as models can begin to overfit in the final training stage.
C.3 Generalization on nuScenes
To evaluate the generalizability of our model, we train the model on PandaSet [87] and evaluate the pre-trained model on nuScenes [11]. The nuScenes [11] dataset contains 1000 driving scenes collected in Boston and Singapore, each with a duration of  $\approx 20$  seconds ( $\approx 40$  frames, sampled at
$^{6}$ https://github.com/fundamentalvision/BEVFormer  
 $^{7}$ https://github.com/fundamentalvision/BEVFormer/blob/master/projects/configs/bevformer/bevformer_small.py  
 $^{8}$ https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingLR.html
2 Hz) acquired by six cameras (Basler acA1600-60gc), one spinning LiDAR (Velodyne HDL32E), and five long-range RADAR (Continental ARS 408-21) sensors. We curate 10 urban scenes from nuScenes [11] characterized by dense traffic and interesting scenarios.
We incorporate front-facing camera and spinning LiDAR and run the neural scene reconstruction module (Section 3.1) to extract the manipulable digital twins for each scene. Then, we utilize the neural lighting estimation module (Section 3.2) to recover the HDR sky domes. This enables us to generate new scenarios and produce the rendering buffers  $\mathbf{I}_{\mathrm{buffer}}$  (Eqn. 4 in the main paper) for scene relighting.
D Additional Discussions
Challenges of inverse rendering on urban scenes: LightSim assumes several simplifications when building lighting-aware digital twins, including approximate diffuse-only reconstruction, separate lighting prediction, and fixed base materials. Those results in imperfect intrinsic decomposition and sim/real discrepancies. Recent concurrent works such as FEGR [83] and UrbanIR [45] make steps towards better decomposition, but it is still a challenging open problem to recover perfect decomposition of materials and light sources for large urban scenes. As shown in Fig. A13 (top right), the recovered materials bear little resemblance to semantic regions in the original scene. These recent relighting works [61, 64, 83, 45] also have shadows baked into the recovered albedo (Fig. A13 left).
We remark that our novelty lies in leveraging neural deferred rendering to overcome the limitations of purely physically-based rendering when the decomposition is imperfect. This allows us to generate better relighting results than prior works that have imperfect decompositions. It is an exciting future direction to incorporate better intrinsic decomposition along with neural deferred rendering for improved relighting.
Prediction-based vs. optimization-based lighting: We explain our design choices in the following two aspects. (a) We use a feed-forward network for lighting estimation, which is more efficient and can benefit from learning on a larger dataset. In contrast, the optimization paradigm is more expensive, requiring per-scene optimization, but has the potential to recover more accurate scene lighting from partial observations. (b) The ill-posed nature of lighting estimation and extreme intensity range make inverse rendering challenging for outdoor scenes [83]. Optimization of the environment map requires a differentiable renderer and high-quality geometry/material to achieve good results. The existing/concurrent state-of-the-art works [64, 83] cannot solve the problem accurately, as shown in Fig. A13 bottom right.
UrbanIR Albedo
NeRF-OSR (top) & Self-OSR (bottom) Albedo
Figure A13: Challenges of inverse rendering on urban scenes for existing works.
NeRF-OSR Env. Map
FEGR Env. Map
Temporal consistency for neural deferred shading: While we do not guarantee temporal consistency, LightSim can produce temporally-consistent lighting simulation videos in most cases. We believe this temporal consistency comes from temporally- and multi-view-consistent inputs during inference (real image, G-buffers), as well as our combination of simulation and real paired relighting data during training. Explicitly enforcing temporal consistency is an interesting direction for future work.
Random shadowed triangles when relighting due to mesh artifacts: We noticed that random shadowed triangles are common in the  $\mathbf{I}_{\mathrm{render}}|_{\mathbf{E}^{\mathrm{src}}}$  due to non-smooth extracted meshes. This is more obvious for the nuScenes dataset where the LiDAR is sparser (32-beam) and the capture frequency is lower (2Hz); for this dataset, we notice many holes and random shadowed triangles. However, thanks to our image-based neural deferred rendering pass trained with mixed sim-real data, our relighting network takes the original image and modifies the lighting, which removes many of those artifacts in the final relit images. We show two nuScenes examples in Fig. A14.
Figure A14: Random shadowed triangles are removed after neural deferred rendering.
E Additional Experiments and Analysis
We provide additional results and analysis for scene relighting, ablation studies, downstream training, and lighting estimation. We then showcase more simulation examples using LightSim.
E.1 Additional Perception Quality Evaluation
Due to the large computational cost of NeRF-OSR [64], we select 15 PandaSet sequences for perceptual quality evaluation in Table 1. Here, we supplement the evaluation at larger scale (47 sequences in total) in Table A3. LightSim achieves perceptual quality (FID and KID) on par with Color Transfer, while the latter approach only adjusts the color histogram and cannot simulate intricate lighting effects properly. Self-OSR and EPE suffer from noticeable artifacts, resulting in significantly worse perception quality and a larger sim-real domain gap.
Method	FID ↓	KID (×103) ↓
Self-OSR [94]	97.3	89.7 ± 11.5
Color Transfer [60]	50.1	18.0 ± 4.5
EPE [61]	79.6	50.0 ± 9.1
Ours	52.3	16.0 ± 4.6
Table A3: Additional perceptual quality evaluation on 47 sequences.
Figure A15: Qualitative comparison against SoTA approaches in scene relighting.
We also provide more qualitative comparisons against SoTA scene relighting approaches in Fig. A15. For each scene, we showcase two different lighting conditions with the inset target HDRs in the leftmost column. We show more scene relighting results of LightSim in Fig. A16 and Fig. A17. We also show results in Fig. A18, where we rotate the HDR skydome and render the shadows at different sun locations, demonstrating controllable outdoor illumination and realistic simulated results. Please refer to the project page for video examples.
Figure A16: Qualitative examples of scene relighting for LightSim (Part 1).