SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ava Pun $^{1,3*†}$  Gary Sun $^{1,3*†}$  Jingkang Wang $^{1,2*}$  Yun Chen $^{1,2}$  Ze Yang $^{1,2}$  Sivabalan Manivasagam $^{1,2}$  Wei-Chiu Ma $^{1,4}$  Raquel Urtasun $^{1,2}$
Waabi<sup>1</sup> University of Toronto<sup>2</sup> University of Waterloo<sup>3</sup> MIT<sup>4</sup>
Different outdoor illumination conditions drastically alter the appearance of urban scenes, and they can harm the performance of image-based robot perception systems if not seen during training. Camera simulation provides a cost-effective solution to create a large dataset of images captured under different lighting conditions. Towards this goal, we propose LightSim, a neural lighting camera simulation system that enables diverse, realistic, and controllable data generation. LightSim automatically builds lighting-aware digital twins at scale from collected raw sensor data and decomposes the scene into dynamic actors and static background with accurate geometry, appearance, and estimated scene lighting. These digital twins enable actor insertion, modification, removal, and rendering from a new viewpoint, all in a lighting-aware manner. LightSim then combines physically-based and learnable deferred rendering to perform realistic relighting of modified scenes, such as altering the sun location and modifying the shadows or changing the sun brightness, producing spatially- and temporally-consistent camera videos. Our experiments show that LightSim generates more realistic relighting results than prior work. Importantly, training perception models on data generated by LightSim can significantly improve their performance. Our project page is available at https://waabi.ai/lightsim/.
Humans can perceive their surroundings under different lighting conditions, such as identifying traffic participants while driving on a dimly lit road or under mild sun glare. Unfortunately, modern camera-based perception systems, such as those in self-driving vehicles (SDVs), are not as robust [52, 32]. They typically only perform well in the canonical setting they were trained in, and their performance drops significantly under different unseen scenarios, such as in low-light conditions [52, 59, 32]. To reduce distribution shift, we could collect data under various lighting conditions for each area in which we want to deploy the SDVs, generating a diverse dataset on which to train the perception system. Unfortunately, this is not scalable as it is too expensive and time-consuming.
Simulation is a cost-effective alternative for generating large-scale data with diverse lighting. To be effective, a simulator should be realistic, controllable, and diverse. Realistic simulation of camera data enables generalization to the real world. Controllable actor placement and lighting allow the simulator to generate the desired training scenarios. Diverse backgrounds, actor assets, and lighting conditions allow simulation to cover the full real-world distribution. While existing game-engine-based self-driving simulators such as CARLA [18, 68] are controllable, they provide a limited number of manually designed assets and lighting conditions. Perception systems trained on this data generalize poorly to the real world [26, 1, 61].
$\dagger$  Work done while a research intern at Waabi.
Figure 1: LightSim builds digital twins from large-scale data with lighting variations and generates high-fidelity simulation videos. Top: LightSim produces realistic scene relighting and shadow editing videos. Bottom: We generate a safety-critical scenario with two vehicles cutting in and perform lighting-aware camera simulation. See Appendix E and project page for more examples.
To build a more realistic, scalable, and diverse simulator, we instead propose reconstructing "digital twins" of the real world at scale from a moving platform equipped with LiDAR and cameras. By reconstructing the geometry and appearance of the actors and static backgrounds to create composable neural assets, as well as estimating the scene lighting, the digital twins can provide a rich asset library for simulation. Existing methods for building digital twins and data-driven simulators [2, 40, 89, 73, 57] bake the lighting into the scene, making simulation under new lighting conditions impossible. In contrast, we create lighting-aware digital twins, which enable actor insertion, removal, modification, and rendering from new viewpoints with accurate illumination, shadows, occlusion, and temporal consistency. Moreover, by estimating the scene lighting, we can use the digital twins for relighting.
Given a digital twin, we must relight the scene to a target novel lighting condition. This is, however, a challenging task. Prior image-based synthesis methods [4, 5] perform relighting via 2D style transfer techniques, but they typically lack human-interpretable controllability, are not temporally or spatially consistent, and can have artifacts. Inverse neural rendering methods [82, 49, 39, 83] aim to decompose the scene into geometry, materials and lighting, which allows for relighting through physically-based rendering [7, 30]. However, the ill-posed nature of lighting estimation and intrinsic decomposition makes it challenging to fully disentangle each aspect accurately, resulting in unrealistic relit images. Both approaches, while successful on synthetic scenes or outdoor landmarks [51] with dense data captured under many lighting conditions, have difficulty performing well on large outdoor scenes. This is primarily due to the scarcity of real-world scenes captured under different lighting conditions. Moreover, most prior works perform relighting on static scenes and have not demonstrated realistic relighting for dynamic scenes where both the actors and the camera viewpoint are changing, resulting in inter-object lighting effects that are challenging to simulate.
In this paper, we present LightSim, a novel lighting simulation system for urban driving scenes that generates diverse, controllable, and realistic camera data. To achieve diversity, LightSim reconstructs lighting-aware digital twins from real-world sensor data, creating a large library of assets and lighting environment maps for simulation. LightSim then leverages physically-based rendering to enable controllable simulation of the dynamic scene, allowing for arbitrary actor placement, SDV location, and novel lighting conditions. To improve realism, we further enhance the physically-based renderer
with an image-based neural deferred renderer to perform relighting, enabling data-driven learning to overcome geometry artifacts and ambiguity in the decomposition due to not having perfect knowledge of the scene and sensor configuration. To overcome the lack of real-world scenes captured under different lighting conditions, we train our neural deferred renderer on a mixture of real and synthetic data generated with physically-based rendering on our reconstructed digital twins.
We demonstrate the effectiveness of our approach on PandaSet [87], which contains more than 100 real-world self-driving scenes covering 20 unique kilometers. LightSim significantly outperforms the state of the art, producing high-quality photorealistic driving videos under a wide range of lighting conditions. We then showcase several capabilities of LightSim, such as its ability to create high-fidelity and lighting-aware actor insertion, scene relighting, and shadow editing (Fig. 1). We demonstrate that by training camera perception models with LightSim-generated data, we can achieve significant performance improvements, making the models more robust to lighting variation. We believe LightSim is an important step towards enhancing the safety of self-driving development and deployment.
Outdoor lighting estimation: As a first step for lighting simulation, lighting estimation aims to recover the  $360^{\circ}$  HDR light field from observed images for photo-realistic virtual object insertion [20, 28, 19, 27, 37, 96, 67, 71, 101, 82, 78, 74, 83]. Existing works generally use neural networks to predict various lighting representations, such as environment maps [71, 101, 78, 82, 74, 83], spherical lobes [8, 43], light probes [37], and sky models [28, 27, 96], from a single image. For outdoor scenes, handling the high dynamic range caused by the presence of the sun is a significant challenge. Moreover, due to the scarcity of real-world datasets and the ill-posed nature of lighting estimation [82, 74], it is challenging to precisely predict peak intensity and sun location from a single limited field-of-view low-dynamic range (LDR) image. To mitigate these issues, SOLDNet [101, 74] enhances the diversity of material and lighting conditions with synthetic data and introduces a disentangled global and local lighting latent representation to handle spatially-varying effects. NLFE [82] uses hybrid sky dome / light volume and introduces adversarial training to improve realism with differentiable actor insertion. In contrast, our work fully leverages the sensory data available in a self-driving setting (i.e., multi-camera images, LiDAR, and GPS) to accurately recover spatially-varying environment maps.
Inverse rendering with lighting: Another way to obtain lighting representations is through joint geometry, material, and lighting optimization with inverse rendering [67, 42, 84, 55, 79, 23, 49, 91, 64, 39, 41, 92, 83]. However, since optimization is conducted on a single scene and material decomposition is the primary goal, the optimized lighting representations are usually not generalizable and do not work well for relighting [67, 55, 23]. Inspired by the success of NeRF in high-fidelity 3D reconstruction, some works use volume rendering to recover material and lighting given image collections under different lighting conditions [64, 49, 39], but they are limited in realism and cannot generalize to unseen scenes. Most recently, FEGR [83], independent and concurrent to our work, proposes a hybrid framework to recover scene geometry, material and HDR lighting of urban scenes from posed camera images. It demonstrates realistic lighting simulation (actor insertion, shadow editing, and scene relighting) for static scenes. However, due to imperfect geometry and material/lighting decomposition, relighting in driving scenes introduces several noticeable artifacts, including unrealistic scene color, blurry rendering results that miss high-frequency details, obvious mesh boundaries on trees and buildings, and unnatural sky regions when using other HDR maps. In contrast, LightSim learns on many driving scenes and performs photorealistic enhanced deferred shading to produce more realistic relighting videos for dynamic scenes.
Camera simulation for robotics: There is extensive work on developing simulated environments for safer and faster robot development [33, 86, 53, 17, 75, 38, 15, 29, 17, 35, 9, 13]. Two major lines of work in sensor simulation for self-driving include graphics-based [18, 68] and data-driven simulation [80, 66, 50, 88]. Graphics-based simulators like CARLA [18] and AirSim [68] are fast and controllable, but they face limitations in scaling and diversity due to costly manual efforts in asset building and scenario creation. Moreover, these approaches can generate unrealistic sensor data that have a large domain gap for autonomy [26, 85]. Data-driven approaches leverage computer vision techniques and real-world data to build simulators for self driving [34, 3, 2, 81, 40]. Unfortunately, existing works tend to fall short of realism, struggle with visual artifacts and domain gap [34, 3, 2, 81,
Figure 2: Overview of LightSim. Given sensor observations of the scene, we first perform neural scene reconstruction and lighting estimation to build lighting-aware digital twins (left). Given a target lighting, we then perform both physically-based and neural deferred rendering to simulate realistic driving videos under diverse lighting conditions (right).
40], and lack comprehensive control in synthesizing novel views [12, 82, 79, 90, 72]. Most recent works [57, 36, 77, 89, 46] use neural radiance fields to build digital twins and represent background scenes and agents as MLPs, enabling photorealistic rendering and controllable simulation across a single snippet. However, these works bake lighting and shadows into the radiance field and therefore cannot conduct actor insertion under various lighting conditions or scene-level lighting simulation. In contrast, LightSim builds lighting-aware digital twins for more controllable camera simulation.
3 Building Lighting-Aware Digital Twins of the Real World
The goal of this paper is to create a diverse, controllable, and realistic simulator that can generate camera data of scenes at scale under diverse lighting conditions. Towards this goal, LightSim first reconstructs lighting-aware digital twins from camera and LiDAR data collected by a moving platform. The digital twin comprises the geometry and appearance of the static background and dynamic actors obtained through neural rendering (Sec. 3.1), as well as the estimated scene lighting (Sec. 3.2). We carefully build this representation to allow full controllability of the scene, including modifying actor placement or SDV position, adding and removing actors in a lighting-aware manner for accurate shadows and occlusion, and modifying lighting conditions, such as changing the sun's location or intensity. In Sec. 3.3, we then describe how we perform learning on sensor data to build the digital twin and estimate lighting. In Sec. 4, we describe how we perform realistic scene relighting with the digital twin to generate the final temporally consistent video.
3.1 Neural Scene Reconstruction
Inspired by [89, 77, 57], we learn the scene geometry and base texture via neural fields. We design our neural field  $F: \mathbf{x} \mapsto (s, \mathbf{k}_d)$  to map a 3D location  $\mathbf{x}$  to a signed distance  $s \in \mathbb{R}$  and view-independent diffuse color  $\mathbf{k}_d \in \mathbb{R}^3$ . We decompose the driving scene into a static background  $\mathcal{B}$  and a set of dynamic actors  $\{\mathcal{A}_i\}_{i=1}^M$  and map multi-resolution spatial feature grids [54] using two MLP networks: one for the static scene and one for the dynamic actors. This compositional representation allows for 3D-aware actor insertion, removal, or manipulation within the background. From our learned neural field, we use marching cubes [47] and quadric mesh decimation [21] to extract simplified textured meshes  $\mathcal{M}$  for the scene. For simplicity, we specify base materials [10] for all the assets and defer material learning to future work. Please see Appendix A.1 for details. Given the desired lighting conditions, we can render our reconstructed scene in a physically-based renderer to model object-light interactions.
3.2 Neural Lighting Estimation
In addition to extracting geometry and appearance, we estimate the scene lighting (Fig. 3, left). We use a high-dynamic-range (HDR) panoramic sky dome  $\mathbf{E}$  to represent the light from the sun and the sky. This representation well models the major light sources of outdoor daytime scenes and is compatible with rendering engines [7, 30]. Unfortunately, estimating the HDR sky dome from sensor data is challenging, as most cameras on SDVs have limited field-of-view (FoV) and do not capture the full sky. Additionally, camera data are typically stored with low dynamic range (LDR) in self-driving datasets, i.e., intensities are represented with 8-bits. To overcome these challenges, we first leverage multi-camera data and our extracted geometry to estimate an incomplete panorama LDR image that
Figure 3: LightSim modules. Left: neural lighting estimation. Right: neural deferred rendering.
captures scene context and available sky observations. We then apply an inpainting network to fill in missing sky regions. Then, we utilize a sky dome estimator network that lifts the LDR panorama image to an HDR sky dome and fuses it with GPS data to obtain accurate sun direction and intensity. Unlike prior works that estimate scene lighting from a single limited-FoV LDR image [82, 94, 74], our work leverages multi-sensor data for more accurate estimation. We now describe these steps (as shown in Fig. 3, left) in detail.
Panorama reconstruction: Given  $K$  images  $\mathbf{I} = \{\mathbf{I}_i\}_{i=1}^K$  captured by multiple cameras triggered close in time and their corresponding camera poses  $\mathbf{P} = \{\mathbf{P}_i\}_{i=1}^K$ , we first render the corresponding depth maps  $\mathbf{D} = \{\mathbf{D}_i\}_{i=1}^K$  from extracted geometry  $\mathcal{M}$ :  $\mathbf{D}_i = \psi(\mathcal{M}, \mathbf{P}_i)$ , where  $\psi$  is the depth rendering function and  $\mathbf{P}_i \in \mathbb{R}^{3 \times 4}$  is the camera projection matrix (a composition of camera intrinsics and extrinsics). We set the depth values for the sky region to infinity. For each camera pixel  $(u', v')$ , we use the rendered depth and projection matrix to estimate 3D world coordinates, then apply an equirectangular projection  $E$  to determine its intensity contribution to panorama pixel  $(u, v)$ , resulting in  $\mathbf{I}_{\mathrm{pano}}$ :
$$
\mathbf {I} _ {\text {p a n o}} = \Theta (\mathbf {I}, \mathbf {D}, \mathbf {P}) = E (\pi^ {- 1} (\mathbf {I}, \mathbf {D}, \mathbf {P})) ， \tag {1}
$$
where  $\Theta$  is the pixel-wise transformation that maps the RGB of limited field-of-view (FoV) images  $\mathbf{I}$  at coordinate  $(u', v')$  to the  $(u, v)$  pixel of the panorama. For areas with overlap, we average all source pixels that are projected to the same panorama  $(u, v)$ . In the self-driving domain, the stitched panorama  $\mathbf{I}_{\mathrm{pano}}$  usually covers a  $360^{\circ}$  horizontal FoV, but the vertical FoV is limited and cannot fully cover the sky region. Therefore, we leverage an inpainting network [93] to complete  $\mathbf{I}_{\mathrm{pano}}$ , creating a full-coverage  $(360^{\circ} \times 180^{\circ})$  panorama image.
Generating HDR sky domes: For realistic rendering, an HDR sky dome should have accurate sun placement and intensity, as well as sky appearance. Following [96, 82], we learn an encoder-decoder sky dome estimator network that lifts the incomplete LDR panorama to HDR, while also leveraging GPS and time of day for more accurate sun direction. The encoder first maps the LDR panorama image to a low-dimensional representation to capture the key attributes of the sky dome, including a sky appearance latent  $\mathbf{z}_{\mathrm{sky}} \in \mathbb{R}^d$ , peak sun intensity  $\mathbf{f}_{\mathrm{int}}$ , and sun direction  $\mathbf{f}_{\mathrm{dir}}$ . By explicitly encoding sun intensity and direction, we enable more human-interpretable control of the lighting conditions and more accurate lighting estimation. The decoder network processes this representation and outputs the HDR sky dome  $\mathbf{E}$  as follows:
$$
\mathbf {E} = \operatorname {H D R d e c o d e r} \left(\mathbf {z} _ {\text {s k y}}, \left[ \mathbf {f} _ {\text {i n t}}, \mathbf {f} _ {\text {d i r}} \right]\right), \text {w h e r e} \mathbf {z} _ {\text {s k y}}, \mathbf {f} _ {\text {i n t}}, \mathbf {f} _ {\text {d i r}} = \operatorname {L D R e n c o d e r} (\mathbf {L}). \tag {2}
$$
When GPS and time of day are available, we replace the encoder-estimated direction with the GPS-derived sun direction for more precise sun placement. Please see Appendix A.1 for details.
3.3 Learning
We now describe the learning process to extract static scenes and dynamic actor textured meshes, as well as training the inpainting network and sky dome estimator.
Optimizing neural urban scenes: We jointly optimize feature grids and MLP headers  $\{f_s, f_{\mathbf{k}_d}\}$  to reconstruct the observed sensor data via volume rendering. This includes a photometric loss on the rendered image, a depth loss on the rendered LiDAR point cloud, and a regularizer, as follows:  $\mathcal{L}_{\mathrm{scene}} = \mathcal{L}_{\mathrm{rgb}} + \lambda_{\mathrm{lidar}}\mathcal{L}_{\mathrm{lidar}} + \lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}$ . Specifically, we have
$$
\mathcal {L} _ {\mathrm {r g b}} = \frac {1}{| \mathcal {R} _ {\mathrm {i m g}} |} \sum_ {\mathbf {r} \in \mathcal {R} _ {\mathrm {i m g}}} \left\| C (\mathbf {r}) - \hat {C} (\mathbf {r}) \right\| _ {2}, \mathcal {L} _ {\text {l i d a r}} = \frac {1}{| \mathcal {R} _ {\text {l i d a r}} |} \sum_ {\mathbf {r} \in \mathcal {R} _ {\text {l i d a r}}} \left\| D (\mathbf {r}) - \hat {D} (\mathbf {r}) \right\| _ {2}. \tag {3}
$$
Here,  $\mathcal{R}$  represents the set of camera or LiDAR rays.  $C(\mathbf{r})$  is the observed color for ray  $\mathbf{r}$ , and  $\hat{C}(\mathbf{r})$  is the predicted color.  $D(\mathbf{r})$  is the observed depth for ray  $\mathbf{r}$ , and  $\hat{D}(\mathbf{r})$  is the predicted depth in the range view. To encourage smooth geometry, we also regularize the SDF to satisfy the Eikonal equation and have free space away from the LiDAR observations [89].
Training panorama inpainting: We train a panorama inpainting network to fill the unobserved regions for stitched panorama  $\mathbf{I}_{\mathrm{pano}}$ . We adopt the DeepFill-v2 [93] network and train on the Holicity [100] dataset, which contains 6k panorama images. During training, we first generate a camera visibility mask using limited-FoV camera intrinsics to generate an incomplete panorama image. The masked panorama is then fed into the network and supervised with the full panorama. Following [93], we use the hinge GAN loss as the objective function for the generator and discriminator.
Training sky dome estimator: We train a sky dome estimator network on collected HDR sky images from HDRMaps [24]. The HDRs are randomly distorted (including random exposure scaling, horizontal rotation, and flipping) and then tone-mapped to form LDR-HDR pairs  $(\mathbf{L},\mathbf{E})$  pairs. Following [82], we apply teacher forcing randomly and employ the  $L_{1}$  angular loss,  $L_{1}$  peak intensity, and  $L_{2}$  HDR reconstruction loss in the log space during training.
4 Neural Lighting Simulation of Dynamic Urban Scenes
As is, our lighting-aware digital twin reconstructs the original scenario. Our goal now is to enable controllable camera simulation. To be controllable, the scene representation should not only replicate the original scene but also handle changes in dynamic actor behavior and allow for insertion of synthetic rare objects, such as construction cones, that are challenging to find in real data alone. This enables diverse creation of unseen scenes. As our representation is compositional, we can add and remove actors, modify the locations and trajectories of existing actors, change the SDV position, and perform neural rendering on the modified scene to generate new camera video in a spatially- and temporally-consistent manner. Using our estimated lighting, we can also use a physically-based renderer to seamlessly composite synthetic assets, such as CAD models [76], into the scene in a 3D- and lighting-aware manner. These scene edits result in an "augmented reality" representation  $\mathcal{M}'$ ,  $\mathbf{E}^{\mathrm{src}}$  and source image  $\mathbf{I}_{\mathrm{src}}'$ . We now describe how we perform realistic scene relighting (Fig. 3 right) to generate new relit videos for improving camera-based perception systems.