Title: Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Gaussian Splatting

RAW images

Computational Photography

Computer vision.
Abstract: Volumetric rendering-based methods, like NeRF, excel in HDR view synthesis from RAW images, especially for nighttime scenes. They suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in dis- tant views; 2) the limited representation capacity of the spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for down- stream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous vol- umetric rendering-based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://srameo.github.io/projects/le3d.

Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis

Abstract

Volumetric rendering-based methods, like NeRF, excel in HDR view synthesis from RAW images, especially for nighttime scenes. They suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in distant views; 2) the limited representation capacity of the spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D ( L ighting E very darkness with 3D GS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for downstream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous volumetric rendering-based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://srameo.github.io/projects/le3d.

1 Introduction

Since the advent of Neural Radiance Fields (NeRF) [37], novel view synthesis (NVS) has entered a period of vigorous development, thereby advancing the progress of related applications in augmented and virtual reality (AR/VR). Existing NVS technologies predominantly utilize multiple well-exposed and noise-free low dynamic range (LDR) RGB images as inputs to reconstruct 3D scenes. This significantly impacts the capability to capture high-quality images in environments with low light or high contrast, such as nighttime settings or areas with stark lighting differences. These scenarios typically necessitate the use of high dynamic range (HDR) scene reconstruction techniques.

Existing HDR scene reconstruction techniques primarily fall into two categories and all are based on volumetric rendering: 1) using multiple-exposure LDR RGB images for supervised training [23], and 2) conducting training directly on noisy RAW data [36]. The first type of method is typically highly effective in well-lit scenes. However, in nighttime scenarios, its reconstruction performance is constrained due to the impact of the limited dynamic range in sRGB data [6]. While RAW data preserves more details in nighttime scenes, it is also more susceptible to noise. Therefore, RawNeRF [36] is proposed to address the issue of vanilla NeRF's lack of noise resistance. However, RawNeRF suffers from prolonged training times and an inability to render in real-time (a common drawback of volumetric rendering-based methods). This significantly limits the application of current scene reconstruction techniques and HDR view synthesis. Enabling real-time rendering for HDR view synthesis is a key step towards bringing computational photography to 3D world.

Recently, 3D Gaussian Splatting (3DGS) has garnered significant attention based on its powerful capabilities in real-time rendering and photorealistic reconstruction. 3DGS utilizes Structure from Motion (SfM) [41] for initialization and employs a set of 3D gaussian primitives to represent the scene. Each gaussian represents direction-aware colors using spherical harmonics (SH) functions and can be updated in terms of color, position, scale, rotation, and opacities through gradient descent optimization. Although 3DGS demonstrates its strong capabilities in reconstruction and real-time rendering, it is not suitable for direct training using nighttime RAW data. This is primarily due to 1) the SfM estimations based on nighttime images are often inaccurate, leading to blurred distant views or the potential emergence of floaters; 2) the SH does not adequately represent the HDR color information of RAW images due to its limited representation capacity; and 3) the finally reconstructed structure, such as depth map, is suboptimal, leading to unsatisfactory performance in downstream tasks like refocus.

To make 3DGS suitable for HDR scene reconstruction, we introduce LE3D that stands for L ighting E very darkness with 3D -GS, addressing the three aforementioned issues. First, to address the issue of inaccurate SfM distant view estimation in low-light scenes, we proposed Cone Scatter Initialization to enrich the COLMAP-estimated SfM. It performs random point sampling within a cone using the estimated camera poses. Second, instead of the SH, we use a tiny MLP to represent the color in RAW linear color space. To better initialize the color of each gaussian, different color biases are used for various gaussian primitives. Thirdly, we propose to use depth distortion and near-far regularization to achieve better scene structure for downstream tasks like refocusing. As shown in Fig. 1 (left), our LE3D can perform real-time rendering with only about 1.5 GPU hours (99 % less than RawNeRF [36]) of training time and at a rate of 100 FPS (about 4000 × faster than RawNeRF [36]) with comparable quality. Additionally, it is capable of supporting downstream tasks such as HDR rendering, refocusing, and exposure variation, as shown in Fig. 1 (right).

In summary, we make the following contributions:

We propose LE3D, which can reconstruct HDR 3D scenes from noisy raw images and perform real-time rendering. Compared with the NeRF-based methods, LE3D requires only 1% of the training time and has 4000 × render speed.

To address the deficiencies in color representation by vanilla 3DGS and the inadequacies of SfM estimation in nighttime scenes, we leverage a Color MLP with primitive-aware bias, and introduce Cone Scatter Initialization to enrich the point cloud initialized by COLMAP.

To improve the scene structure in the final results for achieving better downstream task performance, we introduce depth distortion and near-far regularizations.

2 Related work

Novel view synthesis Since the emergence of NeRF [37], NVS has gained significant advancement. NeRF employs an MLP to represent both the geometry of the scene and the view-dependent color. It utilizes the differentiable volume rendering [28], thereby enabling gradient-descent training through a multi-view set of 2D images. Subsequent variants of NeRF [2, 3, 22] have extended NeRF's capabilities with anti-aliasing features. To overcome the deficiencies of vanilla NeRF in geometry reconstruction, strategies such as depth supervision [10, 12] and distortion loss [3] have been introduced into NeRF. Several methods [9, 46] work on low-light image enhancement with NeRF, and extended its application scenarios. Some methods [5, 7, 16, 39] have conducted experiments with feature-grid based approaches to enhance the training and rendering speeds of NeRF. Although these methods have achieved relatively promising results in novel view synthesis, the training and rendering speeds are still significant bottlenecks. This issue is primarily due to the dense sampling inherently required by volume rendering.

Recently, the advent of 3D Gaussian Splatting [25] has marked a significant advancement in real-time NVS methods. 3DGS represents a scene using a collection of 3D gaussian primitives, each endowed with distinct attributes. Some subsequent works have added anti-aliasing capabilities to 3DGS representations [32, 43, 52]; others have enhanced 3DGS representation capabilities through supervision in the frequency domain [53]. DNGaussian [29] proposed a depth-regularized framework to optimize sparse-view 3DGS, and other works also relying on depth supervision [8, 27]. Additionally, some works [30, 34, 49] have focused on applying 3DGS to dynamic scene representation. However, these methods only accept LDR sRGB data for training, and thus cannot reconstruct the scene's HDR radiance. This limitation suggests they cannot perform downstream tasks such as HDR tone mapping and exposure variation. In contrast, LE3D is specifically designed to reconstruct the HDR representation of scenes from noisy RAW images.

HDR view synthesis and its applications HDR typically refers to a concept in computational photography that focuses on preserving as much dynamic range as possible to facilitate more postprocessing options [11, 14, 20, 24, 33]. The existing HDR view synthesis techniques bear a striking resemblance to the two main approaches in 2D image HDR synthesis: 1) Direct use of multipleexposure LDR images to compute the camera response function (CRF) and synthesize an HDR image [11]. This corresponds to HDR-NeRF [23] which employs an MLP to learn the CRF. 2) Acquisition of noise-free underexposed RAW images, utilizing the characteristics of the linear color space in RAW to manually simulate multiple-exposure images, and synthesize an HDR image. This corresponds to RawNeRF [36], which learns a NeRF representation of RAW linear color space with noisy RAW images to perform both denoising and NVS. VR-NeRF[50] performs a perceptual color space that enables perceptual optimization of high dynamic range images using. Although both methods achieve impressive visual results, the dense sampling required by volume rendering still poses a bottleneck for both training time and rendering efficiency.

LE3D follows the same technical approach as RawNeRF, reconstructing scene representations from noisy RAW images. This means that LE3D does not necessarily require training data with multiple exposures, significantly broadening its range of applications. However, a key distinction of LE3D is its use of differentiable rasterization techniques [25, 26, 44], which enable fast training and real-time rendering . Based on a 3DGS-like representation of the reconstructed scene, LE3D can perform real-time HDR view synthesis. This is a novel attempt to introduce computational photography into the 3D world, as it enables real-time reframing and post-processing (changing white balance, HDR rendering, etc. as shown in Fig. 1).

3 Preliminaries

3D Gaussian Splatting 3D Gaussian Splatting renders detailed scenes by computing the color and depth of pixels through the blending of many 3D gaussian primitives. Each gaussian is defined by its center in 3D space µ i ∈ R 3 , a scaling factor s i ∈ R 3 , a rotation quaternion q i ∈ R 4 , and additional attributes such as opacity o i and color features f i . The basis function of a gaussian primitive is given by Eqn. (1) that incorporates the covariance matrix derived from the scaling and rotation parameters.

COLMAP

Init.

Noisy Images

Cone

Init.

3D-GS Init.

Training

𝑏

!

𝑓

!

Ϝ

!

𝑣

+

𝑐

!

Color MLP

𝑟

!

𝑜

!

𝑠

!

𝑝

!

ℛ

!"

ray

ray

ℛ

#$%&

: Rendering Termination

Rendered RAW

After ISP

Noisy RAW

After ISP

ℒ

F

H

During rendering, the color of a pixel is determined by blending the contributions of multiple gaussians that overlap the pixel's location. This process involves decoding the color features f i to color c i by the SH, and calculating α i of each primitive by multiplying its opacity o i with its projected 2D gaussian G 2 i D on the image plane. Unlike traditional ray sampling strategies, 3D Gaussian Splatting employs an optimized rasterizer to gather the relevant gaussians for rendering. Specifically, the color C is computed by blending N ordered gaussians overlapping the pixel:

HDR view synthesis with noisy RAW images RawNeRF [36], as a powerful extension of NeRF, specifically addresses the challenge of high dynamic range (HDR) view synthesis with noisy images. Different from LDR images, the dynamic range in HDR images can span several orders of magnitude between bright and dark regions, resulting in the NeRF's standard L2 loss being inadequate. To address this challenge, RawNeRF introduces a weighted L2 loss that enhances supervision in the dark regions. RawNeRF applies gradient supervision on tone curve ψ = log( y + ϵ ) with ϵ = 10 -3 , and uses ψ ′ = ( y + ϵ ) -1 as the weighting term on the L2 loss between rendered color ˆ y i and noisy reference color y i . Applying the stop-gradient function sg ( · ) on y , the final loss can be expressed as:

Moreover, RawNeRF employs a variable exposure training method to take advantage of images with varying shutter speeds. The method scales the output color in linear RGB space by a learned factor β c t i for each color channel c with each unique shutter speed t i . In particular, the c -th channel of the output color ˆ y c i will be mapped into min(ˆ y c i · t i · β c t i , 1) as the final output for rendering.

4 Proposed method

The pipeline of our LE3D is shown in Fig. 2. Our main motivations and solutions are as follows: 1) To address the issue of COLMAP's inadequacy in capturing distant scenes during nighttime, we utilize the proposed Cone Scatter Initialization to enrich the point cloud obtained from COLMAP. 2) Experiments show that the original SH in 3DGS is inadequate for representing the RAW linear color space (as shown in Fig. 4 (e) and Fig. 7). Therefore, we replace it with a tiny color MLP. 3) To enhance the scene structure and optimize the performance of downstream tasks, we propose the depth distortion R dist and near-far R nf regularizations.

4.1 Improvements to the vanilla 3DGS representation

Directly applying 3DGS on noisy RAW images set faces the aforementioned two challenges, lack of distant points and inadequate representation of RAW linear color space. To address them, we propose the following improvements to the vanilla 3DGS representation.

Cone Scatter Initialization To enrich the COLMAP-initialized point cloud S = { s i } with distant scenes, we estimate the position and orientation of all cameras. Based on this, we randomly scatter points within a predefined viewing frustum F . To define F , we need to determine: 1) The viewpoint p ; 2) The viewing direction ⃗ n ; 3) The field of view Θ ; and 4) The near and far planes, z = z n and z = z f , respectively. For forward-facing scenes, the viewing direction can be easily determined by averaging the orientations of all cameras, represented as ⃗ n = avg { ⃗ n c i } . To encompass all visible areas in space from the training viewpoints, we use the maximum value of FOV from all cameras, denoted as Θ = max { θ c i } . Additionally, F needs to include all the camera origins { p c i } to ensure complete coverage of the scene from all perspectives. It means that F should encompass a circle with its center at p c = avg { p c i } , radius r = max {∥ p c i -p c ∥ 2 } , and perpendicular to ⃗ n . Therefore, we can establish F :

For near z n and far z f , we use the distance from the nearest and λ F times the distance from farthest points in the COLMAP-initialized point cloud S to p to represent them, respectively. Subsequently, we randomly scatter points within our viewing frustum F = { p , ⃗ n , Θ , z n , z f } to obtain our enriched point cloud S ′ = S ∪ S F , where S F is the scattered point set. Then S ′ is used for initialization of the gaussians, instead of S .

Color MLP with primitive-aware bias To address the issue that SH could not adequately represent the RAW linear color space, we replace it with a tiny color MLP F θ . Each gaussian is initialized with a random color feature f i and a color bias b i . To initialize b i , we project each s ′ i ∈ S ′ onto every training image, obtaining a set of all pixels { c pix } i for each point i ,. The color features f i is concatenated with the camera pose v , and then it is fed into the tiny color MLP F θ to obtain the view-dependent color. Since the HDR color space theoretically has no upper bound on color values, we use the exponent function as the activation function for F θ to simulate it. The final color c i is:

where f (0) i is sampled from a gaussian distribution N (0 , σ f ) and b (0) i is setted by the log value of the average of { c pix } i . This initialization makes c (0) i close to avg { c pix } i . Both f i and b i are learnable parameters and during cloning and splitting, they are copied and assigned to new gaussians.

4.2 Depth distortion & near-far regularizations

Scene structure is crucially important for the downstream applications of our framework, particularly the tasks such as refocusing. Therefore, we propose depth distortion and near-far regularizations to enhance the ability of 3DGS to optimize scene structure. Borrowing from NeRF-based methods [3], we use a depth map and weight map to regularize the scene structure.

Depth and weight map rendering Recently, several 3DGS-based works [29, 8] employ some form of supervision on depth. Also, depth maps are crucial for downstream tasks such as refocus (Sec. 6), mech extraction [19] and relighting [17, 54]. They are achieved by obtaining the rendered average depth map d in the following manner:

where d denotes the depth map, ω i represents the blending weight of the i -th gaussian, [ x i , y i , z i ] T and [ x c i , y c i , z c i ] T represent the position in the world and the camera coordinate system, respectively, and [ W,t ] corresponds to the camera extrinsics. The pixel values in the Weight Map each describe a histogram H of the distribution on the ray passing through this pixel. Similar to Mip-NeRF 360 [3],

we can optimize the scene structure by constraining the gaussian primitives on each ray to be more concentrated. To obtain the Weight Map, we first need to determine the distances to the nearest and farthest gaussian primitives from the current camera pose p c , represented as z c n , z c f , respectively. Subsequently, we transform the interval [ z c n , z c f ) to obtain K intersections, where the k -th intersection is denoted as [ t k , t k +1 ) . Thus, the k -th value in the histogram H ( k ) can be obtained through rendering in the following manner:

Rendering H is essential, as it is effective not only in regularization but also plays a role in the refocusing application.

Proposed regularizations Inspired by Mip-NeRF 360 [3], we proposed similar depth distortion regularization R dist to concentrate the gaussians on each ray:

R dist constrains the weights along the entire ray to either approach zero or be concentrated at the same intersection. However, in unbounded scenes of the real world, the distances ( z c f -z c n ) /K between each intersection are vast. Forcibly increasing the size of K to reduce the length of each intersection also significantly increases the computational burden. This means that our R dist can only provide a relatively coarse supervision for the gaussians on each ray, primarily by constraining them as much as possible within the same intersection.

To further constrain the concentration of gaussians, we propose the Near-Far Regularization R nf . R nf enhances the optimization of scene structure by narrowing the distance between the weighted depth of the nearest and farthest M gaussians on each ray, where the farthest refers to the last M gaussians when the blending weight approaches 1 . First, we extract two subsets of gaussians, N and F , which respectively contain the nearest and farthest M gaussians on each ray. Subsequently, we render the depth maps for both subsets ( d N , d F ), as well as the final blending weight map ( T N , T F ). The blending weight map T is the sum of each ω i . And here comes R nf :

It not only can prune the gaussians at the front or back of each ray through opacity supervision when there is a significant disparity between them (relying on the T N · T F term). Compared to R dist , R nf can also supervise the position of the first and last M gaussians on each ray to be as close as possible (relying on the ∣ ∣ d N -d F ∣ ∣ term). Besides the weighted L2 loss L and proposed regularizations R dist and R nf , we also introduce constraints on the final blending weights T . Given that the LE3D is tested in real-world scenarios, T should ideally approach 1 , meaning that all pixels should be rendered. Thus, we propose R T = -log( T + ϵ ) to penalize the pixels where T is less than 1.

5 Experiments

5.1 Implementation details

Loss functions and regularizations In our implementation, the final loss function is:

where L is the weighted L2 loss, and R T , R dist , and R nf are the proposed T, depth distortion, and near-far regularizations, respectively.

Optimization We set λ F to 10 to enrich the COLMAP-initialized point cloud in distant views. λ T , λ dist , λ nf in the loss function are set to 0 . 01 , 0 . 1 , 0 . 01 respectively. For our color MLP F θ , we use the Adam optimizer with an initial learning rate of 1 . 0 e -4 . The initial learning rates for color features and biases for each gaussians are set to 2 . 0 e -3 and 1 . 0 e -4 , respectively. The learning rates for all three decrease according to a cosine decay strategy to a final learning rate of 1 . 0 e -5 . Besides the color MLP, primitive-aware color bias, and the color features for each gaussians, other settings are the same as those of 3DGS [25]. For scenes captured with multiple exposures, we employ the same multiple-exposure training strategy as RawNeRF [36].

5.2 Datasets and comparisons

Datasets We evaluated LE3D's performance on the benchmark dataset of RawNeRF. It includes fourteen scenes for qualitative testing and three test scenes for quantitative testing. The three test scenes contains 101 noisy images and a clean reference image merged from stabilized long exposures. However, the training data are captured with short exposures, leading to exposure inconsistencies. Therefore, we apply the same affine alignment operation as RawNeRF before testing (detailed in Sec. A.1 of the supplementary materials). All images are 4032 × 3024 Bayer RAW images captured by an iPhone X, saved in DNG format.

Baseline and comparative methods Wecompare two categories of methods, 3DGS-based methods, and NeRF-based methods. The baseline we compare against is RawGS which uses vanilla 3DGS for scene representation and employs the weighted L2 loss and multiple exposure training proposed in RawNeRF [36]. Besides, we remove the SSIM loss, as its computation over local neighborhoods disrupts the noise model present in the RAW data. Additionally, we compare LDR-GS and HDR-GS, both of which are vanilla 3DGS trained on post-processed LDR images and unprocessed RAW images, respectively. The NeRF-based methods include RawNeRF [36] and LDR-NeRF. LDR-NeRF is a vanilla NeRF [37] trained on the post-processed LDR images with L2 loss.

Quantitative evaluation Tab. 1 has shown the quantitative comparisons on the RawNeRF [36] dataset. Although NeRF-based methods have long training times and slow rendering speeds, they exhibit good metrics on sRGB. This indicates that the volume rendering they rely on has strong noise resistance (mainly due to the dense sampling on each ray). In contrast, 3DGS-based methods have inferior metrics compared to RawNeRF, due to their sparse scene representation and poor noise resistance. Additionally, the splitting of gaussians depends on gradient strength, and supervision using noisy raw images affects this process, leading to incomplete structure recovery. LE3D achieves better structure reconstruction suitable for downstream tasks through supervision on structure, depth distortion, and near-far regularizations, as detailed in Sec. 6. Note that the results of LE3D are comparable to the previous volumetric rendering-based method, RawNeRF [36], in both quantitative and qualitative aspects. However, it requires only 1% of the training time and achieves a 3000 × -6000 × rendering speed improvement.

Qualitative evaluation Fig. 3 has shown the qualitative comparisons on the RawNeRF [36] dataset. We selected four scenes for comparison, including two indoor scenes and two outdoor scenes. The data for the first two scenes were collected with a single exposure, while the data for the latter two scenes included multiple exposures. Compared to 3DGS [25]-based methods, LE3D demonstrates stronger noise resistance, particularly in the first two scenes. Additionally, LE3D achieves better results in distant scene reconstruction. For example, in the second scene, LE3D produces a smoother sky compared to RawGS, and in the fourth scene, LE3D recovers distant details more sharply. Compared to RawNeRF, LE3D typically produces smoother results while still effectively preserving details. Most importantly, LE3D offers faster training times and rendering speeds.

Training View

FPS: 0.007

FPS: 186.6

FPS: 0.023

FPS: 225.7

FPS: 137.9

FPS: 0.007

FPS: 256.0

FPS: 0.023

FPS: 137.1

FPS: 110.4

FPS: 0.007

FPS: 219.6

FPS: 0.023

FPS: 204.5

FPS: 87.4

FPS: 0.007

FPS: 128.4

FPS: 0.023

FPS: 135.8

FPS: 80.7

RENSCAI

PENDEAL

LDR-NeRF [37]

LDR-GS [25]

RawNeRF [36] RawGS (Baseline)

LE3D (Ours)

5.3 Ablation studies

Cone Scatter Initialization (CSI) In low-light environments, COLMAP struggles to obtain a high-quality sparse point cloud. Although 3DGS demonstrates its robustness to the quality of the initial point cloud, it still encounters difficulties in achieving optimal geometric reconstruction within insufficient initialized areas. It can be observed from Fig. 4 (b) that the methods without CSI tend to generate gaussians at incorrect depths and lack fine details. Conversely, CSI extends the depth coverage of the scene, enabling 3DGS to generate gaussians at relatively accurate depths and exhibit superior detail representation. A comparative analysis between Fig. 4 (a) and Fig. 4 (b) suggests that our initialization technique plays a pivotal role in achieving accurate and detailed 3D reconstruction.

Color MLP Replacing SH with Color MLP not only enhances the expressiveness of our model but also introduces greater stability during the optimization process. Fig. 4 (e) reveals that the method employing SH rather than Color MLP exhibits strange color representations early in the training phase, due to the inability of SH to adequately represent the RAW linear color space. Although the rendered image may appear similar to those produced by the LE3D, the underlying issues have significantly affected the final structural reconstruction, as depicted in Fig. 4 (c).

Regularizations Superior visual effects in 3D are contingent upon a robust 3D structure reconstruction, which in turn significantly enhances the performance of downstream tasks such as refocusing. To this end, we implement depth distortion regularization R dist and near-far regularization R nf to constrain the gaussians, ensuring their aggregation at the surfaces of objects and thereby improving the quality of structural reconstruction. Fig. 4 (d) underscores the substantial enhancement our proposed regularizations provide in reconstructing the 3D structure of scenes.

6 More applications

Refocus Structural information is crucial for tasks like refocusing. As discussed in Sec. 5.3, LE3D benefits from the inclusion of depth distortion and near-far regularizations, which enhances its ability to learn structural details. As shown in Fig. 5 (b, d), LE3D achieves more realistic refocusing effects due to its superior structural information, as reflected in the depth shown in (c). Conversely, RawGS suffers from foreground and background ambiguity in refocusing due to the lack of structural information. The detailed refocusing algorithm will be released in the supplementary materials.

(a) LE3D

(b) w/o CSI

(c) w/o Color MLP

Training View

(e) Results at 7k iter

w/ Color MLP

w/o Color MLP

Focus at here

Focus at here

(a) LE3D output

(b) LE3D refocus

(c) LE3D output depth

(d) RawGS    refocus

(e) RawGS output depth

Long

Medium

Short

Global & Local Tone-mapping

More Results

⋆

1

(f) Exposure Variation

(g) Changing Tone-mapping, Viewpoint, and Focus

1

Exposure variation and HDR tone-mapping LE3D can easily achieve exposure variation and recover details from overexposed data, as shown in Fig. 5 (f). Fig. 5 (g) showcases the various tone-mapping methods LE3D can implement, including global tone-mapping, such as color temperature and curve adjustments, and local tone-mapping using our re-implemented HDR+ [20] (implementation details will be released in the supplementary materials).

Although RawNeRF [36] can also perform similar applications, its inability to achieve real-time rendering significantly limits its use cases, such as real-time editing described in Sec. C.

7 Conclusion

To address the long training times and slow rendering speeds of previous volumetric rendering-based methods, we propose LE3D based on 3DGS. Additionally, we introduce Cone Scatter Initialization and a tiny MLP to represent color in the linear color space. This addresses the issue of missing distant points in nighttime scenes with COLMAP initialization. It also replaces spherical harmonics with the tiny color MLP, effectively representing the linear color space. Finally, we enhance the structural reconstruction with the proposed depth distortion and near-far regularization, enabling more effective and realistic downstream tasks. Benefiting from the rendering images in the linear color space, LE3D can achieve more realistic exposure variation and HDR tone-mapping in real time, expanding the possibilities for subsequent HDR view synthesis processing.

References

Jonathan T Barron, Andrew Adams, YiChang Shih, and Carlos Hernández. Fast bilateral-space stereo for synthetic defocus. In CVPR , 2015.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

Yuanhao Cai, Zihao Xiao, Yixun Liang, Yulun Zhang, Xiaokang Yang, Yaoyao Liu, and Alan Yuille. Hdr-gs: Efficient high dynamic range novel view synthesis at 1000x speed via gaussian splatting. arXiv:2405.15125 , 2024.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV , 2022.

Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In CVPR , 2018.

Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. Neurbf: A neural fields representation with adaptive radial basis functions. In ICCV , 2023.

Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaussian splatting in few-shot images. arXiv:2311.13398 , 2023.

Ziteng Cui, Lin Gu, Xiao Sun, Xianzheng Ma, Yu Qiao, and Tatsuya Harada. Aleth-nerf: Illumination adaptive nerf with concealing field assumption. In AAAI , 2024.

David Dadon, Ohad Fried, and Yacov Hel-Or. Ddnerf: Depth distribution neural radiance fields. In WACV , 2023.

Paul E Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. Siggraph , 1997.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR , 2022.

Paul HC Eilers and Brian D Marx. Flexible smoothing with b-splines and penalties. Statistical science , 1996.

Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger. Hdr image reconstruction from a single exposure using deep cnns. ACM TOG , 2017.

Zeev Farbman, Raanan Fattal, and Dani Lischinski. Convolution pyramids. ACM TOG , 2011.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing. arXiv:2311.16043 , 2023.

NeRF Studio Group. Viser. https://github.com/nerfstudio-project/viser , 2023.

Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. CVPR , 2024.

Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM TOG , 2016.

Tanner Helland. How to convert temperature (k) to rgb: Algorithm and sample code. https: //tannerhelland.com/2012/09/18/convert-temperature-rgb-algorithm-code.html , 2012.

Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV , 2023.

Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In CVPR , 2022.

Nima Khademi Kalantari, Ravi Ramamoorthi, et al. Deep high dynamic range imaging of dynamic scenes. ACM TOG , 2017.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG , 2023.

Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. Point-based neural rendering with per-view optimization. In Computer Graphics Forum , 2021.

Pou-Chun Kung, Seth Isaacson, Ram Vasudevan, and Katherine A Skinner. Sad-gs: Shape-aligned depth-supervised gaussian splatting. In ICRA Workshops , 2024.

Marc Levoy. Efficient ray tracing of volume data. ACM TOG , 1990.

Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. arXiv:2403.06912 , 2024.

Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. CVPR , 2024.

Zhihao Li, Yufei Wang, Alex Kot, and Bihan Wen. From chaos to clarity: 3dgs in the dark. arXiv:2406.08300 , 2024.

Zhihao Liang, Qi Zhang, Wenbo Hu, Ying Feng, Lei Zhu, and Kui Jia. Analytic-splatting: Anti-aliased 3d gaussian splatting via analytic integration. arXiv:2403.11056 , 2024.

Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. Single-image hdr reconstruction by learning to reverse the camera pipeline. In CVPR , 2020.

Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv:2308.09713 , 2023.

Tom Mertens, Jan Kautz, and Frank Van Reeth. Exposure fusion. In PG , 2007.

Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In CVPR , 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman, Ricardo Martin-Brualla, and Jonathan T. Barron. MultiNeRF: A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF. https://github. com/google-research/multinerf , 2022.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG , 2022.

Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. Photographic tone reproduction for digital images. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 , 2023.

Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR , 2016.

Shreyas Singh, Aryan Garg, and Kaushik Mitra. Hdrsplat: Gaussian splatting for high dynamic range 3d scene reconstruction from raw images. BMVC , 2024.

Xiaowei Song, Jv Zheng, Shiran Yuan, Huan-ang Gao, Jingwei Zhao, Xiang He, Weihao Gu, and Hao Zhao. Sa-gs: Scale-adaptive gaussian splatting for training-free anti-aliasing. arXiv:2403.19615 , 2024.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR , 2022.

Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T Barron, Yael Pritch, and Marc Levoy. Synthetic depth-of-field with a single-camera mobile phone. ACM TOG , 2018.

Haoyuan Wang, Xiaogang Xu, Ke Xu, and Rynson WH Lau. Lighting up nerf via unsupervised decomposition and enhancement. In ICCV , 2023.

Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. https://github.com/XPixelGroup/BasicSR , 2022.

Kaixuan Wei, Ying Fu, Yinqiang Zheng, and Jiaolong Yang. Physics-based noise modeling for extreme low-light photography. IEEE TPAMI , 2021.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv:2310.08528 , 2023.

Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Aljaž Božiˇ c, et al. Vr-nerf: High-fidelity virtualized walkable spaces. In Siggraph Asia , 2023.

Chongjie Ye, Yinyu Nie, Jiahao Chang, Yuantao Chen, Yihao Zhi, and Xiaoguang Han. Gaustudio: A modular framework for 3d gaussian splatting and beyond. arXiv:2403.19632 , 2024.

Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. CVPR , 2024.

Jiahui Zhang, Fangneng Zhan, Muyu Xu, Shijian Lu, and Eric Xing. Fregs: 3d gaussian splatting with progressive frequency regularization. CVPR , 2024.

Tianyi Zhang, Kaining Huang, Weiming Zhi, and Matthew Johnson-Roberson. Darkgs: Learning neural illumination and 3d gaussians relighting for robotic exploration in the dark. arXiv:2403.10814 , 2024.

Xuaner Zhang, Kevin Matzen, Vivien Nguyen, Dillon Yao, You Zhang, and Ren Ng. Synthetic defocus and look-ahead autofocus for casual videography. arXiv:1905.06326 , 2019.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv:1805.09817 , 2018.

Appendix

A Implementation details and more ablation studies

A.1 More detailed implementation

Detailed network architecture of the Color MLP To prevent our color MLP from occupying too much time during rendering, we used a very compact MLP. It consists of only one 16-dimension hidden layer and takes a 19-dimensional input (16 for the color feature of each gaussian and 3 for the viewing direction). With activation function ReLU , the output of the MLP is a 3-dimensional vector as the final color of the gaussian primitive.

Sepcific hyperparameters The λ F in (4) to determine the farthest plane of the training frustum is set to 10. The K intersections of our weight map H is set to 32. The M gaussian number to calculate the near, far subsets N and F if set to 5. The λ T , λ dist , λ nf are set to 0 . 01 , 0 . 1 , 0 . 01 , respectively. And our training and testing resolution is set to 2016 × 1512 .

Experimental environments In our implementation, we use a single Nvidia GeForce 3090 GPU (24 GB VRAM) for both training and testing along with a 20-core CPU and a 64GB RAM. The Pytorch version is 1.12.1, CUDA version is 11.8. Due to the limitation of 24 GB VRAM, RawNeRF cannot be trained on a single Nvidia GeForce 3090 GPU. For the test scenes, we train RawNeRF on 4 GPUs, while for the other scenes, RawNeRF is trained with 2 GPUs. Apart from the difference in the number of GPUs, we did not make any modifications to RawNeRF.

Affine alignment Since all training views in the RawNeRF [36] dataset are captured with a fast shutter, while the test views (ground truth) are captured with a slow shutter, linear enhancement is needed during testing for alignment. However, due to color bias (non-zero-mean noise for high ISO, [48]), direct linear enhancement does not achieve perfect alignment. Therefore, affine alignment is performed on both the output and ground truth during testing. In RawNeRF [36], this process is done as the following procedure:

where x is the mean of x . And x, y are the ground truth and the final output, respectively. This is the least-squares fit of an affine transform ax + b ≈ y . At test time, we first process y with ( y -b ) /a , then calculate the metric. For those methods whose output is in RAW linear color space, the affine alignment process is only done once in the RAW color space. While for other methods (LDR-NeRF [37], LDR-3DGS [25]) which can only output in RGB color space, the affine alignment process is done in the RGB color space.

A.2 More ablation studies

Ablation on number of gaussian with/without CSI While our Cone Scatter Initialization is used to add sparse points for distant scenes, some may be concerned it will influence the total number of gaussians, which is relevant for rendering speed. In fact, due to the optimization capabilities of 3DGS, in all scenarios, CSI not only did not increase the number of points but actually reduced them. Besides, CSI is particularly effective in the outdoor scene (-24.86% for 'bikes' and -13.10% for 'windowlegovary'). Details can be found in Tab. 2.

Ablation on Color Representation In practice, we have observed that employing SH directly can lead to negative color output values, an anomaly in the context of the RAW linear RGB space where such values are not permissible. Noticing that we attempted to directly apply the exponential function after the SH function, turning these negative values to positive. However, this approach did not resolve all the issues encountered. For example, in the 'windowlegovary' scenario, this approach showed a maximum value above 10 3 on one channel while staying below 10 on others, highlighting SH's expressive limitations. Additionally, we observed color issues similar to those in the 'yuccatest' scenario, as shown in Fig. 7. Therefore, using Color MLP instead of SH to represent color is necessary in our method.

Ablation on each of the regularizations Fig. 6 (a,b) presents the visualization of d N and d F adjacent to the depth map. In an ideal scenario, both d N and d F should align with d , ensuring that the weights along each ray are concentrated at the surface. The comparison between Fig. 6 (a,b) demonstrates that the incorporation of near-far regularization indeed encourages d N and d F to progressively align with d . This alignment results in a more refined representation of the threedimensional structure, capturing better details of the scene's geometry. The comparison between Fig. 6 (a,c) elucidates the adverse effects of omitting distortion regularization. Without such constraints, the model would fail to produce depth maps with a natural depth progression, with artifacts such as abrupt depth discontinuities or voids on planes. Such anomalies are indicative of significant issues in the reconstruction of the scene's geometry.

Ablation studies on test scenes As shown in Tab. 3, LE3D has shown superior performance over all ablated methods. The results without Color MLP are the worst because the SH used in vanilla 3DGS is not suitable for representing colors in the RAW linear color space. As shown in Fig. 7, the results without Color MLP are noticeably desaturated and appear gray. Both w/o R nf and w/o R dist show degraded depth, as seen in Fig. 7. Additionally, it can be observed that w/o Color MLP also has poor structural information. This is mainly due to instability during the early stages of training, leading to suboptimal depth map reconstruction, as discussed in Sec. 5.3.

The stability of LE3D Due to the random initialization of our Color MLP, we trained 9 versions of LE3D using 9 different random seeds to test the stability of LE3D. We then compared their metrics on the test set, as shown in Fig. 8. It can be observed that the stability of LE3D is remarkably high, and the overall fluctuations do not impact the experimental conclusions.

B Detailed postprocessing

B.1 Refocus

To implement the refocusing algorithm in LE3D, we adopted a refocusing approach similar to previous methods [1, 45, 55]. Most of these methods require pre-calculating a MultiPlane Image (MPI) representation [56] from the depth to synthesize refocus data. The MPI representation consists of multiple planes arranged from front to back, with each plane being an RGBA image (the RGBA image's color is still in the linear color space for our LE3D). RawNeRF [36] pre-calculates the

Training View

(d) w/o

ℛ

!"#$

, ℛ

%&

(c) w/o

ℛ

!"#$

(b) w/o

ℛ

%&

,

𝑑 ≠ 𝑑

𝐍

𝐅

(a) LE3D,

𝑑 ≈ 𝑑

𝐍

≈ 𝑑

𝐅

LE3D Results

𝑑

𝐍

𝑑

𝐅

𝑑

𝐍

𝑑

𝐅

𝑑

𝑑

𝑑

𝑑

MPI representation from its rendered results and then uses it for refocusing. However, due to its extremely slow rendering speed (about 50 seconds per image), real-time refocusing with RawNeRF is nearly impossible. To obtain our MPI representations in real time, we use the weight distortion map ( H ∈ R K × H × W , where K denotes the plane number) rendered by LE3D as described in Eqn. (7) to directly render the final opacity for each plane in the MPI. The color image ( I ∈ R 3 × H × W ) used for synthesis can also be directly rendered by LE3D. Thanks to the rendering speed of 3DGS, both I and H can be rendered in real time, enabling rapid refocusing.

After obtaining the color image I and the weight distortion map H , we can perform refocusing using the algorithm described in Algorithm 1, similar to the method in [36, 55]. Here, i focus represents the index of the focus plane, and ∆ r controls the simulated aperture size (i.e., the blur intensity). blur(c , r) represents performing a gaussian blur operation on c with a radius of r . This operation is accelerated using 2D Fourier space convolution.

In Fig. 9, we show the refocus result and multiplane image representations of LE3D and

Algorithm 1 Refocus Synthesis in LE3D

RawGS. A more specific and global result of these two methods can be found in Fig. 5 (c) and (e), their depth maps. It can be observed that the lack of structural information leads to inconsistencies in the foreground-background relationship in Fig. 9 (b), resulting in poor refocusing effects. In contrast, LE3D, with its superior structural information, achieves more realistic refocusing effects, accurately depicting the scene's depth relationships.

Rendered Depth

Rendered Image

Training View

LE3D (Ours)

w/o Color MLP

w/o

ℛ

!"

w/o

ℛ

#$%&

58.75

59.00

60.80

61.00

61.20

Values

RAW PSNR

↑

0.9968

0.9969

0.9982

0.9983

0.9984

RAW SSIM

↑

24.05

24.10

24.60

24.65

24.70

RGB PSNR

↑

0

.

605

0

.

606

0

.

607

0

.

608

0

.

609

0

.

610

RGB SSIM

↑

0.495

0.497

0.504

0.506

0.508

RGB LPIPS

↓

LE3D Mean

RawNeRF

Bootstrap CI

58.50

0.9967

24.00

0.493

B.2 Tone-mapping

Default LDR tone-mapping We use the following simplified tone-mapping function to map the RAW image to an LDR sRGB image, which is similar to the ISP pipeline used in RawNeRF [36]:

Normalizing the value of the input RAW image to [0 , 1] .

Apply bilinear demosaicking to the normalized RAW images.

Apply white balance gains for each of the three channels.

Perform the color correction operation with the color matrix from camera RGB-to-sRGB (combined camera RGB-to-XYZ and XYZ-to-sRGB).

(Optional) Exposure adjustment by setting the white level to the 97 percentile.

Clip the value to [0 , 1] , and apply the sRGB gamme correction.

LE3D is trained using the RAW images from step 2, resulting in outputs in the unmodified RAW linear color space. This means that LE3D's outputs are more suitable for various downstream tone-mapping applications.

Global tone-mapping In our implementation, we use two kinds of global tone-mapping techniques: color temperature and curve adjustment. For color temperature adjustment, we refer to Tanner Helland's algorithm [21]. For the curve adjustment algorithm, we adopted a control point strategy. Users first determine several control points, and then we use the 'B-splines' interpolation algorithm [13] to generate a smooth curve from these control points. Since the output of LE3D is a 32-bit float, we quantize it to a 16-bit unsigned integer. The curve obtained earlier is converted into a look-up table (LUT) over the range [0 , 65535] . The LUT is then applied to the output of LE3D to achieve the curve adjustment results. The results of our global tone-mapping can be found in Fig. 10. The color temperature adjustment is directly applied to LE3D's output, while the curve adjustment is performed after applying white balance and the color correction matrix.

Local HDR+ tone-mapping Our local tone-mapping is based on the HDR+ [20] algorithm. However, since LE3D's output is noise-free, we have simplified some of the steps. Our simplified HDR+ algorithm is as follows:

LE3D refocus

LE3D output

Focus At Here

(b) Multiplane Images with Structure of RawGS (Baseline)

10

1

2

3

14

CL

16

18

19

21

25

26

27

28

29

30

31

10

13

18

9

20

23

25

26

2

28

30

(a) Multiplane Images with Structure of

LE3D (Ours)

Apply white balance gains for each of the three channels of the LE3D 's output.

Perform the color correction operation with the color matrix from camera RGB-to-sRGB (combined camera RGB-to-XYZ and XYZ-to-sRGB).

(Optional) Apply the global tone-mapping curve if the curve exists.

Apply the additional digital gain for exposure adjustment, resulting in z .

Based on z , simulate multiple exposures z ev -, z ev 0 , z ev + with different exposure times (in our implementation, we use [0 . 25 , 1 . 0 , 4 . 0] ).

Postporcess z ev -, z ev 0 , z ev + to the sRGB color space, then using ExposureFusion [35] to compress the HDR results into LDR. Then we need to re-map the merged sRGB result to RAW linear color space for further postprocess. This would not cause information loss, due to all the processes are in 32-bit float format.

Perform 'dehazing' like HDR+ [20], which aims at reduceing the effect of veiling glare. This is done by allowing up to 0.1% of pixels to be clamped to zero, but only adjusting pixels below 7% of the white level.

Then we apply the Reinhard tone-mapping [40] and gamma compression to get the sRGB results.

We apply the unsharp masking technique to sharpen the image with a 3-level gaussian pyramid [15].

Convert the result into the HSV color space. Then shifting bluish cyans and purples towards light blue, and increasing the saturation of blues and greens generally. Finally, re-map the result to sRGB to get the final result.

Some sampled local HDR+ tone-mapping results can be found in Fig. 1, Fig. 5 (g) and Fig. 11 (e).

(a) Training View

3000K

6500K

40000K

Smooth Tone Curve

Smooth Tone Curve

Smooth Tone Curve

Smooth Tone Curve

60000

Control Points

50000

Control Points

00009

Control Points

40000

00000

(b) Color Temperature / Curve Adjustment & Viewpoint Changing

(a) Base View,

91.6 FPS

64.6 FPS

(c) Exposure Variation,

91.6 FPS

FPS

91.6

FPS

64.6

FPS

91.6

FPS

14.1

FPS

35.1

FPS

(d) Refocus,

14.1 FPS

(e) Global & Local (HDR+) TM,

35.1 FPS

(f) Novel View,

116 FPS

C Interactive viewer

Fig. 11 has shown some screenshots and downstream tasks results of our interactive viewer which is built upon Viser [18]. Besides refocusing, most of the downstream tasks can be performed in real time. For refocusing, most of the time is spent on the gaussian blur according to Algorithm 1 (due to the large gaussian blur kernel size) rather than on rendering.

(a) Training View

(b) RawNeRF [36]

(c) RawGS

(d)

LE3D (Ours)

D Limitation

Although LE3D can achieve real-time HDR view synthesis, it, like volumetric rendering-based methods (such as RawNeRF [36]), encounters reconstruction failures in extremely dark scenes. For example, in the training view shown in Fig. 12 (a), RawNeRF [36] failed to reconstruct the upper left corner, leaving large artifacts. RawGS, lacking structural supervision, produced many floater gaussians, significantly degrading the reconstruction quality. Although LE3D can reduce these floaters, it, like the other methods, fails to recover the details of the leaves in the upper left corner.

The main cause of this issue is the low scene brightness, resulting in a low signal-to-noise ratio. Potential solutions include: 1) Collecting as much data as possible during scene capture to enable multi-frame fusion '3D' denoising; 2) Using neural networks to denoise images before reconstruction, then using the denoised images to rebuild the scene. However, ensuring consistent denoising results across frames (consistent color space) is challenging, making it difficult to guarantee effectiveness.

E Broader impacts

Potential positive impacts The development of the LE3D method could lead to improvements in virtual and augmented reality applications, providing users with more realistic and immersive experiences. 3D reconstruction capabilities may be utilized for the digital preservation of cultural heritage sites, museums, and art, allowing for greater access and study by a broader audience. The capability of LE3D to reconstruct scenes in low-light environments also facilitates data collection, broadening the application scenarios for 3DGS. This advancement makes it viable to employ 3DGS in situations where establishing adequate lighting conditions is challenging. Most importantly , LE3D enables real-time editing of HDR view synthesis tasks (as shown in Sec. C). This further advances the development of 3D video editors.

Potential negative impacts The fast training reconstruction method like 3DGS could potentially facilitate the implementation of illegal surveillance activities. Enhancing the capabilities of 3DGS in low-light environments may render illicit surveillance practices more insidious, thereby posing a threat to individual privacy and the security of confidential information.

F Licenses for existing assets

Our method is implemented in the style of BasicSR[47], and we encourage further work based on this codebase. The dataset we utilized is sourced from RawNeRF[36]. For rendering the depth map, we employed the rasterizer by GauStudio[51], with modifications made to the rasterizer by 3DGS to implement the R nf regularization. Our real-time viewer is based on Viser[18]. For comparative analysis and reference code, we used RawNeRF from MultiNeRF[38].

G More results

G.1 Detailed quantitative results

Since the metrics in Tab. 1 are calculated after affine alignment, we also provide the metrics without affine alignment in Tab. 5. It can be observed that even without affine alignment, LE3D still ranks

a License can be found at https://github.com/graphdeco-inria/gaussian-splatting/blob/ main/LICENSE.md . This license mentions that licensors grant non-exclusive rights to use the Software for research purposes to research users (both academic and industrial), free of charge, without the right to sublicense.

b http://storage.googleapis.com/gresearch/refraw360/raw.zip

among the top in overall metrics and achieves real-time rendering speeds. Compared to other 3DGS-based methods (all capable of real-time rendering), LE3D demonstrates significantly superior structural information, as shown in Fig. 5 and Fig. 9, enabling more downstream applications.

G.2 More qualitative results

Detailed comparisons between 3DGS-based methods and LE3D As shown in Fig. 13, LE3D achieves better structure reconstruction than our baseline RawGS (3DGS trained with RawNeRF's loss and multiple exposure training strategy). Compared with LDR-GS (trained on the LDR images) and HDR-GS (trained directly on the RAW data), LE3D achieves better color reconstruction results as well as better denoising ability. We also found that LDR-GS and HDR-GS have fewer reconstructed gaussians, resulting in faster rendering speeds but poor overall reconstruction quality. Additionally, LDR-GS, trained on linear brightened LDR images, shows weaker resistance to color bias [48], resulting in severe color shifts in the final output. We also found that the generally low values of RAW images lead to insufficient gradients, reducing the number of gaussian splitting. RawNeRF's weighted L2 loss (Eqn. (3)) strengthens supervision in dark areas but at the cost of structural information. LE3D incorporates both the weighted L2 loss and depth distortion and near-far regularizations to constrain the structure, ultimately achieving the best structural and visual results.

More qualitative results Fig. 7, Fig. 13, Fig. 14, Fig. 15 has shown more qualitative comparisons between LE3D and 3DGS [25]-based methods. From the figures above, it is evident that LE3D demonstrates superior noise resistance and color representation capabilities. Additionally, LE3D produces smoother and more accurate depth maps, which are essential for downstream tasks like refocusing. It is worth noting that volumetric rendering-based methods, such as RawNeRF [36],

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

LE3D (Ours)

RawGS (Baseline)

HDR-GS [25]

LDR-GS [25]

1

1

1

cannot achieve real-time rendering, which significantly limits their applications (including real-time scene editing). Therefore, we do not compare them here. For comparisons between LE3D and volumetric rendering based methods, please refer to Tab. 1, Tab. 5, and Fig. 3.

H Concurrent work

Here list of some concurrent works that focus on HDR view synthesis based on the 3DGS-like method.

Training with RAW images : HDRSplat [42] uses an extra Bayer-space-denoising step to handle noise. 3DGS in the Dark [31] analysis how noise impacts the optimization of 3DGS, and purposes Noise robust reconstruction loss to handle noise.

Training with multi-exposure RGB images : HDR-GS [4] purposes Dual Dynamic Range (DDR) Gaussian point cloud model to jointly fit the HDR and LDR colors, and employs three independent MLPs to learn the tone-mapping operation in RGB channels.

Compared to those methods that need a trained denoising network, our LE3D could just denoise a set of noisy RAW images from a single-blind device. It means we do not need to train the denoising network from scratch for every device like HDRSplat [42] and 3DGS in the Dark [31]. And since our method uses high dynamic range data (RAW images) directly, so we do not need multi-exposure data for training like HDR-GS [4], which simplifies the data capture pipeline.

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

RawGS (Baseline)

LE3D (Ours)

LE3D

, Novel View, Edited

一

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

Rendered Depth

Rendered Image

Training View

RawGS (Baseline)

LE3D (Ours)

LE3D

, Novel View, Edited

一

一

一

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have described our motivation and contributions both in abstract and Sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have described our limitation in Sec. D

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the implementation details can be found in Sec. 5.1 and Sec. A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: No reproducible code is provided in the submission. However, all the codes, both to reproduce the results and the interactive viewer, will be released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the implementation details can be found in Sec. 5.1 and Sec. A.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bar can be found in Sec. A.2 and Fig. 8.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The necessary computer resources LE3D needed can be found in Sec. A.1. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: 1) Our experiments and research are not with human subjects; 2) the RawNeRF [36] dataset is public ( http://storage.googleapis.com/gresearch/ refraw360/raw.zip ); 3) we have discuss our potential broader impacts.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impacts are discussed in Sec. E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Methods and models for reconstruction task is hardly been misused.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Licenses for existing assets are listed in Sec. F.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: No new assets is submitted along with the submission. However, all the codes, both to reproduce the results and the interactive viewer, will be released.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LE3D, an HDR 3D Gaussian Splatting method with Cone Scatter Initialization, Color MLP, and depth distortion. Specifically, this paper introduces the Cone Scatter Initialization to enrich the estimation of SfM. The Color MLP aims to represent the RAW linear color space. The goal of depth distortion and near-far regularizations is to improve the accuracy of scene structure. Experimental results show that the proposed LE3D achieves promising results.\n\nSTRENGTHS:\n1. The motivation of this paper is good. Nighttime 3DGS and HDR 3DGS are very important.\n2. The proposed method includes comprehensive experiments.\n3. The experimental settings are easy to follow.\n4. The video demo in the supplementary material is good.\n\nWEAKNESSES:\n1. The authors mentioned, “the SfM estimations based on nighttime images are often inaccurate.” I agree with this point. However, from my understanding, the initial point clouds can be wrong and can be further optimized by the 3DGS training stage. Therefore, I am wondering if the cone scatter initialization is really important. Could the authors provide ablation studies using the RawNeRF dataset?\n2. 3DGS highly relies on the initial point clouds. One of the examples in Fig. 1 (a) shows that the input image can be very dark. I am wondering if the authors can get the initial SfM estimations from such inputs. From my understanding, COLMAP cannot work well on these dark inputs. If so, how do the authors get the initial SfM estimations?\n3. The authors mentioned, “the SH does not adequately represent the HDR color information of RAW images due to its limited representation capacity.” Why? Could the authors provide any solid evidence?\n4. The depth distortion (Eqn. 8) is quite similar to Eqn. 15 in Mip-NeRF 360. How does the proposed depth distortion differ from the depth distortion of Mip-NeRF 360? Could the authors conduct experiments to compare the two depth distortions?\n5. The authors seem to overstate their methods. Compared to RawNeRF, LE3D can reduce training time to 1%. However, the fast training speed is not actually due to the proposed components, but rather from 3DGS itself. If we compare it to the original 3DGS, the training speed of LE3D increased from 1.05 to 1.53, which is an improvement of approximately 45.71%. Since this is the first HDR 3DGS paper, I do not want to be too strict. But I suggest that the authors do not overstate the advantages in terms of speed.\n6. The quantitative results of the ablation studies should be shown.\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper LE3D proposes training a 3DGS representation with raw images, instead of preprocessed LDR images. This allows more accurate scene recovery in low-light environments and unlocks applications such as HDR rendering or refocusing. While there has been prior work on training neural scene representations with raw images (notably RawNeRF), this work specifically proposes to use a 3DGS representation which allows real-time rendering. To facilitate training a 3DGS representation with raw images, they make several contributions:\n\n1) Cone Scattering Initialization - a method to \"densify\" the Gaussian initialization to overcome inaccuracies in SFM\n2) Replacing the SH in 3DGS with a colour MLP \n3) Introducing space carving losses for better geometry recovery (for downstream tasks)\n\nSTRENGTHS:\nOriginality & Significance: I like that the paper brings several of the contributions in the NeRF literature to 3DGS, specifically I like the problem being attempted in the paper, I think HDR rendering and training with raw images is an important problem and it is timely to attempt this problem using a 3DGS representation. Losses such as the distortion loss from MipNeRF360 are widely used in the NeRF literature and I think it's valuable to have an implementation of them in the 3DGS context. Not a significant research contribution, but I also really liked the interactive viewer in the supplement, I think it would be fun to play around with the demo and see the weaknesses of the method. \n\nQuality: I think the authors are fairly open about the limitations, for example mentioning NeRF-based methods obtain superior sRGB metrics. \n\nClarity: Paper is well organized.\n\nWEAKNESSES:\nOriginality: I think although useful, the contribution of the work isn't huge, from Table 1, we can see RawNeRF + GS performs quite well already. I also believe the Related Works section is a bit sparse, there has been a lot of work on low-light image enhancement with NeRF, which I think is relevant (even though it's not about HDR rendering), see a couple below:\n\n1) Lighting up NeRF via Unsupervised Decomposition and Enhancement, ICCV 2023 - method for training NeRFs with sRGB images taken in the dark \n2) Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption, AAAI 2024 - method for training NeRFs with sRGB images taken in over/under-exposed scenarios \n\nQuality: This is my biggest gripe with the work, some of the contributions are included without significant reasoning. Most importantly:\n\n1) The authors introduce the CSI and say it helps due to the misalignments in the SFM estimation. However, I don't think this claim is sufficiently well-proven. Perhaps a denser initialization helps even in cases with perfect cameras?\n2) The authors do not justify including the colour MLP, apart from saying it performs better than just using the SH (due to the linear scaling). I think this point is specifically important since I think the MLP is making the method 1.7 times slower in rendering (looking at Table 1).\n \nFor 1) I am sympathetic to the authors that this might be hard to justify, but I think 2) should be addressed (see also questions). \n\nClarity: There are quite a few typos/grammatical errors, I list just a few below, but I would implore the authors to use a spell check throughout the text:\n\nL38: to the 3D world \n\nL170: fed \n\nL173: set\n\nQUESTIONS:\n1) Have the authors tried just increasing the rate of the Gaussian splitting instead of CSI? Have they tried anything else apart from CSI for the gaussian density problem? \n\n2) Have the authors tried other strategies apart from a colour MLP? Perhaps the SH are also somehow introducing some bounds/weird gradient updates. Here's a thought: in NeRF if you bound your radiance when it shouldn't be, your density would also become heavily affected by the radiance (as it seems is happening in your case without the colour MLP), perhaps it could be something similar going on? How different is the per channel scaling behaving with/without the colour MLP?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLE3D is a novel method for real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes from RAW images, especially for nighttime scenes. It addresses the limitations of previous volumetric rendering methods by introducing Cone Scatter Initialization to improve SfM estimatied pointclouds, replacing SH with a Color MLP to represent in the RAW linear color space, and introducing depth distortion and near-far regularizations to enhance scene structure accuracy. LE3D significantly reduces training time and increases rendering speed compared to previous methods. The model is tested on the benchmark dataset of the existing RawNeRF paper.\n\nSTRENGTHS:\n1) The paper is well written and has a clear structure that aligns to the contribions which makes it easy to read and understand.\n2) The methods part is technical sound and details are described.\n\nWEAKNESSES:\n1) Instead of using the Spherical Harmonics the authors propose a view-dependence MLP and explain the change that the MLP can better represent the the linear RGB space, however it is unclear to me why the SH are less expressive than a tiny MLP in the linear RGB space. Can the authors provide more insights on this.\n2) Insufficient quantitative investigations of the new view-dependence MLP. I'd expect more evaluation of this part, e.g. the varying size of the MLP and different parameters of the SH model. Moreover it would be good to the effect of switching to MLPs on the render time.\n3) While the cone scatter initialization leads better rendering, it would be good to discuss how the additional sampling point influence the total number of gaussians. This number becomes relevant for the fps measurements and might reduce speed.\n\nQUESTIONS:\nPlease discuss the points mentioned in the weaknesses.\n\nLIMITATIONS:\nSufficiently discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aim to leverage 3D Gaussian Splatting with a few additions and changes in order to perform HDR view synthesis. The authors propose that with the addition of cone-scattering to the Structure from Motion initialization, replacement of Spherical Harmonics with a simple MLP for color representation, and extra distance regularizations, an HDR scene may be very quickly optimized to accurately render novel views in real-time. The approach is evaluated on the data used by RawNeRF (CVPR 2022).\n\nSTRENGTHS:\nThe paper proposes a reasonable set of additions in order to adapt 3DGS for the purpose of HDR scene reconstruction and view synthesis. Though the submission emerges alongside other comparably-aimed papers, the approach proposed is distinct and its authors provide sufficient evidence of the efficacy of their method.\n\n- Originality\nIt is probably not a very distant leap to apply 3DGS to avenues of work done on NeRF, though this work does go further to involve a few novel additions to the method being adapted, in order to address shortcomings that would be present otherwise. Other works with the same aim appear to be emerging at this time, though the approach taken by the authors, affecting the gaussian initialization and performing distance regularizations, appears distinct.\n\n- Quality\nWith consideration to the supplementary materials, the submission adequately covers most points of concern. Claims made are supported by the provided experimental results. Included ablation study results show qualitative and quantitative contribution for each improvement proposed by the authors.\n\nWEAKNESSES:\n- Clarity\nFor the most part, the writing is clear and straightforward. The context for previous efforts improved upon and problems being solved is well provided. Relevant equations are present and adequately described. Implementation and environment details are well expanded upon in the supplementary materials. However, there are a small number of typos remaining in the submission and supplement. Please make sure to correct these.\n\n- Significance\nThe paper is one of several that aim to introduce HDR Gaussian Splatting, though with regards to other work released within this month and the last, no code has been released as of yet, so no direct comparisons between results may be made. However, the stated improvements relative to these other works appear to come out on top.\n\nQUESTIONS:\nIt is not explicitly written, but could it be appropriate to briefly address directly the tradeoff in FPS/Training time with accuracy in regards to the comparison between the proposed method and compared 3DGS methods?\n\nLIMITATIONS:\nThe supplementary material sufficiently addresses the limitations and potential negative societal impact of the submitted work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}