Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis
Authors: Xin Jin
Pengyi Jiao
Zheng-Peng Duan
Xingchao Yang
Chongyi Li
Chun-Le Guo
Bo Ren
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Gaussian Splatting
RAW images
Computational Photography
Computer vision.
Primary area: machine_vision
TL;DR: Fast training and real-time rendering for HDR view synthesis from noisy RAW images using 3DGS-based techniques.

Abstract: Volumetric rendering-based methods, like NeRF, excel in HDR view synthesis from RAW images, especially for nighttime scenes. They suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in dis- tant views; 2) the limited representation capacity of the spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for down- stream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous vol- umetric rendering-based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://srameo.github.io/projects/le3d.

Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis
Xin J
1.2+ Pengyi Jiao 1
Zheng-Peng Duan'
Xingchao Yang²
Chongyi Li
Chun-Le Guo Bo Ren
VCIP, CS, Nankai University
MEGVII Technology
bttpa: Sespa://rrean //wreeeeeeeeeee/eeeeeee/oee/eee gitbub. po/proo.ctta/eddd
Noisy RAW
Exposure Varistion
Changing WB, HDR
LE3D (Ours)
GPUNH
160 FPS
RawNeRF 3i
RawGS (Bascline)
LESD (Ours)
(c) Focus Foreground
(d) Changing WB, Focus on Background
Figure 1: LE3D reconstructs 3DGS representation of scene from a set of multi-view noisy RAW images. As shown on the left, LE3D features fast truining and real-time rendering capabilities com- pared to RawNeRF [36). Morcover, compared to RawGS (a 3DGS [25] we trained with RawNeRF's strategy), LE3D demonstrates superior noise resistance and the ability to represent HDR linear colors. The right side highlights the variety of real-tima downstream tasks LE3D can perform, including (a) exposure variation, (b, d) changing White Balance (WB), (b) HDR rendering, and (c, d) refocus.
Abstract
Volumetric rendering-basee based methods, like NeRF, excel in HDR view synthesis from RAW images, especially for nighttime scenes. They suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor tant views; 2) the limited representation capacity of the spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such refocusing. To address these issues, propose L E3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM and replaces SH with Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for down- stream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering. refocusing, and can-mapping changes. Compared to previous vol- umetric rendering-batt based methods, LE3D reduces training time to % and improves rendering speed by up 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://srameo, http:///rameoggihub..ooo /utbub.io/projees//e33d
couurr--fom-motiio (SIM) estimation in dis-
"Equal Contribution. This project is done during Xin Jin's internship at MEGVII Technology.
tc. L. Guo and B. Ren are corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Since the advent of Neural Radiance Fields (NeRF) [77,, novel view synthesis (NVS) has entered a period of vigorous development, thereby advancing the progress of related applications in augmented and virtual reality (AR/VR). Existing NVS technologies predominantly utilize multiple well-exposed and noise-free low dynamic range (LDR) RGB images as inputs to reconstruct 3D scenes. This significantly impacts the capability to capture high-quality images in environments with low light or high contrast, such nighttime settings or areas with stark lighting differences.. These scenarios typically necessitate the use of high dynamic range (HDR) scene reconstruction techniques.
Existing HDR scene reconstruction techniques primarily fall into two categories and all are based on volumetric rendering: 1) using multiple-exposure LDR RGB images for supervised training 23 and 2) conducting training directly on noisy RAW data [36 The first type of method is typically highly effective in well-lit scenes. However, in nighttime scenarios, its reconstruction performance is constrained due to the impact of the limited dynamic range in sRGB data [6]. While RAW data preserves more details in nighttime scenes, it is also more susceptible to noise. Therefore, RawNeRF R6] is proposed to address the issue of vanilla NeRF's lack of noise resistance. However, RawNeRF suffers from prolonged training times and an inability to render in real-time (a common drawback of volumetric rendering-base based methods). This significantly limits the application of current scene reconstruction techniques and HDR view synthesis. Enabling real-time rendering for HDR view synthesis is key step towards bringing computational photography to 3D world.
Recently, 3D Gaussian Splatting (3DGS) has garnered significant attention based on its powerful capabilities in real-time rendering and photorealistic reconstruction. 3DGS utilizes Structure from Motion (SfM) 41 for initialization and employs a set of 3D gaussian primitives to represent the scene. Each gaussian represents direction-aware colors using spherical harmonics (SH) functions and can be updated in terms of color, position, scale, rotation, and opacities through gradient descent optimization. Although 3DGS demonstrates its strong capabilities in reconstruction and real-time rendering, it is not suitable for direct training using nighttime RAW data. This is primarily due to 1) the SIM estimations based nighttime images often inaccurate, leading to blurred distant views or the potential emergence of floaters; 2) the SH does not adequately represent the HDR color information of RAW images due to its limited representation capacity; and 3) the finally reconstructed structure, such as depth map, is suboptimal, leading to unsatisfactory performance in downstream tasks like refocus.
To make 3DGS suitable for HDR scene reconstruction, we introduce LE3D that stands for Lighting Every darkness with 3D-GS, addressing the three aforementioned issues. First, to address the issue of inaccurate SIM distant view estimation in low-light scenes, we proposed Cone Scatter Initialization to enrich the COLMAP-estimate
SIM. It performs random point sampling within cone using the estimated camera poses. Second, instead of the SH, we use a tiny MLP to represent the color in RAW linear color space. To better initialize the color of each gaussian, different color biases are used for various gaussian primitives. Thirdly, we propose to use depth distortion and near-far far regularization to achieve better scene structure for downstream tasks like refocusing. As shown in Fig. (left), our LE3D can perform real-time rendering with only about 1.5 GPU hours (99% less than RawNeRF (36)) of training time and at rate of 100 FPS (about 4000x faster than RawNeRF (36)) with comparable quality. Additionally, it is capable of supporting downstream tasks such as HDR rendering, refocusing, and exposure variation, shown in Fig. (right).
In summary, we make the following contributions:
We propose LE3D, which can reconstruct HDR 3D scenes from noisy raw images and perform real-time rendering. Compared with the NeRF-based methods, LE3D requires only % of the training time and has 4000 render speed.
To address the deficiencies in color representation by vanilla 3DGS and the inadequacies of SIM estimation in nighttime scenes, we leverage Color MLP with primitive-aware bias, and introduce Cone Scatter Initialization to enrich the point cloud initialized by COLMAP.
To improve the scene structure in the final results for achieving better downstream task performance, we introduce depth distortion and near-far regularizations.

---

2
Related work
Novel view synthesis Since the emergence of NeRF [37], NVS has gained significant ad adaancmeen. NeRF employs an MLP to represent both the geometry of the scene and the view-dependent color. It utilizes the differentiable volume rendering (28], thereby enabling gradient-descent training through multi-view set of 2D images. Subsequent variants of NeRF 2 B 22) have extended NeRF's capabilities with anti- anti-aliasing features. To overcome the deficiencies of vanilla NeRF in geometry reconstruction, strategies such as depth supervision [10] 12] and distortion loss [3] have been introduced into NeRF. Several methods [6] work on low-light image enhancement with NeRF, and extended its application scenarios. Some methods [ 7 16 39 have conducted experiments with feature-grid based approaches to enhance the training and rendering speeds of NeRF. Although these methods have achieved relatively promising results in novel view synthesis, the training and rendering speeds are still significant bottlenecks. This issue is primarily due to the dense sampling inherently required by volume rendering.
Recently, the advent of 3D Gaussian Splatting NVS methods. 3DGS represents scene using a collection of 3D gaussian primitives, each endowed with distinct attributes. Some subsequent works have added anti aliasing capabilities 3DGS repre- sentations 32 43 52 others have enhanced 3DGS representation capabilities through supervision in the frequency domain [33] DNGaussian 29 proposed a depth-regularizzd framework to optimize vierse-view 3DGS, and other works also relying on depth supervision [ 27.. Additionally, some have focused on applying 3DGS to dynamic scene representation. However, these methods only accept LDR sRGB data for training, and thus cannot reconstruct the scene's s HDR radiance. This limitation suggests they cannot perform downstream tasks such as HDR tone mapping and exposure variation. In contrast, LE3D is specifically designed to reconstruct the HDR representation of scenes from noisy RAW images.
25 has marked a significant advancement in real-time
works [30 34 49)
HDR view synthesis and its applications HDR typically refers to concept in computational photography that focuses on preserving as much dynamic range as possible to facilitate more post- processing options I1 14 20 24 [3 The existing HDR view synthesis techniques bear a striking resemblance to the two main approaches in 2D image HDR synthesis: 1) Direct use of multiple- exposure LDR images compute the camera response function (CRF) and synthesize an HDR image [I]. This corresponds to HDR-NeRF [23] which employs an MLP to learn the CRF. 2) Acquisition of frese-free underexposed RAW images, utilizing the characteristics of the linear color space in RAW to manually simulate multiple-exposure images, and synthesize an HDR image. This corresponds to RawNeRF (36), which learns a NeRF representation of RAW linear color space with noisy RAW images to perform both denoising and NVS. VR-NeRF(5O) performs a perceptual color space that enables perceptual optimization of high dynamic range images using. Although both methods achieve impressive visual results, the dense sampling required by volume rendering still poses a bottleneck for both training time and rendering efficiency.
LE3D follows the same technical approach as RawNeRF, reconstructing scene representations from noisy RAW images. This means that LE3D does not necessarily require training data with multiple exposures, significantly broadening its range of applications. However, a key distinction of LE3D is its use of differentiable rasterization techniques (25 26, 44), which enable fast training and real-time rendering. Based on a 3DGS-like representation of the reconstructed scene, LE31) can perform real-time HDR view synthesis. This is novel attempt to introduce computational photography into the 3D world, as it enables real-time reframing and post-processing (changing white balance, HDR rendering, etc. as shown in Fig.
3 Preliminaries
3D Gaussian Splatting 3D Gaussian Splatting renders detailed scenes by computing the color and depth of pixels through the blending of many 3D gaussian primitives. Each gaussian defined by its center in 3D space μ4 E IR3, scaling factor E IR3, a rotation quaternion qa E IR4, and additional attributes such as opacity o and color features ft. The basis function of a gaussian primitive given by Eqn. (I) that incorporates the covariance matrix derived from the scaling and rotation parameters.
G(x) = exp (-ble- ~ (x= 1
(1)
3

---

Randered RAW
RAW
COLMAP Init.
D
Color MLP
Cone
Noisy Images
Scatter Init.
R Rist
F
3D-GS Init.
Rff
Training
Rentering Tennination
Figure 2: Pipeline of our proposed LE3D. 1) Using COLMAP to obtain the initial point cloud and camera poses. Employing Cone Scatter Initialization enrich the point clouds of distant scenes. 3) The standard 3DGS training, where we replace the original SH with our tiny Color MLP to represent the RAW linear color space. 4) We use RawNeRF's weighted L2 loss (Eqn. 3 as image-level supervision, and our proposed Raist (Eqn. 8 as well as Rnf (Eqn. (9) as scene structure regularizations. In this context, fi. b, and & respectively represent the color Teature, bias, and final rendered color of each gaussian Similarly, os, ri. 8i. and denote the opacity, rotation, scale, and position of them.
During rendering, the color of pixel is determined by blending the contributions of multiple gaussians that overlap the pixel's location. This process involves decoding the color features fa color 6₁ by the SH, and calculating a of each primitive by multiplying its opacity o, with its projected 2D gaussian D on the image plane. Unlike traditional ray sampling strategies, 3D Gaussian Splatting employs optimized rasterizer gather the relevant gaussians for rendering. Specifically, the color C' is computed by blending N ordered gaussians overlapping the pixel:
C
Σ C ag II al--yy az); where a; =G G? Oç.
HDR view synthesis with noisy RAW images RawNeRF 336 a powerful extension of NeRF, specifically addresses the challenge of high dynamic range (HDR) view synthesis with noisy images. Different from LDR images, the dynamic range in HDR images can span several orders of magnitude between bright and dark regions, resulting in the NeRF's standard 1.2 loss being inadequate. To address this challenge, RawNeRF introduces a weighted L2 loss that enhances supervision in the dark regions. RawNeRF applies gradient supervision on tone curve & log(y + e) with = 10 and uses & (y + -1 as the weighting term on the L2 loss between rendered color úJ and noisy reference color yı- Applying the stop-gree sap-grrdientt function sg(-) y, the final loss can be expressed as:
(v. Σ
(g( +
Moreover, RawNeRF employs a variable exposure training method to take advantage of images with varying shutter speeds. The method scales the output color in linear RGB space by learned factor 88. for each color channel e with each unique shutter speed t,. In particular, the c-th channel of the output color of will be mapped into min(1ī 4; 38
vf,;11 1) as the final output for rendering.
Proposed method
The pipeline of our LE3D is shown in Fig. 2 Our main motivations and solutions are as follows: 1) address the issue of COLMAP's inadequacy in capturing distant scenes during nighttime, we utilize the proposed Cone Scatter Initialization to enrich the point cloud obtained from COLMAP. 2) Experiments show that the original SH in 3DGS is inadequate for representing the RAW linear color space (as shown in Fig. 4 (e) and Fig. 7 Therefore, we replace it with tiny color MLP. 3) To enhance the scene structure and optimize the performance of downstream tasks, we propose the depth distortion Rulist and naa Ray regularizations.

---

4.1 Improvements to the vanilla 3DGS representation
Directly applying 3DGS on noisy RAW images set faces the aforementioned two challenges, lack of distant points and inadequate representation of RAW linear color space. To address them, we propose the following improvements to the vanilla 3DGS representation.
To enrich the COLMAP -iiitiiiiiillzzed point cloud S = {s,) with distant scenes, we estimate the position and orientation all cameras. Based on this, we randomly seatter points within predefined viewing frustum F. To define F, we need to determine: 1) The viewpoint p: 2) The viewing direction ni; 3) The field of view θ; and 4) The near and far planes, = &n and 2,, respectively. For forward faaiing-anng scenes, the viewing direction averaging the orientations of all cameras, represented as = avg }. To encompass all visible the maximum value of FOV from all cameras,
Cone Seatter Initialization
be easily determined by
areas in space from the training viewpoints, denoted as = max[6% }. Additionally, F needs to include all the camera origins {Pf) to ensure complete coverage of the scene from all perspectives. It means that F should encompass circle with its center at p = (p.) radius r = max ( { - p p(ll)) and perpendicular to ni. Therefore, can establish FF:
ñ
p
ñ avg J. θ (f))
Lan (22)
|
min pll2)
Ef A= max [|s Pll2).
For near 2n and far 2/. we use the distance from the nearest and X times the distance from farthest points in the COLMAP -iittiaizze point cloud S to p to represent them, respectively. Subsequently. we randomly scatter points within our viewing frustum F = (p, n, O 2n, 2f } to obtain our enriched point cloud S' = S U SF,, where S is the scattered point set. Then is used for initialization of the gaussians, instead of S.
Color MLP with primitive-aware bias To address the issue that SH could not adequately represent the RAW linear color space, replace it with a tiny color MIP Fg. Each gaussian is initialized with a random color feature f; and a color bias 6,. To initialize 6, we project each s E S' onto every training image, obtaining a set of all pixels {Cpix]+ & for each point i.. The color features fi is concatenated with the camera pose v, and then it is fed into the tiny color MLP F9 to obtain the view-dependent color. Since the HDR color space theoretically has no upper bound on color values, we use the exponent function as the activation function for Fe to simulate it. The final color is:
C: exp (fi, v) + bx), where (
1:0) (0(,,
(5)
where f!0) is sampled from a gaussian distribution o0, and 6,0h is setted by the log value of the average of (Cpix):. This initialization makes (0) close to avg (rg((pix): J. Both fi and b, are learnable parameters and during cloning and splitting, they are copied and assigned to new gaussians.
4.2 Depth distortion & near-far regularizations
Scene structure is crucially important for the downstream applications of our framework, particularly the tasks such as refocusing. Therefore, we propose depth distortion and near-far regularizations to enhance the ability of 3DGS to optimize scene structure. Borrowing from NeRF-based methods (3). we use a depth map and weight map to regularize the scene structure.
Depth and weight map rendering Recently, several based works 29 g
employ some form
of supervision on depth. Also, depth maps are crucial for downstream tasks such as refocus (Sec. (
mech extraction [19] and relighting [1 (4). They are achieved by obtaining the rendered average depth map d in the following manner:
d E, 2fw where [o vf, 2/1T = W(z,, Vi. 21 + t, and w, a II (1-ay). E,u
(6)
where d denotes the depth map, wi represents the blending weight of the i-th gaussian, [2,, Vis = T and [x.;, yf, = represent the position in the world and the camera coordinate system, respectively, and [W, 4) corresponds to the camera extrinsics. The pixel values the Weight Map each describe histogram H of the distribution on the ray passing through this pixel. Similar to Mip-NeRF 360 (3).
5

---

we can optimize the scene structure by constraining the gaussian primitives on each ray to be more concentrated. To obtain the Weight Map, we first need to determine the distances to the nearest and farthest gaussian primitives from the current camera pose pº, represented as 2 #j, respectively. Subsequently, we transform the interval [= zf) to obtain K' intersections, where the k-th intersection is denoted as [tg, tx+1). ). Thus, the k-th value in the histogram H(k)) can be obtained through rendering in the following manner:
H(k)= = Σ (2 k)ws, where (2 k)) = {
of E [tx, tx-1)
else
Rendering H. is essential, as it is effective not only in regularization but also plays a role in the refocusing application.
Proposed regularizations Inspired by Mip-NeRF 360 B3, proposed similar depth distortion regularization Rdist to concentrate the gaussians on each ray:
Rdist Σ H()))(()) U.V
tu + tu+1
ty
t++1
(8)
2
2
Raist constrains the weights along the entire ray to either approach zero or be concentrated at the same intersection. However, in unbounded scenes of the real world, the distances (2 =///K between each intersection are vast. Forcibly increasing the size of K to reduce the length of each intersection also significantly increases the computational burden. This means that our Roist can only provide relatively coarse supervision for the gaussians on each ray, primarily by constraining them as much as possible within the same intersection.
To further constrain the concentration of gaussians, we propose the Near-Far Regularization Ray- R.f enhances the optimization of scene structure by narrowing the distance between the weighted depth of the nearest and farthest M gaussians on each ray, where the farthest refers to the last M gaussians when the blending weight approaches First, we extract two subsets of gaussians, and F, which respectively contain the nearest and farthest M gaussians on each ray. Subsequently. we render the depth maps for both subsets (dN, dF)) as well as the final blending weight map (T T.). The blending weight map T is the sum of each Wi. And here comes R..f
(9) not only can prune the gaussians at the front or back of each ray through opacity supervision when there is significant disparity between them (relying on the TN T term). Compared Ruist Ru, can also supervise the position of the first and last M gaussians on each ray to be as close as possible (relying on the |dN dd term). Besides the weighted L2 loss L and proposed regularizations Rdist and R.y,, we also introduce constraints on the final blending weights T. Given that the LE3D is tested in real-world scenarios, T should ideally approach meaning that all pixels should be rendered. Thus, we propose RĄ = log(T + e) to penalize the pixels where T is less than 1.
Ras = TNN T d
Experiments
5.1 Implementation details
Loss functions and regularizations In our implementation, the final loss function is:
== C Rr + Ratet + Anf Rnf, where L is the weighted L2 loss, and Rr, Rdiet, and Ray are the proposed T, depth distortion, and near-far regularizations, respectively.
(10)
Optimization We 'y 10 to enrich the COLMAP-initialle point cloud in distant views. Xr, Adist, Anf in the loss function are set to 0.01, 0.1, 0.01 respectively. For our color MLP Fø. we use the Adam optimizer with an initial learning rate of 1.0e 4. The initial learning rates for color features and biases for each gaussians are set to 2.0e 3 and 1.0e - 4, respectively. The learning rates for all three decrease according to cosine decay strategy to a final learning rate of 1.0e 5. Besides the color MLP, primitive-aware color bias, and the color features for each gaussians, other settings are the same as those 3DGS (55. For scenes captured with multiple exposures, we employ the same multiple-cxposurr training strategy as RawNeRF [36].

---

Table 1:
Quantitative results on the test scenes of the RawNeRF [36] dataset. The best result in bold whereas the second best one is in underlined. TM indicates whether the tone-mapping function can be replaced for HDR rendering. For methods where the tone-mapping function can be replaced, the metries on sRGB are calculated using LDR tone-mapping for fair comparison. The FPS measurement is conducted at 2K (2016x 1512) resolution. Train denotes the training time of the method, measured in GPU LE3D achieves comparable performance with previous volumetric rendering based methods (RawNeRF [6]), but with x faster rendering speed.
RAW
sRGB
Method
TM
FPSt
Train
PSNRT
SSIM
PSNR1
SSIMT
LPIPS.
IDR-NeRF
37
X
0.007
13.66
20.0391
0.5541
0,5669
LDR-3DGS 25
X
153
0.75
20.2936
0.5477
0.5344
HDR-3DGS 25
V
238
0.73
56.4960
0.9926
20.3320
0.5286
0.6563
RawNeRF 36
0.022
129.54
58.6920
0.9969
24.0836
0.6100
0.4952
RawGS (Baseline)
176
1.05
59.2834
0,9971
23.3485
0,5843
0,5472
LE3D (Ours)
103
1.53
61.0812
0.9983
24.6984
0.6076
0,5071
5.2 Datasets and comparisons
Datasets We evaluated LE3D's performance on the benchmark dataset of RawNeRF. It includes fourteen scenes for qualitative testing and three test scenes for quantitative testing. The three test scenes contains 101 noisy images and clean reference image merged from stabilized long exposures. However, the training data are captured with short exposures, leading to exposure inconsistencies. Therefore, we apply the same affine alignment operation as RawNeRF before testing (detailed in Sec. A. of the supplementary materials). All images are 4032 x 3024 Bayer RAW images captured by an iPhone X, saved in DNG format.
Baseline and comparative methods We compare two categories methods, 3DGS- based methods, and NeRF-based methods. The baseline we compare against is RawGS which uses vanilla 3DGS for scene representation and employs the weighted L2 loss and multiple exposure training proposed in RawNeRF [36]. Besides, remove the SSIM loss, its computation over local neighborhoods disrupts the noise model present in the RAW data, Additionally, we compare LDR-GS and HDR-GS, both of which are vanilla 3DGS trained
post-processed LDR images and unprocessed RAW images, respectively. The NeRF-based methods include RawNeRF [36] and LDR-NeRF. LDR-NeRF is vanilla NeRF (37] trained on the post-process processed LDR images with L2 loss.
Quantitative evaluation Tab. I has shown the quantitative comparisons the RawNeRF [36] dataset. Although NeRF-basee based methods have long training times and slow rendering speeds, they exhibit good metries sRGB. This indicates that the volume rendering they rely has strong noise resistance (mainly due to the dense sampling on each ray). In contrast, 3DGG-based methods have inferior metrics compared to RawNeRF, due their sparse resistance. Additionally, the splitting of gaussians depends on gradient strength, and supervision using noisy raw images affects this process, leading to incomplete structure recovery. LE3D achieves better structure reconstruction suitable for downstream tasks through supervision on structure, depth distortion, and near-far regularizations, comparable to the previous volumetric rendering- based method, RawNeRF (36), in both quantitative and qualitative aspects. However, requires only 1% of the training time and achieves a 3 6000x rendering speed improvement.
representation and poor noise
detailed in Sec. Note that the results of LE3D
Qualitative evaluation Fig. 3 has shown the qualitative comparisons on the RawNeRF [3] dataset. We selected four scenes for comparison, including two indoor scenes and two outdoor scenes. The data for the first two scenes were collected with a single exposure, while the data for the latter two scenes included multiple exposures. Compared to 3DGS based methods, LE3D demonstrates stronger noise resistance, particularly in the first two scenes. Additionally, LE3D achieves better results in distant scene reconstruction. For example, the second scene, LE3D produces a smoother sky compared to RawGS, and in the fourth scene, LE3D recovers distant details more sharply. Compared to RawNeRF, LF3D typically produces smoother results while still effectively preserving details. Most importantly, LE3D offers faster training times and rendering speeds.
7

---

Training View
LDR- NRRF
LDR-GS 25
RawNeRF [6
RawGS (Baseline) LE3D (Ours)
FPS: 0,007
FPS:: 1866.
EPS: 0,025
EPS. 225
50000
FPS: 0.007
FPS: 256.0
FPS: 0.023
FPS: 137.1
EPS: 1104
FPS: 0.944
FIS 21
2015-
FFS: 0.007
FPS: 128.4
FFS: 0.023
FPS: 135.8
FPS: $0.7
Figure 3: Visual comparison between LE3D and other reconstruction methods (Zoom-in for best view). The training view contains two parts: the post-processed RAW image with linear brightness enhancement (up) and the image directly output by the device (down). By comparison to the 3DGS- based method, LE3D recovers sharper details in the distant scene and is more resistant to noise. Additionally, compared NeRF-based methods, LE3D achieves comparable results with 3000 6000 improvement in rendering speed.
5.3 Ablation studies
Cone Scatter Initialization (CSI) In low-light environments, COLMAP struggles to obtain a high-quality sparse point cloud. Although 3DGS demonstrates its robustness the quality of the initial point cloud, it still encounters difficulties achieving optimal geometric reconstruction within insufficient initialized areas. can be observed from Fig. 4 (b) that the methods without CSI tend to generate gaussians at incorrect depths and lack fine details. Conversely, CSI extends the depth coverage of the scene, enabling 3DGS generate gaussians at relatively accurate depths and exhibit superior detail representation. A comparative analysis between Fig. (a) and Fig. (b) suggests that our initialization technique plays a pivotal role in achieving accurate and detailed 3D reconstruction.
(b)
Color MLP Replacing SH with Color MLP not only enhances the expressiveness of our model but also introduces greater stability during the optimization process. Fig. (e) reveals that the method employing SH rather than Color MLP exhibits strange color representations early in the training phase, due to the inability SH to adequately represent the RAW linear color space. Although the rendered image may appear similar to those produced by the LE3D, the underlying issues have significantly affected the final structural reconstruction, as depicted in Fig. (c).
Regularizations Superior visual effects in 3D are contingent upon a robust 3D structure reconstruc- tion. which in turn significantly enhances the performance of downstream tasks such as refocusing. To this end, we implement depth distortion regularization Raist and near-far regularization R. to constrain the gaussians, ensuring their aggregation the surfaces of objects and thereby improving the quality of structural reconstruction. Fig. (d) underscores the substantial enhancement our proposed regularizations provide in reconstructing the 3D structure of scenes.
6
More applications
Refocus Structural information crucial for tasks like refocusing. As discussed in Sec. 5.3 LE3D benefits from the inclusion of depth distortion and hear-far regularizations, which enhances its ability to learn structural details. As shown Fig. [5] (b. d), LE3D achieves more realistic refocusing effects due to its superior structural information, as reflected in the depth shown in (c). Conversely, RawGS suffers from foreground and background ambiguity refocusing due to lack structural information. The detailed refocusing algorithm will be released in the supplementary materials.

---

MI
Colok MLP
Training View
(a) LE3D
(b) w/o CSI
(c) wio Color MI.P
(d) w/o Regs
(e) Results at 7k iter
Figure 4: Ablation studies on purposed methods (Zoom-in for best view). CSI in (b) and Regs in (d) denote Cone Scatter Initialization and Regularizations, respectively. (e) shows the rendering result of LE3D w/ or w/o Color MLP in the early stages of training.
(a) LE3D output
(b) LE3D refocus
(c) LE3D output depth
(d) RawGS refocus
(e) RawGS output depth
Long
Lovil Treerrapping
Medium
Sboet
Kiet Raibs
(g) Changing Tonc-mapping, Viewpoint, and Focus Figure 5: LE3D supports various applications. RawGS* in (d) denotes using LE3D's rendered image and RawGS's s structure information as input for refocusing. (c, e) are the weighted depth rendered by LE3D and RawGS, respectively. (f) shows the same rendered by LE3D with different exposure settings. In (g), the "-," denotes global tone-mapee one-mapping, while the represents local tone-mapping.
(f) Exposure Variation
Exposure variation and HDR tone-mapping LE3D can easily achieve exposure variation and recover details from overexposed data, as shown in Fig. 5 (f). Fig. [ (g) showcases the various tone-mapping methods LE3D can implement, including global tone-mapping, such color tem- perature and curve adjustments, and local tone- tone-mapping using our re-implemented HDR+ 20 (implementation details will be released in the supplementary materials).
Although RawNeRF (36 can also perform similar applications, its inability to achieve real-time rendering significantly limits its use cases, such as real-time editing described in Sec.
7 Conclusion
To address the long training times and slow rendering speeds of previous volumetric rendering-basee methods, we propose LE3D based on 3DGS. Additionally, we introduce Cone Scatter Initialization and a tiny MLP to represent color in the linear color space. This addresses the issue of missing distant points in nighttime scenes with COLMAP initialization. also replaces spherical harmonies with the tiny color MLP, effectively representing the linear color space. Finally, we enhance the structural reconstruction with proposed depth distortion and near-far regularization, enabling more effective and realistic downstream tasks. Benefiting from the rendering images in the linear color space, LE3D can achieve more realistic exposure variation and HDR tone-mapping in real time, expanding the possibilities for subsequent HDR view synthesis processing.

---

Acknowledgement
This work is funded by National Natural Science Foundation of China (No. 62225604, No. 62272245, No. 62306153), the Fundamental Research Funds for the Central Universities (Nankai University. No. 63243147), and Tianjin science and technology projects (2JJCYBJ001270) The computational devices of this work is supported by the Supercomputing Center of Nankai University (NKSC).
References
Jonathan T Barron, Andrew Adams, YiChang Shih, and Carlos Hernnnde.. Fast bilateral-space stereo for synthetic defocus. In CVPR, 2015.
[2] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[4] Yuanhao Cai, Zihao Xiao, Yixun Liang, Yulun Zhang, Xiaokang Yang. Yaoyao Liu, and Alan Yuille. Hdr-gs: Efficient high dynamic range novel view synthesis at 1000x speed via gaussian splatting, 15555 2024.
[5] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning see in the dark. In CVPR, 2018.
Zhang Chen, Zhong Li. Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. Neurbf: neural fields representation with adaptive radial basis functions. In ICCV, 2023.
Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaussian splatting few-shot images. arXiv:2311.1339) 2023.
[9] Ziteng Cui, Lin Gu, Xiao Sun, Xianzheng Ma, Qiao, and Tatsuya Harada. Aleth-nerf: Illumination adaptive nerf with concealing field assumption. In AAAI, 2024.
[10] David Dadon, Ohad Fried, and Yacov Hel-Or. Ddnerf: Depth distribution neural radiance fields. In WACV, 2023.
[11] Paul E Debevec and Jitendra Malik. Recovering high dynamic range radiance maps from photographs. Siggruph, 1997.
[12] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, 2022.
[13] Paul HC Eilers and Brian D Marx. Flexible smoothing with b-splines and penalties. Statistical science, 1996.
[14] Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K Mantiuk, and Jonas Unger. Hdr image reconstruction from single exposure using deep cuns. ACM TOG, 2017.
[15] Zcev Farbman, Raanan Fattal, and Dani Lischinski. Convolution pyramids. ACM TOG, 2011.
[16] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjao Kanazawa. Plenoxels: Radiance fields without neural networks. CVPR, 2022.
[17] Jian Gao, Chun Gu, Youtian Lin. Hao Zhu, Xun Cao. Zhang, and Yao Yao. Relightable gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing. arXiv:2311. 2023.
[18] NeRF Studio Group. Viser.
2023.
[19] Antoine Guédon Vincent Lepetit. Sugar: Surface-aligned gaussian splatting efficient 3d mesh reconstruction and high-quality mesh rendering. CVPR, 2024.
[20] Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Mare Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM TOG, 2016.
10

---

[21] Tanner Helland. How to convert lemperature (k) to rgb: Algorithm and sample code. //tannerhelland.
https
code. html
2012.
[22] Wenbo Hu, Yuling Wang. Lin Ma, Bangbang Yang. Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: mri- representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023.
[23] Xin Huang, Qi Zhang, Ying Feng. Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In CVPR, 2022.
[24] Nima Khademi Kalantari, Ravi Ramamoorthi, et al. Deep high dynamic range imaging of dynamic scenes. ACM TOG, 2017.
[25] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023.
[26] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. Point-based neural rendering with per-view optimization. In Computer Gruphics Forum, 2021.
[27] Pou-Chun Kung, Seth Isaacson, Ram Vasudevan, and Katherine A Skinner. Sad-gs: Shape-aligned depth-supervised gaussian splatting. In ICRA Workshops, 2024.
[28] Mare Levoy. Efficient ray tracing of volume data. ACM TOG, 1990.
[29] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dugaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. arXiv:2403.06911 2024.
[30] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. CVPR, 2024.
[31] Zhihao Li, Yufci Wang, Alex Kot, and Bihan Wen. From chaos clarity: 3dgs in the dark. arXiv:2406.0833 2024.
[32] Zhihao Liang, Qi Zhang, Wenho Hu, Ying Feng, Lei Zhu, and Kui Jia. Analytic-splatting: Anti-aliased 3d gaussian splatting via analytic integration. arXiv:2403. 1056, 2024.
[33] Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Yu Chuang, and Jia-Bin Huang. Single-image hdr reconstruction by learning to reverse the camera pipeline. CVPR, 2020.
[34] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. wxXiv:2308.09713 2023.
[35] Tom Mertens, Jan Kautz, and Frank Van Reeth. Exposure fusion. In PG. 2007.
[36] Ben Mildenhall, Peter Hedman, Ricardo
Pratul Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. CVPR, 2022.
[37] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[38] Ben Mildenhall, Dor Verhin, Pratul P. Srinivasan, Peter Hedman, Ricardo Bartin-Brualla, and Jonathan T. Barron, MultiNeRF: A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF. httpp://gitta /gthhbbtttuuu 2022.
com/ googlo wwteeeearrcmmmmtiiinee
[39] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 2022.
[40] Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. Photographic tone reproduction for digital images. In Seminal Graphics Pupers: Pushing the Boundaries, Volume 2, 2023.
[41] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-front from-motionn revisited. In CVPR, 2016.
[42] Shreyas Singh, Aryan Garg, and Kaushik Mitra. Hdrsplat: Gaussian splatting for high dynamic range 3d scene reconstruction from raw images. BMVC, 2024.
[43] Xiaowei Song. Jv Zheng, Shiran Yuan, Huan-ang Gao, Jingwei Zhao, Xiang He, Weihao Gu, and Hao Zhao. Sa-gs: Scale-adaptive gaussian splatting for training-free anti-aliasing. 2xXiv:2403.1961 2024.
[44] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super- fast convergence for radiance fields reconstruction. In CVPR, 2022.

---

[45] Neal Wadhwa, Rahul Garg, David E Jacobs, Bryan E Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attis, Jonathan T Barron, Yael Pritch, and Mare Levoy. Synthetic depth-of-field with single-camera mobile phone. ACM TOG, 2018.
[46] Haoyuan Wang. Xiaogang Xu, Ke Xu, and Rynson WH Lau. Lighting up nerf via unsupervised decompo- sition and enhancement. In ICCV, 2023.
[47] Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. https:///githb.. wtw.thhhb.ooooocooccooommm com XPixXXxxeee wxxxxxlrooupp BasicSR 2022.
[48] Kaixuan Wei. Ying Fu, Yinqiang Zheng, and Jiaolong Yang. Physics-based noise modeling for extreme low-light photography. IEEE TPAMI, 2021.
[49] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv:2310.08528. 2023.
[50] Linning Xu, Vasu Agrawal, William Laney, Tony Garcia, Aayush Bansal, Changil Kim, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Aljaž Božič, al. Vr-nerf: High-fidelity virtualized walkable spaces. In Siggraph Asia, 2023.
[51] Chongjie Ye, Yinyu Nie, Jiahao Chang, Yuantao Chen, Yihao Zhi, Xiaoguang Han. Gaustudio: modular framework for 3d gaussian splatting and beyond. D.Xi::2403.19332 2024.
[52] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splattnng Alias-free 3d gaussian splatting. CVPR, 2024.
[53] Jiahui Zhang. Fangneng Zhan, Muyu Xu, Shijian Lu, and Eric Xing. Fregs: 3d gaussian splatting with progressive frequency regularization. CVPR, 2024.
[54] Tianyi Zhang. Kaining Huang, Weiming Zhi, and Matthew Johnson-Roberson. Roberson. Darkgs: Learning neural illumination and 3d gaussians relighting for robotic exploration in the dark. arXiv:2403. 10814, 2024.
[55] Xuaner Zhang, Kevin Matzen, Vivien Nguyen, Dillon Yao, Zhang, and Ren Ng. Synthetic defocus and look-ahead autofocus for casual videography. vrXiv::905.06333 2019.
[56] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffc, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXi::/805.098/7, 2018.
12

---

Appendix
A Implementation details and more ablation studies
A.1 More detailed implementation
Detailed network architecture of the Color MLP To prevent our color MLP from occupying too much time during rendering. we used very compact MLP. It consists of only one 16-dimension hidden layer and takes a 19-dimensional input (16 for the color feature of each gaussian and for the viewing direction). With activation function ReLU, the output of the MLP is 3-dimensional vector as the final color of the gaussian primitive.
Sepcific hyperparameters The 'y in 4 to determine the farthest plane of the training frustum is set to 10. The K intersections of our weight map H. is set to 32. The M gaussian number to calculate the near, far subsets N and F if set to 5. The ÀT, Xdiet; An are set to 0.01, 0.1, 0.01, respectively. And our training and testing resolution is set to 2016 x 1512.
Experimental environments In our implementation, (24 GB VRAM) for both training and testing along with a 20-core CPU and a 64GB RAM. The Pytorch version is 1. 1.1 CUDA version is 11.8. Due the limitation of 24 GB VRAM, RawNeRF cannot be trained on a single Nvidia GeForce 3090 GPU. For the test scenes, we train RawNeRF on 4 GPUs, while for the other scenes, RawNeRF is trained with 2 GPUs. Apart from the difference in the number of GPUs, we did not make any modifications to RawNeRF.
single Nvidia GeForce 3090 GPU
Affine alignment Since all training views in the RawNeRF 36 dataset are captured with fast shutter, while the test views (ground truth) are captured with a slow shutter, linear enhancement is needed during testing for alignment. However, due color bias (non-zero-mean noise for high ISO, 48)), direct linear enhancement does not achieve perfect alignment. Therefore, affine alignment is performed on both the output and ground truth during testing. In RawNeRF [36], this process is done as the following procedure:
TV
ry 2
Cov(x, 8) Var(z) 6 5 at.
(1I)
where E is the mean of z. And x, y are the ground truth and the final output, respectively. This is the least-squares fit of an affine transform az + 6 s y. At test time, we first process y with (y- - ba,a, then calculate the metric. For those methods whose output is in RAW linear color space, the affine alignment process is only done once in the RAW color space. While for other methods (LDR-NeRF (37), LDR-31 3DGS (25) which can only output in RGB color space, the affine alignment process is done in the RGB color space.
A.2 More ablation studies
Ablation on number of gaussian with/without CSI While our Cone Scatter Initialization is used to add sparse points for distant scenes, some may be concerned it will influence the total number of gaussians, which is relevant for rendering speed. In fact, due to the optimization capabilities of 3DGS, in all scenarios, CSI not only did not increase the number of points but actually reduced them. Besides, CSI is particularly effective in the outdoor scene (-24.86% for 'bikes' and -13.13.100 100 for 'windowlegovary') ). Details be found in Tab. 2
Ablation on Color Representation In practice, we have observed that employing SH directly can lead negative color output values, an anomaly in the context of the RAW linear RGB space where such values are not permissible. Noticing that we attempted to directly apply the exponential function after the SH function, turning these negative values positive. However, this approach did not resolve all the issues encountered. For example, in the 'windowlegovary' scenario, this approach showed a maximum value above 10 on one channel while staying below 10 on others, highlighting SH's expressive limitations. Additionally, we observed color issues similar to those in the 'yuccatest' scenario, as shown in Fig. 7 Therefore, using Color MI. instead of SH to represent color is necessary in our method.
13

---

Table 2: The final number of gaussians at convergence: w/o CSI indicates no cone scatter initializa- tion, while LE3D uses full CSI. The change represents the variation in point count between LE3D and w/o CSL. Outdoor indicates if the scene is outdoors. CSI reduces the final point count in all scenarios and is particularly effective in outdoor scenes. This aligns with our motivation improve distant view reconstruction using CSI.
Scene
bikes
candle
candlefiat
gardenlights
livingroom
morningkitchen
w/o CSI
532734
55784
110624
282499
495330
279090
LE3D
400310
49402
105924
246332
428997
260249
Change (%)
-24.86%
-11.44% X
-4.25%
-12.80%
-13,39%
-6.75%
Outdoor
X
X
Scene
nightstreet
notchbush
officetest
parkstatue
pianotest
scooter
w/o CSI
377224
161850
418443
420570
300112
183443
LE3D
328822
148823
404602
336241
294808
148167
Change (%)
-12.83%
-8.05%
-3.31%
-20.05%
-1.77%
-19.23%
Outdoor
X
X
Scene
sharpshadow
stove
streetcorner
windowlegovary
yuccatest
w/o CSI
722148
378176
313398
802336
233571
LE3D
589047
363461
299916
697204
217325
Change (%)
-18.43%
-3.89%
-4.30%
-13.10%
-6.96%
Outdoor V
Ablation on each of the regularizations Fig. 6 (a.b) presents the visualization of dN and d adjacent the depth map. In an ideal scenario, both d' and dF should align with d, ensuring that the weights along each ray are concentrated at the surface. The comparison between Fig. (a,b) demonstrates that the incorporation of near-far regularization indeed encourages dN and d" to progressively align with d. This alignment results in a more refined representation of the three- dimensional structure, capturing better details of the scene's geometry. The comparison between Fig. (a,c) elucidates the adverse effects of omitting distortion regularization. Without such constraints, the model would fail to produce depth maps with natural depth progression, with artifacts such as abrupt depth discontinuities or voids on planes. Such anomalies are indicative of significant issues in the reconstruction of the scene's geometry.
Ablation studies on test scenes As shown in Tab. 3 LE3D has shown superior performance over all ablated methods. The results without Color MIP are the worst because the SH used vanilla 3DGS is not suitable for representing colors in the RAW linear color space. As shown in Fig. 7 the results without Color MLP are noticeably desaturated and appear gray. Both w/o Rnf and w/o Rdist show degraded depth, as seen in Fig. 7 Additionally, it can be observed that w/o Color MLP also has poor structural information. This is mainly due to instability during the early stages of training, leading to suboptimal depth map reconstruction, discussed in Sec. 5.3
The stability of LE3D Due to the random initialization of our Color MLP, we trained 9 versions of LE3D using 9 different random seeds to test the stability of LE3D. We then compared their metrics on the test set, as shown in Fig. g It can be observed that the stability of LE3D is remarkably high, and the overall fluctuations do not impact the experimental conclusions.
B Detailed postprocessing
B.1 Refocus
To implement the refocusing algorithm in LE3D, we adopted refocusing approach similar to
55 Most these methods require pre-caleulating a MultiPlane Image (MPI) representation 566 from the depth to synthesize refocus data. The MP1 representation consists of multiple planes arranged from front to back, with each plane heing an RGBA image (the RGBA image's color is still in the linear color space for LE3D). RawNeRF [36) pre-calculates the
previous methods [1 45
14

---

Table 3:
Quantitative results of the ablation studies. Notice that, since Cone Scatter Initialization (CSI) is used to supplement the point cloud in distant scenes, and the test scenes do not contain distant views (all being indoor scenes). LE3D does not apply CSI in this context. The ablation study of CSI can be found in Fig. | (a, b), which shows significant differences in distant views. Best results is denoted in bold. The rank is indicated in the lower right corner of each metric.
RAW
sRGB
Method
PSNR1
SSIM
PSNRT
SSIMI
LPIPS
w/o Color MLP
59.4483
0.9969 (4)
23.1884(4)
0.5862
0.5635
w/o Rdist
60.5202(3)
0.9981
24.3615
0.6007(3)
0.5087(2)
w/o Rnf
60.7144(2)
0.9982(2)
24.5705
0.6043
0.5096(3)
LE3D (Ours)
61.0812(1)
0.9983(1)
24.6984(1)
0.6077 (1)
0.5071
CN
d
LE3D Results
(a) LE3D, d d
(c) w/o Raise
d
Training View
(b) w/o Ray. d # dN * d
(d) w/o Reistr Res
Figure 6: Ablation on each of the regularizations. Since both Ruistt and Ras are regularization terms intended strengthen the structural representation, we have elected to display only the depth map for the sake of clarity. In addition, to demonstrate the effect of Rey in aligning d, dN d we have also visualized dN and d as mentioned in Sec. 4.2
MPI representation from its rendered results and then uses it for refocusing. However, due to its extremely slow rendering speed (about 50 seconds per image), real-time refocusing with RawNeRF is nearly impossible. To obtain our MPI representations in real time, we use weight distortion map (H E X where K denotes the plane number) rendered by LE3D as described in Eqn. (7) to directly render the final opacity for each plane in the MPI. The color image (I Hx3xHxW) used for synthesis can also be directly rendered by LE3D. Thanks to the rendering speed of 3DGS, both I and H. can be rendered in real time, enabling rapid refocusing.
After obtaining the color image / and the weight distortion map H. we can perform refocusing using the algorithm described in Algorithm | similar to the method in [36 55). Here, foocu represents the index of the focus plane, and A, controls the simulated aperture size (i.e., the blur intensity). blur(c, r) represents performing a gaussian blur operation on with a radius of This operation is accelerated using 2D Fourier space convolution.
Algorithm 1 Refocus Synthesis in LE3D
Require: 1, H. focus A
+ + 0
for i 1 K do
+- A, 1
0; Hi
Φi - I ai
Oblur + blur(as; r)
dwter + blur(
+ f Ublur + abburr
end for
In Fig. 9 we show the refocus result and mul- tiplane image representations of LE3D and RawGS. A more specific and global result of these two methods can be found in Fig. 5 (c) and (e), their depth maps. It can observed that the lack structural information leads to inconsisten- cies in the
return
relationship in Fig. (bg resulting in poor refocusing effects. In contrast, LF3D, with its superior structural information, achieves more realistic refocusing effects, accurately depicting the scene's 's depth relationships.
15

---

LE3D (Uur))
wio Coloe MLP
wio R
wio Rart
Ground Trudh"
Training View
Kondkaa Sandeee Depth
Visualization results for ablation studies on the test scene. The Ground Truth denotes the raw image averaged from a burst with a slow shutter to perform denoising. It can be observed that the results of w/o Color MLP show significant color degradation, while the results of w/o Rnf and w/o Rdist exhibit structural degradation.
Figure 7:
61.20
0.9984
24.70
0.508
0.510
0,9983
24.65
0.509
0.506
60.80
0.9982
34.60
0.534
39.00
0.9999
34.10
0.507
0.497
RowNeRR
Booosrap
58.75
34.05
0.606
0.495
0.505
Figure 8: The error bars of our proposed LE3D and comparison with RawNeRF 33
B.2 Tonc-mapping
Default LDR tone-mapping We use the following simplified tone-mapping function to map the RAW image to an LDR sRGB image, which similar to the ISP pipeline used RawNeRF [36]:
1. Normalizing the value of the input RAW image to [0, I].
2. Apply bilinear demosaicking the normalized RAW images.
3. Apply white balance gains for each of the three channels.
4. Perform the color correction operation with the color matrix from camera RGB-to-sRGB (combined camera RGB-to- tYZ and XYZ-to-sRGB).
5. (Optional) Exposure adjustment by setting the white level to the 97 percentile.
6. Clip the value to [0, 1), and apply the sRGB gamme correction.
LE3D is trained using the RAW images from step 2, resulting in outputs in the unmodified RAW linear color space. This means that LE3D's 's outputs more suitable for various downstream tone-mapping applications.
Global tone-mapping In our implementation, we use two kinds of global tone- tone-mapping techniques: color temperature and curve adjustment. For color temperature adjustment, we refer to Tanner Hel- land's algorithm [21]. For the curve adjustment algorithm, we adopted a control point strategy. Users first determine several control points, and then we use the "B-splines" interpolation algorithm [I3] to generate smooth curve from these control points. Since the output of LE3D is 32-bit float, we quantize it to a 16-bit unsigned integer. The curve obtained carlier is converted into look-up table (LUT) over the range [0, 65535]. The LUT is then applied to the output of LE3D to achieve the curve adjustment results. The results of our global tone-mapping can be found in Fig. 10 The color temperature adjustment is directly applied LE3D's output, while the curve adjustment performed after applying white balance and the color correction matrix.
Local HDR+ tone-mapping Our local tone-mapping is based on the HDR+ 20 algorithm. How- ever, since LE3D's s output noise-free, we have simplified some of the steps. Our simplified HDR+ algorithm is as follows:
16

---

(b) Multiplane Images with Structure of RawGS (Baseline)
RawGS Refocus
Focus At Here
LE3D output
LE3D refocus
(a) Multiplane Images with Structure of LEED (Ours)
Figure 9: Multiplane image representations of (a) LE3D and (b) RawGS, and their corresponding refocused images (Zoom-in for best view). The in (a) and (b) denotes the selected focus plane. The "-"" represents the refocus algorithm in Algorithm I can be observed that the multiplane image representations generated using RawGS structure are noticeably more "noisy" with incorrect depth relationships. This results in ambiguity in the final refocusing result regarding foreground and background relationships.
1. Apply white balance gains for each of the three channels of the LE3D 's output.
2. Perform the color correction operation with the color matrix from camera RGB-to-sRGB (combined camera RGB--o-XYZ, and XYZ-1o-RRBB).
3. (Optional) Apply the global tone-mapping curve if the curve exists.
4. Apply the additional digital gain for exposure adjustment, resulting in z.
5. Based on z, simulate multiple exposures Zev- Zevo, zev+ with different exposure times (in our implementation, we use [0.25, 1.0, 4.0)).
6. Postporcess Zea-, Zev- Zev+ to the sRGB color space, then using ExposureFusion 35 to compress the HDR results into LDR. Then we need to re-map the merged sRGB result to RAW linear color space for further postprocess. This would not cause information loss, due to all the processes are in 32-bit float format.
7. Perform "dehazing" like HDR+ 20), which aims at reduceing the effect veiling glare. This is done by allowing up to 0.1% of pixels to be clamped to zero, but only adjusting pixels below 7% of the white level.
8. Then we apply the Reinhard tone-mapping 40 and gamma compression to get the sRGB results.
9. We apply the unsharp masking technique to sharpen the image with a 3-level gaussian pyramid [15].
10. Convert the result into the HSV color space. Then shifting bluish cyans and purples towards light blue, and increasing the saturation of blues and greens generally. Finally, re-map the result to sRGB to get the final result.
Some sampled local HDR+ tone- mapping results can be found in Fig. Fig. 5
and Fig. II (c).
17

---

3000K
6500K
40000K
(a) Training View
(b) Color Temperature Curve Adjustment & Viewpoint Changing
Figure 10: Global tone-man mappinng and viewpoint changing in a single scene. The first row in (b) shows the color temperature adjustment result and the corresponding Kelvin color temperature. The second row shows the curve adjustment result the corresponding control points and the interpolated curve.
(a) Base View, 91.6FPS
(b) Render Depth, FPS
(c) Exposure Variation, 91.6 FPS
(d) Refocus, 14.1 FPS
(c) Global & Local (HDR+) TM, 35.7 FPS
(f) Novel View, 116 FPS
Figure 11: Some screenshots of interactive viewer, which can perform (b) depth rendering, (c) exposure variation, (d) refocus, (e) global & local tone-mapping, and (f) novel view rendering. The FPS is emphasized by the green bounding box, and the changed rendering parameters emphasized in red bounding box.
C Interactive viewer
Fig. |1 has shown some screenshots and downstream tasks results of our interactive viewer which built upon Viser [18]. Besides refocusing, most of the downstream tasks can be performed in real time. For refocusing, most the time spent on the gaussian blur according to Algorithm T (due to the large gaussian blur kernel size) rather than on rendering.
18

---

(a) Training View
(b) RawNeRF
(c) RawGS
(d) LE3D (Ours)
Figure 12: Failure cases for both volumetric rendering-basee and differentiable rasterization (3DGS)
based methods (Zoom-in for best view). Red bounding box highlights the failure cases.
D Limitation
Although LE3D can achieve real-time HDR view synthesis, it, like volumetric rendering- based methods (such RawNeRF [6]), encounters reconstruction failures in extremely dark scenes. For example, in the training view shown in Fig. 12 (a), RawNeRF (36) failed to reconstruct the upper left corner, leaving large artifacts. RawGS, lacking structural supervision, produced many floater gaussians, significantly degrading the reconstruction quality. Although LE3D can reduce these floaters, it, like the other methods, fails to recover the details of the leaves in the upper left corner.
The main cause of this issue the low scene brightness, resulting in a low signal-to-n toiise ratio. Potential solutions include: 1) Collecting much data possible during scene capture to enable multi-frame fusion "3D" denoising; 2) Using neural networks to denoise images before reconstruction, then using the denoised images rebuild the scene. However, ensuring consistent denoising results across frames (consistent color space) is challenging, making it difficult to guarantee effectiveness.
E Broader impacts
Potential positive impacts The development of the LE3D method could lead to improvements in virtual and augmented reality applications, providing users with more realistic and immersive experiences. 3D reconstruction capabilities may be utilized for the digital preservation of cultural heritage sites, museums, and art, allowing for greater access and study by broader audience. The capability of LE3D to reconstruct scenes in low-light environments also facilitates data collection, broadening the application scenarios for 3DGS. This advancement makes viable to employ 3DGS in situations where establishing adequate lighting conditions is challenging. Most importantly, LE3D enables real-time editing of HDR view synthesis tasks (as shown in Sec. C). This further advances the development of 3D video editors.
Potential negative impacts
The fast training reconstruction method like 3DGS could potentially facilitate the implementation of illegal surveillance activities. Enhancing the capabilities of 3DGS in low-light environments may render illicit surveillance practices more insidious, thereby posing a threat to individual privacy and the security of confidential information.
F Licenses for existing assets
Our method is implemented in the style of BasicSR (77) and we encourage further work based on this codebase. The dataset we utilized is sourced from Raw [366) For rendering the depth map, we employed the rasterizer by GauStudio [5]], with modifications made to the rasterizer by 3DGS to implement the Ray regularization. Our real-time viewer is based on V8ser[T8). For comparative analysis and reference code, used RawNeRF from MultiNeRF33 38).
G
More results
G.1
Detailed quantitative results
Since the metrics in Tab. Taee calculated after affine alignment, we also provide the metrics without affine alignment in Tab. 5 It can be observed that even without affine alignment, LE3D still ranks
19

---

Table 4: Licenses for Existing Assets.
Asset
License
Cite
Rasterizer by 3DGS
Gaussian- Spllttiin License
25
Rasterizer by GauStudio
MIT license
51
BasicSR
Apache-2.0 license
47
Viser
Apache-2 2.0 license
18
MultiNeRF P
Apache-2.0 2.0 license
38
RawNeRF datasets
Apache-2.0 license
36
"License can be found at https://githhb. httplau..ooogggapaaaaa.a http//grapphoooooo inr Aaria, harriaaagasiannna wawpaanssssaatttnl/oooooooeeo main/LICENSE. md This license mentions that licensors grant non-exclusive rights to use the Software for (both academic and industrial), free of charge, without the right sublicense.
research purpasses research att
htttp://storage hww.aatorrggggoggaaap...... googleapis.comm com/ greesarch//eefrr3300//a
'Follow MultNekF'S license.
Table Quantitative results on the test scenes in the RawNeRF [36] dataset w/o affine alignment. The best result is in bold whereas the second best one is in underlined. TM indicates whether the tone-mapping function can be replaced for HDR rendering. For methods where the tone-mapping function can be replaced, the metrics sRGB are calculated using LDR tone-mapping for fair comparison. The FPS mcasurement is conducted at 2K (2016x 1512) resolution. Train denotes the training time of the method, measured in G.UxH.
RAW
sRGB
Method
TM
FPS1
Train
PSNR1
SSIM
PSNR
SSIM
LPIPS
LDR-NeRF
37
X
0.007
13.66
14.4014
0.5541
0.7552
LDR-3DG9 25
x
153
0.75
14.7270
0.5477
0.6977
HDR-3DGS 25
/
238
0.73
56.3537
0.9924
20.4687
0.5286
0.6622
RawNeRF 33
0.022
129.54
1442
0.9956
22.3398
0,5992
0.5363
RawGS (Baseline)
176
1.05
58.7603
0.9963
22.6751
0.5810
0.5603
LE3D (Ours)
103
1.53
59.5099
0.9959
22.4391
0.5924
0.5483
among the top in overall metrics and achieves real-time rendering speeds. Compared to other 3DGS-based methods (all capable of real-time rendering), LE3D demonstrates significantly superior structural information, as shown in Fig. and Fig. enabling more downstream applications.
G.2 More qualitative results
Detailed comparisons between 3DGS-based methods and LE3D As shown in Fig. 13 LE3D achieves better structure reconstruction than our baseline RawGS (3DGS trained with RawN- eRF's loss and multiple exposure training strategy). Compared with LDR-GS (trained the LDR images) and HDR-GS (trained directly on the RAW data), LE3D achieves better color reconstruction results well better denoising ability. We also found that IDR--GS and HDR-GS have fewer reconstructed gaussians, resulting in faster rendering speeds but poor overall reconstruction quality. Additionally. LDR-GS, trained on linear brightened LDR images, shows weaker resistance color bias [48), resulting in severe color shifts the final output. We also found that the generally low values of RAW images lead insufficient gradients, reducing the number of gaussian splitting. RawNeRF's weighted L2 loss (Eqn. (3) strengthens supervision in dark areas but at the cost of structural information. LE3D incorporates both the weighted L2 loss and depth distortion and near-far regularizations to constrain the structure, ultimately achieving the best structural and visual results.
More qualitative results Fig. 7 Fig. isons between LE3D and 3DGS 25 based methods. From the figures above, evident that LE3D demonstrates superior noise resistance and color representation capabilities. Additionally, LE3D produces smoother and more aceurate depth maps, which are essential for downstream tasks like refocusing. It is worth noting that volumetric rendering- based methods, such RawNeRF (36),
Fig. Fig. 15 has shown more qualitative compar-
20

---

LF3D (Ours)
RawGS (Baselinc)
HDR-GS A5
LDR-GS A5
Greand Truih"
Renderel imge
Thaining View
Rendbred Depih
Ground Truch?
Rendened Image
Training View
Kendered Dapih
Groun Tradh"
Rondered Imngge
EPTK TOR
Trainin Vee
Rendered Devih
Figure 13: Comparison between LE3D and other 3DGS-based methods (Zoom-in -nn for best view). All
the results the direct output of each model, not being applied by affine alignment. The Ground Truth denotes the raw image averaged from a burst set with a slow shutter to perform denoising.
cannot achieve real-time rendering, which significantly limits their applications (including real-time scene editing). Therefore, we do not compare them here. For comparisons between LE3D and volumetric rendering based methods, please refer to Tab. Tab. and Fig. 3
H Concurrent work
Here list of some concurrent works that focus on HDR view synthesis based on the 3DGS-like method.
1) Training with RAW images: HDRSplat 42 uses an extra Bayer-space-denoising step to handle noise. 3DGS in the Dark B1 analysis how noise impacts the optimization of 3DGS, and purposes Noise robust reconstruction loss to handle noise.
2) Training with multi-exposure RGB images: HDR- GS 4 purposes Dual Dynamic Range (DDR) Gaussian point cloud model to jointly the HDR and LDR colors, and employs three independent MIPs to learn the tone-mapping operation in RGB channels.
Compared to those methods that need trained denoising network, L.3DD could just denoise set of noisy RAW images from single-blind device. It means we do not need to train the denoising network from scratch for every device like HDRSplat [42] and 3DGS in the Dark [31). And since our method uses high dynamic range data (RAW images) directly, so we do not need multi-exposure data for training like HDR-GS 4. which simplifies the data capture pipeline.
21

---

RawGS (Baseline)
LE3D (Ours)
LE3D, Novel View Edited
Rendered Image
Training View
Rendered Depth
Rendered Image
Training View
Rendered Depth
Rendered Image
Training View
Rendered Depth
Rendered Image
Training View
Rendered Depth
Figure 14: Comparison between LE3D and RawGS (baseline, 3DGS trained with weighted L2 loss in Eqn. 3 and multiple exposure strategy). It can be observed that LE3Dexhibits stronger noise resistance and color representation in low-light scenes. Additionally, it produces smoother and more accurate depth maps across all scenes.
22

---

RawGS (Baseline)
LE3D (Ours)
LE3D, Novel View Edited
Reidard Insu
Teirinng Viewo
Rendad Dapi
Renker Inage
Training
Rarkan Depth
Render Insea
Traiiing
Randard Deptn
Rendard Inago
Taiming View
Raxlar Depth
Figure 15: Comparison between LE3D and RawGS (baseline, 3DGS trained with weighted L2 loss Eqn. 3 and multiple exposure strategy). can observed that LE3Dexhibits stronger noise resistance and color representation in low-light scenes. Additionally. it produces smoother and more accurate depth maps across all scenes.
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We have described our motivation and contributions both in abstract and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have described our limitation in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the implementation details be found in Sec. 5.1 and Sec. A.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [No]
Justification: No reproducible code is provided in the submission. However, all the codes, both to reproduce the results and the interactive viewer, will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr ec/
public/guides. https//oooooSuuossssoooooooo///
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pp wwwppbblccgguiicuceccccceoeooopp /etggudddsse/eebbbinninn Stt.SSabmiiiioooiiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the implementation details be found in Sec. 5.1 and Sec. Sec.A.I A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The error bar can be found in Sec. A.2 and Fig.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The necessary computer resources LE3D needed can be found in Sec. A.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: 1) Our experiments and research are not with human subjects; 2) the
RawNeRF (36) dataset is public (http://storage ceatttorrge.goggllee.. googleapis com/ gresearch/
refraw360/raw ptt.:/aww.ppppp:33 3) we have discuss potential broader impacts.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Methods and models for reconstruction task is hardly been misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Licenses for existing assets are listed in Sec.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: No assets is submitted along with the submission. However, all the codes, both to reproduce the results and the interactive viewer, will be released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "4uiBSSzxfb", "reviewer": "Reviewer_Me2T", "summary": "The authors aim to leverage 3D Gaussian Splatting with a few additions and changes in order to perform HDR view synthesis. The authors propose that with the addition of cone-scattering to the Structure from Motion initialization, replacement of Spherical Harmonics with a simple MLP for color representation, and extra distance regularizations, an HDR scene may be very quickly optimized to accurately render novel views in real-time. The approach is evaluated on the data used by RawNeRF (CVPR 2022).", "questions": "It is not explicitly written, but could it be appropriate to briefly address directly the tradeoff in FPS/Training time with accuracy in regards to the comparison between the proposed method and compared 3DGS methods?", "limitations": "The supplementary material sufficiently addresses the limitations and potential negative societal impact of the submitted work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a reasonable set of additions in order to adapt 3DGS for the purpose of HDR scene reconstruction and view synthesis. Though the submission emerges alongside other comparably-aimed papers, the approach proposed is distinct and its authors provide sufficient evidence of the efficacy of their method.\n\n- Originality\nIt is probably not a very distant leap to apply 3DGS to avenues of work done on NeRF, though this work does go further to involve a few novel additions to the method being adapted, in order to address shortcomings that would be present otherwise. Other works with the same aim appear to be emerging at this time, though the approach taken by the authors, affecting the gaussian initialization and performing distance regularizations, appears distinct.\n\n- Quality\nWith consideration to the supplementary materials, the submission adequately covers most points of concern. Claims made are supported by the provided experimental results. Included ablation study results show qualitative and quantitative contribution for each improvement proposed by the authors.", "weaknesses": "- Clarity\nFor the most part, the writing is clear and straightforward. The context for previous efforts improved upon and problems being solved is well provided. Relevant equations are present and adequately described. Implementation and environment details are well expanded upon in the supplementary materials. However, there are a small number of typos remaining in the submission and supplement. Please make sure to correct these.\n\n- Significance\nThe paper is one of several that aim to introduce HDR Gaussian Splatting, though with regards to other work released within this month and the last, no code has been released as of yet, so no direct comparisons between results may be made. However, the stated improvements relative to these other works appear to come out on top.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our structural regularization and the thoroughness of our experiments! Below, we address the specific points.\n\n1. **About the typos**: thanks for pointing them out! We will definitely fix them in the next version of our paper.\n\n2. **About the difference between LE3D and concurrent papers**: we have discussed the differences between LE3D and other methods in the global rebuttal part, please refer to details there. The main differences between LE3D and other methods lie mainly in two aspects: a) LE3D can perform blind denoising on the data without the need for noise model calibration, which broadens its range of applications. b) LE3D places more emphasis on structural information, making it more suitable for downstream editing tasks.\n\n3. **The tradeoff in FPS/Training time**:  While our assertion may seem somewhat overstated, we confidently advocate for the consistent selection of LE3D, with the disclaimer that this comparison is made relative to LDR-GS, HDR-GS, and RawGS as discussed in our main paper. The main reasons are as follows: a) Visual quality: As shown in the main paper and supplementary figures, LE3D exhibits less noise compared to other methods (e.g., floaters in Fig. 1, 3) and better color shown in Fig. 13. b) Structural information: LE3D captures structural details better than other methods and much better than RawGS (Fig. 13, 14, 15), which benefits downstream tasks, as demonstrated in the defocus tasks shown in Fig. 5 and 9.\n\nWe hope our response addresses your concerns. You can find a reiteration of our motivation, contribution, and potential application in the first section of the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u76Yu6aESr", "reviewer": "Reviewer_PtE5", "summary": "LE3D is a novel method for real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes from RAW images, especially for nighttime scenes. It addresses the limitations of previous volumetric rendering methods by introducing Cone Scatter Initialization to improve SfM estimatied pointclouds, replacing SH with a Color MLP to represent in the RAW linear color space, and introducing depth distortion and near-far regularizations to enhance scene structure accuracy. LE3D significantly reduces training time and increases rendering speed compared to previous methods. The model is tested on the benchmark dataset of the existing RawNeRF paper.", "questions": "Please discuss the points mentioned in the weaknesses.", "limitations": "Sufficiently discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well written and has a clear structure that aligns to the contribions which makes it easy to read and understand.\n2) The methods part is technical sound and details are described.", "weaknesses": "1) Instead of using the Spherical Harmonics the authors propose a view-dependence MLP and explain the change that the MLP can better represent the the linear RGB space, however it is unclear to me why the SH are less expressive than a tiny MLP in the linear RGB space. Can the authors provide more insights on this.\n2) Insufficient quantitative investigations of the new view-dependence MLP. I'd expect more evaluation of this part, e.g. the varying size of the MLP and different parameters of the SH model. Moreover it would be good to the effect of switching to MLPs on the render time.\n3) While the cone scatter initialization leads better rendering, it would be good to discuss how the additional sampling point influence the total number of gaussians. This number becomes relevant for the fps measurements and might reduce speed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our writing and your valuable comments! Below, we address the specific points.\n\n1. **Request for more insights on MLP and SH**: We have analyzed the instability factors of SH during training and provided more statistical data to demonstrate its limited representation capability under high contrast conditions. Please refer to the third section \"What limits the final representation capability of SH on linear color space\" in the global rebuttal for details.\n\n2. **Request for more ablation studies on MLP and SH**:  We have conducted experiments on the size of MLP and the degree of SH. Please see Table 2 in the rebuttal PDF. The experiment demonstrates that modifying the parameters of SH does not address the inherent issues, which underperforms MLP across all metrics. Moreover, adjusting the parameters of the MLP has minimal impact, given the small disparity in the metrics. Consequently, transitioning to an MLP is entirely justified.\n\n3. **Detailed results on gaussian numbers for w/wo CSI**: In some scenes (especially bikes and windowlegovary), the sparse point cloud fails to reconstruct distant points accurately. During training, gaussians tend not to move perpendicular to the camera plane but rather parallel to it. This results in many foreground gaussians attempting to fit the background from different perspectives, as shown in Fig. 4 (b) in our main paper, leading to more foreground split/clone operations and thereby increasing the number of gaussians and leading to worse results in structural reconstruction. CSI is used to add sparse points for distant scenes, and due to the optimization capabilities of 3DGS, in all scenarios, CSI not only did not increase the number of points, but actually reduced them. Besides, CSI is particularly effective in the outdoor scene (-24.86% for bikes and -13.10% for windowlegovary). Details can be found in Table 1 in the rebuttal PDF.\n\nWe hope our response addresses your concerns. Here, I would like to reiterate the potential impact of LE3D: real-time rendering for HDR view synthesis is important because it can support more downstream editing tasks, allowing novel view synthesis technology to be applied to VR/AR or post-production in film and television (reframing and postprocessing). As demonstrated in our demo video, our current progress already supports various subsequent processing techniques in near real-time. These techniques bring LE3D closer to practical applications. Details for more about our motivation can be found in the first section of the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYdlKK0Cj2", "reviewer": "Reviewer_DcQH", "summary": "The paper LE3D proposes training a 3DGS representation with raw images, instead of preprocessed LDR images. This allows more accurate scene recovery in low-light environments and unlocks applications such as HDR rendering or refocusing. While there has been prior work on training neural scene representations with raw images (notably RawNeRF), this work specifically proposes to use a 3DGS representation which allows real-time rendering. To facilitate training a 3DGS representation with raw images, they make several contributions:\n\n1) Cone Scattering Initialization - a method to \"densify\" the Gaussian initialization to overcome inaccuracies in SFM\n2) Replacing the SH in 3DGS with a colour MLP \n3) Introducing space carving losses for better geometry recovery (for downstream tasks)", "questions": "1) Have the authors tried just increasing the rate of the Gaussian splitting instead of CSI? Have they tried anything else apart from CSI for the gaussian density problem? \n\n2) Have the authors tried other strategies apart from a colour MLP? Perhaps the SH are also somehow introducing some bounds/weird gradient updates. Here's a thought: in NeRF if you bound your radiance when it shouldn't be, your density would also become heavily affected by the radiance (as it seems is happening in your case without the colour MLP), perhaps it could be something similar going on? How different is the per channel scaling behaving with/without the colour MLP?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality & Significance: I like that the paper brings several of the contributions in the NeRF literature to 3DGS, specifically I like the problem being attempted in the paper, I think HDR rendering and training with raw images is an important problem and it is timely to attempt this problem using a 3DGS representation. Losses such as the distortion loss from MipNeRF360 are widely used in the NeRF literature and I think it's valuable to have an implementation of them in the 3DGS context. Not a significant research contribution, but I also really liked the interactive viewer in the supplement, I think it would be fun to play around with the demo and see the weaknesses of the method. \n\nQuality: I think the authors are fairly open about the limitations, for example mentioning NeRF-based methods obtain superior sRGB metrics. \n\nClarity: Paper is well organized.", "weaknesses": "Originality: I think although useful, the contribution of the work isn't huge, from Table 1, we can see RawNeRF + GS performs quite well already. I also believe the Related Works section is a bit sparse, there has been a lot of work on low-light image enhancement with NeRF, which I think is relevant (even though it's not about HDR rendering), see a couple below:\n\n1) Lighting up NeRF via Unsupervised Decomposition and Enhancement, ICCV 2023 - method for training NeRFs with sRGB images taken in the dark \n2) Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption, AAAI 2024 - method for training NeRFs with sRGB images taken in over/under-exposed scenarios \n\nQuality: This is my biggest gripe with the work, some of the contributions are included without significant reasoning. Most importantly:\n\n1) The authors introduce the CSI and say it helps due to the misalignments in the SFM estimation. However, I don't think this claim is sufficiently well-proven. Perhaps a denser initialization helps even in cases with perfect cameras?\n2) The authors do not justify including the colour MLP, apart from saying it performs better than just using the SH (due to the linear scaling). I think this point is specifically important since I think the MLP is making the method 1.7 times slower in rendering (looking at Table 1).\n \nFor 1) I am sympathetic to the authors that this might be hard to justify, but I think 2) should be addressed (see also questions). \n\nClarity: There are quite a few typos/grammatical errors, I list just a few below, but I would implore the authors to use a spell check throughout the text:\n\nL38: to the 3D world \n\nL170: fed \n\nL173: set", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our motivation and the structure of our paper, as well as your valuable comments! Below, we address the specific points.\n\n1. **About the related work**: we will add and discuss them in our next version. We will add more discussion for sRGB images (both w/wo multi-exposure) based on novel view synthesis techniques with your referred papers.\n\n2. **Does RawGS (RawNeRF + 3DGS) perform well enough?** We do not believe that RawGS (RawNeRF + 3DGS) performs well enough. The reasons are as follows: 1) Colour reconstruction failure & more floater remaining: As shown in Fig. 1 (left), Fig. 3, and Fig. 13-15 in our main paper and supplementary, these results indicate that RawGS performs worse than LE3D in terms of colour and floaters. 2) Structure reconstruction failure: As shown in Fig. 5 (c, e) and Fig. 13-15 in our main paper and supplementary, it is evident that RawGS fails to reconstruct the structure (depth map). This is disastrous for downstream tasks such as refocusing, as shown in Fig. 5 (b, d) in our main paper and Fig. 9 in supplementary. Therefore, we do not believe RawGS performs well enough, which is also the motivation for LE3D: to make 3DGS available on real-time HDR view rendering from noisy RAW images, as well as downstream editing tasks.\n\n3. **Regarding CSI and dense point cloud initialization**: We have previously conducted experiments using dense point clouds (this is also what we tried before coming up with the idea of CSI), which indeed partially solved the problem of failing to reconstruct distant scenes. However, it led to the following issues: 1) Longer time cost: The time required to reconstruct dense point clouds is four to five times that of sparse point clouds (from 6min 23sec to 28min 53 sec on scene 'bikes'), thereby increasing the overall reconstruction time. 2) Redundant gaussians when converged: An excessive number of points leads some gaussians to fit noise, which negatively increases the count of gaussians. For instance, in the 'bikes' scene, the number of points increased from 4.1e5 to 5.4e5, which is approximately a 30% increase, resulting in longer rendering time and more storge cost. So a balance between sparse and dense point cloud initialization is important, and then here comes our CSI. In fact, it is hard to find out the difference between the visual result of dense initialization and CSI initialization. We can provide proof of this.\n\n4. **Regarding the speed issue between MLP and SH**: We believe this balance is worthwhile because both structure and colour are important for downstream tasks, and the speed of LE3D remains real-time. More discussion on MLP and SH can be found in the third section \"What limits the final representation capability of SH on linear colour space\" in the global rebuttal.\n\n5. **Regarding increasing the splitting threshold and the idea before CSI**: We experimented with increasing the splitting threshold as shown in Fig. 1 (c) in the rebuttal PDF, but it did not effectively solve the issue of distant scenes and actually increased reconstruction noise. The reason it cannot effectively address distant scenes is that our dataset is forward-facing, and gaussians tend to move parallel to the camera plane rather than perpendicular to it. Therefore, if there are not enough gaussians for distant scenes during initialization, additional gaussians will not appear spontaneously. We observed this phenomenon based on previous experiments with dense point clouds. Dense point clouds can reconstruct points in the distance, but this also impacts training time and the number of gaussian. Our CSI effectively balances the number and distance of points.\n\n6. **Regarding colour representation besides MLP and the co-optimization of colour and structure**: Currently our choice of MLP proves sufficiently effective with its learning ability, as shown in Fig. 1 (d) and Table 2 in the rebuttal PDF. However, we agree with the reviewer that trying other strategies is interesting future work. In vanilla 3DGS (with SH), there is indeed a phenomenon where colour and structure do not optimize well together. As we mentioned in the ablation study (Fig. 4 (e) in main paper), gaussians with SH in the early stages lead to colour optimization failure, which in turn causes structure optimization failure before 15,000 iterations (during which gaussian primarily optimize structure through split/clone). This, in part, limits the final colour performance of SH. Please refer to the third section \"What limits the final representation capability of SH on linear color space\" in the global rebuttal for details. Besides, the per channel scaling behaving with/without the colour MLP can also be found in Fig. 1 (d) in the rebuttal PDF.\n\nWe hope our response addresses your concerns. You can find a reiteration of our motivation, contribution, and potential application in the first section of the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bMMWip2WR6", "reviewer": "Reviewer_Vnv5", "summary": "This paper proposes LE3D, an HDR 3D Gaussian Splatting method with Cone Scatter Initialization, Color MLP, and depth distortion. Specifically, this paper introduces the Cone Scatter Initialization to enrich the estimation of SfM. The Color MLP aims to represent the RAW linear color space. The goal of depth distortion and near-far regularizations is to improve the accuracy of scene structure. Experimental results show that the proposed LE3D achieves promising results.", "questions": "Please see the Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is good. Nighttime 3DGS and HDR 3DGS are very important.\n2. The proposed method includes comprehensive experiments.\n3. The experimental settings are easy to follow.\n4. The video demo in the supplementary material is good.", "weaknesses": "1. The authors mentioned, “the SfM estimations based on nighttime images are often inaccurate.” I agree with this point. However, from my understanding, the initial point clouds can be wrong and can be further optimized by the 3DGS training stage. Therefore, I am wondering if the cone scatter initialization is really important. Could the authors provide ablation studies using the RawNeRF dataset?\n2. 3DGS highly relies on the initial point clouds. One of the examples in Fig. 1 (a) shows that the input image can be very dark. I am wondering if the authors can get the initial SfM estimations from such inputs. From my understanding, COLMAP cannot work well on these dark inputs. If so, how do the authors get the initial SfM estimations?\n3. The authors mentioned, “the SH does not adequately represent the HDR color information of RAW images due to its limited representation capacity.” Why? Could the authors provide any solid evidence?\n4. The depth distortion (Eqn. 8) is quite similar to Eqn. 15 in Mip-NeRF 360. How does the proposed depth distortion differ from the depth distortion of Mip-NeRF 360? Could the authors conduct experiments to compare the two depth distortions?\n5. The authors seem to overstate their methods. Compared to RawNeRF, LE3D can reduce training time to 1%. However, the fast training speed is not actually due to the proposed components, but rather from 3DGS itself. If we compare it to the original 3DGS, the training speed of LE3D increased from 1.05 to 1.53, which is an improvement of approximately 45.71%. Since this is the first HDR 3DGS paper, I do not want to be too strict. But I suggest that the authors do not overstate the advantages in terms of speed.\n6. The quantitative results of the ablation studies should be shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the motivation, effectiveness, and thoroughness of our experiments on LE3D. We also appreciate your valuable comments! Below, we address the specific points.\n\n1. **Does CSI really matter?** We have done our ablation studies on CSI in our main paper, please refer to Fig. 4 (b) in our main paper and Fig. 1 in the rebuttal PDF. Compared to LE3D, the removal of CSI leads to a lack of distant scene details and the splitting of a large number of gaussians at incorrect depth levels to fit the distant scene, thereby resulting in failure in structural optimization. Additionally, there is a severe decline in visual quality (please zoom in for the best view). This is because the sparse point cloud cannot reconstruct distant points effectively, whereas CSI can effectively increase the initial point cloud in distant areas, achieving the goal of effective distant scene reconstruction.\n\n2. **Initial SfM on Dark Scenes**: We follow the exact settings of RawNeRF to obtain the initial SfM using COLMAP (except we use the PINHOLE camera model instead of OPENCV). In the RawNeRF dataset, COLMAP works well on calibrating camera poses but not very well on generating point clouds. The lack of point cloud initialization is also one motivation for our purpose of CSI initialization, which enriches the point cloud and extends its depth range. If the scene is too dark in other data, leading to poor COLMAP calibration, we brighten the scene using the DNG data to obtain corresponding JPG images, and then perform COLMAP calibration.\n\n3. **Why SH is less expressive than MLP?** As this is a common problem, please refer to the third section \"What limits the final representation capability of SH on linear color space\" in the global rebuttal for details.\n\n4. **Depth Distortion**: Our depth distortion map is an approximate implementation of equation (14) from Mip-NeRF 360 as we mentioned in the main paper L197-L198. Both of our regularizations are used to improve structure reconstruction. Please refer to Table 2 in the supplementary for a quantitative comparison of the effects of both $ R_{dist} $ and $ R_{nf} $, and Fig. 6, 7 in the supplementary for qualitative comparisons.\n\n5. **Overstated Performance**: Thanks for pointing out this. We apologize and will adjust our wording accordingly in our next version.\n\n6. **Quantitative Results of Ablation Studies**: Please refer to Table 2 in our supplementary.\n\nWe hope our response addresses your concerns. You can find a reiteration of our motivation, contribution, and potential application in the first section of the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First and foremost, we would like to express our gratitude to all the reviewers and the Area Chair for their diligent review. We sincerely appreciate your recognition of our writing, your appreciation of the effects shown in our demo video, and your identification of typos and weaknesses in our paper. We look forward to improving LE3D with all of your help and making it a better paper, project, or maybe a product.

However, given that the confidence scores from our reviewers are predominantly 3, we wish to reiterate to the reviewers and the ACs the motivation, contribution, and potential impact of our LE3D.

**Reiterate the motivation, contribution, and potential impact.** We are aiming at real-time HDR view rendering and downstream editing tasks. With the recent advancements in 3DGS-related work, we attempt to leverage its capabilities to achieve real-time rendering and fast training. However, directly applying 3DGS techniques to noisy raw images presents numerous challenges, such as poor SfM estimation, limited representation capacity of SH and inaccurate structure reconstruction. The primary contribution of LE3D is addressing these issues, enabling 3DGS to excel on noisy RAW images. It enables tons of applications for post-processing, including HDR rendering, refocusing, tone-mapping changes and so on.

Our potential impact lies in the applications of AR/VR technology and related post-production techniques in film and television (reframing and postprocessing). Additionally, it can bring traditional computational photography techniques into the realm of 3D reconstruction. As demonstrated in our demo video, our current progress already supports various subsequent processing techniques in near real-time: HDR tone mapping, refocusing, and color temperature tuning. Therefore, we believe that the potential impact of LE3D is substantial.

**The Difference Between LE3D and Other Concurrent Methods.**  As discussed in the introduction, current HDR reconstruction methods mainly include two approaches: 1) reconstruction from multi-exposure 8-bit RGB images, and 2) direct reconstruction from noisy under/multi-exposure data. The first approach has high data requirements, necessitating changes in exposure during capture, while the second approach needs to overcome the impact of noise.

The most recent concurrent 3DGS methods on similar issues include HDR-GS[1] (RGB), HDRSplat[2], Raw3DGS[3] (RAW). Among them, HDR-GS[1] does not start with RAW data and is of different settings from our method. HDRSplat[2] and Raw3DGS[3] also has substantial differences in their algorithm framework from ours:

1. The Necessity of a Denoising Network: Both HDRSplat[2] and Raw3DGS[3] adopt multi-stage approaches, requiring noise model calibration, denoising, and reconstruction after data collection. This necessitates extra data, training, and tedious noise model calibration when transferring their methods to other devices. In contrast, though we do not have specific module for denoising, our LE3D algorithm is blind to noise, making it potentially adaptable to any device.

2. Better structure for downstream tasks: Unlike other methods, our focus is not only on the reconstruction of visual effects (denoising) but also on the reconstruction of scene structure, due to our CSI initialization and regularizations. This makes our LE3D more suitable for downstream tasks like refocusing.

[1] Cai, Yuanhao, et al. "HDR-GS: Efficient High Dynamic Range Novel View Synthesis at 1000x Speed via Gaussian Splatting." arXiv preprint (2024).

[2] Singh, Shreyas, Aryan Garg, and Kaushik Mitra. "HDRSplat: Gaussian Splatting for High Dynamic Range 3D Scene Reconstruction from Raw Images." arXiv preprint (2024).

[3] Li, Zhihao, et al. "From Chaos to Clarity: 3DGS in the Dark." arXiv preprint (2024).

Next, we will elaborate on a common question raised by several reviewers: 

**What limits the final representation capability of SH on linear color space.** The answer is the extremely high dynamic range in linear color space.

1. We analyze the final convergence statistics for windowlegovary (multi-exposure training with a very high dynamic range). We calculate the mean and variance of each gaussian color from each training view, as shown in Fig. 1 (d) in the rebuttal PDF. It can be observed that: a) The numerical range using color MLP can learn a very high dynamic range in linear color space (maximum value over 200), while the numerical range using SH collapses completely (contains negative values) and fails to learn the especially high dynamic range in linear space. b) Additionally, the final color using SH is significantly affected by the viewing direction (as reflected in the extremely large variance), which is also a manifestation of its instability and finally causes the less color representation ability in linear color space.

2. As for the training process, the SH cannot optimize the color well in the early stages of training, as shown in Fig. 4 (e) in the main paper. This also leads to poor structural optimization before 15,000 iterations (before 15,000 is where 3DGS is used for densifying/cloning), as illustrated in Fig. 4 (c) in the main paper. Poor structure causes color optimization to fail after 15,000 iterations (at this stage, 3DGS mainly optimizes color and does not perform densifying/cloning), further affecting the visual effects, as shown in Fig. 1 (left) in the main paper and Fig. 7 in the supplementary.

Both of the above situations affect the representation capability of SH at final convergence. Details can be found in Table 2 in the rebuttal PDF, which shows MLP always outperforms SH by a large margin.

Meta-review: The paper introduces a new framework that unifies rendering and inverse rendering tasks using a two-stream diffusion model. This approach is innovative and has potential implications for future work in image generation and editing. The paper is well-structured, with solid experimental results demonstrating the effectiveness of the model. However, several concerns were raised by the reviewers, primarily focusing on the limited dataset used for training and the lack of evaluation on real-world data, which raises questions about the model's generalization capabilities.

The paper's handling of lighting estimation was also critiqued, with reviewers noting the need for more comprehensive evaluation in this area. Additionally, the comparisons with state-of-the-art methods were considered somewhat outdated, and reviewers suggested including more recent work to strengthen the evaluation. Despite these concerns, the authors provided detailed rebuttals, addressing most issues and expressing their intention to improve the dataset in future work.

Ultimately, while the paper has some limitations, its novel approach and clear presentation make it a valuable contribution to the field. It has been recommended for poster acceptance, with the expectation that the authors will continue to refine their work and address the highlighted concerns in future iterations.
Decision: Accept (poster)