From Chaos to Clarity: 3DGS in the Dark
Authors: Zhihao Li
Yufei Wang
Alex Kot
Bihan Wen
Venue: NeurIPS 2024 poster (2024)
Keywords: Novel view synthesis; Raw images; 3D Gaussian Splatting (3DGS); Denosing; Self-supervised learning
Primary area: machine_vision
TL;DR: We enhance 3D Gaussian Splatting for HDR reconstruction from noisy raw images, outperforming prior models on the RawNeRF dataset.

Abstract: Novel view synthesis from raw images provides superior high dynamic range (HDR) information compared to reconstructions from low dynamic range RGB images. However, the inherent noise in unprocessed raw images compromises the accuracy of 3D scene representation. Our study reveals that 3D Gaussian Splatting (3DGS) is particularly susceptible to this noise, leading to numerous elongated Gaussian shapes that overfit the noise, thereby significantly degrading reconstruction quality and reducing inference speed, especially in scenarios with limited views. To address these issues, we introduce a novel self-supervised learning framework designed to reconstruct HDR 3DGS from a limited number of noisy raw images. This framework enhances 3DGS by integrating a noise extractor and employing a noise-robust reconstruction loss that leverages a noise distribution prior. Experimental results show that our method outperforms LDR/HDR 3DGS and previous state-of-the-art (SOTA) self-supervised and supervised pre-trained models in both reconstruction quality and inference speed on the RawNeRF dataset across a broad range of training views. We will release the code upon paper acceptance.

From Chaos to Clarity: 3DGS in the Dark
Zhihao Li Yufei Wang" Alex Kot Bihan Wen'
Department of EEE, Nanyang Technology University, Singapore https://lizhihar github Bataab..ooReeee io/Raw3DGS
Abstract
Novel view synthesis from raw images provides superior high dynamic range (HDR) information compared to reconstructions from low dynamic range RGB images. However, the inherent noise in unprocessed raw images compromises the accuracy of 3D scene representation. Our study reveals that 3D Gaussian Splatting (3DGS) is particularly susceptible to this noise, lcading to numerous elongated Gaussian shapes that overfit the noise, thereby significantly degrading reconstruction quality and reducing inference speed, especially in scenarios with limited views. To address these issues, we introduce novel self- sepersupe learning framework designed to reconstruct HDR 3DGS from a limited number of noisy raw images. This framework enhances 3DGS by integrating noise extractor and employing noise-robust reconstruction loss that leverages a noise distribution prior. Experimental results show that our method outperforms LDR/HDR 3DGS and previous state-of of--o-f-thhteee art (SOTA) self-supervised and supervised pre- pre-trained models in both reconstruction quality and inference speed on the RawNeRF dataset across a broad range of training views. Code can be found in/htps: / //1ttps://iizihhoo tithub.io/Raw33
Introduction
Novel view synthesis (NVS) is fundamental to 3D vision, with extensive applications virtual and augmented reality (VR/AR) [16 17), autonomous driving [18] 34), and 3D asset creation [22 4 20
Neural Radiance Fields (NeRF) 24 have revolutionized this field by rendering colors through the accumulation of RGB values along sampling rays, employing implicit MultiLayer Perceptron (MLP) representation. Typically, this method uses low dynamic range (LDR) RGB images processed by image signal processing (ISP) modules. leading to significant loss of crucial scene details, especially in high-contrast areas like highlights and shadows, which can degrade performance in high dynamic range (HDR) environments such as tunnels, sunsets, or dimly lit scenes. Moreover, reliance on ISP-processed RGB images restricts post-capture color and tone adjustments, presenting significant challenges for photographers and modelers during post- poot-production.
In contrast, raw images before ISP offer a higher dynamic range and preserve more scene information. Recent research has indicated that utilizing raw images significantly enhance the performance of downstream computer vision tasks in complex lighting conditions [19] and offer greater flexibility
in post-production adjustments 53 Building on this advantage, RawNeRF (33 first employed
raw images as the optimization target in NeRF, achieving marked improvements over traditional RGB-based LDR NeRF approaches. However, RawNeRF's reliance implicit 3D representation computationally demanding, requiring up to 48 hours to train a single scene and about one minute to render single view, which limits its practicality for real-time applications.
Both authors contributed cqually to this rescarch. i Corresponding author. Email: bihan. wenentu. edu. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Train with
Train with 3=31
30G5 lest R888 PSNR [cle)
Train X
LF69
LF69
22
3013 Task FRS
30
S2PUS
Train X
28
train with clcan X
15
train with deaan X'
LF69
train with naicy &
train with naley &
20 25
Number
Train Number
Namereus Tlin, Flai Coussian Stupes
18LB 1IFPS
22:81/ / 33FPS
(a) Impact of noise on PSNR
(b) Impact of noise on FPS
(c) Visualization of test views
Figure I: Comparative analysis of 3DGS trained with clean raw images, denoted X, versus noisy raw images, denoted X, across various training view counts N. The clean raw images, captured in daylight, are selected from the RawNeRF dataset (23 The noisy raw images are generated from these clean images using the noise model from PMN with calibrated camera noise parameters. (a) Training with noisy raw images results in decreased PSNR in the test views, with widening performance gap the number of training views is reduced. (b) The rendering speed (FPS) shows similar trend to PSNR. (c) Test view visualizations show that training with noisy images causes 3DGS to produce numerous thin, flat Gaussian shapes, leading to visual artifacts and reduced FPS, especially with fewer training views.
Very recently, the advance on 3D Gaussian Splatting (3DGS) [3] offers an explicit 3D representation that employs set of learnable 3D Gaussian points to depict color, shape, and opacity, thereby enabling real-time novel view rendering. However, utilizing raw images directly as the optimization target in 3DGS can introduce significant artifacts and adversely affect rendering speed, especially in scenarios with limited training views- common challenge in real-world applications, as depicted in Fig. D These challenges arise primarily from the inherent noise in raw images, which is more pronounced than in RGB images due to the absence of noise reduction and smoothing processes typically performed by ISPs. As demonstrated in Fig. 2a the presence of noise in raw images is an inevitable consequence of physical phenomena such as the photoelectric effect and hardware limitations like dark current leakage. Unlike NeRF, where the Multilayer Perceptron (MLP) acts low-frequency filter [38] to mitigate high-frequency noise, 3DGS tends to produce numerous thin, elongated Gaussians to represent noise, as seen in Fig. Ic
To address these challenges, we propose a novel self-supervised framework that jointly denoises and constructs HDR 3DGS representations within noisy raw images using a noise robust reconstruction loss. Specifically, our method utilizes a noise extractor to predict the underlying noise in raw images, constrained by predefined noise model. Simultancously, the 3DGS branch is optimized towards pseudo noise-free raw images. This dual-branch approach allows the noise extractor to accurately predict and separate noise from the images. Compared to using the standard RawNeRF loss function with conventional 3DGS, our approach significantly reduces noise artifacts in rendered views and enhances the rendering speed. Our contributions can be summarized as follows:
We discover that noisy raw images significantly degrade the rendering quality and speed of 3DGS, especially in scenarios with limited training views. We provide detailed analysis of how noise impacts the optimization of 3DGS, and model its relationship with the number of training views and the noise distribution.
We propose a novel self-supervised framework that incorporates a noise robust reconstruction loss. This framework leverages a physical- based noise model to jointly denoise and enhance the HDR representation of 3DGS within noisy raw images.
Our method substantially outperforms conventional 3DGS methods that employ RawNeRF loss, along with both self-supervised and pre- rre-trained supervised denoisers, in terms of rendering quality and processing speed across various training view counts.

---

2 Related work
2.1 Neural Radiance Fields and 3D Gaussian Splatting
Recently, Neural Radiance Fields (NeRF) 24 have spearheaded significant advancement in novel-view synthesis by reconstructing 3D scenes. NeRF models achieve scene representation by optimizing coordinate-based multi-layer perceptrons (MLPs) to estimate color and density values through differentiable volume rendering. Typically, NeRF utilizes tone-mapped low dynamic range (LDR) images imputs. These images processed by image signal processing (ISP), which, while reducing noise and smoothing details, also maps high dynamic range (HDR) to LDR, often clipping highlights and shadows. To counter these limitations, RawNeRF (3 modified NeRF with scaling loss to directly train on linear raw images, thus preserving the scene's ' full dynamic range. However, like most previous NeRF-based methods, RawNeRF is computationally demanding, requiring up to 48 hours to train a single scene and about one minute to render a single view, which constrains its usability in real-time applications. To accelerate rendering, various methods have been explored, including space discretization 77 I0 26 29, 33 335, codebooks [28]. and hash tables [25] Yet, these approaches still require a substantial number of queries render a single pixel, rendering them unsuitable for real-time applications. More recently, 3D Gaussian Splatting (3DGS) [I3 introduced novel approach by explicitly representing scenes with learnable 3D Gaussians. Utilizing differentiable splatting and tile-based rasterization, 3DGS enables real- time novel view rendering. However, the initial 3DGS models did not achieve rendering quality comparable to NeRF. To address this. Scaffold-GS [3] introduces an anchor-level feature to capture correlations between adjacent Gaussian points, which significantly enhances the rendering quality of 3DGS, achieving comparable results to NeRF while maintaining real-time rendering speeds.
2.2 Raw image and video denoising methods
Raw image denoising is critical in both photography 9 and scientific imaging [15 12 Traditional ISPs typically employ self-simmi culfs-imilarrtybbbee -based methods such as BM3D [3] to reduce noise. Recent ad- vancements such as SID and SIDD [[] demonstrate that data-driven, deep-Icarning approaches can surpass these traditional techniques. Innovatively, noise model-based methods like the ELD 32 have extended denoising capabilities into extreme low-light scenarios. It challenges conventional Gaussian noise models by introducing row-based noise model that uses the Tukey Lambda distribution, which more accurately captures long-tail noise patterns in low-light conditions. The PMN (6) further investigates Fixed Pattern Noise in dark frames, enhancing alignment with the Poisson-Gauusian (P-G) distribution. These methods, however, often rely on large datasets of paired or clean images for training, addressed by Neighhor2Neighhor [TT), which trains denoising models solely on noisy images without clean counterparts. Building on the Neighbor2Neighbor approach, LGBPN [31] introduces more diverse masks denoise spatially relevant noise. Nevertheless, Neighbor2Neighboo and LGBPN lack integration of a precise noise model or a physical prior of the 3D world, aspects our framework incorporates by merging a noise model with 3DGS to effectively separate noise from noisy images, eliminating the need for clean images. Complementing image- -ased methods, Yue al. (66) expanded the scope raw video. They introduced dataset specifically curated for raw networks that leverage temporal information across
video denoising, along with frames to enhance the denoising process.
Preliminaries
In this section, we describe the noise model that we have adopted to regularize the extracted noise from raw images, provide an overview of the background of 3D Gaussian Splatting (3DGS) and review the RawNeRF scaling loss 23 for handling the wide dynamic range raw images.
Noise model. In the camera imaging process, depicted in Fig. 2a the raw image & derived from the ideal clean raw image x inevitably contains noise n. This relationship is expressed as:
+
(1)
Drawing on existing noise models 32 6 decompose into various noise components as:
Mshot
Dread + Dspp.
(2)
3

---

where Dshot Bread, and nsp denote shot noise, read noise, and fixed pattern noise. respectively. A comprehensive description of each noise component provided in the Supplemental Material. Given that nshot can be approximated from the Poisson distribution P(x) k to a Gaussian distribution (00 k), where represents the camera system gain, can be seamlessly combined with Nyeads which also follows a Gaussian distribution (0, d Feadd). This integration yields heteroscedastic Gaussian noise model:
Bhg (O, o hgg)
o hig = oeead + x k.
Thus, the overall noise model simplifies to:
n Dig + Dsp-
(4)
Utilizing Eq. the prior for noise modeling allows for the potential separation of the clean image X from noise corruption, enabling more precise 3DGS reconstructions.
3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) [33 models 3D scene using a collection of 3D Gaussians, iee,, G = [gi,.... gM), which are rendered from various viewpoints via a differentiable splatting and tile-based rasterization process. Each Gaussian g: is initialized from (SfM) and characterized by 3D covariance matrix Σ, = RR333 position vectors μ4 E R3, color vector C; R and opacity o,, To maintain the positive semi-definiteness of E, it is parameterized as Σ; = R.S.S, R? where R, E R and diagonal matrix S; E g3x33 represent rotation and scaling matrices, respectively. To render pixel &(p) at a pixel position p in the reconstructed image & from a selected viewpoint within the camera projection J. 3DGS utilizes differentiable splatting operation to splat 3D Gaussians onto the 2D surface as follows:
M
i-1
x(p) = Σ (.)(P)
II an)))))))
i=1
f=1
(5)
[((,)) -
1
(p) = 0, exp
J(m)))
All attributes, i.e., = (mr R, Ci.
C; 0; are learnalee and optimized through the loss function:
LIDos (x,x) (1 (&, + ALD-SSIM x).
(6)
RawNeRF scaling loss. As discussed in RawNeRF 23 the dynamic range of colors in raw image is vast, which complicates the application of standard Li or Le loss. These losses tend to overemphasize errors in brighter areas, leading to tone-mapped images with poorly rendered, low-contrast dark regions. To address this, RawNeRF proposes a scaling loss as:
8
CRawNeRF (8,x) =
sg(()) +
where sg(+) is the stop gradient function, and is a small constant to prevent division by zero.
Methodology
4.1 Motivation
Consider of noisy raw images, denoted X (xq,...,&p &yJ, which are captured from different viewpoints using the same camera settings. For any real-world point, e.g., r. that is captured by all cameras, its intensity is recorded at a specific pixel coordinate p, in each raw image x;, with the corresponding pixel value X; (p.). As discussed in Sec.
3 the noisy pixel value is the sum of the
noise-free value and the noise at that location, i.e., x; (p:) = xp)) + n, (p:). Given the same camera settings and real-world point the clean value x(p) remains constant across images.
To further explore the noise impact on 3D Gaussian Splatting (3DGS) [13 optimization, we simplify our analysis by utilizing an L2 loss function. The optimal target for the 3DGS output x(p), which corresponds to different positions p. each image x; after the splatting process with camera projection J, is the following:
N
&p))) x(p) + 1 N
x(p) arg min
Σ n(p.)
(8)
x(P)

---

Photon-voltage
Analog amplification
convertion
Photon
photons
electrons
voltage
value
Noisy Image X
Flux
Arput
SE
shot
pp Bread
Gaussian points
(a) Noise is inevitable in the imaging process
(b) Impact of noise the 3DGS optimal target
Figure 2: illustration of how prevalent noise in raw images impacts the 3DGS optimization. (a) The imaging process inherently introduces additive noise at various stages due to physical principles and hardware limitations, represented as x x + n, where x and x denote the noisy and clean images, respectively. (b) For real-world point r, collection of images X {x, x2) records its intensity at pixel coordinates (PI, P influenced by noise. The optimal target of 3DGS for this point, denoted as xp) {xpp)_.. has discrepancy from the clean pixel intensity x(p). The
variance this discrepancy is detailed in Eq.
iteration 1000
iteration 3000
ileration 10000
ileration 15000
iteration 20000
iteration 30000
Figure 3: Visualization of the 3DGS test view changes across optimization iterations. Initially, the 3DGS model fits the clean signal (at 1,000 and 3,000 iterations). However, as the iterations progress (from 10,000 30,000), the model starts to overfit the noise.
Considering real-world flat surface around point where intensities remain consistent, independent noise will cause variations in the optimal target across the surface as:
1
Var(x(p))
N2 Var(n,(p.))
hg
N22
With the camera settings, it is evident that the variance of the optimal target x(p) is inversely proportional to the number of training views N. Since 3D Gaussians are splatted according to Eq. (5), the splatting process tends to produce numerous thin, flat Gaussian shapes compensate for this variance. With fewer viewpoints, which is a common setting in real-world applications, these elongated Gaussians dominate, illustrated Fig. This dominance not only degrades the reconstruction quality but also affects rendering speed, as the performance is directly linked to the number of Gaussian points M.
Although 3DGS decomposes the variance on a flat surface into numerous elongated Gaussians, we observed that these Gaussian points tend to collapse and are subsequently divided during the later stages of the optimization process. In this process, the low-frequency signals are the first to be approximated before noise, depicted in Fig. 3 which aligns with Deep Image Prior (DIP) 30,, in which the low-frequency image components are typically reconstructed before noise. Based on such an observation, propose incorporating noise model prior in Sec. 4.3 to relax the constraints in the 3DGS optimization framework.
4.2 Lens distortion correction
which employs ray-tracing directly from image data, 3DGS maps 3D Gaussians onto 2D image plane through a splatting operation. To enhance processing efficiency, 3DGS adopts the straightforward pinhole camera model. However, this model does not account for the nonlinear distortions primarily radial- that are typical with real-world camera lenses. These distortions are more significant in raw images, where the typical lens corrections provided by ISP absent.
Unlike NeRF 24
To effectively correct for both radial and tangential lens distortions, we employ a distortion map- ping function ((.). This function transforms the undistorted image coordinates (z, y) into their

---

Ohg
NLL
Noise Extr- actor (())))
3DGS
Render (X)
Noisy X
Pred Noise
Normalized 2
Fp
CRWWNRRF
Ccov
Figure 4: Illustration of the noise-robust reconstruction loss, Lar. which comprises three components: the reconstruction loss LRawNeRF, the negative likelihood loss (NLL), and the covariance loss Leov- noisy raw image, X, first input to the noise extractor F (c; to estimate the noise, á. The estimated noise ñ is then used calculate the NLL loss relative the noise distribution. After that, the normalized noise, 2, undergoes covariance loss, Leov, to minimize spatial dependencies among noise components. Finally, the reconstruction loss, LRawNeRF. is computed between the rendered distorted image D(x) and the pseudo clean image X n.
corresponding distorted coordinates (Id, ya), based the distortion model used in COLMAP (27] The distortion map is derived through an iterative Newton Raphonn method. This computation is performed once prior to the training phase, thereby not affecting the training duration. Detailed explanations of the distortion mapping process are provided the Supplemental Material.
4.3 Noise robust reconstruction loss
To mitigate overfitting to noise in 3DGS, we have revised the reconstruction loss function, now termed the noise-robust reconstruction loss function far, as depicted in Fig. 4 This function defined as follows:
Pur (D) (Rr.NNeR((D(SS
n) + And Lnd (f, n),
(10)
where D.-) denotes the lens distortion mapping as described in Sec. 4.2 The function Lna n) measures the noise divergence between the estimated noise qa and the physical noise model Pn. The term And is a Lagrangian relaxation parameter that balances fidelity and noise reduction. To estimate the noise, we introduce a noise extractor (=;2,,) neural network parameterized by Ω which predicts the noise ñ from the input raw image X as:
F p 52) + n5r
(11)
Here, we subtract the input by the fixed pattern noise nyp before feeding it into the noise extractor Fa (- (=: 22,, due the consistent nature fixed pattern noise across different captures.
For the noise divergence loss L n), we first use the NLL Loss to measure the divergence between the estimated noise and the physical noise in corresponding pixel coordinates. The NLL loss is defined as:
NL Eg lo log Pn (n)), Pn = N ( + n/yr
(12)
where o a + x k is calculated according to Eq. 3
The NLL loss models the distribution of predicted noise across each pixel, but it does not address the inter-pixel correlations, which ideally should be minimal. To tackle this, we then standardize the predicted noise standard Gaussian distribution (0, 1) and introduce covariance loss Loov to minimize spatial dependencies:
Loov (f,, n) = Eg !1 2T2), & = 1/sp//
(13)
where the identity matrix, ensuring the decorrelation noise across pixels. Consequently, the comprehensive noise divergence loss ( n) is formulated as:
La (n. n) NLL n) + >cov And Loov (n, n).
(14)
Finally, the complete noise-robust reconstruction loss can be formulated as:
Lan (D(x), x) (RomNeRF(D((x)) x ii) And NLL(û, n) Açov Loov (oor(n (ú n).
(15)

---

0.50
3DGS Test RGB SSIM
0.40
RGB
16
0.35
0.30
3DGS
ELD
BMBD
Ne2Ne
PNI
Crowterr (Ouns)
0.25
Ne2Ne
PNN
Krantatr
LGBPN
RVIDENet
LGBPN
RVIDENet
Can (Durs)
10
10
16
12
16
18
Train view num
Train view num
(a) PSNR over the number of views in training.
(b) SSIM over the number of views training.
BM 3D
ELD
Cooos
225
BMSD
ELD
Censs
Ne2Ne LGBPN
Ne2No
3DGS Test RGB LPIPS
0.725
RADENE
Par (Ours)
LGBPN
RVIDENet
Lan (Ours)
0.700
175
150
0.650
3DGS
0.625
1.00
75
0.575
12
14
38
12
16
18
20
view num
Train view
(c) LPIPS over the number of views in training.
(d) FPS over the number of views training.
Figure 5: Comparative evaluation of various baselines and our method on rendering quality and speed in limited views training settings. The two-stage denoiser 3DGS methods represented by dotted lines, while training on RGB images is indicated by square markers. All metrics are evaluated on test views within the RGB domain.
Experiments
In this section, we introduce the experimental setup, followed by comparing our methods with existing baselines. Finally, we conduct ablation studies to facilitate a thorough discussion of our approach.
5.1 Implementation details
Our framework is built upon Scaffold-GS 21
Considering the challenges associated with training (: .2) from scratch within only 30,000 iterations, we opted to pretrain it on the SID dataset (2].
which features paired images of noise and noise-free scenes. Notably, the SID dataset is captured using Sony DSLR camera, distinctly different from the iPhone X used in the RawNeRF dataset (23
for evaluation (c.g., bit depth differences: vs 12), thereby mitigating the risk data leakage.
Noise extractor (- 52) is optimized using the Adam optimizer with an initial learning rate of 1 x 10-4 reduced to 10 after 25, 5,000 iterations. The hyperparameters And and Acov as defined in Eq. 15] and Eq. (14) are set to 5 and 20 for full-views settings, and 3 and 20 for limited views settings, respectively. All 3DGS models are trained single NVIDIA RTX 3090 GPU, following the default parameter settings of Scaffold-GS.
5.2 Comparison with baselines
Baselines. We first compare our method with LDR Scaffold-GS [2]), which uses RGB images processed by an ISP as the optimal target, to demonstrate the benefits of training on raw images. The ISP settings are same as those used Raw NeRF. We also benchmark our method against HDR Scaffold-GS optimized with the LRawNeRF loss using raw images as a fair baseline. Additionally, we evaluate our approach against two-stage denoiser+3DGS pipelines optimized using the CROwNeRF loss, where 3DGS is optimized on denoised raw images produced by the denoiser. These comparisons
7

---

FAZI
FAZIO
FAZIC
(4) RMSD
(b) RED
16) PMN
(d) LGRPN
(r) No2No
22.13/0..2//0.000
21, 744///04444/0551
10.90/0.523/0.560
22, 00..8/00.20/0.088
22 Nw..606.677.0.665
FAZKO
EMZIC
FAZIC
FAZIO
(F) WVIDoNNe 20566/0056//..5666
SraflaM-GS 00000000000.0000
(h) HDR Staffai--GS 20 18.18/0.577 18/00.577/.000
Ours
Ground truths PSNR1 SSEM) LPIPS)
20.6//0:51/00521
Figure 6: Visual comparison using ours and competing methods under the 12-view training settings.
(e)
Ne2Ne 22, 22..8/0..50..00.0
28.96/015000 (.00/0000000.4804 48.487
21 21.15,00.51,/M (1..531000.066
19.64/0.5/2/0.544
20. 20..66..551//..GU
Music
(1) RVILNNNet 16.12/0.561/0.500
LOR Scalfald-GS
(h) HDR Soallod-GS 24.01/0.544/0
Oues
Goound trutis PSNR1 SSSM LPIPS)
/0,57
00.47/00002/004477
Figure 7: Visual comparison across various methods in full views (100-view) training settings.
include traditional denoiser BM3D [3. srate-of-the-eeee pretrained image denoisers such as ELD 32 and PMN 6 the pretrained video denoiser RViDeNet (36), and advanced self-superviee methods like Neighbor2Neighbor [11 and LGBPN (31 for thorough evaluations. For the pretrained supervised denoisers, we utilized the checkpoints provided by the authors. For the self-supervised methods, we initially pretrained the denoisers on the SID dataset [2] and subsequently fine-tuned them on the training views. All compared methods apply the lens distortion function D(-) to the rendered images to align with distorted raw images, ensuring fair comparison.
Datasets and metrics. We utilize the RawNeRF dataset which comprises images captured in dark scenes using an iPhone X with various ISO settings. For the full-views training setting, we adhere the same train-test view splits as in RawNeRF. For limited views settings, we randomly select subsets (4, 8, 12, 16, 20 views) from the training views, while maintaining consistent test view across all experiments ensure fair comparison. Following RawNeRF, we assess the rendering quality of various methods using PSNR, SSIM, and LPIPS metrics in the RGB domain. Additionally, we report the frames per second (FPS) to evaluate the real-time rendering capabilities of each method.
Limited views training results. In Fig. 5 we present the quantitative results of our approach compared other baselines in limited views training settings. At first glance, is evident that our method (indicated by red lines) significantly surpasses both the LDR Scaffold-GS, HDR Scaffold-GS. and two-stage methods across all view numbers. Compared LDR Scaffold-GG -S (green lines), which utilizes RGB images processed by an ISP, our method achieves approximately a 4 dB improvement PSNR, demonstrating the benefits training with raw images. Additionally, when both are trained with raw images, our approach outperforms HDR Scaffold-GS (pink lines), optimized by LRR by approximately dB in PSNR across different training views, and about times faster FPS with limited 4-view settings, highlighting the efficiency of our noise robust reconstruction loss Lur- Furthermore, our method also exceeds the two-stage denoiser + 3DGS pipelines by dB, which may lose high-frequency details during the denoising process. Fig. 6 provides visualization comparison. Notably, with 12 views, both LDRR- and HDR- Scaffold-GSs exhibit significant elongated Gaussian artifacts, while our method produces more accurate and detailed reconstructions.
https://bmildd: wit.bb..gitttu......... wtw.uu/...rrraeeeee @ google-research. Licensed under the Apache License, v2.0.

---

Table Quantitative results of our approach and other baselines in full views training settings.
Affine-aligned RGB
Method
Loss
Raw PSNRT
FPS
PSNR1
SSIM
LPIPS,
BM3D
58 38.46
22.70
0.526
0.521
ELD
22
54.70
19.82
0.511
0.544
80
PMN
6
53.69
19.00
0.498
0.584
94
NeighborrNeiighbor
M
RRINNRR
58,40
22.22
0.531
0.518
78
LGBPN BT
53.62
19.09
0.481
0.624
84
RViDeNet (36
57.59
22.05
0.518
0,548
74
LDR Scaffold-GS
Canas
17.34
0.486
0.622
56
HDR Scaffold-GS
CRWNaR
58.08
22.69
0.521
0.513
Ours
Cur
59.49
23.53
0.535
0.499
80
Table 2: Ablation study on the lens distortion D(-).
Table 3: Ablation study on the training time.
All models are trained using full views.
Nc2Ne [I]
LGBPN (3I]
Method
D(-)
PSNR RGB PSNR RGB SSIM RGB LPIPS
Scaffold-GS 21
Ours
w/o
59.33
23.42
0.531
0.509
Time cost
5.3h
w/
59.49
23.53
0.535
0.499
3.th
Full-views training results. We also benchmarked our method against other baselines in views training settings. As shown in Table. and Fig. / our method consistently outperforms all other baselines across PSNR, SSIM, and LPIPS, even with a large number of training views. Besides, visualizations demonstrate that method recovers images with better detail and color accuracy.
Table 4: Ablation study on the selection of And and Acov. All models are trained using full views.
Aua
Azo
Raw PSNR
RGB PSNR
RGB SSIM
RGB LPIPS
0.5
56.36
21.95
0.508
0.591
10
20
22.18
0,509
0,539
55.38
22.41
0.515
0.523
5
57.10
22.52
0.516
0,520
0.5
50
56.72
21.98
0.514
0.576
5
20
59.49
23.53
0.535
0.499
5.3 Ablation study and discussions
Impact of lens distortion function D(-). As detailed in Table 2 the absence of lens distortion correction results in marginal decrease in reconstruction quality. Tn RawNeRF datasets, this effect negligible due the minimal lens distortion inherent the iPhone X. However, for cameras with more pronounced distortion characteristics, the impact could be significantly more substantial.
Impact of pyperparaeeters. The hyperparameters And and Acvv within the Lurr play pivotal role in striking the right balance between denoising efficacy and gradient accumulation. As delineated in Table undersized Aud leads to inadequate constraints on the noise model, culminating in imprecise noise extraction. On the flip side, an over- aeellmell And engenders excessively strong gradients emanating from the noise regularization loss, which can detrimentally impact the optimization process of 3DGS. The parameter Acav when assigned values that are either too low or too high, can incite similar challenges.
Limitations. Our method has several limitations. The noise extractor and covariance loss computation require more training time compared to standard Scaffold-GS. shown in Table 3 Additionally, some high-frequency signal distributions are similar to noise, making difficult to differentiate them using the distribution divergence loss. This can result in over-smoothing in full-views training settings, as discussed the Supplemental Material.
6 Conclusion
this work, we identify the unavoidable noise in raw images as a significant detriment to the rendering quality and speed of HDR 3DGS, particularly in scenarios with limited training views. We provide comprehensive analysis of how noise affects the optimization of 3DGS, modeling its relationship with both the number of training views and the noise distribution. To tackle these challenges,

---

propose a novel self-supervised framework that incorporates a noise-robust reconstruction loss. This framework utilizes a physically- basee noise model simultaneously denoise and enhance the HDR representation within noisy raw images. Our approach markedly outperforms LDR/HDR 3DGS that employs 3DGS/RRwNNRR loss, as well as both self-supervised and supervised pre-trained two-stage methods, in terms of rendering quality and speed across various training view counts.
Acknowledgements
This work was partially supported by the National Research Foundation Singapore Competitive Research Program (award number CRP29-2022-00033.
References
[1] A. Abdelhamed, S. Lin, and M. S. Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1692-1700, 2018.
[2] C. Chen, Q. Chen, J. Xu, and V. Koltun. Learning to in the dark. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 3291-3301 2018
[3] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. Image denoising by sparse 3-d monsform-domaln collaborative filtering, IEEE Transactions on image processing, 16(8):2080 2095, 2007.
[4] Y. Deng, J. Yang. J. Xiang, and X. Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10673-10683) 2022.
[5] G. Eilertsen. R. K. Mantiuk, and Unger. Real-time noise-aware tone mapping. ACM Transactions on Graphics (TOG), 34(6):1-15, 2015.
[6] H. Feng, Wang, Y. Wang. and H. Huang. Learnability enhancement for low-light raw denoising: Where paired real data meets noise modeling. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1436-1444, 2022.
[7] S. Fridovich- Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[8] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2021.
[9] S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy. Burst photography for high dynamic range and low-light imaging mobile cameras. ACM Transactions on Graphics (ToG), 35(6):1 12, 2016.
[10] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875 5884,,8847777 2021.
[11] T. Huang, S. Li, X. Jia, H. Lu, and J. Liu. Neighbor2ncighhor: Self-supervised denoising from single noisy images. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14781-14790, 2021.
[12] M. S. Joens, C. Huynh, J. M. Kasuboski, D. Ferranti, J. Sigal, F, Zeitvogel, M. Obst, C. Burkhardt, K. P. Curran, S. H. Chalasani, et al. Helium ion microscopy (him) for the imaging of biological samples at sub-nanona no--nanometer resolution. Scientific reports, 3(1):3514, 2013.
[13] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[14] S. Koskinen, D. Yang, and J.-K. Kämäräänen. Reverse imaging pipeline for raw rgb image augmentation. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2896-2900. IEEE, 2019.
[15] N. Levin, C. C. Kyba, Q Zhang, A. S. de Miguel, M. O. Román, X. Li, B. A. Portnov, A. L. Molthan, A. Jechow, S. D. Miller, et al. Remote sensing of night lights: A review and an outlook for the future. Remote Sensing of Environment, 237:111445 2020.
10

---

[16] C. Li Y. Zhao, W. Zhu, and Y. Lin. Ri-nerf: Real-time on-device neural radiance fields towards immersive ar/vr rendering. In Proceedings of the 41st IEEE/ACM International Conference on Computer- Aided Design, pages 1-9, 2022.
[17] L. Li, W. Zhu, B. Yu, Zhao, C. Wan, H. You, H. Shi, and Y. Lin. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In Proceedings of the 50th Annual International Symposium on Computer Archilecture, pages 1-13, 2023.
[18] Z. Li,L. L. Li, and J. Zhu. Read: Large-scale neural scene rendering for autonomous driving. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 1522-1529, 2023.
[19] Z. Li, M. Lu, X. Zhang, X. Feng, M. S. Asif, and Z. Ma. Efficient visual computing with camera raw snapshots. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[20] C.-IL Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng. X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[21] T. Lu, M. Yu, L. Xu, Y. Xiangli, L. Wang, D. Lin, and B. Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. arXiv preprint arXiv:2312.00100 2023.
[22] G. Metzer, F. Richardson, O. Patashmik, R. Giryes, and D. Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12663-12677 12673, 2023.
[23] B. Mildenhall, P. Hedman, R. Martin-Brualla, P. Srinivasan, and J. T. Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16190-16199, 2022.
[24] B. Mildenhall, P. P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1);99- 106, 2021.
[25] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution bash encoding. ACM transactions on graphics (TOG), 41 2022.
[26] C. Reiser, S. Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14335-14345 14345, 2021.
[27] J. L. Schonberger and J.M. Frahm. Structureffrommmmotion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
revisits
[28] T. Takikawa, A. Evans, J. Tremblay, T. Müller, M. McGuire, A. Jacobson, and S. Fidler. Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[29] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop. D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358-11361 2021.
[30] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9446-9454, 2018
[31] Z. Wang, Y. Fu, J. Liu, and Y. Zhang. Lg-bpn: Local and global blind-patch network for sell-supervised real-world denoising. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 18156-18165, 2023.
[32] K. Wei, Y. Fu, J. Yang, and H. Huang. A physics-based noise formation model for extreme low-light raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2758-2767, 2020.
[33] X. Wa, J. Xu, Z. Zhu, H. Bao, Q. Huang, J. Tompkin, and W. Xu. Scalable neural indoor scene rendering. ACM transactions on graphics, 41(4), 2022.
[34] J. Xu, L. Peng, H. Cheng, L. Xia, Q. Zhou, D. Deng, W. Qian, W. Wang, and D. Cai. Regulating intermediate 3d features for vision-centric autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6306-6314, 2024.
[35] A. Yu, R. Li, M. Tancik, H. Li, R. Ng. and A. Kanazawa. Plenoctrees for real- time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
11

---

[36] H. Yue, C. Cao, L. Liao, R. Chu, and J. Yang. Supervised raw video denoising with benchmark dataset on dynamic scenes. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 2301-2310, 2020.
[37] X. Zhang, K. Matzen, V. Nguyen. D. Yao. Y. Zhang, and R. Ng. Synthetic defocus and look-ahead autofocus casual videography. arXiv preprint arXiv:1905.0633: 06326, 2019.
[38] H. Zhu, S. Xie, Z. Liu, F. Liu, Zhang, Y. Zhou, Y. Lin, Z. Ma, and X. Cao. Disorder-invariant implicit neural representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024,
[39] Z. Zhu, Z. Fan, Y. Jiang, and Z. Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. In European Conference on Computer Vision, pages 145-163. Springer, 2025.
12

---

From Chaos to Clarity: 3DGS in the Dark - Supplemental Material
In this supplement, we begin by discussing the broader impacts of our method. Next, outline the distribution and calibration of the camera noise model. Following that, we elaborate on the lens distortion function (--). Finally, we address the failure cases of method mentioned in the limitations section.
A Broader Impacts
The broader social impacts of work can be summarized as follows:
1. Enhanced Applications: By significantly improving 3DGS performance in low-light conditions, our work has the potential broaden the scope of its applications across various fields such as medical imaging, autonomous driving, and surveillance systems. These advancements could lead better outcomes in scenarios where lighting a critical factor.
2. Open Access and Collaboration: In the spirit of fostering innovation and collaboration within the research community, we will release our code as open-source. This will allow other researchers and developers to build upon our work, potentially leading to further advancements and new applications.
We have carefully considered the potential impacts of our work and do not foresee any serious negative consequences. Our contributions are intended to promote positive developments and collaboration in the field of 3DGS technology.
B Distribution of the camera noise model
Building on the work of ELD 32 and PMN 16 the noise components in Eq.(4) adhere to specific
distributions:
nshot P
X,
nread N (0, o read):
(16)
fsp ISO fpp + -
where & is the overall system gain associated with the ISO setting, and P and / represent Poisson and Gaussian distributions, respectively. The terms pp and fh R are pixel-wise dark frame noise components. Consistent with ELD and PMN (6. the relationships among k; read, and ISO are expressed as:
k = ISO + bj.
(17)
log (Dread) aread log(k) + bread:
Parameters T. pp Rpp ag. bg. Gread, and bread are calibrated for each camera using series of flat-frame and fark-frame images captured at various ISO levels.
C Calibration of the camera noise model parameters
The calibration process comprises three steps:
1. Calibrating for each ISO level:
Capture 25 flat frames under consistent lighting for each exposure time Exp,-
Calculate the mean and variance for each color block, yielding 24 mean- mean-variance pairs per exposure time.
With three exposure times per ISO, gather 72 mean- variance pairs per ISO.
Model nebot as (xx, x k)), where x the mean and x the variance, and calibrate from the merinvariance relationship. Exclude points with mean values beyond 1/4 saturation due to clipping effects.
2. Calibrating nfp, and Gread::
13

---

FAZIOLI
FAZIOLI
(a) HDR Scaffold-GS (PSNR: 24.59 dB)
(b) Ours (PSNR: 24.85 dB)
Figure 8: Visualization of our failure cases on the RawNeRF dataset 23 Despite achieving a higher PSNR than HDR Scaffold-GS by eliminating noise-induced artifacts, approach resulted in overly smooth outcomes in areas with reflections.
Capture 100 dark frames at each ISO dark room.
The mean of these dark frames gives nyp; representing fixed pattern noise.
Subtract nyp, from all dark frames and calculate variance across the total frame for Bread-
Fitting ISO-related parameters:
Repeat the above steps for different ISO levels to obtain set of parameters fpp, Tread,
Fit aj, by, fpps pb, Gread, bread based on these parameters and equations Eq
D Lens distortion mapping process
To align the distorted coordinates with their accurate positions, we compute reverse mapping from (Ia. ya) to (z. y) using Newton-Raphson iterative method. This method minimizes the residuals between the distorted and true coordinates until the adjustments are beneath a predefined accuracy threshold. Crucially, this distortion mapping is computed once before the training process, thereby minimizing impact on computational efficiency.
using the following equations:
x (1 + kir² + k2r" k3r kgr²) + 2piay + p2 (r2 f 2x²)
(18)
Vd = (1 + kjr + k2r + kgr + k4r + Pı (r2 + 2ny) + 2p2xy,
where 22 y2 represents the radial distance from the center, and k1, k2, kz, Krs P1, P2 are the coefficients for radial and tangential distortions, respectively.
Since and y are not integers, use bilinear interpolation obtain the values of xa and ya-
E
Failure case
Upon examination, we have observed that certain high- figh-frreuency signals exhibit a distribution pattern that closely resembles noise, making it difficult for the noise divergence loss to differentiate between them effectively. Consequently, this has resulted in overly smooth outputs in some areas rich in detail, particularly when training with full views.
Differentiating high-frequenc, signals from noise is an intricate challenge. We acknowledge the complexity of this issue and are committed further exploring potential solutions in our future research endeavors.
14

---

Table 5: Quantitative Comparison on LLFF Datasets (3 Training Views)
1/4*
1/8*
Method
Loss
Raw PSNRT
RGB PSNR
FPS
Raw PSNR1
RGB PSNR
FPS1
BM3D PMN
TRouNeRF
42.902
18.032
163
22.596
17.242
272
LRowNeRF
42.585
18,064
201
23.208
18.572
389
Neighbor2Neighbor
LRmaNeRF
42.774
17.977
152
23.008
18.297
372
FSGS
LRawNeRF
41.197
15.496
65
21.183
15.166
Ours
Lun
43.418
18.753
216
23.799
18.902
370
*denotes the downsample ratio the resolution, as same as (39
Extended evaluation on LLFF dataset
Given the limitations of the RawNeRF dataset, conducted additional quantitative comparisons using the LLFF dataset with simulated noisy raw images and provided qualitative comparisons for the training scenes in the RawNeRF dataset. Specifically, we used the inverse ISP proposed in [14] to convert LLFF RGB images into RAW images, added synthetic noise following the iPhone X noise model, and adhered baseline methods' settings We used FSGS [39 the 3D representation baseline, modifying only the loss function to our proposed Lal.. The results are listed in Table
G Visual comparison between NeRF-based and 3DGS-based methods
Tranin Tuec LS0
Traeiis Tiac Lak
Traenine Tiee
Bonkering
Boskrnng
sSMM: 1557
0,497
Sal::
Liis::
um
LPIS:
FIZIO
FA1IO
FAZIO
(a) RawNeRF
(b) HDR Scaffold-GS
(c) Ours
Figure 9: Visual comparison between NeRF-based and 3DGS- -based methods full views training settings (Zoom in for best view). Noise impact is unique to 3DGS. Compared to RawNeRF, HDR Scaffold-GS is more vulnerable noise, generating many thin-flat 3D Gaussian points to overfit it. The MLP in RawNeRF, acting low-pass filter, does not have this issue. Additionally, the rendering speed 3DGS affected by the number Gaussian points, degrading due to noise points. Our method avoids these noisy Gaussian points, boosting rendering speed and quality.
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction outline the primary contributions and scope of our paper, providing accurate representation of the methodologies and findings discussed in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The limitations our work are discussed the "Limitations" section of paper.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification: We provide detailed list of assumptions and proofs for all theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We detail all the necessary information including experiment settings and hyperparameters reproduce the main experimental results in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Our code is under internal review and will be released upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail all the data splits, hyperparameters, and optimizer settings in the experimental section of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Considering test method across broad range of training views and provide a detailed comparison with existing baselines, the extensive computational resources required to run the entire of experiments several times make infeasible report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We report the GPU type, and time of execution for each experiment in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:
Answer: [Yes]
Justification:
All data and experiments comply with ethical guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our paper in the "Broader impacts" section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no risks that require safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We label the original datasets used in the paper and provide proper credit to the creators.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not introduce new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects in the
рaper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing nor research with human subjects in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "Fe8tHmJsXo", "reviewer": "Reviewer_m8Xp", "summary": "This paper presents a self-supervised learning framework for reconstructing HDR 3DGS from noisy raw images. The proposed method integrates a noise extractor and a noise-robust reconstruction loss to mitigate the effects of noise in raw images. By leveraging a noise distribution prior, the framework improves both the reconstruction quality and inference speed of 3DGS, particularly in scenarios with limited training views. Experimental results on the RawNeRF dataset show that the proposed approach significantly outperforms existing state-of-the-art methods in terms of rendering quality and speed, providing a robust solution for HDR 3DGS reconstruction in challenging lighting conditions.", "questions": "See the weakness.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Proposed to both denoise first and denoise on-the-fly, then using 3DGS based method to reconstruct the scene. Perform better than the other denoising method.\n2. Gain better performance with fewer views.", "weaknesses": "1. It seems that there are only two scenes for qualitative comparison. To the best of my knowledge, there are only three scenes for testing in the RawNeRF dataset, with each scene containing only one view. This suggests a potential lack of evaluation. The author should provide more results, at least for qualitative comparison.\n2. The proposed analysis seems to be not very relevant to the main text, and I did not find its connection with the proposed noise-robust reconstruction loss. Why is a self-supervised denoiser needed if N (in Eq. 9) is small? And if N is large, is a self-supervised denoiser still necessary? Additionally, the proposed analysis is very similar to the motivation of RawNeRF and burst denoising, which uses more (large N in Eq. 9) unstabilized noisy views for denoising. Therefore, I do not see the novelty in the first contribution.\n3. the author seems to have not retrained the SOTA denoisers (e.g., ELD, PMN) with the calibrated noise model of the iPhone X. This presents an unfair comparison. I think the authors should provide results using the iPhone X noise model retrained, given that the proposed method also requires noise calibration. If the results are comparable to those of the proposed method, then why is the proposed method needed? Wouldn't it be sufficient to perform denoising first and then reconstruction simply?\n4. the author did not provide results compared with the original 3DGS. Why is that? It is not a big deal, but I think a comparison with RawNeRF should be provided (at least in Table 1, the results with full training views).\n5. There might be a typo: if the authors directly use the pretrained checkpoint (as mentioned in lines 224-225), why is the loss for ELD, PMN, etc. denoted as $L_\\text{RawNeRF}$ in Table 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your thorough review and valuable feedback on our submission. Here are our detailed responses:\n\n1. **More Quantitative and Qualitative Results:**\n\n   Given the limitations of the RawNeRF dataset, we conducted additional quantitative comparisons using the LLFF dataset with simulated noisy raw images and provided qualitative comparisons for the training scenes in the RawNeRF dataset. Specifically, we used the inverse ISP proposed in \\[1] to convert LLFF RGB images into RAW images, added synthetic noise following the iPhone X noise model, and adhered to baseline methods' settings \\[2]. We used FSGS \\[2] as the 3D representation baseline, modifying only the loss function to our proposed $L_{\\text{nrr}}$. The results are listed in **Table R1**. Our method consistently outperforms others even with limited view settings and diverse training scenes. For qualitative results, please refer to **Fig. R1 in our submitted PDF**, which shows that our method produces smoother results and reduces artifacts.\n\n2. **Connection Between the Analysis and Main Text:**\n\n   Our analysis highlights an overlooked issue: 3DGS is more vulnerable to noise in few-shot settings compared to neural radiance field-based 3D reconstruction. Unlike RawNeRF, 3DGS requires explicit noise regularization. Specifically, in Sec. 4.1, we illustrate the connection between (N) and the bias of the 3DGS optimal target and real-world point color. As (N) decreases, the bias increases, necessitating a denoiser to prevent 3DGS points from overfitting to noisy training views, as shown in Fig. 6 of our manuscript. This issue is common in scenarios like multi-camera 3D reconstruction and street scene reconstruction in autonomous driving. Although the effect diminishes with larger (N), it remains significant, as evidenced by our results with 100-view settings (Fig. 7 and Table 1).\n\n3. **Novelty of First Contribution:**\n\n   **The impact of noise is unique in 3DGS compared to NeRF-based methods.** MLPs in NeRF can perform low-frequency filtering, making it challenging to overfit to high-frequency noise, as explored in \\[3]. RawNeRF requires ~5 days to train a single scene and 2 minutes to render a new view in 4K resolution, whereas 3DGS can achieve training in 2 hours and 50 FPS rendering speed. However, 3DGS tends to generate thin-flat noise, degrading rendering speed and quality. Visual details comparing NeRF and 3DGS can be found in **Fig. R2 of our submitted PDF**. We revise our first contribution in the intro. as follows:\n\n   > We explore the unique impact of noise on 3DGS and its relationship with the number of input views. We highlight that 3DGS is more vulnerable to noise than neural radiance field-based 3D reconstruction, especially in few-shot settings, and provide a detailed analysis of how noise impacts the optimization of 3DGS, modeling its relationship with the number of training views and the noise distribution.\n\n4. **Training ELD/PMN with iPhone X Noise Model:**\n\n   There is a gap between simulated iPhone X noise on DSLR images and real noisy iPhone X images due to **differences in bit depth (14-bits vs. 12-bits)**. Following your suggestion, we simulated noise-clean paired data using the iPhone X noise model and performed denoising first, followed by reconstruction. The results, listed in **Table R2**, do not show significant performance improvement, likely due to sensor differences between the iPhone X and DSLR cameras.\n\n5. **Comparison with 3DGS and RawNeRF:**\n\n   We agree that a comparison with 3DGS and RawNeRF is valuable. We have added this comparison in **Table R2** under full-view settings. Our method shows that the original 3DGS is more prone to noise, resulting in lower reconstruction quality. Although NeRF-based methods achieve similar results due to their low-pass characteristics, they require significantly more time (5000x for rendering a single image and 100x for training) compared to our method, making them impractical for real-world applications.\n\n6. **Pretrained Checkpoint and Loss Function:**\n\n   The denoising method in Table 1 is used to first denoise the input, and then the loss function is applied to 3DGS for the 3D reconstruction loss. **Thus, the loss function is only used during the 3D reconstruction training period.**\n\nOnce again, we sincerely thank you for your constructive feedback.\n\n***\n\n**Tables and References:**\n\n**Table R1:** Comparison on LLFF Datasets (3 Training Views)\n\n|**Method**| **Loss**|**Raw PSNR (1/4\\*)**|**RGB PSNR (1/4\\*)**|**FPS (1/4\\*)**|**Raw PSNR (1/8\\*)**|**RGB PSNR (1/8\\*)**|**FPS (1/8\\*)**|\n|---|---|:---:|:---:|:---:|:---:|:---:|:---:|\n|BM3D|$L_{\\text{RawNeRF}}$|42.902|18.032|163|22.596|17.242|272|\n|PMN|$L_{\\text{RawNeRF}}$|42.585|18.064|201|23.208|18.572|**389**|\n|Neighbor2Neighbor|$L_{\\text{RawNeRF}}$|42.774|17.977|152|23.008|18.297|372|\n|FSGS|$L_{\\text{RawNeRF}}$|41.197|15.496|65|21.183|15.166|132|\n|Ours|$L_{\\text{nrr}}$|**43.418**|**18.753**|**216**|**23.799**|**18.902**|370|\n\n* denotes the downsample ratio of the resolution, as same as [2].\n\n**Table R2:** Comparison on RawNeRF Datasets (Full Training Views)\n\n|Method|Loss|TrainingTime|FPS|RawPSNR|RGBPSNR|RGBSSIM|RGBLPIPS|\n|---|---|:---:|:---:|:---:|:---:|:---:|:---:|\n|ELD|$L_{\\text{RawNeRF}}$|1.5h|80|54.70|19.82|0.511|0.544|\n|PMN|$L_{\\text{RawNeRF}}$|1.4h|**94**|53.69|19.00|0.498|0.584|\n|ELD(fine-tuned)|$L_{\\text{RawNeRF}}$|17.7h|79|55.01|19.96|0.514|0.531|\n|PMN(fine-tuned)|$L_{\\text{RawNeRF}}$|18.6h|92|53.97|19.22|0.503|0.575|\n|HDR3DGS|$L_{\\text{RawNeRF}}$|1.1h|75|57.39|20.83|0.518|0.569|\n|RawNeRF|$L_{\\text{RawNeRF}}$|140h|0.01|59.07|**23.53**|**0.538**|0.500|\n|HDRScaffold-GS|$L_{\\text{RawNeRF}}$|1.6h|73|58.08|22.69|0.521|0.513|\n|Ours|$L_{\\text{nrr}}$|3.1h|80|**59.49**|**23.53**|0.535|**0.499**|\n\n**References:**\n\n[1]  Reverse Imaging Pipeline for Raw RGB Image Augmentation\n\n[2]  FSGS: Real-Time Few-Shot View Synthesis using Gaussian Splatting\n\n[3]  DINER: Disorder-Invariant Implicit Neural Representation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xwz1ML8dPu", "reviewer": "Reviewer_ZDPY", "summary": "This paper investigates the issue of 3DGS overfitting noises in input images, and proposes a self-supervised learning framework as the solution. The paper integrates a noise model as prior to relax the constraints in the 3DGS optimization framework.", "questions": "see weakness", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a detailed analysis of how noise impacts the optimization of 3DGS.\n2. The proposed framework leverages a physical-based noise model to jointly denoise and enhance 3DGS with noisy inputs.\n3. The paper is well-written.", "weaknesses": "1. It is better to introduce comparisons on more sparse-view settings such as LLFF datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your review and valuable suggestions on our paper. In response to your suggestion regarding comparisons on more sparse-view settings, we have conducted additional quantitative comparisons using the LLFF dataset with simulated noisy raw images. Below are the details of our approach and findings:\n\nTo address the more sparse-view settings (**3-views**), we used the inverse ISP proposed in \\[1] to convert the LLFF dataset RGB images into RAW images. We then added synthetic noise following the iPhone X noise model and adhered to the baseline methods' settings \\[2] for a fair comparison. Specifically, we used FSGS \\[2] as the 3D representation baseline and only modified the loss function to our proposed $L_{\\text{nrr}}$. The results, **presented in Tables R1**, demonstrate that our method consistently outperforms other methods even with limited view settings.\n\nOnce again, we appreciate your constructive feedback and hope that our response addresses your concerns.\n\n***\n\n**Tables and References:**\n\n**Table R1:** Quantitative Comparison on LLFF Datasets (3 Training Views)\n\n| **Method**        | **Loss**              | **Raw PSNR (1/4)\\*** | **RGB PSNR (1/4)\\*** | **FPS (1/4)\\*** | **Raw PSNR (1/8)\\*** | **RGB PSNR (1/8)\\*** | **FPS (1/8)\\*** |\n| ----------------- | --------------------- | :----------------: | :----------------: | :-----------: | :----------------: | :----------------: | :-----------: |\n| BM3D              | $L_{\\text{RawNeRF}}$ |      42.902        |       18.032       |      163      |      22.596        |       17.242       |      272      |\n| PMN               | $L_{\\text{RawNeRF}}$ |      42.585        |       18.064       |      201      |      23.208        |       18.572       |     **389**   |\n| Neighbor2Neighbor | $L_{\\text{RawNeRF}}$ |      42.774        |       17.977       |      152      |      23.008        |       18.297       |      372      |\n| FSGS              | $L_{\\text{RawNeRF}}$ |      41.197        |       15.496       |       65      |      21.183        |       15.166       |      132      |\n| Ours              | $L_{\\text{nrr}}$     |    **43.418**      |       **18.753**       |      **216**      |    **23.799**      |     **18.902**     |      370      |\n\n\\* * denotes the downsample ratio of the resolution following the same settings as in \\[2].\n\n**References:**\n\n[1]  Reverse Imaging Pipeline for Raw RGB Image Augmentation; Samu Koskinen, Dan Yang, and Joni-Kristian Kämäräinen, ICIP 2019\n\n[2]  FSGS: Real-Time Few-Shot View Synthesis using Gaussian Splatting; Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang, ECCV 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cV6vD62pSC", "reviewer": "Reviewer_JuHQ", "summary": "This paper proposes a novel self-supervised learning framework to reconstruct HDR 3D Gaussian Splatting (3DGS) from noisy raw images. This addresses the issue of noise degrading reconstruction quality and inference speed in 3DGS, especially in scenarios with limited views. The proposed method demonstrates superior performance compared to LDR/HDR 3DGS and previous state-of-the-art models in both reconstruction quality and inference speed on the RawNeRF dataset.", "questions": "Figure 5 is hard to understand since there are these Loss function symbols. It remains unclear to me what exactly is visible there. Maybe it would be good to show only the most relevant baseline in these figures.", "limitations": "The authors provide a section for discussing the limitiations their methods.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well written, the methodology is well motivated and technical sound.\n2) The overall problem is relevant and the introduction of a noise-free loss functions for 3DGS is plausible and valid contribution. The approach of trying to model the noise for a noise-robust representation is a original idea.", "weaknesses": "1) The experimental evaluation misses a comparison to NeRF-based methods on RAW data, such as RawNeRF. While the NeRF-based methods have their downsides, they are still a good baselines to judge the actual denoising capabilities of 3DGS-based methods.\n2) The impact of the noise extractor F is still unclear to me. The authors mention that this needs to be pre-trained with a paired dataset of noisy and noise-free image pairs. It would be good to investigate this component more by e.g. providing ablation of this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thorough review and valuable suggestions. Here are our detailed responses to your comments:\n\n1. **Comparison with NeRF Baselines:**\n\n   We acknowledge the importance of comparing our method with NeRF-based methods. We have conducted a comparison between NeRF and HDR Scaffold-GS baselines and our proposed method **in Table R1**. We observed that the unmodified 3DGS (HDR Scaffold-GS) exhibited a gap in rendering quality compared to the NeRF-based method (RawNeRF), as 3DGS is more prone to noise. This is discussed in our manuscript and illustrated **in Figure R2 of our submitted PDF**. Our method bridges this gap, achieving similar performance to RawNeRF while significantly **reducing training time (by a factor of 100) and rendering speed (by a factor of 5000)**, making raw 3D reconstruction feasible in real-world applications.\n\n2. **Ablation of the Noise Extractor F:**\n\n   We want to clarify that pretraining the noise extractor F is necessary because U-Net struggles to converge and extract high-frequency noise within 30,000 iterations during 3DGS training. Pretraining F ensures it can effectively extract high-frequency noise. Our pretrained dataset, SID (captured by a DSLR camera), differs significantly from the RawNeRF dataset (captured by an iPhone X) in terms of color space and bit depth. To verify the impact, we also experimented with a self-supervised method like Neighbor2Neighbor (using one output as the clean image) with noise-only images in SID. The results, shown **in Table R2**, indicate no significant difference, supporting **noise-clean paired images are not strictly necessary** during the pretraining stage.\n\n3. **Clarification of Figure 5:**\n\n   We apologize for the unclear presentation in Figure 5. The **solid line** in Figure 5 represents the **one-stage** method, corresponding to the last three rows in Table 1. The **dashed line** denotes the **two-stage** method, corresponding to the rest of Table 1. The **one-stage** method **directly trains the 3DGS on the noisy images** with different loss functions. In the **two-stage** method, the **input is first denoised using a pretrained denoiser**, and then the 3DGS is trained with RawNeRF loss functions. We will clarify this in the revised version of our manuscript for better clarity.\n   \nOnce again, we sincerely thank you for your constructive feedback, which has been crucial in refining our work.\n\n***\n\n**Tables:**\n\n**Table R1:** Quantitative Comparison on RawNeRF Datasets (Full Training Views)\n\n| Method           | Loss                | Training Time | FPS  | Raw PSNR | RGB PSNR | RGB SSIM | RGB LPIPS |\n|------------------|---------------------|:-------------:|:----:|:--------:|:--------:|:--------:|:---------:|\n| RawNeRF          | $L_{\\text{RawNeRF}}$ | 140h          | 0.01 |   59.07  |   **23.53**  |   **0.538**  |   0.500   |\n| HDR Scaffold-GS  | $L_{\\text{RawNeRF}}$ | 1.6h          |  73  |   58.08  |   22.69  |   0.521  |   0.513   |\n| Ours             | $L_{\\text{nrr}}$     | 3.1h          |  80  |   **59.49**  |   **23.53**  |   0.535  |   **0.499**   |\n\n**Table R2:** Quantitative Comparison on RawNeRF Datasets (Full Training Views)\n\n| Method           | F Pretrained Dataset               | F Pretrained Method  | Raw PSNR | RGB PSNR | RGB SSIM | RGB LPIPS |\n|------------------|---------------------|-------------|:--------:|:--------:|:--------:|:---------:|\n| Ours             | SID (noise)     | self-supervised (Neighbor2Neighbor)          | 59.32  |   23.45  |   0.530  |   0.505   |\n| Ours             | SID (noise-clean)     | supervised          |   59.49  |   23.53  |   0.535  |   0.499   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oyKY2yZx9A", "reviewer": "Reviewer_MTeY", "summary": "The paper tries to use raw images with high dynamic range (HDR) for training 3D Gaussian Splatting (3dgs). It first analyzes how noise from raw image affects the optimization of 3dgs especially when the number of training views is small. To address this issue, first it uses a lens distortion correction network before training to correct the distortions of raw images. Then it introduces a noise extractor to predict noise from raw image and presents a novel noise-robust reconstruction loss which consists of a term of RawNeRF loss, a NLL measuring the divergence between the estimated noise and the physical noise, and a term for decorrelating noises across pixels. The method outperforms baselines on RawNeRF dataset on rendering quality and inference speed.", "questions": "Please see the weaknesses.", "limitations": "1. The calibration process for camera noise model parameters seems to be pretty time-consuming and requires capture a lot of images, e.g. in L413 \"capture 100 dark frames at each ISO in a dark room\". Wonder if this method can be applied to images captured in a more home-like setting or it always requires a dataset prepared in a professional setting.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper mainly tries to use a prior noise model to eliminate the effects of noise in raw image, pushing a step forward to utilizing HDR raw images in 3dgs.\n2.The paper includes extensive comparison with baselines including LDR Scaffold-GS, HDR Scaffold-GS and two-stage methods with various denoiser as first stage. And it achieves supreme results compared to these baselines.\n3. The paper analyzes and demonstrates the effect of noises from raw images on 3dgs optimization and its relationship with the number of training views. It does experiments on both full views and limited views.  The whole structure is well-organized and complete.", "weaknesses": "1. The method assumes clean pixel values of a 3d point projected on different images remain the same. But these images are taken from different viewpoints. How does it handle the effects of viewing directions for non-Lambertian surfaces?\n2. The $L_{nd}$ loss consists of two terms. I am especially curious about the effect of the $L_{cov}$. How much would it help with the method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable suggestions and comments. Here are our detailed responses to your feedback:\n\n1. **Lambertian Surface Assumption:**\n\n   **We use the Lambertian surface assumption to simplify our analysis.** However, our method can also be applied to non-Lambertian surfaces, such as mirror reflections on a piano. The core of our method is the integration of a noise model to combine the noise prior with 3DGS reconstruction. **Since 3DGS supports non-Lambertian surfaces, our method is also applicable in such scenarios.**\n\n2. **Effect of $L_{cov}$:**\n\n   The $L_{cov}$ term ensures that noise **remains independent across pixels** and **maintains the standard deviation** of the noise distribution within a pixel. We want to clarify that the ablation of $L_{cov}$ ($\\lambda_{cov}$) is already presented in **Table 4** of our paper. We also provide a visual illustration in **Figure R3** of our submitted PDF. As shown, $L_{cov}$ ensures the noise remains independent across different pixels and adjusts the extracted noise distribution within a pixel to follow a standard Gaussian distribution after normalization. Since $L_{nd}$ only ensures the mean of extracted noise is zero, it does not control the standard deviation. Therefore, constraining the diagonal of the covariance matrix in $L_{cov}$ is necessary to ensure the standard deviation is one.\n\n3. **Home-like Setting:**\n\n   We have found that the noise model can also be calibrated in a home-like setting as follows:\n\n   a. **Flat Frames Capture:** Use a **MacBook screen** to capture flat frames. Adjust the focus of the mobile phone (iPhone X) camera to infinity, attach it tightly to the screen, and adjust the screen brightness to capture a series of flat frames at different ISO and exposure levels. This process is illustrated in **Figure R4(a) of the submitted PDF**.\n\n   b. **Dark Frames Capture:** Use **a drawer and a headphone** to capture dark frames. Connect the mobile phone (iPhone X) with a (wired/wireless) headphone, open the camera app, place the phone in the drawer, close the drawer, and use the headphone's play button to remotely take photos. Capture 50 dark frames at different ISOs. This setup is shown in **Figure R4(b) of the submitted PDF**.\n\n   These settings can be easily achieved in a home environment. We found that the noise parameters calibrated this way are very close to those calibrated in a lab setting (dark room) ($\\sigma_{read}$ 7.38 vs. 7.06). The impact on the final 3DGS reconstruction performance on RawNeRF is less than 0.2dB. Additionally, we have developed **a semi-automatic labeling app for Android mobile phones**, as shown in **Figure R4\\(c) of the submitted PDF**, which we will release upon the paper's acceptance. We hope this tool will enable more users to enjoy 3DGS reconstruction in low-light conditions using their own mobile phones.\n\nOnce again, we sincerely thank you for your constructive feedback, which has been crucial in refining our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qxIvJJq1VC", "reviewer": "Reviewer_h6sN", "summary": "This paper introduces a new framework for 3D reconstruction and denoising from raw images. Specifically, a noise extractor and a noise-robust reconstruction loss are proposed to deal with the overfitting issue of 3DGS to the noises heavily distributed in raw images. Experiments on the rawnerf dataset and ablation studies are provided to demonstrate the effectiveness of the proposed methods.", "questions": "Please refer to the weakness part.", "limitations": "The limitations mentioned in the paper are not easy to address, so it did not affect my rating for this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Intuitive noise decomposition and informative noise removal (or estimation) method\n2. Better reconstruction and denoising performance compared to 3DGS\n3. Good writing and presentation", "weaknesses": "1. The innovation may not be enough for this conference. The major contribution of this paper is the denoising part, where the noise estimator needs pre-training for good performance (extra data is required).\n2. The performance improvement over baseline with rawnerf loss is marginal and even no synthesis quality improvement compared to the reported scores in the original RawNeRF paper (which is based on NeRF).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable suggestions and comments. Here are our detailed responses to your feedback:\n\n1. **Novelty Limited Due to the Need for Noise-Clean Paired Data:**\n\n   We want to clarify that the pretrained model aims to provide a good initialization and **does not have strict restrictions on the data used for training** (e.g., **we can use data from different devices or noise-only data**). Specifically, our pretrained dataset, SID (captured by a DSLR camera), differs significantly from the RawNeRF dataset (captured by an iPhone X) in terms of color space and bit depth. To further verify the impact, we also experimented with a self-supervised pretraining method like Neighbor2Neighbor with noise-only images in SID. The results, shown in **Table R1**, indicate no significant difference, supporting that noise-clean paired images are not strictly necessary during the pretraining stage.\n\n2. **Comparison with Baselines and RawNeRF:**\n\n   We want to clarify that in the few-shot settings (which are more common in real-world scenarios like 3D reconstruction using multi-camera setups or for autonomous driving) on the RawNeRF dataset, **we achieved a 3-4 dB gain compared to the rawnerf loss baselines as presented in Figures 5 and 6**. In full-view settings, we also observed around a 0.9 dB gain compared to 3DGS with the rawnerf loss baseline. While our method's reconstruction quality is similar to RawNeRF, we significantly **reduced the training time by 100x and boosted the rendering speed by 5000x**, as shown in **Table R2**, making it more feasible for real-world applications.\n\nOnce again, we sincerely thank you for your constructive feedback, which has been crucial in refining our work.\n\n***\n\n**Tables:**\n\n**Table R1:** Ablation Study of F Pretraining on RawNeRF Datasets (Full Training Views)\n\n| Method           | F Pretrained Dataset               | F Pretrained Method  | Raw PSNR | RGB PSNR | RGB SSIM | RGB LPIPS |\n|------------------|---------------------|-------------|:--------:|:--------:|:--------:|:---------:|\n| Ours             | SID (noise)     | self-supervised (Neighbor2Neighbor)          | 59.32  |   23.45  |   0.530  |   0.505   |\n| Ours             | SID (noise-clean)     | supervised          |   59.49  |   23.53  |   0.535  |   0.499   |\n\n\n**Table R2:** Quantitative Comparison on RawNeRF Datasets (Full Training Views)\n\n| Method           | Loss                | Training Time | FPS  | Raw PSNR | RGB PSNR | RGB SSIM | RGB LPIPS |\n|------------------|---------------------|:-------------:|:----:|:--------:|:--------:|:--------:|:---------:|\n| RawNeRF          | $L_{\\text{RawNeRF}}$ | 140h          | 0.01 |   59.07  |   **23.53**  |   **0.538**  |   0.500   |\n| HDR Scaffold-GS  | $L_{\\text{RawNeRF}}$ | 1.6h          |  73  |   58.08  |   22.69  |   0.521  |   0.513   |\n| Ours             | $L_{\\text{nrr}}$     | 3.1h          |  80  |   **59.49**  |   **23.53**  |   0.535  |   **0.499**   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Thanks to All the Reviewers for the Insightful Comments

We would like to thank the reviewers for their efforts and insightful comments. We appreciate the reviewers’ acknowledgment of the **novelty**, **performance**, and **presentation** of our proposed method. For example:

- Reviewer JuHQ noted that "the paper is well written, the methodology is well motivated, and technically sound."
- Reviewer MTeY highlighted that "the paper includes extensive comparison ... and achieves supreme results compared to these baselines" and that "the whole structure is well-organized and complete."
- Reviewer h6sN pointed out the "better reconstruction and denoising performance compared to 3DGS" and praised the "good writing and presentation."
- Reviewer ZDPY appreciated the "detailed analysis of how noise impacts the optimization of 3DGS" and remarked that "the paper is well-written."

The questions or weaknesses mentioned by each reviewer are addressed separately in our responses. Please feel free to discuss with us if you have any further concerns or questions.

Meta-review: Following the discussion stage, the paper received four positive and one negative score. Several concerns were raised in the reviews, primarily related to the experiments. These concerns included the limited number of testing scenes, the lack of comparisons with RAWNeRF/3DGS, the need for retraining the state-of-the-art denoiser, and the results from sparse views. The rebuttal addressed these issues by providing additional experiments and results. Other concerns included the impact of the noise extractor, assumptions about diffuse reflection, and the complexity of the calibration process, all of which were also addressed in the rebuttal.

A significant point of discussion was the improvement compared to RAWNeRF. While the improvement is similar to that of RAWNeRF, it might not be crucial for the proposed method to surpass RAWNeRF, as NeRF and Gaussian splatting are fundamentally different approaches. Reviewer m8Xp expressed concerns regarding the grass section in the rebuttal PDF and noted color-shift artifacts. While these artifacts are present and could be improved, the proposed method remains valuable, especially considering that Gaussian splatting has recently gained prominence. The method enhances Gaussian splatting when using RAW images for dark scenes, which could benefit this emerging area.

Please incorporate the experiments and discussions from the rebuttal into the revised paper.
Decision: Accept (poster)