Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction
Authors: Anagh Malik
Parsa Mirdehghan
Sotiris Nousias
Kyros Kutulakos
David B. Lindell
Venue: NeurIPS 2023 spotlight (2023)
Keywords: neural radiance fields
3D reconstruction
single-photon lidar
computational imaging
TL;DR: We develop a NeRF-based approach for view synthesis of transient, picosecond-scale measurements captured by a lidar.

Abstract: Neural radiance fields (NeRFs) have become a ubiquitous tool for modeling scene appearance and geometry from multiview imagery. Recent work has also begun to explore how to use additional supervision from lidar or depth sensor measurements in the NeRF framework. However, previous lidar-supervised NeRFs focus on rendering conventional camera imagery and use lidar-derived point cloud data as auxiliary supervision; thus, they fail to incorporate the underlying image formation model of the lidar. Here, we propose a novel method for rendering transient NeRFs that take as input the raw, time-resolved photon count histograms measured by a single-photon lidar system, and we seek to render such histograms from novel views. Different from conventional NeRFs, the approach relies on a time-resolved version of the volume rendering equation to render the lidar measurements and capture transient light transport phenomena at picosecond timescales. We evaluate our method on a first-of-its-kind dataset of simulated and captured transient multiview scans from a prototype single-photon lidar. Overall, our work brings NeRFs to a new dimension of imaging at transient timescales, newly enabling rendering of transient imagery from novel views. Additionally, we show that our approach recovers improved geometry and conventional appearance compared to point cloud-based supervision when training on few input viewpoints. Transient NeRFs may be especially useful for applications which seek to simulate raw lidar measurements for downstream tasks in autonomous driving, robotics, and remote sensing.

Transient Neural Radiance Fields
for Lidar View Synthesis and 3D Reconstruction
Anagh Malik anagh www.sscoottteeee..ooooooo
Parsa Mirdehghan' 1.2 parsa toronooo...ooeii
Sotiris Nousias' sotiris @ cs. torotoooto
Kiriakos N. Kutulakos
David B. Lindell
kyros
csc kyros@cs.torua
torontocou
lindell @ cs. ws..oronto.cci.
University of Toronto
Vector Institute
Abstract
Neural radiance fields (NeRFs) have become ubiquitous tool for modeling scene appearance and geometry from multiview imagery. Recent work has also begun to explore how use additional supervision from lidar or depth sensor measurements in the NeRF framework. However, previous lidar-supervised NeRFs focus on rendering conventional camera imagery and use lidar-derived point cloud data as auxiliary supervision; thus, they fail to incorporate the underlying image formation model of the lidar. Here, we propose novel method for rendering transient NeRFs that take as input the raw, time-resolved photon count histograms measured by a single-photon lidar system, and we seek to render such histograms from novel views. Different from conventional NeRFs, the approach relies on a time-resolved version the volume rendering equation to render the lidar measurements and capture transient light transport phenomena picosecond timescales. We evaluate method on a first-of-its- kind dataset of simulated and captured transient multiview scans from prototype single-photon lidar. Overall, our work brings NeRFs a new dimension of imaging at transient timescales, newly enabling rendering of transient imagery from novel views. Additionally, show that our approach recovers improved geometry and conventional appearance compared to point cloud- based supervision when training on few input viewpoints. Transient NeRFs may be especially useful for applications which seek to simulate raw lidar measurements for downstream tasks autonomous driving, robotics, and remote sensing.
Introduction
The ability sense and reconstruct 3D appearance and geometry is critical to applications in vision, graphics, and beyond. Lidar sensors M are of particular interest for this task due to their high sensitivity to arriving photons and their extremely high temporal resolution; such, they are being deployed in systems for 3D imaging smart phone cameras [2), autonomous driving, and remote sensing (3). Recent work has also begun explore how additional supervision from lidar [4] or depth sensor measurements [ can be incorporated into the NeRF framework to improve novel view synthesis and 3I) reconstruction. Existing beseed methods that lidar [4] limited to rendering conventional RGB images, and use lidar point clouds (i.e., pre-processed lidar measurements) auxiliary supervision rather than rendering the raw data that lidar systems actually collect. Specifically, lidars capture transient timeres-tim--reslleell picosecond- or nanosecond-sean measurements of pulse of light travelling scene point and back. We consider the problem of how to synthesize such transients from novel viewpoints. In particular, we seek a method that
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

single-photon lidar
multiview lidar seans
lidar novel views
scanning mirrors
pulsed laser
SPAD
CSPC
inlensity
depth
photon count
photon count histogram
returning laser pulse
histograms
Transient NeRF
Figure Overview of transient neural radiance fields (Transient NeRFs). Measurements from single-photon lidar are captured using single-photon -hooton avalanche diode (SPAD), pulsed laser, scanning mirrors, and a time-correlated single photon counter (TCSPC). The lidar scans, consisting of a 2D array of photon count histograms (visualized with maximun-intensity projection), are captured from multiple viewpoints and used to optimize the transient NeRF. After training, we render novel views of time-resolved lidar measurements (r-y and x-1 slices are indicated by the dotted red lines), and we also convert the rendered data into intensity and depth maps.
takes as input and renders transients in the form of time-resolved photon count histograms captured by a single-photon lidar system [ [8]. Lidar view synthesis may be useful for applications that seek to simulate raw lidar measurements for downstream tasks, including autonomous driving, roboties, remote sensing, and virtual reality.
The acquisition and reconstruction of transient measurements has been studied across various different sensing modalities, including holography 9 photonic mixer devices [TO] streak cameras
and single-photon detectors (SPADs) 13
In the context of SPADs and single-photon lidar, transient is measured by repeatedly illuminating a point with pulses of light and accumulating the individual photon arrival times into a time-resolved histogram. After capturing such histograms for each point a scene, one can exploit their rich spatio- temporal structure for scene reconstruction [15] [6]. to uncover statistical properties of captured photons [77 18), and to reveal the temporal profile of the laser pulse used to probe the scene (knowledge of which significantly improve depth resolution [19 20)). These properties motivate transients as a representation and their synthesis from novel views. While existing methods have explored multiview lidar reconstruction tackle lidar view synthesis.
251
they exclusively use point cloud data, and do not
Recently, number of NeRF-based methods for 3D scene modeling have also been proposed to incorporate point cloud data (e.g., from lidar or structure from motion) [26 27 or information from time-of-flight sensors [5]. Again, these methods focus on synthesizing conventional RGB images or depth maps, while our approach synthesizes transient images. Another class of methods combines NeRFs with Lingle-photon lidar data for non-line-of-siee sight imaging [28]; however, they focus very different inverse problem and scene parameterization [29], and do not aim to perform novel view synthesis of lidar data
do.
Our approach, illustrated in Fig. extends neural radiance fields to be compatible with a statistical model of time-resolved measurements captured by single-photon lidar system. The method takes as input multiview scans from a single-photon lidar and, after training, enables rendering lidar measurements from novel views. Moreover, accurate depth maps or intensity images can also be rendered from the learned representation.
In summary, we make the following contributions.
We develop novel time-resolved volumetric image formation model for single photon lidar and introduce transient neural radiance fields for lidar view synthesis and 3D reconstruction.
'Single-photon lidars are closely related to conventional lidar systems based on avalanche photodiodes (6] but they have improved sensitivity and timing resolution (discussed in Section 2 other types of coherent lidar systems ( are outside the scope considered here.

---

We assemble first-of its- dataset of simulated and captured transient multiview scans, con- structed using a prototype multiview single-photon lidar system.
We use the dataset to demonstrate new capabilities in transient view synthesis and state-of-the-art results on 3D reconstruction and appearance modeling from few (2-5) single-photon lidar scans of a scene.
2 Related work
Our work ties together threads from multiple areas of previous research, including methods for imaging with single-photon sensors, and NeRF-based pipelines that leverage 3D information to improve reconstruction quality. Our implementation also builds on recent frameworks that improve the computational efficiency of NeRF training [30] 3T.
Active single-photon imaging. Single-photon sensors output precise timestamps corresponding to the arrival times of individual detected photons. The most common type of single-photon sensor is the single-photon avalanche diode (SPAD). SPADs are based the widely-availabe
CMOS technology ( (which we consider in this work), but other technologies such as superconducting nanowire single-photon detectors 33 and silicon photomultipliers [34), offer different tradeoffs in terms of sensitivity, temporal resolution, and cost.
In active imaging scenarios, pulsed light sources are paired with single-photon sensors to estimate the depth or reflectance of a scene by applying computational algorithms to the captured photon The extreme temporal resolution of these sensors also enables direct capture of interactions of light with scene at picosecond timescales X7 38, and by modeling and inverting the vime-resolved scattering of light, single-photor sensors can be used to see around corners (28) [39-- 41l or through scattering media [42H44]. The extreme sensitivity single-photon sensors has made them attractive technology for autonomous navigation [8] and accurate depth acquisition from mobile phones
timestamps [19 35 36.
22
Our approach differs significantly from all the previous work in that we investigate, for the first time, the problem of lidar view synthesis and multi-view 3D reconstruction in the single-photon lidar regime. We introduce the framework of transient NeRFs for this task and jointly optimize representation of scene geometry and appearance that consistent with captured photon timestamps across all input views.
Finally, we note that while commercial lidars based on SPADs avalanche photodiodes capture photon count histograms or time-resolved intensity, they typically pre-process these raw measurements point cloud format before output. Hence, the raw lidar data that use may have been widely available to previous methods; we have publicly released our dataset of multiview photon count histograms on the project webpage help alleviate this challenge.
3D-informed neural radiance fields. A number of recent techniques for multiview reconstruction using NeRFs leverage additional geometric information (sparse point clouds from lidar or structure from motion [26] (27) to improve the reconstruction quality or reduce the number of required input viewpoints. Similar benefits can obtained by combining volume rendering with monocular depth estimators [45], or using data from lime-of-fligh sensors [5]. Other methods investigate the problem of view synthesis from few input images but leverage appearance priors instead of explicit depth supervision (8). In contrast to the proposed approach, all of these methods focus reconstructing images or depth maps rather than transicnts.
3 Transient Neural Radiance Fields
We describe a mathematical model for transient measurements captured using single-photon lidar and propose time-resolved volume rendering formulation compatible with neural radiance fields.
3.1 Image Formation Model
Consider that a laser pulse illuminates a point scene that imaged onto sensor at position E R² (sce Fig. 2). Assume light from the laser pulse propagates to surface and back to along
3

---

ray casting
time-resolved volume rendering
optimization
r())
(r)
T T 11 rd
tslis,n)
Tli J.m nl
*
laser pulse
71 rendered
x())
captured
7t
Figure 2: Rendering transient neural radiance fields. We cast rays through volume and retrieve the density and color at each point using a neural representation [3) time-resolved measurement is constructed using volume rendering (Equation 3 and we bin the radiance contributions into an array based on distance along the ray. The result is convolved with the impulse response of the lidar (which incorporates the shape of the laser pulse), and we supervise the neural representation based on the difference between the rendered and captured transient measurements.
the same path described by ray r()), where / indicates propagation time. The forward path along the ray is given as r(l) = x(p) + (cw(p), where xpp) E R$ is the ray origin, w(p) E S is the ray direction which maps to p. and is the speed of light. Now, let ft) denote the temporal impulse response of the lidar (including the temporal profile of the laser pulse and the sensor jitter), and let (p)) incorporate reflectance and radiometric falloff factors [77 of the illuminated point at distance z(p) from xp). Then, assuming single-bounc light transport, the photon arrival rate incident on the sensor from the laser pulse is given as
A ji = Jr,
Pr,s J.. alp) (1-))(111)))
(1-2((D)) dt dp,
where Tn and Pi,, indicate the temporal and spatial discretization intervals for the time bin n and pixel j. respectively. The term 22/c gives the time delay for light to propagate point at distance z and back.
Now, describe the measured transient, or the number of photon detections captured by SPAD [7], as
7[i,j. m] PoIsSON (Ny X [i. n| + B),
N(nA +))
where N indicates the number of laser pulses per pixel, 17 E (0. 1) is the detection efficiency of the sensor, and B is the total number of background (non-laser pulse) detections. Background detections in turn depend on A, the average ambient photon rate at pixel [i. . and D, number of false detections produced by the sensor per laser pulse period, also known as the dark count rate. When the number of detected photons is far fewer than the number of laser pulses, SPAD measurements can be modeled according to Poisson process [17] where the arrival rate function varies across space and time. This model is appropriate for our measurements, which have relatively low flux (< 5% detection probability per emitted laser pulse) [9). The resulting measurements F[i,j, n) represent noisy histogram photon counts collected at pixel [i, j] at time bin n.
3.2 Tise-eesolved Volume Rendering
Using the measurements T, we wish to optimize representation of the appearance and geometry of the scene. To this end, we propose a time-resolved version of the volume rendering equation used in NeRF 49 50). Specifically, we model clean (i.e., without Poisson noise) time-resolved histograms T[i j. as (writing r(f) as r for brevity)
-(i.in]] / I
((0) dp,
ST.
where T(1) exx (-((o(r)ds))
We denote by e the radiance of light scattered at a point r(1) the direction w, and a represents the volume density the differential probability of ray termination at r(t). Finally, T(t) is the

---

transmittance from distance to along the ray to t. and this term is squared to reflect the two- way propagation of light [5]. We additionally explicitly account for the inverse-square falloff of intensity, through the term (tc)=2. The definite integrals are evaluated over the extent of time bin
T, = 1.n-1.1m) in), and over p within the area of pixel [i]]] as in Equation T Note that in practice, we
calculate Equation | using the discretization scheme of Max [51] used by Mildenhall et al. [4).
Finally, account for the temporal spread of the laser pulse and sensor jitter, we convolve the estimated transient with the calibrated impulse response of the lidar system f to obtain
T fTT.
Without this step, the volumetric model of Equation does not match the raw data from the lidar system and tends to produce thick clouds of density around surfaces to compensate for this mismatch.
3.3 Reconstruction
To reconstruct transient NeRFs, use lidar measurements 7 [i, ni] of a scene captured from 0 < S - different viewpoints. We parameterize transient NeRF using neural network F consisting of hash grid of features and multi-layer perceptron decoder [30]. The network takes as input a coordinate and viewing direction, and outputs radiance and density, F(r(t), w) = C. a. We use these outputs to render transients (see Fig. 2 The model optimized to minimize the difference between the rendered transient and measured photon count histograms. We also introduce modified loss function to account for the high dynamic range of lidar measurements, and we propose a space carving regularization penalty to help mitigate convergence to local minima in the optimization.
HDR-Informed loss function. Measurements from a single photon sensor can have dynamic
range that spans multiple orders of magnitude, with each pixel recording from zero to thousands of photons. We find that applying two exponential functions to the radiance preactivations (1) enforces non-negativity and (2) improves the dynamic range of the network output. Thus, have exp(exp(ē)) - 1, where the network preactivations are given by è. Following Muller et al. (30) the network also predicts density in log space.
After time-resolved volume rendering using Equation! 3. we apply a loss function in log space to prevent the brightest regions from dominating the loss [52) The loss function is given
L
Σ k.g..
[1.j. 1) In(T (k) [i.j.n] + 1)1[,,
(5)
where the sum is over all images, pixels, and time bins.
Space carving regularization. We find that using the above loss function alone results spurious patches of density in front of dark surfaces in scene. Here, the network can predict bright values on the surface itself, but darkens the corresponding values of T by placing additional spurious density values along the ray. Since the network can predict the radiance of the density to be zero at these points, the predicted transients т can be entirely consistent with the measured transients F, but with incorrect geometry. To address this, we introduce space carving regularization
Ly =
Σ k.j.J (1jj)(B
/ P.s Tm
/
((t)a(r) dp.
(6)
This function penalizes any density along a ray at locations where the corresponding measured transient values less than the expected background level B. This effectively forces space to be empty (i.e., zero density) at regions where the measurements do not indicate the presence of a surface.
The complete loss function used for training is then given
L = Ly + Ase Laes
where Ase controls the strength of the space carving regularization.
5

---

3.4 Implementation Details
Our implementation is based on the NerfAce [1 version of Instant-NGP [30,, which we extend to incorporate our time-resolved volume rendering equation. In particular, we extend the framework to output time-resolved transient measurements, to account for the pixel footprint, and to estimate depth.
Pixel footprint. Captured photon count histograms exhibit multiple peaks where the finite beam width of the laser passes over depth discontinuities. To account for this phenomenon we use a truncated Gaussian distribution to model the spatial footprint of the laser spot and SPAD sensor projected onto the scene. Specifically, we sample rays in the range of 4 standard deviations of the pixel center, weighting their contribution the rendered transient by the corresponding Gaussian probability density function value (after rendering a histogram). We set the standard deviation of the Gaussian to 0. 15 pixels for the simulated dataset and 0.10 pixels for the captured dataset.
To estimate depth we find the distance along each ray that results in the maximum probability ray termination: argmax, (()) Note that when integrating over the pixel footprint occlusion boundaries, multiple local extrema can occur, and so taking the highest peak results in single depth estimate without floating pixel artifacts.
Depth.
Network optimization. We optimize the network using the Adam optimizer [53], learning rate of 1 x 10-3 - and a multi-step learning rate decay of r = 0.33 applied at 100K, 150K, and 180K iterations. We set the batch size to 512 pixels and optimize the simulated results until they appear to converge, or for 250K iterations for the simulated results and 150K iterations for the captured results. For the weighting of the space carving loss, use Age 10-3 3 for the simulated dataset and increase this to Ase 10-2 for captured data, which benefits from additional regularization. We train the network such that с E R². In the captured data c E IR because we illuminate the scene with monochromatic laser light.
single NVIDIA A40 GPU. Also note that simulated results use RGB histograms,
Multiview Lidar Dataset
We introduce first-of-its- dataset consisting of simulated and captured multiview data from single-photon lidar. A description of the full set of simulated and captured scenes is included in the supplemental, and the dataset and simulation code are publicly available on the project webpage.
Simulated dataset We create the simulated dataset using a remolvesolved version of Mitsuba [44] which
pulsed laser
we modify for efficient rendering of lidar measurements.
The dataset consists of one scene from Vicini et al. (55)
and four scenes made available by artists on Blendswap (https://bll httk:p:://beedswappp com/), which we ported from Blender to Mitsuba 2 renderer. The training views are set consistent with the capture setup of our hardware prototype (described below) such that the camera view- point is rotated around the scene at a fixed distance and elevation angle, resulting in 8 synthetic lidar scans used for training. We evaluate on rendered measurements from six viewpoints sampled from the NeRF Blender test set [49]. The renders are used to simulate SPAD measurements by applying the noise model described in Equation [ and setting the mean number of photon counts to 2850 per occupied pixel and the background counts to 0.001 per bin, which we set to approximate our experimentally captured data.
SPAD
Gaiq
beamsplitter
scanning mirrors
laser path
E
Figure 3: Hardware prototype. A pulsed laser shares a path with single-pixel SPAD, and the illumination and imaging path are controlled by scanning mirrors.
Hardware prototype. To create the captured dataset, we built hardware prototype (Fig. 3 consisting of a pulsed laser operating at 532 nm that emits 35 ps pulses of light at repetition rate of 10 MHz. The output power of the laser is lowered to 1 mW to keep the flux low enough

---

Ground Truth
Instant-NGP
DS-NeRF
Urban NeRF
Urban NeRF-M
Proposed
rendered transients
chair
4.5 4.1
2
3.4 (18)
3,8
hotdig
views
3.4 3.8
5 views
legoo
3.2 3.6
3.5 3.9
Figure 4: Results simulated data. We show images from depth supth-supervieed NeRF baselines well as color images and rendered transients from our method after training on 2, 3, and 5 viewpoints. The proposed method produces cleaner results and generates 3D transients for each viewpoint.
Table 1: Simulated results comparing images and depth for the baselines and proposed approach.
PSNR (dB)
LPIPS views
(depth)
Method
2 views
views
views
views
views
views
3 views
5 views
Instant NGP (30
16.62
17.67
19.66
0.520
0.476
0.387
0.238
0.195
0.178
-SS-NeFF 26
21.07
0.436
0.376
0,109
0.115
0.119
Urhan NeRF 4
18.86
18.73
19.80
0.500
0.484
0.406
0.131
0.124
0.101
Urban NeRF
4
0.029
Proposed
21,38
23.48
28.39
0.172
0.151
0.115
0.015
0.011
0,013
(roughly 150, 000 counts per second on average) to prevent pileup, which a non-linear effect that distorts the SPAD measurements [56]. The laser shares an optical path with single-pixel SPAD through a beamsplitter, and a set of 2D scanning mirrors is used to raster scan the scene at resolution of 512x512 scanpoints. A time-correlated single-phote counter is used to record the photon timestamps with a total system resolution of approximately 70 ps.
Captured dataset. We capture multiview lidar scans of six scenes by placing objects on a rotation stage in front of the scanning single-photon lidar and capturing 20 different views in increments of 18 degrees of rotation. For each lidar sean we accumulate photons during a 20 minute exposure time to minimize noise in the transient measurements. We bin the photon counts into histograms with 1500 bins and bin widths set aside 10 views sampled in 36 degree increments for testing and we use 8 of the remaining views for training. Prior to input into the network for training, we normalize the measurement values by the maximum photon count observed across all views.
ps (all raw timestamp data will also made available with the dataset). We
Calibration. We calibrate the camera intrinsics of the system using raxel model [57] with corners detected from two scans of checkerboard translated direction parallel the surface normal. This model calibrates the direction of each ray individually, which is necessary because the 2D scanning mirrors deviate from the standard perspective projection model [58). Extrinsics are calibrated by placing a checkerboard on rotation stage and solving for the axis and center of rotation that best align the 3D positions of the checkerboard corners, where the 3D points are found using the calibrated ray model along with the time of flight from the lidar (see supplemental). Overall, accurate calibration an important and non-trivial task because multiview lidar scans provide two distinct geometric constraints (i.c. stereo disparity and time of flight) that must be consistent for scene reconstruction.
7

---

Results
We evaluate our method on the simulated and captured datasets and use transient neural radiance fields to render intensity, depth, and time-resolved lidar measurements from novel views.
Baselines. The intensity and depth rendered from method are compared to four other baseline methods that combine point cloud-based losses with neural radiance fields. For fairer comparison and to speed up training and inference times, implement the baselines by incorporating their loss terms into the recently introduced frameworks of NerfAce [31] and Instant-NGP (30) adopted by our method. We train the following baselines using intensity images (i.e., the photon count histograms integrated over time) along with point clouds obtained from the photon count histograms using a log- log-matched filter, which the constrained maximum likelihood depth estimate (59].
Instant-NGP 30 used to illustrate performance without additional depth supervision. Depth-Supervised NeRF (DS-NeRF) (26) incorporates an additional loss term to ensure that the expected ray termination distance volume rendering aligns with the point cloud points.
4 incorporates the ray- teyyiimmmnniion loss of DS-NeRF while also adding space carving losses to penalize density along rays before and after the intersection with a point cloud point.
Urban NeRF
Urban NeRF with masking (Urban NeRF-M) modifies Urban NeRF incorporate oracle object mask and extends the space carving loss to unmasked regions, providing stronger geometry regularization (additional details in supplement).
Prior to input into the network, we normalize the images and apply a gamma correction, which improves network fitting the high dynamic range data. Finally, after training with each method, estimate an associated depth map using the expected ray termination depth at each pixel, which is the same metric used in the loss functions of the aforementioned baselines.
5.1 Simulated Results
The method compared to the baselines in simulation across five scenes: chair, ficir, lego, hot dog- and statue. In Fig.[ we show RGB images rendered from novel views using the baselines and our proposed method after training on two, three, and five views. More extensive sets of results on all scenes are included in the supplemental. We find that views rendered from transient neural radiance fields have fewer artifacts and spurious patches of density, as the explicit supervision from the photon count histograms avoids the ill-posedness of the conventional multiview reconstruction problem.
Additional quantitative results are included in Ta- ble [ averaged across all simulated scenes. For the evaluation of rendered RGB images, we nor- malize and gamma-corree the output of the pro- posed method and the ground truth in the same fashion as the baseline methods. Transient NeRF recovers novel views with significantly higher peak signal-to-noise ratio and better performance the learned perceptual image patch similarity (LPIPS) metric [60] compared to baselines. Transient mea- surements provide explicit supervision of the unoc- cupied spaces in the scene, leading fewer floating artifacts, and cleaner novel views.
Ground Truth
Instant-NGP
DS-NeRF
Urban NeRF
Urban NeRF-M
Proposed
The depth maps inferred from Transient NeRF are also significantly more accurate than baselines (see Fig. 5). One key advantage here is that we avoid supervision on point clouds obtained by potentially noisy (and thus view-inconsistent) per-pixel estimates of depth. By training on the raw photon count histograms, the scene's ' geometry is allowed to converge to the shape that best explains all histograms across all views, resulting in much higher geometric accuracy.
Figure 5: Comparison of depth maps recov- ered from simulated measurements trained on views of the lego scene.
8

---

Ground Truth
Instant-NGP
DS-NeRF
Urban NeRF
Urban NeRF-M
Proposed
rendered transients
cinema
7.1
7.3
6.8 (1x))
7.2
3 views
food
7.5
7.5
haskets
5 views
6.5
6.9
6.7
7.1
Figure 6: Results on multiview lidar data captured with the hardware prototype and trained with 2, 3, and 5 viewpoints. For the proposed method we show the rendered transients, intensity image, and individual transients for the indicated pixels.
Table Evaluation of rendered intensity images and depth on captured results.
(dB) f
L.IPS views
(depth)
Method
2 views
views
views
views
views
2 views
3 views
5 views
Instant NGP (30
16.44
16.52
16.39
0.358
0.307
0.274
0.115
0.076
0,053
DS NeRF 26
15.34
15.05
14.86
0.311
0.312
0.325
0.048
0.036
0,036
Urban NeRF 4
16.90
15.91
15.93
0.403
0.328
0.231
0.017
0.015
0.014
Urban NeRF wimask
15.45
18.26
0.269
0.191
0.014
0.006
0.004
Proposed
22.11
21.83
22.72
0.271
0.212
0.172
0.005
0.006
0,010
5.2 Captured Results
In Fig. we show rendered novel views of intensity images from our method and baselines trained on captured data with two, three, and five views. Results are shown on the cinema, and baskets scenes (additional results in the supplemental). The proposed approach results in fewer artifacts and the rendered intensity images are more faithful to reference intensity images captured from the novel viewpoint. Quantitative comparisons of our method to baselines on captured data are shown in Table 2 note that we do not have access to ground truth depth for captured data and instead use depth from log-matched filter on the ground truth transient. We find that the method outperforms the baselines in terms PSNR and LPIPS of intensity images rendered from novel views. While performance data with increasing numbers of viewpoints, we attribute this to small imperfections (=1 mm) in the alignment of the lidar scans after estimating the camera extrinsics.
captured data does not improve much observed on simulated
Since DS-NeRF trains explicitly on depth without additional regularization, it is espe- cially sensitive to camera perturbations and can be outperformed in some cases by In- stant NGP which has no additional geome- try constraints. Our approach appears some- what less sensitive to these issues, perhaps because geometry regularization is done im- plicitly through photometric loss on the lidar measurements.
novel viewpoint
X-d time slice
depth
7.0
7.4
L
renderec
We notice some degradation in depth ac- curacy relative to simulation, likely due to imperfections in the estimated extrinsics. Sub-mm registration of the lidar measure-
Comparison between reference and novel views of lidar measurements, intensity slices, and depth. The r-y intensity slices are visualized for times indi- cated by the red dashed lines.
Figure 7:

---

ments would likely improve results, but achieving such precise registration is non-trivial and beyond the scope our current work. Finally, in Fig. 7 we compare captured measurements to rendered tran- sients and depth rendered for the boots scene trained on 2 viewpoints. We recover the time-resolved light propagation from novel view, shown in z-y slices of the rendered transients over time. The depth map recovered from the novel view appears qualitatively similar to the ground truth (estimated from captured measurements using log-matched filter [17). We show additional 3D reconstruction results in the supplemental.
Discussion
Our work brings NeRF to a new dimension of imaging at transient timescales, offering new opportu- nities for view synthesis and 3D reconstruction from multiview lidar. While work is limited to modeling the direct reflection of laser light to perform lidar view synthesis, our dataset captures much richer light transport effects, including multiple bounces of light and surface reflectance properties that could open avenues for future work. In particular, the method and dataset may help enable techniques for intra-scene non-line-of-siee a scene), and recovery of the bidirectional reflectance distribution function via probing with lidar measurements (65).
imaging [61 (i.e., recovering geometry around occluders within
Our method also limited in that we do not explore more view synthesis more general single- photon imaging setups, such when the lidar and SPAD not coaxial; we hope to explore these configurations in future work. The proposed framework and the ability to render transient measure- ments from novel views may be especially relevant for realistic simulation for autonomous vehicle navigation, multiview remote sensing, and view synthesis of more general transient phenomena.
Acknowledgments and Disclosure of Funding
Kiriakos N. Kutulakos acknowledges the support of the Natural Sciences and Engineering Council of Canada (NSERC) under the RGPIN and RTI programs. David B. Lindell acknowledges the support of the NSERC RGPIN program. The authors also acknowledge Gordon Wetzstein and the Stanford Computational Imaging Lab for loaning the single-photon lidar equipment.
References
[1] Peter Seitz and Albert JP Theuwissen. Single-Photon Imaging. Springer Science & Business Media, 2011.
[2] Ilya Chugunov, Yuxuan Zhang, Zhihao Xia, Xuaner Zhang, Jiawen Chen, and Felix Heide. The implicit values of good hand shake: Handheld multi-frame neural depth refinement. In Proc. CVPR, 2022.
[3] Brent Schwarz. Mapping the world in 3d. Nat. Photonics, 4(7):429 430, 2010.
[4] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan. Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proc. CVPR, 2022.
[5] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O'Toole. Törf: Time-of-fligh radiance fields for dynamic scene view synthesis. Proc. NeurlPS, 34, 2021.
[6] Joe Campbell, Stephane Demiguel, Feng Ma, Ariane Beck, Xiangyi Guo, Shuling Wang, Xiaoguang Zheng, Xiaowei Li, Jelfrey D Beck, Michael A Kinch, et al. Recent advances in avalanche photodiodes. IEEE J. Sel. Top. Quantum Electron., 10(4):777-777, 2004.
[7] Christopher V Poulton, Ami Yaacobi, David B Cole, Matthew J Byrd, Manan Raval, Diedrik Vermeulen, and Michael R Watts. Coherent solid-state lidar with silicon photonic optical phased arryy. Opt. Lett., 42 (20):4091 -0044, 2017.
[8] Joshua Rapp. Julian Tachella, Yoann Altmann, Stephen McLaughlin, and Vivek K Goyal. Advances single-photon lidar for autonomous vehicles: Working principles, challenges, and recent advances. IEEE Signal Process. Mag., 37(4) 71, 2020.
[9] Nils Abramson. Light-in in flight recording by holography. Opt. Lett. 3(4): 123, 1978.
[10] Felix Heide, Matthias B Hullin, James Gregson, and Wolfgang Heidrich. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph., 32(4): 1-10, 2013.
10

---

[11] Achuta Kadambi, Refael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles, ACM Trans. Graph. 32(6);1-10, 2013.
[12] Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chiumaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. Femto-phokography: capturing and visualizing the propagation of light. ACM Trans. Graph., 32(4))1-8, 2013.
[13] Matthew O Tooll, Felix Heide. David B Lindell, Kai Zang. Steven Diamond, and Gordon Wetzstein. Reconstructing transient images from single-photon sensors. In Proc. CVPR, 2017.
[14] Ahmed Kirmani. Tyler Hutchison, James Davis, and Ramesh Raskar. Looking around the corner using transient imaging. In Proc. ICCV, 2009.
[15] David B Lindell, Matthew O OTToole, and Gordon Wetzstein. Singlc-photon 3d imaging with deep sensor fusion. ACM Truns. Graph.,
[16] Jiayong Peng, Zhiwei Xiong, Xin Huang, Sheng-Ping Li, Dong Liu, and Feihu Xu. Photon-efficient 3d imaging with non-local neural network. In Proc. ECCV. Springer, 2020.
[17] Joshua Rapp and Vivek K Goyal. few photons among many: Unmixing signal and noise for photon- efficient active imaging. IEFE Trans. Comput. Imaging, 3(3):445-459, 2017.
[18] Joshua Rapp. Yanting Ma, Robin MA Dawson, and Vivek K Goyal. High-flux single-photon lidar. Optica, 8(1):30-39, 2021.
[19] Felix Heide, Steven Diamond, David B Lindell, and Gordon Wetzstein. Sub-picosecond photon-efficient 3d imaging using single-photon sensors. Sci. Rep., 8(1):17726, 2018.
[20] Joshua Rapp, Robin MA Dawson, and Vivek K Goyal. Dithered depth imaging. Opt. Express, 28(23): 35143-35157, 2020.
[21] Wolfgang Hess, Damon Kohler, Holger Rapp, and Daniel Andor. Real-time loop closure in 2d lidar slam. In Proc. ICRA, 2016.
[22] Qin Zou, Qin Sun, Long Chen, Bu Nic, and Qingquan Li. A comparative analysis of lidar slam-based indoor navigation for autonomous vehicles. IEEE Trans. Intell. Transp. Syst., 23(7):6907-6221, 2021.
[23] Zimo Li, Prakruti C Gogia, and Michael Kaess. Dense surface reconstruction from monocular vision and lidar. In Proc. ICRA, 2019.
[24] Matthew J Leotta, Chengjiang Long, Bastien Jacquet, Matthieu Zins, Dan Lipsa, Jie Shan, Bo Xu, Zhixin Li, Xu Zhang. Shih-Fu Chang, et al. Urban semantic 3d reconstruction from multiview satellite imagery. In Proc. CVPR Workshops, 2019.
[25] Stefan Lionar, Lukas Schmid, Cesar Cadena, Roland Siegwart, and Andrei Cramariue. Neuralblox: Real-time neural representation fusion for robust volumetric mapping. In Proc. 3DV, 2021.
[26] Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth supervised NeRF: Fewer views and faster training for free. In Proc. CVPR, 2022.
[27] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner. Dense depth priors for neural radiance fields from sparse input views. In Proc. CVPR, 2022.
[28] Daniele Faccio, Andreas Velten, and Gordon Wetzstein. Non-line-of sight imaging. Nat. Rev. Phys., 2(6): 918-327, 2020.
[29] Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu. Non-line-offsighh imaging via neural transient fields. IEFE Trans. Pattern Anal, Mach. Intell., 43(7): 2257-2268, 2021.
[30] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Trans. Graph. (SIGGRAPH), 41(4):1-15, 11, 2022.
[31] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerface: A general nerf acceleration toolbox. arXiv preprint Xiv: 10.00477, 2022.
[32] Franco Zappa, Simone Tisa, Alberto Tosi, and Sergio Cova. Principles and features of single-photon avalanche diode arrays. Sensors and Actuators A: Physical, 140(1):103-112, 2007.
[33] Chandra M Natarajan, Michael G Tanner, and Robert H Hadfield. Superconducting nanowire single-photon detectors: physies and applications. Superconductor Sei. Technol., 25(6):063011, 2012.
[34] P Buzhan, B Dolgoshein, L Filatov, A Ilyin, V Kantzerov, V Kaplin, A Karakash, F Kayumov, S Klemin, E Popova, et al. Silicon photomultiplier and its possible applications. Nucl. Insirum. Methods. Phys. Res.
A, 504(1-3):48-52, 2003.
[35] Dongeek Shin, Feihu Xu, Dheera Venkatraman, Rudi Lussana, Federica Villa, Franco Zappa. Vivek K Goyal, Franco NC Wong, and Jeffrey H Shapiro. Photon -fficiefficient imaging with single- camera. Nat. Commun., 7(1::1244, 2016.

---

[36]
Julián Tachella, Yoann Altmann. Nicolas Mellado, Aongus McCarthy, Rachael Tobin, Gerald S Buller. Yves Toumeret, and Stephen McLaughlin. Real-time 3d reconstruction from single-photon lidar data using plug-and-play point cloud denoisers. Nat. Commun.. 10(1):4984, 2019.
[37] Genevieve Gariepy. Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. Single-photon sensitive light-in-fight imaging. Nat. Commuun. 6( ()6211, 2015.
[38] David B Lindell, Matthew O'Toole, and Gordon Wetzstein. Towards transient imaging at interactive rates with single-photon detectors. In Proc. ICCP, 2018.
[39] Joshua Rapp, Charles Saunders, Julián Tachella, John Murray--ruce, Yoann Altmann, Seen-Yves Tourneret, Stephen McLaughlin, Robin MA Dawson, Franco NC Wong, and Vivek K Goyal. Seeing around corners with edge-resolved transient imaging. Nat. Commun., ):(9295,,,, 2020.
[40]
Shumian Xin. Sotiris Nousias, Kiriakos N Kutulakos, Aswin C Sankaranarayanan. Srinivasa Narasimhan, and loannis Gkioulekas. A theory of fermat paths for non- non-line-of lo--lin-off-iihtt shape reconstruction. In Proc. CVPR, 2019.
[41] Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. Learned feature embeddings for non-line line-of-sight imaging and recognition. ACM Trans. Graph. (SIGGRAPH), 39 (6):1-18, 2020.
[42] David B Lindell and Gordon Wetzstein. Three-dimensional imaging through scattering media based on confocal diffuse tomography. Nat. Commun., 1 11(1 44517, 2020.
[43] Yongyi Zhao, Ankit Raghuram, Hyun K Kim, Andreas II Hielscher. Jacob T Robinson, and Ashok Veeraraghavan. High resolution, deep imaging using confocal time-of-fligt diffuse optical tomography. IEEE Trans. Pattern Anal. Mach. Intell., 43(7):2206 2219,, 2021.
[44] Rachael Tobin, Abderrahim Halimi, Aongus McCarthy, Martin Laurenzis, Frank Christnacher, and Gerald S Buller. dinee-dimensional mngle-photon imaging through obscurants. Opt. Express, 27(4):4590 4611, 2019.
[45] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. In Proc. NeurlPS, 2022.
[46] Michael Niemeyer, Jonathan Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan, Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. CVPR, 2022.
[47] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proc. CVPR, 2021.
[48] Samarth Sinha, Jason Y Zhang, Andrea Tagliasacchi, Igor Gilitschenski, and David B Lindell. Sparsepose: Sparse-view view camera pose regression and refinement. In Proc. CVPR, 2022.
[49] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1) 1006 2021.
[50] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH, 18(3)1655-77, 1984.
[51] Nelson Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph., 1(2)999-08,, 1995.
[52] Ben Mildenhall, Peter Hedman, Ricardo BruuilaBrualla, Pratul P. Srinivasan, and Jonathan T. Barron. NeRF in the dark: High dynamic range view synthesis from noisy raw images. CVPR, 2022.
[53] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.
[54] Diego Royo. Jorge Garefa, Adolfo Muñoz, and Adrian Jarabo. Non-line-of-sige transient rendering. Computers & Graphics, 107:84992, 2022.
[55] Delio Vicini, Sébastien Speicrer, and Wenzel Jakob. Differentiable signed distance function rendering. ACM Trans, Graph., -18 2022.
[56] Joshua Rapp. Yanting Ma, Robin MA Dawson, and Vivek K Goyal. Dead time compensation for high-flux ranging. IEEE Trans. Signal Process., 67(13)3347134495 2019.
[57] Michael Grossberg and Shree K Nayar. The raxel imaging model and ray-based calibration. Int. J. Comput. Vis., 612):1199 2005.
[58] Peter Eisert, Konrad Polthier, and Joachim Hornegger. A mathematical model and calibration procedure for galvanometric laser scanning systems. Vision, Modeling, and Visualization, pages 207-214, 1011.
[59] Ahmed Kirmani, Dheera Venkatraman, Dongeek Shin, Andrea Colaço, Franco NC Wong. Jeffrey H Shapiro, and Vivek K Goyal. First-photon imaging. Science, 343(616)) 61, 2014.
12

---

[60] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proc. CVPR, 2018.
[61] Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. Recovering paree-dimensiona, shape around a comer using ultrafast time-of of of-fhg imaging. Nat. Commun., 3(1):745, 2012.
[62] Matthew O O'Toole, David Lindell, and Gordon Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature, 555(7696):338-44 2018.
[63] David Lindell, Gordon Wetzstein, Matthew O"Toole. Wave-based non-line-of-sight imaging using fast fk migration. ACM Trans. Craph., 38(4): 13,, 2019.
[64] Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jaraho, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual optics. Nature, 5720771):600-623, 2019.
[65] Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH Asia), 30(6):1-10, 2011.
13


Reviews:
[{"review_id": "ml6DAYcguW", "reviewer": "Reviewer_tsWB", "summary": "This work proposes a novel-view synthesis method for active sensors (single-photon LiDAR sensors) based on the Neural Radiance Field formulation. To this end, the volume rendering formulation for active sensors is derived, properly taking into account the measurement formation process (two-way distance, intensity fall-off) of LiDAR sensors. Along with that, a new loss functions for empty-space carving is proposed. The proposed method is implemented based on the INGP NeRF model and evaluated on both synthetic dataset of transient multiview scans (one of the contributions of this work), as well as on real-world data that was captured using a prototype single-photon LiDAR system. In these experiments, the proposed method outperforms SoTA NeRF methods (also with depth supervision) across all metrics.", "questions": "- What exactly does **c** denote in Equation 2 and L150?\n- I like that the formulation tries to follow the measurement formation process, but currently only the intensity fall-off and round-trip are considered. Would it make sense to add at least the effects of the incidence angle?\n- Similar to the comment above: why does simulated data simulate RGB and why is the evaluation considering low-view setting?\n- I also like that the beam divergence is modeled through sampling multiple rays (L180), but it would be good to clarify at which point in the volume rendering formulation the contributions are averaged?\n- The depth of the proposed methods is obtained as the argmax across the histogram, however for the baselines the integral along the ray is used. Does that yield better results as taking the argmax of the volume density profile? \n", "limitations": "The limitations are discussed in the conclusion section, while broader societal impacts are not (but I also don't believe that there are any significant ones).", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Original and important problem formulation. Novel-view synthesis for active sensors is an under researched, but a very important problem.\n- A volume rendering formulation for active sensors that properly takes (at least some of the) the properties of the measurement formation process into account. These include, two-way path when computing the transmittance, intensity falloff, beam divergence. \n- Simulated dataset of transient multiview scans that will be made publicly available (along with the scripts used to generate it)\n- A dataset of real-world captures that were acquired using a prototype single-photon LiDAR which, if I understand correctly, will also be made publicly available", "weaknesses": "There are two main weaknesses in my view: \n- **Clarity**: The clarity of the paper could in my opinion be improved, and the main confusion stems from including the RGB images from simulated data in figure 1 and using **c** to denote number of photons (?) in Figure 2. I might have misunderstood something, but if I am not wrong, the actual LiDAR system can only measure the photon count and depth (time-of-flight) and the volumetric rendering formulation should be tied to that. However, in the Equation 3 the radiance is a vector quantity (RGB color?)  same as in L150? The simulated data seems to be full RGB (what is the reason for this?), does that mean that for the simulated data the evaluation is actually similar to the original NeRF (PSNR and LPIPS computed over the RGB images?). \n\n - **Experimental evaluation**:  In my perception, the experimental evaluation seems somewhat biased. The proposed method is compared to the following baselines *INGP*, *DS-NeRF* and *UrbanNeRF* which are supervised through color/intensity supervision (and *DS-NeRF* and *UrbanNeRF* also using the depth). However, the depth supervision only constrains the integral over the density and the supervision signal might even be very noisy (log-matched filter in case of real world data are usually very noisy). On the other hand, the proposed method is supervised with the full histograms per each ray (before integration) which provide additional supervision and helps to constrain the empty space. This is especially important in the low-view setting that was for some reason selected in the evaluation (2-5 views on synthetic data?). I am wondering why the low-view setting was selected for simulated data? In my opinion this pronounces the bias as it emphasizes the differences in the supervision signal between the methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. Please see the above response to all reviewers where we address questions related to the experimental evaluation, effects of incidence angle, consideration of the low-view setting, and the depth evaluation. We address other questions below.\n\n**tsWB: Will the dataset be made public?** Yes, both the captured and simulated datasets will be made public upon publication of the paper. \n\n**tsWB: Clarification on dimensionality of symbol “$\\mathbf{c}$” in rendering equations.** The lidar system measures photon count histograms. We simulated RGB photon count histograms, which could be captured in practice using an RGB laser. However, in our experimental setup, we only had access to a single-wavelength picosecond laser (green; 532 nm), and so our captured results have a single color channel. Hence, in our model, the dimension for $\\mathbf{c}$ (i.e., the radiance output by the neural representation at each point along a ray) is different when applied to simulated and captured data. Specifically, in the simulations we use $\\mathbf{c}\\in\\mathbb{R}^3$, whereas in the captured data $\\mathbf{c}\\in\\mathbb{R}^1$.\n\n**tsWB: Clarification on evaluation metrics**. For evaluation on simulated data, we chose to use three color channels (RGB) and selected the evaluation metrics (LPIPS, PSNR) to be consistent with previous work (Instant NGP, DS-NeRF, Urban NeRF). To generate 2D images from the ground truth photon count histograms (or histograms rendered by the proposed method), we integrate the histograms over the time dimension and apply normalization and gamma correction (L259–262, L270–282). We also evaluated the depth estimates produced by our method as described on L182–185.\n\n**tsWB: Clarification on spatial footprint modeling.** The weighted summation is done as the last rendering step by averaging the corresponding time bins within the photon count histograms produced by each ray for a given pixel. We will clarify this in the revision.\n"}, {"review_id": "8teXqVCZnU", "reviewer": "Reviewer_Zi1e", "summary": "NeRF is a method that has recently become popular for view synthesis. It allows estimation of a 3D scene density and 5D radiance map using a few intensity images and their camera poses. These can then be used to render intensity and depth images from any novel 2D view.\n\nThis paper extends NeRF to allow estimation of scene density and radiance from “scene transients” instead of intensity/depth images. Previous methods have tried to incorporate depth information into NeRF in the form of point clouds, but the proposed method directly uses raw scene transients (histogram of photon counts) captured by a SPAD sensor. This method can then be used to render scene transients given a novel 2D view.\n\nThe authors' main contribution is to extend the volume rendering equation of NeRF to produce time resolved measurement instead of integrated over time. Additionally their training/rendering takes into account the shape of the laser pulse, spatial footprint of the laser spot etc. And lastly they add modifications to the NeRF objective to ensure their method works well for the high dynamic range SPAD data (predict radiances in the double log space, and apply loss in the log space). \nThe authors also apply a regularization (space carving) to encourage the density estimates to be sparse (similar to what other NeRF variants have done e.g urban NeRF).\n\nThe authors also capture a real world multiview LiDAR dataset with their hardware prototype. Using this dataset and another simulation based dataset, they provide quantitative and qualitative comparison of their method with other baselines for intensity and depth reconstruction from novel views. The results from simulation data show their method’s superior performance. The results from captured data also show that their method is superior, but by a smaller margin (which they attribute to alignment imperfections in the captured dataset).", "questions": "1. You mention that you model the spatial footprint of the laser/sensor spot when rendering the transients. Is that necessary? Does performance deteriorate if you just assume an ideal spot?\n2. I'm not convinced about the utility of rendering raw lidar measurements from novel viewpoints. Other than generating simulation data, are there any practical applications?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This is a nice extension to the NeRF setup that allows directly leveraging the raw lidar measurements to learn a 5D representation instead of learning from intensity/depth images which won't contain all the information available in the raw measurements. The improvement in estimation accuracy is evident in the results/experiments shown.\n2. The paper is written very well, with good attention to details (e.g. modeling assumptions, experiment setup). Background work is explained well.\n3. Authors perform exhaustive experiments with baseline methods using both simulated and captured data, and results look impressive.\n", "weaknesses": "From a novelty of algorithm standpoint, the contributions of the paper are limited. From what I understood, It is a straightforward application of the NeRF framework to a new domain, i.e, the authors represent the scene density/radiance just like in NeRF, the only difference is that they modify the rendering logic to produce 3D histograms instead of 2D images. The image formation model of SPADs (which is used in the rendering equation) is also not novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. Please see the above response to all reviewers where we address questions related to the spatial footprint of the laser/sensor spot. We address other questions related to novelty and applications below.\n\n**Zi1e: Novelty.** Our method is the first to apply a NeRF-type approach to multiview rendering of single-photon lidar data. Algorithmically, we extended the NeRF method to properly model a single-photon lidar system, including time-resolved rendering and modeling the temporal response of the system (i.e., laser pulse width and sensor jitter) as well as the footprint of the laser and detector. Further, we demonstrated our approach in practice with a first-of-its kind multiview single-photon lidar setup and captured dataset. We believe the novel hardware prototype and dataset will be appreciated by the community and inspire follow-on work.\n\n**Zi1e: Transient rendering applications**. One compelling application of the method could be generating novel views of lidar measurements for downstream tasks in training autonomous driving or robotics. We also believe that our work is an important first step for the task of transient neural rendering, which could enable advances in material segmentation, recovery of reflectance functions (e.g., BSSRDFs) from sparse views, non-line-of-sight imaging, and free-viewpoint rendering of light propagation and optical phenomena.\n"}, {"review_id": "hI4UOXMD8J", "reviewer": "Reviewer_vJ74", "summary": "The paper describes a variant of NeRF specially designed for the image formation model of a time-of-flight imaging sensor. In contrast to earlier works using ToF sensors in a neural radiance field setting, the method is not designed to merely take depth maps or point clouds from the ToF sensors as input. Instead, the formulation goes one level deeper and presents a NeRF and image formation model on the basis of the time resolved photon count histograms that are used as basis for the final depth measurement by the ToF camera. To this end, the authors propose a time-resolved version of the volume rendering equation for image formation, as well as adapted HDR aware loss functions for training the neural radiance fields on photon histograms. The approach is tested on simulated data as well as real data captured with a ToF prototype setup. The authors show that in particular in a sparse view setting, the new photon count based formulation has advantages over depth-supervised or image-based NeRFs. Overall, I think this is a strong paper showing very innovative ideas. I enjoyed reading it.", "questions": "see the weaknesses section\n\nOne issue with time-of-flight sensors in general is their often non-trivial noise characteristics. Would explicitly incorporating the noise model into the formulation help here ? ", "limitations": "The discussion in the paper is adequate. ", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, I think this is a strong paper. It introduces quite a few clever ideas on how to adapt the neural radiance fields concept to the image formation model of a time-of-flight camera. This ranges from adapting the image formation model to a time-resolved version, to introducing a HDR aware loss and a space-carving regularization tailored to the peculiarities of the imaging properties of the employed sensor. To my knowledge, this is the first paper approaching neural radiance fields in this way.  As the authors suggest in their discussion the formulation on the basis of time resolved photon histograms may enable additional improvements in 3D reconstruction, e.g. of complex shapes, material properties etc. ", "weaknesses": "Not much to say here. One point that could have been discussed more is from what number of viewpoints on the traditional or depth-supervised approaches catch up. At the moment, advantages of the approach are mostly shown in the sparse view setting, which is fine and showcasing an advantage of the approach. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. Please see the above response to all reviewers where we discuss training on more viewpoints. We also discuss noise modeling below.\n\n**vJ74: Forward model choices–noise modeling**. As we note in the paper, the SPADs used in our hardware prototype follow a Poisson noise model (Eq. 2). Previously, we experimented with a loss function based on the corresponding negative log-likelihood (see, e.g., the formulation in O’Toole et al. [13]). Empirically, we found the L1 loss of the log transients (Eq. 5) to work well while also being simple to implement. However, we think incorporating additional reconstruction priors motivated by Poisson statistics could be a very promising direction for future work (see, e.g., Rapp and Goyal [17])."}, {"review_id": "HpOytlsAHy", "reviewer": "Reviewer_kaKw", "summary": "The paper brings NeRFs to a new dimension of imaging at transient timescales enabling the rendering of transient imagery. It incorporates the underlying image formation model of the lidar and recovers improved geometry and conventional appearance when training on a few input viewpoints. The paper also proposes a first-of-its-kind dataset of simulated and captured transient multiview scans from a prototype single-photon lidar and tests the transient NeRF on the dataset.", "questions": "Do you do the ablation study of HDR-Informed loss function and Space carving regularization? Can the method work under natural light?", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper brings NeRFs to a new dimension of imaging at transient timescales and proposes a first-of-its-kind dataset of simulated and captured transient multiview scans from a prototype single-photon lidar. To deal with the new data type, HDR-Informed loss function and Space carving regularization are proposed to further improve the performance. The experiments are sound compared with other NeRF-based methods. The full paper has a clear structure and a clear narrative.", "weaknesses": "It only tests in a dark environment because of the single-photon Lidar. It is hardly used in broad application scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback. Please see the above response to all reviewers where we address questions related to the ablation study of the HDR-informed loss and the space carving regularization. We discuss effects of natural lighting below.\n\n**kaKw: Scanning under natural light.** Our hardware setup is relatively robust to natural light because we place a laser-line spectral filter (Thorlabs FL532-10) in front of the SPAD, which attenuates ambient illumination by a factor of 10,000. Under indoor lighting, we observe roughly 300–3,000 photon counts per second, depending on the albedo of the target, which is <2% of the detected laser photons (~150,000 counts per second). However, operation in much brighter environments (e.g., outdoors under direct sunlight) would result in non-negligible background counts. In that case, it is possible to use other laser wavelengths (e.g., 1550 nm), where sunlight is heavily attenuated because of absorption by the atmosphere. We will note this in the paper."}]

Author rebuttal:
We thank all reviewers for their helpful suggestions and insightful questions.  Our work brings NeRFs to a new dimension of imaging at transient timescales, enabling rendering of photon count histograms from novel views for the first time. Further, we show that supervision on photon count histograms enables improved reconstruction of geometry and novel view synthesis compared to point cloud-based supervision when training on few input viewpoints. Our first-of-its-kind multiview dataset of photon count histograms (including both simulated and captured data) will be made publicly available upon publication.

We address most reviewer concerns in this shared section of the rebuttal and respond to other individual concerns as separate responses to each reviewer. Please see the attached rebuttal PDF for Tables R1–R6.

**Adjustment to simulated results.** After submission, we noticed a slight inconsistency with the pre-processing of the simulated dataset which we have now corrected. Specifically, we created the RGB images used for training by summing the simulated photon count histograms across the time dimension and normalizing by the maximum value _per view\._ We reprocessed the simulated results to use normalization _across all views_ (consistent with the captured results, L230-231). We include the updated results in Table R1 of the rebuttal PDF (compare to Table 1 of the paper). The overall trends and qualitative results are still consistent with what we previously reported, and our method still outperforms all baselines in terms of novel view synthesis and depth estimation. We will use these results in the revision unless reviewers object. Tables R2 and R3 also use normalization across all views.

**tsWB, vJ74: Number of views.** The paper focused on results for sparse views (i.e., 2, 3, and 5 views) since this is the regime where depth supervision improves the most over only using 2D supervision. We tried using 10 simulated training views sampled at equal angles around the Lego scene and our approach still outperforms the baselines when evaluated on the same test views (see Table R2). If reviewers request, we can include results with even more training views in the revision.

**tsWB: Clarification on experimental evaluation.** The reviewer notes that the depth supervision used by previous methods (DS-NeRF, Urban NeRF) only constrains the integral of the density based on point cloud data (though Urban NeRF also includes point cloud-based space carving losses, see their Eq. 15). 

It turns out that supervision using the raw lidar data (i.e., photon count histograms) provides better performance, especially for reconstruction from few input views; **this is a key insight of the paper**. The evaluation uses fair, apples-to-apples comparisons based on the same input photon count histograms. That is, for DS-NeRF and Urban NeRF, we estimate point clouds by applying the constrained maximum likelihood estimate (i.e., log-matched filter) to the photon count histograms. The proposed method uses those same histograms, and we show that supervision with raw data outperforms using pre-processed point clouds in the sparse-view setting considered by DS-NeRF and Urban NeRF. **Importantly,** **this result improves over the standard practice of preprocessing raw lidar data to point clouds.**

**tsWB: Depth evaluation.** DS-NeRF and Urban NeRF calculate depth differentiably using the integral along the ray (i.e., the expected ray termination distance) and incorporate this in their loss functions. Since the baseline methods are supervised with this depth estimate, we opted to use the same approach for evaluation.

For completeness, we provide an updated version of the L1 depth in Table R3 calculated for all methods using the maximum ray termination probability (i.e., using argmax; see L183). Baseline performance using argmax is indeed improved for most of the methods, though Transient NeRF still performs best. 

**tsWB: Normals in forward model**. Estimating the normals requires a noisy finite-difference operation on the predicted density values, which we found results in speckle-like artifacts in the novel views. Instead, we model the effects of incidence angle by conditioning the neural representation on view direction. See Table R4, which ablates using the normals to add cosine factors to the rendering equation (Eq. 3) for the Cinema scene.

**Zi1e: Ablation–laser spatial footprint.** When the laser spot and sensor footprint pass over a depth discontinuity, we observe two peaks in the resulting photon count histogram (corresponding to the two depths across the discontinuity). Assuming an ideal spot (i.e., using a single ray) completely fails to model this effect. Training and rendering the Lego scene (which has many depth discontinuities) with the ideal spot model results in much worse performance across novel views (see Table R5).

**kaKw: Ablation–HDR-informed loss function.** We provide an ablation study of the HDR-informed loss function in Table R6 on the Chef and Food scenes. Incorporating this loss provides some improvement by preventing very bright regions from dominating the loss. Moreover, in the case of 5 input views, performance without the HDR-informed loss drops due to specular highlights appearing in one view, but not an overlapping nearby training view (e.g., on the globe in the Chef scene or the bag of chips in the Food scene). Without the HDR-informed loss, the network has difficulty modeling the large variation in radiance between views, and so the optimization produces spurious patches of density to model this view-dependent effect (despite regularization with the space carving loss).

**kaKw: Ablation–space carving.** An ablation study of the space carving regularization on the Cinema scene is in Supp. Table 2 (reproduced in Table R4). Without space carving regularization we find that spurious clouds of density appear in empty regions, worsening performance across most metrics.


Meta-review: The reviewers are more or less unanimously in favour of acceptance tsWB has not upgraded to a positive score, but in the discussion they have made clear that they find the paper interesting and relevant and that they would increase their rating if the authors include content from the discussion - which they have promised. The AC agrees that the question of supervision by photon histograms vs. discrete range measurements should be elaborated in the paper, please make sure to do this for the final version.
Decision: Accept (spotlight)