TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
Authors: Benjamin Attal
Eliot Laidlaw
Aaron Gokaslan
Changil Kim
Christian Richardt
James Tompkin
Matthew O'Toole
Venue: NeurIPS 2021 Poster (2021)
Keywords: View Synthesis
Reconstruction
Computational Imaging

Abstract: Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g., NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known to be an under-constrained problem, and so methods rely on data-driven priors for reconstructing dynamic content. We replace these priors with measurements from a time-of-flight (ToF) camera, and introduce a neural representation based on an image formation model for continuous-wave ToF cameras. Instead of working with processed depth maps, we model the raw ToF sensor measurements to improve reconstruction quality and avoid issues with low reflectance regions, multi-path interference, and a sensor's limited unambiguous depth range. We show that this approach improves robustness of dynamic scene reconstruction to erroneous calibration and large motions, and discuss the benefits and limitations of integrating RGB+ToF sensors now available on modern smartphones.

TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis
Benjamin Attal" Carnegie Mellon University
Eliot Laidlaw Brown University
Aaron Gokaslan Comell University
Changil Kim
Facebook
Christian Richardt University of Bath
James Tompkin Brown University
Matthew O'Toole Carnegie Mellon University
imaging cs. www.ccu..uuueeeeeemmmmm.mee edu/torfff
Abstract
Neural networks can represent and accurately reconstruct radiance fields for static 3D scenes (e.g.. NeRF). Several works extend these to dynamic scenes captured with monocular video, with promising performance. However, the monocular setting is known data-driven priors for reconstructing dynamic content. We replace these priors with measurements from time-of-flight (ToF) camera, and introduce a neural repre- sentation based on an image formation model for continuous- ToF cameras. Instead of working with processed depth maps, we model the raw ToF sensor mea- surements to improve reconstruction quality and avoid issues with low reflectance regions, multi-path interference, and sensor's limited unambiguous depth range. We show that this approach improves robustness of dynamic scene reconstruction to erroneous calibration and large motions, and discuss benefits and limitations of integrating RGB+ToF sensors that are now available on modern smartphones.
an under-constrained problem, and so methods rely on
1 Introduction
Novel-view synthesis (NVS) is a long-standing problem in computer graphics and computer vision, where the objective is to photorealistically render images of a scene from novel viewpoints. Given number of images taken from different viewpoints, it is possible to infer both the geometry and appearance scene, and then use this information to synthesize images at novel camera poses. One of the challenges associated with NVS is that it requires diverse set of images from different perspectives to accurately represent the scene. This might involve moving a single camera around a static environment [4, 16, 31, 32, 36), or using large multi-camera system to capture dynamic events from different perspectives [2, 7, 24, 38, 44, 56]. Techniques for dynamic NVS from a monocular video sequence have also demonstrated compelling results, though they suffer from various visual artifacts due to the ill-posed nature of this problem [26, 37, 42, 50, 52]. This requires introducing priors, often deep learned, the dynamic scene's 's depth and motion.
In parallel, mobile devices now have camera systems with both color and depth sensors, including Microsoft's 's Kinect and HoloLens devices, and the front and rear RGBD camera systems in the iPhone and iPad Pro. Depth sensors can use stereo or structured light, or increasingly the more accurate time-of-A flight principle for measurements. Although depth sensing technology is more common than ever, many NVS techniques currently do not exploit this additional source of visual information.
To improve NVS performance, we propose TöRF!, implicit neural representation for scene appearance that leverages both color and time-of-flight (ToF) images, as depicted in Figure This
"Correspondence should be addressed to Benjamin Attl:: battal@andren edu.edu. 'TRR== = ToF NeRF. Pronounced just like 'NeRF' but starts with a TT"
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(a) Acquisition process
(b) Color images
(c) Phasor images
(d) Novel Image
Figure 1: Illustration of time-of- flight radiance fields. (a) We move handheld imaging system around a dynamic scene, capturing (b) color images and (c) raw phasor images from a continuous-wan wave time-of-flighh flight (C-ToF) camera. (d) Then, optimize for continuous neural radiance field of the scene that predicts the captured color and phasor images. This allows novel view synthesis.
reduces the number of images required for static NVS problem settings, compared with just using color camera. Further, the additional depth information makes the monocular dynamic NVS problem
more tractable, as directly encodes information about the geometry the scene. Most importantly, rather than using depth directly, we show that using 'raw" ToF data in the form of phasor images [12] that are normally used to derive depth-is more accurate as it allows the optimization to correctly handle geometry that exceeds the sensor's unambiguous range, objects with low reflectance, and regions affected by multi-path interference, leading to better dynamic scene view synthesis. The contributions of our work include:
A physically- -based neural volume rendering model for raw continuous- wave ToF images;
A method to optimize neural radiance field of dynamic scenes with information from color and continuous wavee ToF sensors;
Quantitative and qualitative evaluation on synthetic and real scenes showing better view synthesis than NeRF [32] for few input views, and than two dynamic scene baselines [26, 52].
Related Work
While novel-view synthesis (NVS) is a long-standing problem in computer graphics and vision
[8, 9, 11, 23], recent developments in neural scene representations for NVS have enabled state-ofthe the- art results for a wide variety of settings [46, 49]. The common thread across many of these works is to bring learnable elements together with physics-based models and classical rendering processes.
The designs for neural scene representations often build on standard computer graphics data structures, including voxel grids [34, 47], multiplane images (MPIs) [31, 1, 51, 53, 57], multi-sphere images (MSIs) [3, 7], point clouds [30], and implicit functions scene geometry and appearance [32, 48, 54J. For example, DeepVoxels Voxels [47] represent scene discrete volume of embedded features to encode view-dependent effects, and enable wide baselines that may not be possible with other representations; however, the cost associated with this volumetric representation is that the memory requirements scale cubically with resolution. Alternatively, MPIs can be used to encode appearance from a single stereo pair [57]; the key benefit of this representation is the fast rendering speeds (ideal for interactive VR applications), though performs best for forward-facton scenes.
Implicit neural representations of a scene provide similar flexibility to voxel grids, but circumvent the high memory requirements. These implicit networks therefore have greater capacity to represent the appearance of scene. For example, scene representation networks (SRNs) [48] encode the geometry single neural network, which takes 3D points as input and outputs feature representation of local scene properties (e.g., surface color reflectance); rendering image requires a differentiable ray marching procedure that intersects rays with the implicit volume. Neural radiance fields (NeRFs) [32] encode 5D radiance fields (3D position with 2D viewing direction) to offer higher-fidelity geometry and visual appearance. While these implicit neural representations initially assumed a static scene, recent approaches also demonstrate the ability to perform dynamic NVS from monocular video [26, 37, 42, 50, 52], despite this being a highly ill-posed problem.
Including depth maps has proven beneficial to improve NVS results for long time [41]. However, surprisingly few NVS methods exploit the availability of depth sensors. One reason is that explicitly reconstructing depth maps for NVS [28, 55] may prove problematic, e.g., for thin structures, depth edges, complex reflectance, or noisy depth. We cireumvent these issues by proposing a neural representation that models raw ToF data for better view synthesis for both static and dynamic scenes.

---

Table 1: Mathematical symbol legend for the following equations and explanations.
Symbol
Units
Description
x
A point I
direction; unit vector E S².
nxx)
A normal; direction perpendicular to surface at point x.
Xr
point & units along direction w, X, = x wt.
wi
A direction incoming to point.
Wo
A direction outgoing from point.
L(x, w) LRGB
W-sr mm
Radiance measured by a camera al point in direction
W-sr m
Phasor radiance measured by a C-ToF camera.
w)
W-sr -r-__ -mm mm
Incident radiance point from direction.
[x(x,w)
W-sr -m-
Reflected radiance scattered from point in a direction.
W-sr
Radiant intensity a point light source.
w(x,w) w)
W-sr-1
Reflected radiant intensity scattered from a point in direction w due to a light source collocated with the camera.
o(x)
m- unitless unitless sr
Density function at point.
X(x
Transmittance function, i.e., accumulated density. Discrete approximation of the transmittance function.
xxx,xx)
fp(x. Wi, Wo: mx))
Scattering phase function.
((d)
unitless
Importance function for light path of length d.
3 Neural Volume Rendering of ToF images
A neural radiance field (NeRF) [32] is neural network optimized to predict set of input images. Assuming a static scene, the neural network Fe : (x,,w.) - (a(x),, (x,, w.)) with parameters takes as input a position X; and direction wu. and outputs both the density o(x)) at point x, and the radiance L,(x,, wo) of a light ray passing through x, in direction wa. The volume density function o (x,) controls the opacity at every point large values of a (xx) represent opaque regions and small values represent transparent ones, which allows representation of 3D structures. The radiance function (x,, w.) represents the light scattered at point x; in direction wa. and characterizes the visual appearance of different materials (e.g., shiny or matte). Together, these two functions can be used to render images scene from any given camera pose. The key insight of our work is that NeRFs can be extended to model (and learn from) the raw images of a ToF camera.
NeRF optimization requires neural volume rendering: given the pose of camera, the procedure generates an image by tracing rays through the volume and computing the radiance observed along each ray [32]]
I(x, wo) - 1 x()(x))) w.) dt, where x(x = Sí, ([xx-xxa// was)
-
(1)
describes the transmittance for light propagating from position x to x, =x wat, for near and far bounds [l,, 4r).
In practice, this integral is evaluated using quadrature [32]:
L(x, Wo) Σ fxx x1) -xxx)) [x(x Wo), where T(x, xx) II
(xx))xx (2) j=1
The value for Ax,- ||x++1 -xyll is the distance between two quadrature points.
Generalizing the neural volume rendering procedure for ToF cameras requires two changes. First, because ToF cameras use active light source to illuminate the scene, we must consider the fact that the lighting conditions of the scene change with the position of the camera. In Section 3.1. we derive the scene's 's appearance in response to collocating point light source with camera, which follows a similar derivation to that of Bi et [5]. Second, in Section 3.2, we extend the volume rendering integral to model images captured with a ToF camera. Similar to the approaches taken in transient rendering frameworks [17, 39] and by neural transient fields (NeTFs) [46], we incorporate a path length importance function into our integral that model different types of ToF cameras.

---

For simplicity, we assume that the function L(x, wo) is monochromatic, i.e., it outputs radiance at single wavelength. Later on, we will model output values for red, green, blue, and infrared light (IR). TRGB values correspond to radiance from ambient illumination scattering towards color camera, whereas /TE corresponds to the measurements made by ToF camera with active illumination.
3.1. Collocated Point Light Source. An ideal ToF camera responds only to the light from a collocated IR point source and not to any ambient illumination. With this assumption, we model radiance [x,, Wo) as a function of the source position [5]:
(o) | Fp(X), wi wa: ((x))) wi) dwi,
where the function (x,,, wi) represents the incident illumination from direction Wi. S is the unit sphere of incident directions, and the scattering phase function fp(x,, wi, Wo. n(x,)) describes how light is scattered at point in the volume. Note that the scattering phase function also depends on the local surface shading normal n. For a point light source at x (i.e., collocated with the camera), each scene point only lit from one direction. Thus, the incident radiance is given by
I
x X, xx
Xx wi ),,,)(((xx))
8
(4)
X X
where the scalar 7 represents the emitted radiant intensity of the light source, 1/1xx // is the inverse square light fall-off, and 6(-) is the Dirac distribution used to describe only the light from a single direction. This model ignores forward scattering, which is reasonable if the scene consists mostly of completely opaque surfaces. When substituted into Equation and Equation 3, the resulting forward model is
L(x,wa) = f/M T(x, x1)
(xx) w.) dt where Ixxx,, = fp (x,, Wo: Wo, n(x))) (5)
| x11|
where = w. in the scattering phase function fp as emitted light reflected along the same ray.
This expression is similar to Equation with two key differences: the squared transmittance term, and the inverse square falloff induced by the point light source. Similar to NeRF [32], we can once again numerically approximate the above integral using quadrature, and recover the volume parameters (a(x), [x(,,,)) w))) by training a neural network that depends only on position and direction.
3.2. Continuous- Wave ToF Model. ToF cameras use the travel time of light to compute distances [14]. The collocated point light source sends artificial light signal into environment, and ToF sensor measures the time required for light to reflect back in response. Given the constant speed of light, 3- 10 m/s, this temporal information determines the distance traveled. These devices have found widespread adoption from autonomous vehicles [25] to mobile AR applications [19, 21J.
Photorealistic simulations of ToF cameras involve introducing path length importance function to the rendering equation [17,39 39], and can be just as easily applied to the integral in Equation 5:
LIs w.) = pt T(x, x2)²
1 1n ||x-xx x1
( (xxx w. W xxIl) dt,
(6)
where the function (d) weights the contribution of a light path of length d. Note that light travels twice the distance between the camera's origin x and the scene point x,, As described by Pediredla et al. [39], the function (dd) can be used to represent wide variety of ToF cameras, including both pulsed ToF sensors [18] and continuous-wave ToF (C-ToF) sensors [12, 19, 35]. Here, as proposed system uses C-ToF sensor for imaging, the images modeled using the phasor (dd) exp (izzdt) ). where is the modulation frequency of the signal emitted by the C-ToF camera. Note that, because the function W is complex-valued the radiance (x w.) will also produce a complex-vall
phasor image [12]. In practice, phasor images are created by capturing four real-valued images that are linearly combined (see supplemental document for additional details). In Figure 1(c), we show the real component of the phasor image, with positive pixel values as red, and negative values as blue.

---

ray
ray
Σ ll LiF - Lio
ray 2
camera
t
ray? 2
(a) Ray marching
(b) Integrand of Eq. (5)
(c) Phasor
(d) Loss function
Figure 2: Neural volume rendering of C-ToF images. (a) We start with ray marching to evaluate the radiance and opacity at different points along the ray. (b) These samples represent the continuous integrand of Equation which describes the contribution of every point x; along the ray. For example, ray grazes both bunny ears, producing two distinct responses. (c) As described by Equation 6, we multiply the integrand with complex exponential, with the result represented here in the complex plane. Integrating this result produces single complex phasor, represented here by a vector with magnitude corresponding to reflectance and direction corresponding the phase (or distance). (d) The loss function compares rendered phasors with the raw measurements of C-ToF camera.
Contrasting with ToF-derived depth. ToF cameras typically recover depth by assuming only one point x, reflects light for every ray, i.e., the integrand of Equation is assumed to be zero for all other points x # X, (points in front of Xg reflect no light, and points behind x, are hidden). Under these assumptions, Equation simplifies the phasor:
LXF(X wo) = a W(2 - x, = a exp exp((122 [it
(oexxx1ll) xil).
(7)
where the phasor's magnitude, |LTO wo)l represents the amount of light reflected by this single point, and the phase, 2LToF Wo) Anf |x x, mod 2n, is related to distance ||x xs|l-
In real-world scenarios, it also possible for multiple points along ray to contribute to the signal, resulting in linear combination of phasor radiance values-known as multi multi-path interference. This can degrade the quality of depth measurements for C-ToF camera. For example, around depth edges, pixel integrates the signal from surfaces at two different distances from the camera (e.g., Figure 2), resulting 'flying pixel' artifacts [43] (i.e., 3D points not corresponding to either distance). Similar artifacts occur when imaging semi-transparent or specular objects, where two more surfaces contribute light to a pixel.
Optimizing NeRFs with phasor images via Equation 6 therefore has three distinct advantages over using derived depth maps via Equation 7. (i) For ranges that span values larger than 27. the true range ambiguous, as there are multiple depth values that produce the same phase. For example, typical modulation frequency of f = 30 MHz for a C-ToF camera corresponds to an unambigous range of & R m. By modeling the phasor images directly, we avoid the issues associated with 27 recovering depth images for scenes that exceed this range (Figure 3). (ii) Depth values become unreliable (noisy) when the amount of light reflected to the camera is small. Modeling the phasor images directly makes the solution robust to sensor noise (Figure 4). (iii) For regions near depth edges (Figure 2) or for objects with complicated reflectance properties (e.g., transparent specular surfaces), the light detected may not travel along a single path: this results in mixtures of phasors, producing phase values that do not correspond to single depth. Equation models the response from multiple single-scattering events along a ray, providing us with a better handle over such scenarios.
4 Optimizing Dynamic ToF + NeRF = TöRF
4.1. Dynamic Neural Radiance Fields. One key advantage of working with phasor images is that the method can capture scene geometry from single view, which enables higher-fideliyy novel-view synthesis dynamic scenes from potentially moving color camera and C-ToF camera pair. To support dynamic neural radiance fields, we model the measurements with two neural (x+, 5D function of position and direction, while the second, dynamic network 1- (Xr, Wo. T) (x,, T), /. (x,, wa), pay (x,, wo: r), b(x,, r)) is 6D function of position, direction, and time Instead of directly consuming time т, the dynamic network receives latent code z, which is
networks. The first, static network Fatut (Xr, w.) + (x:t)((;;).. Latal( Xt, Wa), / (x;, (x), is a
5

---

Color (input)
Depth (input)
Color (TöRF)
Depth (TöRF)
Chii
Figure 3: Raw phasor supervision avoids wrap-around errors. Wrap-around phase bounds the range of useful ToF measurements (left), causing errors when depth is used supervision. Our approach of modeling raw phasor measurements within the neural volume alleviates this problem (right). This is because only one phase offset is consistent across multiple camera views.
Color (input)
Depth (input)
Color (ToRF)
Depth (ToRF)
Figure 4: Raw phasor supervision reduces noise in dark objects. The weak signal reflected back by dark objects (e.g., the computer monitor) results in noisy depth measurements. As TöRF does not rely on depth explicitly and instead models raw phasor images, our recovered depth map better captures the scene geometry in comparison ToF-derived depth. This is because when phasor magnitudes small, TöRF falls back on triangulation cues to recover geometry.
optimized per frame, similar Li et al. [24]. Following the approach of Li et al. [26], we blend the outputs of the static and dynamic networks using a position- and time-dependent blending weight b(x,, T) that is predicted by the dynamic network Foyo as in Gao et al. [10]. This produces density blendd radiance L, blend and radiant intensity pblend values to pass into our image formation models:
LRGB(X, Wo:
T blend (x.X,, Xx, T) bland Xx, L) Lblend (xt, Wo: T)
(8)
2 7blend (x, X,, 7 blend (x,, 7) 7) blend (x,, war (2 xx dt.
LIo((x WoiT) T)
(9)
|x x
See the supplemental document for explicit definition of the blending terms.
4.2. Loss Function. Given of color images and phasor images captured of scene different time instances, we sample a set of camera rays from the set of all pixels, and minimize the following total squared error between the rendered images and measured pixel values:
C
Σ |RRBB (x, Wo T) LRGB (x, Wo. T) + Al wo. T) LTOP(x, Wa, T)1 (10)
(x.u0..T)
where the scalar 2 controls the relative contribution of both loss terms, LRGB (x, wo: T) represents the measurements of color camera, and LToF(x, wo. T) represents the phasor measurements of C-ToF camera. At training time, we reduce the weight in later iterations to prioritize the color loss (halved every 125,000 iterations).
4.3. Camera Pose Optimization. In past works, COLMAP [45] has been used to recover camera poses for NVS. However, COLMAP fails to recover accurate camera poses for many real scenes even if we masked dynamic regions [20]. Further, COLMAP only recovers camera poses up unknown scale, whereas our ToF image formation model assumes a known scene scale. As such, for real- world scenes, optimize camera poses from scratch within the training loop. First, weights the static neural network Frtst as well as the camera poses for each video frame and the relative rotation and translation between the color and C-ToF sensor, with a learning rate of 10-3 After 5000 iterations, decrease the pose learning rate to 5 10-1, and optimize our full model.
optimize the

---

4.4. Ray Sampling, Many physical camera systems do not have collocated color and ToF cameras. As such, to train our model, we trace separate rays through the volume for color and ToF measurements. We alternate using the color loss and the ToF loss for every iteration. Further, like NeRF [32], we use stratified random sampling when sampling points along a ray.
5 Experiments
5.1. Hardware. We use an iDS U1-3770CP-C-HQ machine vision camera to provide RGB mea- surements (640x480 @ 30 f0s; downsampled to 320 240), and Texas Instruments OPT8241 sensor to provide phasor measurements (320x 240 @ 30 fps) with an unambiguous range of 5 m. Both cameras are mounted with bascline of 41 mm. We use OpenCV calibrate intrinsics, extrinsics, and lens distortion. See the supplement for details.
For optimization, we use NVIDIA GeForce RTX 2080 Ti with 11 GB RAM. Our model takes 12-24 hours to converge, and 3-5 seconds per frame to generate a novel view (256x256).
5.2. Data. We captured the Bhoth Cuphoard, Photocopier, DeskBox, and StudyBook se- quences with our handheld camera setup. Each is indoors in an office with a person performing dynamic action, and includes view dependence from real-world materials. PhoneBooth includes multi-path interference effects from a glass door, and Photocopier includes wrap-around phase effects in the distance. For comparison, we also captured the Dishwasher sequence on an iPhone 12 Pro, which uses ToF sensor to capture depth (raw measurements are not available). Finally, we create synthetic raw C-ToF sequences Bathroom, Bedroom, and DinoPear by adapting the physically-bann path tracer PBRT [40] to generate phasor images with multi-bounce and scattering effects.
5.3. Few-View Reconstruction of Static Scenes. We demonstrate that integrating raw ToF mea- surements in addition to RGB enables TöRF to reconstruct static scenes from fewer input views, and to achieve higher visual fidelity than standard NeRF [32] for the same number of input views. Table 2 contains quantitative comparison reconstructions from just 2 and 4 input views. To enable the comparison on 10 hold-out views, we use ground-trute camera poses for both methods. With just two input views, TöRF's added phasor supervision better reproduces the scene than NeRF, as one might expect. This closely resembles a camera system that might exist on smartphone, and shows the potential value of ToF supervision for dynamic scenes if we consider a static scene as one time step of a video sequence. For four views, NeRF and TöRF produce comparable RGB results, though depth reconstructions are significantly more accurate (Figure 5).
two synthetic sequences, Bathroom and Bedroom, for
5.4. Dynamic Scenes. We compare reconstruction quality on the synthetic dynamic sequence DinoPear in Table with 30 ground-truth hold-out views and depth maps. Compared to methods that use deep depth estimates (NSFF and VideoNeRF), TöRF produces better depth and RGB views. While ToRF PSNR is slightly lower than NSFF's, the perceptual LPIPS metric is significantly lower for TöRF, which matches the findings from our qualitative results. TöRF also produces better depth and RGB reconstructions than the same methods modified to use ToF-derived depth (NSFF+ToF, VideoNeRF-TToF).
For real-world scenes, we show results and comparisons in Figure 6. VideoNeRF+ToF shows stronger disocclusion artifacts and warped edges near depth boundaries, and cannot recover from depth maps with wrapped range. NSFF suffers from severe ghosting and stretching artifacts that negatively impact the quality of the results. Our results show the highest visual quality and most accurate depth maps. Please see the videos on our website for in-motion novel-view synthesis.
6 Discussion
6.1. Limitations. Introducing ToF sensors into RGB neural radiance fields aims to improve quality by merging the benefits of both sensing modes; but, some limitations also brought in through ToF sensing. C-ToF sensing can struggle on larger-scale scenes; however, using multiple different modulation frequencies can extend the unambiguous range [12]. Using different coding methods can also increase depth precision [13]. While C-ToF sensors typically struggle outdoors, EpiToF [1] has demonstrated the ability perform 15 ranging under strong ambient illumination. Further, for each measurement, C-ToF sensors require capturing four or more images quickly at different times, which can cause artifacts for fast- moving objects.
7

---

Table Phasor supervision aids few-view reconstruction. Each cell contains RGB image similarity measures, and metrics computed on 10 hold-out views. TöRF significantly outperforms NeRF on both synthetic static scenes and produces more accurate depth estimates (Figure 5), particularly from just two imput views. Note that the metric depth error 'MSE (D)' is affected by mirrors, particularly in the bathroom scene, whose depth defined by the plane of the mirror and not the objects in its reflection.
Bathroom
Bedroom
Views Method
MSE (D)
PSNR
SSIM
LPIPS
MSE (D)
PSNR
SSIM
LPIPS
NeRF [32]
0.97
16.56
0.660
0.022
18.43
11.59
0.313
0.056
2
TöRF (ours)
2.12
19.21
0.739
0.015
0.31
22.09
0.840
0.010
NeRF [32]
0.70
24.17
0.864
0.008
0.94
28.29
0.936
0.003
4
TöRF (ours)
0.76
26.18
0.879
0.009
0.27
29.79
0.938
0.002
Bathroom scene
Bedroom scene
Color image
Depth map
Color image
Depth map
(a) Ground truth
(b) NeRF [32]
2-view
(c) TORF (ours)
(d) NeRF [32]
4-view
(c) TöRF (ours)
Figure 5: Adding ToF aids reconstruction for few views in static scenes. NeRF quality suffers as the number of views decreases, but adding ToF data makes view synthesis possible with two RGB views. Note the cleaner depth, sharper edges, and thin geometric details such as the lamps above the mirror.

---

(a) Input
(b) NSFF
(c) VideoNeRF+ToF
(d) TöRF (Ours)
DeskBox
PhoneBooth
Chess
Photocopier
Cupboard
Figure 6: Adding ToF supervision helps improve quality: Compared to two video baselines, Video- TeF-ded depth) and NSFF [26], our approach reduces errors in static scene components and some dynamic components. All models were trained for comparable times. Please see the videos on our website for additional comparisons motion.
NeRF [52] (modified

---

Table 3: Evaluation on ground-truth hold-out views for the dynamic DinoPear sequence shows improved depth and RGB results for our method. Note that NSFF and VideoNeRF are given manually unwrapped ToF depth produced by adding 2 to all phase values below a certain threshold. The TöRF approach of using raw phasor images produces better depth reconstructions. While NSFF
produces the highest PSNR, this does not match the perceived visual quality please see our webpage.
Method
Depth MSE 4
PSNR A
SSIM
LPIPS v
NSFF [26]
0,021 + 0.003
22.64 = 1.46
0.554 = 0.029
0.039 + 0.010
+ ToF depth
0,010 = 0.002
21.84 = 0.72
0.382 = 0.021
0.037 0.014
+ ToF depth (unwrapped)
0.007 + 0.002
21.70 + 0.98
0.387 + 0.028
0.040 + 0.013
VideoNeRF [52]
0.008 0.002
21.32 + 1.03
0.358 + 0.032
0.032 0.017
+ ToF depth
0.011 + 0.002
19.75 + 1.07
0.275 + 0.021
0.041 + 0.016
+ ToF depth (unwrapped)
0.009 0.002
20.72 + 1.03
0.350 + 0.033
0.032 + 0.016
TöRF (ours)
0.005 0.001 22.19 + 1.75
0.561 = 0.052
0.028 + 0.011
Even with ToF data, objects imaged at grazing angles or objects that are both dark (low reflectance) and dynamic remain difficult to reconstruct, e.g. dark
ToRF RGB
TöRF Depth
detee
hair (Figure 7). Further, neural networks have limited capacity to model dynamic scenes. which limits the duration of dynamic sequences. This is limitation of many current neural dynamic scene methods.
6.2. Potential Social Impact. Scene reconstruc- tion and view synthesis are core problems in visual computing for determining the shape and appearance of objects and scenes. Neural approaches to these tasks hold promise to increase accuracy and fidelity. At the methodological level, integrating ToF data im- proves accuracy, but restricts use to scenarios where active illumination is detectable. While the recov- ery of shape and appearance has many applications, negative impact may include synthesizing images from perspectives or time instances that were never captured (falsifying media), extending surveillance through higher-fidelity reconstructions (security), or copying physical objects to 'rip off' designs.
Figure 7: In the Photocopier (top) and Study- Book (bottom) scenes, RGB and depth NVS show that dark and dynamic objects pose diffi- cultics. Dark hair incorrectly extends in front to the side of the face due failure to reconstruct dynamic motion.
Practically, current neural approaches are more computationally expensive in both optimization and rendering than classic image-basee based rendering. Our work required GPUs to optimize for many hours h12-24h), h). Without renewable energy sources, this use will generate CO2 emissions, requiring 1.5-3 kg -Quuiivvllssnss per scene for optimization and 0.01-0.02 kg CO,-cquivalents per sequence for rendering (numbers generated by MI. CO2 Impact (22]). Concurrent work in neural radiance fields reduces this cost using caching, spatial acceleration structures, and more efficient parameterizations, and real-world deployment should exploit these approaches to reduce the CO2 emission impact.
Conclusion
Modern camera systems integrate multiple modes of sensing, and our reconstruction methods should exploit this information to improve quality. To this end, we formulate a neural model for time-of- flight radiance fields based on physical RGB+ToF image formation. We demonstrate an optimization method to recover TöRF volumes, and show that improves novel-view synthesis for few-view scenes and especially for dynamic scenes. Further, we demonstrate that using raw ToF phasor supervision leads to better performance than using derived depth directly, allowing both sensing modes to help resolve errors, limitations, and ambiguities. Future work may extend the combination of additional sensors into neural radiance fields, e.g., dynamic vision sensors [27] or event cameras may be used to measure scenes at higher speeds. Further, a collocated point light source has been shown to be able to render photos of scenes under non-collocated illumination conditions [5]. As a result, we believe ToF images may also serve support relighting applications.
10

---

Acknowledgments and Disclosure of Funding
Thank you to Kenan Deng for developing acquisition software for the time-of-llight camera. For our synthetic data, we thank the authors assets from the McGuire Computer Graphies Archive [29], Benedikt Bitterli for Mitsuba scene files [6]. Davide Tirindelli for Blender scene files, 'Architectural Visualization demo Blender scene by Marek Moravec (CC-0 Public Domain) [33], 'Rampaging T-Rex' from the 3D library of Microsoft's 3D Viewer, and 'Indoor Pot Plant 2' by 3dhaupt from Free3D commercil)a [15].
For funding, Matthew O"Toole acknowledges support from NSF IIS- IIS-2008464, James Tompkin thanks an Amazon Research Award and NSF CNS-2038897, and Christian Richardt acknowledges funding from an EPSRC-UKRI Innovation Fellowship (EP/S001050/1) and RCUK grant CAMERA (EP/M023281/1,, EP/T022523/1).
References
[1] Supreeth Achar, Joseph R. Bartels, William L. 'Red' Whittaker, Kiriakos N. Kutulakos, and Srini- vasa G. Narasimhan. Epipolar time-of-fligt imaging. ACM Trans. Graph., 36(4):37:1-8, 2017. doi: 444337295593377.6666
[2] Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernandez. Sameer Agarwal, and Steven M. Seitz. Jump: Virtual reality video. ACM Trans. Graph., 3586)119:
2016. doi: 1145/2980177 4155//80079.2980011 29.2990257
Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. MarryODShka: Real-time 6DoF video view synthesis using multi-sphere images. ECCV, 2020. 10t110.00077778-3.. 58452-8 26.
[4] Tobias Bertel, Mingze Yuan, Reuben Lindroos, and Christian Richardt. OmniPhotos: Casual 360 VR photography. ACM Trans. Graph., (9(6))267:1-22 2020. doi: 10i:10.1144 11444444444555.. 4685.347708..41777
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold- Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv:2008. 033220 2020.
[6] Benedikt Bitterli. Rendering resources, 2016. URL http:: wwttbbiittteeeeee me/ wtt.://rreeeee/....///....
[7] Michael Broxton. John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with a layered mesh representation. ACM Trans. Graph., (9()))86:1 1-11) 2020. (tl.1444/3365669339244444
[8] Chris Buchler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Unstructured lumigraph rendering. SIGGRAPHI pages 425-432, 2001. 1oi:10.1144/38331 145/383259.888277
[9] Paul Debevec, Yizhou Yu, and George Borshukov. Efficient view-dependent image-based rendering with projective mapping. In Proceedings of the Eurographics Workshop on Rendering, pages 105-116. 1998. doi: 10010077777-37770--655555
10.
[10] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. ICCV, 2021.
[11] Steven J. Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael F. Cohen. The lumigraph. In SIGGRAPHI, pages 43-54, 1996. 104::00.145//37770..22200
[12] Mohit Gupta, Shree K. Nayar, Matthias B. Hullin, and Jaime Martin. Phasor imaging: A generalization of correlation-based time-of-fligt imaging. ACM Trans. Gruph., 34(5):156:1-188 2015, doi: 10. 145/2735702.
[13] Felipe Gutierrez- Barragan, Syed Azer Reza, Andreas Velten, and Mohit Gupta. Practical coding function design for time-of-flight imaging. In CVPR, pages 1566-1574, 2019.
Butrennez-Barrggan,
[14] Miles Hansard, Seungkyu Lee, Ouk Choi, and Radu Patrice Horaud. Time-of-Flight Cameras: Principles, Methods and Applications. Springer, 2012. 10:: 1177779988----47-4653353-
[15] Dennis Haupt. Indoor Pot Plant 2, November 2019. URL indoor-pot- plant 77983. html. Non-commercial only.
cttpddddooommmmmmmeoo modell
[16] Peter Hedman, Suhib Alsisan, Richard Szeliski, and Johannes Kopf. Casual 3D photography. ACM Trans. Graph., 36(6):234:1111 2017. 1et.10.1145/313000 13/11443/3308003333333333

---

[17] Adrian Jarabo, Julio Marco, Adolfo Muñoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. A frame- work for transient rendering. ACM Trans. Graph. 33(6):17711-00, 2014. 4o::10..14//266122.2266
[18] Walter Koechner, Optical ranging system employing a high power injection laser diode. IEEE Transactions Aerospace and Electronic Systems, (11::81-91, 1968. 10.1
[19] Andreas Kolb, Erhardt Barth, Reinhard Koch, and Rasmus Larsen. Time-of-fighh cameras in computer graphies. Comput. Graph. Forum, (9( (1):111-1,, 2010. doi: 10.1 11..11//////........111...11 11j 1111111//17711111111111111 1467-6659. 009..000900110....
[20] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In CVPR, 2021.
[21] George Alex Koulieris, Kaan Akşit, Michael Stengel, Rafal K. Mantiuk, Katerina Mania, and Christian Richardt. Near-eye display and tracking technologies for virtual and augmented reality. Comput. Graph. Forum, 38(2):99-519, 2019. doi:10.111 11cgf..//......... 13654.
[22] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. 1r0.v:1900.09700, 2019.
[23] Mare Levoy and Pat Hanrahan. doi:10.1 15111114//777773711111
Light field rendering. In SIGGRAPH, pages 31-42, 1996.
[24] Tianye Li, Mira Slaveheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tan- ner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3D video synthesis. 2XXi:22103.02555 2021.
[25] You Li and Javier Baanez-Guzmal Lidar for autonomous driving: The principles, challenges, and trends for automotive lidrr and perception systems. IEEE Signal Processing Magazine, 37(4):50-61, 2020. doi: 10.1 2009///..020.2973660
[26] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[27] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. A 128x 128 120 15 ju latency asyn- chronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits, 43(2):566-576, 2008. 1011SSS..0/SSSS20..9999...
[28] Xuan Luo, Jia- Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Trans. Graph., 391):11: 2020. doi: 14/333666666669.999377.
[29] Morgan McGuire. Computer Graphics Archive, July 2017.
[30] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla Neural rerendering in the wild. In CVPR, 2019. doi: 10. 10901000// 2019,00044
[31] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans, Graph., 38 38(4):2:: 1-14, 2019, doi: 1455/33663463322298800
Nima Khademi Kalantari, Ravi Ramamoorthi,
[32] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF; Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 10.1007997 3-030-58452-8 24.
[33] Marek Moravec. Architcctural DinalizzationBBlndee demo scene, November 2019. URL https:
/ //www, blender Sleender..rg/donnnn demo/ files/. CC-0 Public Domain.
[34] Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. HoloGAN: Unsuper-
vised learning of 3D representations from natural images. In ICCV, 2019. doi: 10t.00..19//CC22201199
[35] Matthew O'Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutu- lakos. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Groph., 33(4):87:1-11, 2014. doi: 1455/2601097.260 103.
[36] Ryan Styles Overbeck, Daniel Erickson. Daniel Evangelakos, Matt Pharr, and Paul Debevec. A system for acquiring, compressing, and rendering panoramic light field stills for virtual reality, ACM Trans. Graph., 1-5 2018. 11433...145/322222.3337 327.3333..
[37] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo- Ricardo-Martin Martin Brualla. Nerties: Deformable neural radiance fields. In ICCV, 2021.
12

---

1111..100/PPMML22000307000 [38] Albert Parra Pozo, Michael Toksvig, Tenry Filiba Schrager. Joyse Hsu, Uday Mathur, Alexander Sorkine- Hornung, Rick Szeliski, and Brian Cabral. An integrated 6DoF video camera and system design. ACM Trans. Gruph., 3866):21::1116, 2019. doi:10.1 115//3155099 3335555..
[39] Adithya Pediredla, Ashok Vecraraghavan, and loannis Gkioulckas. Ellipsoidal path connections for time-gated rendering. ACM Trans. Graph., 38(4):38:1-11 2019. (50:100.144/30064633333000
[40] Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically Based Rendering: From Theory to Implemen- tation. Elsevier Science, 3rd edition, 2016. ISBN 9780128007099. URL https: / /////wwpbbbb.. book org/-
[41] Kari Pulli, Michael F. Cohen, Tom Duchamp, Hugues Hoppe. Linda Shapiro. and Wernrr Stuetzle. View- based rendering: Visualizing real objects from scanned range and color data. Proceedings of the Furogrophics Workshop on Rendering, pages 23-34, 1997. 100010..007/78... 7-777988--7711-888---3
[42] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2021.
[43] Malcolm Reynolds, Jozef Doboš, Leto Peel, Tim Weyrich, and Gabriel Brostow. Capturing time-of- data with confidence. In CVPR, pages 945 952, 2011. 1o1.10.110//CVPRR222 201.///C/P...11.995555000
[44] Christopher Schroers, Jcan-Charles Bazin, and Alexander Dorkinc-Hornung An omnistcreoscopic video pipeline for capture and display real-world VR. ACM Trans. Graph., 37())37:1-13, 2018. doi: 10.1 10..114///225550
[45] Johannes L. Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, pages 4104-4113, 2016.
[46] Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu. Non-line-of-sight imaging via neural transient fields. TPAMI, 43(7):2257-2268, 2021.
[47] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias NieBner, Gordon Wetzstein, and Michael Zoll- höfer. Voepls Learning persistent 3D feature embeddings. In CVPR, pages 2437-2446, 2019. doi: 0oi:10.1109/CCS 10t11..11//VVP.22000 .00254.
[48] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Continuous 3D-structure-awwre neural scene representations. In NeurlPS, 2019.
[49] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin- Bruuala, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, and Michael Zollhöfer. State of the art on neural rendering. Comput. Graph. Forum, 39(2)7011-777, 2020. doi:10.11 11 1/egff 14022.
[50] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a deforming scene from monocular video. In ICCV, 2021.
[51] Richard Tucker and Noah Snavely. Single-view view syuthesis with multiplane images. CVPR, 2020. 10::10..1 1091110///99 000.00.22000000000
[52] Wenqi Xian, Jia-Bin Huang. Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
[53] Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. Deep view synthesis from sparse photometric images. ACM Trans. Graph., 38(4))6::1-13 2019. doi: 10: 1145/3306346.3320000
[54] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. NeurlPS, 2020.
[55] Jae Shin Yoon. Kihwan Kim, Orazio Gallo. Hyun Soo Park, and Jan Kautz. Novel view synthe- sis of dynamic scenes with globally coherent depths from monocular camera. In CVPR, 2020.
doi: 10::10.11 109/CCV4422.222200003333
[56] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint video using layered neural representation. ACM Trans. Graph., 2021. $o1:10.1445/450026333999
[57] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifica- tion: Learning view synthesis using multiplane images. ACM Trans. Graph., 37(4):65:1 12, 2018. fof:10.144/1193317220133
13


Reviews:
[{"review_id": "nNSTt5W2Mdj", "reviewer": "Reviewer_SWuj", "summary": "The paper proposes a method to reconstruct dynamic radiance fields from   a single monocular video for novel view synthesis. Different from previous \nmethods that rely on RGB images only, the paper makes use of both RGB and \ntime-of-flight cameras and use the phasor information of the ray to supervise \nthe geometry reconstruction. They derive the image formation model for the \nToF imagee. The experiment results show that the proposed method generates better depth estimation and dynamic scene view synthesis results. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths\n1. The usage of ToF camera for dynamic scene reconstruction is interesting \nand novel. The paper shows improvement over previous method that replies on \nRGB or RGB+depth.\n\n2. The proposed method generates more accurate depth estimations. \n\n3. The paper shows results on diverse synthetic and real scenes to validate \nthe proposed method and design choices. \n\n4. The method can generate better view synthesis results than baseline methods \non sparse inputs. \n\n### Weakness\n\n1. While the paper shows that the proposed method achieves better BRISQUE scores,\nfor most of the video results, the results of the proposed method are \nmore blurry, and the results of VideoNeRF has more fine-grained details. What is the \nreason behind this? Also the proposed method has clear artifacts around depth \ndiscontinuities and obvious distortions (StudyBook, Dishwasher). While the paper \nacknowledges such limitations, they weaken the contributions of the paper. \n\n2. BRISQUE is not a common metric used for this task. I would suggest the paper should \nhave comparison to ground truth on real scenes by capturing multi-view RGB images simultaneously and use some of the views as testing and report the standard metrics such as PSNR, SSIM \nand LPIPS. Currently the paper compares to VideoNeRF on synthetic data (Table 2 in supp) \nand use the GT depth for it, and from the results we can see that VideoNeRF is better than \nTORF consistently. However, we cannot draw the conclusion whether VideoNeRF is still better \nwhen using the ToF depth. Therefore, I would suggest adding quantitative comparisons on \nreal data with GT.  \n\n3. The infrared camera is usually sensitive to background light and sunlight. It's not \nclear to me whether the method works well for outdoor scenes and scenes with visible \nlight sources. Is the method robust the potential corruptions in the ToF information?\n\n4. The synthetic data such as Bedroom and bathroom has a lot of noise. It would be better \nto increase the number of samples during rendering and generate noise-free images. \n\nOverall I like the idea of using ToF cameras to provide additional supervision signals of \nneural radiance fields, and the paper is able to show improvement over previous methods. \nHowever, the quality of the results is not fully convincing to me, and considering that   \nthe paper uses a more complex acquisition setup, I would expect better results than the \nones shown in the paper. Therefore I vote for \"marginally above the acceptance threshold\".\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "Z9OFYwOIeAP", "reviewer": "Reviewer_EBPE", "summary": "The goal is geometry and appearance reconstruction of a dynamic scene from monocular RGB and C-ToF (raw depth) measurements. The proposed method extends dynamic NeRF models by a principled image formation model of C-ToF measurements. It is assumed that only the ToF camera light enters the ToF camera sensor, that both are at the same position, and that there is only one bounce per light path. Experiments on few-view static reconstruction and monocular dynamic reconstruction are included.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The image formation model is a valuable extension of (dynamic) neural radiance fields to raw depth measurements.\n\nThe experiments show good static reconstruction quality from two views already. The dynamic reconstruction quality is similar to Video-NeRF (unpublished at the time of submission I believe) even though Video-NeRF uses additional regularization losses.\n\nI almost missed the index.html file in the supplemental folder and suggest to mention it in the supplemental PDF or the main paper.\n\nExtrinsics:\n\nThe novel view synthesis results seem to be along different camera paths (and resolutions) for different methods. The NSFF camera circles seem to be a bit larger than for ToRF, e.g. Cupboard has a much more aggressive path for NSFF than for ToRF. How are the extrinsics for novel view rendering made comparable between methods? Are quantitative results reported on these differing NVS paths?\n\nComparisons:\n\n- Both NSFF and Video-NeRF originally estimate depth from RGB images during preprocessing and then use that estimated depth during optimization. If computationally feasible, evaluating Video-NeRF and NSFF on both recorded and their respectively own estimated depths would be nice. Measured depth might show other artifacts than estimated depth while measured depth might be more accurate than estimated depth, so it is not clear to me which one is fairer for comparisons.\n- Video-NeRF shows similar quality to ToRF. Is that still the case when using estimated (instead of recorded) depth for Video-NeRF? I.e. are the regularization losses in Video-NeRF sufficient to make up for the lack of recorded depth?\n- In general, it would be nice to show NVS from viewpoints that are more different than currently. The results show NVS that is very close to the input camera path. I understand that this is challenging for this type of method currently, but maybe geometry supervision helps? The reconstructions look like they are not really correct (disappearing hands, torn hair), so I presume that ToRF does not work well for such cases.\n- Similarly, larger camera motion would be nice (in which case Colmap might also get enough baseline to estimate extrinsics). As long as NVS sticks relatively close to the input camera path similar to NSFF, the quality should be decent.\n\n\nSummary:\n\nOverall, the method is a non-trivial extension to dynamic NeRF models and is valuable. For the dynamic setting, I believe that clarifications regarding the extrinsics are necessary and that additional evaluations on NSFF and Video-NeRF would strengthen the paper and provide useful insight but are not strictly necessary for acceptance.\n\nMinor points:\n\n- Not necessary since it is a preprint, but there is another concurrent dynamic NeRF paper: Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, Jiajun Wu. \"Neural Radiance Flow for 4D View Synthesis and Video Processing\", arXiv:2011.13084.\n- line 136: \"component component\"\n- The DinoPear and Dishwasher depth videos of ToRF are in the wrong color coding.\n- Is Eq. 6 or Eq. 7 used for the results? I.e. are multiple paths supported?\n\nAfter rebuttal:\n\nMy points were addressed well enough. My main concern was the comparability of the extrinsics across methods, but I understand the difficulty in achieving that and I don't see a better option than what the authors are promising for the revision.\n\nWhile not mentioned by the authors in their summary of changes, they also promised that \"We will include larger baseline view synthesis results in our revised submission.\"\n\nThere are a few remaining downsides that keep this submission from being outstanding, especially important to me is the comment by reviewer SWuj: \"However, the quality of the results is not fully convincing to me, and considering that the paper uses a more complex acquisition setup, I would expect better results than the ones shown in the paper.\"\n\nOverall, however, I believe that the submission does rise from slightly below to slightly above the threshold for acceptance given its novelty and timeliness.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are addressed and potential negative societal impact is well described.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "R_CegQiRH4t", "reviewer": "Reviewer_XGQC", "summary": "The authors introduce a novel neural representation based on an image formation model for continuous-wave ToF cameras, thereby replacing the data-driven priors used by other approaches on the reconstruction of dynamic scene parts with measurements from a time-of-flight (ToF) camera. Using the modeling of raw ToF measurements instead of depth maps allows better handling challenging scenarios such as regions with low reflectance, multi-path interference, or the limited unambiguous depth range of sensors.\n\n\nThe proposed approach exhibits a potential benefit regarding robust static scene reconstruction from only a few views and regarding increased robustness of dynamic scene reconstruction. Still, the evaluation is lacking in some regards as discussed below.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe proposed approach seems novel. There seems to be not much work yet on Time-of-Flight Radiance Fields. Related work seems to be mostly covered. I would recommend adding the recent work by Shen et al., Non-line-of-sight Imaging via Neural Transient Fields’ as it also considers time-resolved data.\n\nQuality:\n- Besides a discussion of the used hardware, the metrics used for evaluation and the used data the authors provide results for static and dynamic scenes.\n- For static scenes the authors demonstrate in quantitative and qualitative comparisons that fewer views can be used in comparison to NeRF. Detailed experiments on different configurations/poses of the views have not been conducted.\n- For dynamic scenes the authors provide quantitative and qualitative comparisons to NSFF and VideoNeRF, where the proposed method also seems to allow more plausible results in comparison to the other techniques. However, there seems to be a clear shift in the overall color distribution (Figure 5) of the depth which might indicate an offset or scale issue. This needs proper evaluation to allow getting insights on the real potential of the approach (I would also consider raising my voting by 1 point). Highlighting regions with zoom-ins could be another option to demonstrate the benefits for fine-grained structures.\n\nClarity:\n- The paper is well-structured and easy to follow. Figures/tables and captions are informative.\n- The introduction/motivation is good.\n- The supplemental provides a comprehensive overview on different aspects and additional results. \n\nSignificance:\n- The paper seems reproducible by more experienced readers. The authors mention that they aim to release materials for reproduction later on, but this seems not guaranteed.\n\n\nPost-Rebuttal Comments:\n\nThe authors addressed concerns regarding ...\n- the their depth map results (Figure 5) where a the color shift bug and a resolution bug have been resolved,\n- additional zoom-ins to improve the exposition of the results,\n- the influence of the number of view configurations.\nI also appreciate the additional results they provided on other reviewer comments, e.g. regarding the effect of the accuracy of the depth quality for VideoNeRF and the effect of the number of samples used for rendering.\n\nThe authors' promise that \"We will include larger baseline view synthesis results in our revised submission.\" (also highlighted by Reviewer EBPE) is indeed interesting, but at the moment still kept vague, i.e., they did not yet provide details regarding the behavior in this regards.\n\nAs a result of the clarification of bugs and the extension of the evaluation, I also see the submission now above the threshold for acceptance.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a discussion of limitations and the potential societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Q8HsdidSDOi", "reviewer": "Reviewer_rBB3", "summary": "This paper is trying to render novel views of static/dynamic scenes by using RGB and ToF sensors. Rather than directly using depth from ToF sensors, this paper takes a step further to work on phase images, and is more likely to handle challenges of using ToF sensors. Experiment results show that the rendered images look more realistic, with less artifacts, especially for dynamic scenes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. There are quite a lot endeavors trying to extend NeRF to dynamic scenes, which is higly illposed. To merge depth information from a ToF sensor is a nice choice, and this paper seems to be the first in utilizing TOF in the NeRF framework. Furthermore, this paper further works on the phase images, thus is more likely to avoid some commen issues of ToF, in the presence of complex reflectance and depth noise. \n\n2. The rendering model for ToF images has been carefully adapted in the NeRF framework, and those equations look correct to me. \n\n3. As for the results, clear improvement can be observed, when compared with existing methods in which depth information is not used. This is reasonable, since more information is used here. However, the contribution of using phase ToF images, rather than depth directly generated from a ToF sensor, is not clear. Comparison should be added to show the difference, so as to justify the necessity of working from phase images (which is the core claim of this paper). \n\n4. When comparing the results from this paper and those from NeRF, we can see that, the resolution and the sharpness of the proposed method are inferior to those from NeRF. One key reason might be the resolution difference between RGB and ToF. As we know, the rgb camera has high resolution, and the ToF sensor usually has low resolution. So, for a light ray in the ToF framework, there must be multiple corresponding light rays in the RGB framework.  This paper ignores this critical fact. It is highly recommended to examine this multi-resolution fact when using RGB and ToF bi-modality input. \n\n5. As the authors admitted, dark regions indeed have obvious artifacts, which is likely to be enhanced by using some supervised signal enhancement techniques. This can be an interesting task in the future. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It's a nice paper that might interest many people in CV and graphics. The multi-resolution issue should be addressed for better visual quality. Also, the rendering speed should be accelerated. Both might be the main hurdles toward real applications.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed an approach to model geometry and appearance in a dynamic scene based on monocular RGB and ToF measurements. The key novelty is a principled extension of NeRF based on the image formation model of ToF measurements. The paper initially received a mixed rating, with two reviewers rated it below the bar and two above. Two reviewers were satisfied with the rebuttal and upgraded the ratings. While the meta-reviewer agrees that the quality of the results was not impressive, the ToF extension of NeRF for dynamic scenes is a solid technical contribution. The authors are encouraged to incorporate the review feedback in the final manuscript.
Decision: Accept (Poster)