DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
Authors: Jia-Wei Liu
Yan-Pei Cao
Weijia Mao
Wenqiao Zhang
David Junhao Zhang
Jussi Keppo
Ying Shan
Xiaohu Qie
Mike Zheng Shou
Venue: NeurIPS 2022 Accept (2022)
Keywords: dynamic radiance field
volumetric representation
dynamic novel view synthesis.

Abstract: Modeling dynamic scenes is important for many applications such as virtual reality and telepresence. Despite achieving unprecedented fidelity for novel view synthesis in dynamic scenes, existing methods based on Neural Radiance Fields (NeRF) suffer from slow convergence (i.e., model training time measured in days). In this paper, we present DeVRF, a novel representation to accelerate learning dynamic radiance fields. The core of DeVRF is to model both the 3D canonical space and 4D deformation field of a dynamic, non-rigid scene with explicit and discrete voxel-based representations. However, it is quite challenging to train such a representation which has a large number of model parameters, often resulting in overfitting issues. To overcome this challenge, we devise a novel static-to-dynamic learning paradigm together with a new data capture setup that is convenient to deploy in practice. This paradigm unlocks efficient learning of deformable radiance fields via utilizing the 3D volumetric canonical space learnt from multi-view static images to ease the learning of 4D voxel deformation field with only few-view dynamic sequences. To further improve the efficiency of our DeVRF and its synthesized novel view's quality, we conduct thorough explorations and identify a set of strategies. We evaluate DeVRF on both synthetic and real-world dynamic scenes with different types of deformation. Experiments demonstrate that DeVRF achieves two orders of magnitude speedup (**100× faster**) with on-par high-fidelity results compared to the previous state-of-the-art approaches. The code and dataset are released in https://github.com/showlab/DeVRF.

DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
Jia-Wei Wei Liu';Yan Yan-Pei Cao', Weijia Mao', Wenqiao Zhang', David Junhao Zhang', Jussi Keppo Ying Shan', Xiaohu Qie?, Mike Zheng Shou 't
Show Lab, National University of Singapore ARC Lab, Tencent PCG
National University of Singapore Bussness School, National University of Singapore 6 Institute of Operations Research and Analytics, National University of Singapore
Abstract
Modeling dynamic scenes is important for many applications such virtual reality and telepresence. Despite achieving unprecedented fidelity for novel view synthesis in dynamic scenes, existing methods based on Neural Radiance Fields (NeRF) suffer from slow convergence (i.e., model training time measured in days). In this paper, we present De a novel representation to accelerate learning dynamic radiance fields. The core of DeVRF is to model both the 3D canonical space and 4D deformation field of a dynamic, non-rigid scene with explicit and discrete voxel- based representations. However, is quite challenging to train such representation which has a large number of model parameters, often resulting in overfitting issues. To overcome this challenge, we devise novel static -> dynamic learning paradigm together with a new data capture setup that convenient to deploy practice. This paradigm unlocks efficient learning of deformable radiance fields via utilizing the 3D volumetric canonical space learnt from multi-view static images to ease the learning of 4D voxel deformation field with only few-view dynamic sequences. To further improve the efficiency of our DeVRE and its synthesized novel view 's quality, conduct thorough explorations and identify set of strategies. We evaluate DeVRF on both synthetic and real-world dynamic scenes with different types of deformation. Experiments demonstrate that DeVRF achieves orders magnitude speedup (100x faster) with on-par high-fidelity results compared the previous sfate-of-the-art approaches. The code and dataset released in tattps://gitain httphau..oo//hhot.e.eeee www.//hwwwab//e/.......
Introduction
Free-viewpoint photorealistic view synthesis techniques from set of captured images unleash new opportunities for immersive applications such as virtual reality, telepresence, and 3D) animation production. Recent advances in this domain mainly focus static scenes, e.g., Neural Radiance Fields (NeRF) (20), which implicitly represent rigid static scenes using 5D (spatial locations (æ, =) and view directions (0. <p)) neural radiance fields. Although achieving unprecedented fidelity for novel view synthesis. NeRF were mainly exploited under static scenes. To unlock dynamic view synthesis, existing NeRF-based approaches either learn an additional MLP-based deformation field that maps coordinates in dynamic fields to NeRF-based canonical spaces [27] 22 25 38) or model
dynamic scenes as 4D
radiance fields with relatively large MLPs
16 77.
"Work is partially done during internship ARC Lab, Tencent PCG.
1 Conresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a)
(b)
(c)
(d)
a.4
stage
Cansrita point Mélion vector Deformed
stage
V Vdcusity
Hyperteltte
Hypenlenn-22 ttag
NSFF
point
Ou's
color
3.2
Trining Time (Hours)
Figure I: The 3D canonical space (a) and the 4D deformation field (b) of DeVRE for neural modeling of non-rigid scene (c). (d): The comparison between DeVRF and SOTA approaches.
Despite being promising, NeRF is notoriously known for suffering from lengthy optimization time. This issue becomes particularly prominent for non-rigid, dynamic scenes because the aforementioned
NeRF-based approaches require extra computation for the deformation MLPs (27 24 25 38
or
time-varying texture/density querying [6 7
resulting in quite long training time (in "days").
This motivates us to improve the learning efficiency of dynamic radiance fields. Recent advances in show that employing voxel grids, such a volumetric representation, can achieve fast convergence. To adapt for dynamic scenes, one straightforward approach is to incorporate such volumetric representation into the dynamic radiance field for fast neural modeling. In this paper, we present novel deformable voxel radiance field (DcVRF) that models both the 3D canonical space and 4D deformation field of non-rigid, dynamic scene with explicit and discrete voxel-based representations, as illustrated in Fig. [ (a-c). However, we empirically observe that recklessly learning such a representation in dynamic radiance fields tends plunge into the local optimum, Lee, the overfitting issue, due to the large number of parameters in DRVRF.
static NeRF 33 45
To overcome this overfitting issue, we power our DeVRE with two novel designs: (1) We devise an efficient and practical learning paradigm, ie., static - dynamic, for learning deformable radiance fields. The key idea behind this is that the 3D volumetric canonical space learned from multi-view static images can introduce inductive bias [ to unlock efficient learning of deformable radiance fields. Further, with such 3D priors, dynamic scene can effectively modeled with only a few fixed cameras. We argue that such few-fixed-cameras setup for dynamic scene data capture is more convenient than the moving camera (such the setup used in D-NeRF (27) in practice.
(2) Based on the static - dynamic paradigm, we conduct extensive explorations and identify a set of strategies customised for De VRF to improve its efficiency and effectiveness. These include a coarse-te to-fine training strategy for the 4D deformation field to further improve efficiency, and three objectives to encourage our DeVRF to reconstruct dynamic radiance fields with high fidelity: deformation cycle consistency, optical flow supervisions, and total variation regularization.
Fig. [ (d) shows that on five inward facing synthetic scenes, two newwadd-facing real-worle scenes and one inward-facing real-world scene, our approach enables fast dynamic radiance field modeling about 10 minutes on single NVIDIA GeForce RTX3090 GPU. This is 100x faster than SOTA approaches with comparable novel view synthesis quality.
To summarize, the major contributions of our paper are:
A novel perspective of DeVRF is presented that enables fast non-rigid neural scene recon- struction, which achieves an impressive 100x speedup compared to SOTA approaches with on-par high-fidelity.
To the best of knowledge, into dynamic radiance fields.
the first incorporate the 4D voxel deformation field
We devise a static + dynamic learning paradigm that can boost performances with a low-cost yet effective capture setup.
2 Related Work
Novel View Synthesis for Static Scenes. Earlier approaches 14 synthesis by first building an explicit 3D reconstruction static scene, such as voxels and meshes, and then rendering novel views based on the reconstructed model. On the other hand, multi-plane
31 4 6 9 21
tackle novel view

---

images 44 19 represent scene with multiple images at different depths and can reconstruct scenes with complex structures. Recently, NeRF [20] achieves unprecedented fidelity for novel view synthesis by modeling static scenes with neural radiance fields. Subsequent works have extended NeRF to different scenarios, such as few-view novel view synthesis [12], multi-scale representation (2). and larger scenes [36 43 29 However, these methods mainly focus on static scenes while the dynamic radiance fields reconstruction is more practical.
Novel View Synthesis for Dynamic Scenes. In order to capture dynamic scenes with non-rigidly deforming objects, traditional non-rigid reconstruction approaches require depth information as additional input or only reconstruct sparse geometry [23
5
34]. VolumeDeforn [[I]
introduces a unified volumetric representation to encode both the scene's 's geometry and its motion but requires RGB-D sequences as input. Neural Volumes 18 represents dynamic objects with a 3D
voxel grid plus an implicit warp field, but requires an expensive multi-view capture rig and days to train. Recent studies have built upon NeRF [20) and extended it to dynamic neural radiance field reconstruction by learning mapping from dynamic to canonical field [2 [4 25 38 or building
4D spatio-temporal radiance field [2 16 maps coordinates in a dynamic field to a NeRF-based canonical space. Nerfies (24] further associates
[5]. D-NeRF 27 learns a deformation field that
latent codes in the deformation MLP and the canonical NeRF to tackle more challenging scenes such as moving humans. HyperNeRF [25] proposes to model the motion in a higher dimension space, representing the time-dependent radiance field by slicing through the hyperspace. In contrast, radiance fields and addresses motion ambiguity using scene depth. Sharing a similar idea on the 4D apatio-temporal field, NSFF 16 represents a dynamic scene as a time-variant function of geometry and appearance, and warps dynamic scene with 3D scene motion. Lastly, several NeRF-based approaches have been proposed for modeling dynamic humans 88
Video-NeRF 42 models the dynamic scene as 4D apatio-tempon
26 32] but can not directly generalize to other scenes. Although achieving promising results, existing methods require days of GPU training time, which is undesirable in real-world applications.
41
17
NeRF Acceleration. In the light of NeRF's substantial computational requirements for training and rendering. recent papers have proposed methods to improve its efficiency. A line of work 46 (2 [0] focuses NeRF rendering acceleration and has achieved encouraging results. As for training accelcration, DVGO 351 models the radiance field with explicit and discretized volume representations, reducing training time to minutes. Plenoxels [45] employs sparse voxel grids the scene representation and uses spherical harmonics to model view-dependent appearance, reaching a similar training speedup. Finally, Instant-ngp 22 proposes multiresolution hash encoding; together with a highly optimized GPU implementation, it can produce competitive results after seconds of training. However, existing acceleration methods only focus on static scenes, while hardly any research, to our best knowledge, has studied NeRF acceleration for dynamic scenes. Very recently, Fourier PlenOctrees [39] extends PlenOctrees [46 to dynamic scenes by processing mvrriig density and color in the frequency domain: however, the data capturing setup is expensive, and it still requires hours of training. Instead, our proposed algorithm. DRV,F, offers a superior training speed while only requires a few cameras for data capture.
3 Method
3.1
Capture Setup
Deformable scenes undergo various types of deformations and motions, which can result in different scene properties such as object poses, shapes, and occlusions. Therefore, capturing and modeling deformable scenes is nontrivial even for professional photographic studios. Existing approaches attempt to capture 360 inward-facing dynamic scenes with multi-view sequences
33
1 49
and thus require dozens of high-quality cameras. On the other hand, D-NeRt (27] reconstructs deformable radiance fields from a sparse set of synthetic images rendered from a moving monocular camera. However, in practice, it is particularly challenging to capture real-world 360 snward-facing dynamic scenes with a single moving camera due to various types of deformations and resulting occlusions in dynamic scenes, especially for scenes undergoing fast deformations. As a result, only capture forward-facing videos of real-world dynamic scenes with monocular camera.
subsequent studies [24 25 16 7
3

---

Table 1: Comparisons of capture setups for dynamic scenes.
Approach
No. of cameras Cost
Supported real-world use cases
D-NeRF 27,, Nerhes 24
Monocular
Low Forward-facing scenes, slow reconstruction days.
Neural Volumes 18
Multiple (34)
High 360 inward-facing scenes, slow reconstruction in days.
Fourier PlenOctrees 39
Multiple (60)
High
360" seward-fccing scenes, fast reconstruction in 2hrs.
360 inward-facing and lorward-facing scenes, super-fast reconstruction in 10mins.
Ours
Few (4)
Low
(a)
(b)
(e)
V deasiiy
Train
Density / RGB color
Ray Distance
V color
Volume Rendering
Novel view
Canonical space
Multi-view static images (Is)
Forward Voxels Deformation
[X] X
AX 10
4D Motion Cycle Consistency
(c)
(X)
pook Conslstenor
Sampled Ray points
Motion vedor
Start step
Timestep t
End step
Few-view dynamic sequences (Ip)
deformation field
Backward Voxels Deformation
Figure 2: Overview of our method. In the first stage, DeVRF learns 3D volumetric canonical prior (b) from multi-view static images (a). In the second stage, 4D deformation field (d) is jointly optimized from taking few-view dynamic sequences (c) and the 3D canonical prior (b). For ray points sampled from a deformed frame, their deformation to canonical space can be efficiently queried from the 4D backward deformation field (d). Therefore, the scene properties (i.e., density, color) of these deformed points can be obtained through linear interpolation in the 3D volumetric canonical space, and novel views (f) can be accordingly synthesized by volume rendering (e) using these deformed sample points.
Compared to dynamic scenes, it is much casier in practice to do multi-view capture for real- world static scenes with a monocular moving camera. Therefore, we propose to separate the capture process of dynamic scene into two stages: the first stage captures a static state using moving monocular camera, and the second stage captures the scene motion using a few fixed cameras. this capture setup, the multi-view static images provide complete 3D geometry and appearance information of the scene, while few-view dynamic sequences show how the scene deforms 3D space over time; the entire capture process only requires few cameras. Tab. compares our capture process with existing approaches in terms of the number of required cameras, cost, and supported real-world use cases.
3.2 Deformable Voxel Radiance Fields
As illustrated in Fig. 2 present DeVRF to model both the 3D canonical space and 4D deformation field a non-rigid scene with explicit and discrete voxel-based representations. This volumetric representation allows to efficiently query the deformation, density, and color of any 3D point at any time step in a deformable scene, thus largely improving the training and rendering efficiency. In addition, devise static - dynamic learning paradigm that first learns 3D volumetric canonical prior from multi-view static images (Fig. aa) and transfers such prior to dynamic radiance fields reconstruction (Fig. c-)-
3D Volumetric Canonical Space. take inspiration from the volumetric representation of DVGO 35 and model the scene properties such density and color of our 3D canonical space into voxel

---

grids. Such representation enables us to efficiently query the scene property of any 3D point via trilinear interpolation of its neighboring voxels,
Tri-Interp (Iz 2] Vp): (R", RC RCNN..,,....) + RC, vpp E {density, color} (I)
where C' is the dimension of scene property Vp. Na. Ny. and N₂ are the voxel resolutions of Vp in Ty, yr, dimension.
As shown in Fig. 2[a-b), we learn the 3D volumetric canonical prior, i.e., density grid V density and color grid V 'oolor. with multi-view static images {Is} via volumetric rendering. Following DVGO [3]. we employ softplus and post-activation after the trilinear interpolation of 3D point in V deusity: as they are critical for sharp boundary and high-freque free-fre geometry reconstruction. We also apply shallow MLP after the trilinear interpolation of 3D point in V color to enable view-dependent color effects 35 In our static - dynamic learning paradigm, the learned 3D volumetric canonical prior provides critical knowledge of the 3D geometry and appearance of the target dynamic scene, as few-view dynamic sequences alone struggle to reconstruct complete deformable radiance field with high fidelity (as shown in Section 4
4D Voxel Deformation Field. We employ a 4D voxel deformation field V motion to efficiently represent the motion of deformable scene. As shown in Fig. d), the arrow directions represent the motions of voxels, the color denotes the motion direction, and the arrow magnitude denotes the motion scale. To synthesize a novel view time step we shoot rays through image pixels and sample ray points Xg {X}] in 3D space. The 3D motion AX;
{AX; 2g- from X₂ to their
corresponding 3D points in the canonical space Xa = (X X = X + AX! can efficiently
queried through quadruple interpolation of their neighboring voxels at neighboring time steps in the 4D backward deformation field,
Quad- Interpntep (Iz,y, t), V Mootion): (R*, R. R.MCCN,KN,,,,
R
(2)
where C' is the degrees of freedom (DoFs) of the sample point motion. We use C in this paper, ie., assign a displacement vector to each sample point. N₂ is the number of key time steps that can be user-defined based on the scene motion properties.
Therefore, scene properties of X, can then be obtained by querying the scene properties of their corresponding canonical points Xo through trilinear interpolation in the volumetric canonical space. Finally, pixel colors can be calculated through volume rendering with the sampled scene properties along each ray, as illustrated in Fig. [[c-f).
3.3 Optimization
Training the DeVRF is quite challenging, mainly because a large number of model parameters may lead to overfitting suboptimal solutions. This section describes the training strategy and optimization losses that we design to facilitate fast optimization of the DeVRE.
Caars-tt-Fiine Optimization. For dense 4D) voxel deformation field with N₂ x x N, N, N resolution, there could millions free parameters, which are prone to overfitting and suboptimal solutions. To solve this problem, we employ coarse-to-fine fine training strategy. Specifically, in our experiments, we progressively up-scale the r-y-z resolution of the 4D voxel deformation field from 10 10 X 10 to 160 160 160. With this strategy, the 4D voxel deformation field first learns a rough motion at the coarse stage, which is thereafter progressively refined in finer stages. Our experiments demonstrate that the coarse-to-fine strategy can effectively smoothen the optimization landscape of the 4D voxel deformation field and remove most suboptimal solutions, thus largely improving the training efficiency and accuracy.
Re-rendering Loss. With sampled properties at X,, the color of a pixel can be calculated through volume rendering, i.e., by integrating the density and color of X4 along a ray 20
N₂ (r)
Σ
exp ex TN.+I T, = exp
- Σ 05
(3)
where N, is the number of sampled deformed points along the ray, T; represents the probability of light transmitting through ray to the i-th sampled point, and exp (-a,6,) ) is the probability
5

---

that light terminates at the i-th point. ó; is the distance between adjacent sampled points, and o; ca denote the density and color of deformed point i, respectively. Cbg is the pre-defined background color.
Given the few-view training dynamic sequences with calibrated poses {ID), DeVRF is optimized by minimizing the photometric MSE loss between the observed pixels color C' (r) and the rendered pixels color C' (r):
1 RI
Σ
c
LRender
PER
where R is the set of rays in a mini-batch.
4D Deformation Cycle Consistency. As illustrated in Fig. dd) the forward deformation is modeled as 4D forward deformation voxels with the same resolution as the 4D backward deformation voxels, and we enforce 4D deformation cycle consistency between backward and forward motion, which regularizes the learned deformation field. the 4D deformation cycle, backward motion vectors A models the motion from X₂ to Xo; in contrast, forward motion vectors AX models the motion from Xo to their corresponding 3D points the dynamic space X; = {X} X
X AX?-"). The 4D motion cycle consistency can now be realized by minimizing the following cycle consistency loss LCyele (t),
N. Σ
Cyyele (1)
x
X!
(5)
2N
where N₂ is the number sampled 3D points in mini-batch.
Optical Flow Supervision. The DeVRF is indirectly supervised by 2D optical flows estimated from consecutive frames of each dynamic sequence using pre-trained RAFT model [37. For X, and their corresponding Ng. we first compute the corresponding 3D points of Xo at ( 1 time step via forward motion _ = (X: - = X + AX After that, we project & onto the reference camera and get their pixel locations P (P(-1,, and compute the induced optical flow with respect to the pixel location P. = (P?) from which the rays of X, are cast. We enforce the induced flow to be the same the estimated flow by minimizing Lroow (!),
1 R
(P P fp:
CFlow (1)
Wrs
(6)
PER
1=1
where Wr,s = -1 is the ray termination weights from Eq. 33. and fpr is the estimated 2D backward optical flow pixel P,,--
Total Variation Regularization. We additionally employ a total variation prior 30 when training the 4D voxel deformation field to enforce the motion smoothness between neighboring voxels. At time step I
CTV (t)
(v. d) 4 (v. (t), d) A (v, (t). d))
2N
dEC
where A A2, the squared difference of motion vectors between voxel v, and its neighbors along Jr V. axes. N N₂ x N₂ N₂ denotes the number of voxels.
Training Objective. The overall training objective of DeVRF is the combination of per-pixel re- rendering loss CRenders cycle consistency loss Loycles optical flow loss LFlow, and total variation regularization LrV:
- WRender CRender + WCycle
LCyele + WFlow LFlow + WTV LTV,
(8)
where WRender; WCycles WFlow; WTV weights for corresponding losses.

---

Table 2: Averaged quantitative evaluation inward-facing synthetic and real-world scenes against baselines and ablations of our method. We color code each cell as third best
best second best and
SYNTHETIC INWARD-FACING
REAL-WORLD INWARD-FACISS
PSNR1
SSIM; LPIPS] GPU (GB)L
Time!
PSNR
SSIM: LPIPS, GPU (GBy)
Timel
Neural Volumes
9.620
0.532 0,3520
19.1
22.41ur
17.29
0.608 0.310
22.0hus
D-NeRP
31.83
0.960
0.0355
10.0
18.4hrs
29.15
0.946
0.0643
12.4
22.
D-NeRF
stage
28.29
0.945
0.0528
18. 11ur
27.21
0.936
0.0706
22. 22.2hrs
D-NeRF
27
dynamic
17.39
0.839
0,2038
9.8
21.9hrs
21.74
0.911
0.0996
13.5
22 3lrs
Nerfics
0.0132
21.8
18.7
29.58
0.990
0.0576
22.5
19.1hrs
Nerlies
24
stage
32.37
0.991
0,0322
22.0
15.8lrrs
23.93
0.920
0.0878
22.0
19.7hus
Nerfies
dynamic
19.45
0.794
0.1674
22.0
21.3hrs
20.70
0.910
0.1080
22.0
19.6hrs
HyperNeRF
25
33.73
0.965
0.0335
22.5
20.5hrs
28.50
0.944 0.0692
22.0
20.5hrs
NeRP
stage
29.16
0.0555
22.5
19.2hrs
26.53
0.935 0.0802
22.0
19.3hrs
NyRerNeRF
25
-dynmmic
18.00
0.786 0.2173
22.4 4
20.
10:39
0.734 0.3990
22.0
20.5hrs
NSFF 16
27.06 0.936 0,0800
21.4
12.8hrs
28.11
0.939 0.0711
22.7 7
15.3hrs
NSFF
16 dynamic
18.18 0.858 1929
15.0
15.5hrs
19.90
0.909 0.0944
22.7
16.2hrs
Ours (base)
22.44 0.887 0.1173
4.4
Smins
24.56
0.917 0.0844
6.6
10mins
Ours w/ c2f
31.97 0.975 0.0185
4.4
7mins
27.83
0.956 0.0465
6.6
10mins
Ours c2F, tv
32.73 0.963
0.01 72
4.4
7mius
29.35
0.959 0.0434
6.6
10mins
Ours w/ c2f, cycle
33.97
0.981 0.0142
4.4
Smins
31.56
0.971 0.0292
6.6
Imins
Ours w/ c2f, tv, cycle, flow
34.29
0.962 0.0137
4.4
Smins
31 31.68
0.972 0.0289
6.6
11mins
4 Experiments
We extensively evaluate the DcVRF on various types of datasets, including five synthetic 3 360
inward-facing dynamic scenes, two real-world forward-facing dynamic scenes, and one world 360 inward-facing dynamic scene. We run all experiments on a single NVIDIA GeForce RTX3090 GPU. During training, we set Whtender scenes.
1, WCycle 100, WFlow 0.005, and WrV = for all
4.1 Comparisons with SOTA Approaches
To demonstrate the performance of DeVRE, compare DeVRF to various types of SOTA approaches, including volumetric method Neural Volumes [18], NeRF-based methods D-NeRF (27]. Nerfies 224. HyperNeRF [25), and a time-modulated method NSFF [16.. For a fair comparison, since DeVRF follows a static + dynamic learning paradigm, we additionally implement 2-stage versions of D-NeRF, Nerfies, and HyperNeRF learn canonical space prior in the first stage and then optimize a deformation network in the second stage. To show the effectiveness of our low-cost capture strategy for dynamic scenes, we also train these baselines using only few-view dynamic sequences and observe a significant performance drop compared to those trained with both static and dynamic data. For quantitative comparison, peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) [40], and Learned Perceptual Image Patch Similarity (LPIPS) 47 with VGG 33 are employed as evaluation metries
Evaluation on inward-facing synthetic and real-world deformable scenes. We selected five synthetic dynamic scenes with various types of deformations and motions, and rendered synthetic images in 400 x 400 pixels under the 360 inward-facing setup. For each scene, we use 100-view static images and 4-view dynamic sequences with 50 frames (i.e., time steps) as training data for all approaches, and randomly select another 2 views at each time step for test. In addition, we collected one 360 seward-facine
real-worth deformable scene in 540 x 960 pixels. With our data capture setup, only 4 cameras are required to capture dynamic scenes, and we choose 3 views of them as training data and the other view as test data.
We report the metrics of the real-world scene as well as the average metrics of five synthetic scenes for all approaches in Tab. 2 and leave the per-scene metries supplementary material. As shown in Tab. 2 for synthetic and real-world scenes, VRF achieves the best performance in terms of PSNR and LIPP, and the second- or third-best in terms of SSIM among all approaches. Most importantly, our per-scene optimization only takes less than 10mins with 4.4GB to 6.6GB GPU memory on
'The Lego scene is shared by NeRF 20 licensed under the Creative Commons Attribution 3.0 License: https://ccaaivecommnnn.o.op wtw.mmmmon...ggliinnees///////..
Other scenes are purchased from TurboSquid, licensed under the www.sspiiiooooooouuuuuuuuuuuuu
TurboSquid 3D Model License: https:///llo..d http:://blo..uurbbsquiddd wow/turbosquid 3d-model license/.
AAthhoggh improving test-time rendering speed is not the focus of our paper. DeVRF achieves 16x ~ 32x test-time rendering speedup compared with other approaches, averaging 0.875 seconds per 540 960 image.
7

---

For the red box region of each scane, show its zoom-in at the bottom-right of each picture
Lego
Floating robot
Daisy
Kuka
Glove
Plant
Rabbit
Flower-360*
Groundtruth
Ours
D-NeRF
Nerfies
HyperNeRF
NSFF
Figure 3: Qualitative comparisons of baselines and DeVRF on synthetic and real-world scenes.
single NVIDIA GeForce RTX3090 GPU, which is about two orders of magnitude faster than other approaches. The above quantitative comparison demonstrates the efficiency and effectiveness of DeVRF. Besides, the qualitative results of DeVRF and baselines on synthetic and real-world scenes are illustrated in Fig. where DeVRF achieves on-par high-fidelity in comparison to SOTA methods. Please see supplementary video for more results.
For fair comparison, we additionally report the results of the 2-stage versions for D-NeRF 27 Nerfies (24). and HyperNeRF (25) in Tab. [ Since these approaches are not designed separately learn a canonical space and deformation field, there is no significant difference in the results between their 2-stage versions and I-stage versions using the training dataset. Furthermore, report the results of these baselines trained with dynamic data only (baseline-dynantle in Tab. 2 Their performances drop significantly compared to the results trained with both static and dynamic data. In addition, since Neural Volumes [18) requires dozens of dynamic sequences as input, its performance poor with our few-view dynamic sequences. The experimental results not only show the effectiveness of our low-cost data capture process and the proposed DeVRF model; but also validate our observation that few-view dynamic sequences alone fail to provide complete information about the dynamic scene, while static multi-view data can favorably serve supplement.
also

---

Table 3: Quantitative evaluation forward-facing real-world scenes against baselines and ablations
of our system. We color code each cell best second best and third best
PLANT
RABBIT
PSNR1
SSIM:
LPIPS)
GPU (GB)L
Timel
PSNRT
SSIMI
LPIPS,
GPU
Timel
D-NcKF
22
31.94
0.379
0.0251
11.4
21.3brs
33.51
0.974
0,0384
11.4
21.7hrs
Nerfies 24]
31.36
10:901
0.0309
21.9
18.3hrs
24.83
0.952
0.0795
21.9
19.Shrs
HyPerNeRR
25
32.08
0.978
0.0331
22.0
20.0hrs
24.97
0.912
0.0849
22.0
20.7hrs
NSFF 16
14.3ura
0.0851
14.5hrs
Ours (buse)
26.13
0.346
0.0722
8.1
Sudns
25.58
0.910
0.1300
8.9
binins
31.85
26.79
6mins
Ours w/ c2f, tv
31.89
0.980
0.0263
8.1
Smins
29.28
0.951
0.0655
8.9
6mins
Ours c2f, 1v, cycle
31.99
0.981
0.0235
9unins
31,05
0.963 0.0343
7mins
Ours w/ c2f, cycle, flow
32.01
0.981
0.0236
8.1
10mins
32.05
0.966 0.0492
8.9
7mins
Evaluation on forward-facing real-world deformable scenes. We collected two forward-facing real-world deformable scenes in 540 960 pixels using 4 cameras, and chose views of them as training data and the other view as test data. To handle forward-facing scenes, we adapt DRV use normalized device coordinates (NDC) and multi-plane images (MPI) in DVGO [35 As shown in Tab. 3 DeVRF achieves the best result in the plant scene in terms of LPIPS metric and the second-best result in the rabbit scene terms of all metrics. Fig. also demonstrates qualitative comparisons on these two scenes.
4.2 Ablation Study
Ablation study of DeVRF components. We carry out ablation studies on both synthetic and real- world scenes to evaluate the effectiveness of each proposed component in DeVRF. We progressively ablate each component from optical flow, cycle consistency, total variation, to coarse-to- strategy. As shown in Tab. 2 and 3 the performance of DeVRF progressively drops with the disabling of each component, where disabling the coarse-to-fine training strategy causes the most significant performance drop. This is expected since the coarse-to to-fine training strategy is critical to reducing local minimums during optimization.
For the red box region of each scene, we show its zoom-in at the bottom-right of each picture
Lego
Floating robot
Daisy
Kuka
Glove
Groundtruth
View
2 Views
3 Views
4 Views
5 Views
6 Views
Figure 4: Qualitative results of DeVRF with different numbers of dynamic training views.

---

Ablation study of dynamic training views on synthetic dataset. Our static - dynamic learning paradigm is based on a low-cost yet effective capture setup: the multi-view static images provide complete 3D geometry and appearance information of the scene, while few-view dynamic sequences show how the scene deforms in 3D space over time. The entire capture process only requires a few cameras that are convenient to deploy in practice. To evaluate the influence of the number of dynamic views, we conduct additional ablations for DeVRE on five inward-facing synthetic dynamic scenes and report the per-scene metrics as well the average metrics with respect to the number of dynamic training views. As shown Fig. S given the same multi-view static images, the performance of DeVRF largely improves with the increment of dynamic training views and almost saturates six dynamic views, and the four dynamic training views used our paper can yield comparable results compared six dynamic views. We additionally visualize the qualitative results of DeVRF with different numbers of dynamic training views in Fig. 4 Therefore, in our static - dynamic learning paradigm, with static multi-view data supplement, only few (e.g., four) dynamic views required to significantly boost the performance of dynamic neural radiance fields reconstruction. This further demonstrates the effectiveness of our low-cost data capture process and the DeVRF model.
(a) 40
(b)
Lego
0.18
Floating rebot
35
0.16
Daisy
PSNR 17)
Kuka
30
0.12
Avarage
Lego
0.10
25
Floating robot
0.08
Daisy
0.05
Glove
20
Kuka
0.04
Average
0.02
0.00
10
Dynamic Training Views
Dynamic Training Views
Figure 5: Ablation evaluation on the number of dynamic training views: (a) PSNR, (b) LPIPS.
Conclusion
We introduced DeVRF, a novel approach to tackle the challenging task of fast non-rigid radiance field reconstruction by modeling both the 3D canonical space and 4D deformation field of dynamic scene with voxel-based representations. The DeVRF can be efficiently optimized in two major steps. We first proposed a static - dynamic learning paradigm pinpoint that the 3D volumetric canonical prior can be effectively transferred into the 4D voxel deformation field. Second, based on this learning paradigm, we developed a series of optimization strategies, including coarse-to-fine learning. deformation cycle consistency, optical flow supervisions, and total variation priors. Such DeVRF finally produced a 100x faster training efficiency with on-par high-fidelity results in comparison to SOTA approaches. We believe our DeVRY can provide a complement to existing literature and new insights into the view synthesis community.
Limitations and Future Work. Although DeVRF achieves fast deformable radiance field recon- struction, the model size is large due its large number of parameters. In addition, DeVRF currently does not synchronously optimize the 3D canonical space prior during the second stage, and thus may not be able model drastic deformations. We consider these limitations faithful future directions.
6 Acknowledgements
This project is supported by the National Research Foundation, Singapore under its NRFF Award SNNNFFI1-2021-000 Mike Zheng Shou's Start-Up Grant from NUS, and National Research Foundation, Singapore and A *STARR, under its RIE2020 Industry Alignment Fund Industry Col- laboration Projects (IAF-ICP) grant call (Grant No. 12001E0059). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore. Jia-Wei Liu is supported by NUS IDS-ISEP scholarship.
10

---

References
[1] Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan. 4d visualization of dynamic events from unconstrained multi-view videos. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5366-5375, 2020.
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Bartin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2021.
[3] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198, 2000.
[4] Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Un- structured lumigraph rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 425-432, 2001.
[5] Alvaro Collet, Ming Chuang, Pat Sweency, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoint video. ACM Transactions Graphics (ToG), 34(4):1-13, 2015.
[6] Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 11-20, 1996.
[7] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712-5721, 2021.
[8] Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. arXiv preprint 12.0542, 2021.
[9] Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. Scalable inside-out image-based rendering. ACM Trans. Graph., 35(6), nov 2016.
[10] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2021.
[11] Matthias Innmann, Michael Zollhöfer, Mathhis Nießner, Christian Theobalt, and Marc Stam- minger. Volumedeform: Real-time volumetric non-rigid reconstruction. In European Conference on Computer Vision, pages 362-379. Springer, 2016.
[12] Ajay Jain, Matthew Tancik, and Picter Abbeel. Putting nerf diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885-5894, 2021.
[13] Hanqing Jiang, Haomin Liu, Ping Tan, Guofeng Zhang, and Hujun Bao. 3d reconstruction of dynamic scenes with multiple handheld cameras. In European Conference on Computer Vision, pages 601-615. Springer, 2012.
[14] Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International journal of computer vision, 3893):19--218, 2000.
[15] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video. arXiv preprint arXiv:2103.02597 2022.
[16] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6508, 2021.

---

[17] Lingjie Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Transactions on Graphics (TOG), 4006))1-166 2021.
[18] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv: 1906.0770 2019.
[19] Ben Mildenhall, Pratul Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[20] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-48 421. Springer. 2020.
[21] Xin Min. Wenqiao Zhang, Shouqian Sun, Nan Zhao, Siliang Tang, and Yueting Zhuang. Vpmodel: High-fidelit product simulation in a virtual- virtual-physical environment. IEEE transactions on visualization and computer graphics. 25 1):3083-3093, 2019.
[22] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 11:4:110: July 2022.
[23] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 343-352, 2015.
[24] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[25] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Brruala,, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021.
[26] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 14314- 14323, 2021.
[27] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[28] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345. 2021.
[29] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasac- chi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. arXiv preprint arXiv:2111.14643, 2021.
[30] Leonid Rudin and Stanley Osher. Total variation based image restoration with free local constraints. Proceedings of 1st international conference on image processing, volume pages 31-35. IEEE, 1994.
[31] Steven Seitz. and Charles Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35(2):151-173, 1999.
12

---

[32] Ruizhi Shao, Hongwen Zhang, He Zhang, Mingjia Chen, Yanpei Cao, Tao Yu, and Yebin Liu. Doublefield: Bridging the neural surface and radiance fields for high-fidelity human reconstruction and rendering. In CVPR, 2022.
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14099 1556, 2014.
[34] Haoxuan Song, Jiahui Huang, Yan Cao, and Tai-Jiang Mu. Hdr-net-fusions Real-time 3d dynamic scene reconstruction with hierarchical deep reinforcement network. Computational Visual Media, 7(4):419 119 435, 2021.
[35] Cheng Sun, Min Sun, and Hwann- Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. arXiv preprint 1rXi::2111.112155 2021.
[36] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. arXiv preprint 2iXii:2202.0523: 2022.
[37] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pages 402-419. Springer, 2020.
[38] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[39] Liao Wang, Jiakai Zhang. Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Lan Xu, and Jingyi Yu. Fourier plenoctrees for dynamic radiance field rendering in real-time. arXiv preprint arXiv:2202. 08614, 2022.
[40] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. (3()))600- 612, 2004.
[41] Chung-Yr Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan T Barron, and Kemelmacher- Shlizerman. Humannerf: Free-viewe viewpiitn rendering of moving people from monocular video. arXiv preprint arXiv:2201 04112,, 2022.
[42] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- viewpiitn video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021.
[43] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Citynerf: Building nerf at city scale. arXiv preprint 12..5221, 2021.
[44] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from monocular camera. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 5336-5345, 2020.
[45] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131, 2021.
[46] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
[47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
13

---

[48] Tinghui Zhou, Richard Tucker, John Flynn. Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.0968 09817. 2018.
[49] C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. High-quality video view interpolation using a layered representation. ACM transactions on graphics (TOG), 23(3):600-60) 2004.
14


Reviews:
[{"review_id": "FQOQIChhZfS", "reviewer": "Reviewer_yW6h", "summary": "Authors propose an approach for novel view synthesis and reconstruction of dynamic scenes using a NeRF like model. The main idea is to split the capture/training phase between static and dynamic. Objects are first captured in a static setting using a single moving camera to build a canonical space. The dynamic performance is handled relying on multiple views rather than a single camera. A motion field is then learned to handle the dynamic of the scenes. Finally, temporal consistency enforces smooths results.", "questions": "I'd like authors to mostly comment on the weaknesses above. I think the paper has overall some interesting ideas that could be relevant for the community.", "limitations": "Some limitations are described, I would ask author to elaborate more on the actual capture requirements too. For instance, the dynamic part seems to need a multi-view setup, which limits the application in practice.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ Relevant problem, very active research topic in the community. Related work seems adequate.\n\n++ Clarity. The paper is well described and reproducibility should not be a problem.\n\n+/- Contributions: while the idea is definitely interesting, it is not clear the actual improvement over HyperNeRF (or even Nerfies). The results are marginally better and not clear why one should rely on this method instead of HyperNerf for instance. Nevertheless I find this idea of static and dynamic captures is interesting and I would suggest authors to add more results to support the claims. For instance, I would try the method on multi-view human datasets, where a single, multi-view frame could be used to learn the canonical space and then the rest of the sequence should learn a motion field. This could be particularly relevant for performance capture/avatars.\n\n-- Experiments and Results. Authors should show more real-world examples. In particular results with less views for the dynamic part. I appreciated the quantitative results in the supplementary material, but would be good to show qualitatively the effect of using less views.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7tjZTPD4JME", "reviewer": "Reviewer_7cQ7", "summary": "Modeling geometry and motions for a dynamic scene is an important task in computer vision. This paper proposes a volumetric representation to speed up learning dynamic radiance fields. It employs a 4D voxel grid to model deformations while continuing to use a 3D voxel grid to model the canonical space. To save optimization time and provide well initialization, this paper devises a 2-stage capturing procedure – the 1st stage is to capture a static state with a moving monocular camera, and the 2nd stage is to capture dynamic states with a few fixed cameras. Accordingly, the captured images in the 1st stage are utilized for training the canonical space, while the captured images in the 2nd stage are utilized for training the deformation field.", "questions": "1, The design of the backward deformation is well-written, but the process of obtaining the forward motion is not clearly presented. Could you explain how to query the corresponding points of a canonical 3D point at different time steps?\n2, Could you verify that the proposed DeVRF can still receive a well rendering performance without the 2-stages training strategy, like evaluating DeVRF with optical flow, cycle consistency, total variation, as well as a general training process? And compare it with the mainstream frameworks (D-NeRF, Nerfies and HyperNeRF).\n3, If the users are able to employ a few (4) fixed cameras in capturing dynamic states, why do not use the binocular camera for recovering depth and then leverage a fusion strategy to model the deformation (a real-time procedure)?\n4, Could you verify DeVRF in solving some challenging cases, like fast motions or highly variable poses?", "limitations": "The authors have discussed the limitations in the end.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper is well-written, including clarifying the motivation and introducing the experiment setup.\n2.\tThe diagrammatic drawing (Fig. 1 and Fig. 2) is technical and clearly describes the claimed contributions.\n\nWeakness\n1.\tThe setup of capturing strategy is complicated and is not easy for applications in real life. To initialize the canonical space, the first stage is to capture the static state using a moving camera. Then to model motions, the second stage is to capture dynamic states using a few (4) fixed cameras. Such a 2-stage capturing is not straightforward.\n2.\tTo utilize a volumetric representation in the deformation field is not a novel idea. In the real-time dynamic reconstruction task, VolumeDeform [1] has proposed volumetric grids to encode both the geometry and motion, respectively.\n3.\tThe quantitative experiments (Tab. 2 and Tab. 3) show that the fidelity of rendered results highly depends on the 2-stage training strategy. In a general capturing case, other methods can obtain more accurate rendered images. Oppositely, Tab. 2 shows that it is not easy to fuse the designed 2-stage training strategy into current mainstream frameworks, such as D-NeRF, Nerfies and HyperNeRF. It verifies that the 2-stages training strategy is not a general design for dynamic NeRF.\n\n[1] Innmann, Matthias, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. \"Volumedeform: Real-time volumetric non-rigid reconstruction.\" In European conference on computer vision (ECCV), pp. 362-379. Springer, Cham, 2016.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-rJR0llg_TS", "reviewer": "Reviewer_7Cm1", "summary": "The paper proposed a novel view synthesis method for dynamic scenes using a deformable voxel formation. They show that using this representation is hard to train as there are large number of parameters. Sp device a static to dynamic paradigm to train faster resulting in 100x speed over previous dynamic Radiance fields.  ", "questions": "why was such a small dataset being proposed? It is easy to capture a larger data from the current configuration.\n\n", "limitations": "Limitations have been well addressed. It would be great if the authors provided negative societal impact for dynamic radiance fields. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written and easy to follow with key contributions well formulated and experimented.\n- The dataset being used can be helpful in quickly starting the research in the dynamic reconstruction paradigm as it is low cost.\n- Learning Deformable voxel radiance field helps understand dynamic scenes accurately and faster.\n- Using the Static to dynamic learning paradigm is helpful in general for dynamic scene understanding and the coarse to fine optimization is useful in dynamic scenes.\n- The ablation study shows the advantage of using different components in the optimization like optical flow to the overall accuracy of dynamic reconstruction\n\nWeaknesses:\n- The dataset captured in the real world is too small for accurate evaluation. Since the method is dynamic different motions of the same objects can be added to the dataset to make more evaluations and would make the method stronger.\n- Fig 3 shows comparison to different methods but the synthetic examples seems to be very similar to other baselines. It is hard to understand the advantage of the current framework over such examples. Seems like the method performs well on the real videos. May be showing more such examples will be helpful.\n- The method is proposed to be 100x faster, this contribution is mainly because of sparser input data with good flow. Doing more evaluations on what number of inputs is actually sufficient and are needed is will be an interesting insight.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept