NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos
Authors: Jinxi Li
Ziyang Song
Bo Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Physics Learning
Velocity Field
Dynamic Radiance Field
Future Frame Extrapolation
TL;DR: A novel representation of dynamic 3D scenes by disentangling physical velocities from geometry and appearance --> future frame extrapolation, semantic scene decomposition, velocity transfer

Abstract: In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively  train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition.

NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos
Jinxi Li
Ziyang Song Bo Yang
vLAR Group, The Hong Kong Polytechnic University
jinxi Sixxi.11@connettppo. ziyang. song@connect. -ow.mmmeeetttpp..ttt....... bo.yangopolyu.edd edu.
Abstract
In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultameously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition. Our code and data available https///gihuub.oo/LLARR -roopp///Vr
Introduction
The 3D world around us is constantly changing over time, where objects are falling, vehicles moving, and clocks ticking. Humans can effortlessly learn the geometry and physical properties of such dynamic 3D scenes, and further predict their future motions following the learned physics rules, just by watching the things for few seconds. Giving machines such ability automatically infer the geometry and physics of complex dynamic 3D scenes essential for many cutting-edge applications in augmented reality, games, and the movie industry. Recent advances in the emerging area of neural radiance field [37] and its succeeding variants [46, 4, 6, 19, 75, 40] have shown excellent results in modeling dynamic 3D scenes such deformable things [4, 58] and articulated objects [39, 55]. While showing superior performance in interpolating novel views within the observed time period, almost all these methods tend to fit training image sequences, without explicitly learning the physical properties such as object velocities, thus being unable to extrapolate and predict future motion patterns of 3D scenes.
More recently, a few studies [47, 14, 8, 32, 2] start to integrate physics priors into implicit neural representations to model dynamic 3D scenes such floating smoke or simple moving objects. By introducing the governing PDEs, a.k.a,, PINN [48], these methods demonstrate promising reconstruction of 3D scene geometry, appearance, velocity and/or viscosity fields. However, the learned physical properties are either tightly coupled with the target objects [14] in the scene or require additional foreground segmentation masks in the loop [32]. This means that the estimated physics knowledge from dynamic video frames is not clearly disentangled, thereby not transferable from one scene to another.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Joint Keyframe and Interframe Optimization
Keyfrume Dynamic Radiance Field
Interframe Velocity Field
Figure 1: The three major components of our framework: Keyframe Dynamic Radiance Field, Interframe Velocity Field, and the Joint Keyframe and Interframe Optimization module.
In this regard, ask an ambitious question: Can we learn disentangled physical properties alongside recovering geomelry and appearance of dynamic 3D scenes purely from mulii-view video videoframee frames? Among various physical properties in the scene, we choose focus on velocity primarily governs scene movement dynamics. Such disentangled velocity fields, once successfully learned, are expected unlock multiple desirable applications: 1) Future frame extrapolation and prediction beyond the observed time period in training. For example, after watching a football flying in the penalty area, we can predict what will happen next. 2) Dynamics transformation from one scene another. For instance, after watching bird flipping wings, we can imagine the same physical behavior on the body of an airplane. 3) Semantic decomposition of 3D scenes. Intuitively, once the velocity field of an entire dynamic 3D scene is learned, all individual objects or parts undergoing different moving patterns can be easily segmented, without needing any extra human annotations for training.
However, accurately learning the physical velocity of a whole 3D scene space is particularly chal- lenging, primarily due to the lack of ground truth 3D velocity annotations, the unknown object types or materials in the scene, and the sparse yet long-time visual trajectory in training. In addition, when separately learning the velocity field, is usually an under-constrained problem even with the commonly used PINN technique [14].
To tackle these challenges, introduce
general framework to simultaneously model the geometry, appearance, and disentangled velocity of a dynamic 3D scene only from multi-view video frames. In particular, shown in Figure our framework consists of three major components: 1) keyframe dynamic radiance field to learn time-dependent volume density and appearance for every 3D point in space; 2) an interframe velocity field to learn time-dependent 3D velocity for every point as well; and 3) a joint keyframe and interframe optimization method together with physics informed constraints to train both networks. For the first component, is flexible to adopt any of the existing time-dependent NeRF architectures such as HexPlane [6] or K-Planes [19]. For the second component, the neural network actually can be as simple MLPs.
The core of our framework is the third component, where we explicitly apply three types of loss functions to jointly optimize both networks: 1) the keyframe photometric loss, 2) the interframe photometric loss, and 3) the governing PDE losses. With these losses, our framework can precisely learn disentangled velocity fields, without needing additional regularization terms volume density or information on object masks, types, or materials. Overall, our framework models general dynamic 3D scenes by learning neural velocity fields with physical priors. Our method is named NVFi and our contributions are:
We introduce a general framework to model dynamic 3D scenes as physics-informed radiance fields from multi-view videos, without requiring information object types, materials, or masks.
We design neural velocity field together with a joint keyframe and interframe optimization method to effectively train the networks.
We demonstrate three applications for the learned velocity fields on two newly collected dynamic 3D datasets and a challenging real-world dataset, showing superior results in future frame extrapolation, semantic decomposition, and velocity transferring across 3D scenes.
2 Related Works
Static 3D Representations:
Conventional representations for static 3D objects and scenes include
voxels [13, 71, 72], point clouds [17], octrees [57, 64], meshes [28, 23], and primitives [77]. Due to the discretization issue of these explicit representations, the fidelity of 3D shapes is usually limited by spatial resolution and memory footprint. Inspired by the seminal works [10, 36, 41]. recently, there

---

p fe
[6.
gΦ
p =
Dx
By
Keyframe Dynamic Radiance Field
Interframe Velocity Field
Figure 2: The left block illustrates the network architecture of our keyframe dynamic radiance field based on HexPlane [6], and the right block shows the architecture of our interframe velocity field.
has been strong interest in representing 3D data in implicit neural functions. These methods are generally classified as: 1) occupancy fields (OF) [10, 36], 2) signed distance fields (SDF) [41], 3) unsigned distance fields (UDF) [12, 62], and 4) radiance fields (NeRF) [37, 3, 59]. Basically, these neural representations simply take 3D point locations as input and directly regress the corresponding point information, such as occupancy, distance to surface, color, semanties [76, 61], er al., via MLPs. Compared with traditional explicit representations, these implicit neural representations allow for continuous shape and appearance modeling at a particularly low memory footprint.
Dynamic 3D Representations: Recent techniques in this category primarily built the appealing NeRF architecture [37], thanks to its unprecedented level of fidelity in representing various 3D objects and scenes [6. 19]. Given video sequences of dynamic scenes, these NeRF based methods usually take the time as an additional input dimension and then optimize the entire network using the standard photometric loss. Generally, the techniques can be classified as: 1) deformation-bis based methods [42, 58, 4, 46, 5, 18, 74], 2) flow-based methods [21, 16, 63, 35], and 3) direct space-time based methods [69, 31, 43]. In parallel, there are also number of domain-specinf NeRFs to model particular dynamic objects such human bodies [44, 50, 45, 70, 68] and faces [4, 20, 66, 67]. Although all these methods have shown excellent performance in novel view synthesis on wide range of dynamic datasets, they are primarily designed to interpolate frames within the time period of training data, lacking the ability to predict future frames. By contrast, our NVFi aims to model the underlying physical principles of 3D scenes, thus being easily able extrapolate future dynamics.
Physies Informed Deep Learning: Unlike traditional numerical methods such as finite element informed neural networks (PINN) [48] represent tensor fields with neural
methods, recent physics-ifformed networks. converting the PDE solutions into optimizing network weights via PDE-based loss functions. Inspired by the pioneering works [48, 52, 30J, a plethora of succeeding works [51, 49, 25, 38, 65], have been developed, demonstrating excellent results in a wide range of applications such as acoustics [53, 56), fluids [60, 22, 15], 3D scene representations [8, 14, 32, 47], et al., thanks to its mesh-free formulation. More details can found in the recent surveys [27, 24]. In our framework, we find that the commonly-used PINN constraints are insufficient to learn accurate 3D velocity fields. To tackle this issue, we further propose the keyframe based velocity propagation module.
NVFi
3.1 Overview
As shown in Figure 2, our framework consists of two neural networks together with their optimization methods. Given set of images of dynamic scene with known camera poses and intrinsics, the keyframe dynamic radiance field fe simply takes a 3D point p = (x,y,z), viewing angle @), and timestamp / as input, directly regressing the volume density σ and color c b). For the network architecture, we simply adopt the recent HexPlane [6] which shows excellent performance in efficiently modeling dynamic video frames, though other NeRF variants can also be used. Formally, our kryframe dynamic radiance field is defined below and implementation details are in Appendix.
(a,c) = [o(x,,
① are trainable parameters.
For the interframe velocity field ga. it takes the 4D vector (x,,, as input, and predicts the 3D velocity (vx, Vy. for point p at time r. For simplicity, the network go parameterized by simple MLPs, though more advanced architectures can be applied as well. Formally, the velocity field is defined below and implementation details are Appendix.
go (p.r) go (x, 4 are trainable parameters of MLPs.
(2)
3

---

With these two networks and training images sampled from a dynamic 3D scene, the key challenge to effectively optimize these networks, such that the final learned velocity field is precise and disentangled, supporting future frame extrapolation, motion transfer, and semantic decomposition.
3.2 Optimization of Keyframe Dynamic Radiance Fields
Given dynamic video frames of T timestamps (1.......77. there are potential strategies to optimize dynamic neural radiance fields.
Strategy 1- Dense Frame Optimization: This strategy uses all available video frames of specific dynamic 3D scene in the training set to optimize dynamic radiance field. This means that for the network fe. the time dimension is densely sampled during optimization. Formally:
f,, 0,0,1): E {1 (1...-...TT) .T
However, it has two limitations: 1) It is inefficient to learn accurate 3D geometry and appearance because this strategy is somewhat equivalent to modeling dense number of static radiance fields for all timestamps. 2) is hard to obtain a disentangled physical velocity field for the entire 3D scene, as the change of physical geometries is tightly encoded within the dynamic radiance field. Strategy 2 Canonical Frame Optimization: This strategy optimizes canonical representation of 3D geometry and appearance, usually joined with another deformation or transportation network warp the future (7' (7' timestamps back to the first timestamp. It can be as:
0, TE (1}
However, such strategy has a strong assumption that the corresponding point appearances across different timestamps keep unchanged. Therefore, for dramatically changing 3D scenes, the optimized geometry, appearance, and possible jointly learned physies properties tend to be inferior.
Our Strategy Keyframe Optimization: In this regard, we propose to adopt a keyframe based strategy learn the dynamic radiance field fe. particular, we uniformly sample K timestamps out of the total 7' stamps to optimize fe. Formally:
fo(x.y,2,0,000
4 & [TT],
Color images are rendered from the above keyframe dynamic radiance fields by sampling points along rays. For each pixel, i.e, ray r, keyframe at time (s. the appearance C(r, obtained by volume rendering of NeRF [37]. Then the network fe can be optimized by the following keyframe photometric loss. Details of the rendering equation are in Appendix.
Eeryframe |C(r, - C(r,4)ll where C(r,te) ) is the ground truth color observation.
(6)
This keyframe optimization strategy, albeit simple, has two unique advantages: It allows to sufficiently and accurately fit the sparsely sampled dynamic 3D scene geometry and appearance given the same network capacity. 2) It allows the remaining interframes belonging to the (T T7-A K) time stamps to be used for optimizing a disentangled velocity field, as discussed in Section 3.3.
3.3 Optimization of Interframe Velocity Fields
As to separate interframe velocity field ga. it is impossible to directly supervise it using ground truth labels as they cannot be collected practice. However, there are some physics rules to regularize the velocity field. The objects transported by the velocity field, and the velocity field is transported by itself according to some unobservable hidden forces. In order to keep the mass and appearance of the objects, the velocity field needs be divergence-free and obeys the basic law of momentum conservation, whose details are in Appendix. Note that, more complex physics dynamics beyond the daily 3D scenarios are out of the scope of this paper. In this regard, our velocity field, i.e.. = go (p,),, needs to firstly satisfy the following two constraints.
dv dt
V-
+ vpv=a

---

We simply turn these PDEs into the following two PINN losses [48] to optimize the velocity field.
Here we use v(p.t) as the network to avoid abuse notation.
1
M
Edivergence_ free
Σ
Σ
Pa
NM
N
M
av (v(p
Emmmentum
v(Pn:mm) V. p. (( all
(8)
NM
1 dtm
where Pn is uniformly sampled in the whole 3D scene volume, and fm is uniformly sampled from O to the interested maximum extrapolation time fmar. and a is the general acceleration term learned by another MIPP-hased network: (x,), a, whose details such PINN losses, is insufficient to optimize the velocity field itself, since there are infinitely many solutions. To tackle this, we introduce an additional interframe optimization strategy.
Appendix. Nevertheless, with
Interframe Optimization Strategy: Naturally, the 3D scene geometry and appearance encoded in the keyframe dynamic radiance field, once appropriately transported by the velocity field, should be able to render 2D images to match with the ground truth observations in interframes belonging to the remaining (T K) timestamps. To enforce such constraint, the key challenge is to determine the color and density values for all 3D points at each interframe timestamp, such that the volume rendering equation be applied to estimate RGB for each pixel the interframe timestamp, after which the photometric loss can be adopted. To tackle this, we propose the following Algorithm 1.
Algorithm 1 At a specific interframe timestamp li given light ray r; with viewing angle () and S sample points (pr ... P..' along the ray, the objective of this algorithm is to determine the color and density values for the S points along ri, denoted as: {(c), ......................... (.)...-45. ds)}. }. In the meantime, we also have the keyframe dynamic radiance field fe and velocity field go-
Note that, we shall not directly query fe to obtain color and density for the S points because: 1) the dynamic radiance field fe is never trained interframe timestamps, thus the queried values inaccurate; the velocity field g will not be involved, and therefore the interframes cannot provide additional constraints to optimize
Input:
The ray direction (0.0), the interframe timestamp f, the S sample points the ray (Pi (Pi-"-s"" Ps}: Pss'
The K keyframe timestamps {ri -fxx-:--
The initialized and ongoing training networks: fe and go:
Output:
The color and density values for S sample points along the ray:
os)}:
Preliminary step:
Find the nearest keyframe timestamp & for the interframe timestamp f/:
i = arg
Itx -fy
4
for in {p- PP PP,..-Ps' py} do
Transport p, to its corresponding point p', at its nearest keyframe timestamp f.. according to its velocity field. The position of p', can obtained by:
P==P,t P, 1. (t), Runge-Kure Kutta 2 solver [9] is applied in our implementation.
Retrieve the volume density of and view-agnostic color feature vector e, for point p/:
(o), e(, fo(
Note: HexPlane backbone [6] can output view-agnostic color feature vector e'.
Assign the retrieved features of p' to the original point p,: (G,, e,) + (a),e))-
Obtain the color c, for point p,:
C, + foles, 0,0), Note: fe is subnetwork of HexPlane hackhone [6] as detailed in Appendix.
Output (cs, o,) for point P,-
After the above for loop, we obtain all color and density values for S sample points.
Having the color and density values of all 3D points along ray r; in the interframe of timestamp l; the appearance C(r:,i)) hi) is obtained by volume rendering of NeRF [37]. Then both networks fe and go can be optimized by the following interframe photometric loss.
Cinserframe (C(r, - C(r,,)))) where C(r,,t)) is the ground truth color observation.
(9)
5

---

3.4 Joint Keyframe and Interframe Optimization
Technically, the keyframe dynamic radiance field fea can be firstly trained by Exeyframe only, and then the interframe velocity field ga by ddivergence_free + momentuum + finterfome-
Nevertheless. we empirically find that simultameously propagating enrors of interframes back the radiance field fe helps achieve better performance overall. Therefore, we adopt the following joint keyframe and interframe strategy to optimize both networks.
fe f- (Ekeyframe + Cinter
g f- (Edivergence_ + momenntam + finterrrme)
(10)
4 Experiments
Our framework primarily focuses on learning meaningful physical velocity fields for dynamic 3D scenes, instead of simply fitting video frames. Although there number of dynamic 3D scene datasets in the literature, they are mainly collected for the popular task of novel view synthesis within the training time period, i.e., interpolation in time dimension. Besides, the underlying motions of these scenes tend to be chaotic, and estimating their future motions or transferring their motions are hardly meaningful or entertaining in practice. In this regard, we introduce two new synthetic datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset.
Datasets:
1) Dynamic Object Dataset:
This dataset consists of distinct 3D objects, each of which displays a unique motion pattern. including either rigid or deformable movements in 3D space. These 3D objects and their realistic motions all designed by unknown external practitioners from SketchFab, and we purchased their Licenses and will make them available for free use in the community.
For each 3D object, we collect RGB images at 15 different viewing angles over 1 second, where each viewing angle has 60 frames captured. We reserve the first 45 frames at randomly picked 12 viewing angles as the training split, i.e., 540 frames, while leaving the 45 frames at the remaining 3 viewing angles for testing interpolation ability, i.e,, 135 frames for novel view synthesis within the training time period, and keeping the last 15 frames at all 15 viewing angles for evaluating future frame extrapolation, i.e., 225 frames. More details are in Appendix.
2) Dynamic Indoor Scene Dataset: We also collect another synthetic dynamic 3D dataset, which includes 4 indoor scenes with multiple complex 3D objects undergoing different rigid body motions. There are about 4 objects such as tables or chairs in each 3D scene. Basically, such an indoor dataset aims to simulate potential scenarios for robotics or VR applications to understand dynamic 3D scenes.
Since the indoor scene is significantly more challenging, for each 3D scene, we collect RGB images 30 viewing angles over 1 second, where each viewing angle has 60 frames captured. Similarly, we reserve the first 45 frames at randomly picked 25 viewing angles as the training split, i.e., 1 1125 frames, while leaving the 45 frames the remaining viewing angles for testing interpolation ability, i.e., 225 frames, and keeping the last 15 frames at all 30 viewing angles for evaluating future frame extrapolation, i.e., 450 frames. The ground truth object segmentation masks are also collected for evaluating the semantic decomposition capability in Section 4.2. More details are in Appendix.
While existing dynamic 3D scene modeling techniques and the commonly used datasets in literature are mainly designed for novel view cendering/interplaatio within the training time period, rather than for extrapolation beyond the training time period, we evaluate our method two selected scenes from a real-world dataset: NVIDIA Dynamic Scene[73]. It captures real-world dynamic scenes by a static camera rig with 12 cameras. For each scene, we clip 60 frames with reasonable and predictable motions. We reserve the first 46 frames at randomly picked 11 cameras as the training split, iee., 506 frames, while leaving the 46 frames the remaining camera for testing interpolation ability, i.e., 46 frames for novel view synthesis within the training time period, and keeping the last 14 frames at all 12 cameras for evaluating future frame extrapolation, i.e., 168 frames.
We carefully choose three representative groups of methods as our baselines: 1) dense frame optimization method T-NeRF [46] and NSFF [33], 2) canonical frame optimization methods D-NeRF [46] and TiNeu [18], 3) PINN methods T-NeRF pInn and HexPlanepINN: Both methods adapted by us via integrating a separate velocity field supervised by the same PINN losses ours.
Baselines:
F-NeRFPINN
Metrics: For evaluating both interpolation and future frame extrapolation and motion transfer, the standard metrics PSNR, SSIM, and LPIPS scores are reported across testing views. For evaluating

---

Table 1: Quantitative results of all methods for both novel view interpolation and future frame
extrapolation on Dynamic Object Dataset and Dynamic Indoor Scene Dataset.
Dynamic Object Dataset
Dynamic Indoor
Scene Datasel
Interpolation
Extrapolation SSIM
Interpolation
Extrapolation
PSNRT
SSIM
LPIPS|
PSNRT
LPIPS.
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS]
T-NcRF[46]
13.163
0.709
0.353
13,818
0.739
0.324
24.944
0.742
0.336
22.242
0.700
0.363
[-NcRF[46]
14.158
0.697
0.352
14,660
0.737
0.312
25.380
0.766
0.300
20.791
0.692
0.349
[ox
27.988
0.960
0.063
19.612
0.940
0.073
29.982
0.864
0.213
21.029
0.770
0.281
T-NeRFPINN
15.286
0.794
0.293
16.189
0.835
0.230
16.250
0.441
0.638
17.290
0.477
0.618
HexPlanePINN
27.042
0.958
0.057
21.419
0.946
0.067
25.215
0.763
0.389
23.091
0.742
0.401
NSFF[33]
29.365
0.829
0.278
24.163
0.795
0.289
NVFi(Ours) 29.027
0.970
0.039
27.594
0.972
0.036
30,675
0.877
0.211
29.745
0,876
0.204
Table 2: Quantitative results of our method and baselines on the NVIDIA Dynamic Scene dataset.
Truck
Skating
Interpolation SSIM)
Extrapolation
Interpolation SSIM
Extrapolation SSIM?
PSNR1
LPIPS]
PSNRT
SSIM
LPIPS.
PSNR
LPIPS.
PSNR
LPIPS.
ToxeuVox(18) [18] 27.230
0.846
0.229
24.887
0.848
0.209
29.377
0,889
0.202
24.224
0.878
0.220
PlenPlanyy 25.494
0.768
0.337
24.991
0.768
0.325
24.447
0.867
0.225
23,955
0.868
0.232
NVFi(Ours) 27.276
0.840
0.235
28.269
0.855
0.220
26.999
0.848
0.227
28.654
0.896
0.208
semantic decomposition, the Average Precision (AP), Panoptic Quality (PQ) and Fl scores with an IoU threshold of 0.5, together with the mean Intersection over Union (mloU) scores are reported.
4.1 Evaluation of Future Frame Extrapolation
We first evaluate the extrapolation capability of our framework. In particular, our method and baselines except NSFF are trained from scratch on each of the 6 objects in Dynamic Object dataset, and all methods on each of the 4 scenes in Dynamic Indoor Scene dataset, all in scene-specific fashion. In total, (6 x 6) + (7 x 4) 64 models are trained for comparison. The keyframe number K is set as 16 in our method for Dynamic Object Dataset and 4 for Dynamic Indoor Scene Dataset. As for worldworld NVIDIA Dynamic Scene dataset, we evaluate our model and 2 baselines with comparable performance on our own datasets, where the keyframe number K is set as 4 in our method.
Analysis: Table compares all methods regarding the quality of view synthesis for future frame extrapolation. The view synthesis for interpolation is also included for comparison. It can be seen that: 1) our NVFi achieves significantly better results than all baselines on both dynamic datasets, particularly the critical task of future frame extrapolation, although the strong baseline TiNeu Vox shows excellent performance for interpolation. 2) Naïvely adding physics priors into an existing dynamic NeRF tends to be inferior as shown by T-NeRFPINN and HcxPlanePINN- This clearly verifies the effectiveness of our special design of the joint keyframe and interframe optimization strategy. More qualitative results are in Figure 4(a). Table 2 compares our method and two best baselines. can be seen that, even if our NVFi only gets comparable performance in interpolation, it still achieves the best performance in extrapolation without any performance drop compared with interoplation thanks to the accurate motion predictions. More qualitative results are in Figure 3.
4.2 Evaluation of 3D Semantic Scene Decomposition
Having the well-trained keyframe dynamic radiance field so and interframe velocity field go for each 3D scene in the Dynamic Indoor Scene dataset in Section 1, naturally, all individual 3D objects such as chairs and tables undergoing different movements are supposed to be automatically discovered, segmented, and tracked without needing any extra object annotations supervision signals.
In order to achieve this desirable unsupervised object decomposition objective, a naive strategy is to query the velocity values of dense 3D points in space, followed by a velocity clustering module to group points into objects. However, such strategy fundamentally fails to recognize object shapes but just identifies similar motions, thereby tracking objects is also infeasible. A more elegant strategy to directly learn an object code Op (usually one-hot) for each 3D point p within the entire 3D scene volume at the initial timestamp / = 0, without retraining any neural layer of the well-trained networks so and go but just using them. This would clearly allow all dynamic 3D objects in the scene to be segmented and tracked over time. To this end, we simply introduce simple 4-layer MLP as the
7

---

semantic scene decomposition network hy. It takes 3D point p as input, directly regressing its object code op. i.e., Op = hy which optimized using the following steps:
First, given the well-trained keyframe dynamic radiance field Se. we uniformly sample dense 3D points at timestamp = O, obtaining their density values. Only 3D points with sufficiently large densities are kept as valid points for the subsequent steps.
Second, the valid 3D points are fed into our new object segmentation network by (initialized but yet to be trained), obtaining their corresponding object codes.
Third, the valid points are transported to their correspondences at a random timestamp 1' using the well-trained velocity field ga. Motion vectors of these points from O to r' are computed.
Lastly, given per-point motions, employ the dynamic rigid consistency and spatial smoothness losses proposed in OGC [54] to optimize the object segmentation network hy.
Once the object segmentation network by is learned, all dynamic objects at time / = O are segmented. With the aid of the well-trained velocity field, in any subsequent timestamps, all these identified objects can be naturally tracked. In addition, with the aid of well-trained keyframe dynamic radiance field fe. we simply use the accumulated weights computed in volume rendering combine point object codes along a given light ray, thus rendering accurate object segmentation 2D masks for any camera poses at any given timestamps. More details of the implementation are in Appendix.
We evaluate the semantic scene decomposition ability on the Dynamic Indoor Scene dataset. In particular, we render all 2D object segmentation masks from our network at the 30 viewing gles over 60 frames for all scenes, i.e., 7200 2D masks, and then evaluate them against ground truth masks. For fair comparison, we also train similar object segmentation network for the baseline D-NeRF [46] at time 1 = 0, using its learned deformation vectors as supervision signals and tracking signals. Note that, the deformation vectors are converted back as motion vectors. In addition. we include an image-baue based object segmentation method, the powerful Mask2Former [11] pre- trained on COCO [34] dataset, as a fully-supervised baseline. More implementation details are Appendix.
Table Quantitative results of scene decomposi- tion on the Synthetic Indoor Scene dataset.
AP1
PQ1
FIf
mloUf
Mask2Former [11] 65.37
73.14 78.29
64,42
D-NeRF [46] 57.26
46.15
59.02
46.58
NVFi(Ours) 91.21
78.74 93.75
67.64
Analysis: As shown in Table 3, we can see that: 1) Our object segmentation performance is superior to D-NeRF [46], essentially because our velocity field learns better scene dynamics than the deformation field, thus enabling the object segmentation network to be better optimized. 2) We also clearly surpass the powerful pre-trained Mask2Former [11] on all metrics. The reasons are two-fold. First, we fundamentally rely on motion patterns rather than appearances to discover objects, thus being able to generalize to unseen object types ("Genome") or scenes ("Factory") better than Fask2Former [11]. Second, our learned object field inherently leverages multi-view consistency, thus allowing the segmentation of partially occluded objects. Figure 4(b) shows qualitative results.
4.3 Evaluation of Motion Transfer
We further demonstrate the ability of our model to transfer a well-trained velocity field to another separately trained static scene. All objects in the new scene are expected to undergo the same dynamics as learned in the velocity field. The more accurate the learned velocity field, the more realistic and entertaining the new 3D scene will be, after applying the learned dynamics.
In order to evaluate the performance, we create a new 3D scene, called Gnome-new, being similar to the scene Gnome in our Dynamic Indoor Scene dataset. We apply the same dynamics of Gnome on Gnome- new, recording 30*60= = 1800 frames as its ground truth observations. To explicitly show the advantage of our learned disentangled velocity field, we sepa- rately train static TensoRF [7] model for Gnome-new only using its frames at time 0. Note that, any other NeRF variants are also applicable here. We then pick up the well-trained velocity field of Gnome in Section 4, after which we directly apply the learned velocity field on the newly trained static TensoRF model, rendering 30*60 frames for a comparison with the ground truth images. Similarly, we apply the deformation field learned by D-NeRF in Section 4.1 in the same transferring pipeline, rendering 2D images for comparison. More implementation details are in Appendix.
Table 4: Quantitative results of motion trans- fer on Synthetic Indoor Scene dataset.
PSNR1 SSIM LPIPS..
D-NeRF [46] 16.124 0.327
0.550
NVFi(Ours) 16.178
0.334
0.551
8

---

Interpolation
Extrapolation
Interpolation - 05
Extrapolation
(3) Choice of Keyframe Number K: We set the keyframe number as 8 and 32, while we choose : 1
0.5
TiNes
NVR HexPinee (Our))
(PINN))
Gr
Figure 3: Qualitative results of baselines and our method on NVIDIA Dynamic Scene dataset.
Analysis: Figure 4(c) shows that our method clearly keeps the geometry and appearance of the new static object, thanks to the accurately learned disentangled velocity field, whereas D-NeRF [46] fails to do so. However, we observe that in Table our advantage is not so significant. This is because the static reconstruction of Gnome-new lacks supervision signals for ground planes occluded objects, leading to rendering artifacts in these regions when objects are moved away by motion transfer. Our strong ability of motion transfer is further validated by additional experiments our Dynamic Object Dataset in Appendix.
4.4
Ablation Study
(1) Without Joint Optimization: We only use
Table 5: Quantitative results of ablation
Ekeyframe to train the keyframe dynamic radiance field for followed by the (Edivergence__ie "rree & momentuu Cinterframe) to separately train the velocity field go.
studies on Dynamic Object dataset.
Extrapolation
Joint Physies
#K
PSNRT
SSIMt LPIPS..
The PINN losses ( (ddieeggenc__fve + momentuum are removed to train go.
(2) Removing Physics Constraints:
16
27.594
0.972
0.036
x
/
16
24.792
0.955
0.059
X
16
25.537
0.968
0.040
v
8
27.490
0.974
0.036
K = 16 in main experiments.
32
24.902
0.964
0.037
(4) Reducing the Number of Cameras: the number of cameras used in our Dynamic Object datasets half of the number, i.e., cameras, and one quarter of the number, i.e., 3 cameras.
reduce
Table 6:
Quantitative results of ablation studies on Dynamic Object dataset.
Extrapolation
Camera Numbers
PSNR
SSIM1
LPIPST
27.594
0.972
0.036
Table 5 and Table 6 shows the ablation results for future frame extrapolation on our Dynamic Object dataset. We that: 1) The joint keyframe and interframe optimization strategy is critical to enable our method to learn accurate velocity field as well as dynamic radiance field. 2) Once the keyframe number K becomes larger, the extrapolation capability clearly drops, validating that the dense supervision is inferior to help learn physics velocity overall. 3)
25.114
0.959
0.122
21.370
0.917
0.084
The extremely sparse camera views unlikely to capture sufficient visual information for physical motion learning. More ablation results are in Appendix.
Conclusion
In this paper, we extend the appealing radiance field to represent dynamic 3D scenes from multi- view videos. Unlike most of the existing methods which focus on novel view synthesis within the training time period, our method learns to disentangle the physical velocity field from the geometry and appearance of 3D scenes by jointly optimizing two neural networks: the keyframe dynamic radiance field and the interframe velocity field. Extensive experiments on three dynamic datasets demonstrate that our framework learns accurate velocity, enabling successful applications in future frame extrapolation, semantic scene decomposition, and motion transfer.

---

Interpolation
Extrapolation
Interpolation
Extrapolation
[ t= 0.5
t=1 t=1 1
t=
t= 0.5
[= t=
D.NoRF
ToNen
T-NeRF (PINN)
Plaxe (PINN)
NVH (Ours)
Ground Truth
Future Frame Extrapolation
Image
D-NeRF
NVFI (Ours)
GT
Mask2- Fomer
D.NeRF
NVFI (Ours)
1=1
GI
(b) Semantic Scene Decomposition
(c) Motion Transfer
Figure 4: Qualitative results of baselines and our method the three tasks. More qualitative results can be found in Appendix and our project page: https:/vvlargroott wwtuuu.gttuub.iee Watthubio/NVFi..nt
10

---

Acknowledgements: This work was supported in part by Research Grants Council of Hong Kong under Grants 25207822 & 15225522.
References
[1] B. Attal, E. Laidlaw, A. Gokaslan, C. Kim, C. Richardt, J. Tompkin, and M. O'Toole. ToRF: Time-of-Fiin Radiance Fields for Dynamic Scene View Synthesis. NeurIPS, 2021.
[2] D. Baieri, S. Esposito, F. Maggioli, and E. Rodolà. Fluid Dynamics Network: Topology- Agnostic 4D Reconstruction via Fluid Dynamics Priors. arXiv:2303.0987,,200 2023.
[3] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, and R. M.-b. Pratul. Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. ICCV, 2021.
[4] J. T. Barron, K. Park, S. M. Seitz, and R. Martin-brualle Nerfies: Deformable Neural Radiance Fields. ICCV, 2021.
[5] H. Cai, W. Feng, X. Feng, Y. Wang, and Zhang. Neural Surface Reconstruction Dynamic Scenes with Monocular RGB-D Camera. NeurlPS, 2022.
[6] A. Cao and J. Johnson. HexPlane: A Fast Representation for Dynamic Scenes. CVPR, 2023.
[7] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. TensoRF: Tensorial Radiance Fields. ECCV, 2022.
[8] H. Chen, R. Wu, E. Grinspun, C. Zheng, and P. Chen. Implicit Neural Spatial Representations for Time-dependent PDEs. arXiv:2210.00124, 2022.
[9] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural Ordinary Differential Equations. NeurlPS, 2018.
[10] 7. Chen and H. Zhang. Learning Implicit Fields for Generative Shape Modeling. CVPR, 2019.
[11] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention Mask Transformer for Universal Image Segmentation. CVPR, 2022.
[12] J. Chibane, A. Mir, and G. Pons-Moll. Neural Unsigned Distance Fields for Implicit Function Learning. NeurIPS, 2020.
[13] C. B. Choy, D. Xu, Gwak, K. Chen, and S. Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. ECCV, 2016.
[14] M. Chu, L. Liu, Q Zheng. Franz, H. Seidel, Theobalt, and Zayer. Physics informed neural fields for smoke reconstruction with sparse data. TOG, 2022.
[15] Y. Deng, H.-X. Yu. J. Wu, and B. Zhu. Learning Vortex Dynamics for Fluid Inference and Prediction. ICLR, 2023.
[16] Y. Du, Y. Zhang, and J. B. Tenenbaum. Neural Radiance Flow for 4D View Synthesis and Video Processing. ICCV, 2021.
[17] H. Fan, H. Su, and Guibas. A Point Set Generation Network for 3D Object Reconstruction from Single Image. CVPR, 2017.
[18] Fang, X. Wang, and M. Nießner. Fast Dynamic Radiance Fields with Time-Aware Neural Voxels. SIGGRAPHI Asia, 2022.
[19] S. Kridovich-Keil, G. Meanti, F. Warburg, B. Recht, and A. Kanazawa. K-Planes: Explicit Radiance Fields in Space, Time, and Appearance. CVPR, 2023.
[20] G. Gafni, J. Thies, M. Zollhöfer, and M. Nießner, Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. CVPR, 2021.
[21] C. Gao, Saraf, J. Kopf, and J.-B. Huang. Dynamic View Synthesis from Dynamic Monocular Video. ICCV, 2021.

---

[22] F. Gibou, D. Hyde, and R. Fedkiw. Sharp interface approaches and deep learning techniques for multiphase flows. Journal of Computational Physics, 2019.
[23] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry, Papier-Mache Approach to Learning 3D Surface Generation. CVPR, 2018.
[24] Z. Hao, S. Liu, Y. Zhang, C. Ying. Y. Feng, H. Su, and J. Zhu. Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications. arXiv:221..0006. 2022.
[25] Z. Hao, C. Ying, H. Su, J. Zhu, J. Song, and Z. Cheng. Bi-level Physics-Informer Neural Networks for PDE Constrained Optimization Using Broyden's 's Hypergrandients. ICLR, 2023,
[26] R. Z. Horace He. functorch: Jax-like composable function transforms for pytorch. https:
/ cww.ppytorrca/ffuctrrreeeeo 2021.
[27] G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics- informed machine learning. Nature Reviews Physics, 2021.
[28] H. Kato, Y. Ushiku, and T. Harada. Neural 3D Mesh Renderer. CVPR, 2018.
[29] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[30] I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions Neural Networks, 1998.
[31] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, and Z. Lv. Neural 3D Video Synthesis From Multi-View Video. CVPR, 2022.
[32] X. Li, Y.-L. Qiao, P. Y. Chen, K. M. Jatavallabhula, M. Lin, C. Jiang, and C. Gan. PAC- NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnosee System Identification. ICLR, 2023.
[33] Z. Li. S. Niklaus, N. Snavely, and O. Wang. Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes. CVPR, 2021.
[34] T.-Y. Lin, M. Maire, S. Belongie, L.. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft COCO: Common Objects in Context. ECCV, 2014.
[35] Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y.-Y. Chuang, Kopf, and J.-B. Huang. Robust Dynamic Radiance Fields. CVPR, 2023.
[36] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.
[37] B. Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV, 2020.
[38] S. Mishra and R. Molinaro. Estimates the generalization error of physics-informed neural networks for approximating PDEs. IMA Journal of Numerical Analysis, 2023.
[39] A. Noguchi, X. Sun, S. Lin, and T. Harada. Neural Articulated Radiance Field. ICCV, 2021.
[40] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide. Neural Scene Graphs for Dynamic Scenes. CVPR, 2021.
[41] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. CVPR, 2019.
[42] K. Park, U. Sinha, Hedman, T. Barron, Bouaziz, D. B. Goldman, R. Martin-Bruad and
S. M. Seitz. HyperNeRF: A ligeer-Dimeniionall Representation for Topologically Varying Neural Radiance Fields. SIGGRAPH Asia, 2021.
[43] Park, M. Son, S. Jang, Y. Ahn, J.-Y. Kim, and N. Kang. Temporal Interpolation All You Need for Dynamic Neural Radiance Fields. CVPR, 2023.
12

---

[44] S. Peng, J. Dong, Q. Wang, S. Zhang, Q. Shuai, X. Zhou, and H. Bao. Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies. ICCV, 2021.
[45] S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou. Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans. CVPR, 2021.
[46] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. CVPR, 2021.
[47] Y.-L.. Qiao, A. Gao, and M. C. Lin. NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos. NewrIPS, 2022.
[48] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[49] M. Raissi, A. Yazdani, and G. E. Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 2020.
[50] A. Raj, J. Tanke, J. Hays, M. Vo, C. Stoll, and C. Lassner. ANR: Articulated Neural Rendering for Virtual Avatars. CVPR, 2021.
[51] Y. Shin, J. Darbon, and G. E. Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs. #rXiv:2004.0/806, 2020.
[52] J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 175:1339-1364 2018.
[53] V. Sitzmann, N. Martel, A. W. Bergman, D. Lindell, and G. Wetzstein, Implicit Neural Representations with Periodic Activation Functions. NeurIPS, 2020.
[54] Z. Song and B. Yang. OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Clouds. NeurIPS, 2022.
[55] S.-Y. Su, F. Yu, M. Zollhoefer, and H. Rhodin. A-NeRF: Learning Human Shape, Appearance, and Pose. NeurIPS, 2021.
Articulated Neural Radiance Fields for
[56] M. Tancik, P. P. Srinivasan, B. Mildenhall, N. Raghavan, and J. T. Barron. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. NeurlPS, 2020.
[57] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree Generating Networks: Efficient Convolu- tional Architectures for High-resolution 3D Outputs. ICCV, 2017.
[58] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of Deforming Scene from Monocular Video. ICCV, 2021.
[59] A. Trevithick and B. Yang. GRF: Learning a General Radiance Field for 3D Representation and Rendering. ICCV, 2021.
[60] K. Um, R. Brand, Y. R. Fei, P. Holl, and N. Thuerey. Solver-in-the-Loop Learning from Differentiable Physics to Interact with Iterative PDE-Solvers. NeurlPS, 2020.
[61] B. Wang, L. Chen, and B. Yang. DM-NeRF: 3D Scene Geometry Decomposition and Manipu- lation from 2D Images. ICLR, 2023.
[62] B. Wang, Z. Yu, B. Yang. J. Qin, T. Breckon, L. Shao, N. Trigoni, and A. Markham. RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds. arXiv:2204.09138 2022.
[63] C. Wang. B. Eckart, S. Lucey, and O. Gallo. Neural Trajectory Fields for Dynamic Novel View Synthesis. arXiv:2105.0590 05.9499 2021.
[64] P.-.. Wang, Y. Liu, Y.-X, Guo, C.-Y. Sun, and X. Tong. O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis. TOG, 36, 2017.
13

---

[65] S. Wang, H. Wang. and P. Perdikaris. Learning the solution operator of parametric partial differential equations with physics-iffrrmed DeepONets. Science Advances, 7(40). 2021.
[66] T.-C. Wang, A. Mallya, and M.-Y. Liu. One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing. CVPR, 2021.
[67] Z. Wang, T. Bagautdinov, S. Lombardi, T. Simon, J. Saragih, Hodgins, and M. Zollhöfer. Learning Compositional Radiance Fields Dynamic Human Heads. CVPR, 2021.
[68] C.-Y. Weng, B. Curless, P. Srinivasan, J. T. Barron, and I. Kemelmacher--Shizzemaa Human- NeRF: Free-viewpoint Rendering of Moving People from Monocular Video. CVPR, 2022.
[69] W. Xian, J.-B. Huang, J. Kopf, and C. Kim. Space-time Neural Irradiance Fields for Free- Viewpoint Video. CVPR, 2021.
[70] H. Xu, T. Alldieck, and C. Sminchisescu. H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion, NeurIPS, 2021.
[71] B. Yang, S. Rosa, A. Markham, N. Trigoni, and H. Wen. Dense 3D Object Reconstruction from Single Depth View. TPAMI, 2019.
[72] B. Yang, S. Wang, A. Markham, and N. Trigoni. Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction. IJV,, 128:53-73, 2020.
[73] S. Yoon, K. Kim, O. Gallo, H. S. Park, and J. Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera, 2020.
[74] M. You and J. Hou, Decoupling Dynamic Monocular Videos for Dynamic View Synthesis. DoXiv:2304.01700 2023.
[75] W. Yuan, Z. Lv, T. Schmidt, and S. Lovegrove. STaR: Seff-supervised Tracking and Reconstruc- tion of Rigid Objects Motion with Neural Rendering. CVPR, 2021.
[76] S. Zhi, T. Laidlow, S. Leutenegger, and A. J. Davison. In-Place Scene Labelling and Under- standing with Implicit Scene Representation. ICCV, 2021.
[77] C. Zou, E. Yumer, Yang, D. Ceylan, and D. Hoiem. 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks. ICCV, 2017.
14

---

A Appendix
The appendix includes:
Rendering equation.
Implementation details of keyframe dynamic radiance field and velocity field.
Additional details of physics prior.
Implementation details of joint optimization.
Additional details of datasets.
Implementation details about semantic scene decomposition.
Implementation details about motion transfer.
Additional quantitative results for ablation study.
Additional quantitative & qualitative results for future frame extrapolation.
Additional quantitative qualitative results for semantic scene decomposition.
Additional qualitative results for motion transfer.
In addition, we provide demo video for better visualization of qualitative results on multiple tasks, appended in the supplementary materials.
A.1 Rendering Equation
Neural radiance fields were initially introduced [37] to model 3D scenes by associating the coordinate zx) and view direction (0 of each point in space with its color, represented by vector and density, denoted by σ. We keep the same rendering equation to obtain the expected color C(r,,)) of a pixel in the image captured by camera at time fi ray ri(s) = 0; sd, is involved, which marches from the camera's ' center towards the pixel. Here, O, and d, represent the ray's origin and direction, respectively, while signifies the distance from a point the camera, ranging from predefined near bound 5n to a far bound sy. The pixel color is rendered by sampling series of points along the ray and performing classical volume rendering:
((r)
Σ
7:( TXl--exp(o;ã))ee
211
(11)
7, = exp(
Σ o,8,).
where 6 represents the distance between the in and (i+ 1) sample point, and N denotes the number of sampled points. Equation (11) connects real 3D points with image pixels by accumulating colors and densities a, of sample points along the ray.
A.2 Implementation Details of Keyframe Dynamic Radiance Field
We adapt HexPlane to parameterize volumetric keyframes. It is direct extension of static TensoRF[7] a dynamic radiance field. Instead using concatenation to combine the features from different planes as TensoRF and HexPlane, we use Hadamard Production used in K-Plane| 19], which shows stronger ability in representing complex geometries. The formal definition of our Keyframe Dynamic Radiance Field is as follows:
we denote & as the time step corresponding to the th keyframe, we can write:
A (x; (x) Bff (xx.)) Øg (z,, a. @ (2..)))))))))))))))))))))))))))) @ @2 4) (
(12)
and
o (x;, 43) = TT (hr (Ar.y)) @ ki (zs, w (x, @ ka 4e) ha(y))z) @ ka(A,, 4)),
where a (x; Q.) is the predicted sigma for point x,, A (x; R) is the predicted appearance feature. And Fj. h gj and kj are vector-valued grid functions with output dimension Mg Mapp respectively. The
15

---

final density value a is the summation of the Mo dimension density feature, and the final appearance feature weighted sum of the Mapp dimension output feature, where B is a simple linear layer. Note unlike [exPlane[6] and P=Panes[[9]] we only do grid interpolation to find the feature values for spatial coordinates, while we pick the time dimension according to the corresponding keyframe index. The final RGB color is regressed from 2-layer MLP with appearance feature and view direction as inputs. With points' density a and colors c, images are rendered via volumetric rendering Equation (11).
In our implementation, for all the experiments, we set MM as 24 and Mapp as 48, while for different experiments, we use different keyframe numbers. For object extrapolation task, we set keyframe number as 16, due to the more complex dynamics of the deformable objects. For both indoor extrapolation task and segmentation task. we set keyframe number as 4. The color decoder MLP has hidden layers, each with 128 nodes.
A.3 Implementation Details of Velocity Field
The velocity field is defined as follows:
V == go (p.1) = go (x,),, (
① are trainable parameters of ML.Ps. (13)
Here we decompose the velocity field as weight MLP w& along with a velocity basis field. In our experiment, we find this can make the velocity convergence faster. The velocity basis field is defined as:
(x,x,22)
(14)
22 y - y
0
The weight MLP is implemented as 4 hidden layers with 128 nodes, with positional encoder dim 3 as [37]. The output weight vector E R., which could be regarded as the linear velocity in x, directions and angular velocity rounding x, y, Z axis respectively.
A.4 Additional Details of Physics Prior
We enforce the appearance and volume density features, denoted as fi both admit the conservation laws characterized by the divergence-frve implicit velocity field vxx,)):
ff + (vfi) πr
af + +V Ōt
Df: + v= where Dr
D
is the material derivative. Dr
Intuitively, we assume the objects moved by the velocity field have a property of local rigidity, and the implicit velocity field should be incompressible (or in other words free of sources or sinks of mass). Then we can get the divergence-fire free constraint V-v
The momentum of velocity field is defined be pv, per unit volume. Newton's second law of motion states that momentum is conserved by mechanical system masses no forces act on the system. If Fxx,t) is the force acting the velocity, per unit volume, then immediately have
Dv:
P Dr
= F.
where we assume the implicit velocity field has uniform mass density As result, the momentum conservation constraint could be derived as:
Dv: dv
(15)
Dr
A.5 Implementation Details of Joint Optimization
During training, our keyframe radiance field starts with space grid size of 64 and increases its resolution in log scale at 2k, 4k, 6k, 8k, 10k iterations till 200 The learning rate for feature planes 0.02, and the learning rate for color decoding neural network and velocity field is 0.001. All learning
16

---

rates are exponentially decayed to 1/10 at final iteration 30k. We use Adam[29) for optimization with Bi 0.9, f2= = 0.99. We apply Total Variational loss [7. 6, 19] on all feature planes with 2 0.001 for spatial axes.
We sample 262144 points uniformly in the space [-1, 1, and time [0, for dynamic object datasets, and 13 1310672 points for dynamic indoor scene datasets every iteration. The jacobians of velocity required calculated by using autograd from functorch[26]. For all the sampled points, we only evaluate the physics losses at occupied region, where the grid alpha a = - exp o x 0.01 2 0.0001. We set the loss weight for divergence-free loss as 5, and the weight for momentum conservation loss as 0.1. All scenes are trained for 1.5 hours on single NVIDIA RTX 3090 GPU respectively.
A.6 Additional Details of Datasets
This dataset comprises 6 distinct objects 1 displaying variety of motion types, including both rigid and deformable movements. We use a total of 15 views, of which 12 are allocated for training and 3 for testing. Each view consists of 60 frames over 1 second; however, only the first 45 frames are used in the training set, with the remaining 15 frames for evaluation. Details of the 6 dynamic objects are:
Dynamic Object dataset:
Falling Ball: A basketball is falling freely through a hoop. The basketball is accelerated by gravity, which should be learned by the model.
Telescope:
A telescope is given, whose upper part is rotating while the lower part remains
static.
An antique fan is given. The outer part of the fan is static, while the inner fan is rotating. The embedded motion makes this scene difficult.
Fan:
Bat: A bat is flapping its wings. Since the wing is extremely thin, and the motion in great extent, it's hard to reconstruct the motion.
Shark: A shark is shaking its tail left and right.
Whale: A whale is flapping its tail up and down.
Dynamic Indoor Scene dataset: This dataset consists of 4 indoor scenes, each containing several rigid objects 2 undergoing different rigid body motions. Since the indoor scene is more challenging, it comprises 30 views, with 25 for training and 5 for testing. Like the first dataset, use the first 45 frames from the 60-frame views for training and reserved the remaining frames for evaluation. A set of ground-truth segmentation masks is provided for evaluation. The segmentation mask is rendered by blender object index, which is perfectly accurate as ground truth. We have in total 4 scenes:
Gnome House: 3 objects moving outwards from the center of house: gnome, sofa, and a treasure chest.
Chessboard: objects are moving towards each other on a chessboard: a horse statue, marble bust, and a china vase from one side, a wooden elephant and a brass vase from the other side.
Factory: 5 objects moving towards nearly the same direction, among which objects are rotating at the same time: three barrels with different appearance, cardboard box, and a compost bag.
Dining Table: 4 objects are falling down onto dining table from the surrounding air: an apple, a cake, a croissant, and a lime.
A.7 Implementation Details about Semantic Scene Decomposition
NVFi(Ours): Figure 5 illustrates the optimization pipeline of our method for scene decomposition
task:
Basketball is downloaded from TurboSquid. licensed under the TurboSquid 3D Model License: purchased from SketchFab, li-
https://blg..uurboquidd wurbosquii..ooo//uruuuuuu -iw-osuuuidddd-mme---iieee model-license. Other objects censed under the SketchFab Standard I. License: https:/sketchhab.com/// htw.chfaab.cm//cccesssss
AAll objects and textures are freely downloaded from Poly Haven, licensed under Poly Haven asset License: https://pllyhaveencomiiiceee
17

---

J
1=0
0
fe
hxp
dynamic
smooth
M
Figure 5: The optimization pipeline our method for scene decomposition. The solid black lines represent forward inference while the dashed gray lines represent backward supervision.
First, sample dense 3D points at timestamp 0 and query their density values through the well-trained keyframe dynamic radiance field fo. By removing 3D points with low densities, we obtain non-empty point set P with N points timestamp
i.e., P E RR
Second, we feed the point set into the to-be-trained scene decomposition network hy and obtain per-point object codes E (0, N where a predefined number of objects that the network is expected to predict in maximum.
Third, the points in P are transported to their correspondences p' using the well-trained velocity field go. Thus, we compute the per-point motion vectors M 1R M = =P' P. Lastly, we apply two losses proposed in OGC [54] onto the predicted object codes. 1) Dynamic rigid consistency: For the 1th object, We first retrieve its (soft) binary mask O³, and feed the tuple (P. p', O²} into the Beighted-Kabsee algorithm to estimate its transformation matrix Tg E B4x4
belonging to SE(3) group. Then the dynamic loss is defined as:
(TrOP)) - (p (TxoP))-(p++ [
Σ
1 PEP
K
(p+mm)) m2)l
Cdynanic
(16)
where of represents the probability of being assigned to the k object for specific point p. and m" E R23 represents the motion vector of from timestamp 0 to The operation applies the rigid transformation the point. This loss serves discriminate objects with different motions. 2) Spatial smoothness: We first search H nearest neighboring points for each point in P. Then the smoothness loss is defined as:
H Op: 0pr111)
smooth
Σ H
Σ Opp
(17)
N
where op E 1), represents the object assignment of center point and op, E (0, 1)* represents the object assignment of its 4t neighbouring point. This loss servers to avoid the over-segmentation issues. We refer readers to [54] for more details.
For our scene decomposition network hy, we adopt a simple 4-layer MLP with 128 neurons in each hidden layer. The maximum number of predicted objects K is to 8. softmax activation is applied onto the predicted object code. During optimization, we adopt the Adam optimizer with a learning rate of 0.005 and optimize the network for 1000 iterations. The two losses ldynamic and E omooth are weighted by [1.0, 0.1).
D-NeRF: For D-NeRF, we adopt exactly the same architecture of scene decomposition network as ours. The optimizer and other hyperparameters are also consistent with ours.
We use the implementation of Fask2Former from MMDetection 3. where we choose the most powerful Swin-L backbone. The model is pretrained on COCO dataset for panoptic segmentation task. Since we aim to decompose objects with different motions only, the ground plane and the wall in each scene are regarded as a whole static background, while the pre-trained model may separate them into different parts. For a fair comparison, we manually merge the segments in model's 's predictions corresponding to the static background.
Mask2Former:
18

---

Table 7: Quantitative results of ablation study for both interpolation task and extrapolation task on Dynamic Object dataset.
Interpolation SSIMT
Extrapolation SSIMI
Physics
PSNRT
LPIPS
PSNR1
LPIPS]
16
29.027
0.970
0.039
27.594
0.972
0.036
16
27.525
0.959
0.057
24.792
0.955
0.059
16
29.109
0.970
0.039
25.537
0.968
0.040
29.355
0.041
27.490
0.974
0.036
32
28.922
0,969
0.042
24.902
0,964
0.037
Table 8: Quantitative results of ablation study of the keyframe number on our Dynamic Indoor Scene dataset.
Interpolation
Extrapolation
Number Keyframes
PSNRT
SSIM?
LPIPS.
PSNR1
SSIMf
LPIPS]
30.675
0.877
0.211
29.745
0.876
0.204
8
30.321
0.871
0.220
29.093
0.873
0.225
16
28.000
0.862
0.226
26.235
0.839
0.237
32
29.764
0.851
0.255
26.634
0.828
0.247
A.8 Additional Implementation Details about Motion Transfer
There are two models to train for this task. The first one is the velocity field model. The training detail of the velocity field the same as future frame extrapolation task. In fact, we the trained velocity from our future frame extrapolation task. The other model to train is new static TensoRF [7] for the scene be transferred. To be simple, we just set the keyframe number of our NVFi as 1, then we can get static representation as TensoRF. The next step is to load the parameter from the pretrained velocity field to our new NVFi.
A.9
Additional Quantitative Results for Ablation Study
Here we show the total results for ablation study in Tables 7&8&9, both for interpolation and extrapolation.
A.10 Additional Quantitative Results for Future Frame Extrapolation
We report the quantitative results of interpolation and extrapolation tasks for individual objects/scenes in Dynamic Object dataset and Dynamic Indoor Scene dataset. As shown in Tables 10& 1, our method achieves leading performance frequently on interpolation, and significantly outperforms all baselines future frame extrapolation.
A.11 Additional Quantitative Results for Semantic Scene Decomposition
In the main paper, we report the excellent results of model in Table semantic scene decomposition task based on a small integral time step 0.06. We also report the results with much larger integral time step 0.5. As shown in Tables 12&13, 13, when we the integral time step as 0.06, our segmentation results are significantly boosted, surpassing the two baselines by large margins.
A.12 Additional Qualitative results for Future Frame Extrapolation
We show the remaining qualitative results for interpolation and extrapolation tasks Dynamic Objects Datasets in Figures 6&7, on Dynamic Indoor Scene Datasets in Figures 8&9. Images at time 0 and time 0.5 are within the observed time (interpolation), while time is future frames. For each scene, the first three images are novel views, and the fourth one is from seen viewing angle but at future timestamp.
A.13 Additional Qualitative results for semantic scene decomposition
We show more qualitative results semantic scene decomposition on Dynamic Indoor Scene dataset in Figures 10& 1. Images at time 0 till time 0.67 are within observed time interval, while time 0.83 and time for future frames. Besides, we apply the semantic decomposition pipeline onto the Dynamic Object dataset for part decomposition. Qualitative results shown in Figure 12.
19

---

Table 9: Quantitative results of ablation study of the camera number on our Dynamic Objects dataset.
Interpolation PSNRT
Extrapolation PSNR
Number of Cameras
SSIM
LPIPS!
SSIM)
LPIPS.
12
29.027
0.970
0.039
27.594
0.972
0.036
25.689
0.954
0.051
25.114
0.959
0.122
21.460
0.912
0.088
21.370
0.917
0.084
Table 10: Per-scene quantitative results of Dynamic Object dataset.
Falling Ball
Bat
Methods
Interpolation
Extrapolation
Interpolation SSIM
Extrapolation SSIM
PSNR1
SSIMT
LPIPS
PSNRT
SSIM
LPIPS
PSNR1
LPIPS
PSNR
LPIPS.
[-NeRF[46]
14.921
0.782
0.326
15.418
0.793
0.308
13.070
0.836
0.234
13.897
0.834
0.230
D-NcRF[46]
15.548
0.665
0.435
15.116
0.644
0.427
14.087
0.845
0.212
15,406
0.887
0.175
[ixeuVox[[18
35.458
0.974
0.052
20.242
0.959
0.067
16,080
0.908
0.108
16.952
0.930
0.115
T-NcRFPINN
17.687
0.775
0.368
17.857
0.829
0.265
16.412
0.903
0.197
18.983
0.930
0.132
HcxPlancPINN
32.144
0.965
0.065
20.762
0,951
0.081
23,399
0.958
0.057
21.144
0.951
0.064
NVFI(Ours)
35.826
0.978
0.041
31.369
0.978
0.041
23.325
0.964
0.046
25.015
0.968
0.042
Fan
Telescope
Methods
Interpolation
Extrapolation
Interpolation
Extrapolation
PSNR
SSIMt
LPIPS]
PSNR1
SSIM
LPIPS
PSNR1
SSIM
LPIPS
PSNR
SSIM
LPIPS.
[-NeRF[46]
8.001
0.308
0.646
8.494
0.392
0.593
13.031
0.615
0.472
13,892
0.670
0.417
D-NeRF[46]
7.915
0.262
0.690
8.624
0.370
0.623
13.295
0.609
0.469
14,967
0,700
0.385
Vox
24.088
0.930
0.104
20.932
0.935
0.078
31,666
0,982
0.041
20.456
0,921
0.067
T-NeRFPINN
9.233
0.541
0.508
9.828
0.606
0.443
14.293
0.739
0.366
15.752
0.804
0.298
HcxPlanePINN
22.822
0.921
0.079
19.724
0.919
0.080
25.381
0.948
0.066
23.165
0.932
0.074
NVFi(Ours)
25.213
0.948
0.049
27.172
0.963
0.037
26.487
0.959
0.048
27.101
0,963
0.046
Shark
Whale
Methods
Interpolation
Extrapolation SSIM
Interpolation SSIM
Extrapolation
PSNR
SSIM
LPIPS
PSNR1
LPIPS
PSNR
LPIPS)
PSNR
SSIM
LPIPS]
[-NeRF[46]
13.813
0.853
0.223
15.325
0.882
0.193
16. 161
0.860
0.212
15.880
0.860
0.203
[-NeRF[46]
17.727
0.903
0.150
19.078
0.936
0.092
16.373
0.898
0.154
14.771
0.883
0.171
23.178
0.971
0.059
19.463
0.950
0.050
37.455
0.994
0.016
19.624
0.943
0.063
T-NeRFPINN
17.315
0.878
0.177
18.739
0,921
0.115
16,778
0.927
0.141
15.974
0.919
0.127
HexPlanePINN
28.874
0.976
0.040
22.330
0.961
0.047
29.634
0.981
0.035
21.391
0.961
0.053
NVFi(Ours)
32.072
0.984
0.024
28.874
0.982
0.021
31,240
0.986
0.025
26.032
0,978
0.029
A.14 Additional Qualitative Results for Motion Transfer
We have already shown results for Dynamic Indoor Scene datasets in Section 4.3. Here we report three results on objectwise motion transfer in Figure 13:
Whale to starfish. We transfer the flapping tail motion to the starfish. The starfish is supposed to flapping its rays as the whale.
Shark to whale. We transfer the shaking tail motion the whale. The whale used flap its tail, but now it's supposed to shake its tail.
Whale to shark. We transfer the flapping tail motion to the shark. The shark is supposed to flap its tail as whale instead of shaking it.
20

---

Table 11: Per-scene quantitative results of Dynamic Indoor Scene dataset.
Gnome House
Chesshoard
Methods
Interpolation
Extrapolation
Interpolation SSIM
Extrapolation SSIM
PSNR1
SSIM
LPIPS]
PSNR1
SSIM
LPIPS]
PSNR|
LPIPS]
PSNR
LPIPS.
T-NeRF[46]
26.094
0.716
0.383
23.485
0.643
0.419
25.517
0.796
0.294
20.228
0.708
0.365
D-NeRF[46]
27.000
0.745
0.319
21.714
0.641
0.367
24.852
0.774
0.308
19.455
0.675
0.384
[1
30.646
0.831
0.253
21.418
0.699
0.326
33,001
0.917
0.177
19.718
0.765
0.310
T-NeRFFINw
15.008
0.375
0.668
16,200
0.409
0.651
16,549
0.457
0.621
17.197
0.472
0.618
HexPlanePINN
23.764
0.658
0.510
22.867
0.658
0.510
24.605
0.778
0.412
21.518
0.748
0.428
NSFF[33]
31.418
0.821
0.294
25.892
0.750
0.327
32.514
0.810
0.201
21.501
0.805
0.282
NVFi(Our))
30.667
0.824
0.277
30,408
0.826
0.273
30.394
0.888
0.215
27.840
0,872
0.219
Factory
Dining Table
Methods
Interpolation SSIMt
Extrapolation SSIM
Interpolation
Extrapolation
PSNRT
LPIPS|
PSNRT
LPIPS
PSNRT
SSIM
LPIPS
PSNR*
SSIM
LPIPS]
[-NeRF[[6]
26.467
0.741
0.328
24.276
0.722
0.344
21.699
0.716
0.338
20,977
0.725
0.324
D-NeR[[46]
28.818
0.818
0,252
22.959
0.746
0.303
20.851
0.725
0.319
19.035
0.705
0.341
Vox 18]
32,684
0.909
0.148
22.622
0,810
0.229
23,596
0.798
0.274
20.357
0.804
0.258
T-NeRUPINN
16.634
0,446
0.624
17.546
0.480
0,609
16.807
0.486
0.640
18.215
0.548
0.595
HexPlanePINN
27.200
0.826
0.283
24.998
0.792
0.312
25.291
0.788
0.350
22.979
0.771
0.355
NSFF[33]
33,975
0.919
0.152
26.647
0.855
0.196
19.552
0.665
0,464
22.612
0.770
0.351
NVFi(Ours)
32.460
0.912
0.151
31,719
0.908
0.154
29.179
0.885
0.199
29.011
0.898
0.171
Table 12: Overall quantitative results for semantic scene decomposition.
API
PQT
F11
Pret
Reet
mlnUf
Mask2Former(11] [1]
65.37
73.14
78.29
94.83
68.88
64.42
D-NeRF[46]
57.26 46.15
59.02
56.55
62.94
46.58
NVFi(Ours, A 0.5)
75.82
63.34
80.59
80.78
81.11
57.05
NVFROurs, 0.06)
91.21
78.74
93.75
93.76
93.74
67.64
Table 13: Per-scene quantitative results for semantic scene decomposition.
Methods
Gnome
House
Chessboard
AP
PQ*
F11
Pref
Recf
mloU?
APt
PQ1
F1f
Pref
Rect
mloU
Mask2Former[[11
60.89
73.05
77.32
85.32
70.69
66.94
82.68
81.35
90.81
97.54
84.94
76.17
D-NeRF[46]
80.54
62.24
85.28
85.28
85.28
54.82
57.12
48.11
60.22
56.20
64.85
48.97
NVFiOOur,, A 0.5)
99.00
82.99
99.92
99.92
99.92
66.42
48.30
43.14
61.19 61.53 60.84
45.95
NVFi(Ours, A 0.06)
100.00
85.01
100.00
100.00
100,00
68.01
67.97
57.95
76.96
76.96 74.96
56.79
Methods
Factory
Dining Table
AP"
PQ1
F11
Pret
Rect
m
AP1
PQ1
F1
Pref
Rect
mloU>
Mask2Hoomer([L1]
40.25
53.54
57.60
99.01
40.61
37.76
77.65
84.61
87.42
97.44 79.28
76.80
D=NeRF[46]
17.33
17.08
21.29
25.35
18.35
20.72
74.05
57.15
69.30
59.35 83.27
61,82
NVFROurs, & 0.5)
64.82
57.75
77.93
85.72
71.44
49.70
91.17
69.49
83.31
75.96
92.23
66.13
NVFi(Ours, A 0.06)
98.86
80.17
99.09
99.09
99.09
69.07
98.01
91.81
98.95
98.99
98.92
76.68
21

---

Interpolation
Extrapolation
Interpolation
Extrapolation
t= 0
0.5
t=1
t=1
t=
t == 0.5
t=
t=1
T-NoRE
D-NeRF
TiNeu Vox
I-NeRF (PINN)
HexPlane (PINN)
NVTT (Ours)
Grrund Truth
Figure 6: Qualitative results of objects Fan and Telescope.
22

---

Interpolation
Extrapolation 1
Interpolation
Extrapolation [
0.5
[ 0.5
t=1
T-NoRF
D-NeRF
ToNeu
T-NeRF (PINN)
HexPlane (PINN)
NYFi (Ours)
Ground Truth
Figure 7: Qualitative results of objects Shark and Whale.
23

---

Extrapolatio -=1 Extrapolie t=1
Interpolation
Extrapolation E=1
Interpolation
t= 0
t= 0.5
0.5.5
t=1
t= 0
t= 0.5
T-NeRF
D-NeRF
TiNeu
T-NeRt PRINO
T
HexPlannd Hes
(PINN)
NVFI (Our)
Ground Truth
Figure 8: Qualitative results of scenes Gnome House and Chessboard.
24

---

Interpolation
Extrapolation t t=
Interpolation
Extrapolation 1
t= 0.5
=
1
0
t 0.5
t=
t =
T-NeRF
D-NeRF
TiNcu
T-NeRF (PINN)
HexPlane (PINN)
NVEI (Ours)
Ground Truth
Figure 9; Qualitative results of scenes Dining Table and Factory.
25

---

Interpolation t 0.33
Extrapolation
0.17
0.50
t= 0.67
t 0.83
t= 1
Imsage
Mask2- Former
D-NeRF
NVFi (Ours)
GT
aa)
Gnome House
Interpolation
Extrapolation
Image
Mask2- Former
D-NeRF
NVFI
(Ours)
GT
(b) Chessboard
Figure 10: Qualitative results of semantic scene decomposition for Gnome House and Chessboard.
26

---

Interpolation
Extrapolation
t= 0
t= 0.17
t= 0.33
t= 0.50
t= 0.67
t= 0.83
t= 1
Image
Mas12- Forner
D-NeRF
NVEi (Ours)
GT
(c) Dining Table
Extrapolatio Extrapolation
Interpolation
Image
Musk2- Former
D-NeRF
NVFI (Ours)
GT
(d) Factory
Figure 11: Qualitative results semantic scene decomposition for Dining Table and Factory.
27

---

Interpolation
Extrapolation
0
t t= 0.17
t t= 0.33
t= 0.50
t= 0.67
t= 0.83
t= 1
Image
NVTi (Ours)
Bat
Image
NVTI (Ours)
(b) Falling Ball
Imaee
NVTI (Ours)
(c) Fan
ImtRe
NVFI (Ours)
(d) Shark
NVFI (Ours)
(e) Telescope
Imago
NVFI (Ours)
(f) Whale
Figure 12: Qualitative results of semantic scene decomposition for Dynamic Object dataset.
28

---

Interpolation
Extrapolation
t= 0
t= 0.17
t= 0.33
0.50
t= 0.67
t= 0.83
t=1
Source
Target
Source
Target
Source
Target
Figure 13: Qualitative results for objectwise motion transfer.
29


Reviews:
[{"review_id": "PPgeu0IiYt", "reviewer": "Reviewer_e7NB", "summary": "The paper presents the framework to simultaneously learn the geometry, appearance, and velocity from only video frames. The paper's contribution lies in three directions: 1) Keyframe dynamic radiance fields to learn the time-dependent volume density and appearance. 2) Interframe velocity field to learn the time-dependent 3D velocity and 3) joint keyframe and interframe optimization to train keyframe and interframe fields together with physics-informed constraints. The main contribution lies in the joint keyframe and inter-frame optimization, which introduces three loss functions that help precisely learn to disentangle object masks, types, and materials. ", "questions": "Q1) From qualitative results, the dataset used in this work has limited spatial resolution. So, how would the approach scale to real-looking scenes with larger resolution?\n", "limitations": "See the weaknesses section. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) The framework is simple to understand yet elegant in achieving many tasks. \n\nS2) The paper is well written, with detailed derivation of major contributions. Easy to understand. \n\nS3)  The work presents Dynamic Object and Dynamic Indoor Scene datasets. \n\nS4) The evaluation of the proposed approach is well presented on multiple baselines and downstream tasks (Frame extrapolation, Scene decomposition, Motion transfer). The proposed approach outperforms the existing approaches.  \n", "weaknesses": "W1) All the evaluation results are presented on a synthetic dataset where the motion information is constrained ideally. It is unclear from the experiments how well the results translate to real-world scenarios. \n\nW2) All the results presented in this work are evaluated only on the authors' dataset. It is vital for the paper to present results on publicly available datasets to establish benchmarks (at least for a few downstream tasks). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: All the evaluation results are presented on a synthetic dataset where the motion information is constrained ideally. It is unclear from the experiments how well the results translate to real-world scenarios.**\n\n**Q2: All the results presented in this work are evaluated only on the authors' dataset. It is vital for the paper to present results on publicly available datasets to establish benchmarks (at least for a few downstream tasks).**\n\n**A:** Thanks for the suggestions and we agree that establishing a benchmark on a public and real-world dataset is crucial for the field of study. We evaluate our method on the real-world dataset: NVIDIA Dynamic Scene[1]. It captures real-world dynamic scenes by a static camera rig with 12 cameras. For each scene, we clip 60 frames with reasonable and predictable motions. We reserve the first 46 frames at randomly picked 11 cameras as the training split, *i.e.*, 506 frames, while leaving the 46 frames at the remaining 1 camera for testing interpolation ability, *i.e.*, 46 frames for novel view synthesis within the training time period, and keeping the last 14 frames at all 12 cameras for evaluating future frame extrapolation, *i.e.*, 168 frames. As shown in the Table 1 in Author Rebuttal, our method achieves significantly better results in the challenging task of future frame extrapolation. Figure 1 shows the qualitative results in the appended PDF. Due to the time limit, we can only provide scores on two scenes. More experiments are still running and we will add complete results in the next version.\n\n\n**Q3: From qualitative results, the dataset used in this work has limited spatial resolution. So, how would the approach scale to real-looking scenes with larger resolution?**\n\n**A:** This is a very good point. In the newly added NVIDIA Dynamic Scene dataset [1], the spatial scale of the real-world scene ``Truck\" is about 20x10x10 meters, which is clearly larger than the two synthetic datasets in the main paper. From our new results, we can see that our method can easily scale up.\n\n[1] J. S. Yoon. et al, Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera. CVPR, 2020."}, {"review_id": "qjuVs8rJHy", "reviewer": "Reviewer_c2wX", "summary": "The paper aims to model dynamic 3D scenes from multiple-videos. The method simultaneously learns for geometry, appearance, and physical velocity of the 3D scenes from video frames. The show different application like future frame extraploation, unsupervised semantic 3D scene decomposition and dynamic motion transfer using the current framework. They further proposed two dynamic 3D datasets with extensive experiments. ", "questions": "How does the method perform on other dyanmic scene datasets. even using simple multi-view datasets like panoptic studio should be explored to see the effect of the algorithm.\n\nAre the camera poses already provided for the dataset?", "limitations": "limitations have been discussed", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper uses three major components to tackle the dynamic scene, \nthe first is using the keyframe dynamic radiance field to compute radiance field at different time instances would be a good initial 3D estimator. \n\nthe second is to use velocity field to interpolate the intermediate frames instead of recomputing NERF per frame which is very expensive.\n\nFinally a joint keyframe and velocity based interframe optimization produces an smooth trajectory and reconstruction of the dynamic scene.\n\nInstead of just showcasing the advantage of the method, displaying the robustness of the algorithm to multiple downstream tasks like future frame Extrapolation signifies the advantage of this method. \n\nGood ablation study and the introduced datasets are helpful for future research in dynamic scene understanding.", "weaknesses": "There has been many datasets proposed to tackle the problem of dynamic scene understanding, What is the advantage or difference with respect to these datasets has not been well studies. Specifically, Line 187 states that all of the datasets currently available are not useful for this task. It would be interesting to see the effect of the proposed method on one these datasets to understand the advantages and the disadvantages of the proposed algorithm.\n\nEffect of choosing keyframe is specific to the dataset. Since the current dataset has only one moving object may be using 16 frames is sufficient. The ablation should show more experiments to justify the keyframe selection process.\n\nThe effect of the number of cameras is not explored in the analysis. Since you are using dynamic scenes, the effect of fewer frames should be analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: There has been many datasets proposed to tackle the problem of dynamic scene understanding, What is the advantage or difference with respect to these datasets has not been well studies.**\n\n**A:** The primary goal of our method is to model dynamic 3D scenes for future frame extrapolation, by means of learning underlying physical velocities (*i.e.*, meaningful motions) in the continuous 3D space. In contrast, existing dynamic 3D scene modeling techniques and the commonly used datasets in the literature are mainly designed for novel view rendering/interpolation within the training time period, rather than for extrapolation beyond the training time period. For example, the dynamic motion patterns in existing datasets such as DyNeRF dataset [2] are usually chaotic and lack predictable physical movements. This research gap motivates us to propose two synthetic datasets with diverse and predictable motion patterns in the main paper.  \n\nNevertheless, as also suggested by other reviewers, we further pick up a number of meaningful 3D scenes from the real-world NVIDIA Dynamic Scene dataset [1]. The Table 1 in Author Rebuttal shows the superiority of our method for future frame extrapolation. \n\n\n**Q2: Specifically, Line 187 states that all of the datasets currently available are not useful for this task. It would be interesting to see the effect of the proposed method on one these datasets to understand the advantages and the disadvantages of the proposed algorithm.**\n\n**A:** Thanks for the suggestion. As requested, we train our model on two typical scenes of DyNeRF dataset [2]. In these scenes, a man is pouring coffee / flaming a steak, and the hand and coffeemaker / flame are just undergoing random motions. As shown in Figure 3 in the appended PDF, not surprisingly, our method is not able to learn meaningful physical velocities and it is impossible to have correct extrapolations on these chaotic dynamics. Actually, our method simply predicts static frames for future extrapolation.\n\n\n**Q3: Effect of choosing keyframe is specific to the dataset. Since the current dataset has only one moving object may be using 16 frames is sufficient. The ablation should show more experiments to justify the keyframe selection process.**\n\n**A:** We highly appreciate this suggestion and conduct an ablation study on the number of keyframes on our Dynamic Indoor Scene dataset. As shown in the Table 2 in Author Rebuttal, we find that fewer keyframes tend to have better performance, demonstrating that our keyframe based optimization strategy is actually very flexible and effective. More experiments are still running and we will add these results in the next version. \n\n\n**Q4: The effect of the number of cameras is not explored in the analysis. Since you are using dynamic scenes, the effect of fewer frames should be analyzed.**\n\n**A:** Thank you for this advice and we conduct an additional ablation study on the number of training cameras on our Dynamic Objects dataset. The Table 3 in Author Rebuttal shows the quantitative results. As expected, given fewer training camera views, the performance of our method drops sharply, primarily because the extremely sparse views are unlikely to capture sufficient visual information for physical motion learning.  \n\n\n**Q5: How does the method perform on other dynamic scene datasets. even using simple multi-view datasets like panoptic studio should be explored to see the effect of the algorithm.**\n\n**A:** As also suggested by other reviewers, we turn to evaluate our method on another real-world dataset: NVIDIA Dynamic Scene dataset [1]. Results are supplied in above Q1. \n\n\n**Q6: Are the camera poses already provided for the dataset?**\n\n**A:** Yes, all camera poses are given in the datasets. Nevertheless, it is very interesting to explore  simultaneous camera pose estimation and scene modeling in the future.\n\n[1] J. S. Yoon. et al, Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera. CVPR, 2020.\n[2] T. Li. et al, Neural 3d video synthesis from multiview video. CVPR, 2022."}, {"review_id": "75stECzrfQ", "reviewer": "Reviewer_hQnr", "summary": "This paper proposes a model for dynamic 3D scenes from multi-view videos. Different with previous method, this paper proposes a physical velocity field instead of a deformation field. The method uses a dynamic radiance field to model key frames, and use a velocity field to warp the key frame to intermedia frames. For optimization, the photometric loss of rendered keyframes and interframes are optimized together, along with several PINN terms. Also, the authors show some applications of their model: future frame extrapolation, semantic 3D scene decomposition, dynamic motion transfer. Two new synthetic datasets are introduced.", "questions": "1.\tFor motion transfer, how would the learnt velocity field align with new objects? Put original images of Gnome in Figure 3 (c) may help understanding.", "limitations": "The authores addresed the limitations, including no experiment on real dataset.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1.\tAlthough velocity field has been introduced to this area (Neural Radiance Flow for 4D View Synthesis and Video Processing), this paper introduces a more complete solution, including the PINN terms and the warping mechanism.\n2.\tThe implementations of T-NeRF_PINN and HexPlane_PINN are not clear to me, but the future frame extrapolation of the proposed method is impressive.\n3.\tTwo new synthetic datasets are proposed.", "weaknesses": "1.\tThe method shows inconsistent performance on interpolation. \n2.\tFor dense frame optimization method, T-NeRF is not an appropriate method to compare with, since it is a very naïve solution which simply adds a time dimension to MLP input. To prove the effectiveness of the proposed method, I think NSFF (Neural Scene Flow Field) is a reasonable and meaningful method to compare. NSFF also has warping constraints.\n3.\tThe proposed method is basically a canonical based method which separate the whole video into multiple sections and assign a canonical model for each section. From this aspect, it is not so fair to compare with TiNeuVox in this way. How would TiNeuVox perform if also assign the same number of canonical models to it?\n4.\tThe potential application of motion transfer is not clear to me. \n5.\tNo interpolation performance for ablation study.\n6.\tNo experiment on real dataset. NHR dataset (Multi-view Neural Human Rendering) may be an appropriate dataset for this paper.\n7.\tFrom my aspect, the authors spent too much space on applications. The performance of the interpolation is more important for me to identify the ability of the proposed method to model the dynamic scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The method shows inconsistent performance on interpolation.**\n\n**A:** Comprehensive and consistent results for the interpolation of our method are provided in the Appendix. We are not clear about the request of this comment, but we are always open to discussion with the reviewer.\n\n\n**Q2: ... T-NeRF is not an appropriate method to compare with, ... I think NSFF (Neural Scene Flow Field) is a reasonable and meaningful method to compare.**\n\n**A:** Thanks for the valuable suggestion. We add NSFF as a new baseline on our Dynamic Indoor Scene dataset, as NSFF is not suitable for white backgrounds in our Dynamic Object dataset. As shown in the Table 4 in Author Rebuttal, NSFF indeed obtains much better interpolation results than the naive T-NeRF, but it still fails to extrapolate future frames. Figure 4 shows the qualitative results in the appended PDF. We will add this new baseline in the main paper in the next version.\n\n\n**Q3: ... How would TiNeuVox perform if also assign the same number of canonical models to it?**\n\n**A:** This is a very good point to discuss. TiNeuVox is a deformation field based model. If we use several canonical spaces, each canonical space requires a distinct deformation field to deform the latter frames back to it. This means that using multiple canonical spaces is equivalent to slicing the dataset into several pieces, and several TiNeuVoxs are trained on each piece respectively. Then the extrapolation is only related to the final canonical space and deformation field. This strategy actually cannot significantly increase the ability of TiNeuVox. \n\nTo verify this, we conduct additional experiments for TiNeuVox with multiple canonical spaces. From the Table 5 in Author Rebuttal, we can see that TiNeuVox with the same 16 canonical spaces still fails to obtain satisfactory extrapolation results as ours, showing the superiority of our learning of physical velocity fields. Due to time limit, more experiments are still running and we will include complete results in the next version. \n\n\n**Q4: The potential application of motion transfer is not clear to me.**\n\n**A:** One interesting application is character animation using motion transfer. There are some demos in Appendix Figure 12 and in the final part of the demo. Another potential application is to edit the reconstructed scene. For example, we can replace the objects with other ones, which are not in the original dynamic scene. Unarguably, there could be many other exciting use cases and we hope that our method could unlock new opportunities.\n\n\n**Q5: No interpolation performance for ablation study.**\n\n**A:** The interpolation performance is in Appendix Table 5. Thanks for your reminder and we will include this in the main paper in the next version. \n\n\n**Q6: No experiment on real dataset. NHR dataset (Multi-view Neural Human Rendering) may be an appropriate dataset for this paper.**\n\n**A:** Thanks for the suggestion. We find that NHR dataset is a point cloud rendering dataset, which is not suitable for our method. Alternatively, we evaluate our method on the real-world NVIDIA Dynamic Scene dataset[1]. The Table 1 in Author Rebuttal shows the superiority of our method for future frame extrapolation. \n\n\n**Q7: From my aspect, the authors spent too much space on applications. The performance of the interpolation is more important for me to identify the ability of the proposed method to model the dynamic scenes.**\n\n**A:** We agree that interpolation is indeed an important task, which can be seen from a large number of existing research works in recent two years.  In the meantime, we also strongly argue that the future frame extrapolation ability is essential for many intelligent machines, and related research is still in its infancy. Unfortunately, existing interpolation techniques fail to predict future frames as shown in our experiments. In this regard, we hope that our proposed method could inspire more advanced works in the future.\n\n\n**Q8 : For motion transfer, how would the learnt velocity field align with new objects? Put original images of Gnome in Figure 3 (c) may help understanding.**\n\n**A:** A naive motion transfer requires objects to have similar sizes. This is illustrated by Figure 6 in the attached PDF and the video demo Motion Transfer section. For a general motion transfer, more advanced techniques such as shape registration and alignment may be applied to deal with variable sizes of objects.\n\n\n**Q9: The implementations of T-NeRF$ _{PINN} $ and HexPlane$ _{PINN} $ are not clear to me, but the future frame extrapolation of the proposed method is impressive.**\n\n**A:** T-NeRF_pinn is implemented as an original T-NeRF along with a velocity field (MLPs) which is the same as our VeRF. HexPlane_pinn is implemented as a HexPlane along with the same velocity field (MLPs). Different from VeRF, the two models are trained as PINN. \n\nFirst of all, the RGB loss of given supervision is used to train the density and color. In PINN framework, it can be also regarded as boundary conditions. Then we regard the whole dynamics as a transport problem, and more details are in Appendix A.4. We add the transport loss for density and colors, and the physics loss for the velocity field as PINN PDE constraints. The main difference between these two models and VeRF is that, in VeRF, gradients can flow through the velocity field from the RGB loss, while in *$_{PINN}$ models the velocity field is only estimated by PINN losses.\n\nWe will add these explanations in the next version.  \n\n[1] J. S. Yoon. et al, Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera. CVPR, 2020."}, {"review_id": "bL7HvAVBHG", "reviewer": "Reviewer_8aAM", "summary": "This paper proposes a new representation of dynamic scenes, using keyframe NeRF + Velocity Field.\nThis representation disentangles appearance & geometry from velocity, which allows many exciting applications like future frame extrapolation, unsupervised semantic 3D scene decomposition, and dynamic motion transfer.\nObtaining such representation from videos requires a carefully designed system and physics-constrained losses. \nTo validate this model, this paper collects two new synthetic datasets and show impressive results on these datasets.", "questions": "I am generally positive about this paper but I think the limitations of this paper (and the questions in that bullet) should be discussed in detail.", "limitations": "The current version doesn't give a convincing discussion about limation and potential broad impact. See the weakness for detailed comments.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Learning a velocity field from videos is an exciting direction and could be impactful for future research, especially for physics-aware 3D/4D tasks. This paper makes an impressive attempt in this direction.\n2. The method is well-motivated and intuitively reasonable. The whole idea is easy to follow and sound.\n3. The results are impressive, and the applications of the velocity fields are exciting, especially the 3D semantic field segmentation results could clearly show different parts of objects.", "weaknesses": "1. Lacking limitation discussions. The current paper clearly has several limitations and they are expected to be discussed in detail in the paper. (1) As briefly mentioned in the broad impact, this paper doesn't have real-world data to validate whether it could work on real scenes. (2) This method seems to require videos with many views at the same timestamp as inputs. I am curious how this method works with monocular videos/sparse views. (3) How does the proposed method deal with abrupt motions and non-typology changes? For summary, the paper should discuss the potential limitations in detail. \n2. Some presentations could be improved. It would be nice to give a brief explanation about losses and PINN used in this paper. And Algorithm 1 table could be more precise. The current paper seems to omit too many technical details and the meaning of some losses is confusing. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Lacking limitation discussions. The current paper clearly has several limitations and they are expected to be discussed in detail in the paper. (1) As briefly mentioned in the broad impact, this paper doesn't have real-world data to validate whether it could work on real scenes.**\n\n**A:** As also suggested by other reviewers, we conduct additional experiments on the real-world NVIDIA Dynamic Scene dataset. It captures real-world dynamic scenes by a static camera rig with 12 cameras. For each scene, we clip 60 frames with reasonable and predictable motions. We reserve the first 46 frames at randomly picked 11 cameras as the training split, *i.e.*, 506 frames, while leaving the 46 frames at the remaining 1 camera for testing interpolation ability, *i.e.*, 46 frames for novel view synthesis within the training time period, and keeping the last 14 frames at all 12 cameras for evaluating future frame extrapolation, *i.e.*, 168 frames. As shown in the Table 1 in Author Rebuttal, our method achieves significantly better results in the challenging task of future frame extrapolation. Figure 1 shows the qualitative results in the appended PDF. Due to the time limit, we can only provide scores on two scenes. More experiments are still running and we will add complete results in the next version. \n\n\n**Q2: (2) This method seems to require videos with many views at the same timestamp as inputs. I am curious how this method works with monocular videos/sparse views.**\n\n**A:** As requested, we additionally evaluate our method on the truck scene from NVIDIA Dynamic Scenes in a monocular way. In particular, for every timestamp, only one camera from the 12 cameras is used. Due to the depth ambiguity of the monocular video, it is very hard to disentangle the foreground and background of the scene. So the velocity field will influence the background scene as illustrated in Figure 2 in the appended PDF. We leave this challenging monocular setting for our future exploration. \n\nAs suggested by reviewer c2wX, we also make an ablation study on the camera number on our own Dynamic Objects dataset. The Table 3 in Author Rebuttal shows the quantitative results. As expected, given fewer training camera views, the performance of our method drops sharply, primarily because the extremely sparse views are unlikely to capture sufficient visual information for physical motion learning.  \n\n\n**Q3: (3) How does the proposed method deal with abrupt motions and non-typology changes?**\n**A:** This is a good question. We evaluate our model on two scenes from DyNeRF dataset. Since the abrupt motions of arm and flame / coffeemaker in hands are actually not predictable, we clearly fail on the extrapolation task, even though we can get promising results in interpolation. Figure 3 of the appended PDF shows the qualitative results. \n\nAbove all, our model aims to learn the predictable physical dynamics, and is not able to predict abrupt motions. We will discuss this limitation in the next version. \n\n\n**Q4: Some presentations could be improved. It would be nice to give a brief explanation about losses and PINN used in this paper**\n\n**A:** We thank the reviewer for this suggestion. To explain the losses better, we first need to address that the whole dynamics is regarded as a transport problem, as shown in Appendix A.4. In particular, the density and appearance features of objects are transported by a velocity field, and the velocity field is transported by itself according to some unobserved hidden forces. So the losses can be divided into two types. The first type is the RGB rendering loss for both keyframes and interframes, which are MSE between the rendering pixel colors and the ground-truth pixel colors. \n\nThe second type is the PINN PDE losses, where the divergence-free loss is used to constrain the mass conservation of objects in the scene, and the momentum conservation loss is to learn the hidden forces and ensure the extrapolation in a reasonable pattern. \n\nPINN losses are implemented as follows: 1) We uniformly sample points in time and space dimensions. 2) We use torch.autograd to evaluate the jacobian of velocity w.r.t the input position and time. 3) We put those terms to calculate the LHS of Equation 7 and make it an L2 loss. Moreover, since the acceleration (hidden forces) is not observable, it is totally estimated by PINN losses. \n\nWe will add these explanations in the next version.\n\n[1] J. S. Yoon. et al, Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera. CVPR, 2020."}, {"review_id": "09vswwI12u", "reviewer": "Reviewer_3vYE", "summary": "This paper presents an algorithm for the realization of novel view synthesis in dynamic scenarios, leveraging multi-view video data. In order to tackle this task, the paper introduces two key components: a keyframe dynamic field and an interframe velocity field. These fields serve the purpose of accurately representing the motion, geometry, and color information inherent in the recorded scenario. By incorporating these fields into the algorithm, the proposed approach aims to achieve effective synthesis of new views in dynamic scenarios.\n\n\n\n\n", "questions": "### Performance-related questions:\n\n1. When compared to previous algorithms, such as D-Nerf, which rely on a canonical field, this paper employs Hexplane as a backbone. Is the improvement observed in this algorithm attributed to the utilization of Hexplane?\n2. In comparison to HexPlane, which employs a single representation to learn features for the entire scenario, this paper learns multiple key-frame representations. Does the enhancement in performance stem from the usage of multiple representations instead of just one?\n### Innovation-related questions:\n\n1. Instead of directly learning the interframe velocity field, the proposed algorithm initially learns an acceleration field. Why was this approach chosen, and what evidence supports this decision?\n2. Instead of learning the precise positions of key-frames, this paper employs uniform sampling for key-frame selection. Does this decision align with the inherent characteristics of dynamic scenarios, considering that motion within such scenarios may not exhibit uniform speeds?\n\n### Dataset-related question:\n\n1. What are the advantages of the proposed datasets in comparison to previous datasets used in similar studies?\n", "limitations": "The primary limitation of this paper revolves around the level of innovation exhibited by the proposed algorithm and the dissimilarity between the proposed datasets and their predecessors.\n\n\n\n\n", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper introduces two novel 3D datasets for dynamic object scenarios and dynamic indoor scenarios. Given the existing limitations in available datasets for view synthesis in dynamic scenarios, the introduction of these new datasets is expected to significantly enhance the progress and development of algorithms in this field. Additionally, while this paper presents innovative contributions, it also incorporates comprehensive experiments to support its claims. Furthermore, the paper is well-written and maintains a coherent structure, making it easily understandable and accessible to readers.\n\n\n\n\n", "weaknesses": "The primary concept explored in this paper involves the acquisition of a key-frame representation and the establishment of connections between key frames and intra-frames through the utilization of intraframe velocity. However, it is crucial to acknowledge that the notion of learning a canonical field and a deformation field has been previously introduced and extensively discussed in several notable publications. Notably, papers such as \"D-NeRF: Neural Radiance Fields for Dynamic Scenes\" (with over 500 citations) and \"Nerfies: Deformable Neural Radiance Fields\" (also with over 500 citations) have extensively addressed and explored this concept. The concept of learning a deformation field has also been extensively studied and advanced in previous research, as evidenced by the paper \"Dynamic View Synthesis from Dynamic Monocular Video\" which has accumulated over 100 citations. Moreover, while this paper introduces two novel datasets, it is important to note that there exist minimal distinctions between the proposed datasets and preexisting ones such as the Nvidia dataset, Nerfie dataset, and Iphone dataset.\n\n\n\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  ... D-NeRF ... Nerfies ... explored this concept.**\n\n**A:** We agree with the reviewer that both prior works are pioneering in this field. Nevertheless, we observe that only learning the deformation actually does not truly understand the physical motions. That means they are good at interpolation but fail at extrapolation. Therefore, we focus on learning a velocity field. \n\nCompared to the deformation field, the velocity field has several advantages. The first is cycle consistency. A deformation field is always unidirectional - backwardly or forwardly. If we want both deformation fields to be cycle consistent, we need to add another loss to regularize. However, we can simply integrate forwardly or backwardly on one single velocity field to get a consistent position. Secondly, the motion modeled by the velocity field can be accumulated and is continuous. This means we can easily model the difference between any two timestamps. \n\nWith these motivations, we focus on designing an effective framework to estimate the underlying velocity field instead of the deformation field. In order to show the effectiveness, we introduce D-NeRF and TiNeuVox (another SOTA canonical space and deformation model, which outperforms Nerfie in many cases) as our baselines. Extensive experiments demonstrate the superiority of our learned velocity fields over deformation fields. We will include these explanations in the next version.\n\n**Q2: ... \"Dynamic View Synthesis from Dynamic Monocular Video\" ...**\n\n**A:** Thanks for suggesting this paper. It is a successful method that combines a dense-frame model and scene flows. However, the used scene flow can only learn motions for one step/frame. It is not a continuous motion, and there is no way to regularize future (unseen) flows. The reason we do not include this method as our baseline is that it requires depth as input. It relies on too strong priors and is not general enough.\n\n**Q3: Moreover ... Iphone dataset.**\n\n**Q4: What are the advantages ... in similar studies?**\n\n**A:**\nThe primary goal of our method is to model dynamic 3D scenes for future frame extrapolation, by means of learning underlying physical velocities (*i.e.*, meaningful motions). Because most dynamic scenes in existing datasets are usually chaotic and lack predictable physical movements. We turn to propose two synthetic datasets with diverse and predictable motion patterns in the main paper.  Thanks for suggesting the new real-world NVIDIA Dynamic Scene dataset [1]. The Table 1 in Author Rebuttal shows the superiority of our method for future frame extrapolation. \n\n**Q5: When compared to ... utilization of Hexplane?**\n\n**A:** This is a great point and our extensive experiments can respond to this question in the following two aspects. Firstly, we include HexPlane based model (HexPlane\\_PINN) as our baseline, whose performance is worse than ours, so our effectiveness is not just because of a new backbone. Secondly, for those scenes successfully reconstructed by D-NeRF, we can notice a huge gap between interpolation and extrapolation in Appendix Table 6. As shown in Appendix Figures 6-10, D-NeRF also predicts wrong motions in the extrapolation task. Based on this, the main reason why D-NeRF fails is due to its misunderstanding in motions \n\n**Q6: In comparison to HexPlane ... instead of just one?**\n\n**A:**\nA HexPlane model includes feature tensors in dimensions XY, XZ, YZ, XT, YT, ZT, and all of these feature tensors form the single representation. In our method, we take the same representation as HexPlane. The main difference is that the T dimension in HexPlane is always half of the total frames, and for interframes, they use bilinear interpolation to get the values, while the T dimension in VeRF is the number of keyframes, which is much fewer than HexPlane, and we use velocity field to transport interframes to keyframes. In general, both our model and HexPlane employ a single representation. \n\n**Q7:  ... an acceleration field ... decision?**\n\n**A:** For clarity, we do not initially learn an acceleration field. We learn it only by our physics losses along with learning the velocity field simultaneously. Here we briefly discuss why this is needed. Usually, the hidden velocity is not a constant velocity. For example, in a falling ball scene, there is gravitational acceleration, and in scenes with rotation, there is centripetal acceleration. In order to learn a reasonable velocity field following physics rules, we must estimate an accompanied acceleration field to guide our extrapolation. Note that, we do not add any priors and supervision to these accelerations. The only gradients passing through these accelerations are from the momentum conservation PINN loss. More details about the PINN loss are in Appendix A.4. \n\nWe will clarify this point in the next version. \n\n**Q8: Instead of ... uniform speeds?**\n\n**A:** There are two reasons why we use uniform distributed keyframes. Firstly, uniform keyframes can also handle non-uniform motions. For example, in our datasets, the falling ball scene has a gravitational acceleration, and for scenes such as telescope, their rotations require a centripetal acceleration. None of them is uniform motion, and either the direction or the magnitude of the velocity keeps changing over time. As shown in the experiments, our model works well in these cases. \n\nSecondly, uniform keyframes have consistent time intervals between two keyframes. Thus we can easily control the integration step (influence GPU memory and model performance) and the integration stability (influences model performance). However, if the time intervals between keyframes are different, in order to control the GPU memory for a longer integration, we need to sacrifice the integration stability. We evaluate our model with random keyframe places on our Dynamic Scene dataset. Not surprisingly, the Table 6 in Author Rebuttal shows a decrease in performance. We will add these results in the next version."}]

Author rebuttal:
Firstly, we would like to thank all five reviewers for their valuable comments. We genuinely appreciate the time and effort invested in reviewing our paper. Based on the comments provided, we have made significant improvements to our paper:

- Additional experiments on public real-world datasets, including: a)
two distinct scenes derived from NVIDIA Dynamic Scene dataset, b) one scene from the monocular version of NVIDIA Dynamic Scene dataset, c) two scenes from DyNeRF dataset.

- Additional evaluation of a new baseline, Neural Scene Flow Fields (NSFF), on our Dynamic Indoor Scene dataset.

- Additional ablation studies about: a) the number of keyframes used on our Dynamic Indoor Scene dataset, b) the number of cameras used in our Dynamic Object datasets.

Due to the character limitations for responses in review rebuttal, we have cataloged all the relevant tables and details in this author rebuttal. We kindly request reviewers to refer to these tables and sections for a more detailed understanding. 


**Table 1:** Quantitative results of our method and baselines on the NVIDIA Dynamic Scene dataset.
|                   |            |               |   Truck   |            |               |           |
|:-----------------:|:----------:|:-------------:|:---------:|:----------:|:-------------:|:---------:|
|                   |            | **Interpolation** |           |            | **Extrapolation** |           |
|       **Model**       |    **PSNR**    |      **SSIM**     |  **LPIPS**   |    **PSNR**    |      **SSIM**     |   **LPIPS**   |
|      TiNeuVox     |   27.230   |   **0.846**   | **0.229** |   24.887   |     0.848     | **0.209** |
| HexPlane$_{PINN}$ |   25.494   |     0.768     |   0.337   |   24.991   |     0.768     |   0.325   |
|    VeRF (Ours)    | **27.276** |     0.840     |   0.235   | **28.269** |   **0.855**   |   0.220   |
|                   |            |               |  **Skating**  |            |               |           |
|                   |            | **Interpolation** |           |            | **Extrapolation** |           |
|       **Model**       |    **PSNR**    |      **SSIM**     |   **LPIPS**   |    **PSNR**    |      **SSIM**     |   **LPIPS**   |
|      TiNeuVox     | **29.377** |   **0.889**   | **0.202** |   24.224   |     0.878     |   0.220   |
| HexPlane$_{PINN}$ |   24.447   |     0.867     |   0.225   |   23.955   |     0.868     |   0.232   |
|    VeRF (Ours)    |   26.999   |     0.848     |   0.227   | **28.654** |   **0.896**   | **0.208** |

**Table 2:** Ablation study of the keyframe number on our Dynamic Indoor Scene dataset.
|           |            | Interpolation |           |            | Extrapolation |           |
|-----------|------------|---------------|-----------|------------|---------------|-----------|
| **Keyframes** | **PSNR**       | **SSIM**          | **LPIPS**     | **PSNR**       | **SSIM**          | **LPIPS**     |
| 8         | **30.321** | **0.871**     | **0.220** | **29.093** | **0.873**     | **0.225** |
| 16        | 28.000     | 0.862         | 0.226     | 26.235     | 0.839         | 0.237     |
| 32        | 29.764     | 0.851         | 0.255     | 26.634     | 0.828         | 0.247     |

**Table 3:** Ablation study of the camera number on our Dynamic Objects dataset.
|         |            | Interpolation |           |            | Extrapolation |           |
|---------|------------|---------------|-----------|------------|---------------|-----------|
| **cameras** | **PSNR**       | **SSIM**          | **LPIPS**     | **PSNR**       | **SSIM**          | **LPIPS**     |
| 12      | **29.027** | **0.970**     | **0.039** | **27.594** | **0.972**     | **0.036** |
| 6       | 25.689     | 0.954         | 0.051     | 25.114     | 0.959         | 0.122     |
| 3       | 21.460     | 0.912         | 0.088     | 21.370     | 0.917         | 0.084     |

**Table 4:** Quantitative results of NSFF on the Dynamic Indoor Scene dataset.
|       |            | Interpolation |           |            | Extrapolation |           |
|-------|------------|---------------|-----------|------------|---------------|-----------|
| **Model** | **PSNR**       | **SSIM**          | **LPIPS**     | **PSNR**       | **SSIM**          | **LPIPS**     |
| NSFF  | 29.365     | 0.829         | 0.278     | 24.163     | 0.795         | 0.289     |
| VeRF  | **30.321** | **0.871**     | **0.220** | **29.093** | **0.873**     | **0.225** |

**Table 5:** Quantitative results of TiNeuVox on the chessboard of our Dynamic Indoor Scene dataset.
|                         |            | Extrapolation |           |
|-------------------------|------------|---------------|-----------|
| **Model**                   | **PSNR**       | **SSIM**          | **LPIPS**     |
| TiNeuVox (1 Canonical)  | 19.718     | 0.765         | 0.310     |
| TiNeuVox (16 Canonical) | 21.394     | 0.812         | **0.233** |
| VeRF (Ours, K=18)       | **24.160** | **0.837**     | 0.259     |

**Table 6:** Non-uniform sampling for key-frame selection.
|            |        | Interpolation |       |        | Extrapolation |       |
|------------|--------|---------------|-------|--------|---------------|-------|
| **Keyframes**  | **PSNR**   | **SSIM**          | **LPIPS** | **PSNR**   | **SSIM**          | **LPIPS** |
| Uniform 16 | **29.027** | **0.970**         | **0.039** | **27.594** | **0.972**         | **0.036** |
| Random 16  | 28.463 | 0.966         | 0.043 | 24.489 | 0.959         | 0.045 |

Meta-review: The paper presents a new representation for modeling dynamic 3D scenes. The initial reviews were overall positive. There were concerns about the lack of comparisons with other state-of-the-art dynamic NeRF methods and real-world videos. The authors' rebuttal provided the comparisons with TiNeuVox and results on two datasets, effectively alleviating these initial concerns. After discussions, the reviewers converge to accept the paper.
Decision: Accept (poster)