SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinxi Li Ziyang Song Bo Yang
vLAR Group, The Hong Kong Polytechnic University
jinxi.li@connect.polyu.hk ziyang.song@connect.polyu.hk bo.yang@polyu.edu.hk
In this paper, we aim to model 3D scene dynamics from multi-view videos. Unlike the majority of existing works which usually focus on the common task of novel view synthesis within the training time period, we propose to simultaneously learn the geometry, appearance, and physical velocity of 3D scenes only from video frames, such that multiple desirable applications can be supported, including future frame extrapolation, unsupervised 3D semantic scene decomposition, and dynamic motion transfer. Our method consists of three major components, 1) the keyframe dynamic radiance field, 2) the interframe velocity field, and 3) a joint keyframe and interframe optimization module which is the core of our framework to effectively train both networks. To validate our method, we further introduce two dynamic 3D datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset. We conduct extensive experiments on multiple datasets, demonstrating the superior performance of our method over all baselines, particularly in the critical tasks of future frame extrapolation and unsupervised 3D semantic scene decomposition. Our code and data are available at https://github.com/vLAR-group/NVFi
The 3D world around us is constantly changing over time, where objects are falling, vehicles moving, and clocks ticking. Humans can effortlessly learn the geometry and physical properties of such dynamic 3D scenes, and further predict their future motions following the learned physics rules, just by watching the things for a few seconds. Giving machines such ability to automatically infer the geometry and physics of complex dynamic 3D scenes is essential for many cutting-edge applications in augmented reality, games, and the movie industry. Recent advances in the emerging area of neural radiance field [37] and its succeeding variants [46, 4, 6, 19, 75, 40] have shown excellent results in modeling dynamic 3D scenes such as deformable things [4, 58] and articulated objects [39, 55]. While showing superior performance in interpolating novel views within the observed time period, almost all these methods tend to fit training image sequences, without explicitly learning the physical properties such as object velocities, thus being unable to extrapolate and predict future motion patterns of 3D scenes.
More recently, a few studies [47, 14, 8, 32, 2] start to integrate physics priors into implicit neural representations to model dynamic 3D scenes such as floating smoke or simple moving objects. By introducing the governing PDEs, a.k.a., PINN [48], these methods demonstrate promising reconstruction of 3D scene geometry, appearance, velocity and/or viscosity fields. However, the learned physical properties are either tightly coupled with the target objects [14] in the scene or require additional foreground segmentation masks in the loop [32]. This means that the estimated physics knowledge from dynamic video frames is not clearly disentangled, thereby not transferable from one scene to another.
Figure 1: The three major components of our framework: Keyframe Dynamic Radiance Field, Interframe Velocity Field, and the Joint Keyframe and Interframe Optimization module.
In this regard, we ask an ambitious question: Can we learn disentangled physical properties alongside recovering geometry and appearance of dynamic 3D scenes purely from multi-view video frames? Among various physical properties in the scene, we choose to focus on velocity as it primarily governs scene movement dynamics. Such disentangled velocity fields, once successfully learned, are expected to unlock multiple desirable applications: 1) Future frame extrapolation and prediction beyond the observed time period in training. For example, after watching a football flying in the penalty area, we can predict what will happen next. 2) Dynamics transformation from one scene to another. For instance, after watching a bird flipping wings, we can imagine the same physical behavior on the body of an airplane. 3) Semantic decomposition of 3D scenes. Intuitively, once the velocity field of an entire dynamic 3D scene is learned, all individual objects or parts undergoing different moving patterns can be easily segmented, without needing any extra human annotations for training.
However, accurately learning the physical velocity of a whole 3D scene space is particularly challenging, primarily due to the lack of ground truth 3D velocity annotations, the unknown object types or materials in the scene, and the sparse yet long-time visual trajectory in training. In addition, when separately learning the velocity field, it is usually an under-constrained problem even with the commonly used PINN technique [14].
To tackle these challenges, we introduce a new general framework to simultaneously model the geometry, appearance, and disentangled velocity of a dynamic 3D scene only from multi-view video frames. In particular, as shown in Figure 1, our framework consists of three major components: 1) a keyframe dynamic radiance field to learn time-dependent volume density and appearance for every 3D point in space; 2) an interframe velocity field to learn time-dependent 3D velocity for every point as well; and 3) a joint keyframe and interframe optimization method together with physics informed constraints to train both networks. For the first component, it is flexible to adopt any of the existing time-dependent NeRF architectures such as HexPlane [6] or K-Planes [19]. For the second component, the neural network actually can be as simple as MLPs.
The core of our framework is the third component, where we explicitly apply three types of loss functions to jointly optimize both networks: 1) the keyframe photometric loss, 2) the interframe photometric loss, and 3) the governing PDE losses. With these losses, our framework can precisely learn disentangled velocity fields, without needing additional regularization terms on volume density or information on object masks, types, or materials. Overall, our framework models general dynamic 3D scenes by learning neural velocity fields with physical priors. Our method is named NVFi and our contributions are:
- We introduce a general framework to model dynamic 3D scenes as physics-informed radiance fields from multi-view videos, without requiring information on object types, materials, or masks.
- We design a neural velocity field together with a joint keyframe and interframe optimization method to effectively train the networks.
- We demonstrate three applications for the learned velocity fields on two newly collected dynamic 3D datasets and a challenging real-world dataset, showing superior results in future frame extrapolation, semantic decomposition, and velocity transferring across 3D scenes.
Static 3D Representations: Conventional representations for static 3D objects and scenes include voxels [13, 71, 72], point clouds [17], octrees [57, 64], meshes [28, 23], and primitives [77]. Due to the discretization issue of these explicit representations, the fidelity of 3D shapes is usually limited by spatial resolution and memory footprint. Inspired by the seminal works [10, 36, 41], recently, there
Keyframe Dynamic Radiance Field
Interframe Velocity Field
Figure 2: The left block illustrates the network architecture of our keyframe dynamic radiance field based on HexPlane [6], and the right block shows the architecture of our interframe velocity field.
has been a strong interest in representing 3D data in implicit neural functions. These methods are generally classified as: 1) occupancy fields (OF) [10, 36], 2) signed distance fields (SDF) [41], 3) unsigned distance fields (UDF) [12, 62], and 4) radiance fields (NeRF) [37, 3, 59]. Basically, these neural representations simply take 3D point locations as input and directly regress the corresponding point information, such as occupancy, distance to surface, color, semantics [76, 61], et al., via MLPs. Compared with traditional explicit representations, these implicit neural representations allow for continuous shape and appearance modeling at a particularly low memory footprint.
Dynamic 3D Representations: Recent techniques in this category are primarily built on the appealing NeRF architecture [37], thanks to its unprecedented level of fidelity in representing various 3D objects and scenes [6, 19]. Given video sequences of dynamic scenes, these NeRF based methods usually take the time  $t$  as an additional input dimension and then optimize the entire network using the standard photometric loss. Generally, the techniques can be classified as: 1) deformation-based methods [42, 58, 4, 46, 5, 18, 74], 2) flow-based methods [21, 33, 16, 63, 35], and 3) direct space-time based methods [69, 1, 31, 43]. In parallel, there are also a number of domain-specific NeRFs to model particular dynamic objects such as human bodies [44, 50, 45, 70, 68] and faces [4, 20, 66, 67]. Although all these methods have shown excellent performance in novel view synthesis on a wide range of dynamic datasets, they are primarily designed to interpolate frames within the time period of training data, lacking the ability to predict future frames. By contrast, our NVFi aims to model the underlying physical principles of 3D scenes, thus being easily able to extrapolate future dynamics.
Physics Informed Deep Learning: Unlike traditional numerical methods such as finite element methods, recent physics-informed neural networks (PINN) [48] represent tensor fields with neural networks, converting the PDE solutions into optimizing network weights via PDE-based loss functions. Inspired by the pioneering works [48, 52, 30], a plethora of succeeding works [51, 49, 25, 38, 65], have been developed, demonstrating excellent results in a wide range of applications such as acoustics [53, 56], fluids [60, 22, 15], 3D scene representations [8, 14, 32, 47], et al., thanks to its mesh-free formulation. More details can be found in the recent surveys [27, 24]. In our framework, we find that the commonly-used PINN constraints are insufficient to learn accurate 3D velocity fields. To tackle this issue, we further propose the keyframe based velocity propagation module.
3 NVFi
As shown in Figure 2, our framework consists of two neural networks together with their optimization methods. Given a set of images of a dynamic scene with known camera poses and intrinsics, the keyframe dynamic radiance field  $f_{\Theta}$  simply takes a 3D point  $\pmb{p} = (x,y,z)$ , viewing angle  $(\theta, \phi)$ , and timestamp  $t$  as input, directly regressing the volume density  $\sigma$  and color  $\pmb{c} = (r,g,b)$ . For the network architecture, we simply adopt the recent HexPlane [6] which shows excellent performance in efficiently modeling dynamic video frames, though other NeRF variants can also be used. Formally, our keyframe dynamic radiance field is defined below and implementation details are in Appendix.
$$
(\boldsymbol {\sigma}, \boldsymbol {c}) = f _ {\Theta} (x, y, z, \theta , \phi , t) \quad \Theta \text {a r e t r a i n a b l e p a r a m e t e r s}. \tag {1}
$$
For the interframe velocity field  $g_{\Phi}$ , it takes the 4D vector  $(x,y,z,t)$  as input, and predicts the 3D velocity  $\pmb{v} = (\nu_x,\nu_y,\nu_z)$  for point  $\pmb{p}$  at time  $t$ . For simplicity, the network  $g_{\Phi}$  is parameterized by simple MLPs, though more advanced architectures can be applied as well. Formally, the velocity field is defined below and implementation details are in Appendix.
$$
\boldsymbol {v} = g _ {\Phi} (\boldsymbol {p}, t) = g _ {\Phi} (x, y, z, t) \quad \Phi \text {a r e t r a i n a b l e p a r a m e t e r s o f M L P s .} \tag {2}
$$
With these two networks and training images sampled from a dynamic 3D scene, the key challenge is to effectively optimize these networks, such that the final learned velocity field is precise and disentangled, supporting future frame extrapolation, motion transfer, and semantic decomposition.
3.2 Optimization of Keyframe Dynamic Radiance Fields
Given dynamic video frames of  $T$  timestamps  $\{1\cdots t\cdots T\}$ , there are two potential strategies to optimize dynamic neural radiance fields.
- Strategy 1 - Dense Frame Optimization: This strategy uses all available video frames of a specific dynamic 3D scene in the training set to optimize a dynamic radiance field. This means that for the network  $f_{\Theta}$ , the time dimension  $t$  is densely sampled during optimization. Formally:
$$
f _ {\Theta} (x, y, z, \theta , \phi , t): \quad t \in \{1 \dots t \dots T \} \tag {3}
$$
However, it has two limitations: 1) It is inefficient to learn accurate 3D geometry and appearance because this strategy is somewhat equivalent to modeling a dense number of static radiance fields for all timestamps. 2) It is hard to obtain a disentangled physical velocity field for the entire 3D scene, as the change of physical geometries is tightly encoded within the dynamic radiance field.
- Strategy 2 - Canonical Frame Optimization: This strategy optimizes a canonical representation of 3D geometry and appearance, usually joined with another deformation or transportation network to warp the future ( $T - 1$ ) timestamps back to the first timestamp. It can be seen as:
$$
f _ {\Theta} (x, y, z, \theta , \phi , t): \quad t \in \{1 \} \tag {4}
$$
However, such a strategy has a strong assumption that the corresponding point appearances across different timestamps keep unchanged. Therefore, for dramatically changing 3D scenes, the optimized geometry, appearance, and possible jointly learned physics properties tend to be inferior.
Our Strategy - Keyframe Optimization: In this regard, we propose to adopt a keyframe based strategy to learn the dynamic radiance field  $f_{\Theta}$ . In particular, we uniformly sample  $K$  timestamps out of the total  $T$  stamps to optimize  $f_{\Theta}$ . Formally:
$$
f _ {\Theta} (x, y, z, \theta , \phi , t _ {k}): \quad t _ {k} \in \left\{\left[ T / K \right], 2 \left[ T / K \right], 3 \left[ T / K \right], \dots T \right\} \tag {5}
$$
Color images are rendered from the above keyframe dynamic radiance fields by sampling points along rays. For each pixel, i.e. ray  $\mathbf{r}$ , in a keyframe at time  $t_k$ , the appearance  $\mathbf{C}(\mathbf{r},t_k)$  is obtained by volume rendering of NeRF [37]. Then the network  $f_{\Theta}$  can be optimized by the following keyframe photometric loss. Details of the rendering equation are in Appendix.
$$
\ell_ {\text {k e y f r a m e}} = \left| \left| \boldsymbol {C} (\boldsymbol {r}, t _ {k}) - \bar {\boldsymbol {C}} (\boldsymbol {r}, t _ {k}) \right| \right|, \quad \text {w h e r e} \bar {\boldsymbol {C}} (\boldsymbol {r}, t _ {k}) \text {i s} \tag {6}
$$
This keyframe optimization strategy, albeit simple, has two unique advantages: 1) It allows to sufficiently and accurately fit the sparsely sampled dynamic 3D scene geometry and appearance given the same network capacity. 2) It allows the remaining interframes belonging to the  $(T - K)$  time stamps to be used for optimizing a disentangled velocity field, as discussed in Section 3.3.
3.3 Optimization of Interframe Velocity Fields
As to our separate interframe velocity field  $g_{\Phi}$ , it is impossible to directly supervise it using ground truth labels as they cannot be collected in practice. However, there are some physics rules to regularize the velocity field. The objects are transported by the velocity field, and the velocity field is transported by itself according to some unobservable hidden forces. In order to keep the mass and appearance of the objects, the velocity field needs to be divergence-free and obeys the basic law of momentum conservation, whose details are in Appendix. Note that, more complex physics dynamics beyond the daily 3D scenarios are out of the scope of this paper. In this regard, our velocity field, i.e.,  $\pmb{v} = g_{\Phi}(\pmb{p}, t)$ , needs to firstly satisfy the following two constraints.
$$
\nabla_ {\boldsymbol {p}} \cdot \boldsymbol {v} = 0, \quad \frac {\partial \boldsymbol {v}}{\partial t} + \boldsymbol {v} \cdot \nabla_ {\boldsymbol {p}} \boldsymbol {v} = \boldsymbol {a} \tag {7}
$$
We simply turn these PDEs into the following two PINN losses [48] to optimize the velocity field. Here we use  $\pmb{\nu}(\pmb{p},t)$  as the network to avoid abuse of notation.
$$
\begin{array}{l} \ell_ {\text {d i v e r g e n c e \_ f r e e}} = \frac {1}{N M} \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M} | | \nabla_ {\boldsymbol {p} _ {n}} \cdot \boldsymbol {v} (\boldsymbol {p} _ {n}, t _ {m}) | | \\ \ell_ {\text {m o m e n t u m}} = \frac {1}{N M} \sum_ {n = 1} ^ {N} \sum_ {m = 1} ^ {M} | | \frac {\partial \boldsymbol {v} \left(\boldsymbol {p} _ {n} , t _ {m}\right)}{\partial t _ {m}} + \boldsymbol {v} \left(\boldsymbol {p} _ {n}, t _ {m}\right) \cdot \nabla_ {\boldsymbol {p} _ {n}} \boldsymbol {v} \left(\boldsymbol {p} _ {n}, t _ {m}\right) - \boldsymbol {a} | | \tag {8} \\ \end{array}
$$
where  $\pmb{p}_n$  is uniformly sampled in the whole 3D scene volume, and  $t_m$  is uniformly sampled from 0 to the interested maximum extrapolation time  $t_{max}$ , and  $\pmb{a}$  is the general acceleration term learned by another MLP-based network:  $(x,y,z,t) \to \pmb{a}$ , whose details are in Appendix. Nevertheless, with such PINN losses, it is insufficient to optimize the velocity field itself, since there are infinitely many solutions. To tackle this, we introduce an additional interframe optimization strategy.
Interframe Optimization Strategy: Naturally, the 3D scene geometry and appearance encoded in the keyframe dynamic radiance field, once appropriately transported by the velocity field, should be able to render 2D images to match with the ground truth observations in interframes belonging to the remaining  $(T - K)$  timestamps. To enforce such a constraint, the key challenge is to determine the color and density values for all 3D points at each interframe timestamp, such that the volume rendering equation can be applied to estimate RGB for each pixel at the interframe timestamp, after which the photometric loss can be adopted. To tackle this, we propose the following Algorithm 1.
Algorithm 1 At a specific interframe timestamp  $t_i$ , given a light ray  $r_i$  with viewing angle  $(\theta, \phi)$  and  $S$  sample points  $\{p_1 \cdots p_s \cdots p_S\}$  along the ray, the objective of this algorithm is to determine the color and density values for the  $S$  points along  $r_i$ , denoted as:  $\{(c_1, \sigma_1) \cdots (c_s, \sigma_s) \cdots (c_S, \sigma_S)\}$ . In the meantime, we also have the keyframe dynamic radiance field  $f_\Theta$  and velocity field  $g_\Phi$ .
Note that, we shall not directly query  $f_{\Theta}$  to obtain color and density for the  $S$  points because: 1) the dynamic radiance field  $f_{\Theta}$  is never trained on interframe timestamps, thus the queried values are inaccurate; 2) the velocity field  $g_{\Phi}$  will not be involved, and therefore the interframes cannot provide additional constraints to optimize  $g_{\Phi}$ .
- The ray direction  $(\theta, \phi)$ , the interframe timestamp  $t_i$ , the  $S$  sample points on the ray  $\{\pmb{p}_1 \cdots \pmb{p}_s \cdots \pmb{p}_S\}$ ;
- The  $K$  keyframe timestamps  $\{t_1\dots t_k\dots t_K\}$
- The initialized and ongoing training networks:  $f_{\Theta}$  and  $g_{\Phi}$ ;
- The color and density values for  $S$  sample points along the ray:  $\{(c_1, \sigma_1) \cdots (c_s, \sigma_s) \cdots (c_S, \sigma_S)\}$ ;
Preliminary step:
- Find the nearest keyframe timestamp  $\hat{t}_k$  for the interframe timestamp  $t_i$ :
$$
\hat {t} _ {k} = \underset {t _ {k}} {\arg \min } | t _ {k} - t _ {i} |
$$
for  $p_s$  in  $\{\pmb{p}_1\cdots \pmb{p}_s\cdots \pmb{p}_S\}$  do
- Transport  $\pmb{p}_s$  to its corresponding point  $\pmb{p}_s'$  at its nearest keyframe timestamp  $\hat{t}_k$ , according to its velocity field. The position of  $\pmb{p}_s'$  can be obtained by:
$$
\boldsymbol {p} _ {s} ^ {\prime} = \boldsymbol {p} _ {s} + \int_ {t _ {i}} ^ {\hat {t} _ {k}} g _ {\Phi} (\boldsymbol {p} _ {s} (t), t) d t, \quad \text {R u n g e - K u t t a 2 s o l v e r [ 9 ] i s a p p l i e d i n o u r i m p l e m e n t a t i o n}.
$$
- Retrieve the volume density  $\sigma_s'$  and view-agnostic color feature vector  $\pmb{e}_s'$  for point  $p_s'$ :
$(\sigma_s', \pmb{e}_s') \gets f_\Theta(\pmb{p}_s', \hat{\imath}_k)$ , Note: HexPlane backbone [6] can output a view-agnostic color feature vector  $\pmb{e}_s'$ .
- Assign the retrieved features of  $\pmb{p}_s'$  to the original point  $\pmb{p}_s$ :  $(\sigma_s, \pmb{e}_s) \gets (\sigma_s', \pmb{e}_s')$ .
- Obtain the color  $c_{s}$  for point  $p_{s}$ :
$\pmb{c}_s \gets \tilde{f}_{\Theta}(\pmb{e}_s, \theta, \phi)$ , Note:  $\tilde{f}_{\Theta}$  is a subnetwork of HexPlane backbone [6] as detailed in Appendix.
- Output  $(\pmb{c}_s, \pmb{\sigma}_s)$  for point  $\pmb{p}_s$ .
After the above for loop, we obtain all color and density values for  $S$  sample points.
Having the color and density values of all 3D points along ray  $\mathbf{r}_i$  in the interframe of timestamp  $t_i$ , the appearance  $\mathbf{C}(\mathbf{r}_i, t_i)$  is obtained by volume rendering of NeRF [37]. Then both networks  $f_{\Theta}$  and  $g_{\Phi}$  can be optimized by the following interframe photometric loss.
$\ell_{interframe} = ||\pmb {C}(\pmb {r}_i,t_i) - \bar{\pmb{C}} (\pmb {r}_i,t_i)||,$  where  $\bar{\pmb{C}} (\pmb {r}_i,t_i)$  is the ground truth color observation. (9)
3.4 Joint Keyframe and Interframe Optimization
Technically, the keyframe dynamic radiance field  $f_{\Theta}$  can be firstly trained by  $\ell_{keyframe}$  only, and then the interframe velocity field  $g_{\Phi}$  by  $\ell_{divergence\_free} + \ell_{momentum} + \ell_{interframe}$ .
Nevertheless, we empirically find that simultaneously propagating errors of interframes back to the radiance field  $f_{\Theta}$  helps achieve better performance overall. Therefore, we adopt the following joint keyframe and interframe strategy to optimize both networks.
$$
f _ {\Theta} \leftarrow \left(\ell_ {\text {k e y f r a m e}} + \ell_ {\text {i n t e r f r a m e}}\right) \quad g _ {\Phi} \leftarrow \left(\ell_ {\text {d i v e r g e n c e - f r e e}} + \ell_ {\text {m o m e n t u m}} + \ell_ {\text {i n t e r f r a m e}}\right) \tag {10}
$$
Datasets: Our framework primarily focuses on learning meaningful physical velocity fields for dynamic 3D scenes, instead of simply fitting video frames. Although there are a number of dynamic 3D scene datasets in the literature, they are mainly collected for the popular task of novel view synthesis within the training time period, i.e., interpolation in time dimension. Besides, the underlying motions of these scenes tend to be chaotic, and estimating their future motions or transferring their motions are hardly meaningful or entertaining in practice. In this regard, we introduce two new synthetic datasets: 1) Dynamic Object dataset, and 2) Dynamic Indoor Scene dataset.
1) Dynamic Object Dataset: This dataset consists of 6 distinct 3D objects, each of which displays a unique motion pattern, including either rigid or deformable movements in 3D space. These 3D objects and their realistic motions are all designed by unknown external practitioners from SketchFab, and we purchased their Licenses and will make them available for free use in the community.
For each 3D object, we collect RGB images at 15 different viewing angles over 1 second, where each viewing angle has 60 frames captured. We reserve the first 45 frames at randomly picked 12 viewing angles as the training split, i.e., 540 frames, while leaving the 45 frames at the remaining 3 viewing angles for testing interpolation ability, i.e., 135 frames for novel view synthesis within the training time period, and keeping the last 15 frames at all 15 viewing angles for evaluating future frame extrapolation, i.e., 225 frames. More details are in Appendix.