SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lukas Uzolas Elmar Eisemann Petr Kellnhofer
{1.uzolas, e.eisenemann, p.kellnhofer}@tudelft.nl
Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains. In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reproducible 3D reconstructions without the need of object-specific skeletal templates. The project website can be found at https://lukas.uzolas.com/Articulated-Point-NeRF/.
Synthesizing novel photo-realistic views of captured 3D scenes is important for many domains including virtual/augmented reality, video games or movie productions. In recent years, Neural Radiance Fields (NeRFs) [1] have proved their remarkable capacity to represent complex view-dependent effects, captured in photographs and videos, sparsely sampled from natural light fields [2]. Follow-up works have extended the scope to dynamic scenes [3-10], facilitating rendering of unseen views at different timestamps. Despite progress in reconstruction quality and speed [11], manipulating learned scenes remains a challenge, but would be a highly desirable feature, as it can enable downstream applications, such as the creation of avatars for virtual presence or 3D assets for games and movies.
The key challenge of reposing a NeRF is inverting the backward-warping function that maps individual observations to a shared canonical representation [3]. It is an inversion of traditional kinematic animation, such as Linear Blend Skinning (LBS) [12], where a canonical shape is forward-warped to a desired pose. Such inverse mapping often requires resolving ambiguous situations as it is difficult to guarantee bijectivity (see Fig. 2). A common remedy are parametric templates, typically built for narrow application domains, such as human heads and bodies [13-16], but they are difficult to generalize. Alternatively, object shapes can be retrieved from videos as ensembles of geometric parts, yet existing techniques provide limited image-synthesis fidelity [17-19]. Our work aims to combine these different lines of work and enable joint learning of the NeRF representation and its pose parameterization from sparse or dense multi-view videos. We aim for time-efficient class-agnostic view synthesis ofposable models with high image-synthesis quality without access to a template or pose annotation, which is a combination not currently covered by existing work (see Table 1).
Figure 1: Overview of our method: a) First, we pre-train a NeRF backbone to initialize a feature point cloud  $P^c$  for a selected canonical timestamp and to extract an initial skeleton. b) During the main training stage,  $P^c$  is forward-warped using LBS consisting of learned time-invariant skinning weights  $\hat{w}_i$  and time-dependent pose transformations from an MLP regressor  $\Phi_r$ . The image is obtained by integration and decoding of features aggregated from points along each camera ray. In summary, we fine-tune the initial neural point features  $\mathbf{f}_i$ , skinning weights  $\hat{w}_i$ , joints  $J$ , density and color regressor  $\Phi_d$  and  $\Phi_c$  of the backbone. We fully train the pose regressor  $\Phi_r$  and feature point decoder  $\Phi_p$  from scratch. In test time, we modify the pose transformations to obtain novel poses.
Figure 2: Examples of poses difficult for backward warping. (a) Ill-defined projection from an observation to the canonical space. Both ambiguous solutions (green and magenta) correctly pass through the semantically corresponding surface points B and C. (b) Projection ambiguity for points of contact between two surfaces. Note that in contrast, it is trivial to forward warp the object points from a well-chosen canonical space to any observation space.
Method	Pose	Generic	Fidelity	Training
LASR [17]	No	Yes	Shape	≤ 2 h
ViSER [18]	No	Yes	Shape	2-12 h
BANMO [22]	No	Yes	Low	≥ 12 h
D-NeRF [3]	No	Yes	High	≥ 12 h
TiNeuVox [11]	No	Yes	High	≤ 2 h
CASA [23]	Yes	No	Shape	≤ 2 h
LASSIE [19]	Yes	Yes	Low	≥ 12 h
WIM [20]	Yes	Yes	High	≥ 12 h
Ours	Yes	Yes	High	≤ 2 h
Table 1: A comparison of representative dynamic 3D representations learned from 2D videos. We analyze reproducibility, agnosticism to object class, image synthesis fidelity and training time. Papers that demonstrate reposing are shown below the bar. "Shape" denotes shape-only reconstruction without texture details.
Recent work by Noguchi et al. [20] addresses a similar problem. However, we exploit the structure-free nature of point-based NeRF representations [21], enabling direct forward warping of the canonical object to any desired pose, while maintaining the capacity and flexibility of NeRF-like rendering to capture highly detailed image features. Together with our automatically extracted and jointly-optimized LBS skeletal pose parameterization, our work allows for faster training and achieves state-of-the-art visual quality in a much shorter time. Finally, we demonstrate how to easily repose learned representations of varied objects by directly editing joint angles of the forward LBS model.
To summarize, we make the following contributions: 1) We propose a novel method for learning articulated neural representations of dynamic objects using forward projection of neural point clouds. 2) We demonstrate state-of-the-art novel view synthesis for a variety of multi-view videos with training times lower than comparable methods. 3) We jointly learn a skeletal model without domain-specific priors or user annotations and demonstrate reposing of the reconstructed 3D objects.
Neural Radiance Fields and Parameterizations Implicit neural networks have recently emerged as a powerful tool for building differentiable representations of 3D scenes [24-47] and for learning view-dependent Neural Radiance Fields (NeRFs) [1, 2, 48-57] enabling photo-realistic novel view synthesis. Their high computational cost has been dramatically reduced by spatial decomposition [58-60] and hybrid representations [61-65]. Recently, neural point clouds have combined expressive neural functions with the flexibility of explicit geometry [21, 66-69]. We also use neural point clouds, but unlike previous work, we jointly learn a forward kinematic model along with the associated soft part segmentation and, thus, convert fast voxel-based representations to reposable NeRFs.
Dynamic Neural Radiance Fields Equivalent to 2D video, dynamic NeRFs capture temporal changes in scenes by encoding each frame separately [70], expanding the radiance field into the temporal domain [7, 8, 71, 72], or time-dependent interpolation in a compact latent representation [6, 9, 10]. Alternatively, a single canonical NeRF can be animated by backward warping from the canonical space to individual time-varying observation spaces [3-6, 9, 11]. However, such warps rely on the bijectivity of the mapping which is difficult to guarantee for all observed poses (see Fig. 2). Forward mapping only requires a single well-posed canonical representation, which we exploit.
Object Reposing Directly reposing high-dimensional deformation fields of dynamic NeRFs is impractical. Instead, parametric templates can sparsely represent prominent deformation modes for faces [13, 14], bodies [15, 16], hands [73], and also non-human objects, such as animals [74]. Together with skeleton-based LBS, they enable articulated neural representations of human heads [69, 75-81] or bodies [82-97], as well as modeling distributions in generative models [98-101]. However, the assumption of piece-wise rigid motion and the availability of a skeletal template restrict the applicable object classes. The former issue has been addressed by physically inspired but computationally expensive deformation models [102]. To remedy the latter issue, the skeletal model can be retrieved from a database [23], adapted from a generic graph [19, 103], fitted as a template to 2D observations [104-108] or fully learned during a 3D shape recovery [20, 109]. An external cage can also be used for re-animation of static NeRFs [110]. Alternatively, a surface embedding can be learned without a skeleton if reposing is not required [17, 18, 22].
Similarly to Noguchi et al. [20], we jointly learn a 3D object representation, a skeletal model, and observed poses. However, we utilize a point-based NeRF representation to benefit from its computational efficiency, ease of geometric transformation, and capacity to learn complex light fields.
We share our point-based approach with a contemporaneous work MovingParts [111] which, however, relies on inverting the backward flow. We demonstrate the reconstruction of large transformations in the Robots dataset [20], not demonstrated in MovingParts. Furthermore, our approach does not enforce binary segmentation and, hence, can represent non-rigid part transitions (see Fig. 6 right).
Our method builds upon available NeRF reconstruction methods as a backbone for learning a reposable dynamic representation. In principle, any NeRF-like representation is a suitable initial point for our training. In practice, we use TiNeuVox [11] in all experiments, as it enables fast reconstruction with both sparse and dense multi-view supervision in dynamic scenes. Next, we describe the core concepts of NeRF [1] and TiNeuVox [11] to provide context for our method in Sec. 4.
Time-Aware Neural Voxels NeRF [1] maps a point  $\mathbf{p} = (x,y,z)$  and view direction  $\mathbf{d} = (\theta ,\phi)$  to color  $\mathbf{c} = (r,g,b)$  and density  $\sigma$  . The mapping function often takes form of a Multi-Layer-Perceptron (MLP)  $\Phi$  , such that  $(\mathbf{c},\sigma) = \Phi (\mathbf{p},\mathbf{d})$  . The color of an image pixel  $C(\mathbf{r})$  can be obtained via volume rendering along the ray  $r(t) = \mathbf{o} + t\mathbf{d}$
TiNeuVox [11] expands NeRF to dynamic scenes and significantly increases training speed by utilizing an explicit volume representation. To this extent, each point  $\mathbf{p}$  at time  $t$  in observation space is backward-warmed to canonical space as
$$
\mathbf {p} ^ {c} = \Phi_ {b} (\hat {\mathbf {p}}, \hat {\mathbf {t}}), \text {w h e r e} \hat {\mathbf {p}} = \gamma (\mathbf {p}), \hat {\mathbf {t}} = \Phi_ {t} (\gamma (t)), \tag {1}
$$
Here,  $\gamma$  is the positional encoding [1],  $\Phi_t$  a small color-decoding MLP,  $\Phi_b$  a backward-warping MLP, and  $\hat{\mathbf{p}}$  and  $\hat{\mathbf{t}}$  the encoded point and time respectively. The canonical point  $\mathbf{p}^c$  is used to sample a feature vector  $\mathbf{v}_m \in \mathcal{R}^V$  from the explicit feature volume  $\mathbf{V} \in \mathcal{R}^{X \times Y \times Z \times V}$  by means of trilinear interpolation interp with varying stride  $s_m$  at each scale:  $\mathbf{v}_m = \gamma (\text{interp}(\mathbf{p}^c, \mathbf{V}, s_m))$ . The feature vectors across all scales are concatenated as  $\mathbf{v} = \mathbf{v}_1 \oplus \ldots \oplus \mathbf{v}_m \oplus \ldots \oplus \mathbf{v}_M$ , where  $\oplus$  is the concatenation operator. Finally, the view-dependent color and density are regressed by additional MLPs  $\Phi_f$ ,  $\Phi_d$  and  $\Phi_c$ :
$$
\mathbf {f} = \Phi_ {f} (\mathbf {v}, \hat {\mathbf {p}}, \hat {\mathbf {t}}), \text {w h e r e} \sigma = \Phi_ {d} (\mathbf {f}), \mathbf {c} = \Phi_ {c} (\mathbf {f}, \mathbf {d}). \tag {2}
$$
The volume rendering equation [112] integrates density/color of points  $\mathbf{p}_i$  sampled along the ray  $r$ :
$$
\hat {C} (\mathbf {r}) = \sum_ {i = 1} ^ {N} T _ {i} (1 - \exp (- \sigma_ {i} \delta_ {i})) \mathbf {c} _ {i}, \text {w h e r e} T _ {i} = \exp (- \sum_ {j = 1} ^ {i - 1} \sigma_ {j} \delta_ {j})), \tag {3}
$$
Here,  $\delta_{i}$  is the distance between  $\mathbf{p}_i$  and  $\mathbf{p}_{i + 1}$ . The feature volume  $\mathbf{V}$  and all MLPs are optimized end-to-end and supervised by the Mean Squared Error (MSE) and the corresponding ground-truth pixel color  $C(\mathbf{r})$ :  $\mathcal{L}_{photo} = ||\hat{C} (\mathbf{r}) - C(\mathbf{r})||_2^2$ .
Note that because  $\Phi_f$  (Eq. 2) is conditioned by the time  $\hat{\mathbf{t}}$ , space and time are entangled and the feature volume  $V$  does not represent a single static canonical shape. Therefore, we cannot directly learn a forward warping function to invert  $\Phi_b$  as proposed by Chen et al. [102]. In the next section, we show that this is not necessary and we can learn a forward kinematic model directly without relying on specific properties of the backbone, which will allow us to replace it in the future.
While dynamic NeRF models such as TiNeuVox [11] can reproduce motion in a dynamic scene, they do not enable reposing. Furthermore, it is impractical to post hoc reparametrize their motion representation due to the ambiguities of backward flow inversion (see Fig. 2).
We instead propose a completely new representation based on a neural point cloud and an explicit kinematic model for forward warping, and we use the backbone only as an initialization for our training procedure (see Fig. 1). First, we extract a feature point cloud from a selected canonical frame of the backbone Sec. 4.1. Second, we describe the underlying kinematic model of our 3D representation and its initialization Sec. 4.2. Consecutively, we introduce how the 3D point cloud is warped from a canonical space to an observation space and rendered Sec. 4.3. Lastly, we describe our losses Sec. 4.4.
4.1 Canonical Feature Point Cloud
We first pre-train a backbone NeRF model (TiNeuVox [11]) using its default parameters. To initialize the feature point cloud, we sample the canonical density function of TiNeuVox  $\sigma = \Phi_d(\mathbf{f})$  (Eq. 2) on a uniform coordinate grid and discard empty samples through thresholding. The grid resolution is adaptively chosen, such that  $|P|\approx 10000$ , similarly to other explicit NeRFs [11, 64, 113]. After thresholding, we obtain points  $P = \{\mathbf{p}_i|i = 1,\dots,N\}$ . Furthermore, we extract a feature vector  $\mathbf{f}_i$  for each point  $\mathbf{p}_i$  (see Eq. 2).
4.2 Kinematic Model
We forward-warp our feature point cloud from canonical space to observation space using an LBS kinematic model to match the training images. Here, we describe how we initialize, use, and simplify our kinematic model.
Skeleton Initialization We do not rely on a class-specific template to initialize our kinematic skeleton. Instead, we extract an approximate initial skeleton tree by Medial Axis Transform (MAT) and refine it during training.
Let  $M = \{\mathbf{p}_m|m = 1,\dots,M\}$  be the set of medial axis points extracted by applying a MAT on  $P^c$ . We choose the root of our kinematic model  $\mathbf{p}_{root}$  as the medial axis point that is closest to all other medial-axis points, i.e.,  $\mathbf{p}_{root} \coloneqq \arg \min_{i}\sum_{j}||\mathbf{p}_{i} - \mathbf{p}_{j}||_{2},\forall \mathbf{p}_{i}\in M,i\neq j.$
Next, we leverage dense sampling-grid neighborhoods and define a graph  $G_{M}$  connecting neighboring points in  $M$ . Points disconnected from  $\mathbf{p}_{root}$  are removed. We then use a heuristic to select sparse joints (nodes) as a subset of  $G_{M}$  and define the bones (edges) based on their connectivity. To this extent, we traverse  $G_{M}$  starting from  $\mathbf{p}_{root}$  in a breadth-first manner and mark  $\mathbf{p}_m$  as a joint  $\mathbf{j}_b$  if its geodesic distance from the preceding joint exceeds a threshold  $B_{length} = 10$ . As a result, we obtain a set of time-invariant canonical joint positions  $J = \{\mathbf{j}_b\}$  which we further optimize during training.
While this skeleton is usually over-segmented, we show in our experiments that this does not hamper the training. We propose a pruning strategy to enable easier pose manipulation afterwards (see Fig. 7).
Blend Skinning Weights For each point  $\mathbf{p}_i$  we initialize its raw skinning weight vector  $\hat{\mathbf{w}}_i$  by an exponential decay function of the distance  $dist$  to each bone line  $b_{j}$  such that  $\hat{w}_{i,j} = 1 / e^{dist(\mathbf{p}_i^c,b_j)}$ . Before skinning, we obtain the final blend skinning weight vector  $\mathbf{w}_i$  through scaling by a global learnable parameter  $\alpha$  and applying a softmax:  $w_{i,j} = e^{\hat{w}_{i,j} / \alpha} / \sum_{k}e^{\hat{w}_{i,k} / \alpha}$ . During the training, we optimize the initial  $\hat{\mathbf{w}}_i$  as well as  $\alpha$ . Accounting for the per-point weights, we define our full canonical feature point cloud as  $P^{c} = \{(p_{i}^{c},f_{i},\hat{w}_{i})|i = 1,\dots,N\}$ .
Point Warping We forward-warp the canonical point cloud  $P^c$  to an observation space of timestamp  $t$  via LBS [12]. The local transformation matrix  $\hat{T}_b^t$  of each bone  $b$  is defined by a rotation  $R_b^t$  around its parent joint  $\mathbf{j}_b$ . Consequently, each point  $\mathbf{p}_i^c$  is transformed by a linear combination of bone transformations as:
$$
\mathbf {p} _ {i} ^ {t} = \bar {T} _ {i} ^ {t} p _ {i} ^ {c} = \sum_ {b = 1} ^ {| B |} w _ {i, b} T _ {b} ^ {t} \mathbf {p} _ {i} ^ {c}, \text {w i t h} T _ {b} ^ {t} = T _ {p _ {b}} ^ {t} \hat {T} _ {b} ^ {t} \text {a n d} \hat {T} _ {b} ^ {t} = \left[ \begin{array}{l l} R _ {b} ^ {t} & \mathbf {j} _ {b} + R _ {b} ^ {t} \mathbf {j} _ {b} ^ {- 1} \\ \mathbf {0} & 1 \end{array} \right], \tag {4}
$$
where  $T_{b}^{t}$  is defined recursively by its parent bone  $p_{b}$ .  $T_{p_{b}}^{t}$  of the skeleton root is identity.
We express rotations  $R_{b}^{t} \in SO(3)$  using the Rodrigues' formula, where  $\hat{\mathbf{r}} = \mathbf{r} / ||\mathbf{r}||$  is the axis of rotation. However, we learn the rotation angle  $\theta$  directly as an additional parameters because we find it leads to a better pose regression than using  $\theta = ||\mathbf{r}||$ . The time-dependent rotations  $\mathbf{r}_b, \theta_b$  for each bone  $b$  are regressed by an MLP:  $\Phi_r(\gamma(t)) = \mathbf{r}_1^t, \theta_1^t, \dots, \mathbf{r}_b^t, \theta_b^t, \dots, \mathbf{r}_B^t, \theta_B, \mathbf{r}_r^t, \theta_r^t, \mathbf{t}_r^t$ , where  $\mathbf{r}_r^t, \theta_r^t$  and  $\mathbf{t}_r^t$  are the time-dependent root rotation and translation.
Skeleton simplification The initial skeleton's over-segmentation does not hamper pose reconstruction during training, but we optionally simplify the skeleton after training to ease pose editing (see Fig. 7). We prune or merge bones based on the size of rotation angles  $\theta_{b}^{t}$  produced by the transformation regressor  $\Phi_r$ . Joints, which do not exhibit a rotational change from the rest pose above the threshold of  $t_r$  deg in more than  $5\%$  of the observed timestamps, are marked static. We then merge the bones of such joints and their corresponding weights  $\hat{\mathbf{w}}$ . See the supplement for details.
4.3 Dynamic Point Rendering
We adopt the point cloud rendering from [21] but extend it to explicitly model rotational invariance of the radiance field. For each sampling point  $x$ , we find up to  $N = 8$  nearest observation feature points  $\mathbf{p}_i^t$  within a radius of 0.01 and roto-translate them into their canonical frames. This enables the feature embedding MLP  $\Phi_p$  to learn spatial relations in a pose-invariant coordinate frame as:
$$
\mathbf {f} _ {i, x} ^ {t} = \Phi_ {p} \left(\mathbf {f} _ {i}, x _ {\mathbf {p} _ {i}} ^ {t}\right), \text {w i t h} x _ {\mathbf {p} _ {i}} ^ {t} = \gamma \left(R _ {i} ^ {t ^ {- 1}} \left(x - \mathbf {p} _ {i} ^ {t}\right)\right), \tag {5}
$$
where,  $R_{i}^{t}$  is the rotation component  $\bar{T}_i^t$  (Eq. 4) for point  $\mathbf{p}_i^t$  and  $\gamma(.)$  is the positional encoding [1].
The neighboring embeddings  $\mathbf{f}_{i,x}^{t}$  are aggregated by inverse distance weighting, which produces the final feature input for  $\Phi_d$  and  $\Phi_c$  (see Eq. 2) and the consequent volume rendering (Eq. 3):
$$
\mathbf {f} _ {x} ^ {t} = \sum_ {i} ^ {N} \frac {d _ {i}}{\sum_ {j} ^ {| B |} d _ {j}} f _ {i, x} ^ {t}, \text {w i t h} d _ {i} = \left| \left| p _ {i} ^ {t} - x \right| \right| ^ {- 1}; \sigma = \Phi_ {d} \left(\mathbf {f} _ {x} ^ {t}\right); \mathbf {c} = \Phi_ {c} \left(\mathbf {f} _ {x} ^ {t}, \mathbf {d}\right). \tag {6}
$$
4.4 Losses
Next to the photometric loss  $\mathcal{L}_{photo}$  (Sec. 3), we utilize a 2D chamfer loss to penalize the difference between the point cloud projected into a training view  $I(P^t)$  and the corresponding 2D ground truth
Figure 3: Qualitative comparison displaying two held-out views-frames of scenes from the Robots rendered by WIM [20] and our method after 2 and 10 hours of training, and the PSNR scores.
object mask  $M^t$ :  $\mathcal{L}_{mask} \coloneqq \mathcal{L}_{chamf}(I(P^t), M^t)$ . The chamfer loss is defined as:
$$
\mathcal {L} _ {c h a m f} \left(P ^ {1}, P ^ {2}\right) = \frac {1}{| P ^ {1} |} \sum_ {i} ^ {| P ^ {1} |} \min  _ {j} \| \mathbf {p} _ {i} ^ {1} - \mathbf {p} _ {j} ^ {2} \| _ {2} ^ {2} + \frac {1}{| P ^ {2} |} \sum_ {j} ^ {| P ^ {2} |} \min  _ {i} \| \mathbf {p} _ {i} ^ {1} - \mathbf {p} _ {j} ^ {2} \| _ {2} ^ {2}. \tag {7}
$$
To prevent the skeleton from diverging too much from the medial axis  $M$ , we further minimize the chamfer loss between  $M$  and the joints  $J$  (see Sec. 4.2):  $\mathcal{L}_{skel} \coloneqq \mathcal{L}_{chamf}(M, J)$ . In addition, we minimize the transformation angles and the root translation:  $\mathcal{L}_{tranf} = \left( \sum_{i} |\theta_{i}^{t}| \right) + |\mathbf{t}_{r}^{t}|$ . Local rigidity is further enforced upon points after deformation via as-rigid-as-possible regularization:
$$
\mathcal {L} _ {a r a p} = \sum_ {i} ^ {| P ^ {t} |} \sum_ {j} ^ {N (p _ {i})} \left| \left| \left| p _ {i} ^ {c} - p _ {j} ^ {c} \right| \right| _ {2} ^ {2} - \left| \left| p _ {i} ^ {t} - p _ {j} ^ {t} \right| \right| _ {2} ^ {2} \right|. \tag {8}
$$
Lastly, we apply two regularizers on the blend skinning weights. To encourage smoothness, we penalize divergence of skinning weights in the rendering neighborhood  $N$ :  $\mathcal{L}_{smooth} = \sum_{i}^{|P^t|} \sum_{j \in N(p_i)} |w_i - w_j|$ , and, to encourage sparsity, we apply:
$$
\mathcal {L} _ {\text {s p a r s e}} = - \sum_ {i} ^ {| P ^ {c} |} \sum_ {j} ^ {B} w _ {i, j} \log \left(w _ {i, j}\right) + (1 - w _ {i, j}) \log \left(1 - w _ {i, j}\right). \tag {9}
$$
In total, our training loss is  $\mathcal{L} = \omega_0\mathcal{L}_{photo} + \omega_1\mathcal{L}_{mask} + \omega_2\mathcal{L}_{skel} + \omega_3\mathcal{L}_{tranf} + \omega_4\mathcal{L}_{smooth} + \omega_5\mathcal{L}_{sparse} + \omega_6\mathcal{L}_{ARAP}$  where  $\omega = \{200, 0.02, 1, 0.1, 10, 0.2, 0.005\}$  in our experiments.
Here, we evaluate our work, which obtains state-of-the-art view-synthesis quality with a lower training cost than other articulated methods. We also demonstrate class-agnostic reposing capability and we evaluate contribution of method components in an ablation study. Video examples can be seen on the project website.
Datasets We chose three multi-view video datasets that are commonly used for the evaluation of dynamic multi-view synthesis methods and pose articulation. First, the Robots dataset [20] features multi-view synthetic videos of 8 topologically varied robots, making it well suited for testing pose articulation performance (see Fig. 4). We use 18 views for training and 2 for evaluation. Second, the Blender dataset [3] is a sparse multi-view synthetic dataset with 5 humanoid and 2 other articulated objects<sup>1</sup>. Its visual quality benchmarks are used to test the fidelity of image detail reconstruction (see
<sup>1</sup>We do not use the multi-component Bouncing balls scene.