Title: Template-free Articulated Neural Point Clouds for Reposable View Synthesis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Radiance Fields

View Synthesis

Kinematics

Reposing

NeRF
Abstract: Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains.  In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates.

Template-free Articulated Neural Point Clouds for Reposable View Synthesis

Abstract

Dynamic Neural Radiance Fields (NeRFs) achieve remarkable visual quality when synthesizing novel views of time-evolving 3D scenes. However, the common reliance on backward deformation fields makes reanimation of the captured object poses challenging. Moreover, the state of the art dynamic models are often limited by low visual fidelity, long reconstruction time or specificity to narrow application domains. In this paper, we present a novel method utilizing a point-based representation and Linear Blend Skinning (LBS) to jointly learn a Dynamic NeRF and an associated skeletal model from even sparse multi-view video. Our forward-warping approach achieves state-of-the-art visual fidelity when synthesizing novel views and poses while significantly reducing the necessary learning time when compared to existing work. We demonstrate the versatility of our representation on a variety of articulated objects from common datasets and obtain reposable 3D reconstructions without the need of object-specific skeletal templates. The project website can be found at https://lukas.uzolas.com/Articulated-Point-NeRF/ .

1 Introduction

Synthesizing novel photo-realistic views of captured 3D scenes is important for many domains including virtual/augmented reality, video games or movie productions. In recent years, Neural Radiance Fields (NeRFs) [1] have proved their remarkable capacity to represent complex view-dependent effects, captured in photographs and videos, sparsely sampled from natural light fields [2]. Follow-up works have extended the scope to dynamic scenes [3-10], facilitating rendering of unseen views at different timestamps. Despite progress in reconstruction quality and speed [11], manipulating learned scenes remains a challenge, but would be a highly desirable feature, as it can enable downstream applications, such as the creation of avatars for virtual presence or 3D assets for games and movies.

The key challenge of reposing a NeRF is inverting the backward-warping function that maps individual observations to a shared canonical representation [3]. It is an inversion of traditional kinematic animation, such as Linear Blend Skinning (LBS) [12], where a canonical shape is forward-warped to a desired pose. Such inverse mapping often requires resolving ambiguous situations as it is difficult to guarantee bijectivity (see Fig. 2). A common remedy are parametric templates, typically built for narrow application domains, such as human heads and bodies [13-16], but they are difficult to generalize. Alternatively, object shapes can be retrieved from videos as ensembles of geometric parts, yet existing techniques provide limited image-synthesis fidelity [17-19]. Our work aims to combine these different lines of work and enable joint learning of the NeRF representation and its pose parameterization from sparse or dense multi-view videos. We aim for time-efficient class-agnostic view synthesis of reposable models with high image-synthesis quality without access to a template or pose annotation, which is a combination not currently covered by existing work (see Table 1).

NeRF

Extracted

Skeleton

Weights

Feature

/gid00064/gid00010/gid00001/gid00049 /gid00078/gid00072/gid00077/gid00083/gid00001/gid00036/gid00075/gid00078/gid00084/gid00067/gid00001/gid00038/gid00087/gid00083/gid00081/gid00064/gid00066 /gid00083/gid00072/gid00078/gid00077

/gid00065/gid00010/gid00001/gid00044/gid00072/gid00077/gid00068/gid00076/gid00064/gid00083/gid00072/gid00066/gid00001/gid00047/gid00068/gid00084/gid00081/gid00064/gid00075/gid00001/gid00049 /gid00078/gid00072/gid00077/gid00083/gid00001/gid00036/gid00075/gid00078/gid00084/gid00067/gid00001 /gid00053

/gid00081/gid00064/gid00072/gid00077/gid00072/gid00077/gid00070

Rendered Image and Weights

Rt

-1

c

, σ

Color and Density

Regressor

γ(x)

t

{R

t

, t

t

}

P c

wi

J

p

1

p

2

p

3

p

4

川

Observation

space

Canonical

space

Observation

space

Canonical

space

A

(a)

(b)

D

B C

A

D

?

C B

Recent work by Noguchi et al. [20] addresses a similar problem. However, we exploit the structurefree nature of point-based NeRF representations [21], enabling direct forward warping of the canonical object to any desired pose, while maintaining the capacity and flexibility of NeRF-like rendering to capture highly detailed image features. Together with our automatically extracted and jointlyoptimized LBS skeletal pose parameterization, our work allows for faster training and achieves state-of-the-art visual quality in a much shorter time. Finally, we demonstrate how to easily repose learned representations of varied objects by directly editing joint angles of the forward LBS model.

To summarize, we make the following contributions: 1) We propose a novel method for learning articulated neural representations of dynamic objects using forward projection of neural point clouds. 2) We demonstrate state-of-the-art novel view synthesis for a variety of multi-view videos with training times lower than comparable methods. 3) We jointly learn a skeletal model without domainspecific priors or user annotations and demonstrate reposing of the reconstructed 3D objects.

2 Related Work

Neural Radiance Fields and Parameterizations Implicit neural networks have recently emerged as a powerful tool for building differentiable representations of 3D scenes [24-47] and for learning view-dependent Neural Radiance Fields (NeRFs) [1, 2, 48-57] enabling photo-realistic novel view synthesis. Their high computational cost has been dramatically reduced by spatial decomposition [5860] and hybrid representations [61-65].Recently, neural point clouds have combined expressive neural functions with the flexibility of explicit geometry [21, 66-69]. We also use neural point clouds, but unlike previous work, we jointly learn a forward kinematic model along with the associated soft part segmentation and, thus, convert fast voxel-based representations to reposable NeRFs.

Dynamic Neural Radiance Fields Equivalent to 2D video, dynamic NeRFs capture temporal changes in scenes by encoding each frame separately [70], expanding the radiance field into the temporal domain [7, 8, 71, 72], or time-dependent interpolation in a compact latent representation [6, 9, 10]. Alternatively, a single canonical NeRF can be animated by backward warping from the canonical space to individual time-varying observation spaces [3-6, 9, 11]. However, such warps rely on the bijectivity of the mapping which is difficult to guarantee for all observed poses (see Fig. 2). Forward mapping only requires a single well-posed canonical representation, which we exploit.

Object Reposing Directly reposing high-dimensional deformation fields of dynamic NeRFs is impractical. Instead, parametric templates can sparsely represent prominent deformation modes for faces [13, 14], bodies [15, 16], hands [73], and also non-human objects, such as animals [74]. Together with skeleton-based LBS, they enable articulated neural representations of human heads [69, 75-81] or bodies [82-97], as well as modeling distributions in generative models [98-101]. However, the assumption of piece-wise rigid motion and the availability of a skeletal template restrict the applicable object classes. The former issue has been addressed by physically inspired but computationally expensive deformation models [102]. To remedy the latter issue, the skeletal model can be retrieved from a database [23], adapted from a generic graph [19, 103], fitted as a template to 2D observations [104-108] or fully learned during a 3D shape recovery [20, 109]. An external cage can also be used for re-animation of static NeRFs [110]. Alternatively, a surface embedding can be learned without a skeleton if reposing is not required [17, 18, 22].

Similarly to Noguchi et al. [20], we jointly learn a 3D object representation, a skeletal model, and observed poses. However, we utilize a point-based NeRF representation to benefit from its computational efficiency, ease of geometric transformation, and capacity to learn complex light fields.

We share our point-based approach with a contemporaneous work MovingParts [111] which, however, relies on inverting the backward flow. We demonstrate the reconstruction of large transformations in the Robots dataset [20], not demonstrated in MovingParts . Furthermore, our approach does not enforce binary segmentation and, hence, can represent non-rigid part transitions (see Fig. 6 right).

3 Preliminaries

Our method builds upon available NeRF reconstruction methods as a backbone for learning a reposable dynamic representation. In principle, any NeRF-like representation is a suitable initial point for our training. In practice, we use TiNeuV ox [11] in all experiments, as it enables fast reconstruction with both sparse and dense multi-view supervision in dynamic scenes. Next, we describe the core concepts of NeRF [1] and TiNeuVox [11] to provide context for our method in Sec. 4.

Time-Aware Neural Voxels NeRF [1] maps a point p = ( x, y, z ) and view direction d = ( θ, ϕ ) to color c = ( r, g, b ) and density σ . The mapping function often takes form of a Multi-Layer-Perceptron (MLP) Φ , such that ( c , σ ) = Φ( p , d ) . The color of an image pixel C ( r ) can be obtained via volume rendering along the ray r ( t ) = o + t d .

TiNeuVox [11] expands NeRF to dynamic scenes and significantly increases training speed by utilizing an explicit volume representation. To this extent, each point p at time t in observation space is backward-warped to canonical space as

Here, γ is the positional encoding [1], Φ t a small color-decoding MLP, Φ b a backward-warping MLP, and ˆ p and ˆ t the encoded point and time respectively. The canonical point p c is used to sample a feature vector v m ∈ R V from the explicit feature volume V ∈ R X × Y × Z × V by means of trilinear interpolation interp with varying stride s m at each scale: v m = γ ( interp ( p c , V , s m )) . The feature vectors across all scales are concatenated as v = v 1 ⊕ ... ⊕ v m ⊕ ... ⊕ v M , where ⊕ is the concatenation operator. Finally, the view-dependent color and density are regressed by additional MLPs Φ f , Φ d and Φ c :

The volume rendering equation [112] integrates density/color of points p i sampled along the ray r :

Here, δ i is the distance between p i and p i +1 . The feature volume V and all MLPs are optimized end-to-end and supervised by the Mean Squared Error (MSE) and the corresponding ground-truth pixel color C ( r ) : L photo = || ˆ C ( r ) -C ( r ) || 2 2 .

Note that because Φ f (Eq. 2) is conditioned by the time ˆ t , space and time are entangled and the feature volume V does not represent a single static canonical shape. Therefore, we cannot directly learn a forward warping function to invert Φ b as proposed by Chen et al. [102]. In the next section, we show that this is not necessary and we can learn a forward kinematic model directly without relying on specific properties of the backbone, which will allow us to replace it in the future.

4 Method

While dynamic NeRF models such as TiNeuVox [11] can reproduce motion in a dynamic scene, they do not enable reposing. Furthermore, it is impractical to post hoc reparametrize their motion representation due to the ambiguities of backward flow inversion (see Fig. 2).

We instead propose a completely new representation based on a neural point cloud and an explicit kinematic model for forward warping, and we use the backbone only as an initialization for our training procedure (see Fig. 1). First, we extract a feature point cloud from a selected canonical frame of the backbone Sec. 4.1. Second, we describe the underlying kinematic model of our 3D representation and its initialization Sec. 4.2. Consecutively, we introduce how the 3D point cloud is warped from a canonical space to an observation space and rendered Sec. 4.3. Lastly, we describe our losses Sec. 4.4.

4.1 Canonical Feature Point Cloud

We first pre-train a backbone NeRF model (TiNeuVox [11]) using its default parameters. To initialize the feature point cloud, we sample the canonical density function of TiNeuVox σ = Φ d ( f ) (Eq. 2) on a uniform coordinate grid and discard empty samples through thresholding. The grid resolution is adaptively chosen, such that | P | ≈ 10000 , similarly to other explicit NeRFs [11, 64, 113]. After thresholding, we obtain points P = { p i | i = 1 , ..., N } . Furthermore, we extract a feature vector f i for each point p i (see Eq. 2).

4.2 Kinematic Model

We forward-warp our feature point cloud from canonical space to observation space using an LBS kinematic model to match the training images. Here, we describe how we initialize, use, and simplify our kinematic model.

Skeleton Initialization We do not rely on a class-specific template to initialize our kinematic skeleton. Instead, we extract an approximate initial skeleton tree by Medial Axis Transform (MAT) and refine it during training.

̸

Let M = { p m | m = 1 , ..., M } be the set of medial axis points extracted by applying a MAT on P c . We choose the root of our kinematic model p root as the medial axis point that is closest to all other medial-axis points, i.e., p root := arg min i ∑ j || p i -p j || 2 , ∀ p i ∈ M,i = j .

Next, we leverage dense sampling-grid neighborhoods and define a graph G M connecting neighboring points in M . Points disconnected from p root are removed. We then use a heuristic to select sparse joints (nodes) as a subset of G M and define the bones (edges) based on their connectivity. To this extent, we traverse G M starting from p root in a breadth-first manner and mark p m as a joint j b if its geodesic distance from the preceding joint exceeds a threshold B length = 10 . As a result, we obtain a set of time-invariant canonical joint positions J = { j b } which we further optimize during training.

While this skeleton is usually over-segmented, we show in our experiments that this does not hamper the training. We propose a pruning strategy to enable easier pose manipulation afterwards (see Fig. 7).

Blend Skinning Weights For each point p i we initialize its raw skinning weight vector ˆ w i by an exponential decay function of the distance dist to each bone line b j such that ˆ w i,j = 1 /e dist ( p c i ,b j ) . Before skinning, we obtain the final blend skinning weight vector w i through scaling by a global learnable parameter α and applying a softmax: w i,j = e ˆ wi,j /α / ∑ k e ˆ w i,k /α . During the training, we optimize the initial ˆ w i as well as α . Accounting for the per-point weights, we define our full canonical feature point cloud as P c = { ( p c i , f i , ˆ w i ) | i = 1 , ..., N } .

Point Warping Weforward-warp the canonical point cloud P c to an observation space of timestamp t via LBS [12]. The local transformation matrix ˆ T t b of each bone b is defined by a rotation R t b around its parent joint j b . Consequently, each point p c i is transformed by a linear combination of bone transformations as:

where T t b is defined recursively by its parent bone p b . T t p b of the skeleton root is identity.

We express rotations R t b ∈ SO (3) using the Rodrigues' formula, where ˆ r = r / || r || is the axis of rotation. However, we learn the rotation angle θ directly as an additional parameters because we find it leads to a better pose regression than using θ = || r || . The time-dependent rotations r b , θ b for each bone b are regressed by an MLP: Φ r ( γ ( t )) = r t 1 , θ t 1 , ..., r t b , θ t b , ..., r t B , θ B , r t r , θ t r , t t r , where r t r , θ t r and t t r are the time-dependent root rotation and translation.

Skeleton simplification The initial skeleton's over-segmentation does not hamper pose reconstruction during training, but we optionally simplify the skeleton after training to ease pose editing (see Fig. 7). We prune or merge bones based on the size of rotation angles θ t b produced by the transformation regressor Φ r . Joints, which do not exhibit a rotational change from the rest pose above the threshold of t r deg in more than 5% of the observed timestamps, are marked static. We then merge the bones of such joints and their corresponding weights ˆ w . See the supplement for details.

4.3 Dynamic Point Rendering

We adopt the point cloud rendering from [21] but extend it to explicitly model rotational invariance of the radiance field. For each sampling point x , we find up to N = 8 nearest observation feature points p t i within a radius of 0 . 01 and roto-translate them into their canonical frames. This enables the feature embedding MLP Φ p to learn spatial relations in a pose-invariant coordinate frame as:

where, R t i is the rotation component ¯ T t i (Eq. 4) for point p t i and γ ( . ) is the positional encoding [1].

The neighboring embeddings f t i,x are aggregated by inverse distance weighting, which produces the final feature input for Φ d and Φ c (see Eq. 2) and the consequent volume rendering (Eq. 3):

4.4 Losses

Next to the photometric loss L photo (Sec. 3), we utilize a 2D chamfer loss to penalize the difference between the point cloud projected into a training view I ( P t ) and the corresponding 2D ground truth

23.3 dB

29.4 dB

26.1 dB

32.4 dB

21.8 dB

26.8 dB

24.0 dB

28.7 dB

21.2 dB

27.5 dB

22.5 dB

28.6 dB

Ground Truth

2 hour training

Watch-It-Move

Ours

10 hour training

Watch-It-Move

Ours

object mask M t : L mask := L chamf ( I ( P t ) , M t ) . The chamfer loss is defined as:

To prevent the skeleton from diverging too much from the medial axis M , we further minimize the chamfer loss between M and the joints J (see Sec. 4.2): L skel := L chamf ( M,J ) . In addition, we minimize the transformation angles and the root translation: L tranf = ( ∑ i | θ t i | ) + | t t r | . Local rigidity is further enforced upon points after deformation via as-rigid-as-possible regularization:

Lastly, we apply two regularizers on the blend skinning weights. To encourage smoothness, we penalize divergence of skinning weights in the rendering neighborhood N : L smooth = ∑ | P t | i ∑ j ∈ N ( p i ) | w i -w j | , and, to encourage sparsity, we apply:

In total, our training loss is L = ω 0 L photo + ω 1 L mask + ω 2 L skel + ω 3 L tranf + ω 4 L smooth + ω 5 L sparse + ω 6 L ARAP where ω = { 200 , 0 . 02 , 1 , 0 . 1 , 10 , 0 . 2 , 0 . 005 } in our experiments.

5 Experiments

Here, we evaluate our work, which obtains state-of-the-art view-synthesis quality with a lower training cost than other articulated methods. We also demonstrate class-agnostic reposing capability and we evaluate contribution of method components in an ablation study. Video examples can be seen on the project website.

Datasets We chose three multi-view video datasets that are commonly used for the evaluation of dynamic multi-view synthesis methods and pose articulation. First, the Robots dataset [20] features multi-view synthetic videos of 8 topologically varied robots, making it well suited for testing pose articulation performance (see Fig. 4). We use 18 views for training and 2 for evaluation. Second, the Blender dataset [3] is a sparse multi-view synthetic dataset with 5 humanoid and 2 other articulated objects 1 . Its visual quality benchmarks are used to test the fidelity of image detail reconstruction (see

WIM

Ours

PSNR [dB]

LPIPS

Training time [hours]

Training time [hours]

Time Method

PSNR

↑

SSIM

↑

LPIPS

↓

2 h

WIM [20]

25.05

0.952

0.059

Ours

28.81

0.968

0.047

10 h

WIM [20]

27.61

0.964

0.046

Ours

30.19

0.973

0.041

Time Backbone [Iterations]

PSNR [dB]

SSIM

Time Backbone [Iterations]

Time Backbone [Iterations]

LPIPS

Fig. 6 right). We use the original training-test split. Third, ZJU-MoCap dataset [82] is a common test suite for dynamic human reconstruction and we evaluate 5 of its multi-view sequences with the same 6 training views as used in Watch-It-Move [20]. Finally, we evaluate image quality using peak signal-to-noise ratio (PSNR), structural similarity (SSIM) [114], and learned perceptual image patch similarity (LPIPS) [115] image metrics.

Implementation details We pre-train the TiNeuVox [11] backbone using the authors' implementation and an additional distortion loss [116], as implemented in [117]. Our method is implemented in Pytorch and we train each scene using the Adam optimizer for 160k ( Blender and Robots ) or 320k ( ZJU-MoCap ) iterations with a batch size of 8192 rays, sampled randomly from multiple views and a single timestamp. We choose the canonical timestamp by visual inspection, and gradually increase the number of observed timestamps during training. We adjust ray sampling and scheduling for the ZJU-MoCap dataset (see the Supplement). All experiments were done on a single Nvidia GPU RTX 3090Ti. See the supplementary materials for details. For more details, see the Appendix.

Baselines We compare our method to state-of-the-art non-articulated and articulated methods for high-fidelity multi-view video synthesis (see Table 1 for an overview). D-NeRF [3] extends NeRF to the temporal domain by backward warping a static canonical NeRF. TiNeuVox [11] improves performance of D-NeRF using voxel grids. Hexplane [72] and K-Planes [71] decompose to the space-time volume to several hyper-planes. Finally, Watch-It-Move [20] (WIM) jointly learns a surface representation and LBS model for articulation. Note, that because we aim at time-efficient

WIM

Ours

Ground Truth

Ours

WIM

Ground Truth

b) Simplified

Kinematic Model

a) Learned Kinematic

Model

17.

learning, training of WIM was stopped after 11 hours (i.e., 80k of the original 400k optimization steps).

Novel view synthesis We provide results for the Robots view synthesis without the skeleton simplification; quantitatively in Fig. 4 and qualitatively in Fig. 3. More visual examples are available in the supplement. After the initial pre-training, our method quickly surpasses the image quality of WIM [20]. The mean image scores obtained for our method after 2 hours of training (incl. pretraining) are higher than those that WIM achieves after 10 hours. We attribute this to NeRF's capability to approximate even complex shapes, which are difficult to fit using the signed distance function utilized by WIM. Nevertheless, for visually simple objects with long nested pose transformation chains, this capacity might encourage false explanations of the articulation and cause artifacts, as visible in Fig. 6 left. However, this is not a common issue. We illustrate it in the Blender , where WIM struggles to represent fine details (see Fig. 6 right), while our method achieves image scores close to those of non-reposable baselines (see Table 2 and the supplement).

Furthermore, Fig. 5 shows that our image quality is not highly dependent on the pre-training phase. We observe that mere 100 iterations of TiNeuVox pre-training provide an initialization sufficient for achieving PSNR scores over 30 dB by our method. However, while fine geometric details are still recovered, the coarse initial shape limits the skeleton complexity and, therefore, we opt for 20k pre-training steps in all other experiments.

ZJU-MoCap In Fig. 10, we compare our method to WIM in the ZJU-MoCap dataset. We observe that both methods are able to recover the 3D shape and the skeletal motion despite the difficulty of an accurate fine texture detail reconstruction. This can partly be attributed to imperfections in camera calibrations (see Supplement F of [96]) and partly by soft deformations of clothes which are modeled neither method. Notably, with a small modification our method learns to partially compensate for this. In Ours pose , we condition the feature embedding network Φ p by the skeletal poses jointly learned from scratch by our method. This improves the image quality by modeling the residual deformations. See the Supplement for details. Finally, Ours SMPL shows that replacing our skeleton in Ours pose with a ground truth from an annotated SMPL template [15] does not affect the performance. This validates our skeleton initialization based on Medial Axis Transform.

No

L

No

L

No

L

a)

b)

c)

ARAP

smooth

Skel

GT

WIM

Ours

GT

WIM

Ours

Method

PSNR

↑

SSIM

↑

LPIPS

↓

WIM [20]

31.08

0.963

0.053

Ours

29.60

0.958

0.063

Ours

pose

32.05

0.967

0.056

Ours

SMPL

32.01

0.967

0.056

Reposing In Fig. 7, we visualize the learned blend-skinning weights ˆ w i and skeletons with and without the skeleton simplification. The algorithm is able to substantially reduce the skeleton complexity, while largely preserving semantic articulations observed in the training data. However, it is not able to remove all unnecessary skeletal branches when complex geometry is present (see Fig. 7 right). In Fig. 8, we show that such post-processed skeletons allow for animating of novel poses by smoothly interpolating between user-defined poses. More animations can be found in the Supplement.

Loss ablation Our experiments suggest the regularization does not improve the image quality, but it improves the quality of the kinematic model, which is important for our main goal of reposing (see Fig. 9). Specifically, L ARAP avoids non-rigid distortions (a), L smooth leads to a more semantically meaningful part segmentation (b), L skel encourages functional placement of the skeleton joints inside the object volume even for thin parts (c), and L tranf leads to a reduction of necessary object parts after simplification ( 62 components are removed instead of 49 for Atlas ). More details are provided in the Supplement.

Backbone The TiNeuVox backbone allows us to transfer parameters learned during the pre-training and initialize the features f i and the regressors Φ d and Φ c . Interestingly, our experiment shows that such off-the-shelf features f i often do not need any further fine-tuning (see Table 4). Despite this, our method is agnostic to the backbone choice by design and the end-to-end training procedure of the entire model is essential for this. We demonstrate it by training with a random initialization. This way, only the point positions p c i obtainable by any 3D reconstruction method are needed. Table 3 shows that this leads to only a negligible drop in reconstruction quality in the Robots dataset. Here, the weight of L skel ( w 2 = 1 ) was adjusted to prevent drift of the skeleton.

Table 3: Results with a TiNeuVox initialization and with a random initialization of the features f i and the regressors Φ d and Φ c in the Robots dataset.

6 Discussion

Limitations and Future Work We demonstrate fast learning of articulated NeRFs for state-ofthe-art view synthesis and straightforward skeletal reposing for objects with piece-wise rigid pose transformations. While the LBS allows for fitting partially non-rigid deformations (see Fig. 6 right), representing fully non-rigid, topologically varying, and multi-component objects would benefit from a higher-dimensional parameterization. Chen et al. [102] aim in that direction, but an intuitive and cost-effective reposing still favors skeleton-based techniques such as ours.

The performance of our method depends on the quality of the backbone reconstruction. While TiNeuVox [11] supports reconstruction from even sparse data, a different backbone could offer a more robust starting geometry and improve the skeleton initialization. Although our approach works well even for highly structured shapes and thin structures (see Fig. 6 right), we observe incorrect joint placement for objects with long chains of highly nonlinear geometrical transformations (see Fig. 6 left). Moreover, our approach successfully reduces the number of extraneous bones but it is not able to completely eliminate all superfluous skeletal elements (see Fig. 7).

Our method is restricted to the kinematic motion space exhibited in the training sequences. While extrapolation is possible to some extent, the proposed method is not able to generalize to arbitrary unseen poses. Finally, we focus on image synthesis for user-defined poses rather than a direct fitting of unseen skeletal poses from images or motion capture. An inverse fitting of skeletal poses to novel 2D observations or transfer of poses from one object to another remain opportunities for future research.

Conclusion We presented a method for fast learning of articulated models for high-quality novel view synthesis of captured 3D objects. Our forward-warped neural point clouds avoid the pitfalls of backward warping and elegantly integrate a skeleton-based LBS. As a result, our method merges straightforward reposing even of strongly animated inputs, as present in the Robots dataset, with high visual fidelity of NeRF rendering. Our work is a significant step towards low-cost acquisition of animatable 3D objects for games, movies, and education.

Ethical Considerations Our method renders novel views and poses of 3D objects including human bodies. However, we do not focus on this class and we note that many human-specific methods exist (see Sec. 2). Nevertheless, we acknowledge that our method could potentially be used to produce fake images of people and that research is needed to understand the risks and their mitigation.

Table 4: Results with and without fine-tuning of the feature points f i in the Blender dataset.

References

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV) , 2020.

Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum , volume 41, pages 703-735. Wiley Online Library, 2022.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. arXiv preprint arXiv:2011.13961 , 2020.

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR , 2021.

Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In ICCV , 2021.

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV , 2021.

Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR , pages 9421-9431, 2021.

Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In CVPR , 2021.

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. (SIGGRAPH Asia) , 40(6), 2021.

Tianye Li, Mira Slavcheva, Michael Zollhöfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lv. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5521-5531, June 2022.

Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers , pages 1-9, 2022.

John P Lewis, Matt Cordner, and Nickson Fong. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , pages 165-172, 2000.

Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In ACM SIGGRAPH , page 187-194, 1999.

Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. (SIGGRAPH Asia) , 36(6):194:1-194:17, 2017.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (SIGGRAPH Asia) , 34(6):248:1-248:16, 2015.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR , pages 10975-10985, 2019.

Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15980-15989, 2021.

Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems , 34:19326-19338, 2021.

Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery. arXiv preprint arXiv:2207.03434 , 2022.

Atsuhiro Noguchi, Umar Iqbal, Jonathan Tremblay, Tatsuya Harada, and Orazio Gallo. Watch it move: Unsupervised discovery of 3d joints for re-posing of articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3677-3687, 2022.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5438-5448, 2022.

Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul. Banmo: Building animatable 3d neural models from many casual videos. In CVPR , 2022.

Yuefan Wu*, Zeyuan Chen*, Shaowei Liu, Zhongzheng Ren, and Shenlong Wang. CASA: Categoryagnostic skeletal animal reconstruction. In Neural Information Processing Systems (NeurIPS) , 2022.

SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science , 2018.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In IEEE International Conference on Computer Vision (ICCV) , 2019.

Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Matan Atzmon and Yaron Lipman. SAL: Sign agnostic learning of shapes from raw data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In International Conference on Machine Learning (ICML) , 2020.

Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. Overfit neural networks as a compact shape representation. arXiv preprint arXiv:2009.09808 , 2020.

Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In European Conference on Computer Vision (ECCV) , 2020.

Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision (ECCV) , 2020.

Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: Adaptive coordinate networks for neural representation. ACM Transactions on Graphics (SIGGRAPH) , 2021.

Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. SDFDiff: Differentiable rendering of signed distance fields for 3D shape optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. DeepVoxels: Learning persistent 3D feature embeddings. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In IEEE International Conference on Computer Vision (ICCV) , 2019.

Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. arXiv preprint arXiv:1911.00767 , 2019.

Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (SIGGRAPH) , 2019.

Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, and Markus Steinberger. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum , 40(4), 2021.

Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

David B Lindell, Julien NP Martel, and Gordon Wetzstein. AutoInt: Automatic integration for fast neural volume rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. arXiv preprint arXiv:2104.00670 , 2021.

Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In IEEE International Conference on Computer Vision (ICCV) , 2021.

Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. FastNeRF: High-fidelity neural rendering at 200fps. arXiv preprint arXiv:2103.10380 , 2021.

Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. Eurographics Association , 2020.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In IEEE International Conference on Computer Vision (ICCV) , 2021.

Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14153-14161, 2021.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable large scene neural view synthesis. arXiv , 2022.

Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In IEEE International Conference on Computer Vision (ICCV) , 2021.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In IEEE International Conference on Computer Vision (ICCV) , 2021.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv:2201.05989 , January 2022.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide. Neural point light fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1841918429, 2022.

Jad Abou-Chakra, Feras Dayoub, and Niko Sünderhauf. Particlenerf: Particle based encoding for online neural radiance fields in dynamic scenes. arXiv preprint arXiv:2211.04041 , 2022.

Ruofan Liang, Jiahao Zhang, Haoda Li, Chen Yang, and Nandita Vijaykumar. Spidr: Sdf-based neural point fields for illumination and deformation. arXiv preprint arXiv:2210.08398 , 2022.

Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. arXiv preprint arXiv:2212.08377 , 2022.

Shengze Wang, Alexey Supikov, Joshua Ratcliff, Henry Fuchs, and Ronald Azuma. Inv: Towards streaming incremental neural videos. arXiv preprint arXiv:2302.01532 , 2023.

Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12479-12488, 2023.

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 130-141, 2023.

Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Trans. Graph. (SIGGRAPH Asia) , 36(6), November 2017.

Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In CVPR , pages 6365-6373, 2017.

Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In CVPR , 2021.

Ziyan Wang, Timur Bagautdinov, Stephen Lombardi, Tomas Simon, Jason Saragih, Jessica Hodgins, and Michael Zollhofer. Learning compositional radiance fields of dynamic human heads. In CVPR , pages 5704-5713, 2021.

Yudong Guo, Keyu Chen, Sen Liang, Yongjin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV , 2021.

Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. I M Avatar: Implicit morphable head avatars from videos. In Computer Vision and Pattern Recognition (CVPR) , 2022.

Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzci´ nski, and Andrea Tagliasacchi. CoNeRF: Controllable Neural Radiance Fields. In CVPR , 2022.

Yiyu Zhuang, Hao Zhu, Xusen Sun, and Xun Cao. Mofanerf: Morphable facial neural radiance field. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III , pages 268-285. Springer, 2022.

ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20364-20373, 2022.

Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR , 2021.

Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV , 2021.

Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In NeurIPS , 2021.

Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. In NeurIPS , 2021.

Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In ICCV , 2021.

Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neural human performer: Learning generalizable radiance fields for human performance rendering. In NeurIPS , 2021.

Haotong Lin, Sida Peng, Zhen Xu, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields with learned depth-guided sampling. In arXiv , 2021.

Tao Hu, Tao Yu, Zerong Zheng, He Zhang, Yebin Liu, and Matthias Zwicker. Hvtr: Hybrid volumetrictextural rendering for human avatars. In arXiv , 2021.

Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM Trans. Graph.(ACM SIGGRAPH Asia) , 2021.

Tianhan Xu, Yasuhiro Fujita, and Eiichi Matsumoto. Surface-aligned neural radiance fields for controllable 3d human synthesis. In CVPR , 2022.

Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, and Lan Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7743-7753, 2022.

Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira KemelmacherShlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In CVPR , 2022.

Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang. Selfrecon: Self reconstruction your digital avatar from monocular video. In CVPR , 2022.

Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11594-11604, 2021.

Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In European Conference on Computer Vision , pages 419-436. Springer, 2022.

Yihao Zhi, Shenhan Qian, Xinhao Yan, and Shenghua Gao. Dual-space nerf: Learning animatable avatars and scene lighting in separate spaces. In 2022 International Conference on 3D Vision (3DV) , pages 1-10. IEEE, 2022.

Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: A real-time nerf-based parametric head model. In CVPR , 2022.

Alexander W Bergman, P Kellnhofer, Wang Yifan, Eric R Chan, David B Lindell, and Gordon Wetzstein. Generative neural articulated radiance fields. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022): NeurIPS Proceedings . 2022.

Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Unsupervised learning of efficient geometry-aware neural articulated representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 597-614. Springer, 2022.

Suyi Jiang, Haoran Jiang, Ziyu Wang, Haimin Luo, Wenzheng Chen, and Lan Xu. Humangen: Generating human radiance fields with explicit priors. arXiv preprint arXiv:2212.05321 , 2022.

Hsiao-yu Chen, Edith Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. Virtual elastic objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15827-15837, 2022.

Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. arXiv preprint arXiv:2211.12497 , 2022.

Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7122-7131, 2018.

Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2252-2261, 2019.

Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7763-7772, 2019.

Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo. Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 84-93, 2020.

Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5253-5263, 2020.

Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble. arXiv preprint arXiv:2212.11042 , 2022.

Tianhan Xu and Tatsuya Harada. Deforming radiance fields with cages. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII , pages 159-175. Springer, 2022.

Kaizhi Yang, Xiaoshuai Zhang, Zhiao Huang, Xuejin Chen, Zexiang Xu, and Hao Su. Movingparts: Motion-based 3d part discovery in dynamic radiance field. arXiv preprint arXiv:2303.05703 , 2023.

James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics , 18(3):165-174, 1984.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , pages 333-350. Springer, 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 586-595, 2018.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206.05085 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method for dynamic radiance fields of subjects whose motion can be described by skeletal animation. The method automatically extracts a skeleton using medial axis transform. Further it obtains a object feature point cloud from a pre-trained NeRF and expresses their position as a function of the skeleton via linear blend skinning.\nThe animatable point cloud (including skinning weights and animation) is optimized using the PointNeRF renderer using losses for RGB, rigidity, animation smoothness and 2D CD against the mask. After optimization, it can be manually reposed and rendered from novel views.\n\nThe method is evaluated on the Robots and Blender datasets where it compares favorably against previous work.\n\nSTRENGTHS:\n- Using a point based representation to solve the given task is a good idea and makes sense, given that deformations might be very local and not restricted to a smaller set of rigid parts.\n- The ability for automatic part decomposition and reposing are interesting contributions that adds value with respect to other dynamic NeRFs that might produce better visual results.\n- The method compares favorably against the most relevant previous work WIM.\n- The extracted part decompositions and skinning weights look very good and intuitive\n- The paper is well written and very clear.\n- The authors clearly discuss limitations, show failure cases and provide ablations\n\nWEAKNESSES:\n- The whole work is purely constructive (not many novel conceptual insights) and most parts of the presented method are existing concepts: The method combines NeRF (as TiNeuVox), PointNeRF (adding rotation equivariance), MAT, and ARAP. However, there is novelty in the joint system. I think the positive aspects of this work outweigh this.\n- The method in general is pretty close to some of the methods for animatable human radiance fields [80-93]. It differs mostly in that it does not rely on a human template but extracts one from scratch and thus, also works on other subjects. The paper would profit from a comparison with one of them on a human subject though.\n- In general, while relevant related work is mentioned, it is not always compared against. Some relevant examples are NeRFies [6] and NSFF [4], which both are known for producing better results than D-NeRF, which is used in the comparisons.\n\nQUESTIONS:\nIt does not become clear to me what exactly is done with the features $\\mathbf{f}_i$ coming from TiNeuVox. Are they just used as point feature initialization and then further optimized using PointNeRF or are they kept fixed? I can imagine that the PointNeRF renderer could adapt to these features without modifying them. This boils down to the question of which parameters of the model are optimized when, which seems to be not completely clear in the paper.\n\nLIMITATIONS:\nLimitations and potential negative societal impact are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors presents a method to learn articulated model from multi-view video. And demonstrate an ability for efficient learning skeleton pose along as view synthesis model for dynamical structures.  Moreover the suggested method drastically improve convergence compared to naive approaches. To extract the skeleton authors apply Medial Axis Transform for initalization and RBF weights for the joints.\n The results on the D-NeRF dataset are demonstrate the superiority of this model over predecessor.\n\nSTRENGTHS:\nI appreciate the ideas in the method especially skeleton construction and mask loss.\n- The approach doesn’t take into account pre-defined joint structure\n- Convergence is quite fast\n- Can handle non-rigid motions (in theory) \n- This a first method learning articulation model purely from the input images\nThe description of the method is clear and solid.\n\nWEAKNESSES:\nThe main idea is interesting, but the comparison is poor. \nThe idea of skelton initializing is mostly based on the previous works, and the it is not clear how much the final quality degrades, it is necessary to compare it with the ideal human skeleton, for example.\nTo apply this technique you first have to pre-train a dynamic version of NeRF, that complicates method compared to others. \nAll experiments are done for a single backbone and extremely limited variety of data\nAs you mentioned this method assume single object in the centre of the scene.\nSimplified model produces unrealistic results (figure 6) the skeleton even cannot cover actions from training.\nThere are some works (e.g. TAVA) that incorporate idea of non-template methods (fairly they utilize keypoints) that have to be mentioned\nHuman related baselines can be interesting here.\n\nQUESTIONS:\nWhy do you need  time-dependent rotations?\nDoes your training time include pre-training of the dynamic scene in Tab 2.\nDo you need ARAP and transformation loss, how do they influence on the training.\nHave you consider using something different to TiNeuVox method, there are plenty of much faster and accurate versions (e.g. HexPlane, Tensor4D)?\n\nLIMITATIONS:\nHighly relying on the retrained method\nSpeed and memory consumption for higher resolution \nSimplification and skeleton construction itself is based on heuristic and can be very poor on some scenes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

comments: SUMMARY:
The method reconstructs a reposable Dynamic NeRF of an articulated object from multiview videos. This is achieved by using linear blend skinning (LBS) of an automatically extracted skeleton to represent the deformation from canonical to observation space.

STRENGTHS:
Combining LBS kinematics with NeRF appearance model effectively produces compelling reposing results.
Yielding a posable skeleton enables numerous traditional animation workflows.

WEAKNESSES:
A quick literature search yielded https://arxiv.org/abs/2208.14851 which talks about reposing with a different way of avoiding inverse mapping issues. It may warrant a mention in related work. Could that approach be combined with this paper's automatic skeleton extraction (they use a known rigged template mesh) to yield similar results?
 
Given that the method yields a posable skeleton, the results could have been much more compelling with artist-made or even mocap-driven animations (rather than just blending between a couple of poses).

QUESTIONS:
Thank you for the supplemental video. I think the Ground Truth T-Rex needs to also be animated.

Line 95 mentions Fi_d as a backward-warping MLP, did you mean Fi_b?

Line 98 uses p' in an expression for v_m, was that supposed to be p^c?

Specific descriptions of various MLPs listed in line 101 would be helpful.

In line 103, did you mean to use delta_i instead of just delta?

Is the N in equation 8 (L_arap) the same N as in L_smooth?

How come the simplified skeletons for the man and dinosaur are so sparse, compared to the two robots?

LIMITATIONS:
The authors address the main limitations of their approach, namely that LBS restricts the method to individual rigidly-linked bodies (as opposed to deformable objects or full scenes).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach to articulated view synthesis, introducing the concept of Template-free Articulated Neural Point Clouds. The authors utilize a structure-free point-based NeRF representation which supports forward-warping of canonical objects to any poses through Linear Blend Skinning (LBS). Such a representation allows for the joint optimization of LBS pose parameters as well as dynamic NeRFs in a short training time. The method is evaluated on two datasets and compared with existing methods, such as D-NeRF, TiNeuVox, and WIM, showing superior or comparable performance on dynamic novel view synthesis and novel pose synthesis.\n\nSTRENGTHS:\n(+) The paper provides thorough explanations of the model and experiments.\n\n(+) The paper proposed an effective point-based NeRF representation to support the forward-warping of the canonical space for template-free objects.\n\n(+) The method achieves better novel view synthesis and reposing results in a shorter training time on two datasets compared to existing approaches. The visualized learned LBS weights fields and poses are cleaner than WIM.\n\nWEAKNESSES:\n(-) Inconsistent Motivation: The paper's motivation appears to be inconsistent throughout the text. Initially, the primary motivations are identified as 1) **Reposability** and 2) **Efficiency**. However, in the model section, the method is built upon a pre-trained TiNeuVox as initialization, aiming to animate TiNeuVox to new poses. Consequently, the efficiency of the proposed method heavily depends on the backbone/representation used, and the main motivation here seems to be **Reposability**. In the experimental section, the method primarily compares with WIM to demonstrate its efficiency over WIM on novel view/pose synthesis. A more convincing demonstration of the **Reposability** performance of the proposed method would involve comparing it with WIM in __its original training iterations__. To prove the **Efficiency** of the method, it would be beneficial to investigate the influence of different backbones/representations on the proposed method.\n\n(-) Performance and Efficiency: The performance and efficiency of the proposed method do not seem promising compared to TiNeuVox. Given that the method builds upon a pre-trained TiNeuVox as initialization and aims to animate TiNeuVox to new poses, one would expect the method to perform at least similarly to TiNeuVox on the novel view synthesis task. However, as shown in Table 1 of the supplementary material, the method performs worse than TiNeuVox on 5 out of 7 scenes and requires significantly longer training time.\n\n(-) Dependence on Pre-trained Model: The proposed method relies on a pre-trained model as initialization, whereas WIM does not require such a pre-trained model. For a fair comparison, it would be advisable to also compare with a modified version of WIM that is initialized from a pre-trained model. This comparison would provide a more balanced view of the strengths and weaknesses of the proposed method.\n\nQUESTIONS:\n* Please refer to the weaknesses section.\n\n* How to optimize blend skinning weight vector $w_i$? And what is the motivation for defining another $\\alpha$ for scaling the weights?\n\n* $\\Phi_d$ should be $\\Phi_b$ in line 95 to represent the backward-warping MLP.\n\nLIMITATIONS:\n* The authors could discuss how variations in image quality, such as resolution, lighting conditions, and the presence of noise, might impact the performance of their method.\n\n* How well does the method perform when applied to longer sequences or more complex scenes?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the task of dynamic novel view synthesis from multiview videos and aims for the ability of reposing. It tackles the problem with a point-based rendering approach. More importantly, it does not need any pre-defined or class-specific template/skeletons and learns a per-video data-driven skeleton. The effectiveness is verified on two datasets, i.e., Robots and Blender.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n\n2. The proposed skeleton learning is quite interesting: using a data-driven point cloud from TiNeuVox; then approximating an initial skeleton via Medial Axis Transform (MAT); finally using another data-driven component to refine the skeleton from MAT.\n\nWEAKNESSES:\n### 1. Robustness to Initialization\n\na. The proposed approach heavily relies on the NeRF pretraining: 1) geometry-wise, the initial skeleton needs the pre-trained NeRF's density function to sample points, which will be later used to run MAT; 2) appearance-wise, the points' features come from the pre-trained NeRF.\n\nb. A natural question is how robust the proposed approach is to the initialization. One way to demonstrate this could be: 1) train TiNeuVox until convergence; 2) initialize the proposed approach from different checkpoints of TiNeuVox's training; 3) draw a plot of the final performance wrt different initializations.\n\nc. What is related is: can authors clarify what the statement \"we uniformly sample the density $\\sigma$\" (L124) means? Does it mean that you will first compute densities on plenty of points to get a sense of min and max densities? And how will you sample the point cloud with such min/max values?\n\n### 2. Evaluation of Human Dataset\n\nArguably, one of the most important articulated view synthesis scenarios is human-related rendering. I am wondering whether authors can provide quantitative and qualitative results for the proposed approach on some human-related datasets, e.g., ZJU-MoCap used in [20]. Such results could provide a more complete assessment of the proposed approach.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe authors provide a discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}