Title: TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D reconstruction

Template-free NeRF

Semantic reconstruction

Multiple entity interactions
Abstract: Despite advancements in Neural Implicit models for 3D surface reconstruction, handling dynamic environments with interactions between arbitrary rigid, non-rigid, or deformable entities remains challenging. The generic reconstruction methods adaptable to such dynamic scenes often require additional inputs like depth or optical flow or rely on pre-trained image features for reasonable outcomes. These methods typically use latent codes to capture frame-by-frame deformations. Another set of dynamic scene reconstruction methods, are entity-specific, mostly focusing on humans, and relies on template models. In contrast, some template-free methods bypass these requirements and adopt traditional LBS (Linear Blend Skinning) weights for a detailed representation of deformable object motions,
although they involve complex optimizations leading to lengthy training times. To this end, as a remedy, this paper introduces TFS-NeRF, a template-free 3D semantic NeRF for dynamic scenes captured from sparse or single-view RGB videos, featuring interactions among two entities and more time-efficient than other LBS-based approaches. Our framework uses an Invertible Neural Network (INN) for LBS prediction, simplifying the training process. By disentangling the motions of interacting entities and optimizing per-entity skinning weights, our method efficiently generates accurate, semantically separable geometries. Extensive experiments demonstrate that our approach produces high-quality reconstructions of both deformable and non-deformable objects in complex interactions, with improved
training efficiency compared to existing methods. The code and models will be available on our github page.

TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene

Abstract

Despite advancements in Neural Implicit models for 3D surface reconstruction, handling dynamic environments with interactions between arbitrary rigid, nonrigid, or deformable entities remains challenging. The generic reconstruction methods adaptable to such dynamic scenes often require additional inputs like depth or optical flow or rely on pre-trained image features for reasonable outcomes. These methods typically use latent codes to capture frame-by-frame deformations. Another set of dynamic scene reconstruction methods, are entity-specific, mostly focusing on humans, and relies on template models. In contrast, some templatefree methods bypass these requirements and adopt traditional LBS (Linear Blend Skinning) weights for a detailed representation of deformable object motions, although they involve complex optimizations leading to lengthy training times. To this end, as a remedy, this paper introduces TFS-NeRF, a template-free 3D semantic NeRF for dynamic scenes captured from sparse or single-view RGB videos, featuring interactions among two entities and more time-efficient than other LBS-based approaches. Our framework uses an Invertible Neural Network (INN) for LBS prediction, simplifying the training process. By disentangling the motions of interacting entities and optimizing per-entity skinning weights, our method efficiently generates accurate, semantically separable geometries. Extensive experiments demonstrate that our approach produces high-quality reconstructions of both deformable and non-deformable objects in complex interactions, with improved training efficiency compared to existing methods. The code and models will be available on our github page .

1 Introduction

With the rapid advancements in deep learning, the field of 3D geometry reconstruction of static and dynamic scenes and objects has experienced significant transformation, largely due to its crucial role in diverse applications such as Augmented Reality (AR), Virtual Reality (VR), robotics, autonomous navigation systems, and human-robot interactions (HRI). While primarily focused on novel view synthesis, Neural Implicit models (NeRFs) [1] have recently made remarkable strides in 3D surface reconstruction due to their ability to learn detailed geometry without needing prior knowledge about the shape of the scene elements or direct 3D supervisions [2, 3, 4, 5, 6]. Despite these advancements, challenges remain in developing models that effectively generalize across varied real-world dynamic environments, particularly those involving arbitrary rigid, non-rigid, or deformable entities (such as humans or animals) engaged in complex interactions . Moreover, achieving semantic reconstruction that accurately captures the geometry and positioning of each semantic scene element independently is essential for enhancing functionalities in applications like AR, VR, and HRI.

In NeRF-based dynamic scene reconstruction, the focus has predominantly been on human reconstruction [7, 8, 9, 10], utilizing template models such as SMPL [11], and CAPE [12]. However, these methods struggle with generalizability to arbitrary deformable entities and primarily concentrate on single-entity reconstructions, neglecting interactions among multiple entities, such as interactions of humans with scene objects. Notably, HOSNeRF [13] by Liu et al . addresses human-object interactions but remains dependent on the SMPL model.

Concurrently, some dynamic NeRFs [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26] aim beyond human and focus on either rendering or geometry reconstruction of generic scene/object. The surface reconstruction methods mostly require additional data, such as depth or optical flow, to optimize the underlying 3D. For instance, [16, 17] focus on building shape-prior-free 3D models for deformable entities. However, besides requiring a large number of casual videos with diverse view coverage of the subject, these approaches heavily depend on highquality optical flow for supervision, which makes their reconstruction quality subject to the accuracy of the pre-trained models for generating such data. Moreover, generally, the dynamic NeRF methods use latent codes to learn per-frame deformations [15, 27], which may not be sufficient for capturing accurate articulation of arbitrary deformable ob-

A

B

C

D

A

B

C

D

A

C

B

D

jects. In a concurrent work, Li et al . [28] bypass the need for such additional information and develop a template-free model from only sparse view RGB videos. They also learn a more accurate representation of body deformation, the forward LBS [11] ( i.e ., mapping of canonical points to posed or view space), which helps them achieve impressive shape reconstruction results for different deformable entities. However, their method suffers from a lengthy training convergence time. Furthermore, none of these template-free approaches have focused on interactions between more than one entity nor achieved semantic reconstruction of scenes, highlighting a gap in the current methodologies.

Our contributions: In this paper, we aim to develop a time-efficient 3D semantic NeRF for dynamic scenes captured from sparse view RGB videos involving interactions between two rigid, non-rigid, or deformable objects. Built upon [28], we propose a novel framework to learn the forward LBS by utilizing INN [29] to bypass the computationally demanding root-finding approach used in [28]. This adjustment boosts the efficiency of our training process, which is shown with an experiment in the results section. Additionally, our proposed framework can produce semantically separable reconstructions for the scene entities. The challenge of building template-free 3D models for two dynamic entities engaged in complex interactions is amplified by occlusions and their diverse shapes or deformations. To tackle these challenges, we propose a strategy to disentangle the motions of distinct entities within the scene. Specifically, we first perform semantic-aware ray sampling and learn the independent transformation of each entity from the deformed space to the canonical space to optimize per-entity skinning weight. This process allows us to accurately generate the individual Signed Distance Fields (SDF) from the disentangled canonical points, enhancing our model's ability to handle complex dynamic environments under interactions effectively. The efficacy of our proposed method is demonstrated through comprehensive experiments conducted on several datasets featuring human-object, hand-object interactions, and animal movements. Results consistently show superior performance compared to the best-performing state-of-the-art methods. Our noteworthy technical contributions are, therefore:

-We introduce a time-efficient template-free NeRF-based 3D reconstruction method for dynamic

scenes from sparse multi-view/single videos featuring two interacting entities.

-Our approach extends to the semantic reconstruction of dynamic scenes, emphasizing the detailed capture of each entity's explicit geometry within the scene.

-The efficacy of our proposed method is demonstrated through comprehensive experiments conducted on diverse datasets featuring interaction between rigid, and non-rigid entities.

2 Related Works

Human-object reconstruction: Several model-based approaches [30, 31, 32, 33, 34, 35, 36] have explored 3D reconstruction of humans and objects under interactions. However, a common limitation is their reliance on parametric models ( e.g ., SMPL) for human reconstruction. While SMPL provides a robust base, it limits flexibility in reconstructing diverse deformable entities and cannot generalize beyond humans. Also, it does not capture finer details such as hair dynamics and clothing deformations. In contrast, NeRF-based reconstruction offers a promising alternative for capturing detailed geometric information but mainly focuses on human reconstruction under a dynamic environment. Yet, only a few NeRF-based methods [13, 37, 38] have addressed the reconstruction involving multiple objects. For example, Jiang et al . [37] consider modeling the background, along with dynamic human by designing two separate NeRFs i.e ., human NeRF and background NeRF . In a recent work, HOSNeRF, Liu et al . [13] introduce a NeRF-based approach for free-viewpoint rendering of dynamic scenes featuring human-object interactions. Even using the NeRF-based approach, these methods still rely on the SMPL model for initialization. Moreover, unlike our method, most of these methods focus on novel view or pose synthesis and do not emphasize detailed geometry reconstruction.

Generic Scene Reconstruction: Some NeRF-based methodologies [14, 15, 27, 40, 39, 41, 25, 26] offer generic scene reconstruction under a dynamic environment without relying on any parametric template models, making them favorable for extending to any deformable object reconstruction. Pumarola et al . [14] extend the traditional NeRF framework by integrating time as an additional input, which facilitates the mapping of each frame's deformed scene to a canonical space. Building on this, T-NeRF [42] utilizes time-

varying latent codes to condition the NeRF, improving training speed and rendering quality. Cai et al . [15] propose using an INN to learn the mapping between deformed and canonical space and show impressive surface reconstruction results optimized from RGB-D videos. HyperNeRF [27] incorporates an additional Multilayer Perceptron to learn the frame-specific topology variations via ambient codes, capturing scene deformations more effectively. However, these methods mainly rely on latent codes to capture the relation between frames at different timestamps or topological variations within a frame. A recent approach Tensor4D [25] represents dynamic scenes as a 4D spatiotemporal tensor but does not account for the relative motions between scene elements. In contrast, our approach considers learning the structural relations between different parts of the scene or the object under reconstruction e.g ., body parts of humans in case of human reconstruction, etc .

Reconstruction with predicted LBS: Several methods have recently emerged that aim to develop generic NeRFs focusing on arbitrary deformable object reconstruction [16, 17, 28]. These methods utilize more specific topological representation, i.e ., LBS, that helps understand how different parts of the deformable body are connected and how they deform from the canonical pose under motion. Specifically, given the forward LBS weight w ( x c ) for a canonical point x c on the surface, the corresponding deformed point x v (in the viewing space, i.e ., per-frame observations) is computed using a weighted combination of bone transformation matrices B [11] defined as,

!

!

"

!

!

#"

Ω

!

"

: Deformable SDF

!

$

#"

∇Ω

"

: Normal

Ω

!

#"

: Non-deformable SDF

∇Ω

#"

: Normal

!

$

"

Volume

Rendering

Inverse

Warping

Inverse

Warping

ℱ

!→%

"

ℱ

!→%

#"

$

&→!

#"

$

&→!

"

ℱ

%→'()

A

B

C

D

m

where n b represents the total number of bones. Unlike human reconstruction NeRFs [7, 8, 9, 10], these methods learn this skinning weight function w ( x c ) , that allows for the flexible adaptation of these models to any deformable objects. Yang et al . [16] learns a forward-backward deformation field for mapping between canonical and deformed space and represents the skinning weight using a set of Gaussians defined around the bones. Whereas Li et al . [28] and Chen et al . [41] learn the skinning weights and formulate the mapping from deformed to canonical spaces as an iterative root-finding problem. They solve this problem using Broyden's method, which involves matrixvector multiplications and computationally expensive matrix inversions at each iteration, leading to a lengthy training convergence time. Additionally, these methods consider reconstructing only a single entity within the scene. In contrast, our method addresses the reconstruction of two moving objects under complex interactions, which is more challenging due to the highly unconstrained nature of the problem.

3 Methodology

Overview. Given sparse-/single-view videos of deformable objects, such as hands, humans, or animals, interacting with non-rigid/rigid objects like balls or boxes, our goal is to accurately learn the semantically separable geometry of each entity under interactions. Fig. 2 provides an illustrative overview of our methodology, highlighting the following key components and steps involved in the process. A) Semantic-aware ray casting and sampling to distinguish individual elements within the scene (Fig. 2A), B) INN to learn the forward skinning and deformation of each element, enhancing the efficiency of the training process (Fig. 2B), C) SDF module to independently learn the geometry of each element (Fig. 2C) and D) RGB Renderer module to generate RGB values from individual SDF volumes for the final rendering (Fig. 2D).

A. Semantic-aware ray sampling. To reconstruct semantically separable geometry, it is essential that each 3D point in the reconstructed surface is tagged with a semantic label to denote its object affiliation. Previous efforts [5] propose to model a compositional scene within a shared network structure, which is challenging in the dynamic scenario for the following two reasons. First, it is hard to supervise the SDF of each entity from different frames by 2D semantic segmentation due to its dynamic nature. Second, the interaction between humans and objects in different frames is complex, considering the occlusion between each other. These issues provoke us to design a network structure with natural disentanglement for the compositional modeling.

To address these challenges, our network is designed with distinct modules for SDF prediction for each entity, maintaining semantic continuity from the input space to facilitate the prediction of semantically separable 3D geometry and joining them with the rendering module for composite rendering. Specifically, leveraging the 2D semantic mask of the input image, we sample two sets of rays surrounding the interacting objects. Along these rays, we sample 3D points to generate

two distinct sets: one representing the deformable object, { x d v } N i =1 , and the other representing the non-deformable object, { x nd v } N i =1 . However, only 2D semantic-aware ray sampling is not sufficient for accurate disentanglement of the individual entities, as the rays can intersect both entities in their path through the 3D space due to the occlusion of one object by the other. Hence, we also perform an encoding of the 3D points based on their distance from the 3D skeletons of individual entities. Please refer to Section 3C for more details about the semantic encoding of the 3D points. These points are then processed through separate networks tailored to each object type to get separate geometry and finally merged at the rendering module. This approach enhances our ability to effectively separate and reconstruct individual elements, even in scenarios of strong occlusions. This meticulous sampling strategy guarantees that supervision from individual entity's observation only influences each SDF prediction module.

B. View space to canonical space deformation. In dynamic NeRF, a common approach is mapping each frame to a fixed canonical frame to learn the correspondence between frames. For deformable objects, like humans, this is generally achieved by using pre-defined LBS weight from template models like SMPL etc . [7, 8, 39, 40]. LBS is a technique used to deform the surface points on a human mesh at the canonical pose to each frame's pose based on the positions and orientations of an underlying skeleton (Eqn. 1). A few methods, e.g ., TAVA [28] and SNARF [41] extend this approach beyond humans and learn the forward LBS for arbitrary deformable entities without using any template model. However, for learning the forward LBS, defined in canonical space, the main challenge lies in finding the correct correspondence in the canonical space for a given deformed point because of the implicitly defined correspondences without an analytical inverse form of the Eqn. 1. Hence, they formulate the problem of learning this correspondence as a root finding problem and solve for x c s.t ., LBS ( w ( x c ) , B , x c ) -x v = 0 . They solve it numerically using iterative Broyden's method [43], which involves matrix-vector multiplications and computationally expensive matrix inversions at each iteration and must be optimized for multiple iterations for each iteration of NeRF optimization. Hence, this makes the training process time-consuming. To overcome this challenge, we replace this root finding problem and instead use an INN [29] for transforming the view space points to canonical space and learn the skinning weight w ( x c ) simultaneously. Conditioned by each frame's pose, INN can correctly learn the bijective mapping between the deformed and canonical space through a consistency loss and bypassing the iterative optimization process, resulting in time-efficient training. Fig. 3 shows the workflow for the view to canonical space conversion framework.

First, function G INN v -> c is used to transform the points from view space to canonical space using an INN defined as [29]. To enhance the convergence

!

$

%

Inverse Warping

!

!

Invertible

Neural

Network

Skinning

Weight

MLP

"′

!

!

"

"

#

!

#$

$

%

"

(

INN Loss

!

!

!

!

"

&

#

!

#$

$

%

$

)

$

&→!

*++

$

,

!

. →-

!

$

*#*+

" $&#' and provide a better initialization for the INN network, we first transform the deformed space points x v to give an approximation of the canonical space points. Under the intuition that the sampled points x v near the posed skeleton will remain close to the canonical skeleton at canonical space, x v are transformed to x init c = ( ∑ n b i =1 w ′ . B i ) -1 . x v , where, w ′ represents one-hot vectors defining the nearest joint of the posed 3D skeleton J p ∈ R n b from the view space points x v . An ablation study is presented in Tab. 6 to show the effectiveness of this initialization. The INN, G INN v -> c is conditioned on per frame 3D skeleton J p . Then a skinning weight prediction network G x W c -> w is used to predict the skinning weight w s at canonical space from the predicted canonical points x ′ c .

Finally, the canonical points are calculated as, x c = ( ∑ n b i =1 w s . B i ) -1 . x v , that are given as input to the SDF prediction network. This helps to constrain the skinning weight prediction network. As skinning weight defines the weightage of underlying skeleton joints for the deformation of each vertex on the mesh surface; it helps capture better articulation or deformation and surface reconstruction under motion. Moreover, it implicitly learns the spatial relationship between the surface points, resulting in smoother reconstruction compared to the approaches where no skinning weight representation is learned. This is evident in Fig 5, given a comparison with such methods.

C. Learning the SDF volume of individual elements. Given our aim to generate semantically separable geometries of the scene entities, we use two independent SDF networks defined as F j c -> Ω ( θ Ω ) for deformable and non-deformable objects j ∈ { d, nd } . The individual set of transformed canonical points x c are passed through these geometry prediction networks, defined as F j c -> Ω ( θ ) : R 3+3+ n b → R 1+256 (SDF and a global feature representation of dimension 256) produces surface reconstruction at canonical space, Ω j c .

The SDF prediction network is conditioned on the canonical skeletons J p0 ∈ R n b . Only semanticaware (based on a 2D semantic map) ray sampling is not sufficient to differentiate the points in 3D. For this purpose, to provide the SDF prediction network information about which object the point belongs to, we assign a semantic label to each 3D point. This is defined as a weightage, ω j = exp( -dist 2 /σ 2 ) based on the distance of the point from the nearest 3D joints in the per entity canonical skeletons J j p0 . The points far from the canonical skeleton of either of the entities are assigned to the background, with weight defined as ω bg = 1 . 0 -clamp ( ∑ j ω j , 1 . 0) . This semantic label with dimensionality 3 is concatenated with canonical point and pose input ( 3 + 3 + n b ) and passed through the SDF prediction network that predicts SDF and a global feature representation.

D. Compositing for final rendering. We use a single RGB prediction network to predict the final rendering. For this purpose, the predicted canonical points (3), normals calculated from SDFs (3) [39], geometric features (256) and posed skeletons J p0 ( n b ) for individual elements are concatenated before sending through a unified RGB prediction network F Ω ->rgb Following Guo et al . [39] we predict the texture in canonical space and condition the texture generation network with normal calculated from SDFs to consider the deformation from view space points to canonical points. This is defined as F Ω ->rgb : R 3+3+ n p +256 → R 3 .

Training: All the modules defined above are trained jointly over all the frames of the given video. The training losses used for the global optimization are defined as follows:

Reconstruction loss: For optimizing the NeRF, reconstruction loss is defined between the rendered RGB ˆ C ( r ) and the RGB C ( r ) of input pixel along the ray r , L rgb = ∑ r ∈ R ∥ ˆ C ( r ) -C ( r ) ∥ .

Due to the lack of direct supervision of the skinning weight prediction network or INN, several incorrect combinations of canonical points and skinning weights can satisfy the forward LBS Eq. 1. Hence, the following supervisions are used on the skeleton space to constrain these two networks.

Pose loss: To ensure that the INN network learns a correct mapping between the view space and the canonical space, a loss is defined on two sets of points ( X J p 0 , X J p ) sampled around the bones of canonical ( J p0 ) and deformed skeletons ( J p ) respectively. The following loss is applied to ensure that the INN correctly transforms the point set X J p to X J p 0 ,

where P is the total number of points sampled around the bones from each skeleton.

Skinning weight loss: To constrain the skinning weight prediction network, a loss is applied to ensure that the predicted skinning weight for the canonical joints is a one-hot vector (1 for the respective joint and 0 for rests), ˆ w .

Cycle loss: Conventional cycle consistency loss is used for optimizing the INN,

where H is the transformation learned by the INN between view and canonical space. The inverse function H -1 ( x v , J p 0 ) transforms the view space point to canonical points, and the forward function H ( x c , J p ) transforms back the canonical points to deformed space conditioned by the posed skeleton.

Consistency loss: To ensure that the INN transformed point x ′ c and the final skinning weight conditioned canonical points x c (Fig. 3) are close to each other we also minimize the L 2 distance between these two sets of points.

In shape loss: Following Guo et al . [39], to accelerate the learning process, a loss ( L shape ) is defined around a point cloud initialization for the individual entities. The point cloud is defined around the canonical skeleton of individual elements. This loss ensures that the transformed points x c that fall within this point cloud have the sum of weights predicted from NeRF densities α = 1

[39]. The full loss, minimized over each video frame, is defined as follows, where λ skel , λ W , λ INN , L consis , λ shape are empirically set to 2,10,1,1,0.03 respectively.

Evaluation: As we learn the object geometry at canonical space, at inference time, we can directly sample points around the canonical skeleton and pass through the SDF prediction network F j c -> Ω ( θ ) (Fig. 2C) to predict the canonical SDF. Then, for the canonical mesh vertices, we can predict the skinning weights using the skinning weight prediction network G x W c -> w (Fig. 3) and generate the final posed mesh (viewing/deformed space) by following Eqn. 1.

Implementation Details: The overall network proposed in Fig 2 is trained end-to-end, with a learning rate of 5 . 0 e -4 . We use ADAM optimizer for the SGD optimization and PyTorch library for the implementation of our method. All training and inference have been performed in NVIDIA RTX 4090 GPU. Details about the network architecture are presented in the Appendix.

4 Experiments

In this section, we first demonstrate the effectiveness of our method for precise 3D reconstructions of deformable and non-deformable objects, as well as their interactions. Next, we present qualitative and quantitative comparisons of our approach with relevant state-of-the-art methods. Additionally, we conduct a comprehensive ablation study to analyze the impact of different network design choices and loss formulations on our model's performance. Finally, we discuss the efficiency of our method in terms of training convergence time compared to existing approaches in the literature. Datasets. To evaluate our reconstruction under multiple entity interactions, we select the BEHA VE [31] with human-object interactions and HO3D-V3 [44] with hand-object interactions. We tested sequences featuring large motions of humans and objects, training with 45-50 images per camera view to optimize 3D geometry. Following [28], all methods utilize dataset-provided camera poses, body poses, and masks for training. We also evaluate our method for reconstructing single deformable entities (only human/animal) and use a similar setup as proposed in TAVA [28]. For this purpose, we test performance on two datasets: the ZJU-MoCap dataset [9] for human reconstruction and a synthetic dataset for animal reconstruction from [28].

Baseline. For human-object reconstruction, we choose methods that focus on generic scene reconstruction without using any template 3D models for a fair comparison of our template-free model. Specifically, we benchmark our approach against state-of-the-art generic scene reconstruction methods such as Tensor4D [25], NDR [15], HyperNeRF [27], and D-NeRF [14]. Additionally, we consider the recent approach ResFields, which models large and complex temporal motions by introducing temporal residual layers into the general NeRF architectures, showing significant improvements over the baselines [45]. We train NDR [15], HyperNeRF [27], and D-NeRF [14] with and without ResField layers on RGB videos. We haven't trained our method with ResField. For human reconstruction, we compare with methods that learn the skinning weight field [28, 27]. TAVA [28] learns the skinning weight from scratch without using any template model, whereas AnimatableNeRF [9] initializes the skinning weight from SMPL and learns a residual weight to optimize the final skinning weight field.

Evaluation metrics. For quantitative evaluation, following previous 3D reconstruction NeRFs [5, 6], we measure distance metrics (after registration between predicted and ground-truth meshes) such as Average Distance Accuracy (Dist. Acc.), Completeness, and Chamfer Distance (CD), along with Precision, Recall, and F-score (defined with a threshold of 5cm). Following previous methods [39], we use SMPL meshes provided in the ZJU-MoCap and BEHAVE datasets as ground-truth to evaluate human reconstruction quality to obtain the quantitative results. Similarly, we utilize the ground-truth object meshes within the BEHAVE dataset for object reconstruction evaluation.

Results and discussions. - Reconstruction of scenes with two entities under interactions: Our evaluation methodology for human-object reconstruction focuses on two critical dimensions: 1) Holistic Scene Reconstruction : We assess the entire scene's reconstructed mesh quality (human+object) against the ground

truth, providing a comprehensive measure of the scene's accuracy (Tab. 3). 2) Semantic Reconstruc-

tion : We asses the reconstruction quality of individual scene elements by comparing the accuracy of reconstructed humans and objects to their respective ground truth meshes (Tab. 2). Our method excels in holistic scene reconstruction across all metrics (Tab. 3) and delivers superior results in semantic reconstruction, especially for humans (Tab. 2, upper rows) and most objects (Tab. 2, lower rows). Competing methods, focusing on the prior-free linkage between observation and canonical space, often neglect the topological relationships essential for dynamic scenes. While they are effective for rigid entities, they underperform in reconstructing high-quality, deformable entities. Our approach leverages learning skinning weights to capture the intricate relationships between body parts, enhancing reconstructions for both deformable and non-deformable objects (Fig. 5).

-Reconstruction of scenes with hand-object reconstruction: We evaluate our method also on the HO3D-V3 dataset [44] and present comparative results in Tab. 4. For this purpose, we choose two baseline methods, i.e ., NDR and HyperNeRF trained with ResField, that best perform on the BEHAVE dataset. Tab. 4 compares semantic reconstruction quality. We use the ground-truth mesh for hand from the HO3D-V3 dataset and the ground-truth mesh for objects from YCB-Video 3D models for calculating the metrics. Our method achieves better reconstruction, showing superior performance on most metrics.

Also, we evaluate our method for single-object reconstruction, including arbitrary deformable entities,

-Human reconstruction: We compare our human surface reconstruction results

with TAVA [28] and AnimatableNeRF [7] on the ZJU-MoCap dataset [9] (Tab. 5, upper table). TAVA employs a template-free approach, while AnimatableNeRF uses the SMPL body model. Our method surpasses both TAVA and AnimatableNeRF in performance. The qualitative comparison, shown in Fig. 4, highlights the superior surface reconstruction of our method on the ZJU-MoCap dataset. SDF modeling contributes smoother surface reconstructions compared to models like [28, 7].

Ours

TAVA[28]

AnimatableNeRF[7]

I/P

Reconstruction of other deformable entities: To evaluate our reconstruction method on other deformable entities, we employ a similar experimental setup as used in TAVA [28]. Tab. 5 (lower table) presents the quantitative comparison using the synthetic animal dataset introduced by TA V A. Our method shows comparable performance with the baseline method. Fig. 1 displays qualitative results from one of the animal subjects in the dataset.

Ours

NDR [15] + ResField[45]

NDR [15]

I/P

HyperNeRF [27] + ResField [45]

HyperNeRF [27]

Ablation studies: We present an ablation study for different network design choices and losses in Tab. 6. Without initializing the INN with x init c : In this experiment, we use an MLP to learn the initialization, following traditional INN networks [47], instead of initializing based on the distance of the deformed points from the posed skeleton (see Section 3, B.). Our initialization method leads to better reconstruction performance. With the same geometry head : Here, we use a unified network architecture with a single INN for both deformable and non-deformable objects, for predicting skinning weights and SDF. Rather than using semantic-aware ray sampling in image space for disentangling motions and geometries of entities (see Section 3, A), a semantic logit is predicted from the SDF and optimized with a semantic loss to produce semantically separable geometries following [5]. This approach struggles with high occlusion and complex interactions, resulting in poor reconstructions (Figure 6). Without L W and L pose loss : These losses are crucial for constraining the prediction networks. The L W has a significant impact, effectively constraining the INN and skinning weight prediction, improving the reconstruction quality. Training convergence for W and W/o Broyden method: We experiment to assess the efficacy of our INN network compared to the Broyden-based LBS learning approach. For this purpose, we train our network to replace the INN network and use the Broyden equation solver to solve the Eqn. 1 and report the progress of CD with respect to time (Fig. 7). Our network, designed with INN, converges much faster than the Broyden method used in [28, 41]. Also, our approach takes around 0.3 sec/frame with INN compared to 0.9 sec/frame with the Broyden approach.

Further study and limitations: Although we successfully disentangle motions of the interacting entities, our framework currently employs separate networks for each entity, which is not scalable for scenarios involving more than two entities. This limitation affects its practicality for more com-

3.0

W Broyden

W/o Broyden

2.0

Chamfer

1.5

1.0

2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0

Time (hrs.)

(a)

(b)

plex scenes with numerous interacting objects. A potential solution could involve the integration of an occlusion map to better manage interactions among multiple entities.

Also, we further analyze the dependency of reconstruction quality on 3D pose and semantic map accuracy (Table 8). With predicted masks: For this purpose, we have used a combination of a state-of-the-art object detection network, YOLOv8 [48], for first detecting the objects under reconstruction and then SAM [49] for segmenting the respective objects

within the predicted bounding boxes given by YOLOv8. Results show that our method can generate similar reconstruction results as dataset-given masks, because, in our method, the reconstruction quality for the semantically separable geometries is not solely dependent on the quality of input semantic masks. We utilize information from both 2D semantic masks and 3D skeletons. While the rays are sampled in image space within 2D bounding boxes around each entity, we also perform an encoding for every 3D point on the sampled rays based on its distance from the 3D skeletons of individual elements under reconstruction. With predicted pose: We generate these 3D skeletons by using the state-of-the-art 3D pose estimation network [50]. As the results show, our method needs a good quality 3D skeleton to constrain the shape and motions of the individual elements. Even though the reconstruction quality is not affected, the inaccuracies come from the predicted pose (Figure 7). So, this is another limitation of our method, that it requires good quality 3D pose for correct reconstruction.

5 Conclusion

This paper introduces TFS-NeRF, a 3D semantic NeRF framework for dynamic scene reconstruction using sparse/single-view RGB videos. Utilizing INN, our approach significantly streamlines the training process, addressing the typically long convergence times associated with existing template-free methods. Our method effectively disentangles the motions of multiple entities, whether

(a)

(b)

(c)

(d)

rigid, non-rigid, or deformable, and optimizes per-entity skinning weights for accurate and semantically distinct 3D reconstructions. We conducted extensive experiments across various datasets, showcasing our approach's superior capability to manage complex interactions between multiple entities while ensuring high-quality reconstructions.

Broader Impact. Positive implications include advancements in digital media, robotics, and medical imaging, to name a few. Potential negative impacts involve privacy concerns and bias in model training, which can be mitigated by implementing strict data policies and ensuring diverse training datasets. Our proposal promises significant benefits for various fields, though it requires careful consideration of ethical and societal impacts.

Acknowledgments. This work has been partially funded by The Australian Research Council Discovery Project (ARC DP2020102427). We acknowledge the partial sponsorship of our research by the DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR) program, under award number FA8750-23-2-1016.

References

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems , 34:4805-4815, 2021.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems , 34:27171-27183, 2021.

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems , 35:25018-25032, 2022.

Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Objectcompositional neural implicit surfaces. In European Conference on Computer Vision , pages 197-213. Springer, 2022.

Qianyi Wu, Kaisiyuan Wang, Kejie Li, Jianmin Zheng, and Jianfei Cai. Objectsdf++: Improved objectcompositional neural implicit surfaces. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21764-21774, 2023.

Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV , 2021.

Sida Peng, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Animatable implicit neural representations for creating realistic avatars from videos. TPAMI , 2024.

Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR , 2021.

Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems , 34:14955-14966, 2021.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 , pages 851-866. 2023.

Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning to Dress 3D People in Generative Clothing. In Computer Vision and Pattern Recognition (CVPR) , June 2020.

Jia-Wei Liu, Yan-Pei Cao, Tianyuan Yang, Eric Zhongcong Xu, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Hosnerf: Dynamic human-object-scene neural radiance fields from a single video. arXiv preprint arXiv:2304.12281 , 2023.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10318-10327, 2021.

Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang. Neural surface reconstruction of dynamic scenes with monocular rgb-d camera. Advances in Neural Information Processing Systems , 35:967-981, 2022.

Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2863-2873, 2022.

Gengshan Yang, Chaoyang Wang, N Dinesh Reddy, and Deva Ramanan. Reconstructing animatable categories from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16995-17005, 2023.

Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers , pages 1-9, 2022.

Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5712-5721, 2021.

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5865-5874, 2021.

Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498-6508, 2021.

Chonghyuk Song, Gengshan Yang, Kangle Deng, Jun-Yan Zhu, and Deva Ramanan. Total-recon: Deformable scene reconstruction for embodied view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 17671-17682, 2023.

Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12959-12970, 2021.

Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16632-16642, 2023.

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR , 2023.

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG) , 40(6):1-12, 2021.

Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In European Conference on Computer Vision , pages 419-436. Springer, 2022.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations , 2016.

Zhenzhen Weng and Serena Yeung. Holistic 3d human and scene mesh estimation from single view images. arXiv preprint arXiv:2012.01591 , 2020.

Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, jun 2022.

Xi Wang, Gen Li, Yen-Ling Kuo, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Reconstructing action-conditioned human-object interactions using commonsense knowledge priors. In International Conference on 3D Vision (3DV) , 2022.

Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: Contact, human and object reconstruction from a single rgb image. In European Conference on Computer Vision , pages 125-145. Springer, 2022.

Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In European Conference on Computer Vision (ECCV) , 2020.

Hongwei Yi, Chun-Hao P. Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus Thies, and Michael J. Black. Human-aware object placement for visual environment reconstruction. In Computer Vision and Pattern Recognition (CVPR) , pages 3959-3970, June 2022.

Rishabh Dabral, Soshi Shimada, Arjun Jain, Christian Theobalt, and Vladislav Golyanik. Gravity-aware monocular 3d human-object reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12365-12374, 2021.

Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In European Conference on Computer Vision , pages 402-418. Springer, 2022.

Yuheng Jiang, Suyi Jiang, Guoxing Sun, Zhuo Su, Kaiwen Guo, Minye Wu, Jingyi Yu, and Lan Xu. Neuralhofusion: Neural volumetric rendering under human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6155-6165, 2022.

Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Vid2avatar: 3d avatar reconstruction from videos in the wild via self-supervised scene decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12858-12868, 2023.

Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition , pages 16210-16220, 2022.

Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11594-11604, 2021.

Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5521-5531, 2022.

Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation , 19(92):577-593, 1965.

Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In CVPR , 2020.

Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, and Siyu Tang. Resfields: Residual neural fields for spatiotemporal signals. In The Twelfth International Conference on Learning Representations , 2023.

Qi Fang, Qing Shuai, Junting Dong, Hujun Bao, and Xiaowei Zhou. Reconstructing 3d human pose by watching humans in the mirror. In CVPR , 2021.

Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6624-6634, 2022.

Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolo, january 2023. URL https://github. com/ultralytics/ultralytics .

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV , 2021.

A Appendix / supplemental material

A.1 Neural Network architectures:

Invertible Neural Network: It transforms the view space points to canonical space (Fig. 3), x ′ c = G INN v -> c ( x init c , J ) . We use realNVP [29] as the baseline of our Invertible Neural Network. This network consists of 2 Coupling layers [29], each with scaling and translation prediction modules. Each of these modules consists of 3 linear layers with dimensions 331 × 512 , 512 × 512 , and 512 . The input to the INN network is the deformed space points and it is conditioned on the skeleton pose. The input points are first transformed using a projection layer to dimension 256 which is concatenated with the skeletal pose (72) and passed as input to the scale and translation prediction networks. The predicted scale and translation transform the deformed space points to canonical space.

Skinning weight prediction network: The transformed canonical points are passed through the skinning weight prediction network w s = G x W c -> w ( x ′ c ) (Fig. 3). The skinning weight prediction network consists of 3 linear layers with dimensions 3 × 256 , 256 × 256 , 256 × 24 . The skinning weight prediction network takes the transformed canonical points as input, hence, the dimension is

3 . The output defines the weightage of each skeleton joint on a 3D point for its deformation from canonical space to deformed space. Hence, the dimension of the output layer is 256 × 24 . The output activation layer is defined as softmax as the sum of individual weight should be 1. Two similar architecture weight prediction networks are used for skinning weight prediction for individual entities.

SDF prediction network: Transformed canonical points are passed through the SDF prediction network Fig. 2 for geometry prediction, F j c -> Ω ( θ ) : R 3+3+ n b → R 1+256 . The SDF prediction network consists of 8 linear layers each with a hidden size of 256. A skip connection is added at layer 4. The dimensions of each layer are as follows 114 × 256 , 256 × 256 , 256 × 256 , 256 × 217 , 256 × 256 , 256 × 256 , 256 , 256 × 257 . The input canonical points are transformed by a frequency layer and mapped to dimension 39 , which is concatenated with canonical joints represented as 24 × 3 (each entity is represented by 24 skeleton joints). Moreover, as discussed in the main paper, each point is assigned a semantic label denoting which entity it belongs to i.e ., deformable, non-deformable object or background. The dimension of the semantic label is 3. Hence, the input dimension is 39 + 72 + 3 = 114 . The SDF prediction network generates an SDF value for each point (dimension 1) and a feature representation of dimension 256, resulting in a total output dimension of 257. We use separate networks for the SDF prediction of different entities. However, use similar architecture for both entities.

RGB rendering network: The RGB rendering network (Fig. 2) consists of 5 linear layers with dimensions of 270 × 256 , 256 , 256 , 256 × 256 , and 256 × 3 . The input to the rendering network is canonical points with dimension 3 , normals (calculated from predicted SDF) with dimension 3 , and per-frame skeleton pose (concatenated skeletal joints from both the entities with dimension of ( 72 ⊕ 72 ) and the SDF predicted feature vector of dimension 256 . A linear layer first transforms the skeleton pose to a lower dimension ( 8 ). Hence, the input dimension of the rendering network is 3 + 3 + 8 + 256 = 270 . The output of the network is the RGB value for each sampled point, hence, the dimension of the output is 3 . The output layer activation is defined as sigmoid.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, to correctly reflect the paper's contribution and scope, the main claims are made clear in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, the paper discusses the limitation of the proposed work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results, hence no proof or assumptions are provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, the paper disclose all the information required to reproduce the results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The paper does not open access the code during submission for review.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper discusses all details regarding the experimental setup.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer:

Justification: We do not produce error bars or statistical studies for our results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper mention about the computation resources used during training and testing the method.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in the paper conform with the NeuRIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes the paper discuss the potential positive and negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for approach optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not produce research in the above-mentioned areas, so there is no risk of misuse of released data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper discuss about all the technical details of the proposed methods.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper experiments with open source datasets, but does not capture any new data involving the human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper experiments with open source datasets, but does not capture any new data involving the human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposed a novel, time-efficient, template-free NeRF-based method for 3D dynamic scene reconstruction, focusing on capturing detailed explicit geometry for each entity in the scene. Extensive experiments demonstrate the efficiency of the proposed method.\n\nSTRENGTHS:\n1. The paper is well-organized, and the font size in the figures is appropriate, making it easy for readers to follow.\n2. The method section is straightforward, providing a detailed introduction to each part, including sufficient details on the network, training, and loss weights.\n\nWEAKNESSES:\n1. The author mentioned that one of the contributions is time efficiency. Based on the description in lines 183-202, it seems the proposed method reduces computation to accelerate training. However, Figure 6 only shows the convergence speed. It would be more informative if the author could report the time consumed per iteration.\n\n2. The information on how to obtain the semantic masks is missing. For example, in RoDynRF [1], the author uses optical flow and Mask R-CNN to obtain the segmentation masks.\n\n3. The work focuses more on dynamic scenes with human subjects, but the title gives the impression that the method is designed for general dynamic scenes.\n\n4. From Figures 2 and 3, it appears that the proposed method requires a 3D skeleton as part of the inputs. Does this mean that the proposed method does not work for scenes without humans or datasets lacking 3D skeleton information?\n\n5. In the bottom part of Table 3, under the Dist. Acc column, HyperNeRF is the second-best method.\n\n[1] Liu, Yu-Lun, et al. \"Robust dynamic radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose TFS-NeRF to solve the semantic reconstruction of dynamic scenes.\n\nSTRENGTHS:\nExperimental results on multiple entity and deformable entity reconstructions are good.\n\nWEAKNESSES:\nIt is very hard to follow the storyline of the introduction.\n\nUnclear contributions. In Lines 81 and 91, the paper claims it could reconstruct multiple dynamic entities with multiple complex interactions, but in the experiment parts, the presented results are mostly from one person/animal with a single object, it is hard to judge whether the proposed model could process multiple dynamic entities with complex interactions. Also, the authors claim the proposed model could address the occlusion challenge in Line 81, but there is no experiments about it. Besides, the paper claims that it could reconstruct semantics of dynamic scenes by using semantic masks. However, such semantic reconstruction has been solved by previous works like Semantic Flow [1], it is hard to judge the contribution of this paper in this situation.\n\nMissing important method details. This paper proposes a semantic-aware ray sampling strategy and illustrates the points are separated in to deformable object set and non-deformable object set, but does not present any details about the separation process. After reading the paper, the reviewer guesses it separates the points on the rays based on the semantic labels of the pixels traveling through the rays. However, it may not be correct since different points in the same camera ray may belong to different set (like a ray travels through a dynamic person and a static wall behind the person). It is very hard to understand the entire pipeline of semantic reconstruction without such details.\n\nWriting should be significantly improved. There are many informal and unclear expressions in this paper:\n\n- Line 405: Takeaways -> Conclusion\n- Figure 2: Overview of the system-\n- Line 229: unclear expression of R^{1+256}\n- Line 221: why \"it helps capture better articulation or deformation and surface reconstruction under motion?\"\n- Tables 5 and 6: what is the mearning of \"->\"\n- Line 216: Given,\n- Lines 98, 112, 136: the usage of : should be the same\n\nReference:\n[1] Tian, Fengrui, Yueqi Duan, Angtian Wang, Jianfei Guo, and Shaoyi Du, Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos, ICLR 2024.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nUnclear expressions and limited contributions", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

comments: SUMMARY:
The paper addresses the problem of reconstructing dynamic environments for arbitrary rigid, non-rigid, or deformable entities. The authors propose a template-free 3D semantic NeRF for dynamic scenes, which employs an Invertible Neural Network (INN) for LBS prediction, and optimizing per-entity skinning weights based on semantic masks. The experimental results show high-quality reconstructions of complex interactions with improved training efficiency compared to existing methods.

STRENGTHS:
- The proposed method is able to reconstruct arbitrary non-rigid objects by utilizing TAVA framework.
- Disentanglement of objects improves the reconstruction quality by predicting LBS for each entity.
- The proposed framework outperforms existing template-free NeRF methods on various datasets containing different non-rigid and rigid objects.
- The paper is clearly written and easy to follow.

WEAKNESSES:
- The paper emphasizes semantic-aware ray sampling as a key contribution but lacks details on how the semantic masks are generated.
- The framework heavily relies on existing methods (TAVA, INN), limiting the novelty and distinctiveness of its contribution.

QUESTIONS:
- How is the semantic masks calculated? How is objects classified as rigid or non-rigid?
- During the training, are the same number of rays sampled for rigid and non-rigid objects?

LIMITATIONS:
- The method is not scalable for the scenes more than two entities.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TFS-NeRF, a semantic-NeRF framework leveraging no prior templates of dynamic scenes for 3D reconstruction. Guided by INN-driven LBS prediction and semantic-aware ray sampling, TFS-NeRF separately consider deformable and non-deformable parts during geometric learning but composite them to learn apperance to benefit from self-supervised RGB reconstruction loss. Experiments on human-object and animal cenric dynamic videos demosntrate the advantages of the proposed method against chosen baselibes.\n\nSTRENGTHS:\n-The overall paper is well motivated and easy to follow. The qualitative reuslts and supplement videos demonstrate the promising reconstruction performance of dynamic scenes with multiple entities.\n\n-The experiment are extensively evalauted on several public benchmarks and the ablation studies highlight the unqiue contribution of several design choices.\n\nWEAKNESSES:\nMy main concerns comes from the lack of clarification on several key components:\n\n(1)As a semantic-nerf framework, how abou the influence of quality of input masks? As to labels from an imperfect 2D predictor, what is the impact on final results, since the label quality may affect the sampling quality for deformable and non-deformable rays.\n(2)Similarly, what is the robustness to pose accuracy as poses are involved with both ray generation and the pose loss?\n\nDiscussion on the above two points could better strenthen the practicalibility of TFS-NeRF.\n\n(3)As a rapidly growing area, the chosen baselines do not include latest methods such as HexPlan,Tensor4D or GS-based ones to highlight the advantages of using INNs, compositional rendering as well as reconstruction quality.\n\n(4)As to LBS predicton, what is the unique advantages of applying INN over other types of NN modules (e.g, lossy CNN auto-encoders)? What is the efficiency or performance boosts compared to standard CNNs?\n\n(5)How to get the initial values of B_i to compute the initialization value of x_c for arbitrary objects?\n\n(6)As the paper is claimed to template free and able to deal with generic scenes, it would be more exciting and convincing to see other types of interactions? For example, a common daily objects (e.g. a door or a cabinet, opening and closing) to really show the advantages of prior-free deformations of general scenes.\n\n(7)It would be good to aldo add the appearance quality or view synthesis performanc, though I understand the focus of TFS-NeRF is on 3D reconstruction.\n\n(8)Formatting needs to be improved. For example, it is wierd that in Sec3.B, authors put much related work descriptions. There are also improper indents like Line 282 and Line284,etc.\n\nQUESTIONS:\nPlease see the weakness section above on the clarification on the robustness to masks and pose qualities, as well as more in-depth discussions towards recent baselines and more general scenes.\n\nLIMITATIONS:\nLimitations are properly mentioned in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}