TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene
Authors: Sandika Biswas
Qianyi Wu
Biplab Banerjee
Hamid Rezatofighi
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D reconstruction
Template-free NeRF
Semantic reconstruction
Multiple entity interactions
Primary area: machine_vision
TL;DR: Dyanmic template-free NeRF for 3D semantic reconstruction of multiple entities interactions

Abstract: Despite advancements in Neural Implicit models for 3D surface reconstruction, handling dynamic environments with interactions between arbitrary rigid, non-rigid, or deformable entities remains challenging. The generic reconstruction methods adaptable to such dynamic scenes often require additional inputs like depth or optical flow or rely on pre-trained image features for reasonable outcomes. These methods typically use latent codes to capture frame-by-frame deformations. Another set of dynamic scene reconstruction methods, are entity-specific, mostly focusing on humans, and relies on template models. In contrast, some template-free methods bypass these requirements and adopt traditional LBS (Linear Blend Skinning) weights for a detailed representation of deformable object motions,
although they involve complex optimizations leading to lengthy training times. To this end, as a remedy, this paper introduces TFS-NeRF, a template-free 3D semantic NeRF for dynamic scenes captured from sparse or single-view RGB videos, featuring interactions among two entities and more time-efficient than other LBS-based approaches. Our framework uses an Invertible Neural Network (INN) for LBS prediction, simplifying the training process. By disentangling the motions of interacting entities and optimizing per-entity skinning weights, our method efficiently generates accurate, semantically separable geometries. Extensive experiments demonstrate that our approach produces high-quality reconstructions of both deformable and non-deformable objects in complex interactions, with improved
training efficiency compared to existing methods. The code and models will be available on our github page.

TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene
Sandika Biswas Qianyi Wu', Biplab Banerjee", and Hamid Rezatofighi'
Faculty of IT, Monash University
Indian Institute of Technology (IIT), Bombay
Abstract
Despite advancements in Neural Implicit models for 3D surface reconstruction, handling dynamic environments with interactions between arbitrary rigid, non- rigid, or deformable entities remains challenging. The generic reconstruction methods adaptable to such dynamic scenes often require additional inputs like depth or optical flow or rely on pre-trained image features for reasonable outcomes. These methods typically use latent codes to capture frame-by-frame deformations. Another set of dynamic scene reconstruction methods, are entity-specific, mostly focusing humans, and relies template models. In contrast. some template- free methods bypass these requirements and adopt traditional LBS (Linear Blend Skinning) weights for detailed representation of deformable object motions, although they involve complex optimizations leading to lengthy training times. To this end, remedy, this paper introduces TFS-NeRF, template-free 3D semantic NeRF for dynamic scenes captured from sparse or single-view RGB videos, featuring interactions among two entities and more time-efficient than other LBS-based approaches. Our framework uses an Invertible Neural Network (INN) for LBS prediction, simplifying the training process. By disentangling the motions of interacting entities and optimizing per-entity skinning weights, our method efficiently generates accurate, semantically separable geometries. Extensive experi- ments demonstrate that our approach produces high-quality reconstructions of both deformable and non-deformable objects complex interactions, with improved training efficiency compared to existing methods. The code and models will be available on our github page
Introduction
With the rapid advancements in deep learning, the field of 3D geometry reconstruction of static and dynamic scenes and objects has experienced significant transformation, largely due to its crucial role in diverse applications such Augmented Reality (AR), Virtual Reality (VR), robotics, autonomous navigation systems, and human-robot interactions (HRI). While primarily focused on novel view synthesis, Neural Implicit models (NeRFs) 1 have recently made remarkable strides in 3D surface reconstruction due to their ability to learn detailed geometry without needing prior knowledge about the shape of the scene elements direct 3D supervisions 2 3 4 5).. Despite these advancements, challenges remain in developing models that effectively generalize across varied real-world dynamic environments, particularly those involving arbitrary rigid, non-rigid, or deformable entities (such as humans or animals) engaged in complex interactions. Moreover, achieving semantic reconstruction that accurately captures the geometry and positioning of each semantic scene element independently is essential for enhancing functionalities in applications like AR, VR, and HRI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In NeRF-based dynamic scene reconstruction, the focus has predominantly been on human reconstruction
I0]. utilizing template models such as SMPL [I], and CAPE [[2]
However, these methods struggle with generalizability to arbitrary deformable entities and
primarily concentrate on single-entity reconstructions, neglecting interactions among multiple entities, such as interactions of humans with scene objects. Notably, HOSNeRF [3] by Liu et al. addresses human-object interactions but remains dependent on the SMPL model. Concurrently, some dynamic NeRFs
14
15
16
17
18
20
21 22
23
26 aim beyond human and focus on either rendering or geometry reconstruction of generic scene/object. The surface reconstruc- tion methods mostly require addi- tional data, such as depth or opti- cal flow, to optimize the underlying 3D. For instance, [16] 17 focus on building shape-prior-free 3D mod- els for deformable entities. How- ever, besides requiring a large num- ber of casual videos with diverse view coverage of the subject, these approaches heavily depend on high- quality optical flow for supervision, which makes their reconstruction quality subject to the accuracy of the pre-trained models for generat- ing such data. Moreover, generally, the dynamic NeRF methods use la- tent codes to learn per-frame defor- mations [15 (2)) which may not be sufficient for capturing accurate ar- ticulation of arbitrary deformable ob-
24
25
A
A
B
D
Figure 1: Existing dynamic-NeRF models struggle to generate plausible 3D reconstructions for generic dynamic scenes fcaturing humans and objects engaged in complex interactions, In this work, introduce Neural Radiance Field model designed for 3D recon- struction such generic scenes, captured using a sparse/single- video, capable of producing plausible geometry for each semantic element within the scene. In this figure, A: Input RGB, B: predicted normal map, C: predicted semantic reconstruction, and D: predicted skinning weight.
jects. In a concurrent work, Li et al. [28] bypass the need for such additional information and develop accurate represen- tation of body deformation, the forward LBS [1] (i.e., mapping of canonical points to posed or view space), which helps them achieve impressive shape reconstruction results for different deformable entities. However, their method suffers from a lengthy training convergence time. Furthermore, none of these template-free approaches have focused interactions between more than one entity nor achieved semantic reconstruction of scenes, highlighting a gap in the current methodologies.
template-free model from only sparse view RGB videos. They also learn
Our contributions: In this paper, we aim develop time-efficient 3D semantic NeRF for dynamic scenes captured from sparse view RGB videos involving interactions between two rigid, non-rigid, deformable objects. Built upon 28 propose novel framework learn the forward LBS by utilizing INN (29] to bypass the computationally demanding root-finding approach used in (28). This adjustment boosts the efficiency of our training process, which is shown with experiment in the results section. Additionally, our proposed framework can produce semantically separable reconstructions for the scene entities. The challenge of building template-free 3D models for two dynamic entities engaged in complex interactions is amplified by occlusions and their diverse shapes or deformations. To tackle these challenges, we propose a strategy to disentangle the motions of distinct entities within the scene. Specifically, we first perform semantic-awan ray sampling and learn the independent transformation of each entity from the deformed space to the canonical space to optimize per-entity skinning weight. This process allows us to accurately generate the individual Signed Distance Fields (SDF) from the disentangled canonical points, enhancing our model's ability to handle complex dynamic environments under interactions effectively. The efficacy of our proposed method is demonstrated through comprehensive experiments conducted on several datasets featuring human-object, hand-object interactions, and animal movements. Results consistently show superior performance compared to the rest-performing state-of-the-art methods. Our noteworthy technical contributions are, therefore:
We introduce a time-efficient template-fin NeRF-based 3D reconstruction method for dynamic

---

scenes from sparse multi-view/single videos featuring two interacting entities.
Our approach extends to the semantic reconstruction of dynamic scenes, emphasizing the detailed capture of each entity's ' explicit geometry within the scene.
The efficacy of our proposed method demonstrated through comprehensive experiments conducted on diverse datasets featuring interaction between rigid, and non-rigid entities.
2 Related Works
Human-object reconstruction:
Several model-based approaches 30 31 32
35
36 have
explored 3D reconstruction of humans and objects under interactions. However, a common limitation is their reliance on parametric models (e.g., SMPI.) for human reconstruction. While SMPI. provides a robust base, it limits flexibility in reconstructing diverse deformable entities and cannot generalize beyond humans. Also, it does not capture finer details such as hair dynamics and clothing deforma- tions. In contrast, NeRF-based reconstruction offers a promising alternative for capturing detailed geometric information but mainly focuses human reconstruction under a dynamic environment. Yet, only a few NeRF- based methods [13
37 38 have addressed the reconstruction involving multi- ple objects. For example, Jiang et al. [37 consider modeling the background, along with dynamic human by designing two separate NeRFs i.e. human NeRF and background NeRF. a recent work, HOSNeRF, Liu ef al. [13] introduce NeRF-based approach for free- viewpontt rendering of dynamic scenes featuring human-object interactions. Even using the NeRF-based approach, these methods still rely the SMPL model for initialization. Moreover, unlike our method, most of these methods focus on novel view or pose synthesis and do not emphasize detailed geometry reconstruction.
Generic Scene Reconstruction:
Methods
Template- free
No pre- trained features
Reconstructs
Some NeRe-based methodologies
15 27 40
39
41 25 26] offer
generic scene reconstruction under dynamic environment without relying on any parametric template models, making them favorable for extending to any deformable object reconstruction. Pumarola et al. [14) extend the traditional NeRF framework by integrating time as an additional input, which facilitates the mapping of each frame's 's deformed scene to canonical space. Building on this, T-NeRF 42] utilizes time- varying latent codes to condition the NeRF, improving training speed and rendering quality. Cai er al. (15) propose using an INN to learn the mapping between deformed and canonical space and show impressive surface reconstruction results optimized from RGB-D videos. HyperNeKF [27] incorporates an additional Multilayer Perceptron to learn the frame-specific topology variations via ambient codes, capturing scene deformations more effectively. However, these methods mainly rely on latent codes to capture the relation between frames at different timestamps or topological variations within frame. A recent approach Tensor4D [25] represents dynamic scenes spatiotemporal tensor but does not account for the relative motions between scene elements. In contrast. approach considers learning the structural relations between different parts of the scene or the object under reconstruction e.g., body parts of humans in case of human reconstruction, etc.
Vid2Avatar
30
X
single cntity single entity
AnimstableNeRtF
SDF-PDF
X
X
single cutity
HumanNeRF
40
single entity
HOSNeRF
13
X
multiple entities
NDR 15
single entity
HyperNeRF 27
single couity
D-NeRF
single entity
BANMO [6
X
single cotity
RAC 17
single entity
TAVA
single entity
Ours
multiple entities with semuntic
Table 1: Our approach vs. existing dynamic NeRFs
4D
Reconstruction with predicted LBS: Several methods have recently emerged that aim to develop generic NeRFs focusing on arbitrary deformable object reconstruction [16] 17 28). These methods utilize more specific topological representation, i.e., LBS, that helps understand how different parts of the deformable body are connected and how they deform from the canonical pose under motion. Specifically, given the forward LBS weight w(xe) for canonical point X. on the surface, the corresponding deformed point x, (in the viewing space, i.e., per-frame observations) computed using a weighted combination of bone transformation matrices B
[I] defined as,
78b
X, -LB((W(x)).
B, B,
w(xx).Boxx
(1)
6=1
3

---

Inverse
Deformable SDF
Warping
y c--
92%: Normal
fume
Volume Rendering
F RGB
VOTh: Normal
nd
F Rid
X
Warping
DE Non-deformuble SDF
9tmmc
D
Figure 2: Overview of the system. A: To produce semantically separable reconstruction of each element, first, we perform semantic-aware ray sampling. Given 2D semantic segmentation mask, we shoot two sets of rays and sample two sets of 3D points differentiating the deformable and non-deformable entities of the scene, [a]] {a N under interactions. B: Next, each set of points is transformed from the deformed/view space (imput frame) to its respective canonical space by inverse warping enabled by the learned forward LBS (Details are presented in Fig. 3 C: Then the individual geometry is predicted at the canonical space in the form of canonical SDFs by two independent SDF prediction networks F (0) for the deformable and non-deformable entities denoted as E (d, nd). D: Finally, the output SDFs are used to predict composite scene rendering. Both these branches are optimized jointly using the RGB reconstruction loss.
where ny represents the total number of bones. Unlike human reconstruction NeRFs g g
these methods learn this skinning weight function w(Xe). that allows for the flexible adaptation of deformation field for mapping between canonical and deformed space and represents the skinning weight using a set of Gaussians defined around the bones. Whereas L. el al. (28) and Chen al. [4]] learn the skinning weights and formulate the mapping from deformed to canonical spaces as an iterative root-finding woot-fnning problem. They solve this problem using Broyden'ss 'ss method, which involves matrix- vector multiplications and computationally expensive matrix inversions at each iteration, leading to lengthy training convergence time. Additionally, these methods consider reconstructing only a single entity within the scene. In contrast, our method addresses the reconstruction of two moving objects under complex interactions, which is more challenging due to the highly unconstrained nature of the problem.
these models to any deformable objects. Yang el al. [66 learns a forward-backwand
3 Methodology
Overview. Given sparse-/single-view videos of deformable objects, such as hands, humans, or animals, interacting with non-rigid/rigid objects like balls or boxes, our goal is to accurately learn the semantically separable geometry of each entity under interactions. Fig. 2 provides an illustrative overview of our methodology. highlighting the following key components and steps involved in the process. A) Semantic-aware ray casting and sampling to distinguish individual elements within the scene (Fig. A B) INN to learn the forward skinning and deformation of each element, enhancing the efficiency of the training process (Fig. 2B), C) SDF module to independently learn the geometry of each element (Fig. 2C) and D) RGB Renderer module to generate RGB values from individual SDF volumes for the final rendering (Fig. (P).
A. Semantic-aware ray sampling, To reconstruct semantically separable geometry, it is essential that each 3D point in the reconstructed surface tagged with a semantic label to denote its object affiliation. Previous efforts (5) propose to model a compositional scene within shared network structure, which is challenging in the dynamic scenario for the following two reasons. First, it is hard to supervise the SDF of each entity from different frames by 2D semantic segmentation due to its dynamic nature. Second, the interaction between humans and objects in different frames is complex, considering the occlusion between each other. These issues provoke us to design network structure with natural disentanglement for the compositional modeling.
To address these challenges, our network is designed with distinct modules for SDF prediction for each entity, maintaining semantic continuity from the input space to facilitate the prediction of semantically separable 3D geometry and joining them with the rendering module for composite rendering. Specifically, leveraging the 2D semantic mask of the input image, we sample two sets of rays surrounding the interacting objects. Along these rays, we sample 3D points to generate

---

two distinct sets: one representing the deformable object, { and the other representing the non-deformable object, { Ji=1- However, only 2D semantic-aware ray sampling is not sufficient for accurate disentanglement of the individual entities, as the rays can intersect both entities in their path through the 3D space due to the occlusion of one object by the other. Hence, we also perform an encoding of the 3D points based on their distance from the 3D skeletons of individual entities. Please refer to Section 3C for more details about the semantic encoding of the 3D points. These points are then processed through separate networks tailored to each object type to get separate geometry and finally merged at the rendering module. This approach enhances our ability to effectively separate and reconstruct individual elements, even scenarios of strong occlusions. This meticulous sampling strategy guarantees that supervision from individual entity's 's observation only influences each SDF prediction module.
B. View space to canonical space deformation. In dynamic NeRF, common approach is mapping each frame to fixed canonical frame to learn the correspondence between frames. For deformable objects, like humans, this is generally achieved by using pre-defined LBS weight from template
LBS a technique used to deform the surface points on a human mesh at the canonical pose to each frame's pose based on the positions and orientations of an underlying skeleton (Eqn. [T. A few methods, e.g., TAVA (2] and SNARF 41 extend this approach beyond humans and learn the forward LBS for arbitrary deformable entities without using any template model. However, for learning the forward LBS, defined in canonical space, the main challenge lies in finding the correct correspondence in the canonical space for a given deformed point because of the implicitly defined correspondences without an analytical inverse form of the Eqn. [ Hence, they formulate the problem of learning this correspondence as root finding problem and solve for Xc s.f., (BBS(((e)). B, xe) Xv O. They solve it numerically using iterative multiplications and computationally expensive
models like SMPL etc. 7
39 40
Broyden's 's method [43] which involves matrix matrix inversions at each iteration and must be optimized for multiple iterations for each iteration of NeRF optimization. Hence, this makes the training process time-consuming. To overcome this challenge, we replace this root finding problem and instead use an INN (29] for transforming the view space points canonical space and learn the skinning weight w (xe) simultaneously. Conditioned by each frame's ' pose, INN can correctly learn the bijective mapping between the deformed and canonical space through consistency loss and bypassing the iterative optimization process, resulting in time-efficient training. Fig. 3 shows the workflow for the view to canonical space conversion framework.
First, function GINN >> is used to transform the points from view space to canonical space using an INN defined as (29). To en- hance the convergence and provide a better initialization for the INN network, we first transform the deformed space points Xv to give an approximation of the canonical space points. Under the intuition that the sampled points Xv near the posed skeleton will remain close the canonical skeleton at canonical space, Xv are transformed to x- = (E w' .B,) .xy, where, w' represents one- hot vectors defining the nearest joint of the posed 3D skeleton Jp E R from the view space points X.. An ablation study is presented in Tab. to show the effectiveness of this initialization. The INN, GINN is conditioned on per frame 3D skeleton Jp. Then a skinning weight prediction network GW -x- is used to predict the skinning weight w, at canonical space from the predicted canonical points x
Natwork
MILP
W
X
Inverse Warping
Figure Overview of the transformation from view space to canonical space.
X GINN init -v-ve (x
J)
Wa 9xc > (xx))
(2)
Finally, the canonical points are calculated as, x. = EEA 11g 1i= W Bl. Xy, that are given as input to the SDF prediction network. This helps to constrain the skinning weight prediction network. As skinning weight defines the weightage of underlying skeleton joints for the deformation of each vertex on the mesh surface; it helps capture better articulation or deformation and surface reconstruction under motion. Moreover, it implicitly learns the spatial relationship between the surface points, resulting in smoother reconstruction compared to the approaches where no skinning weight representation is learned. This is evident in Fig 5 given a comparison with such methods.
5

---

C. Learning the SDF volume of individual elements. Given our aim to generate semantically separable geometries of the scene entities, we use two independent SDF networks defined as F2->0(0p) for deformable and non-deformable objects j E {d, nd]. The individual set of trans- formed canonical points Xe are passed through these geometry prediction networks, defined as Fe-sa R + II+256 (SDF and global feature representation of dimension 256)
produces surface reconstruction at canonical space, 02%.
The SDF prediction network conditioned on the canonical skeletons po E Only semantic- aware (based
2D semantic map) ray sampling is not sufficient to differentiate the points in 3D. For this purpose, to provide the SDF prediction network information about which object the point belongs to, we assign semantic label to each 3D point. This is defined
weightage, w = exp(-dist* /a²) based on the distance of the point from the nearest 3D joints in the per entity canonical skeletons Jpo- The points far from the canonical skeleton of either of the entities are assigned the background, with weight defined why 1.0 clamp(E, w 1.0). This semantic label with dimensionality 3 is concatenated with canonical point and pose input (3 + 3 + r1s)) and passed through the SDF prediction network that predicts SDF and a global feature representation.
D. Compositing for final rendering, We use single RGB prediction network to predict the final rendering. For this purpose, the predicted canonical points (3), normals calculated from SDFs (3) 39.. geometric features (256) and posed skeletons J po (1a) for individual elements are concatenated before sending through unified RGB prediction network Fn->rgh Following Guo el al. (39 we predict the texture in canonical space and condition the texture generation network with normal calculated from SDFs to consider the deformation from view space points to canonical points. This
defined as Fe->rah 13+33+n +56 + R
Training: All the modules defined above are trained jointly over all the frames of the given video. The training losses used for the global optimization are defined as follows:
Reconstruction loss: For optimizing the NeRF, reconstruction loss is defined between the rendered
RGB C(r) and the RGB Crr) of input pixel along the ray r, Lrgh Eren l|((r) - C(r)
Due to the lack of direct supervision of the skinning weight prediction network or INN, several incorrect combinations of canonical points and skinning weights can satisfy the forward LBS Eq. Hence, the following supervisions are used on the skeleton space to constrain these two networks. Pose loss: To ensure that the INN network learns a correct mapping between the view space and the canonical space, loss is defined two sets of points (0 Xx,)) sampled around the bones of canonical (J (Jpo) and deformed skeletons (Jp) respectively. The following loss is applied to ensure that the INN correctly transforms the point set X.,, to X_
Cpose
[Xp0 (X,,) PEP
where P is the total number of points sampled around the bones from each skeleton. Skinning weight loss: To constrain the skinning weight prediction network, a loss is applied to ensure that the predicted skinning weight for the canonical joints is one-hot vector (1 for the respective joint and 0 for rests), w.
Gw = ( w 1
(4)
Cyylle loss: Conventional cycle consistency loss is used for optimizing the INN,
LINN =
Jno), Jp) xv11
(5)
where H is the transformation learned by the INN between view and canonical space. The inverse function (xx, transforms the view space point to canonical points, and the forward function H(xe, Jn) transforms back the canonical points to deformed space conditioned by the posed skeleton. Connsittnncy loss: To ensure that the INN transformed point x and the final skinning weight conditioned canonical points Xe (Fig. 3 are close to each other we also minimize the L2 distance between these two sets of points.
Loonsie = || -and-xxlll 2el12
In shape loss: Following Guo et al. [39, to accelerate the learning process, a loss (Cshape) is defined around a point cloud initialization for the individual entities. The point cloud is defined around the canonical skeleton of individual elements. This loss ensures that the transformed points Xe that fall within this point cloud have the sum of weights predicted from NeRF densities =

---

391. The full loss, minimized over each video frame, is defined as follows, where Asket Xw, AINN, Coonsiss Ashape are empirically set to 2,10,1,1,0.03 respectively.
L = Lrgb + SaketCaket + Aw Lw + AINNCINN - LconsisCoonst + Ashape Sahape
(7)
Evaluation: As we learn the object geometry canonical space, inference time,
directly
sample points around the canonical skeleton and pass through the SDF prediction network J (0) (Fig. ( to predict the canonical SDF. Then, for the canonical mesh vertices, we can predict the skinning weights using the skinning weight prediction network GW - (Fig. and generate the final posed mesh (viewing/deformed space) by following Eqn. [
2 is trained end-to-end, with a learning rate of 5.0c - 4. We use ADAM optimizer for the SGD optimization and Py PyTorch library for the implementation of our method. All training and inference have been performed in NVIDIA RTX 4090 GPU. Details about the network architecture are presented in the Appendix.
Implementation Details: The overall network proposed in Fig
4 Experiments
In this section, first demonstrate the effectiveness of our method for precise 3D reconstructions of deformable and non-deformable objects, as well as their interactions. Next, we present qualitative and quantitative comparisons of our approach with relevant state-of-the-art methods. Additionally, we conduct a comprehensive ablation study to analyze the impact of different network design choices and loss formulations on our model's performance. Finally, we discuss the efficiency of our method in terms of training convergence time compared to existing approaches in the literature. Datasets. To evaluate our reconstruction under multiple entity interactions, we select the BEHAVE [31] with human-objes interactions and HO3D- V3 44 with hand-object interactions. We tested sequences featuring large motions of humans and objects, training with 45-50 images per camera view to optimize 3D geometry. Following (28). all methods utilize dataset-prowe patasee-provided camera poses, body poses, and masks for training. We also evaluate our method for reconstructing single deformable entities (only human/animal) and use a similar setup as proposed in TAVA (28) For this purpose, we test performance on two datasets: the ZJU-MoCap dataset [9] for human reconstruction and a synthetic dataset for animal reconstruction from (28).
Baseline. For human-object reconstruction, we choose methods that focus on generic scene recon- struction without using any template 3D models for a fair comparison of our template-free model. Specifically, we benchmark our approach against state-of-the-art generic scene reconstruction meth- ods such as Tensor4D 225 NDR [15]. HyperNeRF 277. and D-NeRF [1]. Additionally, we consider the recent approach ResFields, which models large and complex temporal motions by introducing temporal residual layers into the general NeRF architectures, showing significant improvements over the baselines (45) We train NDR 15. HyperNeRR (27), and D-NeRF [14] with and without ResField layers RGB videos. We haven't 't trained our method with ResField. For human reconstruction, we compare with methods that learn the skinning weight field 28 27). TAVA 28 learns the skinning weight from scratch without using any template model, whereas AnimatableNeRR [9] initializes the skinning weight from SMPL and learns a residual weight to optimize the final skinning weight field.
Evaluation metrics. For quantitative evaluation, following previous 3D reconstruction NeRFs (5) b] we measure distance metrics (after registration between predicted and ground -oound-truth meshes) such as Average Distance Accuracy (Dist. Acc.), Completeness, and Chamfer Distance (CD), along with Precision, Recall, and F-score (defined with a threshold of 5cm). Following previous methods (39. we use SMPL meshes provided in the ZJU-MoCan and BEHAVE datasets as ground-truth to evaluate human reconstruction quality to obtain the quantitative results. Similarly, we utilize the ground-truth object meshes within the BEHAVE dataset for object reconstruction evaluation.
Results and discussions. Recon- struction of scenes with two enti- ties under interactions: Our evalu- ation methodology for numan-objee reconstruction focuses on two crit- ical dimensions: 1) Holistic Scene Reconstruction: We assess the entire scene's reconstructed mesh quality (human-object) against the ground truth, providing a comprehensive measure of the scene's accuracy (Tab. 3) 2) Semantic Reconstruc-
Methood
Metric
Trained with ResFied
Dist Acc.
Camp.
Prec.
Recal:
E-scrr
Chamfer:
(cws)
(cm))
(%)
(7)
(3)
(1x))
Temnest BS
4.152
2.441
69:413
9L.642
78,993
1.297
NUR [t
4.203
3.527
73.045
75.446
75.999
1.865
3.362
75/661
3342
D.NeNE
NDR [E]
3,591
3.195
78.564
82,672
96.355
1,399
HeperNest
33570
1.313
76 (20
81.578
71,019
3,566
B.Ne
3.927
3.354
74,315
3.466
Oues
2.72
3.442
10.100
933833
91,315
2441
Table 3: Scene reconstruction results on BEHAVE [3] dataset.
7

---

Metric
Method
Dist. Acc.
Comp-
Prec:
Recal.
F-score
Chamfer
ResField 15
[em)
[cm))
(86
(%6
(86
[cmn)
Tensor4D 25
x
4.409
2,419
69.402
91.680
79,000
4.414
NDR 11
4.865
3.546
69.653
76,991
72.728
4.205
HyperNeRF
D-NeRF 14
6.315
6.132
53.013
52.745
52.071
6.324
NDR [1
4,681
3,186
72,914
81,025
76.553
3,931
HyperNeRF
4,265
3.394
73,864
79,642
76.331
3.831
D-NoRR 14
4,729
3,403
71,211
79.244
74.706
1.761
1,863
97.225
93.624
95.343
1,812
TensordD D
25
4,390
2.523
56,953
91.683
70.260
3,956
[13
75.675
HyperNeRIF
221
3.647
3.508
78.171
76.892
77.144
3,586
D-NeRF
1
4,675
5.529
64.88
54.095
57,748
5,102
NDR [1
3.442
3.531
81.114
76.612
78.641
3.485
HyperNeRF
3.451
80.551
50.720
80.174
3.379
D-NeRF
14
3.565
3.362
79.379
79.155
78.875
3,464
Ours
X
3,571
2,121
N2.762
92.410
86.991
2,741
Table 2: Semantic reconstruction results on BEHAVE
31
dataset.
The upper and lower tables represent
quantitative human and object reconstruction evaluation.
tion: We asses the reconstruction quality of individual scene elements by comparing the accuracy of reconstructed humans and objects their respective ground truth meshes (Tab. 2 Our method excels in holistic scene reconstruction across all metrics (Tab. (3) and delivers superior results in semantic reconstruction, especially for humans (Tab. 2 upper rows) and most objects (Tab. 2 lower rows). Competing methods, focusing on the prior-free linkage between observation and canonical space, often neglect the topological relationships essential for dynamic scenes. While they are effective for rigid entities, they underperform in reconstructing high-quality, deformable entities. Our approach leverages learning skinning weights to capture the intricate relationships between body parts, enhancing reconstructions for both deformable and non-deformable objects (Fig. 5
Reconstruction of scenes with hand-object reconstruction: We evaluate our method also on the HO3D-V3 dataset (44) and present comparative results in Tab. 4 For this purpose, we choose two baseline methods, i.e., NDR and HyperNeRF trained with Res- Field, that best perform on the BEHAVE dataset. Tab. 4 compares semantic reconstruc- tion quality. We use the ground-truth mesh for hand from the HO3D-V3 dataset and the ground-truth mesh for objects from YCB- Video 3D models for calculating the metrics.
Our method achieves better reconstruction, showing superior performance on most metrics. Also, we evaluate our method for single-object reconstruc- tion, including arbitrary deformable entities,
recenstruction
Object reconstrucion
Methed
Metric
Trained with with RedFadd
Diet. Are.
Foecor:
Chamfer
Did. Aec.
Focore
Chamder
(ero)
(sm)
[cm]
(cw)
NDR BS
94.051
1.156
1.170
Human reconstruction:
HypeNeRI
1.159
1,42
Ours
1.33
95.396
1.294
0.530
99,900
0,463
We compare our human surface reconstruction results with TAVA (28 and AnimatableNeRF 7 on the ZJU-MoCap dataset 9 (Tab. upper table). TAVA employs template-fre free approach, while AnimatableNeRF uses the SMPL body model. Our method surpasses both TAVA and AnimatableNeRF in performance. The qualitative comparison, shown in
Table Reconstruction results on HO3D-V3 dataset
44
Fig. 4 highlights the superior surface reconstruction of method the ZJU-MoCap dataset. SDF modeling contributes smoother surface reconstructions compared to models like 28 7
Dist. Acc.com)
Comp. (em)
Prec. ?
Recal.
F-score
Chamfer
(%6
(50)
(%)
(cm)
TAVA [X
2.99
2.14
90.40
95.66
92.95
2.47
AnimatileNeRFF
3,63
2.39
81.57
93.32
88.30
2,81
Ours
2,01
92.15
95.99
93.97
2.24
TAVA
0.92
0.53
99.65
100.00
99.83
0.73
Figure Qualitative comparison on ZJU-
0.61
99.99
100.00
99.99
0.71
Mocap dataset (9]
Tuble 5: Reconstnuction ZJU-Mocpp (upper) ( 46) and symthetic unimal dataser 28 (lower).
Reconstruction of other deformable entities: To evaluate our reconstruction method on other deformable entities, we employ similar experimental setup as used TAVA (28) Tab. 5 (lower table) presents the quantitative comparison using the synthetic animal dataset introduced by TAVA. Our method shows comparable performance with the baseline method. Fig. T displays qualitative results from one of the animal subjects in the dataset.

---

ResFieM[455
NDR
HypeeNoRR
Rrpartcar Rasiiaa HC!
Figure 5: Qualitative comparison with SoTA methods on BEHAVE dataset.
Ablation studies: We present an ablation study for different network design choices and losses in Tab. 6 Without initializing the INN with xinit, In this experiment, we use an MLP to learn the initialization, following traditional INN networks [7]. instead of initializing based on the distance of the deformed points from the posed skeleton (see Section 3 B.). Our initialization method leads to better reconstruction performance. With the same geometry head: Here, architecture with a single INN for both deformable and non- non-deformable objects, for predicting skinning weights and SDF. Rather than using semantic- aware ray sampling in image space for disentangling motions and geometries of entities (see Section 3 A), a semantic logit is predicted from the SDF and optimized with semantic loss to produce semantically separable geometries following (5) This approach struggles with high occlusion and complex interactions, resulting in poor reconstructions (Figure(B). 6 Without Lw and Lyose loss: These losses are crucial for constraining the prediction networks. The Lw has significant impact, effectively constraining the INN and skinning weight prediction, improving the reconstruction quality. Training convergence for W and W/o Broyden method: We experiment to assess the efficacy of our INN network compared to the Broyden- LBS learning approach. For this purpose, we train our network replace the INN network and use the Broyden equation solver to solve the Eqn. and report the progress of CD with respect to time (Fig. 7. Our network. designed with INN, converges much faster than the Broyden
unified network
method used in 28 41 Also, our approach takes around 0.3 sec/frame with INN compared to 0.9
sec/frame with the Broyden approach.
W Booyke
Metric
Dist.
F-score (%)
Chamfer
Wia Broyla
Method
Acc. (cm)
(%)
2.0
Wlo siit (Section 3
B
3,92
83.50
3.51
Same geometry heads (Section 3 A,B.C)
7.83
41.18
8.42
W/o Lw (Equ.
3.83
82.67
3.53
Wio Epmee (Equ.
2.73
90.17
2.67
Ours
2.72
91.34
2.43
Table 7: INN vs Broyden formu-
Table 6: Ablation on BEHAVE for holistic reconstruction.
lation.
Further study and limitations: Although we successfully dis- entangle motions of the inter- acting entities, our framework currently employs separate net- works for each entity, which is not scalable for scenarios involving more than two Figure 6: (a) Using single SDF network for both entities. (b) Using separate SDF networks for individual elements.
tities. This limitation affects its practicality for more com- plex scenes with numerous interacting objects. A potential solution could involve the integration of an occlusion map to better manage interactions among multiple entities.

---

Also, we further analyze the depen- dency of reconstruction quality on 3D pose and semantic map accuracy (Ta With predicted masks: For this purpose, we have used a com- bination of a state-of-the-rrt object detection network, YOLOv8 [48),
Evalestien
of MadiPPose
Dis.
Prec.
Chanfer
Proc.
(cm)
[cm)]
(25)
(75)
[55]
(sm1)
GT
Rod. mak
2236
966003
50.619
93,233
2.125
Pied. mask
5.290
3.392
$1.851
83.512
12.602
3,34]
Chjaz Reeen
GT
A578
2421
82.762
92.418
$6.91
2741
Prid mask prse.
$1.159
$1.500
$6.000
2.001
Recon.
GT
93.853
2,01
Prid mask
4.88
83365
164
35.00
3211
for first detecting the objects under reconstruction and then SAM [49]
Table 8: Reconstruction results on the BEHAVE dataset with pre-
dicted semantic masks and predicted pose.
for segmenting the respective objects within the predicted bounding boxes given by YOLOv8. Results show that our method generate similar reconstruction results as dataset-given masks, because, in our method, the reconstruction qual- ity for the semantically separable geometries is not solely dependent the quality of input semantic masks. We utilize information from both 2D semantic masks and 3D skeletons. While the rays are sampled in image space within 2D bounding boxes around each entity, we also perform encoding for every 3D point on the sampled rays based on its distance from the 3D skeletons of individual elements under reconstruction. With predicted pose: We generate these 3D skeletons by using the state-of-the-ar 3D pose estimation network [50]. As the results show, our method needs a good quality 3D skeleton to constrain the shape and motions of the individual elements. Even though the re- construction quality is not affected, the inaccuracies come from the predicted pose (Figure 77. So, this another limitation of our method, that requires good quality 3D pose for correct reconstruction.
5
Conclusion
This paper introduces TFS-NeRF, a 3D semantic NeRF framework for dynamic scene reconstruction using sparse/single-veww RGB videos. Uti- lizing INN, our approach signifi- cantly streamlines the training pro- cess, addressing the typically long convergence times associated with existing template-fres methods. Our method effectively disentangles the motions of multiple entities, whether rigid, non-rigid, deformable, and optimizes per-entity skinning weights for accurate and seman- tically distinct 3D reconstructions. We conducted extensive experiments across various datasets, showcasing our approach's 's superior capability to manage complex interactions between multiple entities while ensuring high-quality reconstructions.
(4)
(H)
(d)
Figure 7: Input image, (b), (c) Reconstruction with ground-truth and predicted pose, (d) pose inaccuracy.
Broader Impact. Positive implications include advancements in digital media, robotics, and medical imaging, to name a few. Potential negative impacts involve privacy concerns and bias in model training, which can be mitigated by implementing strict data policies and ensuring diverse training datasets. Our proposal promises significant benefits for various fields, though it requires careful consideration of ethical and societal impacts.
Acknowledgments. This work has been partially funded by The Australian Research Council Discovery Project (ARC DP2020102427). We acknowledge the partial sponsorship of our research by the DARPA Assured Neuro Symbolic Learning and Reasoning (ANSR) program, under award number FA8750-23-2-1016
References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)99-100, 2021.
[2] Lior Yariy, Jiatoo Gu. Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805 4815, 2021.
10

---

[3] Peng Wang, Lingjie Liu. Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171-27183 2021.
[4] Zehao Yu, Songyou Peng. Michael Niemeyer, Torsten Sattler, and Andreas Geiger, Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 35:25018 25032, 2022.
Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Object- compositional neural implicit surfaces. In European Conference on Computer Vision, pages 197-213. Springer, 2022.
[6] Qianyi Wu, Kaisiyuan Wang, Kejie Li, Jianmin Zheng, and Jianfei Cai. Objectsdf++: Improved object- compositional neural implicit surfaces. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 21764-21774, 2023.
[7] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In ICCV, 2021.
[8] Sida Peng, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang. Qing Shuai, Hujun Bao, and Xiaowei Zhou. Animatable implicit neural representations for creating realistic avatars from videos. TPAMI, 2024.
[9] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. CVPR, 2021.
[10] Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 1114955-19666 2021.
[11] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. Seminal Graphics Pupers: Pushing the Boundaries, Volume 2, pages 2023.
[12] Qianli Ma, Jinlong Yang. Amurag Ranjan, Sergi Pujades. Gerard Pons-Moll, Siyu Tang, and Michael J. Black. Learning Dress 3D People in Generative Clothing. In Computer Vision and Pottern Recognition (CVPR), June 2020.
[13] Jia Wai Liu, Yan-Pei Cao, Tianyuan Yang, Eric Zhongcong Xu, Jussi Keppo, Ying Shan, Xiaohu Qic, and Mike Zheng Shou. Hosnerf: Dynamic human human-object-scom neural radiance fields from a single vidco. arXiv preprint ar arXiv:2304. 122.22214..2.. 2023.
[14] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[15] Hongrui Cai, Wanquan Feng. Xuetao Feng. Yan Wang, and Juyong Zhang. Neural surface reconstruction of dynamic scenes with monocular rgb-d camera. Advances in Neural Information Processing Systems, 35:967-960 2022.
[16] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vodaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2863-2873, 2022.
[17] Gengshan Yang, Chaoyang Wang, N Dinesh Reddy, and Deva Ramanan. Reconstructing animatable categories from videos. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 16995-17005, 2023.
[18] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias NieBner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 2022.
[19] Chen Gao, Ayush Saraf. Johannes Kopf, and Jia- Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7712-5721, 2021.
11

---

[20] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman. Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IELE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[21] Zhenggi Li, Qianqian Wang. Forrester Cole, Richard Tucker, and Noah Snavely. Dynihar: Neural dynamic image-based rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[22] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic seenes, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6508, 2021.
[23] Chonghyuk Song. Gengshan Yang, Kangle Deng, Jun-Yan Zhu, and Deva Ramanan. Total-recon: De- formable scene reconstruction for embodied view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17671-17682, 2023.
[24] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11959-12978 2021.
[25] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu, Tensor4d: Efficient neural 4d decompasition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16632-16642, 2023.
[26] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR, 2023.
[27] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ri- cardo Martin Brualla, and Steven M Seitz. Hypemert: a higher- dinger--imensionall representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 40(6): 12, 2021.
[28] Ruilong Li, Julian Tanke, Minh Vo, Michacl Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. European Conference on Computer Vision. pages 419-436. Springer, 2022.
[29] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. International Conference on Learning Representations, 2016.
[30] Zhenzhen Weng and Serena Yeung. Holistic 3d human and scene mesh estimation from single view images. arXiv preprint 20X2::00120015:: 2020.
[31] Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theohalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In IEEF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2022.
[32] Xi Wang, Gen Li, Yen-Ling Kuo, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Reconstructing action-conditioned human-object interactions using commonsense knowledge priors. In International Conference on 3D Vision (3DV), 2022.
[33] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: Contact, human and object recon- struction from a single rgb image. In European Conference on Computer Vision, pages 125-145. Springer, 2022.
[34] Jason Y. Zhang, Sam Pepase, Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d bujjnn-obecct spatial arrangements from single image in the wild. In European Conference Computer Vision (ECCV). 2020.
[35] Hongwei Yi, Chun-Hao P. Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus Thies, and Michael J. Black. Human-aware object placement for visual environment reconstruction. In Computer Vision and Pattern Recognition (CVPR), pages 3959-3970, June 2022.
[36] Rishabh Dabral, Soshi Shimada, Arjun Jain, Christian Thcobalt, and Vladislav Golyanik. Gravity-aware monocular 3d human-object reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 12365-12374, 2021.
[37] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In European Conference on Computer Vision, pages 402 418. Springer. 2022.
12

---

[38] Yuheng Jiang, Suyi Jiang. Guoxing Sun, Zhuo Su, Kaiwen Guo. Minye Wu, Jingyi Yu, and Lan Xu. Neuralhofusion: Neural volumetric rendering under human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6155-6165, 2022.
[39] Chen Guo, Tianjian Jiang. Xu Chen, Jie Song, and Otmar Hilliges. Vid2avatar: 3d avatar reconstruction from videos the wild via sulf-super scene decomposition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12858-1 128881 2023.
[40] Chung- Yi Weng, Brian Curless, Pratul Srinivasan, Jonathan Barron, and Ira Gemelmacher--Shiizeein Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference on compiter vision and pattern Recognition, pages 16210-1000 162-0,20, 2022.
[41] Xu Chen, Yufeng Zheng. Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11594-11666 11604, 2021.
[42] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
[43] Charles G Broyden. class of methods for solving nonlinear simultaneous equations. Mathematics of computation, 19(92):577-5999 1965.
[44] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: method for 3d annotation of hand and object poses. In CVPR, 2020.
[45] Marko Mihajlovic, Sergey Prokudin, Mare Pollefeys, and Siyu Tang. Resfields: Residual neural fields for spatiotemporal signals. In The Twelfth International Conference on Learning Representations. 2023.
[46] Qi Fang, Qing Shuai, Junting Dong, Hujun Bao, and Xiaowei Zhou. Reconstructing human pose by watching humans in the mirror. In CVPR, 2021.
[47] Jiahui Lei and Kostas Danillidis. Cadex: I Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6624-6634, 2022.
[48] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolo, january 2023. URL https://github.
comfultaalyiissuurralttiss
[49] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Speneer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[50] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV, 2021.
A Appendix / supplemental material
A.1 Neural Network architectures:
Invertible Neural Network: It transforms the view space points to canonical space (Fig. 3
x = (x (xinit J). We use reaINVP 229 as the baseline of our Invertible Neural Network. This network consists of 2 Coupling layers 29 each with scaling and translation prediction modules. Each of these modules consists
lincar layers with dimensions 331 x 512, 512 x 512, and 512. The input to the INN network is the deformed space points and it is conditioned on the skeleton pose. The input points are first transformed using a projection layer to dimension 256 which is concatenated with the skeletal pose (72) and passed as input to the scale and translation prediction networks. The predicted scale and translation transform the deformed space points to canonical space.
Skinning weight prediction network: The transformed canonical points are passed through the
skinning weight prediction network w, GW -x- >w (x() (Fig. 3 The skinning weight prediction network consists of linear layers with dimensions 3 x 256, 256 256, 256 x 24. The skinning
weight prediction network takes the transformed canonical points input, hence, the dimension is
13

---

3. The output defines the weightage of each skeleton joint on 3D point for its deformation from canonical space deformed space, Hence, the dimension the output layer is 256 24. The output activation layer is defined as softmax the sum of individual weight should be 1. Two similar architecture weight prediction networks are used for skinning weight prediction for individual entities.
SDF prediction network: Transformed canonical points are passed through the SDF prediction
network Fig. 2 for geometry prediction, F ) 3++ + R 1+256 The SDF prediction
network consists of 8 linear layers each with a hidden size of 256. A skip connection is added at layer 4. The dimensions of each layer are as follows 114 256, 256 256, 256 x 256, 256 217. 256 x 256, 256 x 256, 256, 256 x 257. The input canonical points are transformed by a frequency layer and mapped to dimension 39, which concatenated with canonical joints represented as 24 x 3 (each entity is represented by 24 skeleton joints). Moreover, paper, each point is assigned semantic label denoting which entity it belongs to i.e. deformable, non-deformable object background. The dimension of the semantic label is 3. Hence, the input dimension 39 + 72 +3 3 = 114. The SDF prediction network generates an SDF value for each point (dimension 1) and a feature representation of dimension 256, resulting in total output dimension of 257. We use separate networks for the SDF prediction of different entities. However, use similar architecture for both entities.
discussed in the main
RGB rendering network: The RGB rendering network (Fig. consists of 5 linear layers with dimensions of 270 x 256, 256, 256, 256 256, and 256 The input to the rendering network is canonical points with dimension 3, normals (calculated from predicted SDF) with dimension 3, and per-frame skeleton pose (concatenated skeletal joints from both the entities with dimension of (72 @ 72) and the SDF predicted feature vector of dimension 256. A linear layer first transforms skeleton pose to lower dimension (8). Hence, the input dimension of the rendering network 3 + 8 + 256 270. The output of the network is the RGB value for each sampled point, hence, the dimension of the output is 3. The output layer activation defined as sigmoid.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, correctly reflect the paper's contribution and scope, the main claims made clear in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitation of the proposed work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results, hence no proof or assumptions are provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper disclose all the information required to reproduce the results.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [No]
Justification:
The paper does not open access the code during submission for review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
criiionnoiccccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp..cppulli///uuii wwwtii/gguie///eeeeeeoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper discusses all details regarding the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
Justification:
We do not produce error bars or statistical studies for our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper mention about the computation resources used during training and testing the method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted in the paper conform with the NeuRIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Yes the paper discuss the potential positive and negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for approach optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not produce research in the above-mentions areas, so there is no risk of misuse of released data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper discuss about all the technical details of the proposed methods.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper experiments with open source datasets, but does not capture any new data involving the human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper experiments with open source datasets, but does not capture any new data involving the human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "4KaEICSZpv", "reviewer": "Reviewer_Rbt4", "summary": "This paper proposes TFS-NeRF, a semantic-NeRF framework leveraging no prior templates of dynamic scenes for 3D reconstruction. Guided by INN-driven LBS prediction and semantic-aware ray sampling, TFS-NeRF separately consider deformable and non-deformable parts during geometric learning but composite them to learn apperance to benefit from self-supervised RGB reconstruction loss. Experiments on human-object and animal cenric dynamic videos demosntrate the advantages of the proposed method against chosen baselibes.", "questions": "Please see the weakness section above on the clarification on the robustness to masks and pose qualities, as well as more in-depth discussions towards recent baselines and more general scenes.", "limitations": "Limitations are properly mentioned in the main paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-The overall paper is well motivated and easy to follow. The qualitative reuslts and supplement videos demonstrate the promising reconstruction performance of dynamic scenes with multiple entities.\n\n-The experiment are extensively evalauted on several public benchmarks and the ablation studies highlight the unqiue contribution of several design choices.", "weaknesses": "My main concerns comes from the lack of clarification on several key components:\n\n(1)As a semantic-nerf framework, how abou the influence of quality of input masks? As to labels from an imperfect 2D predictor, what is the impact on final results, since the label quality may affect the sampling quality for deformable and non-deformable rays.\n(2)Similarly, what is the robustness to pose accuracy as poses are involved with both ray generation and the pose loss?\n\nDiscussion on the above two points could better strenthen the practicalibility of TFS-NeRF.\n\n(3)As a rapidly growing area, the chosen baselines do not include latest methods such as HexPlan,Tensor4D or GS-based ones to highlight the advantages of using INNs, compositional rendering as well as reconstruction quality.\n\n(4)As to LBS predicton, what is the unique advantages of applying INN over other types of NN modules (e.g, lossy CNN auto-encoders)? What is the efficiency or performance boosts compared to standard CNNs?\n\n(5)How to get the initial values of B_i to compute the initialization value of x_c for arbitrary objects?\n\n(6)As the paper is claimed to template free and able to deal with generic scenes, it would be more exciting and convincing to see other types of interactions? For example, a common daily objects (e.g. a door or a cabinet, opening and closing) to really show the advantages of prior-free deformations of general scenes.\n\n(7)It would be good to aldo add the appearance quality or view synthesis performanc, though I understand the focus of TFS-NeRF is on 3D reconstruction.\n\n(8)Formatting needs to be improved. For example, it is wierd that in Sec3.B, authors put much related work descriptions. There are also improper indents like Line 282 and Line284,etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Robustness to mask accuracy:** Please refer to the *Author Rebuttal* section.\n\n**Robustness to pose accuracy:** We agree that for the practicability of TFS-NeRF, it is important to evaluate it from this aspect, and apologize for not presenting the same in the initial submission. We have added quantitative results for the reconstruction quality of our method using predicted 3D poses (**Table 1 in the attached pdf in Author Rebuttal section**). We generate these 3D poses by using the state-of-the-art 3D pose estimation network [3].  As the results show, our method needs a good quality 3D pose to constrain the shape and motions of the individual elements. Even though the reconstruction quality does not get affected, the inaccuracies come from the predicted pose (**Figure 1 in the attached pdf in Author Rebuttal section**). We will include this evaluation in the final paper.\n\n**Comparison with current methods (Hexplane, Tensor4D, GS-based methods):** Thank you for highlighting these methods. We primarily compared our approach with techniques that emphasize geometry reconstruction, such as NDR, ResField, D-NeRF, and HyperNeRF, which predict Signed Distance Functions (SDF) for underlying geometries. We agree that including Tensor4D in the comparison is valuable and have conducted additional experiments to assess its performance relative to our method (**Figure 2 and Table 2 in attached pdf in Author Rebuttal section**).\n\nOur experiments show that our method provides superior reconstruction quality compared to Tensor4D. While Tensor4D's key innovation lies in its efficient 4D tensor decomposition, which represents dynamic scenes as a 4D spatiotemporal tensor, it does not account for the relative motions between scene elements. Instead, it captures the scene's overall motion. In contrast, our approach models the motion of individual elements and employs skeleton-guided reconstruction, leading to more accurate geometry.\n\nAdditionally, current GS-based methods for dynamic scenes primarily focus on rendering rather than surface reconstruction. Research indicates that methods emphasizing geometry reconstruction often fall short in rendering quality, and vice versa (refer to Table 3 in 2DGS [4], Table 1 in SuGaR [5]). Similarly, Hexplane prioritizes image rendering over geometry reconstruction which is a NeRF-based method. Similar arguments are applicable for the NeRf-based method [7]. Therefore, these methods may not be directly comparable. Hence, given the limited time frame for rebuttal, we are concentrating on the geometry-based method. However, we are open to including these methods in the final comparison.\n\n**INN vs Lossy auto-encoder:** We appreciate your feedback and would like to clarify any confusion. We chose INN because we expect to maintain the invertible mapping between the 3D point space and the deformation space, which is the property that INN naturally has. A lossy auto-encoder could also become a choice only if it has the same property. We left the model structure exploration as future work to enhance the semantic reconstruction task further.\n\n**Computation of $B_i$ for arbitrary objects:** We apologize for the lack of clarity in the submission. For arbitrary non-articulated objects, $B_i$​ is computed based on the 3D bounding boxes, which are derived from multiview semantic masks. Specifically, for non-articulated objects, $B_i$​ represents the rigid transformation between the bounding boxes of the canonical frame (keyframe) and each input frame. We will ensure that this explanation is included in the final version of the paper to provide clearer context.\n\n**Evaluation on more datasets (daily life objects with door close, opening):** Thank you for highlighting this important aspect. Given the limited time frame for rebuttal, we are unable to evaluate additional datasets at this moment. Moreover, we have already evaluated several benchmark datasets (including datasets like BEHAVE containing daily life objects) to showcase the efficacy of our method. We appreciate your understanding and interest in expanding the scope of our evaluation.\n\n**Rendering quality:** We appreciate your feedback regarding the importance of appearance quality. We have compared our rendering quality with Tensor4D and obtained the following results: PSNR of 30.09 and SSIM of 0.970, whereas Tensor4D achieved a PSNR of 35.78 and an SSIM of 0.985. Methods, that show very good geometric reconstruction quality do not necessarily perform well in rendering and vice versa ([4], [5], [7]). The improvements in geometric detail often reduce the rendering quality, as the optimization process becomes more complex and computationally demanding.​ In our work, we aim to do good reconstruction but we will not care about rendering quality.\n\n**Formatting issues:** Thank you for the detailed reviews and feedback. We will rectify all these errors in the final paper\n\n[3] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV, 2021.\\\n[4] Huang B, Yu Z, Chen A, Geiger A, Gao S. 2d gaussian splatting for geometrically accurate radiance fields. InACM SIGGRAPH 2024 Conference Papers 2024 Jul 13 (pp. 1-11).\\\n[5] Guédon A, Lepetit V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 5354-5363).\\\n[7] Martin-Brualla R, Radwan N, Sajjadi MS, Barron JT, Dosovitskiy A, Duckworth D. Nerf in the wild: Neural radiance fields for unconstrained photo collections. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2021 (pp. 7210-7219).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RmDmGdYdED", "reviewer": "Reviewer_KrvP", "summary": "The paper addresses the problem of reconstructing dynamic environments for arbitrary rigid, non-rigid, or deformable entities. The authors propose a template-free 3D semantic NeRF for dynamic scenes, which employs an Invertible Neural Network (INN) for LBS prediction, and optimizing per-entity skinning weights based on semantic masks. The experimental results show high-quality reconstructions of complex interactions with improved training efficiency compared to existing methods.", "questions": "- How is the semantic masks calculated? How is objects classified as rigid or non-rigid?\n- During the training, are the same number of rays sampled for rigid and non-rigid objects?", "limitations": "- The method is not scalable for the scenes more than two entities.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method is able to reconstruct arbitrary non-rigid objects by utilizing TAVA framework.\n- Disentanglement of objects improves the reconstruction quality by predicting LBS for each entity.\n- The proposed framework outperforms existing template-free NeRF methods on various datasets containing different non-rigid and rigid objects.\n- The paper is clearly written and easy to follow.", "weaknesses": "- The paper emphasizes semantic-aware ray sampling as a key contribution but lacks details on how the semantic masks are generated.\n- The framework heavily relies on existing methods (TAVA, INN), limiting the novelty and distinctiveness of its contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**How is the semantic masks calculated:** Please refer to the *Author Rebuttal* section.\n\n**Limiting novelty and distinctiveness w.r.t TAVA, INN:** Our method focuses on the research gap of producing semantic 3D reconstruction of dynamic scenes under multiple object interactions. While our problem setup of reconstructing the scene guided by a 3D skeleton is similar to TAVA, extending this concept for various object interactions is not trivial. Moreover, TAVA relies on an iterative method for solving the forward LBS, which is not time-efficient. In our method, we utilize INN to make the solution more time efficient, which makes it even more challenging under multiple object interactions setup, causing occlusion and different motions. \n\n**Classifier for rigid, non-rigid objects:** Thank you for your question. Based on the semantic masks, it is indeed possible to infer whether an object is rigid or non-rigid. We do not explicitly employ a separate classifier for this purpose. Instead, our approach leverages the information provided by the semantic masks to distinguish between rigid and non-rigid objects.\n\n**Number of rays sampled for rigid and non-rigid objects:** The rays are sampled in a 60:40 ratio for the non-rigid and rigid objects, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WTenld51lN", "reviewer": "Reviewer_3yt5", "summary": "This paper propose TFS-NeRF to solve the semantic reconstruction of dynamic scenes.", "questions": "See above", "limitations": "Unclear expressions and limited contributions", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Experimental results on multiple entity and deformable entity reconstructions are good.", "weaknesses": "It is very hard to follow the storyline of the introduction.\n\nUnclear contributions. In Lines 81 and 91, the paper claims it could reconstruct multiple dynamic entities with multiple complex interactions, but in the experiment parts, the presented results are mostly from one person/animal with a single object, it is hard to judge whether the proposed model could process multiple dynamic entities with complex interactions. Also, the authors claim the proposed model could address the occlusion challenge in Line 81, but there is no experiments about it. Besides, the paper claims that it could reconstruct semantics of dynamic scenes by using semantic masks. However, such semantic reconstruction has been solved by previous works like Semantic Flow [1], it is hard to judge the contribution of this paper in this situation.\n\nMissing important method details. This paper proposes a semantic-aware ray sampling strategy and illustrates the points are separated in to deformable object set and non-deformable object set, but does not present any details about the separation process. After reading the paper, the reviewer guesses it separates the points on the rays based on the semantic labels of the pixels traveling through the rays. However, it may not be correct since different points in the same camera ray may belong to different set (like a ray travels through a dynamic person and a static wall behind the person). It is very hard to understand the entire pipeline of semantic reconstruction without such details.\n\nWriting should be significantly improved. There are many informal and unclear expressions in this paper:\n\n- Line 405: Takeaways -> Conclusion\n- Figure 2: Overview of the system-\n- Line 229: unclear expression of R^{1+256}\n- Line 221: why \"it helps capture better articulation or deformation and surface reconstruction under motion?\"\n- Tables 5 and 6: what is the mearning of \"->\"\n- Line 216: Given,\n- Lines 98, 112, 136: the usage of : should be the same\n\nReference:\n[1] Tian, Fengrui, Yueqi Duan, Angtian Wang, Jianfei Guo, and Shaoyi Du, Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hard to follow introduction:** Thank you for your feedback regarding the introduction. We apologize if the storyline was challenging to follow. We will carefully review and revise the introduction to ensure it is clearer and more cohesive. Your input is valuable, and we appreciate your efforts to help us improve our work.\n\n**Claim on multiple dynamic entities:** Please refer to the *Author Rebuttal* section.\n\n**Claim on addressing occlusion challenge, but no experiment:** \nThank you for the insightful review. We have included an experiment (**Figure 3 in the attached PDF in Author Rebuttal section**).\n\nOur goal is to generate a 3D reconstruction of the scene without using any template model. Unlike methods that use the T-pose or A-pose of SMPL models as canonical representations to handle self-occlusions, we select a keyframe from the sequence as the canonical pose. Consequently, if there is no frame in the sequence without occlusion, the SDF network struggles to capture the occluded geometry in the canonical space.\n\nTo address this occlusion challenge, we use a strategy that disentangles the motions of distinct entities. This involves semantic-aware sampling and distance-based encoding of 3D points on the rays. Additionally, we employ separate SDF prediction networks for each entity, which helps mitigate the issue. Our overall design helps mitigate the occlusion problem.\n\n**Contribution compared to SemanticFlow** Thank you for mentioning this method. SemanticFlow primarily focuses on semantic rendering and scene editing. In contrast, our work is focused on semantic geometry reconstruction. Techniques that excel in rendering may not necessarily perform well in 3D geometry reconstruction ([4], [5], [7]). Therefore, our primary emphasis has been on methods that are strong in geometry reconstruction. However, we are considering evaluating this method to validate its effectiveness and to determine if it could be integrated into our final submission if appropriate.\n\n**Separates the points on the rays based on the semantic labels of the pixels** We apologize for not being clear in explaining this approach. We utilize information from both 2D semantic maps and 3D skeletons. While the rays are sampled in image space within 2D bounding boxes around each entity, we also perform an encoding for every 3D point on the sampled rays based on their distance from the 3D skeletons of individual elements. As mentioned in lines 230-238 of the initial submission for details, to provide the SDF prediction network information about which object the point belongs to, we assign a semantic label to each 3D point. This is defined as a weightage, $\\omega^j = \\exp(-dist^2/\\sigma^2)$ based on the distance of the point from the nearest 3D joints in the per entity canonical skeletons $\\mathbf{J^j_{p0}}$.\n\n**Formatting issues** Thank you for the detailed reviews and for pointing out these errors in the manuscript. We will rectify all these errors in the final paper.\n\n[4] Huang B, Yu Z, Chen A, Geiger A, Gao S. 2d gaussian splatting for geometrically accurate radiance fields. InACM SIGGRAPH 2024 Conference Papers 2024 Jul 13 (pp. 1-11). \\\n[5] Guédon A, Lepetit V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 5354-5363).\\\n[7] Martin-Brualla R, Radwan N, Sajjadi MS, Barron JT, Dosovitskiy A, Duckworth D. Nerf in the wild: Neural radiance fields for unconstrained photo collections. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition 2021 (pp. 7210-7219).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qXCYwbmmjk", "reviewer": "Reviewer_UY9B", "summary": "The author proposed a novel, time-efficient, template-free NeRF-based method for 3D dynamic scene reconstruction, focusing on capturing detailed explicit geometry for each entity in the scene. Extensive experiments demonstrate the efficiency of the proposed method.", "questions": "See weaknesses.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized, and the font size in the figures is appropriate, making it easy for readers to follow.\n2. The method section is straightforward, providing a detailed introduction to each part, including sufficient details on the network, training, and loss weights.", "weaknesses": "1. The author mentioned that one of the contributions is time efficiency. Based on the description in lines 183-202, it seems the proposed method reduces computation to accelerate training. However, Figure 6 only shows the convergence speed. It would be more informative if the author could report the time consumed per iteration.\n\n2. The information on how to obtain the semantic masks is missing. For example, in RoDynRF [1], the author uses optical flow and Mask R-CNN to obtain the segmentation masks.\n\n3. The work focuses more on dynamic scenes with human subjects, but the title gives the impression that the method is designed for general dynamic scenes.\n\n4. From Figures 2 and 3, it appears that the proposed method requires a 3D skeleton as part of the inputs. Does this mean that the proposed method does not work for scenes without humans or datasets lacking 3D skeleton information?\n\n5. In the bottom part of Table 3, under the Dist. Acc column, HyperNeRF is the second-best method.\n\n[1] Liu, Yu-Lun, et al. \"Robust dynamic radiance fields.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Time consumed per iteration:** Thank you for bringing this to our attention. Our approach takes around 0.3 sec/frame with INN while 0.9 sec/frame with the Broyden approach used in TAVA. We will include this information in our final paper.\n\n**Information on how we obtain semantic masks:** Please refer to the *Author Rebuttal* section for the response.\n\n**Gives impression of general dynamic scenes, but works on human subjects:** Please refer to the *Author Rebuttal* section for the response. \n\n**Method does not work for scenes without humans or datasets lacking 3D skeleton information:** \nWe acknowledge that our method relies on 3D skeleton information to constrain the network and produce accurate shapes and geometries of the scene objects. However, this 3D skeleton information can be easily obtained for both rigid and non-rigid objects. For rigid objects, skeleton joints are derived from uniformly sampled 3D points along the x, y, and z axes, passing through the center of the 3D bounding box (obtained from multiview semantic masks) surrounding the object. For non-rigid objects, 3D skeletons can also be easily acquired using available tools (e.g. [3],[6], etc.).\n\n**Table 3, HyperNeRF is second best:** Thank you for pointing out the error. We will rectify the error in the final submission.\n\n[3] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Black Michael J., and Tao Mei. Monocular, One-stage, Regression of Multiple 3D People. In ICCV, 2021. \\\n[6] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie W. Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful for the constructive feedback from the reviewers and are pleased that they found our paper "well-motivated, well-organized, and easy to follow" (Rbt4, KrvP, UY9B). They highlighted that our paper includes a "detailed introduction, sufficient details, and extensive evaluation on several public benchmarks" (UY9B, Rbt4). Additionally, they noted that our method produces "promising reconstruction under multiple entities" (Rbt4, 3yt5). We appreciate these insights and will incorporate all the suggestions and additional experiments to enhance the writing and presentation of the paper. In this section, we try to address all the common concerns.

**How masks are generated and the influence of the quality of masks on results [Rbt4, UY9B]:** We appreciate for highlighting this point. Following the baseline method TAVA,  we use the semantic masks given in respective datasets, to generate the quantitative results for ours as well as the comparing methods. \

As per the suggestion, to evaluate the robustness of our method in terms of the accuracy of semantic masks, we have further provided an experiment with predicted semantic masks (**Table 1 in attached pdf**). For this purpose, we have used a combination of a state-of-the-art object detection network, YOLOv8 [1], for first detecting the objects under reconstruction and then SAM [2] for segmenting the respective objects within the predicted bounding boxes given by YOLOv8. Results show that our method can generate similar reconstruction results as *dataset given masks*, because, in our method, the reconstruction quality for the semantically separable geometries is not solely dependent on the quality of input semantic masks. We utilize information from both 2D semantic masks and 3D skeletons. While the rays are sampled in image space within 2D bounding boxes around each entity, we also perform an encoding for every 3D point on the sampled rays based on its distance from the 3D skeletons of individual elements under reconstruction (Please refer to lines 230-238 of the initial submission for details). Hence, our method does not need a very accurate semantic mask as input. We will include this evaluation in the final paper.

**Evaluation dataset (mentioned generic scene but results given only on two entities, mostly focused on the human, or a single entity) [3yt5, UY9B]** We apologize for the lack of clarity and confusion in our presentation. By *multiple dynamic entities* in our paper, we mean more than one entity interacting with each other. Our method considers only two entities as of now. We have discussed this limitation under the *limitations and future directions* section (Lines 398-404 in the submission). The results on single entities (only human or animal subjects) are presented to demonstrate the capability of our reconstruction methods for *arbitrary deformable subjects* without using any template.\
For datasets with more than one entity, we selected two types: 1) human-object and 2) hand-object interaction datasets. To the best of our knowledge, there is no available dataset where an animal interacts with any object.\
We agree with the reviewer that the phrase multiple dynamic entities could be misleading, suggesting that our method can handle more than two entities. We will revise this phrase in the final paper to ensure clarity.

**[1]** Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics YOLO, January 2023.\
**[2]** Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao,
Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.

Meta-review: The authors and reviewers engaged in a detailed discussion. Thank you all for your time!
After the discussion most concerns could be addressed by additional explanation and results. Some points still remain open and the score average borderline, with one weak reject. All reviewers provided detailed reviews, with two of them increasing their initial rating (Rbt4 from 4 to 6 and UY9B 4 to 5). The one negative review remains critical about the motivation, clarity, and overall writing. The AC agrees that these points could be improved but points are relatively minor, with the setting (two objects interacting) already being well explained. Hence, the paper can be accepted with minor revisions to the final version concerning clarity. Please incorporate the issues raised by 3yt5 carefully.
Decision: Accept (poster)