Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera
Authors: Hongrui Cai
Wanquan Feng
Xuetao Feng
Yan Wang
Juyong Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Dynamic Reconstruction
Monocular RGB-D Reconstruction
Neural Implicit Function

Abstract: We propose Neural-DynamicReconstruction (NDR), a template-free method to recover high-fidelity geometry and motions of a dynamic scene from a monocular RGB-D camera. In NDR, we adopt the neural implicit function for surface representation and rendering such that the captured color and depth can be fully utilized to jointly optimize the surface and deformations. To represent and constrain the non-rigid deformations, we propose a novel neural invertible deforming network such that the cycle consistency between arbitrary two frames is automatically satisfied. Considering that the surface topology of dynamic scene might change over time, we employ a topology-aware strategy to construct the topology-variant correspondence for the fused frames. NDR also further refines the camera poses in a global optimization manner. Experiments on public datasets and our collected dataset demonstrate that NDR outperforms existing monocular dynamic reconstruction methods.

Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera
Hongrui Cail, Wanquan Feng', Xuetao Feng', Yan Wang', Juyong Zhang
University of Science and Technology of China !Alibbba Group
https://ustcddv. github.io/nnt io/ndr
Abstract
We propose SuurallDyyaamcceecosttttttooooo (NDR), a template-free method to recover high-fidelity geometry and motions of dynamic scene from monocular RGB-D camera. NDR, we adopt the neural implicit function for surface repre- sentation and rendering such that the captured color and depth can be fully utilized to jointly optimize the surface and deformations. To represent and constrain the non-rigid deformations, we propose novel neural invertible deforming network such that the cycle consistency between arbitrary two frames is automatically satisfied. Considering that the surface topology of dynamic scene might change over time, we employ topology-aware strategy to construct the topology-varin correspondence for the fused frames. NDR also further refines the camera poses in global optimization manner. Experiments on public datasets and our col- lected dataset demonstrate that NDR outperforms existing monocular dynamic reconstruction methods.
Introduction
Reconstructing 3D geometry shape, texture and motions of the dynamic scene from monocular video a classical and challenging problem in computer vision. has broad applications in many areas like virtual and augmented reality. Although existing methods [63, 65] have demonstrated impressive reconstruction results for dynamic scenes only with 2D images, they are still difficult to recover high-fidelity geometry shapes, especially for solutions exist without depth constraints. Only with 2D measurements, dynamic reconstruction methods require that motions of interested object hold in nearby z-plane. Meanwhile, it is difficult to construct reliable correspondences in areas with weak texture, which causes error accumulation in the canonical space.
casually captured data abundant potential
To solve this under-constrained problem, some methods propose to utilize shape priors for some special object types. For example, category-specifie parametric shape models like 3DMM [6], SMPL [41] and SMAL [72] are first constructed and then used to help the reconstruction. However, templated-basee methods could not generalize unknown object types. On the other hand, some methods utilize annotations, like keypoints and optical flow, obtained from manual annotators fffftf tools [31, 33, 63, 65]. The motion trajectories of sparse dense 2D points can effectively help recover the exact motion the whole structure. However, it needs human labeling for supervision or highly depends on the quality of learned priors from a large-scale dataset.
One straightforward solution to this under-constrained problem to reconstruct the interested object based on observations from RGB-D cameras like Microsoft Kinect [69] and Apple iPhone X. Existing iusion-based methods [44, 27, 54] utilize a dense non-rigid warp field and a canonical truncated signed
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

distance (TSDF) volume to represent motion and shape, respectively. However, these fusion-based methods might fail due accumulated tracking errors, especially for long sequences. To alleviate this problem, some learning-basee methods [9, 8, 39] utilize more accurate correspondences which are annotated or learned from synthesis datasets to guide the dynamic fusion process. However, the captured color and depth information is not represented together within one differentiable framework these methods. Recently, neural implicit representation based method [3] has been proposed to reconstruct room-scale scene from RGB-D inputs, but it is only designed for static scenes and can not be directly applied to dynamic scenes.
Time
Figure I: Examples of reconstructed (right) and rendered (left) results by NDR. Given a monocular RGB-D video sequence, NDR recovers high-fidelit, geometry and motions of dynamic scene.
In this paper, we present Veural-DynamicRcoostuuttio (NDR), a neural dynamic reconstruction method from monocular RGB-D camera (Fig. 1). To represent the high-fidelity geometry and texture of deformable object, NDR maintains a neural implicit field as the canonical space. With extra depth constraint, there still exist multiple potential solutions since the correspondences between different frames are still unknown. this paper, we propose the following strategies constrain and regularize the solution space: (1) integrating all RGB-D frames to a high-fidelity textured shape in the canonical space: (2) maintaining eycle consistency between arbitrary two frames; (3) surface representation which can handle topological changes.
Specifically, adopt the neural SDF and radiance field to respectively represent the high-fidelity geometry and appearance the canonical space instead of the TSDF volume frequently used in fusion-based methods [28, 44, 54, 9, 8, 39]. In our framework, each RGB-D frame can be integrated into the canonical representation. We propose a novel neural deformation representation that implies continuous bijective map between observation and canonical space. The designed invertible module applies a cycle consistency constraint through the whole RGB-D video; meanwhile, it fits the natural properties of non-rigid motion well. To support topology changes of dynamic scene, we adopt the topology-awe correspondence, our framework can handle topology changes while existing deformation graph based methods [44, 39, 65] could not. NDR also further refines camera intrinsic parameters and poses during training. Extensive experimental results demonstrate that NDR can recover high-fidelity geometry and photorealistic texture for monocular category- RGB-D videos.
network in HyperNeRF [47]. Thanks for modeling topology-variant

---

2
Related Works
RGB based dynamic reconstruction. Dynamic reconstruction approach can be divided into cemplate-based and template-free types. Templates [6, 41, 50, 72] are category-specife statistical models constructed from large-scale datasets. With the help of pre-constructed 3D morphable models [6, 12, 36], some researches [5, 11, 26, 57, 21, 25, 19] reconstruct faces or heads from RGB inputs. Most of them need 2D keypoints as extra supervisory information to guide dynamic tracking [71, 17, 19]. With the aid of human parametric models [l, 41J, some works [7, 62, 23, 24, 70, 29] recover digital avatars based on monocular image or video cues. However, it is unpractical to extend templates to general objects with limited 3D scanned priors, such as articulated objects, clothed human and animals. Non-rigid structure from motion (NR-SFM) algorithms [10, 51, 15, object from 2D observations. Although NR-SFM
34, 53] are to reconstruct category-agnosie can reconstruct reasonable result for general dynamic scenes, it heavily depends on reliable point trajectories throughout observed sequences [52, 56]. Recently, some methods [63, 64, 65] obtain promising results from a long monocular video or several short videos of a category. LASR [63] and ViSER [64] recover articulated shapes via differentiable rendering manner [40], while BANMo [65] models them with the help of Neural Radiance Fields (NeRF) [43]. However, due to the depth ambiguity of input 2D images, the reconstruction might fails for some challenging inputs.
RGB-D based dynamic reconstruction. Recovering 3D deforming shapes from a monocular RGB video is highly inder-constrained problem. On the other hand, The progress in consumer-grade RGB-D sensors has made depth map capture from a single camera more convenient. Therefore, is quite natural to reconstruct the target objects based on RGB-D sequences. DynamicFusion [44], the seminar work RGB-D camera based dynamic object reconstruction, proposes to estimate a template- free 6D motion field to warp live frames into TSDF surface. The surface representation strategy has also been used in KinectFusion [28]. VolumeDeform [27] represents motion in a grid and incorporates global sparse SIFT [42] features during alignment. Guo et al. [20] coheres albedo, geometry and motion estimation in an optimization pipeline. KillingFusion [54] and SobolevFusion [55] are proposed deal with topology changes. During deep learning era, DeepDeform [9] and Bozic et al. [8] aim to learn more accurate correspondences for tracking improvement of faster and more complex motions. OcelusionFusion [39] probes and handles the occlusion problem via LSTM- involved graph neural network but fails when topology changes. Although these methods obtain promising reconstruction results with the additional depth cues, their reconstructed shapes mainly depend on the captured depths, while the RGB images are not fully utilized to further improve the results.
Dynamic NeRF. Given a range of image cues, prior works on NeRF [43] optimize an underlying
continuous scene function for novel view synthesis. Some NeRF-like methods [37, 49, 58, 18, 46, 47] achieve promising results on dynamic scenes without prior templates. Nerfies [46] and AD-NeRF [22] reconstruct free-vie free-viewpoint selfies from monocular videos. HyperNeRF [47] models an ambient slicing surface to express topologically varying regions. Recent approaches [60, 3] introduce neural representation for static object/scene reconstruction, but theirs can not be used for non-rigid scenes.
Cycle consistency constraint. To maintain cycle consistency between deformed frames is an important regularization in perceiving and modeling dynamic scenes [61]. However, recent meth- ods [64, 37, 65] try to leverage loss term to constrain estimated surface features or scene flow, which is a weak but not strict property. Therefore, constructing invertible representation for deformation field reasonable design. Several invertible networks are proposed to represent deformation, such as Real-NVP [16], Neural-ODE [13], I-ResNet [4]. Based on these manners, there exist some methods modeling deformation in space [30, 66, 48] or time [45, 35] domain. CaDeX [35] is a novel dynamic surface representation method using real-valued non-volume preserving module [16]. Different from these strategies, we propose novel scale-invariant binary map between observation space and 3D canonical space to process RGB-D sequences, which is more suitable for modeling non-rigid motion.
3 Method
The input of NDR is an RGB-D sequence (LI, D,), 1.... N} captured by monocular RGB-D camera (c.g., Kinect and iPhone X), where I, E 12 kxx is the i-th RGB frame and
3

---

(a)
Results
RGB-D Frame
tj
Topology aarrggg--wwaree. Network Fq
Appearance
1/;
Deformation
MUP
View Direction V;
Color C2
sDr d(xc)
ti
Implliit Canonical Goometry and Appearance
Bijective Map Fh
Bijective Map Fi
Block
PP:
Rg
y/
Figure 2: (a) Pipeline of NDR. (b) The structure bijective map Fh the deformation field.
D; E #xxwxx is the corresponding aligned depth map. To optimize a canonical textured shape and motion through the sequence, leverage full N color frames I, as well as corresponding depth frames D. Specifically, we first adopt video segmentation methods [14, 38] to obtain the mask M, of interested object. Then, we integrate RGB-D video sequence into a canonical hyper-space composed of a 3D canonical space and topology space. We propose continuous bijective representation between the 3D canonical and observation space such that the cycle consistency can be strictly satisfied. The implicit surface is represented by neural SDF and volume rendering field, as function of input hyper-coordinate and camera view. The geometry, appearance, and motions of dynamic object are optimized without any template or structured priors, like optical flow [65], 2D annotations [9] and estimated normal map [29]. The pipeline of NDR is shown in Fig. 2(a).
3.1 Bijective Map in Space-time Synthesis
Invertible representation. Given 3D point sampled in the space of i-th frame, recent meth- ods [44, 65, 47] model its motion as 6D transformation in SE(3) space. Nerfies [46] and Hyper- NeRF [47] construct continuous dense field to estimate the motion. To reduce the complexity, DynamicFusion [44] and BANMo [65] define warp functions based on several control points. The lat- ter designs both 2D and 3D cycle consistency loss terms to apply bijective constraints to deformation representation, but is just a guide for learning instead of rigorous inference module. Similar the previous works, also construct the deformation between each current frame and the 3D canonical space. Further, we employ a strictly invertible bijective mapping, which is naturally compatible with the cycle consistency strategy. Specifically, we decompose the non-rigid deformation into several reversible bijective blocks, where each block represents the transformation along and around certain axis. In this manner, our deformation representation is strictly invertible and fits the natural properties of non-rigid motion well, which helpful for the reconstruction effect.
We denote p; [xg Vi, 21] 3 as position of the observation space at time f,, in which a deformed surface U, embedded. points. A continuous homeomorphic mapping H4 R - R maps p; back to the 3D canonical position p [x. y. 2]. Supposes that there exists a canonical shape U of the interested object, which is independent of time and is shared across the video sequence. Notes that map H; is invertible, and thus we can directly obtain the deformed surface at time 4;;
noticeable that P; represents any position, both surface and free-space
U: [H, ([x, y, 2,,,),))///,.. U}.
(I)

---

Then, the correspondence of p, can be expressed by the bijective map, factorized as:
y Uý. 2,] (1r,, Vi, z:]) Mí 74 Ves 2:))
(2)
The deformation representation G is cycle consistent strictly, since it is invariant on deforming path (95k o Giy Gix) As a composite function of two bijective maps (Eq. 2), is a appoloyy-ivariian function between arbitrary double time stamps.
Implementation. Based on these observations, we implement the bijective map H. by novel invertible network h. While Real-NVP [16] seems suitable network structure, its scale-variant property limits its usage our object reconstruction task. Inspired by the idea of Real-NVP to split the coordinates, we decompose our ocale-invarian deformation into several blocks. In each block, we set an axis and represent the motion steps as simple axis-related rotations and translations, which are totally shared by the forward and backward deformations. In this manner, the inverse deformation H-1 can be viewed as the composite of the inverse of these simple rotations and translations in H. On the other hand, this map also regularizes the freedom of deformation.
Fig. 2(b) shows the detailed structure of each block. Given a latent deformation code sp binding with time, we firstly consider the forward deformation, where the 3D positions [u, v, w] E R of observation space input, and the positions ju' v', w"] E R of 3D canonical space is output. The cause of the invertible property is that after specifying certain coordinate axis, each block predicts the movement along and rotation around the axis in turn, and the process of predicting the deformation is reversible, owing to coordinate split. In the inverse process, each block can infer the rotation around and movement along the axis from [u', , w'] and invert them in turn to recover the original [u, v, w/.
Without loss of generality, let the w-axis to be the chosen axis. With [u, u] fixed, we compute a displacement 8w and update w' as w + 8w. With [w'] fixed, we then compute the rotation R. and translation 8. for [u, v] and update them as [u', e"). Oppositely, for the backward deformation, we apply - R and - in turn recover [u', v', w/] back [u, u, w]. We refer the reader to supplementary material for the inverse process. Therefore, if the network h consists of these invertible blocks, it can represent bijective map as well. At time t,, h(-lyp:) IR3 IR3 maps 3D positions P. of observation space back to 3D canonical correspondences p. where cP denotes the deformation code of i-th frame. In our experiment, we use a Multi-Layer Perceptron (MLP) as the implementation of h, so we design a continuous bijective map Fn for space-time synthesis.
3.2 Deformation Field
Although the proposed deformation representation is a continuous homeomorphic mapping that satisfies the cycle consistency between different frames, it also preserves the surface topology. However, several dynamic scenes (e.g., varying body motion and facial expression) may undergo topology changes. Therefore, combine topology- aware design [47] into our deformation field. 3D positions p. observed at time t; are mapped to topology coordinates qpp,) through a network q: R + R". We regress topology coordinates from an MLP Fq. Then the corresponding coordinate of p, in the canonical hyper-space is represented as:
x [p. {(p))] = [Pr,p; Fal; E 33+m
conditioned on tire-narying deformation si:-
3.3 Implicit Canonical Geometry and Appearance
Inspired by NeRF [43], we consider that sample point E RR3+m in the canonical hyper-space is associated with two properties: density a and color c E R33
Neural SDE. Notes that the object embeds in the (3 + m)-D canonical hyper-space. In this work, represent its geometry the zero-level set of SDF:
S = (x E 3 [d(x) 0}.
(4)
Following NeuS [60], we utilize probability function to calculate the density value (x) based on the estimated signed distance value, which an unbiased and pcclusion-awar approximation. We refer the reader to their paper for more details.
5

---

Implicit rendering network. We utilize a neural renderer Fe as the implicit appearance network. At time 4, takes in 3D canonical coordinate p. its corresponding normal, a canonical view direction as well as a geometry feature vector, then outputs the color of the point, conditioned on time-varying appearance code th;- Specifically, we first compute normal пp Vpd(x) by gradient calculation. Then, the view direction Vp in 3D canonical space can be obtained by transforming the view direction Vi in observation space with the Jacobian matrix (p;):) = 0p/Ôp. of the 3D canonical map p w.r.t p:: Vp = Jp (p.)v,, Except the SDF value, we adopt larger MLP Fx(x) (d(x), z(x)) to compute the embedded geometry feature Zx = z(x) to help the prediction global shadow [67]. Finally, noticing p; is the correspondence of time &,
formulate its color C;
(p. Dp: Vp: Zx, v/,) Fe V Vpd(x), Jp (p.)v, (xx. vb.).
(5)
can be seen that the color of point p: viewed from direction v, depends on the deformation field, canonical representation, deformation code well appearance code combined with time.
3.4
Optimization
N). the optimizable parameters include MLPs (Fh. F Fd, Fe). learnable codes { 1b.). RGB and depth camera intrinsies (Kagb, Keepth) ). well as SE(3) camera pose Ti each time t,. Our target is to design the loss terms to match input masks, color images and depth images. Since we leverage neural implicit functions for representing the geometry, appearance and motion of dynamic object, we divide all constraints into two parts, on free-space points and on surface points:
Given an RGB-D sequence with the masks of interested object ((I, M.),
L Lrusk + >2 + XaLetetth + >x + (Aahear + Ag.ivisible
(6)
space
surface
where (i = ....
.6) are balancing weights.
Constraints free-space. Given ray parameterized r(s) + sv (pass through pixel), sample the implicit radiance field at points lying along this ray to approximate its color and depth:
(*1 C(r) = f (()))(())(()dd) ds, D(r) [ ((s)o
where Sn and sy represent near and far bounds, and T(s) = exp(- (u) du) denotes the accumulated transmittance along the ray. The density and color calculation are described in Sec. 3.3. Then the color and depth reconstruction loss are defined as:
Leolor
Σ [IM(r)(C(r) C(r))l,,
(8)
(Eggh
Σ (((())((r)) - D(r))lI,
Laepth
RK
where R(Kggh. Tr) and R (Katephh Ti) represent the set of rays to RGB and depth camera, respectively. M(r) E {0, 1} the object mask value, while C(r) and D(r) are the observed color and depth value. To focus dynamic object reconstruction. also define mask loss
Luusk (CE(
(10)
where (x) = ((s)o(s) ds is the density accumulation along the ray, and BCE is the binary cross entropy loss.
An Eikonal loss is introduced to regularize d(x) to be signed distance function of p. and it has the following form:
Lieg Σ (IV 1)
(11)
XEX
where x are points sampled in the canonical hyper-space X. In our implementation, to obtain first sample some points P: on the observed free-space and then deform sampled points back X using Eq. 3. We constrain points sampled by uniform and importance sampling strategy.

---

Constraints on surface. Except for the losses on the free-space, we also constrain the property of points lying on the depth images D. We add an SDF loss term:
L Σ
Id(x)]l1-
(12)
P,ED,
To avoid the deformed surface at each time fuses into the canonical space which causes multi-surfaces phenomenon, design a visible loss term to constrain surface:
Lissible
Σ
max(
np
1|Vp12 0),
Vp
(13)
P.ED.
where ) denotes the inner product. The visible loss term is to constrain the angle between the normal vector of the sampled point depth map and the view direction to be larger than 90 degrees, which aims to guide depth points to be visible surface points under the RGB-D camera view.
Experiments
4.1 Experimental Settings
Implementation details. We initialize d(x) such that it approximates a unit sphere [2]. We train our neural networks using the ADAM optimizer [32] with a learning rate 5 x 10-1, We run most of our experiments with x 1 iterations for 12 hours on single NVIDIA A100 40GB GPU. On free-space, sample 2, 048 rays per batch (128 points along each ray). Following NeuS [60]. we first uniformly sample 64 points, and then adopt importance sampling iteratively for 4 times (16 points each iteration). On depth map, we uniformly sample 2, 048 points per batch. For coarse-to-fin to-fine training, we utilize an incremental positional encoding strategy on sampled points, similar with Nerfies [46]. The weights in Eq. 6 are set as: A 0.1, A2 1.0, A3 0.5, A = 0.1,As 0.5, AG 0.1.
For non-rigid object segmentation, we leverage fff-the-shel methods, RVM [38] for human and MiVOS [14] for other objects. Since we assume the region of object is inside unit sphere, we normalize the points back-projected from depth maps first. If the collected sequence implies larger global rotation, we leverage Robust ICP method [68] for per-frame initialization of poses T,
Datasets. To evaluate our NDR and baseline approaches, we scenes from DeepDeform [9] dataset, 7 scenes from KillingFusion [54] dataset, 1 scene from AMA [59] dataset and 11 scenes captured by ourselves. The evaluation data contains 6 classes: human faces, human bodies, domestic animals, plants, toys, and clothes. It includes challenging cases, such as rapid movement, self-rotation motion, topology change and complex shape. DeepDeform [9] dataset is captured by an iPad. Its RGB-D streams are recorded and aligned at a resolution of 640 x 480 and 30 frames per second. Since our NDR does not need any annotated or estimated correspondences, we only leverage RGB-D sequences and camera intrinsics as initialization when evaluating NDR, without scene flow or optical flow data. We choose 6 scenes from the whole dataset, including human bodies, dogs, and clothes. All sequences in KillingFusion [54] dataset were recorded with Kinect vl, also aligned to 640 x 480 resolution. We choose scenes from it, which contain toys and human motions. For evaluation on synthetic data, we use AMA [59] dataset, which contains reconstructed mesh corresponding to each video frame. To construct synthetic depth data, we render meshes to a chosen camera view. In the experiment, we do not utilize any multi-view messages but only monocular RGB-D frames. To increase the data diversity, especially for adding more challenging but routine conditions (e.g.. topology change and complex details), (resolution 480 x 640 at 30 fps). When capturing head data, we ask the person to rotate the face while freely varying expressions. When capturing plant data, record the states of leaf swings.
capture some human head and plant videos with iPhone
Comparison methods. (1) A widely-used classical fusion-based method, DynamicFusion [44]: It is the pioneering work that estimates and utilizes the motion of hierarchical node graph for deforming guidance, and assumes the shape inside canonical TSDF volume. (2) Two recent musion-based methods, DeepDeform [9] and Bozic et al. [8]: These methods utilize the learning- based correspondences to help handle challenging motions. (3) A state-of-the-ar fusion-based method, OcclusionFusion [39]: It computes occlusion-aware 3D motion through neural network for modeling guidance. (4) state-of-the-aa RGB reconstruction method from monocular video, BANMo [65]: models articulated 3D shapes in neural blend skinning and differentiable rendering framework. For
7

---

comparison with RGB-D based methods, we use our re-implementation of DynamicFusion [44] and the results provided by the authors of OcclusionFusion [39].
Input RGB
DynamicFusion OcclusionFusion
Ours
Input RGB
DynamicFusie
OcclusionFusion
Ours
Figure 3: Qualitative comparisons with DynamicFusion [44] and OcclusionFusion [39] dataset (a sequence 600 frames) and KillingFusion [54] dataset (a sequence of 200 frames).
Input RGB
DynamicFusion
DeepDeform
Bozic al. OcclusionFusion
Ours
Ours Normal
Figure 4: Qualitative comparisons with DynamicFusion [44], DeepDeform [9], Bozic et al. [8] and OcclusionFusion [39] on DeepDeform [9] dataset (a sequence of 500 frames). The results of DeepDeform [9] and Bozic al. [8] are copied from the video of Bozic et al. [8].
4.2 Comparisons
RGB-D based methods. For qualitative evaluation, we exhibit some comparisons with Dynamic- Fusion [44] and OcclusionFusion [39] in Fig. 3, also with DeepDeform [9] and Bozic et al. [8] in Fig. 4. Specifically, results of detailed modeling verify that bijective deformation mapping help match photometric correspondences between observed frames. As Fig. 3 shown, our NDR models geometry details while fusion-based methods [44, 39] are casy to form artifacts on the reconstructed surfaces. NDR also achieves considerable reconstruction accuracy on handling rapid movement (Fig. 4),
For quantitative evaluation, cal- culate geometry errors on some test- ing sequences, following previous works [9, 8, 39]. The geometry met- ric is to compare depth values inside the object mask to the reconstructed geometry. The sequences are on be- half of various class objects and cases, including domestic animal (seq. Dog from DeepDeform [9]), rotated body, human-object interaction, general object (seq. Alex, Hat, Frog from KillingFusion [54], separately), and human heads (seq. HumanI, Human2 from our collected dataset). The quantitative results are shown in Tab. 1. We can see that our NDR outperforms previous works [44, 39], owing to jointly optimizing geometry, appearance and motion on a total video. On seq. Alex, the geometry error of OccluionFusion [39] is lower than that of ours. However, NDR can handle topology varying well, as shown in the corresponding qualitative results on the right of Fig, 3.
Metric
Methad
DeepDeform Dog
KillingFaaia
Fasion
Human Head
Ales
Hst.
Frog
Human
Human2
Dynumiccuuioo
Mean
Occlusionf Fusioon
3.61
3.75
6.77
1.61
1.30
1.25
Ours
3.42
4.24
4.93
1.34
1.26
1.08
DynamicFusion
12.35
5.19
6.72
1.7
1.87
1.39
Occlusion
Ours
3.39
4.11
1.33
1.25
1,07
Table 1:
Quantitative results on sequences. The geometry error represents the difference between reconstructed shape and depth values inside mask. All values are mm.

---

Input RGB
BANMo
Ours (only RGB)
Ours (full)
Figure 5: Qualitative comparison with BANMo [65] and ablation study on optimization without depth images on DeepDeform [9] dataset and KillingFusion [54] dataset (both sequences of 200 frames). Each sample is shown in two view directions (the camera view and a novel view).
RGB based method. Fig. 5 exhibits several comparisons with recent RGB based method- BANMo [65]. BANMo takes the RGB sequence as input and optimizes the geometry, appearance and motion based on the precomputed annotations, including the camera pose and optical flow. For fair comparison, we also compare BANMo [65] with our NDR with only RGB supervision, where we provide them with the same camera initialization and frame- wise mask. For the RGB-only situation, both our method and BANMo may make some structural mistakes, such as the human arm in ours and the Snoopy's ears in BANMo. Moreover, compared to our RGB-only results, BANMo suffers more from the local geometry noise, which should be due to the error caused by incorrect precomputed annotations. Meanwhile, our method does not rely on any precomputed annotations and achieves flat results. With the RGB-D sequence as input, our NDR full model performs robust and well in modeling geometry details and rapid motions.
4.3 Robustness on Camera Initialization
In order to systematically analyze the perfor- mance of our camera pose optimization ability, we add an experiment to test. the robustness un- der various degrees of noise on both real and synthetic data. We choose 2 sequences of small rigid motion separately from DeepDeform [9]
10
20
40
60
Moving Joints
2.95
4.58
9.17
11.07
30.28
Sambu Dancer
3.50
5,49
8.18
11.60
13.46
27.02
Table 2: Robustness
camera pose initialization. Values of the first row represent degrees of added Gaussian noises. Other values are mean geometry errors on the sequences, in mm.
dataset (a body with moving joints, 200 frames)
and AMA [59] dataset (a Samba dancer, 175
monocular frames). As Tab. 2, we add Gaussian noises with 5, 10, 20, 40, 60 degrees of standard deviation to initial Euler angles and calculate mean geometry errors (0 denotes without adding noises). The results show that NDR is robust against noisy camera poses to certain extent, owing to its neural implicit representation and abundant optimization with RGB-D messages. If the standard deviation of Gaussian Noises is over 20 degrees, the reconstruction quality will be obviously affected (geometry error is over cm). We refer the reader to supplementary material for qualitative results.
Input RGB
Ours (only depth)
Ours (6D motion)
Ours (full)
2
Figure 6: Ablation studies on optimization without RGB images and using 6D motion instead of the bijective map in the deformation field. The example sequence contains 200 frames.

---

4.4 Ablation Studies
We evaluate components of our NDR regarding their effects on the final reconstruction result.
Depth cues. We evaluate the reconstruction results with only RGB supervision, i.c. removing depth images and only supervised with loss terms Luusk, Lealors Legg As shown in Fig. 5, the reconstruction results with only RGB information are not correct (especially seen from a novel view) since monocular camera scenes exist the ambiguity of depth.
RGB cues. We also evaluate the reconstruction results with only depth supervision, i.e. removing RGB images and color loss term Lookr- As shown in Fig. 6, the reconstructed shapes lack geometrical details as color messages are not used.
Bijective map Fi. To verify the effect of our proposed bijective map Fi (Sec. 3.1), we change it to 6D motion representation in SE(3) space. As shown in Fig. 6, since Fh, can satisfy the cycle consistency strictly, it is less prone to accumulate artifacts and thus performs better in local geometry. In comparison, the inreversible transformation is easy to fail in preserving high-quality surfaces.
4.5 Evaluation of Cycle Consistency
We perform a numerical experiment for cycle consistency evaluation on the whole deformation field. In the experi- ment, we randomly select 3 frames (indexed by i, j. k) as group in a video sequence. Given points on one frame, we calculate the corresponding coordinates on another frame and record this scene flow as f. Then it includes 2 deforming paths from frame to k, based on the direct flow fūx or the composite flow fúz + fjx. To evaluate the cycle consistency, we calculate the Euclidean norm of fy + f - f, as the error. The error smaller, the cycle consistency (invariant deforming path) maintains better. We conduct experiments on a human body rotated in 360 degrees (200 frames) from KillingFusion [54] dataset and a talking head (300 frames) from our captured dataset. In the experiment, we randomly select 1, 000 groups of frames and calculate the mean error on depth points of object surface. Since the topology-aware network is irreversible, we optimize the corresponding positions with fixed network parameters and ADAM optimizer [32]. As a comparison, we also evaluate them on our framework with 6D motion. As Tab. 3 shown, eycle consistency of the whole deformation field among frames is maintained by bijective map Fn quite well, although it might be affected by irreversible topology-awar network.
w/ //
6D motion
Rotated Body
5.29
473.25
Talking Head
4.97
289.41
Table 3: Evaluation of cycle consistency, Each value is the mean error (x10-1) per point in the unit coordinate system.
Conclusion
We have presented NDR, new approach for reconstructing the high-fidelity geometry and motions of dynamic scene from a monocular RGB-D video without any template priors. Other than previous works, NDR integrates observed color and depth into a canonical SDF and radiance field for joint optimization of surface and deformation. For maintaining cycle consistency throughout the whole video, we propose invertible bijective mapping between observation space and canonical space, which fits perfectly with non-rigid motions. To handle topology change, we employ a topology-awan network to model lopology-variant correspondence. On public datasets and our collected dataset, NDR shows a strong empirical performance in modeling different class objects and handling various challenging cases. Negative societal impact and limitation: like many other works with neural implicit representation, our method needs plenty of computation resources and optimization time, which can be a concern for energy resource consumption. We will explore alleviating these in future work.
Acknowledgements. This research was partially supported by the National Natural Science Foundation of China (No.62122071, No. (6662227,33), the Fundamental Research Funds for the Central Universities (No. WK3470000021), and Alibaba Group through Alibaba Innovation Research Program (AIR). The opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies or the government. We thank the authors of OcclusionFuussoo for sharing the fusion results of several RGB-D sequences. We also thank the authors of BANMo for their suggestions on experimental parameter settings. Special thanks to Prof. Weiwei Xu for providing some help.
10

---

References
[1] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH 2005 Papers, pages 408-416. 2005. 3
[2] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2020. 7
[3] Dejan Azinović, Ricardo Martin-Brualla Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022. 2,3
[4] Jens Behrmann. Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning (ICML), pages
573-582. PMLR, 2019. 3
[5] Volker Blanz, Curzio Basso, Tomaso Poggio, and Thomas Vetter. Reanimating faces in images and video. In Computer graphics forum (CGF), volume 22, pages 641-650. Wiley Online Library, 2003.
[6] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th amnual conference on Computer graphics and interactive techniques (SIGGRAPH), pages 187-194, 1999. 1,3
[7] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep it smpl: Automatic estimation of human pose and shape from a single image. In European conference on computer vision (ECCV), pages 561-578. Springer, 2016.
[8] Aljaz Bozic, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, and Matthias NieBner. Neural non-rigid tracking. Advances in Neural Information Processing Systems (NeurIPS), 33:18727-18737, 2020. 7,8
[9] Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner. Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7002-7012, 2020. 4, 9
[10] Christoph Bregler, Aaron Hertzmann, and Henning Biermann. Recovering non non-rigid 3d shape from image streams. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2. pages 690-696. IEEE, 2000.
[11] Chen Cao. Qiming Hou, and Kun Zhou. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions graphics (TOG), 33(4):1-10, 2014. 3
[12] Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Facewarehouse: 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics (TVCG), 20(3):413 425, 2013. 3
[13] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems (NeurlPS), 31, 2018. 3
[14] Ho Kei Cheng, Yu-Wing Tai, and Chi- Chi-Keung Tang. Modular interactive video object segmen- tation: Interaction-to-me -makk propagation and difference-aware fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5559-5568,
2021. 4.7
[15] Yuchao Dai, Hongdong Li, and Mingyi He. A simple prior-free method for non-rigid structure- from-motion factorization. Int. J. Comput. Vis. (IJCV), 101(2):101-122, 2014. 3
[16] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In
ICLR (Poster). OpenReview.net, 2017. 3.5
[17] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart. Learning an animatable detailed
3d face model from in-the-wild images. ACM Transactions on Graphics (TOG), 40(4):1-13,
2021.
11

---

[18]
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5712-5721, 2021. 3
[19]
Mailip-Willians Grassal, Malte Prinzler. Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. Neural head avatars from monocular rgb videos. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022. 3
[20] Kaiwen Guo, Feng Xu, Tao Yu, Xiaoyang Liu, Qionghai Dai, and Yebin Liu. Real-time geometry, albedo. and motion reconstruction using a single rgb-d camera. ACM Transactions on Graphics (TOG), 36(4):1, 2017.
[21] Yudong Guo, Jianfei Cai, Boyi Jiang, Jianmin Zheng, et al. Cnn-based real-time dense face reconstruction with reveseerendered photo-realistic face images. IEEE transactions on pattern analysis and machine intelligence (TPAMI), 41(()) 1294-13777 2018. 3
[22] Yudong Guo, Keyu Chen, Sen Liang. Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In IEÉE/CVF International Conference on Computer Vision (ICCV), pages 5764-5774. IEEE, 2021. 3
[23] Mare Habermann, Weipeng Xu, Michael Zollhoefer, Gerard Pons-Moll, and Christian Theobalt. Livecap: Real-time human performance capture from monocular video. ACM Trunsactions On Graphics (TOG), 38(2):1-17, 2019. 3
[24] Mare Habermann, Weipeng Xu, Michael Zollhofer, Gerard Pons-Moll, and Christian Theobalt. Deepcap: Monocular human performance capture using weak supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5052-5063, 2020. 3
[25] Yang Hong. Bo Peng. Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: real-time nerf-based parametric head model. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[26] Alexandru Eugen Ichim. Sofien Bouaziz, and Mark Pauly, Dynamic 3d avatar creation from hand-held video input. ACM Transactions on Graphics (TOG), 34(4):1-14, 2015.
[27] Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stam- minger. Volumedeform: Real-time volumetric non-rigid reconstruction. In European Conference on Computer Vision (ECCV), pages 362-379. Springer, 2016. 3.3
[28] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559-568, 2011. 2, 3
[29] Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang. Selfrecon: Self reconstruction your digital avatar from monocular video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3,
[30] Chiyu Jiang, Jingwei Huang, Andrea Tagliasacchi, and Leonidas Guibas. Shapeflow: Learn- able deformations among 3d shapes. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 3
[31] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pages 371-386, 2018. 1
[32] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. 7, 10
[33] Filippos Kokkinos and Iasonas Kokkinos. To the point: Correspondence-drivee monocular 3d category reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021. 1
[34] Chen Kong and Simon Lucey. Deep non-rigid structure from motion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1558-1567, 2019.
[35] Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

[36]
Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph. (TOG), 36(6):194-1, 2017.
[37] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-tima simce-time-o view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021. 3
[38] Shanchuan Lin, Linjie Yang. Imran Saleemi, and Soumyadip Sengupta. Robust high-resolution
video matting with temporal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 238-247, 2022. 4,7
[39] Wenbin Lin, Chengwei Zheng. Jun-Hai Yong, and Feng Xu. Occlusionfusion: Occlusion-aware motion estimation for real-time dynamic 3d reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 7.
[40] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEF/CVE International Conference on Computer Vision (ICCV), pages 7708-7717, 2019. 3
[41] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015. 3,
[42] David G Lowe. Distinctive image features from scale-invae invariant keypoints. International journal of computer vision (ICCV), 60(2):91-110, 2004. 3
[43] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European
conference on computer vision (ECCV), pages 405-421. Springer, 2020. 3,5
[44] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pages 343-352, 2015. 1, 2, 3, 4, 7,8
[45] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF international
conference on computer vision (ICCV), pages 5379-5389, 2019. 3
[46] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz. Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5865-5874,
2021. 3, 4,7
[47] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. (TOG), 40(6), 2021. 2, 4,
[48] Despoina Paschalidou, Angelos Katharopoulos, Andreas Geiger, and Sanja Fidler. Neural parts: Learning expressive 3d shape abstractions with invertible neural networks. In Proceedings of the IFEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3204-3215, 2021.3 3
[49] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10318-10327, 2021. 3
[50] Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: modeling and capturing hands and bodies together. ACM Trans. Graph. (TOG), 3666):245:1-24::17 2017. 3
[51] Chris Russell, João Fayad, and Lourdes Agapito. Dense non-rigid structure from motion. In 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, Zurich, Switzerland, October 13-15, 2012, pages 509-516. IEEE Computer Society, 2012. 3
[52] Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. International Journal of Computer Vision (IJCV), 80(1):72-91, 2008. 3
[53] Vikranjit Sidhu, Edgar Tretschk, Vladislav Golyanik, Antonio Agudo, and Christian Theobalt. Neural dense non-rigid structure from motion with latent space constraints. In European
Conference Computer Vision (ECCV), pages 224-222.6 Springer, 2020. 3
13

---

[54] Miroslava Slavcheva, Maximilian Baust, Daniel Cremers, and Slobodan llie. Killingfusion: Non-rigid 3d reconstruction without correspondences. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR), pages 1386-1395, 2017. 7,
10
[55] Miroslava Slavcheva, Maximilian Baust, and Slobodan Ilie. Sobolevfusion: 3d reconstruction of scenes undergoing free non non-rigid motion. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 2646 2655, 2018.
[56] Narayanan Sundaram, Thomas Brox, and Kurt Keutzer. Dense point trajectories by gpu- accelerated large displacement optical flow. In European conference on computer vision
(ECCV), pages 438-451 Springer, 2010. 3
[57] Justus Thies, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 2387-2395, 2016. 3
[58] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12959-12970, 2021. 3
[59] Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popovic, Articulated mesh animation from multi-view silhouettes. ACM Trans. Graph. (TOG), 27(3)997, 2008. 7.9
[60] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advunces in Neural Information P Processing Systems 34: Annual Conference Neural Information Processing Systems (NeurIPS), virtual, pages 27171-27182 2021. 3,5,7
[61] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2566-2576, 2019. 3
[62] Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin, Dushyant Mehta, Hans- Peter Seidel, and Christian Theobalt. Monoperícap: Human performance capture from monocu- lar video. ACM Transactions on Graphics (TOG), 37(2):1-15, 2018.
[63] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from a monocular video. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15980-15989, 2021. 1,3
[64] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[65] Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 2, 3, 4, 7.9
[66] Guandao Yang. Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[67] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems (NeurIPS), 33:2492-2502. 2020.
[68] Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2021. 7
[69] Zhengyou Zhang. Microsoft kinect sensor and its effect. IEEE multimedia, 19(2):410, 2012. 1
[70] Zerong Zheng, Tao Yu, Yebin Liu, and Qionghai Dai. Pamir: Parametric condll-codd
implicit representation for image-based human reconstruction. IEEE transactions pattern analysis and machine intelligence (TPAMI), 2021. 3
[71] Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Mare Stamminger, Matthias Nießner, and Christian Thcobalt. State of the art monocular 3d
14

---

face reconstruction, tracking, and applications. In Computer Graphics Forum (CGF), volume 37, pages 523-550. Wiley Online Library, 2018.
[72] Silvia Zuffi, Angjoo Kanazawa, David W Jacobs, and Michael J Black. 3d menagerie: Modeling the 3d shape and pose of animals. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 6365-6373, 2017. 1,3
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations the end of Sec. and the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We also discuss potential negative societal impacts at the end of Sec. 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL.)? [Yes] We upload them as a part of the supplemental material in submission.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We discuss them Sec, 4, including the number of video frames, optimization time, the number of iterations, batch size, hyperparameters, and so on. We choose the hyperparameters empirically.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We discuss the total amount of compute and the type of resources Sec. 4.
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite related papers or URLs.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We include related URLs in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "m9v1Y24WmXi", "reviewer": "Reviewer_1SCx", "summary": "This paper proposes a template-free RGB-D based 3D scene reconstruction method that handles non-rigid changes with deformation in dynamic scenes.\nThe proposed method, Neural Dynamic Reconstruction (NDR) follows the similar steps with the classic dynamic scene reconstruction methods.\nIt uses a neural implicit function for surface representation, and thus using neural signed distance fields (SDF) and proposes a novel neural invertible deformation network that utilizes a bijective map between frames and a canonical space in order to constrain the non-rigid deformation of observed surfaces.\nThe paper also adds a topology aware network that tackles the well known challenges of dynamic scene reconstruction of the free-form deformation, where dramatic changes of topology (or assumptions of topology) could make handling deformation/motion constraints hard.\nThe experiments show that the proposed method outperforms (in the context of the accuracy of surface reconstruction over time) the other existing monocular dynamic scene reconstruction methods.\n", "questions": "To make the paper more solid and to help the readers understand the article more clearly, I enumerated several questions below. Some questions may focus on how far the use of the proposed method is from the real world applications.\n\n-How much offset the method can handle refining camera poses, and how much error or residual that wrongly initialized camera pose could be handled?\n\n-How the segmentation results affect the overall results? How do the residuals in the boundary of target surfaces affect the quality of the reconstruction results?\n\n-What is the sole inference time (including optimization steps) for each example? Particularly, providing some numbers, settings (settings, pose accuracy, # samples for each example demonstrated in the result sections would be very helpful to understand the correlation between the complexity of the scene/topology/motion.\n\n-Is there any number or visualization that shows how accurately the bijective map is constructed?\n\n-How wrongly handled topology affects the bijective map quality?\n\nRegarding the importance of the major contribution of this paper, rather than the comparison to the 6D motion, extra discussion on the bijective map and its dependency on the actual topology aware network would make the paper more solid.\n\n-The method is obviously very expensive. Is there any potential idea or open discussion to reduce the computational expenses? For example, not completely evaluate cycle consistency all over the frames, making some steps sequentially updatable. etc.\n\n-In Figure 3. the results from DynamicFusion do not look right. What is the main source of the stripe artifacts in the most of results?", "limitations": "As addressed in the conclusion, the major limitation of the method is probably the computational expense.\nAt least adding a small section about the discussion of how to make the method scalable (even with the trade off between quality) or how to make it easier to use the method in the general use cases (end to end scenario) would make the paper more solid.\nFinally, as the major contribution also lies in the use of bijective map and topology aware network, providing more discussion and detailed evaluation (in addition to ablation test) would make the paper even more solid.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I appreciate the research effort from the authors. The results look very impressive and the contribution look very clear. Here are the +/- of the proposed methods and the submitted article.\n\n+Convincing results compared to existing methods\n\n+Provides solutions of each challenging limitation of classic methods (and other latest methods).\n\n+Great idea on the use of bijective map together with topology aware network\n\n-Need more detailed discussion and the evaluation of the bijective map and topology aware network\n\n-Need more detailed discussion of computational expenses and how to handle them.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ch-IGmlL8n5", "reviewer": "Reviewer_NfVp", "summary": "This paper proposed a method for surface reconstruction from a sequential RGBD input. The main strategy for this method is the cycle consistency between canonical and observation space. This approach seems to represent non-rigid deformation. Moreover, to be topology-aware, this paper employed [45]. The results show some comparisons with DynamicFusion[42], OcclusionFusion[58], and so on. Both qualitative and quantitative results show that the method proposed in this paper is better than the methods for the comparison in some parts. The ablation study is also conducted but it is qualitative results only. ", "questions": "As for the canonical space, how is it decided? Moreover, does the choice, deciding, or learning of canonical space affect the performance?\nIn a case of a long sequence with big motion like a 360 rotation, does the cycle consistency correctly work?", "limitations": "If there are some motions that this method cannot apply, I think it should be mentioned as a limitation of this method.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The idea of a cycle consistently between observation and canonical space is reasonable.\n- Topology-awareness improves the results but it is basically the previously proposed method. \n- The organization and writing are good.\n\nWeakness\n- As written in the questions and limitations of this review, I cannot find how to define the canonical space.  Also, I have some concerns in the limitation of this method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aec0hGZTsmH", "reviewer": "Reviewer_BjMr", "summary": "This paper introduces a template-free method to reconstruct a high quality geometry and motion of a dynamic scene from a single RGB-D camera. It proposes a bijective deformation map to preserve the cycle consistency between two frames, thus it doesn’t require any scene or optical flow map. To handle the topology changes, the deformation network is combined with a topological-aware network. Experimental results show that the proposed method overcomes the state-of-the-art RGBD methods, such as DynamicFusion and OcclusionFusion, and RGB methods, such as BANMO.", "questions": "- While the proposed method utilizes depth from Kinect as the ground truth, is there any effect if the captured depth information is noisy? Note that an active depth camera could not be used as the ground truth due to its noisy characteristic?\n- Why don’t use a synthetic dataset for the ablation study to prove the proposed idea?\n", "limitations": "The authors have addressed the limitations in the conclusion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts\n\n- A novel 3D reconstruction network of a dynamic scene from a single RGB-D camera. The combination between topology-aware network and deformation-based network makes the network to model the geometry and motion of a dynamic object.\n- The proposed bijective map is able to preserve the cycle-consistency because it models the points in 3D observation space to the points in 3D canonical space.\n\n# Weaknesses\n\n- There is only a qualitative ablation study. The overall framework is like an extended version of HyperNERF for RGBD videos. Thus, it is essential to perform a quantitative ablation study, especially compared to HyperNERF.\n- Lack of quantitative experimental results. The proposed method only performs a qualitative evaluation for various comparison methods. While qualitative evaluation can be subjective, it is essential to perform quantitative evaluation, especially with BANMo, HyperNERF, VolumeDeform, etc.\n- It is unclear about the cycle-consistency performance of the proposed bijective map. The evaluation method only focuses on a single frame. There should be a way to evaluate the consistency between frames because it is also a part of the proposed contribution.\n- It is recommended to follow the HyperNERF Fig. 8 to show the performance of the proposed method. It is unclear how the topology-aware and bijective map-based deformation networks affect the overall performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_Lw-xdHJm0b", "reviewer": "Reviewer_1XXC", "summary": "The paper tackles the problem of dynamic object reconstruction from a single RGBD video. It solves the problem in a differentiable rendering pipeline and designs a novel 3D warping function that is guaranteed to be invertible. It achieves high-quality reconstruction results on KillingFusion, DeepDeform and iPhone videos. ", "questions": "- For the implementation of invertible wrapping fields in 155-165, if points in the canonical space happen to have the same $w'$ coordinate, their predicted $R_{uv}$ is constrained to be the same. Similarly if points in the observation space have the same $(u, v)$ coordinates, their predicted $\\delta_w$ is constrained to be the same. Does this cause undesirable artifacts, for instance, when dealing with an object containing a flat surface with same $w'$ coordinates?\n- Eq(13) is not clearly explained. What is v_p? What does it mean to force points on depth images to be visible from camera? \n- Fig. 6: the difference between 6D motion fields and full is not obvious. Consider adding more descriptive captions to highlight the difference or choose the example more carefully.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The method is conceptually simple and effective. The authors did well in integrating the best existing solutions (neural surface, canonical hyper-space, and invertible NNs) into a system that not only works well but also stays clean. \n- The bijective warping field is an interesting technical improvement over SE(3) fields. it ensures 3D warping functions to be invertible by design. \n- The results are high-quality and state-of-the-art.\n- The paper is well-written with adequate amount of details. Design choices are also well-motivated. \n\n**Weakness**\n- Some details can be clarified. As there is genuine ambiguity between camera motion and object motion, it is worth explaining the camera pose initialization in more detail and analyze the failure modes. When does Robust ICP fail? For instance, does it fail when the object exhibits rotational motion? How robust is the method to inaccurate camera pose initialization? \n- Some writing could be improved (also see questions). The technical part of sec. 3.1 is not easy to follow possible due to lack of concise equations in l148-165. It is also not obvious what design choice made it invertible. The key idea of coordinate splitting is only mentioned in l150 and the high-level intuition is not conveyed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept