Neural SDF Flow for 3D Reconstruction of Dynamic Scenes
Authors: Wei Mao
Richard Hartley
Mathieu Salzmann
miaomiao Liu
Venue: ICLR 2024 poster (2024)
Keywords: 3D reconstruction
NeRF
dynamic scene
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: In this paper, we tackle the problem of 3D reconstruction of dynamic scenes from multi-view videos. Previous dynamic scene reconstruction works either attempt to model the motion of 3D points in space, which constrains them to handle a single articulated object or require depth maps as input. By contrast, we propose to directly estimate the change of Signed Distance Function (SDF), namely SDF
flow, of the dynamic scene. We show that the SDF flow captures the evolution of the scene surface. We further derive the mathematical relation between the SDF flow and the scene flow, which allows us to calculate the scene flow from the SDF flow analytically by solving linear equations. Our experiments on real-world multi-view video datasets show that our reconstructions are better than those of the state-of-the-art methods. Our code is available at https://github.com/wei-mao-2019/SDFFlow.git.

Published as a conference paper at ICLR 2024
NEURAL SDF FLOW FOR 3D RECONSTRUCTION OF DYNAMIC SCENES
Wei Mao
Richard Hartley
Australian National University
Australian National University & Google richard. Sechard.hartleydeee www.tlyyaaa edu.
wef. .aw.@anu .dw.uuuuduuu...
Mathieu Salzmann
Miaomiao Liu
CVLab, EPFL & SDSC, Switzerland mathieu. zmanneepf1.cn
Australian National University miaomiao. liu@anu. edu, au
ABSTRACT
In this paper, tackle the problem of 3D reconstruction of dynamic scenes from multi-viww videos. Previous dynamic scene reconstruction works either attempt model the motion of 3D points in space, which constrains them to handle a single articulated object or require depth maps as input. By contrast, we propose directly estimate the change of Signed Distance Function (SDF), namely SDF flow, of the dynamic scene. We show that the SDF flow captures the evolution of the scene surface. We further derive the mathematical relation between the SDF flow and the scene flow, which allows us to calculate the scene flow from the SDF flow analytically by solving linear equations. Our experiments on real-world multi-view video datasets show that our reconstructions are better than those of the state-of-the-as methods. Our code is available at http:::://gttt....oo com/ wei mao 2019/ Stt///FFllo.....
INTRODUCTION
The 3D reconstruction of dynamic scene from multi-view videos is very challenging research topic compared to its counterpart for static scenes. Yet, it has many important applications ranging from virtual/augmented reality to videos games, where is required to model changes in the 31) environment, i.e., surface deformations. To handle such deformations, traditional non-rigid structure
from motion methods (Bregler et al. 2000; AKhter et al. 2008) require 2D correspondences across time. While more recent works (Pumarola et al.) 2021 this problem with neural rendering techniques, i.c., NeRF (Mildenhal et al. 2021). almost all those works (Pumarola et al. 202T Park et al. 202 et al.
Park et al., 2021a Li et al. 202 tackle
2021) directly model the movements of 3D points. Despite their great success, these methods mainly focus on synthesizing photo-realistic novel views and cannot obtain good 3D geometry due the shape radiance ambiguity (Zhang al. 2020). To resolve such ambiguity, a commonly used strategy is to parameterize the density with Signed Distance Function (SDF) (Wang et al. 20216 Yariv al. 202Tb.
In this work, we aim for the reconstruction of an unconstrained dynamic scene and for recovering the 3D motion of the scene, i.e., scene flow, using NeRF (Mildenhall al. 202T). Previous works (Yang
et al.
2022 Xu et al. 2021 Wang et al. 2022 Grassal et al. [022] Hong et al. 2022 Guo et al.
2023 on dynamic object reconstruction are restricted to either single articulated object such as
human (Yang et al. 2022 Xu et al. 2021
Wang et al., 2022 Guo et al.) 2023 or a pre-defined
2022 Hong et al. 2022). To handle an unconstrained scene that may contain multiple (non- Jrigid objects, existing NeRF-based methods (Cai al. 2022
template such as human head (Grassal et al.
Shao et al. 2023) either require mapping the 3D points to a higher dimensional space to account for
topology changes (Cai al. 2022 directly predict the SDF at each time step (Shao al.) 2023
thus preventing them to recover the scene flow.
By contrast, propose novel representation, namely SDF flow, which naturally captures the topology changes and allows us to infer the scene flow. Based on the observation that the SDF of any given point in dynamic scene is continuous and almost everywhere smooth with respect to

---

Published as a conference paper at ICLR 2024
time, our SDF flow is defined as the first-order derivative of the SDF with respect to time. Given such an SDF flow, the SDF at any point and at any given time is simply the integral of its SDF flow. We then develop NeRF-based method that, instead of directly predicting the SDF, is trained to estimate the SDF flow, allowing us to extract the 3D scene motion.
To obtain such 3D scene motion, derive the mathematical relationship between the SDF flow and the scene flow. Specifically, we show that the SDF flow of 3D point can be expressed as a linear function of its location, surface normal, and the scene flow. We then demonstrate that, without any supervision, we can analytically compute the scene flow from the SDF flow. Although such scene flow can be noisy due to the reconstruction error. we showease that the linear relationship provides a good regularization on both the scene flow and SDF flow resulting better scene flow estimation and 3D reconstruction. We believe that revealing this relationship will valuable for future research.
Our contributions can be summarized as follows: i) we propose the SDF flow as novel 3D rep- resentation of unconstrained dynamic scenes; ii) we unify the SDF flow and the scene flow with a linear function; iii) With our SDF flow representation, we introduce a NeRF-based pipeline that can reconstruct the 3D geometry of dynamic scene given multi-view videos. We evaluate our method on real world multi-view videos, and our model reconstructs more accurate surfaces than those of the state-of-the dynamic scene reconstruction methods.
RELATED WORK
Neural radiance field for dynamic scenes. Given multi-view images, a neural radiance field (NeRF) (Mildenhall et al. 202T) optimizes continuous function that maps any 3D location to its density and radiance. Such a function has been proven to be effective for novel view synthesis
of static scenes. Recent works (Park al. 2021a Pumarola al. 2021 Li al. 2021 Tretschki
et al.
102T] Du et al. 2021 Wang et al. 202 a
Song et al. [022 Lang et al. 2022 Li et al. 2022]
2023 further extend it to dynamic scenes. Most of them propose to optimize additional functions
that deform the observed points to canonical space (Park et al. 202 la b Pumarola et al. 1
Tretschk et al. 2021] Fang et al. 2022p or over time (Li et al. 2021; Wang et (ai]]2221a Du al,
2021 Li et al. 2023). Despite their good quality of novel view synthesis, these methods cannot
reconstruct faithful 3D scene geometry due to the "shape-radiance ambiguity" (Zhang el al. 2020).
3D reconstruction of dynamic scenes. Traditional non-rigid structure from motion (NRSfM) meth- ods (Bregler et al., 2000; AAkhter et al.] 2008) reconstruct deformable 3D shapes from set of 2D correspondences. Such correspondences
sometimes hard to obtain, making these methods not
suitable to complex real-world scenes. Although some works (Blanz et al. 2003 [Co et al., 2014
Ichim et al. 2015 Thics et al., 2016 Guo et al.) 2018 Gafni et al. 202 Yang et al, 202 la b
Xu et al. 2021: Yang et al. 2022]] Wang et al. [202]] Hong et al. 2022; Grassal et al. 2022 Guo
al.] 2023)
reconstruct non-rigid objects without requiring 2D correspondences, they assume
the reconstructed object to be either articulated or follow certain pre-defined templates such as, CCo
al.
(2013; Li al. (2017); Blanz & Vetter (023). Such assumptions make these methods not
suitable for unconstrained scenes where there may be multiple non-rigid moving objects. Other
works (Newcombe et al., 2013 Innmann et al., 2016; Slaveheva et al.] 2017 Lin et al.) [022) that can handle unconstrained scenes, require depth map inputs. Two existing NeRF-based methods can nonetheless handle unconstrained scenes without depth information: NDR (Cai et al. 2022)
and Tensor4]) (Shao et al. 2023). NDR (Cai et al. 2022) introduces a bijective function that maps the points in observation space to a canonical space. It requires to extend the 3D input to a higher dimensional space to account for the topology changes (Park et al. 2021b). Tensor4D (Shao et al. 2023) represents the dynamic scene with a 4D tensor and further decomposes the tensor into several 2D planes to speed up training and inference. However, since their method directly estimate the SDF at each time step, it cannot recover the scene flow. Our SDF flow naturally captures the smooth deformations of the surface and handles topology changes by design. Given the SDF flow, we can further obtain the scene flow.
OUR APPROACH
In this section, we first brietly introduce the neural radiance field and the SDF-based parameteri- zation of the density (Section B.I.. We then describe our SDF flow to capture the dynamic scenes (Section 3.2). Lastly, we derive the mathematical relationship between the SDF flow and the scene flow (Section
2

---

Published as a conference paper at ICLR 2024
SSD = (1 x sx t)ll2 - 1)2
Ōs
Ot
V B
C
RGGB =
GT||
Figure 1: Pipeline. We propose the SDF flow
ds to capture the deformation of dynamic scene.
3.1 PRELIMINARIES
Neural radiance field (NeRF). The main idea of NeRF (Mildenhall et al. 2021 is to represent static scene as 5D continuous function that maps 3D location æ E R and a viewing direction d E R22 to the RGB radiance e. E R3 and the density R, i.e.,
C, Se d)
where the function is typically implemented Multi-Layer Perceptron (MLP) with O as train- able parameters.
Given such function, for each ray r(r) o + rd shooting from the camera origin E R along direction d, one can obtain the pixel intensity via the volume rendering function
C(r) - I T())(((()))d(((),.c dd)r,
(2)
where Tn and ry the bound of the 3D scene; T(r) - a(r())d is the accumulated opacity; C(r) E R is the rendered color of this ray.
The model can then be trained by minimizing the loss between the rendered color C(r) and the ground-truth C(r), i.e.,
CrGB |(C) - C(r)
Volume rendering with SDE. It has been shown that NeRF may not recover the correct 3D geometry due to shape radiance ambiguity (Zhang et al. 2020 address this issue, a few works have proposed to regularize the density by parameterizing as an SDF (Wang et al. 2021b Yariv et al, 2021. Taking VolSDF (Yariv et al. 202[ as an example, the density a is defined as
v V)(s())))
where T is the Cumulative Distribution Function (CDF) of the Laplace distribution with zero mean and scale B3. Instead of directly estimating the density in Equation the function f outputs the SDF s(x:).
To extend NeRF to dynamic scenes, the most commonly adopted strategy is to jointly optimize an additional function that models the deformation in 3D space such function either maps all observa- tion spaces to canonical one such as Cai et al. (2022) or models the temporal motion of the scene such Li et (202T). In the next section, we propose drastically different representation, i.c., SDF flow, which tries to directly model the change of the dynamic scenes over time.
3.2
SDF FLOW
Let Q C R represent the 3D space occupied by a scene and 0S2 be the scene surface. The Signed
minyeao || yll
if E Ω
Distance Function (SDF) s(xr is defined as s(x) mingean || 7l|2
otherwise.
For any given 3D point in dynamic scene, treat its SDF s(x, t) as function of time t. As shown in Figure / such a function is always continuous and almost everywhere differentiable in the real world scenario. To model the continuous function, propose to estimate its first-order

---

Published as a conference paper at ICLR 2024
derivative 0s öt (SDF flow) as 0s(w.t)
f(r,t) where f is the function to be optimized during
öt
training.
The SDF of point at time t is then the integral of its SDF flow, i.c.
I f(x, t)dt + s(x,
s(x t)
(5)
where s(x, (g) the SDF of the initial time lo. which can be produced by another function as s(a, (a) = fo(.:). We can also obtain its normal as
V S / Val(se n(x,
n(x,
where nxxxtt) (o) = s tn) = 2o)f(())) R
Figure provides overview of our pipeline. Given the SDF s(x, t) of point time t, we can compute its density (s((x,t) t)) using Equation We follow the neural rendering pipeline to further optimize another function that produces the radiance Given the density and the radiance, the volume rendering equation defined in Equation 2 to obtain the final rendered RGB color C(r, t).
Our training loss consists of two parts:
C= LRGB ALSDF
where CROB = Eree,eer|| C(r, C(r,t))li t)]l is the average color loss over all sampled rays P across all times T, and LSDF (eeex,teT([[n](( 1)ll2 - 1)2 is the eikonal constraint of the SDF for all sampled points X across all times. is balancing weight.
Since is often beneficial obtain the 3D correspondences described by the scene flow for many applications downstream tasks, in the next section, we derive the mathematical relation between the proposed SDF flow and the scene flow.
3.3 RELATION BETWEEN SDF FLOW AND SCENE FLOW
Given any point æ on the surface an, we first define its e-neighbor local region on the surface that contains that point, i.e., (x) = {yllyy - x 2| < e, E 82, E 02).
When considering the scenario where the surface is evolving within time At > O. make the following assumption and derive a theorem, which we will illustrate via toy example at the end of the section.
Assumption time period At approaches zero, with sufficiently small e, the motion of a surface point æ's c-neighbor (x(x) is rigid and can be represented as rotation AR E SO(3) and translation AT I23 Thus, we can obtain the corresponding location x' of point æ after At as
(8) Theorem 2 Given 3D location x which is on locally smooth surface deforming smoothly the SDF change of x thus the first order derivative of its SDF with respect time is the negative projection of its scene flow on to its normal. Specifically.
x' = ARx AT
0s
As
lim
01
A1+0
At
Ox T
n(x),
(10)
where at
dz E 3 is the scene flow and nxx) E R is the surface normal at location æ.
We provide the proof in Section A. and illustrate the proof in Figure 3
Combining the Assumption and Theorem 2
we have
0s
01. - x
(II)
4

---

Published as a conference paper at ICLR 2024
n(x)
n(x)
X
As
S0F
2-
Figure 3: 2D example of the relation between the scene flow (x' - x) and the SDF flow As. The solid curve is the surface around The dashed one is the deformed surface after a very short time period.
Figure 2:
Considering an ellipse mov- ing right at constant speed (top), the SDF of the point as a function of time is always differentiable (bottom).
w WER
R 2
as/at
V E
Eq.
Eq.
0.0
0.5
1.0
1.5
2.0
0
angle (m)
(a)
(b)
Figure 4: (a) 2D toy example of an ellipse moving with angular velocity w and velocity v. o is the origin. (b) We plot the SDF flow computed from Equation 9 and [ Here,
the polar coordinate system to represent points on the ellipse. The z-axis represents the points on the ellipse of different polar angles. The SDF flow computed from the scene flow (Equation 11 well matches that from the definition (Equation for any point on the ellipse.
where w = 10 a0, 84, is the angular velocity of the surface (0,, 0y,00, O2 are the 3 rotation angles); 8T
0r ot 0t E R is the velocity. The angular velocity and velocity define the 3D surface motion and thus the scene flow. The detailed derivation provided Section A.2 As will be demonstrated in Section we would also like to compute scene flow directly from SDF flow with the derived relation linking them. To this end, we first transform Equation 11 to
Os a
w
(12)
Øt
v
where
E 12,
a13 x x (xx)) and a = n(x). principle, given the SDF flow of least
points that are moving rigidly, one can solve for the scene flow In practice, to handle the noise, select more than 6 points and obtain the optimal scene flow by minimizing the least-square error (details are in Section A.8.
Toy example. In Figure a), provide 2D toy example verify Equation IT where the initial position of an ellipse as well as its angular velocity w E and velocity v E IR² are given. As shown in Figure b), the SDF flow computed from the scene flow (Equation 1 closely matches that from the definition (Equation
4 EXPERIMENTS
4.1 DATASETS
We evaluate our method quantitativly on the CMU Panoptic dataset (Joo al. 2017) and qualita- tively on the Tensor4D dataset (Shao et al. 2023).
The CMU Panoptic dataset (Joo al. 2017) captures various kinds of scenes including multi-
person activities and human object interactions using multiple RGB(-D) cameras. Each scene is
Note that, there exists exceptions where the solution may not match the true scene motion. We discuss such exceptions in Section A.3
5

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results on the CMU Panoptic dataset. We report the accuracy (top), com- pleteness (middle), and overall (bottom) in millimeter. For each sequence, the accuracy and com- pleteness are averaged across 24 frames, and the "avg" column is the average over all 5 scenes.
acc (mm)
lan3
Haggling.b2
Band1
Pizzal
Cellol
avg
NDR (Cai et al. 2022
21.8
12.5
15.9
17.7
23.1
18.2
Tensor4D (Shao al.
2023
15.4
13.7
17.1
18.3
17.9
16.5
Ours
14.1
8.3
13.0
11.5
12.3
11.8
comp (mm)
lan3
Haggling b2
Band1
Pizzal
Cellol
avg
NDR Cai et al. 2022
20.7
22.8
23.7
25.0
19.5
22.3
Tensor4D Shao al.
2023
22.8
25.3
29.2
27.4
23.5
25.6
Ours
17.5
18.6
21.4
20.6
15.2
18.7
overall (mm)
lan3
Haggling
Band1
Pizzal
Cellol
avg
NDR CCai al.
2022
21.3
17.7
19.8
21.3
21.3
20.3
Tensor4D Shao et al. 2023
19.1
19.5
23.2
22.9
20.7
21.1
Ours
15.8
13.5
17.2
16.1
13.7
15.2
captured by 10 RGB-D and hundreds of RGB cameras. this paper, we only use the images from 10 RGB-D cameras. We obtain the ground-truth point cloud at each time step by registering the depth maps taken from those cameras using the provided camera poses and intrinsics. We select challenging clips: "Ian3", "Haggling_be "Bandl". "Pizzal", and "Cellol". Our selected sequences cover activities like multi-person socializing ("Haggling b2"), band with multiple persons and musical instruments ("Bandl") and mother playing with little child ("Ian3"). Each clip contains 24 frames from 10 camera view thus 240 images. The resolution of the image is 1920 x 1080. Since our goal is 3D reconstruction, we use 10 camera views for training and only evaluate the meshes.
The Tensor4D dataset (Shao et al., 2023 is captured by sparse-view system with RGB cameras. contains single person performing different actions like thumbs-up and waving hands. We select the 3 sample sequences provided on their official github page: "Boxing_v12", "Dance_v4" and "Thumbsup_v4" The "Boxing_vi v12" sequence is captured by 12 cameras in a circle surrounding the human. The "Dance v4", and "Thumbsup. v4" sequences are taken by 4 forward-facine cameras. For each sequence, we select a clip of 12 frames. The image resultion is 1024 x 1024. Since no ground-truth geometry is available, only provide qualitative comparison this dataset.
4.2 METRICS, BASELINES & IMPLEMENTATION
Metrics. We follow the standard evaluation protocol in the multi-view stereo literature Yao et al.) 2018) to evaluate our method with accuracy, completeness and overall distance. Specifically, given the ground-truth point cloud P, and the predicted point cloud P, the accuracy and completeness defined as
1
Acc
Σ
min Ip -pl1a PEP
(13)
P
pEp
Comp
1
Σ
min
[p -pll2-
(14)
P
PEP
pep
The overall distance is the average of the accuracy and completeness.
Baselines. We compare our method with two recent NeRF-based dynamic scene reconstruction methods: NDR (Cai et al. 2022) and Tensor4D (Shao et al. 2023). NDR (Cai et al. 2022) attempts to find bijective mapping between the observation space and the canonical space. Tensor4D (Shao
jet al., 2023) decomposes the 4D space into several 2D planes to speed up the model training. For both methods, their official implementation.

---

Published as a conference paper at ICLR 2024
NDR
Tensor4D
Ours
(a) "lan3"
NDR
Tensor4D
Ours
(b) "Haggling b2"
Figure 5: Qualitative results the CMU Panoptic dataset. Best viewed screen.
Implementation details. We implement our method using Pytorch (Pazkee et al. 2017) and use the Adam optimizer (Kingma Ba 2014) to train our model with a 0.0005 learning rate, The batch size is set to 1024. We use the second-order Runge- Kunge-Kutta Kutta method to solve the integration in Equation 5 We train our model for 2000 epochs, which takes around 7 days on NVIDIA 4090 GPUs for ten 1920 x 1080 videos of 24 frames. The rendering of one ray takes around 1.5 ms. The balancing weight À is set to 0.1. During testing, for all baselines and our method, we construct a 3D grid of resolution 512 512 512 and query the SDF of each voxel in this grid from the trained model. We then use the marching cube algorithm (Lorensen & Cline 1998) to obtain the mesh. We uniformly sample 10000 points from the 3D mesh and compare them to the ground-truth point clouds.

---

Published as a conference paper at ICLR 2024
NDR
Tensor4D
Ours
Figure 6: Qualitative results on 3 samples of the Tensor4D dataset.
4.3 RESULTS
Quantitative results. We provide quantitative results on the CMU Panoptic dataset (Joo al. 2017) in Table I Our method consistently outperforms the baselines for all scenes in all metrics. For each scene, the reported accuracy (top), completeness (middle), and overall (bottom) averaged across all frames, and we also report the average distance over all scenes (last column).
the CMU Panopic dataset. We compare results those of the baselines on the CMU Panopic dataset in Figure [ Here, we show the reconstruction results of different time steps for 2 scenes: "Ian3" and "Haggling.br
Qualitative comparisons
As highlighted by the red box, the baselines (second and third columns) sometimes reconstruct overly smooth surfaces or even produce non-existing geometry. By contrast, the reconstructed meshes from our method are sharper with more details. More qualitative results on this dataset are provided in Section A.4 and the supplementary video.
Qualitative comparisons on the Tensor4D dataset. The results are shown in Figure C Although NDR (Cai at. 2022) performs well on "Dance v4" (middle), sometimes fails on "Thumbsup v4" (right) and its reconstructions of "Boxing_ve -12ni (left) are over smoothed. As for Tensor4D (Shao) ct al. 2023), ) although it can sometimes reconstruct more details, such as the face in 'Thumbsup all of its results share similar artifacts which we conjecture are due to the tensor decomposition. Our method performs comparably to the baselines, with fewer artifacts.
Table 2: Optical flow evaluation on the CMU Panoptic dataset. We report the average end-point-
(EPR) in pixel.
EPE (pixel)
Ian3
Haggling.b2
Bandl
Pizzal
Cello1
avg
NDR (Cai et al. 2022
6.86
4.43
1.66
3.11
2.41
3.69
Ours
3.49
2.97
1.18
1.18
1.34
2.04
8

---

Published as a conference paper at ICLR 2024
Figure 7: Scene flow of toy example. We show the images two different time steps (the first and third column) and their scene flow (the second and forth column). Different colors represent different scene flows.
(b) eseudo-ground truth Figure 8: The comparison of optical flow from NDR Cai et al. 2022 and SDF flow. The
(a) image
NDR
(d) ours
optical flow computed using our SDF flow better matches the ground truth.
Scene flow from SDF flow. We further demonstrate that using our SDF flow lets us derive the scene flow, i.e. the angular velocity and velocity. We first show the derived scene flow of toy example where two rigid objects are moving in the scene. For any surface point, we use the surrounding surface points to compute the scene flow. The results shown in Figure 7 Note that, here, do not have any prior knowledge of the number of rigid objects in the scene. As shown in the second and fourth column, the scene flow clearly distinguishes the two moving objects. We provide quantitative evaluation on the scene flow of this synthetic sequence in Section A.9
We also compare our scene flow that of NDR (Cai al. 2022) on the real-world sequences of the CMU Panopic dataset. Since the ground-truth scene flow is not available, we evaluate the optical flow instead and use the optical flow estimated from RAFT (Teed Deng 2020) as sgeudo-ground truth. Specifically, we project our scene flow to the image plane to obtain the corresponding optical flow. For NDR (Cai et al., 2022). we use the bijective mapping to obtain the scene flow. We report the average sud-point-erro of the optical flow. The evaluation results is shown in Table 2 As also evidenced in Figure S our optical flow better matches the ground truth. We also visualize the 3D scene flow in Section A.5 and the supplementary video. Our scene flow better reflects the real scene motion.
5 CONLUSION
In this paper, we have proposed to exploit the SDF flow to represent dynamic scene with the first- order derivative of its SDF with respect time. Our SDF flow naturally captures the deformations of the scene surface. We have designed a NeRF-based pipeline using our SDF flow to reconstruct a dynamic scene from multi-view videos. Our experiments show that our method yields state-of- the-art performance. We have further derived a mathematical relation between the SDF flow and the scene flow. Such relation allows us to calculate the scene flow from the SDF flow analytically by solving linear equations. We have demonstrated that the resulting scene flow correctly reflects the real motion. the future, we would like to explore the potential of applying our method to monocular videos.
ACKNOWLEDGEMENTS
This research supported in part by the Australia Research Council DECRA Fellowship (DE180100628) and ARC Discovery Grant (DP200102274). The authors would like to thank NVIDIA for the donated GPU (Titan V).

---

Published as a conference paper at ICLR 2024
REFERENCES
Ijaz Akhter, Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Nonrigid structure from motion in trajectory space, Advances in neural information processing systems, 21, 2008.
Volker Blanz and Thomas Vetter. morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 157-164. 2023.
Volker Blanz, Curzio Basso, Tomaso Poggio, and Thomas Vetter. Reanimating faces images and video. In Computer graphics forum, volume 22, pp. 641-650 Wiley Online Library, 2003.
Christoph Bregler, Aaron Hertzmann, and Henning Biermann. Recovering non-rigid 3d shape from image streams. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), volume 2, pp. 690-696. IEFF, 2000.
Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang. and Juyong Zhang. Neural surface re- construction of dynamic scenes with monoeular rgb-d camera. Advances in Neural Information Processing Systems, 35:967-981, 2022. 2022.
Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. Pacewarehouse: 3d facial expression database for visual computing. IEEE Transactions Visualization and Computer Graphics, 20(3):413-425, 2013.
Chen Cao, Qiming Hou, and Kun Zhou. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on graphics (TOG), 33(4):1-10, 2014.
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 14304-14314. IEEE Computer Society, 2021.
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Pupers, pp. 1-9, 2022.
Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8649-8658, 2021.
Philip- Williamm Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Jus- tus Thies. Neural head avatars from monocular rgb videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18653-18664, 2022.
Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Vid2avatar: 3d avatar recon- struction from videos the wild via self-supervised scene decomposition. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12858-12868, 2023.
Yudong Guo, Jianfei Cai, Boyi Jiang, Jianmin Zheng, et al. Cnn-based real-time dense face recon- struction with inverse-rendered photo-realistic face images. IEEE transactions on pattern analysis and machine intelligence, 41(6): ((66:12944 1221-13943777 2018.
Yang Hong, Bo Peng. Haiyao Xiao, Ligang Liu, and Juyong Zhang. Headnerf: real-time nerf- based parametric head model. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 20374-20384, 2022.
Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. Dynamic 3d avatar creation from hand- held video input. ACM Transactions on Graphies (ToG), 34(4):1-14, 2015.
Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Mare Stamminger. Volumedeform: Real-time volumetric non-rigid reconstruction. In Computer Vision-ECCo 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pp. 362-379. Springer, 2016.
10

---

Published as a conference paper at ICLR 2024
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Scott Godisart, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panop- tic studio: A massively multiview system for social interaction capture. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4d scans. ACM Trans. Graph., 1,
Tianye Li. Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5521-5531, 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6498-6508, 2021.
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neu- ral dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273-4284, 2023.
Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, and Feng Xu. Occlusionfusion: Occlusion-aware motion estimation for real-time dynamic 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1736-1744 2022.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pp. 347-353. 1998.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-185 2021.
Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and track- ing of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 343-352, 2015.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 5865-5874, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, and Steven M Seitz. Hypernerf: nigher-dimensione representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 4016) 2021b.
Ricardo
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Ten- sor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16632-16642. 2023.
Miroslava Slavcheva, Maximilian Baust, Daniel Cremers, and Slobodan Ilie. Killingfusion: Non- rigid 3d reconstruction without correspondences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386-1395, 2017.
11

---

Published as a conference paper at ICLR 2024
Liangchen Song, Xuan Gong, Benjamin Planche, Meng Zheng, David Doermann, Junsong Yuan, Terrence Chen, and Ziyan Wu. Pref: Predictability regularized neural motion fields. In European Conference Computer Vision, pp. 664-681. Springer, 2022.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer ESsonnEECV220 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2face: Real-time face capture and reenactment of rgh videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2387-2395, 2016.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 12959-12970, 2021.
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105... 059944, 2021a.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171 27118,, 2021b.
Shaofei Wang. Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah: Animatable volume rendering of articulated human sdfs. In European conference on computer vision, pp. 1-19. Springer, 2022.
Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for render- ing and temporal reconstruction of humans in motion. Advances in Neural Information Processing Systems, 11445-559666 2021.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William Freeman, and Ce Liu. Lasr: Learning articulated shape reconstruction from monocular video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15980-15989, 2021a.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan. Viser: Video-specific surface embeddings for articulated 3d shape reconstruction. Advances in Neural Information Processing Systems, 34: 19326- 2021b.
Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2863-2873, 2022.
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European conference computer vision (ECCV), pp. 767-783, 2018.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805 2021.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
A APPENDIX
A.1 PROOF OF THEOREM
Proof. Let us first prove the absolute value of SDF flow,
òs
As
lim
(15)
A1-0
A/
22
lim
(16)
A1-00
AI
12

---

Published as a conference paper at ICLR 2024
where x" is the closest point on the evolved surface after time At (At 0).
Since the surface locally smooth around æ and its deformation is also smooth, as At approaches zeros, x" is infinitely close to the corresponding point (denoted æ') of the evolved surface
(as shown in Figure Thus, we can regard æ" as on the tangent plane to the evolved surface at a'.
xll2
<x' x) "((x())
lim A1-0
lim -0
(17)
At
At
Ox T
n(x))),
(18)
where nxx" is the normal of w' on the evolved surface and limas- n((x') = n(x).
As to the sign of SDF flow, it is easy to see that if the angle between o' - x and r2(x") is obtuse, the SDF change is positive, otherwise it is negative.
We provide another form of the Theorem
2 wishing to help the readers better understand it.
Theorem Let V be a smooth surface in R3 with a a point on the surface and n(xx) the normal to the surface x. Let (t) be smooth curve Y -220 RR be smooth path in R& such that ((0) = x. Let s(t) be the signed distance from the curve to the surface. Then
s' s() = (t)² n(()..
(19)
This theorem is effectively saying that at time / - O, the point (tt) is moving away from the surface at speed equal to the projection of the velocity > (0) onto the normal direction n(x). The proof can be made rigorous by invoking the implicit function theorem, which states that in chosen local coordinates, the surface can be expressed graph, locally around
A.2 DETAILED DERIVATION OF THE RELATION BETWEEN SCENE FLOW AND SDF FLOW
Let us start from Equation when Al approaches zero, the rotation angles are very small. Accord- ing Rodrigues' rotation formula, we can approximate the rotation as
AR = I AK,
(20)
0
A
AO,
where AK-
A0
0
A0,
(A0.. Ayy. AB, are the Euler angles). We can then com-
-
A0
0
pute the scene flow
dx
x' x
lim
(21)
Øt
Ar-10
At
AKr + AT
lim
(22)
AC-0
At
U
(23)
where E R is the angular velocity and into Equation T0 we have
R3 is the velocity. Substituting the above equation
0s Øt
- (w x x x)
(24)
A.3 EXCEPTION
Given the SDF flow of 6 points on a surface that moves smoothly and rigidly, one can obtain the scene flow of the surface by solving a linear equation v
d = A
(25)
13

---

Published as a conference paper at ICLR 2024
where d R is the SDF flow of those 6 points. A R6x6 whose normals and locations of those points. To the best of our knowledge, the only exception when we cannot obtain the correct scene flow is that when the scene motion does not lead to any SDF changes to any points on the surface, thus, = 0. In this case, there are many solutions to the linear equation. For example, plane moving along any of its tangent direction its centre. For the former, any velocity on the tangent plane could be the solution. For the latter, any angular velocity by the ball centre will work. Note that, the linear relation between scene flow and SDF flow defined in Equation I1 still holds on these exceptions. In fact, as long as the scene is piece-wise rigid and scene motion is smooth, Equation holds.
computed from the
ball rotating around
A.4 ADDITIONAL QUALITATIVE RESULTS ON THE CMU PANOPTIC DATASET
We provide qualitative results the rest sequences of the CMU Panoptic dataset in Figure 10
and | below.
NDR
Tensor4D
Ours
Figure 9: Qualitative results on "Pizzal" sequence of the CMU Panoptic dataset.
14

---

Published as a conference paper at ICLR 2024
NDR
Tensor4D
Ours
Figure 10: Qualitative results on "Bandl" sequence of the CMU Panoptic dataset.
NDR
Tensor4D
Ours
Figure 11: Qualitative results on "Cellol" sequence of the CMU Panoptic dataset.
A.5 MORE QUALITATIVE RESULTS ON SCENE FLOW
We visualize the scene flow for all 5 sequences of the CMU Panoptic dataset in Figure L 1 and [6] below.
3 14
15

---

Published as a conference paper at ICLR 2024
NDR
Ours
Figure 12: Scene flow of the "Ian3" sequence. We show the reconstructed meshes well as the scene flow (blue arrows) at different time steps. Note that, the blue arrows are scaled for better visualization. The scene flow from NDR are wrong while the scene flow from SDF flow correctly reflects the true 3D motion.
NDR
Ours
Figure 13: Scene flow on "Hanggling.br b2" of the CMU Panoptic dataset.
A.6 MULTI-VIEW IMAGES
In Figure 17 show the 10 images from different camera views of "Ian3". The cameras are fixed for all the 5 sequences of the CMU Panoptic dataset cross all time steps. It is a challenging setup due to the large view changes.
16

---

Published as a conference paper at ICLR 2024
NDR
Ours
Figure 14: Scene flow on "Pizzal" of the CMU Panoptic dataset.
NDR
Ours
Figure 15: Scene flow on "Band1" of the CMU Panoptic dataset.
A.7 SCENE FLOW REGULARIZATION
We believe that revealing the linear relationship between the SDF flow and scene flow will be valu- able for modeling dynamic scenes. We evidence this by the toy example shown in Figure / In particular, we train model (named "Ours w/ reg") that outputs not only the SDF flow but also the scene flow
as(m, 1) Øt w(x v(x, fxx
(26)
Given the estimated SDF flow and scene flow, we can then apply several priors.
SDF flow and scene flow consistency prior. According to Equation for any point the surface, we have
t) + as(x, t)
Lst = ([ x x v(x, rz(x,
112
(27)
Ot
17

---

Published as a conference paper at ICLR 2024
NDR
Ours
Figure 16: Scene flow on "Cellol" of the CMU Panoptic dataset.
Piece-wise rigid prior. We also encourage the scene flow of neighbouring points to be similar.
RRggid
w(x, w t)] + (v v(y,t)]
(28)
(e)
Optical flow loss. For any point æ at frame t1, we can obtain its corresponding point at next frame 12 as
x <'(x)= = =/ w(x
+ v( t)dt
(29)
Given the 3D correspondences, we can use the camera parameters and the standard volume rendering techniques to compute the optical flow for any pixel r the image.
f(r) ["" 1 (())((()))(((((((()) II(x))dr,
(30)
where I1(-) is the prospective camera project. The optical flow loss then defined CoF = [f((r) f(r)
(31)
where f(r) the ground truth optical flow.
The final loss is the weighted sum of all losses
L = LRGB + ALSDE + AsECsF + ARIgid CRigid + Xop CoF
(32)
We report the reconstruction results in Table The scene flow regularization helps to improve the reconstruction quality.
A.8 SCENE FLOW FROM SDF FLOW
Given a set of points sampled uniformly from a reconstructed mesh, for any point æ in this set, first obtain its K nearest neighbours {y where y; P. For every neighbouring points yi, we query its surface normal (3:) and SDF flow d(y:) from the MLPs. According to Equation 17,
obtain matrix A E R Kx6 where the i-th row A; [( nyy))) ( and the SDF flow vector d E R² where d, d(y:). We can then solve such problem as
w
w
arg min v ||d
A
(33)
2
-A
A)
ATd
(34)
18

---

Published as a conference paper at ICLR 2024
Figure 17: Multi-view images used in our experiments. It is a challenging setup due to the small overlap between different views. All 5 sequences of the CMU Panoptic dataset use the same camera setup. The images are zoomed and cropped to better show the content.
where w*, 0² is the optimal angular velocity and velocity.
In practice, we uniformly sample around 2000 points from a reconstructed mesh and choose around 200 neighbouring points to compute the scene flow of one surface point. To further study the influ- ence the number of neighbouring point, we compare the scene flow snd-point-error(ERR with respect to the number of neighbouring points in Table on the synthetic sequence.
mm)
A.9 SCENE FLOW EVALUATION
To directly evaluate the scene flow, we use the synthetic sequence shown Figure 7 where we have the ground truth scene flow across all frames. The evaluation results are shown in Table We compare the results of three models: NDR (Cai al. 2022), Ours, Ours with regularization which will introduce in the next section. Our model without any explicitly scene flow regularization performs better than NDR (Cai et al. 2022). With further regularization, the scene flow can be drastically improved. We also evaluate the long-term correspondences up frames. In particular,
19

---

Published as a conference paper at ICLR 2024
Table 3: Reconstruction results on the synthetic sequence. We compare the results from NDR (Cai et al. 2022), our model with only the SDF flow ("Ours"), and with scene flow regu- larization ("Ours reg").
acc
comp
overall
NDR (Cai et al. 2022)
33.5
25.3
29.4
Ours
31.8
23.3
27.6
Ours w/ reg
27.2
19.1
23.2
Table Scene flow error v.s. number of neighbouring points.
K
50
100
200
500
1000
EPE (mm)
13.7
12.6
11.7
11.3
12.0
for every surface point in frame, we compute the scene flow not only to the next frame but also to further future frames. The scene flow error increases with respect to frame gaps.
Table Scene flow evaluation on the synthetic data. We report the average end-point-error (EPE) in millimeter. We evaluate the long-term correspondences up to 5 frames. In particular, for every surface point in a frame, we compute the scene flow to the future 1, 2, and frames.
EPE (mm)
+1
+2
+5
NDR (Cai et al., 2022)
16.8
34.8
90.2
Ours
11.7
24.7
71.4
Ours w/ reg
5.8
13.0
51.9
A.10 LIMITATIONS
Similar to all other multi-view reconstruction works, the reconstruction quality highly rely on the number of views. Another limitation of our SDF flow the time complexity. To solve for the integral (Equation 5 for any query point, the number of function evaluation grows with respect to time. We aim address those limitations in the future work. now have explicit relation between the scene geometry and scene motion, we may be able to achieve better geometry with extra regularization from motion such as optical flow. For the second limitation, potential solution could be that instead of integrating from the initial time step, can integrate within a fixed time window.
the first limitation, since we
A.11 SDF V.S. SDF FLOW
We compare the reconstruction results of our model directly output the SDF and SDF flow. During training we set aside several frames to evaluate the interpolation ability of both representations. In particular, we train the model on {1, 4, 7, 10)-th frames and the evaluation results on those frames are refered to as reconstruction and those on 3, 3, 5, 6, 8, 9)-th frames are referred to as interpolation. Our model with SDF flow representation consistently outperforms that with SDF representation especially at the unseen frames (interpolation).
20

---

Published as a conference paper at ICLR 2024
Table 6: SDF v.s. SDF flow with the same model on the CMU Panoptic dataset.
Reconstruction
Interpolation
acc
comp
overall
acc
comp
overall
Ours w/ SDF
12.4
19.0
15.7
13.0
20.8
16.9
Ours w/ SDF flow
11.6
18.9
15.2
11.5
18.9
15.2
21


Reviews:
[{"review_id": "eeJODEwEZS", "reviewer": "Reviewer_nuCZ", "summary": "This paper proposes an algorithm for implicit surface reconstruction in dynamic scenes. The key is to bridge the SDF flow and sceneflow by assuming the two hypothesis. Also, typically this paper requires RGBD data which is to sample points nearby surface and to make this algorithm feasible. Accordingly, this paper could be understood as 4D surface tracker.", "questions": "Please refer to the question above. Especially for W-3, if there are some things that I misunderstood, please let me know.\n\nOverall, I am quite positive to this paper. Depending on the rebuttal, let me change my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper brilliantly bridge the concept of sceneflow and the proposed SDF flow. Based on the two assumptions, one for rigidity and the other for the linearlization (?), the proposed SDF flow is differentiably convertible into the sceneflow, which enables the network to encode sceneflow from RGBD frames. As far as my understanding, this is the pioneering work. For experiment parts, comparison with NDR (Neurips 2022) is reasonable.", "weaknesses": "W-1. Low fidelity results\n\nDespite the novel idea, the reconstruction quality is far below my expectation. Even though the results from NDR (Neurips 2022) are also not that good enough, there are not that much dramatic change after the authors applied the proposed SDF flow.\n\nOf course, I can see the sceneflow visualization in Fig8 of the manuscript, the reconstruction quality is not really good enough. Moreover, __we cannot judge whether the quality of the sceneflow is correct or not.__ \n\nW-2. Validity on the assumption 2.\n\nCan the authors further elaborate the validity of the proposed 2nd assumption? What geometric insight reside within this hypothesis?\n\n_\"[Assumption 2] As the time period Δt approaches zero, the absolute SDF change |Δs| of a surface point x equals the distance from x to the tangent plane to the evolved surface at the corresponding point x′ and the sign of Δs is determined by the angle between that tangent plane’s normal and the scene flow (as shown in Figure 3).\"_\n\nW-3. Necessity of Sec 3.3\n\nWhile training the network, does the understanding of sec 3.3 is needed? While this paper proposes an algorithm for bridging the SDF flow and sceneflow, there are not much material or experiments that clearly demonstrate the accuracy of the sceneflow. Moreover, the proposed assumptions are not used when training the methods. Accordingly, it is quite confusing me to understand the precise pipeline of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vbh5NIbjtJ", "reviewer": "Reviewer_jfGg", "summary": "This paper proposes to predict the first-order derivative of a signed distance function (SDF) at a point, termed as SDFflow. By using SDFlow, the dynamic recontruction to recover a SDF at time t can be treated as space-time integration of the SDFflow from time t_0 to t and its starting SDF at time t_0. The authors further demonstrate how locally rigid scene-flow can be recovered least-square optimization given the locally rigid assumption for small motions. The results demonstrate the method is outperforming previous baselines using alternative shape and flow representation.", "questions": "1. Though the method is outperforming previous methods by big margin in the qualitative comparisons, the difference in qualitative comparison is less clear to me from all figures in the paper. In particular compared to Tensor4D, I can see each method wins in different level of details being captured. The paper currently summarizes it as \"all of its results share similar artifacts which we conjecture are due to the tensor decomposition. Our method performs comparably to the baselines, with fewer artifacts.\" I am not sure I can capture this from the current presentation. Will love to know a concrete summary of the areas where the model increase performance best. \n2. I did not see any discussions about the limitations of existing methods or representations. As all current multi-view dynamic reconstruction work, I assume this work will face the same limitations in  large motion (which will also break the linear motion relation between scene flow and sdf flow) and number of views. But I'd like to some more technical insights of potential downside using SDF flow in some scenarios compared to other representation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The approach is technically sound. It is kinda intuitive that this representation should work. \n* From the quantitative comparison, it is clear the method is outperforming previous methods by a relatively large margin. \n* The author also provides connection of SDF flow to scene flow with a math derivation. It is helpful for readers to understand its connection if without background in optimization based motion estimation.", "weaknesses": "* The flow evaluation in the paper is very weak. The method only compares to NDR on 2D project error using RAFT as pesudo ground truth. This should not be hard to achieve if using a synthetic dataset, if using the rigid moving toy example being shown in the paper. \n* The flow estimation also lacks details. Solving the least-square optimization requires sample multiple points, which the author say \"we select more than 6 points\" and sometimes it still be ill-conditioned depending on the property of the sampled points, but I don't see any discussion related to this and how the results are generated. \n* The main contribution of this paper is the SDFlow and a claim that it is beating alternative representation (SDF with warping field for example). This can be much clearer if the authors can provide more ablations studies on the contrast of the two representations. Two small experiments they author can simply do is to 1) have the network predict the integral of flow in eq. (7), or 2) predict the time dependent SDF s(x, t), all with fixed hyper-parameters. The contrasts in this ablation can reflect the performance difference in predicting SDF flow. Though the authors provide comparisons to alternative papers that the other papers compound too many other terms, and I don't think I can get the insights of why SDF flow works better here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "USJUHH8iOO", "reviewer": "Reviewer_hwJN", "summary": "This paper proposes a novel representation, namely SDF Flow, for solving multi-view dynamic scene reconstruction by representing the dynamic scene as a 4-D SDF field and modelling its derivative w.r.t. time instead of the SDF value. The proposed representation has several nice properties and can be used to compute the scene flow analytically. Experimental results on public datasets have shown competitive reconstruction quality compared to state-of-the-art, and better scene flow estimation.", "questions": "Please see weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed representation is novel. I’ve never seen similar representations before. The idea is clear and elegant but effective. It has the potential to open up a new paradigm for dynamic scene reconstruction and provide great insight for the community.\n\n2. The authors also revealed the relationship between SDF flow and scene flow through mathematical derivation, and have shown that the scene motion can be computed analytically by solving linear equations.\n\n3. Quantitative and qualitative experiments have shown the proposed method could achieve promising reconstruction quality. The scene flow estimation result also looks promising.", "weaknesses": "1. In Sec 3.3, the authors made two assumptions when deriving the computation of scene motion from SDF flow, which is justified by a 2D toy example. However, it would be great to have a more in-depth theoretical analysis and experiments to understand its convergence behaviour. Especially for assumption 2, the difference between assumed $\\Delta_S$ and real value looks quite big.\n\n2. The optimisation takes too long, which limits its practical usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5qpEHunzq", "reviewer": "Reviewer_uMrz", "summary": "The paper tackles dynamic surface reconstruction and scene-flow estimation from multi-view RGB video with known camera parameters. It uses a global coordinate-based MLP that outputs the SDF in canonical space (t=0), which then gets converted to density via VolSDF's formula, enabling NeRF-style rendering. Training uses an RGB reconstruction loss and an Eikonal regularizer for the SDF. The method departs from prior dynamic NeRFs in its deformation model: a separate MLP is trained to predict the temporal change (derivative) of the SDF at any point in space. These changes get integrated with Runge-Kutta over time to obtain the time-dependent SDF. Furthermore, the paper shows how this SDF flow can be converted into scene flow. Experiments show that the geometry is either less noisy or more detailed than prior methods on the CMU Panoptic dataset and another existing dataset.", "questions": "I have listed my concerns in Weaknesses. The concerns under \"Minor\" don't need to be addressed in a rebuttal, except that clearing up [Input Assumptions] would help me. For all others, I'd appreciate a response. In particular, the most important concerns I have are [Topology Change] and [Scene Flow Evaluation]. If these are not addressed in a rebuttal in some form, I am against accepting the paper. I would still want the rebuttal to address the other major concerns to feel like I have a decent grasp of the submission.\n\nOverall, I lean towards reject. Even though the results are not that impressive, the paper shows nice technical contributions. If the rebuttal addresses my two main concerns well, I could increase my score to acceptable.\n\n=====\n\nPost-rebuttal justification: The rebuttal addressed all concerns very well.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Expanding the dynamic NeRF field in the direction of reconstruction/geometry is worthwhile in my opinion, as it enables more applications than novel view synthesis alone would. Correspondences are also interesting for the same reason. Thus, the problem setting is a big strength of the paper.\n\nThe method's SDF flow parametrization is novel, as far as I know. It seems generic enough to be potentially valuable for problem settings very different from the setting in the submission.\n\nThe illustrations are done well and the paper in general is written very well.\n\nThe method outperforms prior methods quantitatively. Qualitatively, it is better in at least one aspect compared to each prior method, while on par in the other aspects, and hence overall better.", "weaknesses": "*** Method:\n\n[Tangential Motion] The paper should more clearly state that tangential motion causes issues when converting a time-dependent SDF into scene flow.\n\n[Topology Change] The results don't show any good instances of topology change. I do not understand this claim. The best example I could find is at 0:45. Are there any other good instances? I'm also lost as to which aspect of the method helps with topology change. Please mark that more clearly.\n\n*** Experiments:\n\n[Scene Flow Evaluation] The paper doesn't show any appearance/novel view results. That's okay since it focuses on the geometry, and appearance is only used as an auxiliary to deal with the RGB input. However, I would still be curious to see video results where the appearance is taken from t=0 and propagated via the SDF flow. That would allow to visualize longer-term correspondence drift and whether tangential motion causes issues in practice. Currently, the SDF flow (and thus the derived scene flow) only cares about short-term \"geometrical\" correspondences. - This is relevant because the paper doesn't just claim that the SDF flow is a helpful parametrization but rather that it enables scene flow, i.e. correspondences. Currently, the qualitative and quantitative scene flow evaluation is only short-term, even though the method requires long-term correspondences (equation 7). The quantitative scene flow evaluation is also only a single sequence. \n\n[Integral Evaluation] How does the temporal integration scale with scene length? Presumably linearly? Is that why only 24 frames are used? How many function evaluations does the Runge-Kutta solver need for the last (24th) frame? \n\n[Weak Qualitative Geometry] Tensor4D gives detailed, noisy results and NDR is less noisy but lacks detail. The submission's results have the level of detail of Tensor4D and the low noise level of NDR. Still, none of these results are overwhelming or that impressive. (Doesn't need to be addressed in the rebuttal.)\n\n*** Paper:\n\n[Related Work] The related work section isn't that thorough. For example, the dynamic NeRF papers Fang et al. Fast dynamic radiance fields with time-aware neural voxels and Li et al. Neural 3D Video Synthesis are missing. Li et al., like NSFF, can also handle topology changes. Furthermore, shape-from-template methods aren't mentioned. Scene flow methods (e.g. Song et al. PREF: Predictability Regularized Neural Motion Fields) aren't discussed. Also, since NDR is compared to, mentioning some of the papers in the RGB-D line of work would be good, e.g. DynamicFusion, VolumeDeform, KillingFusion, OcclusionFusion. And TARS (Duggal et al. Topologically-aware deformation fields for single-view 3d reconstruction) seems relevant since it isn't restricted to blend skinning (a typo (\"blender\") in the submission). \n\n[Limitations] Please discuss limitations.\n\n*** Minor:\n\n[Input Assumptions] Since the mathematical derivation matters here, please state what assumptions go into the paragraph between equation 5 and 6, which talks about the continuity and differentiability of a time-dependent SDF. In theory, there is nothing preventing an object from appearing out of nowhere, an SDF does not inherently have any restrictions on its temporal evolution (while it does have restrictions in space, namely the Eikonal equation). In real-world cases, geometry noise (say, due to the sensor or an imperfect reconstruction) pops randomly into existence and vanishes randomly over time, which leads to discontinuities w.r.t. the time parameter. Unless point x is meant to be a Lagrangian particle rather than an Eulerian grid coordinate? Figure 2 looks Eulerian though. --- Please state the assumptions that go into that paragraph.\n\n[Wrong Argument] The argument at the end of Sec. 3.1 is that most dynamic NeRF methods have issues with topology changes. That's not the case for most dynamic NeRF methods that handle general objects since most condition the canonical model in some manner on time (e.g. HyperNeRF). An extreme case of that is Neural Scene Flow Fields. The argument only holds for D-NeRF-style methods like Nerfies or NR-NeRF. For the others, the better argument would be to say that they use density to parametrize geometry rather than SDF and hence the geometry tends to be very noisy and lack a clearly defined surface.\n\n[Labelling] Figure 2 would benefit from labelling the two ellipses with their respective timesteps (t=0 and t=6?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)