Class-agnostic Reconstruction of Dynamic Objects from Videos
Authors: Zhongzheng Ren
Xiaoming Zhao
Alex Schwing
Venue: NeurIPS 2021 Poster (2021)
Keywords: Dynamic reconstruction
implicit network
3D vision

Abstract: We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due to occlusion or camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape; 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different  categories  of  objects  with  one  unified  framework. To  address  these challenges, we develop two novel modules.  First, we introduce a canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop a 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VOS 3D and DeformingThings4D++,  and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.

Class-agnostic Reconstruction of Dynamic Objects from Videos
Zhongzheng Ren, Xiaoming Zhao; Alexander G. Schwing
University of Illinois at Airana-Champaln https://jason718 github.io/red
Abstract
We introduce REDO, class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due occlusion camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape: 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different categories of objects with one unified framework. To address these challenges, develop two novel modules. First, introduce canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VON VOS 3D and and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.
Introduction
4D (3D space + time) reconstruction of both the geometry and dynamics of different objects is a long-standing research problem, and is crucial for numerous applications across domains from robotics to augmented/virtui reality (AR/VR). However, complete and accurate 4D reconstruction from videos remains a great challenge for mainly three reasons: 1) partial visibility of objects due occlusion camera settings (e.g., out-of-view parts, non-observable surfaces); 2) complexity of the dynamics including rigid-motion (e.g., translation and rotation), non-rigid motion (deformation caused by external forces), and articulation; and 3) variability within and across object categories.
Existing work addresses the above challenges by assuming complete visibility through a multi-view setting [33, 4], or by recovering only the observable surface rather than the complete shape of an object [57]. or by ignoring rigid object motion and recovering only the articulation [59], or by building shape templates priors specific to particular object category like humans [47]. However, these assumptions also limit applicability of models to unconstrained videos in the wild, where these challenges either infeasible only met when taking special care during video capture.
In contrast, we aim to study the more challenging unconstrained 4D reconstruction setting where objects may never be entirely visible. Specifically, we deal with visual inputs that suffer from: 1) occlusion: the moving occluder and self-articulation cause occlusion to change across time; 2) cropped view: the camera view is limited and often fails capture the complete and consistent appearance across time; 3) front-view only: due to limited camera motion, the back side of the objects are often not captured at all in the entire video. Moreover, focus different dynamic object-types
Indicates equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Person (syuthctic) Articulation, rigid motion
CarlMotoryycll (symtletie)) Non-rigid motion, rigid motion
Animal (synthetic) Articulation, rigid motion
Person (rcal-world) Articulation, rigid motion
Figure I: We present REDO, 4D reconstruction framework that predicts the geometry and dynamics of various objects for a given video clip. Despite objects being either occluded (e.g.. car/motorcycle) or partially-observed, REDO recovers relatively complete and temporally smooth results.
with complex motion patterns. These objects could either move in 3D space, or be deformed due external forces, or articulate themselves. Importantly, we aim for class-agnostic reconstruction framework which can recover the accurate shape at each time-step.
To achieve this we develop REDO. As illustrated in Fig. 1, REDO predicts the shape of different objects (e.g., human, animals, car) and models their dynamics (e.g.. articulation, non-rigid motion, rigid motion) given input video clips. Besides the RGB frames, REDO takes as input the depth map, masks of the objects of interest, and camera matrices. In practice these inputs are realistic as depth-sensors are increasingly prevalent [19, 78] and segmentation models [44, 17] are increasingly accurate and readily available, e.g., on mobile devices, If this data isn't It accessible, off-the-sheil tools are applicable (e.g., SIM [71], instance segmentation [26]. video depth estimation [5]].
To address the partial visibility challenge introduced by occlusion or camera settings, REDO predicts temporally coherent appearance in canonical space. To ensure that the same model is able to reconstruct very different object types in a unified manner we introduce a pixel-aligned 4D implicit representation, which encourages the predicted shapes closely align with the 2D visual inputs (& 3.1.. The visible parts from different frames of the video clip are aggregated to reconstruct the same object ($ 3.2). During inference, the reconstructed object in canonical space is propagated to other frames to ensure a temporally coherent prediction (6 3.3).
REDO achieves state-of-the-art results on various benchmarks ($ 4). We first conduct experiments on two synthetic RGBD video datasets: SAIL-VOS 3D [27] and REDO improves over prior 4D reconstruction work [59, 70] by great margin (+5.9 mloU, 0.085 mCham., -0.22 mACD on SAIL-VOS 3D and +2.2 mloU, -0.063 mCham., -0.047 mACD on Deforming/Things4Dtt over OFlow [59]). We then test on the real-world calibrated video dataset 3DPW [86]. We find that REDO generalizes well and consistently outperforms prior 4D reconstruction methods (+ 10.1 mloU, -0. 124 mCham., -0.061 mACD over OFlow). We provide comprehensive analysis to validate the effectiveness of each of the introduced components.
[42].
2 Related work
In this section, we first discuss possible geometry representations. We then review fusion-basse and learning-basee 4D reconstruction approaches, followed by brief introduction of Motion Capture methods. Lastly, we discuss works of dynamics modelling and related 4D reconstruction datasets.
Geometric representations. Representations to describe 3D objects can be categorized into two groups: discrete and continuous. Common discrete representations are voxel grids [22, 90, 8], octrees [68, 82], volumetric signed distance functions (SDFs) [10, 30, 56], point-clouds [1. 18, 67], and meshes [23, 34, 55, 38, 88]. Even though being widely used, these representations pose important challenges. Voxel grids and volumetric-SDFs can be easily processed with deep learning frameworks, but are memory inefficient [76, 52, 64]. Point-clouds are more memory efficient to process [65, 66), but do not contain any surface information and thus fail to capture fine details. Meshes more expressive, but their topology and discretization introduce additional challenges.
To overcome these issues, continuous representations, i.e., parametric implicit functions, are intro- duced to describe the 3D geometry of objects [7, 60, 53] and scenes [54, 73). These methods are not constrained by discretization and can thus model arbitrary geometry at high resolution [70, 81]. In

---

Canenical space é ($3..1
9e Dynamic feat xp
Occupancy
How-field + (S 3.2c)
Hlw-field + 3.2c)
By Fax fg]
Transformer Lux 3.2m)
12
Velocity
embedding
Neural
Static fent Zp
ODE
Transformer
fags
Figure 2: Framework overview. For a query point in canonical space, REDO first computes the pixel-aligned features from the feature maps of different frames using the flow-field &. It then aggregates these features using the temporal aggregator Jagg- The obtained dynamic feature Xp is eventually used compute the occupancy score for shape reconstruction.
practice, discretized representations like a mesh can be easily extracted from implicit functions via algorithms like Marching Cubes [49].
Fusion-based 4D reconstruction. DynamicFusion [57] the pioneering work for reconstructing non-rigid scenes from depth videos in real-time. It fuses multiple depth frames into a canonical frame recover the observable surface and adopt a dense volumetric 6D motion field for modeling motion. Another early work [14] reconstructs dynamic objects without explicit motion fields. Improvements of DynamicFusion leverage more RGB information [28], introduce extra regularization [74, 75], develop efficient representations [20], and predict additional surface properties [24]. Importantly, different from the proposed direction, these works only recover the observable surfaces rather then the complete shape. Moreover, these works often fail to handle fast motion and changing occlusion. Note, these models often have no trainable parameters and are more geometry-based.
Learning-based 4D reconstruction. Reconstruction of the complete shape of dynamic objects is considered hard due big intra-class variations. For popular object categories such as humans or certain animals, supervised learning of object template parameters has been utilized to ease reconstruction [36, 94, 47, 96, 97]. These templates are carefully designed parametric 3D models, which restrict learning to low-dimensional solution space. However, the expressiveness of templates limited. For instance, the SMPL [47] human template struggles to capture the clothing or hair styles. In addition, systems relying on templates are class-specific as different object categories need to be parameterized differently. We think it remains illusive to construct a template for every object category. To overcome these issues, OFlow [59] directly learns 4D reconstruction from scratch. However, it predicts articulation in normalized model space and thus overlooks rigid-motion of the object. It also struggles to handle occlusion. When ground-trute truth 3D models not available, dynamic neural radiance fields (NeRF) [54, 43, 63, 61] learn an implicit scene representation from videos, but are often scene-specifie and don't scale to a class-agnostic setting. Self-supervised methods [91, 35, 41] are promising and learn 4D reconstruction via 2D supervision of differentiable rendering [37, 48]. In contrast, in this work, present class class-agnostic and template free framework which learns to recover the shape and dynamics from input videos.
capture. When restricting ourselves to human modeling, 4D reconstruction is often referred to as motion capture (MoCap). Marker and scanner based MoCap systems [62, 46, 40] are popular in industry, yet inconvenient and expensive. Without expensive sensors, camera-based methods are more applicable but often less competitive. Standard methods rely on a multi-view camera setup and leverage 3D human body models [32, 95, 21, 85]. Active research problems in the MoCap field include template-fir methods [16. 15, 11], single-view depth camera methods [2, 92, 93], monocular RGB video methods [25, 79], and fine-grained body parts (e.g., hand, hair, face) methods [72, 50, 89, 33]. In contrast, this paper studies a more generic class-agnostic setting where multiple different non-rigid objects reconstructed using unified framework.
Motion/Performant
Dynamics modeling. different communities, the task of dynamics modeling is named differently, For 2D dynamics in images, optical flow methods [13, 80, 83] have been widely studied and used. Scene flow methods [43, 45, 31] aim to capture the dynamics of all points in 3D space densely, and are often timecconsuming and inefficient. For objects, non-rigid tracking matching methods [4, 77, 9,
3

---

study the dynamics of non-rigid objects. However, these methods often only track the observable surface rather then the complete shape space. Recently, neural implicit functions [43, 59] have been applied to estimate 3D dynamics, which we adapt and further improve through conditioning on pixel-aligned semporally-aggregated visual features.
4D reconstruction datasets. To support 4D reconstruction, a dataset needs to have both ground- truth 3D models and accurate temporal correspondences. Collecting such dataset real-world setting is extremely challenging as it requires either expensive sensors or restricted experimental settings. To simplify the setting. existing data is either class maass-peecffic (e.g., human) [29, 87, 3]. or of extremely small scale [74, 28], or lacking ground-truth 3D models [5]. Given the progress in computer graphics. synthetic data is becoming increasingly photo-realistic and readily available. Deforming DeformingThings4D [42] provides large collections of humanoid and animal 4D models, but lacks textures and background. SAIL-VOS 3D [27] contains photo-realistic game videos together with various ground-truth. For more details, please see the comparison table in Appendix $ B.
3 REDO
We aim recover the 3D geometry of a dynamic object over space and time given RGBD video together with instance masks and camera matrices. To achieve this we develop REDO which is illustrated in Fig. 2. Specifically, REDO reconstructs space, which we detail in $ 3.1. For this REDO employs a temporal feature aggregator, pixel-aligned visual feature extractor, and flow-field, all of which we discuss in 3.2. These three modules help align dynamics of the object across time and resolve occlusions to condense the most useful information into the canonical space. Lastly, detail inference ($ 3.3) and training ($ 3.4).
object shape and its dynamics in canonical
Notation.
The input of REDO is a fixed-length video clip denoted via , 1y}. consists of N RGBD frames 13 E [0, 1]4xW H (i {1,.... N) }) of width W and height H, each recorded at time- step f4. For each frame 7, we also assume the camera matrix and instance mask maj E {0, 1] [W*H given, where maj indicates the of pixels that correspond object j. For readability, we define the following operations: 1) propagate: transforms temporally in 3D; 2) project: transforms from 3D 2D (image space); and 3) lift: transforms from 2D (image space) to 3D.
3.1 Canonical 4D implicit function
Dynamic objects deform and move across time and are often occluded or partially out-of-views In addition, the depth-size ambiguity also chal- condense
points liftnng
lenges reconstruction algorithms. information about the object across both space and time, we leverage a canonical space C R-3 which aims to capture holistic geometry repre- sentation centered around the object of interest. In our case C' denotes volume-constrained 3- dimensional space. Temporally, the canonical space corresponds to the center frame I, at time- step to with e = [11 + N/22
Canoniaal spocod
Zyar
Figure 3: Canonical Space. Depth points from dif- ferent frames are lifted and aggregated to form an enlarged space for capturing the dynamic objects.
To construct the canonical space, we infer 3D volume around the dynamic object j. For all frames I, E (1....,N) N). we first lift the pixels of instance mask m,j of object j to the world space using the depth map and the camera matrices. These 3D points are then aggregated into canonical space using the camera matrix of the canonical frame. From the aggregated point clouds we infer the horizontal and vertical bounding box coordinates well the object's 's closest distance from the camera, i.e., Znear- However, since the visible pixels only represent the front surface of the observed object, the largest distance of the object from the camera, i.e., Ziar is unknown. To estimate this value, we simply set Ziar to stay a fixed distance from Zucar- We illustrate an example of this canonical space in Fig. where the inferred canonical space (blue volume) is relatively tight and big enough to capture the complete shape and dynamics of the observed object.
To more precisely capture an object inside the volume-constraines canonical space, we use 4D implicit function. For each 3D query point canonical space, this function conditions on temporal feature to indicate whether that point is inside outside the object. Specifically, for query point

---

inside the canonical space, i.e., for E C, the 4D implicit function is defined as:
go xp) RK - [0, 1].
(1)
Intuitively, the higher the score go(:;-), the more likely the point p is part of the object. Here, 0 subsumes all trainable parameters in the framework and Xp E IR is a K K-dimensional dynamic feature which summarizes information from all input frames. Concretely,
Xp = Sage ({Senc (Φ tos 4, va)) (4))4))))))))
(2)
where (+)) R transforms point in canonical space to world space. Meanwhile, + is the flow-field which propagates the point p from t. to t; in canonical space, leveraging a learned 3-dimensional velocity-fies vg. Note, le refers to the time-step of the canonical frame I. while 4 denotes the time-step of frame J;. Given the transformed point ( ley li, vo)) E IRS. we use pixel-aligned feature encoder Sane to extract a set of visual representations, each of which comes from one of the N frames 4, i... Vi E {l.... .N. The set function fagg is temporal feature aggregator which merges the temporal information of different time-steps. We'll discuss details of temporal aggregator fagg. encoder fonc. flow-field & and velocity-field vp in 3.2.
This canomical 4D implicit function propagates points across time and extracts the pixel-aligned visual representations from different frames. This differs from: prior works that only consider static objects. Actually, setting N = 1 in Eq. (2) will simplify Eq. (1) to g(p. ( (p). recovering static Pixel-aligned Implicit Function (PIFu) used in [69, 70]; 2) OFlow [59] which encodes the
whole video clip (I,,.., JN] } into one single feature vector, thus loosing spatial information.
3.2 Framework design
In this section, we introduce the temporal aggregator Sage. the feature extractor Senc. and the flow-field & used in Eq. (2). These modules help REDO aggregate information from different time-steps (frames) and model the complex dynamics.
a) Temporal aggregator fags. To deal with partial visibility caused by occlusions camera settings, we develop a transformer-basee temporal aggregator fage as shown in Fig. 2. fage is a set function which computes K di-imensional point features Xp E RRK Assume the flow-field +b is given and object j is of interest. We first propagate query point within the canonical space, i.e., the point E C, to every time-step t,, obtaining locations z (p, te, t,, vo)) E R Vi E {1...., N). We then project each 3D point v ($(p, le: 44, vo)) back to the corresponding 2D image frame using the associated camera matrix. For all points that are projected into the mask area maj. we extract visual representation using the pixel-aligned feature extractor Senc, which we detail below. Note that due to partial visibility, we may not be able to extract features from every frame in the clip. To cope with this, the aggregator fres is designed as transformer-basse [84] set function. For more information, please see the implementation details in $ 4.1 and Appendix A.
b) Pixel-aligned feature extractor fenc. Pixel-aligned features help REDO make 3D predictions that are aligned with the visual 2D input. To achieve this, we develop fenc (q, I,) where the first argument is point in 3D world space and the second argument is frame I,. We first project the point q in world space to the frame I,, and then use a pre-trained convolutional neural net [58] extract the 2D feature map of video frame 7. For points that fall within the instance mask of frame, we extract its visual representation using bi-linear interpolation at the We then append a positional encoding [84] of the frame time-step &, this feature which helps to retain temporal information. We illustrate this process in Fig. 2. The resulting feature is combined with visual cues from other frames to serve as input to the temporal aggregator faes.
location.
c) Flow-field Φ. The flow-field models object dynamics in space and time. For this let (pp, f1, 2g) E C denote flow-field function in canonical space. It computes the position at time-step to of 3D point, whose location is E at time-step t1. To compute the displacement, we define velocity field un(-) which represents the 3D velocity vectors in space and time via
Ue (p Zp f) C x RR
R + C.
Here, E C is a point in canonical space with corresponding static feature Zp E RK computed as
Up = Sage
(4)
5

---

Here, fene (t)),, 14) is the feature of world coordinate point (p)p) extracted from frame I,. Note, Zp differs from Xp defined in Eq. (2). The feature zp summarizes information of static locations from all frames 4. This feature is heneficial as it helps capture whether point remains static or whether it moves. The velocity network then leverages this feature 7p to predict object dynamics.
Using the velocity field, we compute the target location at time-step &2 of point originating from location p time-step by integrating the velocity field over the interval [ti, via
(p, t2 vo) = P /, No (p, 11, v0), Zp, t).t..t
(5)
Note that ((p, (1, 12, (1) can represent both forward (42 > (1) or backward motion (12 < (1) given the initial location and velocity-field. To solve the flow-field for discrete video time-steps, approximate the above continuous integral equation using a neural-ODE solver [6].
3.3 Inference
To reconstruct objects densely in a clip, the reconstruction/nffereee procedure is summarized as:
Step 1: We construct the canonical space the center frame I, and sample query point set P C uniformly from the inferred space. This step requires no network inference and is very efficient.
Step 2:
We extract static features Zp for all p P using Eq. (4), Next, compute the trajectory of point p. i.e. (p, fo; bi, vi)) for all time-steps t associated with frames L,, E {I,..., NN,, using a neural ODE solver which solves Eq. (5).
Step 3: We then compute dynamic features Xp for all p using Eq. (2).
Step 4: Using Eq. (1) we finally compute the occupancy scores for all p E P. which are then transformed into triangle mesh via Multi-resolution Iso-Surface Extraction (MISH) [53]. Note, the mesh is constructed in canonical space.
Step 5: To obtain the mesh associated with frame I, following Step 2, we use the flow-field + to propagate all vertices of the extracted mesh from the time-step to of the canonical space to time-step which corresponds non-canonical frame 1, After using the function (5() defined in Eq. (2), we obtain the mesh in 3D world space. While various discrete representations could be obtained from implicit representation,
meshes for evaluation and visualization purposes.
3.4 Training
REDO is fully differentiable containing the following parametric components: the temporal aggrega- tor fagy. the feature extractor fene, the velocity-fiele network ug. and the reconstruction network go- For simplicity, we use f subsume all trainable parameters of REDO. To better extract shape and dynamics from given video clips, we train REDO end-to-end using
min Shhppe (D, 8) + Cump (D,
(6)
where D is the training set. Lahape (D, 0) is the shape reconstruction loss in canonical space which encourages REDO to recover the accurate 3D geometry. Cemp (2, 0) is a temporal coherence loss defined on temporal point correspondences. This loss encourages the flow-field Φ to capture the precise dynamics of objects. We detail training set, sampling procedure, and both losses next.
Training set. We train REDO on a dataset D that consists of entries from different videos and of various object instances. Formally, D {(((T,, 4)) V, v. Specifically, data entry for an N-frame video clip includes three components: 1) set of RGBD frames {4)) associated time-steps [t+). as well as corresponding camera matrices E [I....,N : 2) an instance ground-truth mesh V that provdeesttmpooral
aligned supervision. For every vertex E V, we use V; to denote its position at time-step f; 3) the ground occupancy ) in the canonical space. Concretely, for point p C in canonical space, occupancy label y(p) E {0, 1) indicates whether p is inside the object (y = 1) or outside the object (y = 0). As mentioned $ 3.1, the canonical space is chosen so that it corresponds time-step l.., where [(1 + N)/2].
Sampling procedure. To optimize for the parameters during training we randomly sample a set of points p P(V) within the canonical space. P(V) contains mixture of uniform sampling and

---

importance sampling around the ground-truth mesh's 's surface V at time-step &c. Similar strategies are also used in prior works [69, 70].
Shape reconstruction loss. To encourage that the canonical 4D implicit function go accurately captures the shape of objects use the shape reconstruction loss
Shappe (D,0)
E
Σ
BCE (sp(p.
P(V)
PEP(V)
where BCE(..)) represents the standard binary cross-entropy loss.
Temporal coherence loss. REDO models dynamics of objects explicitly through the flow-field Φ, which leverages the velocity-field network up. As the ground-truth correspondences across time are available in V, we define the temporal correspondence loss via the squared error
Ciepp (D,0)
Σ
pV(ve ti -vili..
(8)
NIV (WYEDD
VEV i=
4
Experiments
We first introduce the key implementation details 4.1) and the experimental setup 4.2), followed by the quantitative results ($ 4.3), qualitative results ($ 4.5), and an in-depth analysis ($ 4.4).
4.1 Implementation details
We briefly introduce the key implementation details. Check Appendix $ for more detailed version.
We assume all input clips are trimmed to have N = 17 frames following [59]. During training, clips randomly sampled from original videos which have any length. For videos that shorter than 17 frames, we pad at both ends with duplicated starting and ending frames to form the clips. The validation and split development into manageable pieces, and allows fair comparison with prior work. In practice, dense reconstruction the entire video is achieved via sliding window method.
Input:
consist of fixed 17-frame clips. This simplified input setting allows to
Reconstruction network: Following [69], the reconstruction network ge is implemented as a 6-layer MLP with dimensions (259, 1024, 512, 256, 128, and -kip-connections, The first layer's 's dimension of 259 is due to the concatenation of visual features (256-dim) and query point locations (3-dim).
Temporal aggregator: Sag uses transformer model with 3 nulti-headed self-attention blocks and a I-layer MLP. Group normalization and skip-connections are applied for each block, and we set the hidden dimension to be 128. To compute the time-encoding, use positional-encoding [84] with 6 exponentially increasing frequencies.
fene is implemented as a 2-stack hourglass network [58] following PIFu [69]. Given the instance mask, we take out the object of interest the picture and resize it to 256 256 before providing it as input to feae. The output feature map has dimensions of 128 x 128 x 256 of 256.
Feature extractor:
spatial resolution 128 128 and feature dimension
Velocity-field network: Up uses a 4-layer MIP with skip-connections following [59], where the intemal dimension is fixed to 128. It takes query points input and adds the visual features the activations after the 1st block. For ODE solvers, we use the Dormand-Prince method (dopri5) [12).
Training: In each training iteration we sample 2048 query points for shape reconstruction and 512 vertices for learning temporal coherence. We train REDO end-to-end using the Adam optimizer [39] for 60 epochs with batch size of 8. The learning rate is initialized 0.0001 and decayed by 10x at the thth and 55th epochs.
4.2 Experimental setup
Dataset. We briefly introduce the three datasets used our experiments below. For more details (e.g., preparation, statistics, examples), please check Appendix $ B.
7

---

Datasct Metrics
SAIL-VOS 3D
DeformingThingsDD..
3DPW
m
mCham
mACD
mloUt
mCham.l mACD) m
mCham.
MACDI
Static reconstruction
ONet [53]
24.5
0.951
60.2
0.260
29.8
0,440
PIFuHD [70]
25.6
0.724
43.8
0,511
37.4
0.363
Dynamic reconstruction
Surfe/Warp [20]
1.03
2.13
3.75
6.53
OFlow [59]
26.0
0.732
1.69
55.2
0,412
0.812
31.5
0.461
0.907
REDO
31.9
0.647
1.47
57.4
0.349
0.765
41.6
0.337
0.846
Table 1: Quantitative results. For both shape reconstruction (mloU and mCham.) and dynamics modeling (mACD), REDO demonstrates significant improvements over prior methods. mACD is not available for static methods and SurfelWarp which don't 't predict temporally corresponding meshes.
1) SAIL-VOS 3D [27]: photo-realistic synthetic dataset extracted from the game GTA-V. It consists of RGBD videos together with ground-truth (masks and cameras). Out of the original 178 object categories. we use 7 dynamic ones: human, car, truck, motorcycle, bicycle, airplane, and helicopter. During training, we randomly sample clips from 193 training videos. For evaluation, we sample 291 clips from 78 validation videos. We further hold out 2 classes (dog and gorilla) as an unseen test set.
EeftommnngThingsAAee [42] is synthetic dataset containing 39 object categories, As the original dataset only provides texture-less meshes, we render RGBD video and corresponding ground-trum (mask and camera) using Blender. Because the original dataset doesn't t provide dataset splits, we create our own. Specifically, during training, we randomly sample clips from 1227 videos. For evaluation, we create a validation set of 152 clips and test set of 347 clips. We hold out class puma with 56 videos as zero-shot test set. We dub this dataset Deforming DeformingThings444444 Things4D++.
2)
3) 3D Poses in the Wild (3DPW) [86]: to test the generalizability of our model, test on this real- world video dataset. Unfortunately, no real-world multi-class 4D dataset is available. Therefore, we test REDO in a class-specific, i.e., class human, setting using 3DPW. This dataset contains calibrated videos, i.e., known camera, and 3D human pose annotation. However; it doesn't 't provide ground-true -tuuh mesh and depth. To extract a mesh, we fit the provided 3D human pose using the SMPL [47] template. To compute depth, we use Consistent Video Depth (CVD) [51] with ground-truth camera data to get temporally consistent estimates. The dataset contains 60 videos (24 training, 12 validation, and 24 testing). During training, we randomly sample clips from all training videos. For evaluation, we evaluate at uniformly sampled clips (10 clips per video) using the validation and test set.
Baselines. We consider the following baselines: 1) Static reconstruction: we adopt state-of-the-eee methods ONet [53] and PIFuHD [70], and train them for per-frame static reconstruction. For fair comparison, we train these two networks in a class-agnostic setting using all the frames in the training videos. 2) Fusion-based dynamic reconstruction: most fusion based methods [57, 74, 75] are neither open-sourced nor reproduced. Among the available ones, we adapt the author-released SurfelWarp [20] due to its superior performance. Since this method is non-parametric and requires no training, we directly apply it on the validation/testing clips. 3) Supervised dynamic reconstruction: REDO learns to reconstruct dynamic objects in a supervised manner. OFlow [59] also falls into this category but it doesn't 't handle the partial observation and rigid-motion Note, REDO uses each clip's s center frame as the canonical space while OFlow uses the initial one. Therefore, for a fair comparison. we set OFlow's 's first frame to be the center of our input clip.
Metrics. To evaluate the reconstructed geometry, report the mean volumetric Intersection over Union (mloU) and the mean Chamfer l distance (mCham.) over different classes at one time- step. i.e., the center frame of the test clip. To evaluate the temporal motion prediction, we compute the mean Averaged (over time) Correspondence &g Distance (mACD) following [59]. As stated before, OFlow's starting frame set to the center frame. For fair comparison, we report mACD on the latter half of each testing clip. We compute mCham. and mACD error in a scale-invariant way following [53, 59, 18]: we use 1/10 times the maximal edge length of the object's 's bounding box as unit one. Even though our network is class-agnostic, we report the mean values over different object categories. Namely, all 'mean' operations are conducted over categories.
4.3 Quantitative results
We present results on all three datasets in Tab. 1. For a fair comparison, we test on the center frame (canonical frame of REDO) of the validation/testing clips. We observe that:
8

---

1) REDO improves upon the static methods for shape reconstruction on SAIL-VOS 3D and 3DW (+6.3/4.2 mloU and -0.077/-0.006 mCham. over best static method). This is because the static methods cannot capture the visual information from other frames in the video clip and thus fail to handle partial visibility. However, REDO performs slightly worse than ONet on due to the unrealistic simplified visual input: 1) the pictures have only one foreground object with neither occlusion nor background, 2) the rendered color is determined by the vertices order and hence provides a visual short-cut for 2D to 3D mapping. Without modeling dynamics, the static baselines are hence easier to optimize. In contrast, SAIL-VOS 3D renders photo-realistic game scenes with diverse dynamic objects, which are much closer to a real-world setting.
2) REDO outperforms basien method SurfelWarp greatly on all benchmarks, as Surfel Warp only recovers the observable surface rather than the complete shape. We didn't 't run SurfelWary on 3DPW as it relies on precise depth as input and crashes frequently using the estimated depth values.
3) REDO improves upon OFlow (+5.9/2.2/10. mloU and -0.085/0.063/0.. 10.01.22. 124 mCham.) for shape reconstruction due to the pixel-aligned 4D implicit representation, whereas OFlow encodes the whole image as a single feature vector and looses spatial information.
4) Regarding dynamics modeling, REDO improves upon OFlow (-0.22/0.047/)) 047/0.061 mACD) thanks to the pixel-aligne implicit flow-field. Note that OFlow normalizes the 3D models at each time-step into the first frame's 's model space and hence fails to capture rigid motion like translation. In contrast, our canonical space is constructed for the entire clip in which REDO predicts a complete trajectory.
As stated in Sec. 3.3, the reconstructed mesh on the center frame is propagated to other frames in the video clip for dense reconstruction. We thus report the per-frame results in Appendix addition, all above results are mean values averaged over object categories to avoid being biased towards the most frequent class, Class-wise results on SAIL-VOS 3D are reported in Appendix $ C.
In
Zero-shot reconstruction. To test the generalizability of REDO, we further test on unseen categories with no fine- tuning in Tab. 2. The result is averaged over three unseen classes: dog and gorilla from SAIL-VOS 3D. and puma from Deforming' DeformingThings4. Things4+++. REDO still greatly outper- forms baselines and doesn't 't fail catastrophically. The per- class results are provided in Appendix Tab. S5.
m
mCham.l
MACD,
ONet
23.1
0.764
PIFu
21.2
0.911
SurfeIWarp
2.06
1.23
OFlow
26.7
0,931
1.18
REDO
38.5
0.479
1.07
Table 2: Zero-shot reconstruction.
4.4 Analysis
In Tab. 3, we provide an ablation study of different com- ponents in REDO using SAIL-VOS 3D data. 1) We first replace the temporal aggregator fuge with average pool- ing layer where features of different frames are averaged and fed into the shape reconstruction and velocity field net- work. The results are shown in the 1st row of Tab. 3 (avg. pooling). The performance drops by -3.6 IoU, +0.065 mCham., and +0.13 mACD. 2) We then study pixel-aligned feature representations Xp. Zp- We replace these two features with the feature map of the entire input frame following OFlow [59] but still keep the transformer to aggregate these feature maps. Results of this ablation are reported in Tab. 3 (w/o alignment). Compared to REDO, this setting greatly hurts the results (-7.8 mloU, +0.290 mCham -0.38 mACD) as the network can longer handle partial observations and the 3D predictions don't 't well align with the visual input, 3) In many real-world tasks, ground-trutt -rrth meshes of different time-steps are not corresponded. Conceptually, REDO could adapt to this setting. This is because all components are differentiable and the flow-field network could be used as a latent module for shape reconstruction. To mimic this setting, we train REDO using only the shape reconstruction loss Lshupe. As shown in Tab. 3 (w/o Lemp)), the model still recovers objects at the canonical frame. However, mACD increases significantly (+1.65).
mloU?
mCham.
MACD]
avg. puoling
28.3
0.712
1.60
wio alignment
24.1
0.937
1.85
w/o Cung
29.4
0.685
3.12
REDO 31.9
0.647
1.47
Table 3: Ablation studies.
4.5 Qualitative results
Fig. 4 shows a few representative examples of REDO predictions on SAIL-VOS 3D and Deform- ing Things4D+++ Please check Appendix & F for more results on real-world data and additional analysis. From Fig. 4 we observe that: 1) REDO able to recover accurate geometry and dynamics of different objects from input video frames. It completes the occluded parts and hallucinates invisible

---

Input
PIFuHD
SurfelWarp
OFlow
Ours
Figure 4: Qualitative results. We illustrate the input frames with the object of interest highlighted. the reconstructed meshes obtained from different methods, and the ground-truth mesh.
Frame
Frame i+1
GT
Pred
Figure 5: Flow-field visualization. REDO accurately recovers the non-rigid motion (e.g., moving forward) but is less precise for small-scale articulation (e.g., hand).
components (e.g., legs and back of humans; tire of the motorcycle) by aggregating temporal information and due to large-scale training. 2) REDO improves upon baselines methods. E.g., PIFuHD struggles handle occlusion and non-human objects, SurfelWarp only predicts the visible surface, and OFlow results are over-smooth as it ignores the spatial information. 3) REDO predictions are still far from perfect compared ground-truth meshes. Many fine- fne-grained details are missing (e.g., clothing of the human, car's 's front-light and tires, etc.).
We also visualize the predicted and ground-truth motion vectors in Fig. 5. REDO successfully models the rigid motion, ie., moving forward, over the entire human body, egg., head, leg, chest, etc. It's less accurate in capturing the very fine-gra grained dynamies, e.g., the hand motion where the ground-truth indicates the hand will open while the prediction doesn't t.
Conclusion
We present REDO, a novel class-agnostic method to reconstruct dynamic objects from videos. REDO implemented canonical 4D implicit function which captures the precise shape and dynamics and deals with partial visibility. We validate the effectiveness of REDO on synthetic and real-world datasets. We think REDO could be generalized wide variety of 4D reconstruccion tasks.
10

---

Acknowledgements & funding transparency statement. This work was supported in part by NSF under Grant #1718221, 2008387, 2045586, 2106825, MRI #1725729, NIFA award 2020-67021- 32799 and Cisco Systems Inc. (Gift Award CG 1377144 thanks for access to Arcetri). 7R is supported by a Yee Memorial Fund Fellowship.
References
[1] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018.
[2] A. Baak, M. Müller, G. Bharaj, H.-P. Seidel, and C. Theobalt. A data-driven approach for real-time full body pose reconstruction from depth camera. Consumer Depth Cameras for Computer Vision, 2013.
[3] F. Bogo, J. Romero, G. Pons-Moll, and M. J. Black. Dynamic FAUST: Registering human bodies in motion. In CVPR, 2017.
[4] A. Bozic, P. Palafox, M. Zollöfer, A. Dai, J. Thics, and M. Nießner, Neural non-rigid tracking. In NeurlPS, 2020.
[5] A. Božić, M. Zollhöfer, C. Theobalt, and M. NieBnerr Deepdeform: Leaming oonriiiid rgb-d reconstruction with semi-supervised data. In CVPR, 2020.
[6] R. Т. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018.
[7] 7. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
[8] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.
[9] A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Eyseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. TOG, 2015.
[10] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In SIGGRAPH, 1996.
[11] E. de Aguiar, C. Stoll, C. Theobalt, N. Ahmed. H.-P. Scidel, and S. Thrun. Performance capture from sparse multi-view video. TOG, 2008.
[12] J. R. Dormand and J. Prince. A family of embedded runge-kutta formulae. J. Comput. Appl. Math, 1980.
[13] A. Dasovitskiy, P. Fischer, E. 1lg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
[14] M. Dou, J. Taylor, H. Fuchs, A. Fitzgibbon, and S. Izadi. 3d scanning deformable objects with single rgbd sensor. In CVPR, 2015.
[15] M. Dou, Khamis, Y. Degtyarev, Davidson, S. Fanello, A. Kowdle, S. Orts, C. Rhemann, D. Kim, Taylor, Kohli, Tankovich, Izadi. Fusion4d: real-time performance capture challenging scenes. TOG, 2016.
[16] Dou, P. Davidson, S. R. Fanello, S. Khamis, A. Kowdle, C. Rhemann, Tankovich, and S. Izadi. Motion2fusion: Real-time volumetric performance capture. TOG, 2017.
[17] Facebook-Ar AL D2go brings detectron2 to mobile.
https://ai www.aaaw.....o.aooooaaaoooa
d2go- wew.ing-ddettctronttooo-oo-teoeo
2021.
[18] H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017.
[19] C. Franklin. Apple unveils new iPad Pro with breakthrough LiDAR Scanner and brings trackpad support to iPadOS. https: : //www/ htwp::wpwpp.....cm/////////./... com/ 2020.
[20] W. Gao and R. Tedrake. Surfelwarp: Efficient non-volumetric single view dynamic reconstruction. In RSS, 2018.
[21] D. Gavrila and L. Davis. Tracking of humans in action: A 3-d model-based approach. ARPA Image Understanding Workshop, 1996.

---

[22] G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In ICCV, 2019.
[23] T. Groueix, M. Fisher, V. G. Kim, B. Russell, and M. Aubry. AtlaNNet: A Papier- Pacher-Mäché Mäché Approach to Learning 3D Surface Generation, In CVPR, 2018.
[24] K. Guo, F. Xu, T. Yu, X. Liu, Q. Dai, and Y. Liu. Real-time geometry. albedo, and motion reconstruction using single rgb-d camera. TOG, 2017.
[25] M. Habermann, W. Xu, M. Zollhoefer, G. Pons-Moll, and C. Theobalt. Livecap: Real-time human performance capture from monocular video. TOG, 2019.
[26] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask In CVPR, 2017.
[27] Y.-T. Hu, J. Wang. R. A. Yeh, and A. G. Schwing. SAIL-VOS 3D: Synthetic Dataset and Baselines for Object Detection 3D Mesh Reconstruction from Video Data. CVPR, 2021.
[28] M. Innmann, M. Zollhöfer, M. Nicßner, Theobald, and M. Stamminger. Volumedeform: Real-time volumetric non-rigid reconstruction. In ECCV, 2016.
[29] C. lonescu, D. Papava, Olaru, and Sminchisescu. Human3, 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2013.
[30] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. A. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman,
A. J. Davison, and A. W. Fitzgibbon. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In UIST, 2011.
[31] M. Jaimez, M. Souiai, J. González, and D. Cremers. A primal-dual framework for seal-time dense rgb-d scene flow. In ICRA, 2015.
[32] H. Joo, H. Liu, L. Tan, L. Gui, B. C. Nabbe, I. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh, Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
2015.
[33] H. Joo, T. Simon, and Y. Sheikh. Total Capture: 3D Deformation Model for Tracking Faces, Hands, and Bodies. In CVPR, 2018.
[34] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-to-end Recovery of Human Shape and Pose. In CVPR, 2018.
[35] A. Kanazawa, S. Tulsiani, A. Efros, and Malik. Learning Category-Specite Mesh Reconstruction from Image Collections. In ECCV, 2018.
[36] A. Kanazawa, J. Y. Zhang, P. Felsen, and Malik, Learning 3d human dynamics from video. CVPR, 2019.
[37] H. Kato, Y. Ushiku, and Harada. Neural mesh renderer. In CVPR, 2018
[38] Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Eurggaaphic, 2006.
[39] D. Kingma and J. Ba. Adam: method stochastic optimization. /CLR, 2015.
[40] Li, B. Adams, L. Guibas, and Pauly. Robust single-view geometry and motion reconstruction. TOG, 2009.
[41] X. Li. Liu, K. Kim, S. Mello, Jampani, M.-H. Yang, and Kautz. Self-supervised single-view 3d reconstruction via semantic consistency. In ECCV, 2020.
[42] Y. Li, H. Takehara, Takafumi, Taketomi, B. Zheng, and M. NieBner. 4dcomplete: Non-rigid motion estimation beyond the observable surface. In ICCV, 2021.
[43] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural Scene Flow Fields for Space-Timi Time View Synthesis of Dynamic Scenes. In CVPR, 2021.
[44] H. Liu, R. A. R. Soto, F. Xiao, and Y. J. Lee. Yolactedge: Real-time instance segmentation on the edge. In ICRA, 2021.
[45] X. Liu, C. R. Qi, and L. J. Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, 2019.
[46] M. Loper, N. Mahmood, and M. J. Black. Mosh: Motion and shape capture from sparse markers. TOG, 2014.
12

---

[47] M. Loper. N. Mahmood, J. Romero, G. Pons-Moll, and M. Black. SMPL: A skinned multi-person linear model. TOG, 2015.
[48] M. M. Loper and M. J. Black. Opendr: An approximate differentiable renderer. In ECCV, 2014.
[49] W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm.
TOG, 1987.
[50] L. Luo, H. Li, and S. Rusinkiewicz. Structure-awart aware hair capture. TOG, 2013.
[51] X. Luo, J. Huang, R. Szeliski, K. Matzen, and J. Kopf. Consistent video depth estimation. TOG, 2020.
[52] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition.
In IROS, 2015.
[53] L. Mcseheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[54] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields view synthesis. ECCV, 2020.
[55] C. Nash, Y. Ganin, A. Eslami, and P. Battaglia. Polygen: An autoregressive generative model 3d meshes. In /CML, 2020.
[56] A. Newcombe, S. Izadi. O. Hilliges. Molyneaux, Kim, Davison, Kohli, Shotton,
S. Hodges, and A. W. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In ISMAR, 2011.
[57] R. A. Newcombe, D. Fox, and S. M. Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015.
[58] A. Newell. K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016.
[59] M. Niemeyer, L. Mescheder, M. Occhsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In ICCV, 2019.
[60] J. J. Park, P. R. Florence. J. Straub, R. A. Newcombe, and S. Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR, 2019.
[61] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[62] S. I. Park and J. K. Hodgins. Capturing and animating skin deformation in human motion. TOG, 2006.
[63] A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2020.
[64] C. Qi. H. Su, M. NieBner, A. Dai, M. Yan, and J. Guibas. Volumetric and multi-view cans for object classification on 3d data. CVPR, 2016.
[65] C. Qi. segmentation. In CVPR, 2017.
Mo,
Guibas. Pointuet: Deep leamnngg on point sets for 3d classification and
[66] Qi. L. Yi, H. Su, metric space. In NeurlPS, 2017.
Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in
[67] Z. Ren, I. Misra, A. G. Schwing, and R. Girdhar. 3d spatial recognition without spatially labeled 3d. In CVPR, 2021.
[68] G. Riegler, A. Osman Ulusoy, and A. Geiger. Octnet: Learning deep 3d representations at high resolutions. In CVPR, 2017.
[69] S. Saito, Z. Huang. R. Natsume, S. Morishima, A. Kanazawa, and H. Li. PlFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In ICCV, 2019.
[70] S. Saito, T. Simon, J. Saragih, and H. Joo. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for Hiig-Resollttinn 3D Human Digitization. In CVPR, 2020.
[71] J. L. Schönberger and J.-M. Frahm. Structure-frome motion revisited. In CVPR, 2016.
13

---

[72] T. Simon, H. Joo, and Y. Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR. 2017.
[73] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. In NeurlPS, 2019.
[74] M. Slavcheva, M. Baust, D. Cremers, and S. Ili.. KillingFusion: Non-rigid 3D Reconstruction without Correspondences. In CVPR, 2017.
[75] M. Slavcheva, M. Baust, and S. Ilie. Sobolevfusion: 3d reconstruction of scenes undergoing free non-rigid motion. In CVPR, 2018.
[76] S. Song and J. Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In CVPR, 2016.
[77] Starck and A. Hilton. Surface capture for performance- animation. CG&A, 2007.
[78] S. Stein. Lidar on the iPhone 12 Pro. https: :/wp::///wwwww cnet. .owt.com/. 2020.
[79] Z. Su, W. Wan, T. Yu, L. Liu, L. Fang, W. Wang, and Y. Liu. Mulaycap: Multi-layer human performance capture using a monocular video camera. TVCG, 2020.
[80] D. Sun. X. Yang. M.-Y. Liu, and J. Kautz. Pwe-net: Cans for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[81] T. Takikawa, Litalien. K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. CVPR, 2021.
[82] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In ICCV, 2017.
[83] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[84] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In NeurlPS, 2017.
[85] D. Vlasic, I. Baran, W. Matusik, and J. Popović. Articulated mesh animation from multi-view silhouettes. TOG, 2008.
[86] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
[87] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
[88] N. Wang, Y. Zhang, 7. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018.
[89] T. Weise, S. Bouaziz, H. Li, and M. Pauly. Realtime performance-hased facial animation. TOG, 2011 1.
[90] J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adveersaiill modeling. In NeurlPS, 2016.
[91] G. Yang, D. Sun, V. Jampani, D. Vlasic, F. Cole, H. Chang, Ramanan, T. Freeman, and C, Liu. Lasr: Learning articulated shape reconstruction from a monocular video. CVPR, 2021.
[92] T. Yu, K. Guo, F. Xu, Y. Dong, 7. Su, J. Zhao, J. Li, Q. Dai, and Y. Liu. Bodyfusion: Real-time capture of human motion and surface geometry using a single depth camera. In ICCV, 2017.
[93] T. Yu. Z. Zheng, K. Guo, J. Zhao, Q Dai, H. Li, G. Pons-Moll, and Y. Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In CVPR, 2018.
[94] J. Y. Zhang, Felsen, A. Kanazawa, and J. Malik. Predicting 3d human dynamics from vidco. In ICCV, 2019.
[95] M. Zollhöfer, M. Nicßner, Izadi, C. Rhemann, C. Zach, M. Fisher, C. Wu, A. W. Fitzgibbon, C. T. Loop, C. Thcobalt, and M. Stamminger. Real-time non-rigid reconstruction using an rgb-d camera. TOG, 2014.
[96] S. Zuffi, A. Kanazawa, and M. J. Black. Lions and tigers and bears: Capturing non-rigid 3d articulated shape from images. In CVPR, 2018.
[97] S. Zuffi, A. Kanazawa, T. Berger-Wood Wolf, and M. J. Black. Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images" In the Wild". In ICCV, 2019.
14


Reviews:
[{"review_id": "k-m9cRaHHg", "reviewer": "Reviewer_nToe", "summary": "This paper presents an approach for 4D (space + time) reconstruction from RGBD video sequences with instance masks. It uses an implicit function based representation with explicit modeling of velocity fields (similar to Occupancy Flow) to model dynamic objects. demonstrates superior performance on the SAIL VOS 3D dataset compared to prior works. The authors claim their key contributions to be 1) a pixel-aligned neural representation for non-rigid dynamic objects; and (2) an end-to-end trainable framework (FREDO) for learning this representation from monocular video clips showing partially observed objects.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper combines two key recent ideas which have been shown to be very effective for representing 3D / 4D objects - pixel aligned representations (PiFU / PiFUHD) and occupancy flow. The benchmarks show clear improvement over these works in isolation.\n\n- The paper is clearly written and easy to understand. The notation is clear and abstractions are used when necessary to improve clarity (e.g. avoiding basic 3D<->2D and 3D <->3D transform equations).\n\n- The illustrations in the paper are very helpful and make understanding the work much easier. Fig 4 and the associated discussion regarding quality of flow fields recovered is also quite interesting.\n\nWeaknesses\n- I am concerned that the novelty of the work is limited by the fact that it is a straightforward combination of pixel aligned implicit functions and occupancy flow. The contribution of dealing with partially observed instances is rather weak as it involves just aggregating point clouds over a few frames and constructing the canonical space. \n\n- The supervision used by the method is very extensive. Not only does it require RGBD inputs with instance segmentations during inference, it also requires temporal 3D correspondences (scene flow) during training as well. This type of supervision is only possible in synthetic datasets and raises questions on the general applicability of the method.\n\n- I am also concerned about the baselines reported in this paper and how fair the comparisons are. Based on my knowledge of PiFU HD, it does not require depth maps as input during inference while this work does. Similarly, its unclear what the input is for the OFlow baseline. In general, the experiments section lacks details about the baselines (what data they were retrained on, what was the input etc.).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- I believe the limitations of this work around the nature of supervision required and comparisons of inputs required vs baselines is not discussed adequately. A detailed discussion around the baselines would significantly improve the manuscript.\n\n- The authors are also encouraged to comment on the novelty of this method in the context of works like PiFU and OFlow. Specifically, what are the technical contributions of this work beyond combining ideas from these two works?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "jYypLVGPUwN", "reviewer": "Reviewer_SY53", "summary": "This paper presents a supervised algorithm for 4D reconstruction of objects. The input to the system are ground truth instance masks, ground truth depth maps and RGB images. The system outputs a 3D model in a canonical space. The algorithm has three main components, first a feature extractor that convert 2.5D observations into dense feature vectors. Then, A flow field is learnt jointly, which maps observations in each view into the canonical space. Finally, a aggregation network is used to combine information from all observations, and generate an implicit field in canonical space. \n\nThe training signals used by the method are ground truth shapes and dense correspondences in 3D.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "----\n\nOriginality(4/10)\n\nThis paper presents a new method to the challenging problem of 4D reconstruction, which is a new framework inspired by OccFlow and PiFU. The new components are the aggregation network, the flow network and the strategy of moving window for generating and advancing the canonical space.\n \n My major concern to the presented method, is that it does not discuss any non-deep learning method at all. For example, DynamicFusion [1] uses a commodity RGBD sensor that reconstructs dynamic scenes in real time. Of course, it has limitations: it could only handle moderate motion without topology changes (which are not addressed in this paper either), but it is still powerful enough to be the backbone of many real world applications. Therefore, missing such discussions and comparisons greatly hurts the originality of the presented method, since those methods are, essentially, class-agnostic 4D reconstructions from Monocular RGBD videos.  The authors claim that in L3 ,Compared to prior works, our problem setting is more realistic yet more challenging for two reasons: 1) partial observation 2) class-independent reconstruction. Those claims are not true since[1], compared to this work, tackles both challenges well and is more realistic than the presented work since it works on real world data with a RGBD sensor. \n\n[1]Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. \"Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time.\" CVPR 2015.\n\n----\n\nQuality(4/10)\n\nI'm not convinced by the results of this paper. On the contrary, I believe the results presented in this paper indicates the presented method is missing important parts, prohibiting it from solving the actual problem.\n\nFirst of all, this method takes **ground truth** depth and **ground truth** instance masks as input. Since the observed data is noise free, it is expected that the final reconstruction should at least have good quality when rendered at the input view ,which is not the case as shown in Fig.5, for the motorbike, the truck, the airplane, the person on the floor and the helicopter result. Why this is the case?  This also defeats the purpose of having bold numbers in the table, since if the method is having hard time reproducing the input GT observations, then it is essentially broken somehow. Also it shows that it is less likely to work when only estimated depth/ instance masks are used. This point is further demonstrated in L347, where the authors used GT flow-field. GT flow-field should, in theory produce near perfect results: isn't putting the gt depth into 3D space and deform them using the gt flow-field already gives you the result? What's missing here? \n\nSecondly, since the input are GT signals, the problem reduces to 3D correspondence estimation. Registering deforming surfaces is not a new problem, and many solutions exist. For example, [1] uses a deformation grid for regularizing motion, together with ICP for registering global translations and rotations. How does this strategy compare to flow estimation presented in the paper?  Is Neural ODE style training of dynamics really necessary?\n\nThirdly, I wonder if the aggregation part really needs neural networks. Usually fusion can be done in many forms, the mostly used case is deforming a canonical space to the current view and fuse the observation using signed-distance functions. I understand the design choice in this paper is tailored to a MLP-based continuous occupancy field representation, but how much do you lose, or even gain, if switched to a discrete space with simple fusion strategy?\n\nFinally, a minor issue is about the flow-field prediction. As I understand it, the flow-field is defined in the continuous space. The figures showed the in prediction (Fig(4), bottom right) seems to use the GT mesh for sampling the flow-field.  Shouldn't it be using the predicted surface instead?\n\n\n\n\n---\n\nClarity(5/10)\n\nThere are some confusing descriptions which makes it hard to understand. For example:\n\n1. L97:  instance masks and camera matrices -> does camera matrices mean intrinsic matrices? or the entire projection matrices?\n2. the use of $\\theta$ is a bit misleading: it is used both in v and g, but nowhere else. what does that mean? \n3. Eq.2 is a bit misleading. According to Sec.3.2, there's nontrivial dependencies between each component: $\\Phi$ depends on a integration of $\\Phi$ and $v$, where $v$ depends on aggregated features. Though it makes the presentation simple to look at, it hides all the necessary details to understand the structure of the method. \n\n\n\n---\nSignificance (3/10)\n\nI'm negative about the significance of this paper. Major concerns are:\n\n1. The setup is too simple. Assuming GT depth maps and instance masks is okay, but the quality of the results have to be much better to justify the setup\n\n2. The quality is poor even the setup is way too realistic. Even with GT depth, mask and flow-field the method doesn't solve the problem near perfectly, which makes me believe there's something missing in the method or it is partially broken.\n\n3. I don't see how this could generalize to more realistic setups, such as using a RGBD sensor as input, with estimated instance masks. RGBD sensors are noisy and instance segmentation are not perfect; I would expect the method to degrade even further to such setups, since it is supervised by synthetic data with Gt inputs. \n\n4. Existing method already show real world examples of 4D reconstruction with a RGBD sensor, in real time. I'm not sure how this paper compare to that, or what it adds to the current progress of that problem. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not discuss about limitations of the method.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "f_HclkC6d3m", "reviewer": "Reviewer_7c6w", "summary": "This paper recovers dense 3D shape of a possibly articulating and occluded object given a RGB-D video with known instance masks and camera intrinsics/extrinsics. The paper proposes an approach that relies on two learned modules:\n\na) An implicit velocity field v(p,t) which predicts velocity at point p at time t, using additional image based features for the location p across all timesteps.\nb) A canonical space occupancy f(p), which aggregates pixel-aligned features from the various frames using projection of 3D coordinates obtained by integrating the velocity field (essentially a generalization of PixelNeRF and PIFu, but with a velocity-field).\n\n\nThe approach relies on strong supervision in the form of known 3D shapes as well as dense 3D scene flow (so both the above modules can be directly supervised) and presents results on 5 categories using data from GTA-V.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the positive side, I really like the formulation using the implicit velocity field to model motion. This is a simple and elegant idea which was also used in the OccupancyFlow work, but differs from some other recent approaches which directly predict canonical-to-view deformations at each time-step instead of predicting a velocity field. In context of this, this paper introduces some novel deisgn and architecture choices, and one that future efforts could build upon e.g. integrating pixel-aligned features, and predicting velocity using features from across timesteps - both sensible choices that can help generalization.\n\nFurther, the empirical results do clearly show improvements over static-scene prediction methods, thus demonstrating the benefits of using multiple video frames (although I am not sure if the baselines were modified to provide depth as additional input)\n.\nHowever, although aspects of the technical approach are novel, I would not argue for acceptance. In particular:\n\na) The approach requires very strong supervision for learning i.e. full 3D across timesteps as well as 3D scene flow are required, and even at inference, calibrated RGB-D videos with instance masks are needed. I think this requirement makes the approach not very scalable, and this is in contrast to recent work e.g. NeRFies, NSFF etc which accomplish similar goals, but without such strong training/test time annotations.\n\nb) While the paper claims to perform \"class-agnostic\" reconstruction, the experiments are really on a limited set of categories and I suspect the model does infact rely on category-level priors. To support claims of category-agnostic reconstruction, the work should report experiments where the learned model can generalize to new categories.\n\nc) On a note related to the above, I think the results shown are not of a very high-quality, specially for non-human categories (e.g. the helicopter in main text, truck in appendix)\n\nd) Assuming calibrated RGB-D videos and instance masks is a rather strong supervision, and I would be curious to see comparisons against recent NeRF based methods (e.g. NeRFies can easily to use depth and flow supervision) as well as more classical methods e.g. DynamicFusion.\n\ne) As alluded to above, the key contributions of this work are to integrate PIFu-like design choices into the OccupancyFlow framework. While this is certainly desirable, I do not think it is a significant contribution on its own.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think more discussion on the limitations e.g. what categories can be handled, and the strong supervision required, might be beneficial.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BpMxl1FvAOd", "reviewer": "Reviewer_Jqx8", "summary": "The paper proposed a neural network architecture for reconstructing shape and motion from RGBD sequence. It mainly consists of two modules, (i) infer a canonical space which permits to aggregate visual cues across time. (ii) a 4D transformation module to capture correspondences.  The experiment is done on a synthetic dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tries to solve for a challenging problem, i.e. reconstruct shape and motion from RGBD sequence, which in literature we tend to refer as “dynamic fusion”. The best paper in CVPR15 by Newcombe et al. is perhaps one of the most famous works which try to solve this task. But the authors somehow neglect this line of works, which raises questions of the credibility of the paper. Moreover, the title of the paper is misleading, “... from Monocular videos” by convention refers to using video captured by a single rgb camera, which is an even more challenging setup compared to what is actually used in this paper, i.e. rgbd sensor. In addition, the result presented in this paper looks far less desirable, especially when compared to the visual result presented in the dynamic fusion paper. Also the experiment is only conducted on synthetic data, and the result for non-human subject does not look good enough to back up the “class agnostic” claim.\n\nMy other critics / comments are:\n1) Page 3 Ln. 77 when discussing NRSfM, missing reference to a few most classical paper, e.g. “A Simple Prior-free Method for Non-Rigid Structure-from-Motion Factorization”, Dai te al. CVPR12. \n\n2) Ln. 79, it claims “does not rely on any templates or priors”. I think this is a overclaim, as this work learns from a synthetic dataset, thus actually uses a data-driven prior, which is no less when compared to what is used in NRSfM.\n\n3) I am not fully convinced by the proposed framework. The convention wisdom is that solving the 3D scene flow and 3D shape is a joint task, as both modules should help each other. However, it seems that in this paper, the flow field inference module is sort of an independent process. I think the authors should try to polish their method, and improve the result to be at least comparable to the real state-of-the-arts (e.g. Newcombe et al.).  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received four negative reviews from expert reviewers, who all chose to main their negative recommendations.  The shared concern is the lack of discussion and comparison with multiple related methods, in particular those along the line of DynamicFusion.  While the AC understands the authors' argument on the lack of publicly released code, the AC agrees with the reviewers that given the large number of follow-up methods and the similarities between the approaches, such comparisons are crucial, and the authors should have included a thorough comparison. The authors are encouraged to revise the paper based on the reviews.
Decision: Accept (Poster)