Class-agnostic Reconstruction of Dynamic Objects from Videos
Authors: Zhongzheng Ren
Xiaoming Zhao
Alex Schwing
Venue: NeurIPS 2021 Poster (2021)
Keywords: Dynamic reconstruction
implicit network
3D vision

Abstract: We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due to occlusion or camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape; 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different  categories  of  objects  with  one  unified  framework. To  address  these challenges, we develop two novel modules.  First, we introduce a canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop a 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VOS 3D and DeformingThings4D++,  and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.

Class-agnostic Reconstruction of Dynamic Objects from Videos
Zhongzheng Ren, Xiaoming Zhao; Alexander G. Schwing
University of Illinois at Airana-Champaln https://jason718 github.io/red
Abstract
We introduce REDO, class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due occlusion camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape: 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different categories of objects with one unified framework. To address these challenges, develop two novel modules. First, introduce canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VON VOS 3D and and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.
Introduction
4D (3D space + time) reconstruction of both the geometry and dynamics of different objects is a long-standing research problem, and is crucial for numerous applications across domains from robotics to augmented/virtui reality (AR/VR). However, complete and accurate 4D reconstruction from videos remains a great challenge for mainly three reasons: 1) partial visibility of objects due occlusion camera settings (e.g., out-of-view parts, non-observable surfaces); 2) complexity of the dynamics including rigid-motion (e.g., translation and rotation), non-rigid motion (deformation caused by external forces), and articulation; and 3) variability within and across object categories.
Existing work addresses the above challenges by assuming complete visibility through a multi-view setting [33, 4], or by recovering only the observable surface rather than the complete shape of an object [57]. or by ignoring rigid object motion and recovering only the articulation [59], or by building shape templates priors specific to particular object category like humans [47]. However, these assumptions also limit applicability of models to unconstrained videos in the wild, where these challenges either infeasible only met when taking special care during video capture.
In contrast, we aim to study the more challenging unconstrained 4D reconstruction setting where objects may never be entirely visible. Specifically, we deal with visual inputs that suffer from: 1) occlusion: the moving occluder and self-articulation cause occlusion to change across time; 2) cropped view: the camera view is limited and often fails capture the complete and consistent appearance across time; 3) front-view only: due to limited camera motion, the back side of the objects are often not captured at all in the entire video. Moreover, focus different dynamic object-types
Indicates equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Person (syuthctic) Articulation, rigid motion
CarlMotoryycll (symtletie)) Non-rigid motion, rigid motion
Animal (synthetic) Articulation, rigid motion
Person (rcal-world) Articulation, rigid motion
Figure I: We present REDO, 4D reconstruction framework that predicts the geometry and dynamics of various objects for a given video clip. Despite objects being either occluded (e.g.. car/motorcycle) or partially-observed, REDO recovers relatively complete and temporally smooth results.
with complex motion patterns. These objects could either move in 3D space, or be deformed due external forces, or articulate themselves. Importantly, we aim for class-agnostic reconstruction framework which can recover the accurate shape at each time-step.
To achieve this we develop REDO. As illustrated in Fig. 1, REDO predicts the shape of different objects (e.g., human, animals, car) and models their dynamics (e.g.. articulation, non-rigid motion, rigid motion) given input video clips. Besides the RGB frames, REDO takes as input the depth map, masks of the objects of interest, and camera matrices. In practice these inputs are realistic as depth-sensors are increasingly prevalent [19, 78] and segmentation models [44, 17] are increasingly accurate and readily available, e.g., on mobile devices, If this data isn't It accessible, off-the-sheil tools are applicable (e.g., SIM [71], instance segmentation [26]. video depth estimation [5]].
To address the partial visibility challenge introduced by occlusion or camera settings, REDO predicts temporally coherent appearance in canonical space. To ensure that the same model is able to reconstruct very different object types in a unified manner we introduce a pixel-aligned 4D implicit representation, which encourages the predicted shapes closely align with the 2D visual inputs (& 3.1.. The visible parts from different frames of the video clip are aggregated to reconstruct the same object ($ 3.2). During inference, the reconstructed object in canonical space is propagated to other frames to ensure a temporally coherent prediction (6 3.3).
REDO achieves state-of-the-art results on various benchmarks ($ 4). We first conduct experiments on two synthetic RGBD video datasets: SAIL-VOS 3D [27] and REDO improves over prior 4D reconstruction work [59, 70] by great margin (+5.9 mloU, 0.085 mCham., -0.22 mACD on SAIL-VOS 3D and +2.2 mloU, -0.063 mCham., -0.047 mACD on Deforming/Things4Dtt over OFlow [59]). We then test on the real-world calibrated video dataset 3DPW [86]. We find that REDO generalizes well and consistently outperforms prior 4D reconstruction methods (+ 10.1 mloU, -0. 124 mCham., -0.061 mACD over OFlow). We provide comprehensive analysis to validate the effectiveness of each of the introduced components.
[42].
2 Related work
In this section, we first discuss possible geometry representations. We then review fusion-basse and learning-basee 4D reconstruction approaches, followed by brief introduction of Motion Capture methods. Lastly, we discuss works of dynamics modelling and related 4D reconstruction datasets.
Geometric representations. Representations to describe 3D objects can be categorized into two groups: discrete and continuous. Common discrete representations are voxel grids [22, 90, 8], octrees [68, 82], volumetric signed distance functions (SDFs) [10, 30, 56], point-clouds [1. 18, 67], and meshes [23, 34, 55, 38, 88]. Even though being widely used, these representations pose important challenges. Voxel grids and volumetric-SDFs can be easily processed with deep learning frameworks, but are memory inefficient [76, 52, 64]. Point-clouds are more memory efficient to process [65, 66), but do not contain any surface information and thus fail to capture fine details. Meshes more expressive, but their topology and discretization introduce additional challenges.
To overcome these issues, continuous representations, i.e., parametric implicit functions, are intro- duced to describe the 3D geometry of objects [7, 60, 53] and scenes [54, 73). These methods are not constrained by discretization and can thus model arbitrary geometry at high resolution [70, 81]. In

---

Canenical space é ($3..1
9e Dynamic feat xp
Occupancy
How-field + (S 3.2c)
Hlw-field + 3.2c)
By Fax fg]
Transformer Lux 3.2m)
12
Velocity
embedding
Neural
Static fent Zp
ODE
Transformer
fags
Figure 2: Framework overview. For a query point in canonical space, REDO first computes the pixel-aligned features from the feature maps of different frames using the flow-field &. It then aggregates these features using the temporal aggregator Jagg- The obtained dynamic feature Xp is eventually used compute the occupancy score for shape reconstruction.
practice, discretized representations like a mesh can be easily extracted from implicit functions via algorithms like Marching Cubes [49].
Fusion-based 4D reconstruction. DynamicFusion [57] the pioneering work for reconstructing non-rigid scenes from depth videos in real-time. It fuses multiple depth frames into a canonical frame recover the observable surface and adopt a dense volumetric 6D motion field for modeling motion. Another early work [14] reconstructs dynamic objects without explicit motion fields. Improvements of DynamicFusion leverage more RGB information [28], introduce extra regularization [74, 75], develop efficient representations [20], and predict additional surface properties [24]. Importantly, different from the proposed direction, these works only recover the observable surfaces rather then the complete shape. Moreover, these works often fail to handle fast motion and changing occlusion. Note, these models often have no trainable parameters and are more geometry-based.
Learning-based 4D reconstruction. Reconstruction of the complete shape of dynamic objects is considered hard due big intra-class variations. For popular object categories such as humans or certain animals, supervised learning of object template parameters has been utilized to ease reconstruction [36, 94, 47, 96, 97]. These templates are carefully designed parametric 3D models, which restrict learning to low-dimensional solution space. However, the expressiveness of templates limited. For instance, the SMPL [47] human template struggles to capture the clothing or hair styles. In addition, systems relying on templates are class-specific as different object categories need to be parameterized differently. We think it remains illusive to construct a template for every object category. To overcome these issues, OFlow [59] directly learns 4D reconstruction from scratch. However, it predicts articulation in normalized model space and thus overlooks rigid-motion of the object. It also struggles to handle occlusion. When ground-trute truth 3D models not available, dynamic neural radiance fields (NeRF) [54, 43, 63, 61] learn an implicit scene representation from videos, but are often scene-specifie and don't scale to a class-agnostic setting. Self-supervised methods [91, 35, 41] are promising and learn 4D reconstruction via 2D supervision of differentiable rendering [37, 48]. In contrast, in this work, present class class-agnostic and template free framework which learns to recover the shape and dynamics from input videos.
capture. When restricting ourselves to human modeling, 4D reconstruction is often referred to as motion capture (MoCap). Marker and scanner based MoCap systems [62, 46, 40] are popular in industry, yet inconvenient and expensive. Without expensive sensors, camera-based methods are more applicable but often less competitive. Standard methods rely on a multi-view camera setup and leverage 3D human body models [32, 95, 21, 85]. Active research problems in the MoCap field include template-fir methods [16. 15, 11], single-view depth camera methods [2, 92, 93], monocular RGB video methods [25, 79], and fine-grained body parts (e.g., hand, hair, face) methods [72, 50, 89, 33]. In contrast, this paper studies a more generic class-agnostic setting where multiple different non-rigid objects reconstructed using unified framework.
Motion/Performant
Dynamics modeling. different communities, the task of dynamics modeling is named differently, For 2D dynamics in images, optical flow methods [13, 80, 83] have been widely studied and used. Scene flow methods [43, 45, 31] aim to capture the dynamics of all points in 3D space densely, and are often timecconsuming and inefficient. For objects, non-rigid tracking matching methods [4, 77, 9,
3

---

study the dynamics of non-rigid objects. However, these methods often only track the observable surface rather then the complete shape space. Recently, neural implicit functions [43, 59] have been applied to estimate 3D dynamics, which we adapt and further improve through conditioning on pixel-aligned semporally-aggregated visual features.
4D reconstruction datasets. To support 4D reconstruction, a dataset needs to have both ground- truth 3D models and accurate temporal correspondences. Collecting such dataset real-world setting is extremely challenging as it requires either expensive sensors or restricted experimental settings. To simplify the setting. existing data is either class maass-seecffic (e.g., human) [29, 87, 3]. or of extremely small scale [74, 28], or lacking ground-truth 3D models [5]. Given the progress in computer graphics. synthetic data is becoming increasingly photo-realistic and readily available. Deforming DeformingThings4D [42] provides large collections of humanoid and animal 4D models, but lacks textures and background. SAIL-VOS 3D [27] contains photo-realistic game videos together with various ground-truth. For more details, please see the comparison table in Appendix $ B.
3 REDO
We aim recover the 3D geometry of a dynamic object over space and time given RGBD video together with instance masks and camera matrices. To achieve this we develop REDO which is illustrated in Fig. 2. Specifically, REDO reconstructs space, which we detail in $ 3.1. For this REDO employs a temporal feature aggregator, pixel-aligned visual feature extractor, and flow-field, all of which we discuss in 3.2. These three modules help align dynamics of the object across time and resolve occlusions to condense the most useful information into the canonical space. Lastly, detail inference ($ 3.3) and training ($ 3.4).
object shape and its dynamics in canonical
Notation.
The input of REDO is a fixed-length video clip denoted via , 1y}. consists of N RGBD frames 13 E [0, 1]4xW H (i {1,.... N) }) of width W and height H, each recorded at time- step f4. For each frame 7, we also assume the camera matrix and instance mask maj E {0, 1] [W*H given, where maj indicates the of pixels that correspond object j. For readability, we define the following operations: 1) propagate: transforms temporally in 3D; 2) project: transforms from 3D 2D (image space); and 3) lift: transforms from 2D (image space) to 3D.
3.1 Canonical 4D implicit function
Dynamic objects deform and move across time and are often occluded or partially out-of-views In addition, the depth-size ambiguity also chal- condense
points liftnng
lenges reconstruction algorithms. information about the object across both space and time, we leverage a canonical space C R-3 which aims to capture holistic geometry repre- sentation centered around the object of interest. In our case C' denotes volume-constrained 3- dimensional space. Temporally, the canonical space corresponds to the center frame I, at time- step to with e = [11 + N/22
Canoniaal spocod
Zyar
Figure 3: Canonical Space. Depth points from dif- ferent frames are lifted and aggregated to form an enlarged space for capturing the dynamic objects.
To construct the canonical space, we infer 3D volume around the dynamic object j. For all frames I, E (1....,N) N). we first lift the pixels of instance mask m,j of object j to the world space using the depth map and the camera matrices. These 3D points are then aggregated into canonical space using the camera matrix of the canonical frame. From the aggregated point clouds we infer the horizontal and vertical bounding box coordinates well the object's 's closest distance from the camera, i.e., Znear- However, since the visible pixels only represent the front surface of the observed object, the largest distance of the object from the camera, i.e., Ziar is unknown. To estimate this value, we simply set Ziar to stay a fixed distance from Zucar- We illustrate an example of this canonical space in Fig. where the inferred canonical space (blue volume) is relatively tight and big enough to capture the complete shape and dynamics of the observed object.
To more precisely capture an object inside the volume-constraines canonical space, we use 4D implicit function. For each 3D query point canonical space, this function conditions on temporal feature to indicate whether that point is inside outside the object. Specifically, for query point

---

inside the canonical space, i.e., for E C, the 4D implicit function is defined as:
go xp) RK - [0, 1].
(1)
Intuitively, the higher the score go(:;-), the more likely the point p is part of the object. Here, 0 subsumes all trainable parameters in the framework and Xp E IR is a K K-dimensional dynamic feature which summarizes information from all input frames. Concretely,
Xp = Sage ((Senc (Φ tos 4, va)) (4))4))))))))
(2)
where (+)) R transforms point in canonical space to world space. Meanwhile, + is the flow-field which propagates the point p from t. to t; in canonical space, leveraging a learned 3-dimensional velocity-fies vg. Note, le refers to the time-step of the canonical frame I. while 4 denotes the time-step of frame J; Given the transformed point ( ley li, vo)) E IRS. we use pixel-aligned feature encoder Sane to extract a set of visual representations, each of which comes from one of the N frames 4, i... Vi E {l.... .N. The set function fagg is temporal feature aggregator which merges the temporal information of different time-steps. We'll discuss details of temporal aggregator fagg. encoder fonc. flow-field & and velocity-field vp in 3.2.
This canomical 4D implicit function propagates points across time and extracts the pixel-aligned visual representations from different frames. This differs from: prior works that only consider static objects. Actually, setting N = 1 in Eq. (2) will simplify Eq. (1) to g(p. ( (p). recovering static Pixel-aligned Implicit Function (PIFu) used in [69, 70]; 2) OFlow [59] which encodes the
whole video clip (I,,.., JN] } into one single feature vector, thus loosing spatial information.
3.2 Framework design
In this section, we introduce the temporal aggregator Sage. the feature extractor Senc. and the flow-field & used in Eq. (2). These modules help REDO aggregate information from different time-steps (frames) and model the complex dynamics.
a) Temporal aggregator fags. To deal with partial visibility caused by occlusions camera settings, we develop a transformer-basee temporal aggregator fage as shown in Fig. 2. fage is a set function which computes K di-imensional point features Xp E RRK Assume the flow-field +b is given and object j is of interest. We first propagate query point within the canonical space, i.e., the point E C, to every time-step t,, obtaining locations z (p, te, t,, vo)) E R Vi E {1...., N). We then project each 3D point v ($(p, le: 44, vo)) back to the corresponding 2D image frame using the associated camera matrix. For all points that are projected into the mask area maj. we extract visual representation using the pixel-aligned feature extractor Senc, which we detail below. Note that due to partial visibility, we may not be able to extract features from every frame in the clip. To cope with this, the aggregator fres is designed as transformer-basse [84] set function. For more information, please see the implementation details in $ 4.1 and Appendix A.
b) Pixel-aligned feature extractor fenc. Pixel-aligned features help REDO make 3D predictions that are aligned with the visual 2D input. To achieve this, we develop fenc (q, I,) where the first argument is point in 3D world space and the second argument is frame I,. We first project the point q in world space to the frame I,, and then use a pre-trained convolutional neural net [58] extract the 2D feature map of video frame 7. For points that fall within the instance mask of frame, we extract its visual representation using bi-linear interpolation at the We then append a positional encoding [84] of the frame time-step &, this feature which helps to retain temporal information. We illustrate this process in Fig. 2. The resulting feature is combined with visual cues from other frames to serve as input to the temporal aggregator faes.
location.
c) Flow-field Φ. The flow-field models object dynamics in space and time. For this let ((p, f1, 2g) E C denote flow-field function in canonical space. It computes the position at time-step to of 3D point, whose location is E at time-step t1. To compute the displacement, we define velocity field un(-) which represents the 3D velocity vectors in space and time via
Ue (p Zp f) C x RR
R + C.
Here, E C is a point in canonical space with corresponding static feature Zp E RK computed as
Up = Sage
(4)
5

---

Here, fene (t))), 14) is the feature of world coordinate point (p)p) extracted from frame I,. Note, Zp differs from Xp defined in Eq. (2). The feature zp summarizes information of static locations from all frames 4. This feature is heneficial as it helps capture whether point remains static or whether it moves. The velocity network then leverages this feature 7p to predict object dynamics.
Using the velocity field, we compute the target location at time-step &2 of point originating from location p time-step by integrating the velocity field over the interval [ti, via
(p, t2 vo) = P /, No (p, 11, v0), Zp, t).t..
(5)
Note that ((p, (1, 12, (1) can represent both forward (42 > (1) or backward motion (12 < (1) given the initial location and velocity-field. To solve the flow-field for discrete video time-steps, approximate the above continuous integral equation using a neural-ODE solver [6].
3.3 Inference
To reconstruct objects densely in a clip, the reconstruction/nffereee procedure is summarized as:
Step 1: We construct the canonical space the center frame I, and sample query point set P C uniformly from the inferred space. This step requires no network inference and is very efficient.
Step 2:
We extract static features Zp for all p P using Eq. (4), Next, compute the trajectory of point p. i.e. (p, fo; bi, vi)) for all time-steps t associated with frames L,, E {I,..., NN,, using a neural ODE solver which solves Eq. (5).
Step 3: We then compute dynamic features Xp for all p using Eq. (2).
Step 4: Using Eq. (1) we finally compute the occupancy scores for all p E P. which are then transformed into triangle mesh via Multi-resolution Iso-Surface Extraction (MISH) [53]. Note, the mesh is constructed in canonical space.
Step 5: To obtain the mesh associated with frame I, following Step 2, we use the flow-field + to propagate all vertices of the extracted mesh from the time-step to of the canonical space to time-step which corresponds non-canonical frame 1, After using the function (5() defined in Eq. (2), we obtain the mesh in 3D world space. While various discrete representations could be obtained from implicit representation,
meshes for evaluation and visualization purposes.
3.4 Training
REDO is fully differentiable containing the following parametric components: the temporal aggrega- tor fagy. the feature extractor fene, the velocity-fiele network ug. and the reconstruction network go- For simplicity, we use f subsume all trainable parameters of REDO. To better extract shape and dynamics from given video clips, we train REDO end-to-end using
min Shhppe (D, 8) + Cump (D,
(6)
where D is the training set. Lahape (D, 0) is the shape reconstruction loss in canonical space which encourages REDO to recover the accurate 3D geometry. Cemp (2, 0) is a temporal coherence loss defined on temporal point correspondences. This loss encourages the flow-field Φ to capture the precise dynamics of objects. We detail training set, sampling procedure, and both losses next.
Training set. We train REDO on a dataset D that consists of entries from different videos and of various object instances. Formally, D {(((T,, 4)) V, v. Specifically, data entry for an N-frame video clip includes three components: 1) set of RGBD frames {4)) associated time-steps [t+). as well as corresponding camera matrices E [I....,N : 2) an instance ground-truth mesh V that provdees-tmpooral
aligned supervision. For every vertex E V, we use V; to denote its position at time-step f; 3) the ground occupancy ) in the canonical space. Concretely, for point p C in canonical space, occupancy label y(p) E {0, 1) indicates whether p is inside the object (y = 1) or outside the object (y = 0). As mentioned $ 3.1, the canonical space is chosen so that it corresponds time-step l.., where [(1 + N)/2].
Sampling procedure. To optimize for the parameters during training we randomly sample a set of points p P(V) within the canonical space. P(V) contains mixture of uniform sampling and

---

importance sampling around the ground-truth mesh's 's surface V at time-step &c. Similar strategies are also used in prior works [69, 70].
Shape reconstruction loss. To encourage that the canonical 4D implicit function go accurately captures the shape of objects use the shape reconstruction loss
Shappe (D,0)
E
Σ
BCE (sp(p.
P(V)
PEP(V)
where BCE(..)) represents the standard binary cross-entropy loss.
Temporal coherence loss. REDO models dynamics of objects explicitly through the flow-field Φ, which leverages the velocity-field network up. As the ground-truth correspondences across time are available in V, we define the temporal correspondence loss via the squared error
Ciepp (D,0)
Σ
pV(ve ti -vili..
(8)
NIV
VEV i=
4
Experiments
We first introduce the key implementation details 4.1) and the experimental setup 4.2), followed by the quantitative results ($ 4.3), qualitative results ($ 4.5), and an in-depth analysis ($ 4.4).
4.1 Implementation details
We briefly introduce the key implementation details. Check Appendix $ for more detailed version.
We assume all input clips are trimmed to have N = 17 frames following [59]. During training, clips randomly sampled from original videos which have any length. For videos that shorter than 17 frames, we pad at both ends with duplicated starting and ending frames to form the clips. The validation and split development into manageable pieces, and allows fair comparison with prior work. In practice, dense reconstruction on the entire video is achieved via sliding window method.
Input:
consist of fixed 17-frame clips. This simplified input setting allows to
Reconstruction network: Following [69], the reconstruction network ge is implemented as a 6-layer MLP with dimensions (259, 1024, 512, 256, 128, and -kip-connections, The first layer's 's dimension of 259 is due to the concatenation of visual features (256-dim) and query point locations (3-dim).
Temporal aggregator: Sag uses transformer model with 3 nulti-headed self-attention blocks and a I-layer MLP. Group normalization and skip-connections are applied for each block, and we set the hidden dimension to be 128. To compute the time-encoding, use positional-encoding [84] with 6 exponentially increasing frequencies.
fene is implemented as a 2-stack hourglass network [58] following PIFu [69]. Given the instance mask, we take out the object of interest the picture and resize it to 256 256 before providing it as input to feae. The output feature map has dimensions of 128 x 128 x 256 of 256.
Feature extractor:
spatial resolution 128 128 and feature dimension
Velocity-field network: Up uses a 4-layer MIP with skip-connections following [59], where the intemal dimension is fixed to 128. It takes query points input and adds the visual features the activations after the 1st block. For ODE solvers, we use the Dormand-Prince method (dopri5) [12).
Training: In each training iteration we sample 2048 query points for shape reconstruction and 512 vertices for learning temporal coherence. We train REDO end-to-end using the Adam optimizer [39] for 60 epochs with batch size of 8. The learning rate is initialized 0.0001 and decayed by 10x at the thth and 55th epochs.
4.2 Experimental setup
Dataset. We briefly introduce the three datasets used our experiments below. For more details (e.g., preparation, statistics, examples), please check Appendix $ B.
7

---

Datasct Metrics
SAIL-VOS 3D
DeformingThingsDD..
3DPW
m
mCham
mACD
mloUt
mCham.l mACD) m
mCham.
MACDI
Static reconstruction
ONet [53]
24.5
0.951
60.2
0.260
29.8
0,440
PIFuHD [70]
25.6
0.724
43.8
0,511
37.4
0.363
Dynamic reconstruction
Surfe/Warp [20]
1.03
2.13
3.75
6.53
OFlow [59]
26.0
0.732
1.69
55.2
0,412
0.812
31.5
0.461
0.907
REDO
31.9
0.647
1.47
57.4
0.349
0.765
41.6
0.337
0.846
Table 1: Quantitative results. For both shape reconstruction (mloU and mCham.) and dynamics modeling (mACD), REDO demonstrates significant improvements over prior methods. mACD is not available for static methods and SurfelWarp which don't 't predict temporally corresponding meshes.
1) SAIL-VOS 3D [27]: photo-realistic synthetic dataset extracted from the game GTA-V. It consists of RGBD videos together with ground-truth (masks and cameras). Out of the original 178 object categories. we use 7 dynamic ones: human, car, truck, motorcycle, bicycle, airplane, and helicopter. During training, we randomly sample clips from 193 training videos. For evaluation, we sample 291 clips from 78 validation videos. We further hold out 2 classes (dog and gorilla) as an unseen test set.
EeftommnngThingsAAee [42] is synthetic dataset containing 39 object categories, As the original dataset only provides texture-less meshes, we render RGBD video and corresponding ground-trum (mask and camera) using Blender. Because the original dataset doesn't t provide dataset splits, we create our own. Specifically, during training, we randomly sample clips from 1227 videos. For evaluation, we create a validation set of 152 clips and test set of 347 clips. We hold out class puma with 56 videos as zero-shot test set. We dub this dataset Deforming DeformingThings444444 Things4D++.
2)
3) 3D Poses in the Wild (3DPW) [86]: to test the generalizability of our model, test on this real- world video dataset. Unfortunately, no real-world multi-class 4D dataset is available. Therefore, we test REDO in a class-specific, i.e., class human, setting using 3DPW. This dataset contains calibrated videos, i.e., known camera, and 3D human pose annotation. However; it doesn't 't provide ground-true -tuuh mesh and depth. To extract a mesh, we fit the provided 3D human pose using the SMPL [47] template. To compute depth, we use Consistent Video Depth (CVD) [51] with ground-truth camera data to get temporally consistent estimates. The dataset contains 60 videos (24 training, 12 validation, and 24 testing). During training, we randomly sample clips from all training videos. For evaluation, we evaluate at uniformly sampled clips (10 clips per video) using the validation and test set.
Baselines. We consider the following baselines: 1) Static reconstruction: we adopt state-of-the-eee methods ONet [53] and PIFuHD [70], and train them for per-frame static reconstruction. For fair comparison, we train these two networks in a class-agnostic setting using all the frames in the training videos. 2) Fusion-based dynamic reconstruction: most fusion based methods [57, 74, 75] are neither open-sourced nor reproduced. Among the available ones, we adapt the author-released SurfelWarp [20] due to its superior performance. Since this method is non-parametric and requires no training, we directly apply it on the validation/testing clips. 3) Supervised dynamic reconstruction: REDO learns to reconstruct dynamic objects in a supervised manner. OFlow [59] also falls into this category but it doesn't 't handle the partial observation and rigid-motion Note, REDO uses each clip's s center frame as the canonical space while OFlow uses the initial one. Therefore, for a fair comparison. we set OFlow's 's first frame to be the center of our input clip.
Metrics. To evaluate the reconstructed geometry, report the mean volumetric Intersection over Union (mloU) and the mean Chamfer l distance (mCham.) over different classes at one time- step. i.e., the center frame of the test clip. To evaluate the temporal motion prediction, we compute the mean Averaged (over time) Correspondence &g Distance (mACD) following [59]. As stated before, OFlow's starting frame set to the center frame. For fair comparison, we report mACD on the latter half of each testing clip. We compute mCham. and mACD error in a scale-invariant way following [53, 59, 18]: we use 1/10 times the maximal edge length of the object's 's bounding box as unit one. Even though our network is class-agnostic, we report the mean values over different object categories. Namely, all 'mean' operations are conducted over categories.
4.3 Quantitative results
We present results on all three datasets in Tab. 1. For a fair comparison, we test on the center frame (canonical frame of REDO) of the validation/testing clips. We observe that:
8

---

1) REDO improves upon the static methods for shape reconstruction on SAIL-VOS 3D and 3DW (+6.3/4.2 mloU and -0.077/-0.006 mCham. over best static method). This is because the static methods cannot capture the visual information from other frames in the video clip and thus fail to handle partial visibility. However, REDO performs slightly worse than ONet on due to the unrealistic simplified visual input: 1) the pictures have only one foreground object with neither occlusion nor background, 2) the rendered color is determined by the vertices order and hence provides a visual short-cut for 2D to 3D mapping. Without modeling dynamics, the static baselines are hence easier to optimize. In contrast, SAIL-VOS 3D renders photo-realistic game scenes with diverse dynamic objects, which are much closer to a real-world setting.
2) REDO outperforms basien method SurfelWarp greatly on all benchmarks, as Surfel Warp only recovers the observable surface rather than the complete shape. We didn't 't run SurfelWary on 3DPW as it relies on precise depth as input and crashes frequently using the estimated depth values.
3) REDO improves upon OFlow (+5.9/2.2/10. mloU and -0.085/0.063/0.. 10.01.22. 124 mCham.) for shape reconstruction due to the pixel-aligned 4D implicit representation, whereas OFlow encodes the whole image as a single feature vector and looses spatial information.
4) Regarding dynamics modeling, REDO improves upon OFlow (-0.22/0.047/)) 047/0.061 mACD) thanks to the pixel-aligne implicit flow-field. Note that OFlow normalizes the 3D models at each time-step into the first frame's 's model space and hence fails to capture rigid motion like translation. In contrast, our canonical space is constructed for the entire clip in which REDO predicts a complete trajectory.
As stated in Sec. 3.3, the reconstructed mesh on the center frame is propagated to other frames in the video clip for dense reconstruction. We thus report the per-frame results in Appendix addition, all above results are mean values averaged over object categories to avoid being biased towards the most frequent class, Class-wise results on SAIL-VOS 3D are reported in Appendix $ C.
In
Zero-shot reconstruction. To test the generalizability of REDO, we further test on unseen categories with no fine- tuning in Tab. 2. The result is averaged over three unseen classes: dog and gorilla from SAIL-VOS 3D. and puma from Deforming' DeformingThings4. Things4+++. REDO still greatly outper- forms baselines and doesn't 't fail catastrophically. The per- class results are provided in Appendix Tab. S5.
m
mCham.l
MACD,
ONet
23.1
0.764
PIFu
21.2
0.911
SurfeIWarp
2.06
1.23
OFlow
26.7
0,931
1.18
REDO
38.5
0.479
1.07
Table 2: Zero-shot reconstruction.
4.4 Analysis
In Tab. 3, we provide an ablation study of different com- ponents in REDO using SAIL-VOS 3D data. 1) We first replace the temporal aggregator fuge with average pool- ing layer where features of different frames are averaged and fed into the shape reconstruction and velocity field net- work. The results are shown in the 1st row of Tab. 3 (avg. pooling). The performance drops by -3.6 IoU, +0.065 mCham., and +0.13 mACD. 2) We then study pixel-aligned feature representations Xp. Zp- We replace these two features with the feature map of the entire input frame following OFlow [59] but still keep the transformer to aggregate these feature maps. Results of this ablation are reported in Tab. 3 (w/o alignment). Compared to REDO, this setting greatly hurts the results (-7.8 mloU, +0.290 mCham -0.38 mACD) as the network can longer handle partial observations and the 3D predictions don't 't well align with the visual input, 3) In many real-world tasks, ground-trutt -rrth meshes of different time-steps are not corresponded. Conceptually, REDO could adapt to this setting. This is because all components are differentiable and the flow-field network could be used as a latent module for shape reconstruction. To mimic this setting, we train REDO using only the shape reconstruction loss Lshupe. As shown in Tab. 3 (w/o Lemp)), the model still recovers objects at the canonical frame. However, mACD increases significantly (+1.65).
mloU?
mCham.
MACD]
avg. puoling
28.3
0.712
1.60
wio alignment
24.1
0.937
1.85
w/o Cung
29.4
0.685
3.12
REDO 31.9
0.647
1.47
Table 3: Ablation studies.
4.5 Qualitative results
Fig. 4 shows a few representative examples of REDO predictions on SAIL-VOS 3D and Deform- ing Things4D+++ Please check Appendix & F for more results on real-world data and additional analysis. From Fig. 4 we observe that: 1) REDO able to recover accurate geometry and dynamics of different objects from input video frames. It completes the occluded parts and hallucinates invisible

---

Input
PIFuHD
SurfelWarp
OFlow
Ours
Figure 4: Qualitative results. We illustrate the input frames with the object of interest highlighted. the reconstructed meshes obtained from different methods, and the ground-truth mesh.
Frame
Frame i+1
GT
Pred
Figure 5: Flow-field visualization. REDO accurately recovers the non-rigid motion (e.g., moving forward) but is less precise for small-scale articulation (e.g., hand).
components (e.g., legs and back of humans; tire of the motorcycle) by aggregating temporal information and due to large-scale training. 2) REDO improves upon baselines methods. E.g., PIFuHD struggles handle occlusion and non-human objects, SurfelWarp only predicts the visible surface, and OFlow results are over-smooth as it ignores the spatial information. 3) REDO predictions are still far from perfect compared ground-truth meshes. Many fine- fne-grained details are missing (e.g., clothing of the human, car's 's front-light and tires, etc.).
We also visualize the predicted and ground-truth motion vectors in Fig. 5. REDO successfully models the rigid motion, ie., moving forward, over the entire human body, egg., head, leg, chest, etc. It's less accurate in capturing the very fine-gra grained dynamies, e.g., the hand motion where the ground-truth indicates the hand will open while the prediction doesn't t.
Conclusion
We present REDO, a novel class-agnostic method to reconstruct dynamic objects from videos. REDO implemented canonical 4D implicit function which captures the precise shape and dynamics and deals with partial visibility. We validate the effectiveness of REDO on synthetic and real-world datasets. We think REDO could be generalized wide variety of 4D reconstruccion tasks.
10

---

Acknowledgements & funding transparency statement. This work was supported in part by NSF under Grant #1718221, 2008387, 2045586, 2106825, MRI #1725729, NIFA award 2020-67021- 32799 and Cisco Systems Inc. (Gift Award CG 1377144 thanks for access to Arcetri). 7R is supported by a Yee Memorial Fund Fellowship.
References
[1] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas. Learning representations and generative models for 3d point clouds. In ICML, 2018.
[2] A. Baak, M. Müller, G. Bharaj, H.-P. Seidel, and C. Theobalt. A data-driven approach for real-time full body pose reconstruction from depth camera. Consumer Depth Cameras for Computer Vision, 2013.
[3] F. Bogo, J. Romero, G. Pons-Moll, and M. J. Black. Dynamic FAUST: Registering human bodies in motion. In CVPR, 2017.
[4] A. Bozic, P. Palafox, M. Zollöfer, A. Dai, J. Thics, and M. Nießner, Neural non-rigid tracking. In NeurlPS, 2020.
[5] A. Božić, M. Zollhöfer, C. Theobalt, and M. NieBnerr Deepdeform: Leaming oonriiiid rgb-d reconstruction with semi-supervised data. In CVPR, 2020.
[6] R. Т. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equations. In NeurIPS, 2018.
[7] 7. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019.
[8] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.
[9] A. Collet, M. Chuang, P. Sweeney, D. Gillett, D. Eyseev, D. Calabrese, H. Hoppe, A. Kirk, and S. Sullivan. High-quality streamable free-viewpoint video. TOG, 2015.
[10] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In SIGGRAPH, 1996.
[11] E. de Aguiar, C. Stoll, C. Theobalt, N. Ahmed. H.-P. Scidel, and S. Thrun. Performance capture from sparse multi-view video. TOG, 2008.
[12] J. R. Dormand and J. Prince. A family of embedded runge-kutta formulae. J. Comput. Appl. Math, 1980.
[13] A. Dasovitskiy, P. Fischer, E. 1lg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, 2015.
[14] M. Dou, J. Taylor, H. Fuchs, A. Fitzgibbon, and S. Izadi. 3d scanning deformable objects with single rgbd sensor. In CVPR, 2015.
[15] M. Dou, Khamis, Y. Degtyarev, Davidson, S. Fanello, A. Kowdle, S. Orts, C. Rhemann, D. Kim, Taylor, Kohli, Tankovich, Izadi. Fusion4d: real-time performance capture challenging scenes. TOG, 2016.
[16] Dou, P. Davidson, S. R. Fanello, S. Khamis, A. Kowdle, C. Rhemann, Tankovich, and S. Izadi. Motion2fusion: Real-time volumetric performance capture. TOG, 2017.
[17] Facebook-Ar AL D2go brings detectron2 to mobile.
https://ai www.aaaw.....o.oooooaaaoooa
d2go- wew.ing-ddettctronttooo-oo-teoeo
2021.
[18] H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017.
[19] C. Franklin. Apple unveils new iPad Pro with breakthrough LiDAR Scanner and brings trackpad support to iPadOS. https: : //www/ htwp::wpwpp.....cm/////////./... com/ 2020.
[20] W. Gao and R. Tedrake. Surfelwarp: Efficient non-volumetric single view dynamic reconstruction. In RSS, 2018.
[21] D. Gavrila and L. Davis. Tracking of humans in action: A 3-d model-based approach. ARPA Image Understanding Workshop, 1996.

---

[22] G. Gkioxari, J. Malik, and J. Johnson. Mesh r-cnn. In ICCV, 2019.
[23] T. Groueix, M. Fisher, V. G. Kim, B. Russell, and M. Aubry. AtlaNNet: A Papier- Pacher-Mäché Mäché Approach to Learning 3D Surface Generation, In CVPR, 2018.
[24] K. Guo, F. Xu, T. Yu, X. Liu, Q. Dai, and Y. Liu. Real-time geometry. albedo, and motion reconstruction using single rgb-d camera. TOG, 2017.
[25] M. Habermann, W. Xu, M. Zollhoefer, G. Pons-Moll, and C. Theobalt. Livecap: Real-time human performance capture from monocular video. TOG, 2019.
[26] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask r-cnn. In CVPR, 2017.
[27] Y.-T. Hu, J. Wang. R. A. Yeh, and A. G. Schwing. SAIL-VOS 3D: Synthetic Dataset and Baselines for Object Detection 3D Mesh Reconstruction from Video Data. CVPR, 2021.
[28] M. Innmann, M. Zollhöfer, M. Nicßner, Theobald, and M. Stamminger. Volumedeform: Real-time volumetric non-rigid reconstruction. In ECCV, 2016.
[29] C. lonescu, D. Papava, Olaru, and Sminchisescu. Human3, 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2013.
[30] S. Izadi, D. Kim, O. Hilliges, D. Molyneaux, R. A. Newcombe, P. Kohli, J. Shotton, S. Hodges, D. Freeman,
A. J. Davison, and A. W. Fitzgibbon. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In UIST, 2011.
[31] M. Jaimez, M. Souiai, J. González, and D. Cremers. A primal-dual framework for seal-time dense rgb-d scene flow. In ICRA, 2015.
[32] H. Joo, H. Liu, L. Tan, L. Gui, B. C. Nabbe, I. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh, Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
2015.
[33] H. Joo, T. Simon, and Y. Sheikh. Total Capture: 3D Deformation Model for Tracking Faces, Hands, and Bodies. In CVPR, 2018.
[34] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-to-end Recovery of Human Shape and Pose. In CVPR, 2018.
[35] A. Kanazawa, S. Tulsiani, A. Efros, and Malik. Learning Category-Specite Mesh Reconstruction from Image Collections. In ECCV, 2018.
[36] A. Kanazawa, J. Y. Zhang, P. Felsen, and Malik, Learning 3d human dynamics from video. CVPR, 2019.
[37] H. Kato, Y. Ushiku, and Harada. Neural mesh renderer. In CVPR, 2018
[38] Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Eurggaaphic, 2006.
[39] D. Kingma and J. Ba. Adam: method stochastic optimization. /CLR, 2015.
[40] Li, B. Adams, L. Guibas, and Pauly. Robust single-view geometry and motion reconstruction. TOG, 2009.
[41] X. Li. Liu, K. Kim, S. Mello, Jampani, M.-H. Yang, and Kautz. Self-supervised single-view 3d reconstruction via semantic consistency. In ECCV, 2020.
[42] Y. Li, H. Takehara, Takafumi, Taketomi, B. Zheng, and M. NieBner. 4dcomplete: Non-rigid motion estimation beyond the observable surface. In ICCV, 2021.
[43] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural Scene Flow Fields for Space-Timi Time View Synthesis of Dynamic Scenes. In CVPR, 2021.
[44] H. Liu, R. A. R. Soto, F. Xiao, and Y. J. Lee. Yolactedge: Real-time instance segmentation on the edge. In ICRA, 2021.
[45] X. Liu, C. R. Qi, and L. J. Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, 2019.
[46] M. Loper, N. Mahmood, and M. J. Black. Mosh: Motion and shape capture from sparse markers. TOG, 2014.
12

---

[47] M. Loper. N. Mahmood, J. Romero, G. Pons-Moll, and M. Black. SMPL: A skinned multi-person linear model. TOG, 2015.
[48] M. M. Loper and M. J. Black. Opendr: An approximate differentiable renderer. In ECCV, 2014.
[49] W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3d surface construction algorithm.
TOG, 1987.
[50] L. Luo, H. Li, and S. Rusinkiewicz. Structure-awart aware hair capture. TOG, 2013.
[51] X. Luo, J. Huang, R. Szeliski, K. Matzen, and J. Kopf. Consistent video depth estimation. TOG, 2020.
[52] D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition.
In IROS, 2015.
[53] L. Mcseheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[54] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields view synthesis. ECCV, 2020.
[55] C. Nash, Y. Ganin, A. Eslami, and P. Battaglia. Polygen: An autoregressive generative model 3d meshes. In /CML, 2020.
[56] A. Newcombe, S. Izadi. O. Hilliges. Molyneaux, Kim, Davison, Kohli, Shotton,
S. Hodges, and A. W. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In ISMAR, 2011.
[57] R. A. Newcombe, D. Fox, and S. M. Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In CVPR, 2015.
[58] A. Newell. K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016.
[59] M. Niemeyer, L. Mescheder, M. Occhsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In ICCV, 2019.
[60] J. J. Park, P. R. Florence. J. Straub, R. A. Newcombe, and S. Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR, 2019.
[61] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[62] S. I. Park and J. K. Hodgins. Capturing and animating skin deformation in human motion. TOG, 2006.
[63] A. Pumarola, E. Corona, G. Pons-Moll and F. Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In CVPR, 2020.
[64] C. Qi. H. Su, M. NieBner, A. Dai, M. Yan, and J. Guibas. Volumetric and multi-view cans for object classification on 3d data. CVPR, 2016.
[65] C. Qi. segmentation. In CVPR, 2017.
Mo,
Guibas. Pointuet: Deep leamnngg on point sets for 3d classification and
[66] Qi. L. Yi, H. Su, metric space. In NeurlPS, 2017.
Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in
[67] Z. Ren, I. Misra, A. G. Schwing, and R. Girdhar. 3d spatial recognition without spatially labeled 3d. In CVPR, 2021.
[68] G. Riegler, A. Osman Ulusoy, and A. Geiger. Octnet: Learning deep 3d representations at high resolutions. In CVPR, 2017.
[69] S. Saito, Z. Huang. R. Natsume, S. Morishima, A. Kanazawa, and H. Li. PlFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. In ICCV, 2019.
[70] S. Saito, T. Simon, J. Saragih, and H. Joo. PIFuHD: Multi-Level Pixel-Aligned Implicit Function for HiiglResollttinn 3D Human Digitization. In CVPR, 2020.
[71] J. L. Schönberger and J.-M. Frahm. Structure-frome motion revisited. In CVPR, 2016.
13

---

[72] T. Simon, H. Joo, and Y. Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In CVPR. 2017.
[73] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations. In NeurlPS, 2019.
[74] M. Slavcheva, M. Baust, D. Cremers, and S. Ili.. KillingFusion: Non-rigid 3D Reconstruction without Correspondences. In CVPR, 2017.
[75] M. Slavcheva, M. Baust, and S. Ilie. Sobolevfusion: 3d reconstruction of scenes undergoing free non-rigid motion. In CVPR, 2018.
[76] S. Song and J. Xiao. Deep sliding shapes for amodal 3d object detection in rgb-d images. In CVPR, 2016.
[77] Starck and A. Hilton. Surface capture for performance- animation. CG&A, 2007.
[78] S. Stein. Lidar on the iPhone 12 Pro. https: :/wp::///wwwww cnet. .owt.com/. 2020.
[79] Z. Su, W. Wan, T. Yu, L. Liu, L. Fang, W. Wang, and Y. Liu. Mulaycap: Multi-layer human performance capture using a monocular video camera. TVCG, 2020.
[80] D. Sun. X. Yang. M.-Y. Liu, and J. Kautz. Pwe-net: Cans for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[81] T. Takikawa, Litalien. K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. CVPR, 2021.
[82] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In ICCV, 2017.
[83] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[84] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In NeurlPS, 2017.
[85] D. Vlasic, I. Baran, W. Matusik, and J. Popović. Articulated mesh animation from multi-view silhouettes. TOG, 2008.
[86] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
[87] T. von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
[88] N. Wang, Y. Zhang, 7. Li, Y. Fu, W. Liu, and Y.-G. Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018.
[89] T. Weise, S. Bouaziz, H. Li, and M. Pauly. Realtime performance-hased facial animation. TOG, 2011 1.
[90] J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adveersaiill modeling. In NeurlPS, 2016.
[91] G. Yang, D. Sun, V. Jampani, D. Vlasic, F. Cole, H. Chang, Ramanan, T. Freeman, and C, Liu. Lasr: Learning articulated shape reconstruction from a monocular video. CVPR, 2021.
[92] T. Yu, K. Guo, F. Xu, Y. Dong, 7. Su, J. Zhao, J. Li, Q. Dai, and Y. Liu. Bodyfusion: Real-time capture of human motion and surface geometry using a single depth camera. In ICCV, 2017.
[93] T. Yu. Z. Zheng, K. Guo, J. Zhao, Q Dai, H. Li, G. Pons-Moll, and Y. Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In CVPR, 2018.
[94] J. Y. Zhang, Felsen, A. Kanazawa, and J. Malik. Predicting 3d human dynamics from vidco. In ICCV, 2019.
[95] M. Zollhöfer, M. Nicßner, Izadi, C. Rhemann, C. Zach, M. Fisher, C. Wu, A. W. Fitzgibbon, C. T. Loop, C. Thcobalt, and M. Stamminger. Real-time non-rigid reconstruction using an rgb-d camera. TOG, 2014.
[96] S. Zuffi, A. Kanazawa, and M. J. Black. Lions and tigers and bears: Capturing non-rigid 3d articulated shape from images. In CVPR, 2018.
[97] S. Zuffi, A. Kanazawa, T. Berger-Wood Wolf, and M. J. Black. Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture from Images" In the Wild". In ICCV, 2019.
14


Reviews:
[{"review_id": "qYPvOxKL19", "reviewer": "Reviewer_tafs", "summary": "This paper proposes the novel task of unconstrained reconstruction of non-rigid objects from RGBD videos. With respect to previous works the authors do not use any class specific prior (e.g., human bodies templates) nor assume the objects to be completely visible. Given a RGBD video sequence of an object, enriched with camera poses and segmentation masks, the proposed FREDO system can generate for each frame a full 3D mesh of the object of interest and a dense scene flow field connecting the frames in the sequence. The proposed method builds upon Occupancy Flow and extends it with two main novelties. First, it uses a canonical space where features coming from different frames are projected, then it introduces an aggregation network, based on a transformer, to fuse information across frames before using them to predict shape and flow of the model. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Summary\nThe paper is well written and achieves quite impressive results when compared to the competitors, that however were designed for slightly different tasks. The proposed formulation has quite strong assumptions in terms of the expected inputs to the system and has been proved to work only on a synthetic dataset, however the task addressed is very challenging and yet not well explored. This work does some first steps along this exploration and already achieves quite good results. For these reasons and for the general formulation that seems quite sound I’m currently rating this paper as 7. I will highlight below some of the strengths and weaknesses of the method, as well as leave some questions and suggestions for the authors.\n\n### Strengths\n+ The task explored in this work is very challenging and not yet very well investigated. Part of the reason for this might be that the main dataset used in this work has been proposed only very recently, but nevertheless it’s a very interesting and relatively new direction of research.\n+ The gap in performance between the this method and the baselines/competitors is quite significant. It is however true that this method was the first one specifically designed for this task on this dataset. \n+ The paper organization and the presentation of the method are very good, especially considering that the method has many different components and having a clear explanation in a few pages can be complicated.\n+ I liked how the authors spent some space to report failure cases and briefly comment on them\n\n### Weaknesses\n\n1. The assumptions of the framework are quite strong: at inference time the input of the system is a RGBD video sequence with per-frame camera position and instance mask of the object that needs to be reconstructed. It is true that all these inputs can be generated using other models (e.g., segmentation networks) or sensor (e.g., depth sensor + IMU), however all the experiments in the paper use GT information as input, therefore it is not clear how well the model will work in presence of noisy inputs.\n\n2. On top of the inference time assumption described at weakness (1) at training time also GT meshes and scene flows of the object to be reconstructed are required. Given all these requirements it is not clear how this method could be trained in a dataset different from a synthetic one. \n\n3. The whole framework is trained and tested in a single (synthetic) dataset, it would be interesting to check if and how well does the model generalize to real data with noisy input estimations.\n\n4. The authors set the dimension of the canonical space as fixed along the depth dimension, i.e., the space has always a fixed distance between $z_{near}$ and $z_{far}$.. This seems a bit naive as it does not take into account the dimension of the object being reconstructed nor the “type”, I’m assuming that the movement of a car would be bigger than the one of a human. Moreover from the text it is not clear if the distance between $z_{near}$ and $z_{far}$ is fixed across all the categories or it’s recalibrated for every training. These details should be discussed in the manuscript or supplementary material.\n\n5. Some implementation details could be made a bit more clear and some hyperparameters study could have made the work even more strong. More on this in Question 1 and 2 to the authors\n\n\n### Questions\n1. Some implementations details are currently not very clear in this version of the paper, could you clarify if:\n    1. Has one model been trained across the different categories or one model is trained per category?\n    2. Does the number of frames $N$ seen at train time need to match the number of frames used at test time?\n    3. What is the computational efficiency of the method? How long does it take to generate a mesh aligned to one frame? This should also be discussed in the main paper.\n2. Can you clarify the impact of some hyperparameters on the method? In particular:\n     1. What is the impact on performance of using more frames per reconstruction? Do performance drops with few frames or do they degrade nicelly?\n     2. Is the temporal positional encoding used as input to the transformer network useful? Is there an ablation on it’s impact on performance? Intuitively it shouldn’t add much to the feature being pulled from the frames.\n3. Can you clarify the experiment “Oracle experiment to study flow-field” discussed between line 338 and 350? The initial part of the paragraph seems to imply the use of GT meshes and predicted flow, while the last part the use of predicted meshes and GT flow. Which one is the correct interpretation? Please also make this more clear in the text.\n4. How does the flow predicted by the model compare to frame to frame discrete scene-flow? Could the two be interchanged or does the formulation need continuous fields? \n\n### Suggestions\n* I would add some of the qualitative comparison to OFlow directly in the main paper and not only in the supplementary.\n* Typo: line 550 supplementary: “helpss” -> “helps”\n\n----------------------------------------------------------------------------------------------------------------------------------------------------------------\n\n### Post Rebuttal Update\nI would like to thank the authors for replying to my questions. I'm keeping my original score for this paper and would suggest acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors showed some of the limitations of the current work as well as provided insights on what may cause these limitations. There is no discussion on potential societal impact but the work doesn't raise any flag at the current stage since it's mainly targetd at being evaluated in very controlled environment (so far only on synthetic datasets).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ef7fMlU9l0x", "reviewer": "Reviewer_Gayd", "summary": "The authors propose a pixel-aligned implicit function together with a pixel-aligned implicit flow field to address reconstruction of non-rigid objects in video.\n\nThe authors make several smart and well-motivated architectural choices, specifically around aggregating pixel-aligned features over time via a transformer, which, combined with strong results, make this paper a significant contribution.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I argue for acceptance of this paper. \n\nExposition. I find the paper expertly written, with the method section as well as the architecture figure being crystal clear. The paper is well structured by bold paragraphs and a step-by-step description of the inference procedure.\n\nContributions. The contributions are clear: To the best of my knowledge, this is the first paper that tackles prior-based, class-agnostic reconstruction of non-rigidly moving objects from 4D video. The results look quite reasonable for such a first attempt, and outperform previously proposed baselines. Algorithmically, contributions are solid. The extension of the pixel-aligned feature idea to non-rigidly deformable objects is non-obvious - the key insights here are to transport a query point with the flow field in the canonical coordinate space, and subsequently transform into world coordinates, to finally project them into context images. The resulting context set is then elegantly aggregated via a transformer that aggregates features over time. This is a rather complex system, as it also includes the NeuralODE solver for the flow field, and betrays considerable engineering skill.\n\nRelated work. I was satisfied with the discussion of related work.\n\nLimitations. I was glad to see the authors included failure cases.\n\nI rank this paper in the top 50% of accepted papers. The only reason why I don't rank it higher is that while considerable, the algorithmic contributions aren't _fundamental_, in that they are (expertly!) executed combinations of previously known methods. In particular, pixel-aligned implicits, flow fields, and template implicits are well-known and explored tools. While I think this will encourage follow-up work, it seems that the methods proposed in this work are mostly specific to this application.\n\nAltogether: Fantastic work!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors could point to prior discussions of the potential negative societal impacts of neural rendering and 3D reconstruction, which will allow easy re-targeting, but this is not a strong point.\n\nI find that limitations are addressed well.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UYf-_lnqMfO", "reviewer": "Reviewer_FhQ8", "summary": "The paper proposes FREDO that aims to predict complete and temporally coherent implicit 3D shapes from an RGBD video given tracked instance masks and camera parameters. The model is trained and evaluated on a synthetic dataset SAIL-VOS with 3D ground-truth. It shows improvement over prior methods such as Occupancy flow. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance and originality\n- The problem of dynamic scene reconstruction from monocular RGBD videos has been studied in the past [1,2]. Further discussion and comparison on related work is needed. Specifically, [1,2] use a canonical space and implicit surface representation to aggregate depth scans, and are class-agnostic.\n- The proposed solution seem to have the potential to reconstruct surfaces that are entirely occluded in a video, due to its data-driven nature. This could be a significant difference from the prior work of [1,2]. However, I don't see much discussion on its ability to handle entirely occluded surfaces.\n\n\nClarity\n- For the method section,  I'm able to follow the specific architecture being proposed. However, I find the intuitions of design choices missing. For instance, I don't understand the role of image features. If depth are already used as inputs, why do we need to aggregate image features to predict 3D shapes? \n- In terms of approach, can the authors clarify the difference between FREDO and Oflow? It seems the significant difference is the way of using image features.\n- How much degree of visibility does FREDO aim to address? Does it work for objects that are mostly occluded in all frames, or an object that only appear in one frame? \n\nExperiment\n- I like the evaluation on the synthetic 3D dataset, which provides 3D groudtruth for a wide variety of objects in a realistic environment. \n- However, the comparison are not convincing. For one, I found the comparison with PiFUHD to be unfair. PiFUHD takes RGB as input but FREDO takes RGBD. Second, it's not clear how is OFlow adopted to the task of RGBD reconstruction. \n- The experiments are done on synthetic dataset, SAIL-VOS, but not on real data. It would be convincing if it shows generalization ability to real data, such as Youtube-YOS or DAVIS, which provide tracked masks. Can the authors provide such results or discuss what are the challenges of applying it to real data? I'd also suggest  to include comparison with RGBD fusion baselines on existing datasets [1,2] \n\nMinor\n- l35: per-frame segmentation would not be enough. There is another level of difficulty in associating segmentation across frames that is usually handled by object-tracking or video-object segmentation [3].\n- An experiment that ablates the effect of temporal aggregation and single-frame prediction would be helpful to understand the approach.\n\n[1] Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. \"DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\n[2] Bozic, Aljaz, et al. \"Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[3] Perazzi, Federico, et al. \"A benchmark dataset and evaluation methodology for video object segmentation.\" CVPR. 2016.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "8hQGRtoqX2L", "reviewer": "Reviewer_Lqt3", "summary": "In this paper, the authors present a class-agnostic framework to reconstruct the dynamic objects (FREDO) from unconstrained monocular videos. The proposed pipeline is designed to recover the accurate shape and articulation and deal with partial visibility. Experimental results on SAIL-VOS 3D dataset show that each proposed module is effective on reconstructing the object.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the motivation of the proposed system is described clearly, considering the necessity of a unified (or generalized) framework for the reconstruction of any types of non-rigid objects. However, it is hard to find the explicit contributions of the proposed model from the following reasons. \n(1) The framework is composed of complicated stages which make it difficult to understand where the pure novelty lies. Especially, the main algorithm for finding correspondences between temporal frames is highly dependent on Flow-field, which seems not much different from the previous scene flow estimation such as “Self-Supervised Multi-Frame Monocular Scene Flow (CVPR’21)”. \n(2) Compared to the works [3,31,51], leveraging shape priors and semantic correspondences, it is hard to find the superiority of the proposed system. I would like to see the quantitative and qualitative results between the methods with & without shape priors.\n\n### UPDATE AFTER REBUTTAL ###\nI appreciate the authors' feedback and valuable comments from other reviewers. Parts of my main concerns (e.g., the difference from scene flow estimation) are eased with the authors' fair reasonings. However, the experiments with only a synthetic dataset are limited to highlight their contributions. It needs more extensive validations with other datasets as commented in my initial review, and comments from FhQ8 and TAFS. Overall, I update my final score from 5 to 6, leaning towards acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "[Dependency on canonical frame]\nThe performance of the overall framework is highly dependent on the geometric perception of the canonical frame, $I_c$, during the inference step. Can we selectively optimize the canonical space by capturing the whole frames?\n[Limited experiments]\nThe validation is conducted only on SAIL-VOS 3D dataset, which is hard to show its scalability. Considering the advantage of the proposed model, which is a unified framework without a class information, it would be better to show its capability on the generalization. Also, I think it is necessary to demonstrate some results on a real-world dataset to show its efficacy on the real-world application.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three expert reviewers were initially positive about the paper, whereas a fourth reviewer was borderline negative. After rebuttal the positive reviewers became more positive, especially with additional results on real-world data. The more negative reviewer had requested these as well but did not interact in the reviewing process further.

Overall this paper seems provides a positive step in trying to tackle the very difficult problem of spatio-temporal class-agnostic object reconstruction from video so I am recommending acceptance.
Decision: Accept (Poster)