Class-agnostic Reconstruction of Dynamic Objects from Videos

Abstract: We introduce REDO, a class-agnostic framework to REconstruct the Dynamic Objects from RGBD or calibrated videos. Compared to prior work, our problem setting is more realistic yet more challenging for three reasons: 1) due to occlusion or camera settings an object of interest may never be entirely visible, but we aim to reconstruct the complete shape; 2) we aim to handle different object dynamics including rigid motion, non-rigid motion, and articulation; 3) we aim to reconstruct different  categories  of  objects  with  one  unified  framework. To  address  these challenges, we develop two novel modules.  First, we introduce a canonical 4D implicit function which is pixel-aligned with aggregated temporal visual cues. Second, we develop a 4D transformation module which captures object dynamics to support temporal propagation and aggregation. We study the efficacy of REDO in extensive experiments on synthetic RGBD video datasets SAIL-VOS 3D and DeformingThings4D++,  and on real-world video data 3DPW. We find REDO outperforms state-of-the-art dynamic reconstruction methods by a margin. In ablation studies we validate each developed component.


Reviews and author responses:

In this paper, the authors present a class-agnostic framework to reconstruct the dynamic objects (FREDO) from unconstrained monocular videos. The proposed pipeline is designed to recover the accurate shape and articulation and deal with partial visibility. Experimental results on SAIL-VOS 3D dataset show that each proposed module is effective on reconstructing the object.  Overall, the motivation of the proposed system is described clearly, considering the necessity of a unified (or generalized) framework for the reconstruction of any types of non-rigid objects. However, it is hard to find the explicit contributions of the proposed model from the following reasons. 
(1) The framework is composed of complicated stages which make it difficult to understand where the pure novelty lies. Especially, the main algorithm for finding correspondences between temporal frames is highly dependent on Flow-field, which seems not much different from the previous scene flow estimation such as “Self-Supervised Multi-Frame Monocular Scene Flow (CVPR’21)”. 
(2) Compared to the works [3,31,51], leveraging shape priors and semantic correspondences, it is hard to find the superiority of the proposed system. I would like to see the quantitative and qualitative results between the methods with & without shape priors.

### UPDATE AFTER REBUTTAL ###
I appreciate the authors' feedback and valuable comments from other reviewers. Parts of my main concerns (e.g., the difference from scene flow estimation) are eased with the authors' fair reasonings. However, the experiments with only a synthetic dataset are limited to highlight their contributions. It needs more extensive validations with other datasets as commented in my initial review, and comments from FhQ8 and TAFS. Overall, I update my final score from 5 to 6, leaning towards acceptance.
 [Dependency on canonical frame]
The performance of the overall framework is highly dependent on the geometric perception of the canonical frame, $I_c$, during the inference step. Can we selectively optimize the canonical space by capturing the whole frames?
[Limited experiments]
The validation is conducted only on SAIL-VOS 3D dataset, which is hard to show its scalability. Considering the advantage of the proposed model, which is a unified framework without a class information, it would be better to show its capability on the generalization. Also, I think it is necessary to demonstrate some results on a real-world dataset to show its efficacy on the real-world application.


This paper proposes the novel task of unconstrained reconstruction of non-rigid objects from RGBD videos. With respect to previous works the authors do not use any class specific prior (e.g., human bodies templates) nor assume the objects to be completely visible. Given a RGBD video sequence of an object, enriched with camera poses and segmentation masks, the proposed FREDO system can generate for each frame a full 3D mesh of the object of interest and a dense scene flow field connecting the frames in the sequence. The proposed method builds upon Occupancy Flow and extends it with two main novelties. First, it uses a canonical space where features coming from different frames are projected, then it introduces an aggregation network, based on a transformer, to fuse information across frames before using them to predict shape and flow of the model.   ### Summary
The paper is well written and achieves quite impressive results when compared to the competitors, that however were designed for slightly different tasks. The proposed formulation has quite strong assumptions in terms of the expected inputs to the system and has been proved to work only on a synthetic dataset, however the task addressed is very challenging and yet not well explored. This work does some first steps along this exploration and already achieves quite good results. For these reasons and for the general formulation that seems quite sound I’m currently rating this paper as 7. I will highlight below some of the strengths and weaknesses of the method, as well as leave some questions and suggestions for the authors.

### Strengths
+ The task explored in this work is very challenging and not yet very well investigated. Part of the reason for this might be that the main dataset used in this work has been proposed only very recently, but nevertheless it’s a very interesting and relatively new direction of research.
+ The gap in performance between the this method and the baselines/competitors is quite significant. It is however true that this method was the first one specifically designed for this task on this dataset. 
+ The paper organization and the presentation of the method are very good, especially considering that the method has many different components and having a clear explanation in a few pages can be complicated.
+ I liked how the authors spent some space to report failure cases and briefly comment on them

### Weaknesses

1. The assumptions of the framework are quite strong: at inference time the input of the system is a RGBD video sequence with per-frame camera position and instance mask of the object that needs to be reconstructed. It is true that all these inputs can be generated using other models (e.g., segmentation networks) or sensor (e.g., depth sensor + IMU), however all the experiments in the paper use GT information as input, therefore it is not clear how well the model will work in presence of noisy inputs.

2. On top of the inference time assumption described at weakness (1) at training time also GT meshes and scene flows of the object to be reconstructed are required. Given all these requirements it is not clear how this method could be trained in a dataset different from a synthetic one. 

3. The whole framework is trained and tested in a single (synthetic) dataset, it would be interesting to check if and how well does the model generalize to real data with noisy input estimations.

4. The authors set the dimension of the canonical space as fixed along the depth dimension, i.e., the space has always a fixed distance between $z_{near}$ and $z_{far}$.. This seems a bit naive as it does not take into account the dimension of the object being reconstructed nor the “type”, I’m assuming that the movement of a car would be bigger than the one of a human. Moreover from the text it is not clear if the distance between $z_{near}$ and $z_{far}$ is fixed across all the categories or it’s recalibrated for every training. These details should be discussed in the manuscript or supplementary material.

5. Some implementation details could be made a bit more clear and some hyperparameters study could have made the work even more strong. More on this in Question 1 and 2 to the authors


### Questions
1. Some implementations details are currently not very clear in this version of the paper, could you clarify if:
    1. Has one model been trained across the different categories or one model is trained per category?
    2. Does the number of frames $N$ seen at train time need to match the number of frames used at test time?
    3. What is the computational efficiency of the method? How long does it take to generate a mesh aligned to one frame? This should also be discussed in the main paper.
2. Can you clarify the impact of some hyperparameters on the method? In particular:
     1. What is the impact on performance of using more frames per reconstruction? Do performance drops with few frames or do they degrade nicelly?
     2. Is the temporal positional encoding used as input to the transformer network useful? Is there an ablation on it’s impact on performance? Intuitively it shouldn’t add much to the feature being pulled from the frames.
3. Can you clarify the experiment “Oracle experiment to study flow-field” discussed between line 338 and 350? The initial part of the paragraph seems to imply the use of GT meshes and predicted flow, while the last part the use of predicted meshes and GT flow. Which one is the correct interpretation? Please also make this more clear in the text.
4. How does the flow predicted by the model compare to frame to frame discrete scene-flow? Could the two be interchanged or does the formulation need continuous fields? 

### Suggestions
* I would add some of the qualitative comparison to OFlow directly in the main paper and not only in the supplementary.
* Typo: line 550 supplementary: “helpss” -> “helps”

----------------------------------------------------------------------------------------------------------------------------------------------------------------

### Post Rebuttal Update
I would like to thank the authors for replying to my questions. I'm keeping my original score for this paper and would suggest acceptance.
 The authors showed some of the limitations of the current work as well as provided insights on what may cause these limitations. There is no discussion on potential societal impact but the work doesn't raise any flag at the current stage since it's mainly targetd at being evaluated in very controlled environment (so far only on synthetic datasets).

The paper proposes FREDO that aims to predict complete and temporally coherent implicit 3D shapes from an RGBD video given tracked instance masks and camera parameters. The model is trained and evaluated on a synthetic dataset SAIL-VOS with 3D ground-truth. It shows improvement over prior methods such as Occupancy flow.   Significance and originality
- The problem of dynamic scene reconstruction from monocular RGBD videos has been studied in the past [1,2]. Further discussion and comparison on related work is needed. Specifically, [1,2] use a canonical space and implicit surface representation to aggregate depth scans, and are class-agnostic.
- The proposed solution seem to have the potential to reconstruct surfaces that are entirely occluded in a video, due to its data-driven nature. This could be a significant difference from the prior work of [1,2]. However, I don't see much discussion on its ability to handle entirely occluded surfaces.


Clarity
- For the method section,  I'm able to follow the specific architecture being proposed. However, I find the intuitions of design choices missing. For instance, I don't understand the role of image features. If depth are already used as inputs, why do we need to aggregate image features to predict 3D shapes? 
- In terms of approach, can the authors clarify the difference between FREDO and Oflow? It seems the significant difference is the way of using image features.
- How much degree of visibility does FREDO aim to address? Does it work for objects that are mostly occluded in all frames, or an object that only appear in one frame? 

Experiment
- I like the evaluation on the synthetic 3D dataset, which provides 3D groudtruth for a wide variety of objects in a realistic environment. 
- However, the comparison are not convincing. For one, I found the comparison with PiFUHD to be unfair. PiFUHD takes RGB as input but FREDO takes RGBD. Second, it's not clear how is OFlow adopted to the task of RGBD reconstruction. 
- The experiments are done on synthetic dataset, SAIL-VOS, but not on real data. It would be convincing if it shows generalization ability to real data, such as Youtube-YOS or DAVIS, which provide tracked masks. Can the authors provide such results or discuss what are the challenges of applying it to real data? I'd also suggest  to include comparison with RGBD fusion baselines on existing datasets [1,2] 

Minor
- l35: per-frame segmentation would not be enough. There is another level of difficulty in associating segmentation across frames that is usually handled by object-tracking or video-object segmentation [3].
- An experiment that ablates the effect of temporal aggregation and single-frame prediction would be helpful to understand the approach.

[1] Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. "DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[2] Bozic, Aljaz, et al. "Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] Perazzi, Federico, et al. "A benchmark dataset and evaluation methodology for video object segmentation." CVPR. 2016.
 Yes

 Thanks the authors for putting a dedicated effort on the rebuttal. My remaining concerns were adequately addressed and therefore I'm adjusting the score to 7.

Here are a few notes:
* The qualitative results on 3DPW shows FREDO generalizes well to real-world data, with comparable or even better results than PiFUHD, plus the ability of predicting temporally consistent surfaces. 
* Although I'm still surprised that consistent-video-depth would provide sufficiently-accurate depth inputs in highly-dynamic scenes, this could be explained by the fact that FREDO attends to RGB information more than depth, and depth is mostly useful for aligning the scale over different frames. 
* The comparison with SurfelWarp clearly shows that the data-driven FREDO is able to track larger deformation, which is challenging for classic dynamic 3D reconstruction methods.

 We thank all reviewers for detailed and constructive comments. We are encouraged that the reviewers agree that the task of class-agnostic 4D reconstruction is *“challenging and not yet very well investigated”* (R2-tafs) and that we make a *“significant contribution”* (R3-Gayd). We appreciate that the reviewers think *“the proposed system is described clearly”* (R1-Lqt3, R2-tafs, R3-Gayd) and *“shows a significant performance gap over prior work”* (R2-tafs, R4-FhQ8).  We first address the common questions in this section.

---

### Note on the references

In this material, we use [number] to refer to the related work in the submission, and [character] for new citations. We thank the reviewers for suggesting relevant work which we’ll incorporate in the revision.

---

### Robustness and generalizability: test on real world dataset

To demonstrate the effectiveness of FREDO on real world data, and to alleviate the GT assumption of depth and instance masks, we conduct additional experiments. We use FREDO to reconstruct meshes on the 3DPW dataset [a]. 3DPW is a real-world dataset which contains RGB video sequences for people only. To obtain depth and instance masks, we utilize Consistent Video Depth Estimation (CVD) [b] for monocular depth estimation and Mask RCNN from Detectron2 [c] for instance segmentation. The dataset itself doesn’t provide GT mesh. Instead it labels human 3D key-points, which we feed into the SMPL template for generating GT meshes. Following the same experimental setting in our paper, we adopt PIFuHD as the static reconstruction baseline and Oflow as the dynamic reconstruction baseline. 

**Quantitative results**: we obtain the following results summarized in the table below. We observe the proposed method to significantly improve upon both baselines.

|Metrics|IoU|Chamfer|L2-corr|
| --------- | ------- | ------ | ------- | 
| PIFuHD| 42.3  |0.242|  N/A  |
| Oflow   |39.7   |0.265|0.374 |
| ours     | 46.2  |0.167|0.203 | 

**Qualitative results**: we provide additional qualitative results via this anonymous link: https://anonymous.4open.science/r/fredo_vis-D712/vis1.pdf (the webpage may take a while to load). We find the proposed method to outperform the other two baselines clearly. The proposed method is able to predict detailed, complete, and clean meshes of the dynamic objects (moving humans in this dataset).

[a] Marcard et al., Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera, ECCV 2018  (https://virtualhumans.mpi-inf.mpg.de/3DPW/)

[b] Luo et al., Consistent Video Depth Estimation, SIGGRAPH 2021 (https://github.com/facebookresearch/consistent_depth)

[c] Detectron2 (https://github.com/facebookresearch/detectron2)




 We are happy to hear that our previous responses answered some of the reviewer’s questions. Thanks for the additional comments and questions which we answer next:

### Qualitative results on real data

We provide additional qualitative results on the real-world “3D person in the wild” (3DPW) dataset via this anonymous link: https://anonymous.4open.science/r/fredo_vis-D712/vis1.pdf (the webpage may take a while to load). We find the proposed method to outperform the other two baselines clearly. The proposed method is able to predict detailed, complete, and clean meshes of the dynamic objects (moving humans in this dataset).

---

### Qualitative comparison to dynamic 3D reconstruction methods (e.g., [A,B,C])

We appreciate the reviewer's understanding of the difficulty of comparing to DynamicFusion due to the lack of an open-source codebase.

**Regarding KillingFusion [A]**: there isn't an official implementation. The most-starred repo (https://github.com/saurabheights/KillingFusion) lacks several critical components, such as “Rigid Registration” as the author specifies in the README (see “What needs work” in https://github.com/saurabheights/KillingFusion/blob/1076239/README.md).

**Regarding SurfelWarp [B]**: After one week of debugging to fix code issues (e.g., out of bounds memory accesses) and to ensure that the comparison is fair (e.g., provide exactly identical input to SurfelWarp and our method), we are finally able to run SurfelWarp on SAIL-VOS 3D. Qualitative results can be found at: https://anonymous.4open.science/r/fredo_vis-D712/vis2.pdf (the webpage may take a while to load). We first elaborate on qualitative results (1) before providing more information about how we obtained the results (2).

1. **Qualitative results**: we display meshes generated from point clouds that produced by SurfelWarp. It is apparent that SurfelWarp lacks the ability to hallucinate unseen parts of objects. Moreover, SurfelWarp cannot estimate motion correctly on SAIL-VOS 3D. For example, duplicated hands/feet are observed. This is because SAIL-VOS 3D contains relatively larger motion between images, making inference more challenging. Specifically:

    - VolumeDeform’s scenes, which SurfelWarp is developed upon, contain 1200 frames (UPPER BODY), 552 frames (HOODIE), 876 frames (SUNFLOWER), and 727 frames (UMBRELLA) and only span several seconds (see Supplementary of VolumeDeform https://www.lgdv.tf.fau.de/files/2018/05/innmann2016deform_supplemental.pdf).

    - SAIL-VOS 3D’s scenes are recorded at 9 frames per second.

We think this clearly shows the benefits of the developed method and its ability compared to existing work.

2. **Implementation Details**:
- Note, there are unsolved bugs in the official code repo, as reported in the official issues listed below. We spent considerable time and effort to pinpoint and fix these problems:
    - https://github.com/weigao95/surfelwarp/issues/16
    - https://github.com/weigao95/surfelwarp/issues/34
    - https://github.com/weigao95/surfelwarp/issues/51

- We also identified additional unreported issues (e.g., out of bounds memory accesses) which we fixed.

- The VolumeDeform dataset that the SurfelWarp is developed upon is currently not available for download. See https://www.lgdv.tf.fau.de/publicationen/volumedeform-real-time-volumetric-non-rigid-reconstruction/. We have contacted the dataset authors and they confirmed in an email conversation that the server is down and that they do not have a copy anywhere else. To quote the email from the dataset authors: 
> Sorry that the link does not work. I checked with the administrator of the site and unfortunately there are some technical problems … I don't have the data on a different location, so unfortunately I don't have any solution
>
Unfortunately, we cannot test on the VolumeDeform dataset at this point.

- To ensure that the comparison is fair and rigorous, we adjusted SurfelWarp [B] to use GT masks and GT camera poses from SAIL-VOS 3D. Hence both our method and SurfelWarp [B] use exactly the same data.

**Regarding the neural tracker [C]**: as we clarify in “**Clarification about FREDO uses more information than dynamic 3D reconstruction**”, it needs an additional dynamic 3D reconstruction pipeline. However, the author does not release the code for reconstruction, as mentioned in this issue (https://github.com/DeformableFriends/NeuralTracking/issues/2#issuecomment-756700497).

---


### Clarification about inappropriate baselines (PIFuHD/Oflow)

For **learning based** 4D reconstruction, there aren’t published works that exactly match the problem setting, i.e., dynamic mesh reconstruction from RGBD videos. We are aware of the test time difference between PIFuHD/Oflow and ours, but these are the SOTA methods on the market for 3D static reconstruction and 4D reconstruction respectively. 

In fact, we believe utilizing depth has several advantages: 1) the predictions of FREDO don’t suffer from scale-depth ambiguity whereas PIFuHD/Oflow do; 2) use of depth can help to reduce the domain gap between synthetic data and real-world data; 3) avoiding input noise and splitting development into manageable pieces reveals the importance of individual components.

Moreover, when estimated depth is used (see 3DPW experiments in general response), the comparison between Oflow/PIFuHD and ours is strictly fair since RGB videos are the only input data. To quote the results on 3DPW (also provided in the general response):
 
1. for IoU, FREDO yields 46.2 IoU while PIFuHD and Oflow achieve 42.3 and 39.7 respectively;

2. for Chamfer distance, FREDO’s 0.167 result outperforms both PIFuHD (0.242) and Oflow (0.265);

3. for L2-correlation error, FREDO provides 0.203, which is better than Oflow’s 0.374. Note, PIFuHD does not have the ability to model dynamics.

---


### Clarification about FREDO uses more information than dynamic 3D reconstruction

The differences in the problem setting are as follows:

- **Online v.s. Batched**: We think these two are different settings and a direct comparison is challenging. In our experiments, we adopt mainly “batch” methods (e.g., Oflow) which are closer to our problem setting.

- **RGBD v.s. Depth**: In practice, RGB information is often easily accessible. Importantly, we find RGB information improves temporal coherence significantly (L191-193). This is beneficial for both reconstructing smooth temporal motion/articulation and for handling occlusion. We find “depth-only” methods cannot handle these scenarios. Moreover, contrary to the reviewers statement “[A, B, C] use depth as input and produce a 4D reconstruction”, we want to note that [B, C] also use RGB information. See the following statements for more information on this:

- **DynamicFusion and KillingFusion [A]**: Conceptually, there are pros and cons of using either RGBD [B, C, ours] or only depth [A] as input. However, we think, if available, RGB provides additional useful information. No official code or reliable reproduced version is available to compare to [A] as also stated above.

- **SurfelWarp [B]**: FREDO does not use more information. Indeed, SurfelWarp uses [e] to find correspondences between adjacent observations. [e] in turn utilizes RGB information. See “Run Instructions” in README (https://github.com/weigao95/surfelwarp). For the comparison we provided above, we made sure that both SurfelWarp [B] and the proposed version are based on exactly the same data.

- **Neural tracker [C]**: We want to clarify that Neural tracker also uses RGB-D input. See Sec. 3 in [C] “we are given a pair of source and target RGB-D frames”. Furthermore, note that Neural tracker cannot reconstruct 3D meshes directly as it only tracks correspondences between observations. To obtain a reconstruction, [C] needs to be integrated into an additional dynamic 3D reconstruction pipeline. See Sec. 4.5 in [C] “We introduce our differentiable tracking module into the non-rigid reconstruction framework of Newcombe et al.“.

[e] Wang et al., The Global Patch Collider. CVPR 2016.

---

### Use DeepDeform dataset

Thanks a lot for the suggestion. We have submitted an application to access the DeepDeform dataset. Unfortunately we haven’t yet received a reply.


 First of all, thanks the authors for providing a rebuttal. It clears some of my questions and I have a few more questions based on the reply. In short, it would be nice to see some **qualitative** results on real data. And if possible, a qualitative comparison to any prior work of dynamic 3D reconstruction (such as [A,B,C]) in any test scene that can showcase the benefit of the proposed solution, for example, the ability of completing the non-visible surface. 

Here's my reasoning:
* In terms of evaluation, the authors provided **quantitative** comparisons on a real world dataset (3DPW). However, it is hard to understand the quality of the results without seeing the reconstructed surface. I'm a bit surprised that consistent video depth (CVD) would work to estimate the depth of the highly-dynamic human bodies as in 3DPW. My conjecture is that FREDO would not perform well if the depth estimation is not accurate, given it is trained using groud-truth depth as input. It would be nice to see some **qualitative results** and comparisons with PiFUHD and Oflow. Even the result may not be perfect, it would reveal some potential issues/future direction if someone wants to use FREDO on real-world data. 

* For comparison, the authors provided a detailed discussion on the difference between FREDO and dynamic fusion, which would be nice to add to the paper. I also agree it is difficult to compare to dynamic fusion due to the lack of open-sourced codebase. However, I still think PiFUHD and Oflow are not the most appropriate baselines, both of which does not use depth as test time inputs but FREDO does. In contrast, prior works on dynamic 3D reconstruction (such as dynamic fusion and [A,B,C]) use depth as input and produce a 4D reconstruction. Although there are difference in RGB+depth vs depth, online vs batched, FREDO seem to always use more information than those methods. 

Additional comment (would not affect my rating)
* Instead of using 3DPW, I think it is also fine to use the deep-deform dataset in [C] which provide depth and RGB frames.

[A] Slavcheva, Miroslava, et al. "Killingfusion: Non-rigid 3d reconstruction without correspondences." CVPR. 2017.

[B] Wei Gao and Russ Tedrake, "SurfelWarp: Efficient Non-Volumetric Dynamic Reconstruction." RSS. 2018.

[C] Božič, Aljaž, et al. "Neural non-rigid tracking." NeurIPS. 2020.

 
### [Significance and originality] Compare to DynamicFusion

We tried our best to compare to DynamicFusion and we’ll continue to do so. Please take note of the following:

1. FREDO as a learning-based data-driven approach differs in key aspects from SLAM-based DynamicFusion:
    - **task setting**: DynamicFusion considers a stream-based reconstruction while FREDO focuses on a batch setting. Specifically, DynamicFusion observes input frames one by one while FREDO obtains all input frames.
    - **flow estimation**: DynamicFusion discritizes the non-rigid motion-fields into a predetermined set of deformation nodes (similar to voxels). In contrast, FREDO incorporates a continuous representation, i.e., implicit network, which predicts a velocity field for **arbitrary** points in canonical space (Eq. (4)). In principle, ours could be generalized to higher-resolution space and model finer motion details.
     - **input**: The input of DynamicFusion is solely the depth map of each object, while we use RGBD videos. The color information could help learn smooth articulation and consistent temporal transformation of objects.
     - **reconstruction**: as evidenced in qualitative results provided by DynamicFusion (https://youtu.be/i1eZekcc_lM), it cannot complete the backside of the object if the camera only moves in the front. In many cases (e.g., short video clips), the reconstructed meshes are merely the front-view surfaces even for highly symmetric geometric shapes. In contrast, as a learning-based model, FREDO is able to complete the geometry of the observed objects as shown in the multi-view visualization of Fig. 5 main paper and Fig.S2/S3 in our Appendix. 

2. Due to patent issues or the authors's choice, there is no official code available for DynamicFusion. Unfortunately, it turned out to be very hard to reproduce DynamicFusion results for the entire community. We have tried our best to find, fix and modify third-party-reimplementations. However, after diving into each repo, we find there does not exist a suitable one:
    - The most-starred repo (https://github.com/mihaibujanca/dynamicfusion) doesn’t implement a warp field estimation let alone surface fusion (https://github.com/mihaibujanca/dynamicfusion/blob/master/Report.md). Therefore, it cannot be used to reconstruct either static or dynamic objects. 
    - Another version (https://github.com/swarth100/dynfu) does not implement surface fusion (see Sec. 6.7 of the project report in https://spina.me/assets/files/dynamic-fusion.pdf). This means they do not update geometry in the canonical space. Thus the repo is not able to reconstruct non-rigid objects.
    - The third-party-contributed OpenCV-contrib module (https://github.com/opencv/opencv_contrib/tree/master/modules/rgbd) does not have a correct implementation for the dense warp field interpolation (https://github.com/opencv/opencv_contrib/pull/2719). Therefore, the non-rigid object’s motion estimation is incorrect.
    - Neural Deformation Graphs (NDG) [b] compares to their self-implemented DynamicFusion. However, they have not released their implementation of DynamicFusion (https://github.com/AljazBozic/NeuralGraph/issues/1#issuecomment-866293129).

---

###  [Significance and originality] Compare to Neural Deformation Graphs (NDG)

We kindly disagree that NDG targets the monocular setting. NDG requires the input to be a signed distance field (SDF) grid. To obtain the grid, NDG requires “four calibrated cameras” and converts four depth maps to an SDF with “static volumetric reconstruction” (Sec. 3.1 of NDG paper). In contrast, FREDO assumes **monocular** input. We test the real-world monocular setting where only RGB videos are given as input. Please check the general responses.

Moreover, the datasets used in NDG aren’t available at the time of submission. See answer **"Apply FREDO on existing datasets from [a, b]"**  in a later section of this rebuttal. Consequently, it isn’t possible to compare to NDG.

---

###  [Significance and originality] Potential for reconstructing entirely-occluded surfaces in a video

Thanks for highlighting this. We will note it in the revised version.

---

### [Clarity] Intuition for using image features

Pure depth maps do not provide correspondences between frames. We find image features provide better information for temporal coherence (L191-193). This is beneficial for both reconstructing smooth temporal motion/articulation and for handling occlusion. 

---

### [Clarity] Clarify the difference between FREDO and Oflow

FREDO has several key differences compared to Oflow: 1) a canonical space representation to tackle object occlusions (Sec. 3.1) which Oflow does not consider; 2) we utilize image features to provide more information about temporal coherence (L191-192). Without such information, performance decreases from 32.5 to 26.4 (IoU) (L324-328); 3) we use a transformer to aggregate information from sequences (L164), which improves IoU from 29.7 to 32.5 compared to avg-pool (L316-323) . 

---

### [Clarity] How much degree of visibility does FREDO aim to address?

The videos in the SAIL-VOS 3D dataset are trimmed in a way that the object of interest is always (at least partially) visible. We leave the problem of handling the challenging “gone and re-appear” scenario to future work. 

However, our model is able to hallucinate missing parts of objects if these parts are visible in neighboring frames. We qualitatively show representative results in Fig. 5. To further study FREDO’s ability to handle occlusion, we split results into categories according to the object’s level of visibility. We categorize the validation dataset into 3 different occlusion levels: light (occlusion ratio < 0.25), medium (occlusion ratio $\in [0.25, 0.75]$), and heavy (occlusion ratio > 0.75)  where occlusion ratio is defined as the average ratio between the visible area and the amodal whole region across all frames in the videos.  

The results are shown in the table below. As expected, the more occluded, the more challenging the task. Specifically, IoU, Chamfer distance, and L2 correspondence error worsened from 37.4, 1.02, 1.07 to 26.4, 1.57, 1.44 for heavily occluded objects. We will add a discussion in the revised version. 

| Occlusion | Light | Medium |  Heavy | All |
| --- | --- | --- | --- |  --- |
| IoU | 37.4 | 30.6 | 26.4 | 32.5 |
| Chamfer | 1.02 | 1.35 | 1.57 |  1.27 |
| L2-corr | 1.07 | 1.30 | 1.44 |  1.19 |

---

### [Experiment] PIFuHD only takes RGB

We respectfully disagree. During training, PIFu requires depth as input (Eq.2 and Fig. 2 in [37]).  However, they normalize the depth value into a pre-defined range, i.e., $z_\text{near} = 1.0$ and $z_\text{far} = 10.0$.

Therefore, during inference, PIFuHD uses the normalized depth range and does not require depth as input. Note, in this way, the reconstruction is scale-ambiguous.

On real-world data, we compare ours to the same setting by only taking RGB as input and predict depth maps from RGB images. Ours still outperform PIFuHD by 3.9 IoU / 0.075 Chamfer distance (see general part).

---

### [Experiment] Clarification on baseline Oflow

We use the vanilla version of Oflow as a baseline for dynamic reconstruction (Appendix L516-518). Rather than adapting Oflow to RGBD reconstruction, we devote the effort to developing our own framework, which is more suitable for non-rigid 4D reconstruction. Specifically, constructing canonical space helps tackle occlusion of objects; utilizing image features provides more knowledge about temporal coherence; and transformer-based temporal aggregator improves information aggregation.

To ensure a fair comparison, we instead suggest to use estimated depth from RGB for FREDO. Specifically, we compare to Oflow on a real world dataset 3DPW, which only contains RGB sequence. Please see the responses in the general part.

---

### [Experiment] FREDO on real world dataset

Please see the general responses.

---

### [Experiment]  FREDO on existing datasets from [a, b]

Thanks a lot for mentioning this. Unfortunately we cannot apply FREDO on datasets from [a, b]. For DynamicFusion [a], the original paper does not provide access to their dataset. For Neural Deformation Graphs (NDG) [b], it discusses two datasets, a synthetic one and a self-collected real-world one. 1) **For synthetic data**, NDG uses DeformingThings4D [d]. DeformingThings4D has not been released yet. We have submitted an application for downloading the data but have not received a response. Moreover, it doesn’t contain the RGB sequence of the dynamic meshes but only the non-textured shapes. Therefore, we cannot run these experiments. 2) **For real world data**, NDG created its own dataset with pure depth sensors and does not provide RGB sequences. Besides, this dataset isn’t available yet either.

---

### [Minors] Statement in L35

Thanks for pointing out, we’ll clarify.

---

### [Minors] Ablate the effect of temporal aggregation and single-frame prediction

We interpret “single-frame model” to mean a variant that only operates on one frame during training and testing. Essentially, single-frame prediction defaults to PIFu w/o texture (L155-156). Results in Tab. 1 have already verified the benefits of FREDO. Essentially, to conduct single-frame prediction, we need to remove the canonical space, temporal aggregator, and flow-field estimation. This would remove the key technical contributions of our work.

[a] Newcombe et al., DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time, CVPR 2015

[b] Bozic et al., Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction." CVPR 2021

[c] Perazzi et al., A benchmark dataset and evaluation methodology for video object segmentation, CVPR 2016

[d] Li et al., 4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface, ArXiv 2105.01905


 We thank the reviewer for finding our work “well-motivated”, “expertly executed”, and “well-structured”. We are encouraged and happy to receive positive feedback on the amount of efforts we devote to this work.

We agree with the reviewer that the field of 4D reconstruction is rarely explored, especially in the deep learning era. Template-based methods have greatly pushed the progress of 3D/4D reconstruction of certain objects but also greatly constrained the generalizability of corresponding frameworks. We hope to target a more practical setting where we develop a class-agnosic model for non-rigid objects beyond person. 

As the reviewer pointed out, our work is one of the first attempts to tackle this problem using learning-based methods, i.e., implicit networks. The proposed framework is far from optimal, but we hope it encourages future research efforts. 


 ### [Weakness 1] Strong assumptions on GT camera, mask, and depth

We agree that the use of GT information as input is a strong assumption. However, we think use of ground truth is insightful to the community when developing an end-to-end trainable framework for recovering a pixel-aligned representation of non-rigid dynamic objects which may only be partially visible within a frame. By ignoring input noise and by splitting development into manageable pieces, we can reveal the importance of individual parts. For instance, 1) accurate camera data isolates the development of temporal propagation and fusion (Sec. 3.2a) and a pixel-aligned representation (Sec. 3.2b) from other factors; 2) GT depth and mask let us focus on developing a canonical 4D implicit function (Sec. 3.1). Through experiments, we validated the effectiveness of individual modules for the task of 4D reconstruction (Sec. 4.4).

In practice, these assumptions can be relaxed for real-world data. To demonstrate the robustness of FREDO to non-GT inputs, we evaluate FREDO on a real world dataset which only contains RGB sequences. Please see the general responses for more details.

---

### [Weakness 2] Supervision of the GT mesh and flow

Three trends led to the decision to study the use of ground truth meshes and flows for the challenging task of non-rigid dynamic object-agnostic mesh reconstruction. 1) Supervised learning has been shown again and again to be remarkably successful. In fact, most 3D reconstruction works (e.g., ONet [26], PIFu [36], etc.) are trained using GT synthetic meshes and then generalized to real-world data. 2) Recent advances have shown the benefits of large-scale synthetic data, e.g., SunCG [a], ShapeNet [b], Playing for Benchmark [c], etc. 3) The ability of transfer learning has also been verified in recent synthetic platforms, e.g., embodied AI [d], optical flow [e], and autonomous driving [f] etc. 

Moreover, large-scale real world data for non-rigid dynamic object-agnostic mesh reconstruction isn’t available. Instead, people either apply costly motion capture systems to limited dynamic object categories [g] or utilize 2D supervision and only evaluate on 2D metrics [h]. Instead, we conduct experiments on large-scale 3D datasets and evaluate the results rigorously using perfect pixel-aligned 3D GT.

[a] Song et al., Semantic Scene Completion from a Single Depth Image. CVPR 2017.

[b] Chang et al., ShapeNet: An Information-Rich 3D Model Repository. CVPR 2015.

[c] Richter et al., Playing for Benchmarks. ICCV 2017.

[d] Savva et al., Habitat: A Platform for Embodied AI Research. ICCV 2019.

[e] Butler et al., A Naturalistic Open Source Movie for Optical Flow Evaluation. ECCV 2012.

[f] Dosovitskiy et al., CARLA: An Open Urban Driving Simulator. CoRL 2017.

[g] Slavcheva et al., KillingFusion: Non-rigid 3D Reconstruction without Correspondences. CVPR 2017.

[h] Yang et al., LASR: Learning Articulated Shape Reconstruction from a Monocular Video. CVPR 2021.

---

### [Weakness 3] Generalize to real world dataset

Thanks a lot for this suggestion. We answered in the general responses.

---

### [Weakness 4] Clarification on $z_\text{near}$ and $z_\text{far}$

For each video sequence, we first aggregate all pixels within the instance masks to a world coordinate system using depth maps and camera poses. We then specify $z_\text{near}$ as the smallest $z$ value obtained from the points in the world coordinate system. Next, since we use a **fixed** $z$ range, we obtain  $z_\text{far} = z_\text{near}$ + fixed-range. See discussion in L124-130.

---

### [Weakness 4] Motivation of fixed z range

Note, our goal is a **class-agnostic** framework. Hence we use a fixed distance between $z_\text{near}$ and $z_\text{far}$. We’d expect a category-specific choice or a data-driven selection to improve results. We agree this would be an interesting study in future work.

---

### [Question 1.1] Is model category-specific or category-agnostic

FREDO is class/category-agnostic. We use a **unified** model for all classes. Namely, a **single** model is trained on a dataset consisting of many classes. We then use the same set of trained parameters for 3D model reconstruction of all object classes.

---

### [Question 1.2 & Question 2.1] About the number of frames (window size) $N$.

 [Question 1.2] For simplicity, we keep $N$ the same during training and testing.

 [Question 2.1 Impact of $N$] In our experiments, window size $N$=11 is empirically selected based on validation set performance. To further understand the impact of $N$, we conduct an additional ablation experiment where we test the original model which is trained using $N_\text{train}=11$ with different test-time window sizes $N_\text{test} \in \{ 3,7,11,15,19\} $. The results are shown in the below table. We observe that setting $N$ to a very low value hurts performance as the temporal window fails to capture enough information. Setting $N$ to larger values is also slightly harmful as temporal errors will be propagated and accumulate for longer sequences.  We will incorporate this ablation study of training-time window size in the revised version.

| $N_\text{test}$ |   3    |  7     |  11   |  15   |   19   |
| ------------------- | ------ | ------ | ------ | -------| ------ |
| IoU                  | 24.3  | 30.1 | 32.5 | 31.6 | 30.9 |
| Chamfer          | 1.96  | 1.40 | 1.27 | 1.35 | 1.37 |
| L2-corr            | 1.75  | 1.37 | 1.19  | 1.22 | 1.35 |

---

### [Question 1.3] Computational efficiency

Generation of a mesh from a video during inference consists of two steps: 1) Use FREDO to infer an **implicit** representation for that video (see Step 1-3 in Sec. 3.3); 2) use MISE [26] to generate an explicit mesh representation (marching cube). We report the timing for the above two procedures here. On a GeForce GTX 1080 Ti GPU, the total time to generate the mesh for one object is 1.07s (0.29s inference + 0.78s marching cube). We’ll add this. 

---

### [Question 2.2] Ablation on positional encoding

To verify the effectiveness of positional encoding, we ablate by replacing the positional encoding with the absolute time value. We find this to slightly hurt the results as it achieves 31.6 (-0.9, higher is better) IoU, 1.31 (+0.04, lower is better) Chamfer distance, and 1.26 (+0.07, lower is better) L2 correlation error. 

---

### [Question 3] Clarification on the oracle experiment

Sorry for the confusion. The intention of this oracle experiment is to study the accuracy of the predicted flow-field. During inference, to reconstruct N consecutive frames, we use the GT mesh of the center/canonical frame and propagate this mesh to other frames using the predicted flow-field. If GT flow-fields were used, we would achieve perfect reconstruction results for all N frames. In comparison, we obtained 72.6 IoU and 0.59 Chamfer distance indicating that the current flow-field estimate is good but could be further improved. 

---

### [Question 4] Compare to frame-to-frame discrete scene flow

Conceptually, our implicit flow-field can better model the motion of objects with different moving speed or articulation, as we estimate 3D flow at any continuous point in time $t$, irrespective of whether an image observation at this time is available. In contrast, frame-to-frame scene-flow methods [16, 24, 43] estimate at discrete time-steps. Moreover, discrete scene-flow methods often compute dense flow maps, which is time-consuming and less efficient (L81-83).

Specifically, in practice, the aforementioned discrete scene-flow methods are not suitable for FREDO due to their high computational cost (e.g., one dense 3D flow volume of $512^3$ dimensions would require ~1.5 GB ($3*512^3$ floating numbers) memory and such flow volumes need to be saved between any two consecutive frames) and due to the implicit and continuous nature of our model. Instead, we provide the result of a continuous scene-flow variant in Sec. 4.4 Ablation 3 (L329-334) where the scene-flow is directly estimated rather than through a velocity-field network, which follows the designs in [22]. Even though being simpler, ours clearly outperforms this baseline as it predicts velocity locally and avoids predicting the entire trajectory across long time intervals.

Please also see our related answer to Reviewer Lqt3 in the section of **"[Main review] Difference between our flow-field and scene-flow estimation”**. 

---

### Writing 

Thanks for the detailed comments. We will provide more implementation details, rearrange the content in the Appendix, and correct typos.


 ### [Main review] Contributions

As pointed out in the main paper (L48-50), our main contributions are: 1) we propose a pixel-aligned representation for non-rigid dynamic objects which may only be partially visible in each frame, and 2) we develop an end-to-end trainable framework for this task.

---

### [Main review] Framework is composed of complicated stages

We respectfully disagree. On a high-level, FREDO contains two stages: 1) shape reconstruction in canonical space and 2) motion modeling across time. For the task of 4D reconstruction, we improve prior works on each stage by developing 1) a pixel-aligned canonical 4D implicit representation (Sec. 3.1) and 2) a transformer-based fusion module for temporal modeling (Sec. 3.2a). 

Specifically, FREDO improves over Oflow [30] by benefiting from a canonical space, a pixel-aligned representation, and a transformer-based temporal aggregation. The effectiveness and importance of each component are verified through an ablation study (Sec. 4.4). Concretely, the pixel-aligned representation improves performance by +6.1 IoU/-0.65 Chamfer/-0.24 L2 correlation error (Tab.2 w/o alignment). The transformer-based temporal aggregation improves performance by +2.8 IoU/-0.27 Chamfer/-0.11 L2 correlation error over the average pooling baseline (Tab.2 avg. pooling).

Also note, compared to loosely-related template-based [19] or self-supervised [a] 4D reconstruction, FREDO is significantly simpler as it is end-to-end trainable and requires fewer training losses and stages. Note that [a] appeared on arxiv shortly after the NeurIPS submission deadline.

[a] Yang et al., LASR: Learning Articulated Shape Reconstruction from a Monocular Video, CVPR 2021


---

### [Main review] Difference between our flow-field and scene-flow estimation [b]

Most importantly, we estimate scene flow at any continuous point in time $t$ irrespective of whether an image observation at this time is available. This is due to the fact that we only use an aggregated feature $\mathbf{z}_\mathbf{p}$ to compute the velocity field (Eq. (4)). Note, $\mathbf{z}_\mathbf{p}$ does not depend on any specific time step or visual frame. Instead, we use a Neural ODE [6] to integrate the velocity field to any specified time (Eq. (5)). In contrast, scene-flow works [22, 43, b] estimate scene flow at discrete frames only because the corresponding image is needed. 

Moreover, our flow-field is implemented as a memory-efficient implicit network. In contrast, as also mentioned in the paper (L81-83), scene-flow methods [16, 22, 43, b] compute dense scene-flow maps for all pixels or points explicitly, which is time-consuming and less efficient. For example, one dense 3D flow volume of $512^3$ dimensions requires $3\times 512^3$ float numbers (~1.5GB) and we need to save the flow volumes between any two consecutive frames in the video. 

To validate the effectiveness of the proposed scene-flow, we provide the result of a scene-flow baseline in the main paper Sec. 4.4 Ablation 3 (L329-334, Tab. 2 “implicit flow”). The scene-flow of the baseline is directly estimated rather than being obtained through a velocity-field network. This baseline is similar to the scene-flow module used in [22]. Other scene-flow methods [16, 43, b] require dense flow-prediction which is not suitable for FREDO due to its high computational/memory cost and the implicit and continuous nature of our model.

Finally, we’d like to mention that [b] became publicly available only two weeks prior to the NeurIPS abstract deadline.

[b] Hur and Roth, Self-Supervised Multi-Frame Monocular Scene Flow, CVPR 2021

---

### [Main review] Compare to Non-Rigid Structure From Motion (NR-SFM) methods 

NR-SFM methods [3, 31, 51] use pose information in the form of 2D key points, which requires human annotation. Such pose information needs to be explicitly defined, and thus only works for a few object categories like humans. In contrast, we aim for a more generic setting and reconstruct any object (e.g., truck, airplane) whose 2D key points are hard to define. Moreover, the output of these methods are 3D key-points. To reconstruct the surface mesh, 3D key-points need to be combined with shape templates (e.g., SMPL), and thus fail to generalize to most SAIL-VOS 3D object categories. 

Hence, NR-SFM methods aren’t applicable to SAIL-VOS 3D which does not provide 2D key points.

---

### [Limitations] Dependency on canonical frame

Thanks a lot for the suggestion. As explained in L205, we select the center frame as the canonical frame time because we expect the ODE flow estimation error to increase super-linearly over time. To verify, instead of selecting the center frame as the canonical frame time (L205), we also tried to construct the canonical space at the 1st frame. Performance decreases by 2.4 IoU, 0.16 Chamfer distance, and 0.2 L2-correlation error. 

We agree that it is exciting research to find the optimal time for the canonical space during​​ training or possibly even during inference. However, we think this complicates the framework and we prefer to study the overall setup first. We’d like to defer this optimization to future work.

---

### [Limitations] Limited experiments

Please see responses in the general part. 


The authors propose a pixel-aligned implicit function together with a pixel-aligned implicit flow field to address reconstruction of non-rigid objects in video.

The authors make several smart and well-motivated architectural choices, specifically around aggregating pixel-aligned features over time via a transformer, which, combined with strong results, make this paper a significant contribution.  I argue for acceptance of this paper. 

Exposition. I find the paper expertly written, with the method section as well as the architecture figure being crystal clear. The paper is well structured by bold paragraphs and a step-by-step description of the inference procedure.

Contributions. The contributions are clear: To the best of my knowledge, this is the first paper that tackles prior-based, class-agnostic reconstruction of non-rigidly moving objects from 4D video. The results look quite reasonable for such a first attempt, and outperform previously proposed baselines. Algorithmically, contributions are solid. The extension of the pixel-aligned feature idea to non-rigidly deformable objects is non-obvious - the key insights here are to transport a query point with the flow field in the canonical coordinate space, and subsequently transform into world coordinates, to finally project them into context images. The resulting context set is then elegantly aggregated via a transformer that aggregates features over time. This is a rather complex system, as it also includes the NeuralODE solver for the flow field, and betrays considerable engineering skill.

Related work. I was satisfied with the discussion of related work.

Limitations. I was glad to see the authors included failure cases.

I rank this paper in the top 50% of accepted papers. The only reason why I don't rank it higher is that while considerable, the algorithmic contributions aren't _fundamental_, in that they are (expertly!) executed combinations of previously known methods. In particular, pixel-aligned implicits, flow fields, and template implicits are well-known and explored tools. While I think this will encourage follow-up work, it seems that the methods proposed in this work are mostly specific to this application.

Altogether: Fantastic work! The authors could point to prior discussions of the potential negative societal impacts of neural rendering and 3D reconstruction, which will allow easy re-targeting, but this is not a strong point.

I find that limitations are addressed well.


Meta-review: Three expert reviewers were initially positive about the paper, whereas a fourth reviewer was borderline negative. After rebuttal the positive reviewers became more positive, especially with additional results on real-world data. The more negative reviewer had requested these as well but did not interact in the reviewing process further.

Overall this paper seems provides a positive step in trying to tackle the very difficult problem of spatio-temporal class-agnostic object reconstruction from video so I am recommending acceptance.

Decision: accept