Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors present a class-agnostic framework to reconstruct the dynamic objects (FREDO) from unconstrained monocular videos. The proposed pipeline is designed to recover the accurate shape and articulation and deal with partial visibility. Experimental results on SAIL-VOS 3D dataset show that each proposed module is effective on reconstructing the object.
main_review: Overall, the motivation of the proposed system is described clearly, considering the necessity of a unified (or generalized) framework for the reconstruction of any types of non-rigid objects. However, it is hard to find the explicit contributions of the proposed model from the following reasons. 
(1) The framework is composed of complicated stages which make it difficult to understand where the pure novelty lies. Especially, the main algorithm for finding correspondences between temporal frames is highly dependent on Flow-field, which seems not much different from the previous scene flow estimation such as “Self-Supervised Multi-Frame Monocular Scene Flow (CVPR’21)”. 
(2) Compared to the works [3,31,51], leveraging shape priors and semantic correspondences, it is hard to find the superiority of the proposed system. I would like to see the quantitative and qualitative results between the methods with & without shape priors.

### UPDATE AFTER REBUTTAL ###
I appreciate the authors' feedback and valuable comments from other reviewers. Parts of my main concerns (e.g., the difference from scene flow estimation) are eased with the authors' fair reasonings. However, the experiments with only a synthetic dataset are limited to highlight their contributions. It needs more extensive validations with other datasets as commented in my initial review, and comments from FhQ8 and TAFS. Overall, I update my final score from 5 to 6, leaning towards acceptance.

limitations_and_societal_impact: [Dependency on canonical frame]
The performance of the overall framework is highly dependent on the geometric perception of the canonical frame, $I_c$, during the inference step. Can we selectively optimize the canonical space by capturing the whole frames?
[Limited experiments]
The validation is conducted only on SAIL-VOS 3D dataset, which is hard to show its scalability. Considering the advantage of the proposed model, which is a unified framework without a class information, it would be better to show its capability on the generalization. Also, I think it is necessary to demonstrate some results on a real-world dataset to show its efficacy on the real-world application.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes the novel task of unconstrained reconstruction of non-rigid objects from RGBD videos. With respect to previous works the authors do not use any class specific prior (e.g., human bodies templates) nor assume the objects to be completely visible. Given a RGBD video sequence of an object, enriched with camera poses and segmentation masks, the proposed FREDO system can generate for each frame a full 3D mesh of the object of interest and a dense scene flow field connecting the frames in the sequence. The proposed method builds upon Occupancy Flow and extends it with two main novelties. First, it uses a canonical space where features coming from different frames are projected, then it introduces an aggregation network, based on a transformer, to fuse information across frames before using them to predict shape and flow of the model. 
main_review: ### Summary
The paper is well written and achieves quite impressive results when compared to the competitors, that however were designed for slightly different tasks. The proposed formulation has quite strong assumptions in terms of the expected inputs to the system and has been proved to work only on a synthetic dataset, however the task addressed is very challenging and yet not well explored. This work does some first steps along this exploration and already achieves quite good results. For these reasons and for the general formulation that seems quite sound I’m currently rating this paper as 7. I will highlight below some of the strengths and weaknesses of the method, as well as leave some questions and suggestions for the authors.

### Strengths
+ The task explored in this work is very challenging and not yet very well investigated. Part of the reason for this might be that the main dataset used in this work has been proposed only very recently, but nevertheless it’s a very interesting and relatively new direction of research.
+ The gap in performance between the this method and the baselines/competitors is quite significant. It is however true that this method was the first one specifically designed for this task on this dataset. 
+ The paper organization and the presentation of the method are very good, especially considering that the method has many different components and having a clear explanation in a few pages can be complicated.
+ I liked how the authors spent some space to report failure cases and briefly comment on them

### Weaknesses

1. The assumptions of the framework are quite strong: at inference time the input of the system is a RGBD video sequence with per-frame camera position and instance mask of the object that needs to be reconstructed. It is true that all these inputs can be generated using other models (e.g., segmentation networks) or sensor (e.g., depth sensor + IMU), however all the experiments in the paper use GT information as input, therefore it is not clear how well the model will work in presence of noisy inputs.

2. On top of the inference time assumption described at weakness (1) at training time also GT meshes and scene flows of the object to be reconstructed are required. Given all these requirements it is not clear how this method could be trained in a dataset different from a synthetic one. 

3. The whole framework is trained and tested in a single (synthetic) dataset, it would be interesting to check if and how well does the model generalize to real data with noisy input estimations.

4. The authors set the dimension of the canonical space as fixed along the depth dimension, i.e., the space has always a fixed distance between $z_{near}$ and $z_{far}$.. This seems a bit naive as it does not take into account the dimension of the object being reconstructed nor the “type”, I’m assuming that the movement of a car would be bigger than the one of a human. Moreover from the text it is not clear if the distance between $z_{near}$ and $z_{far}$ is fixed across all the categories or it’s recalibrated for every training. These details should be discussed in the manuscript or supplementary material.

5. Some implementation details could be made a bit more clear and some hyperparameters study could have made the work even more strong. More on this in Question 1 and 2 to the authors


### Questions
1. Some implementations details are currently not very clear in this version of the paper, could you clarify if:
    1. Has one model been trained across the different categories or one model is trained per category?
    2. Does the number of frames $N$ seen at train time need to match the number of frames used at test time?
    3. What is the computational efficiency of the method? How long does it take to generate a mesh aligned to one frame? This should also be discussed in the main paper.
2. Can you clarify the impact of some hyperparameters on the method? In particular:
     1. What is the impact on performance of using more frames per reconstruction? Do performance drops with few frames or do they degrade nicelly?
     2. Is the temporal positional encoding used as input to the transformer network useful? Is there an ablation on it’s impact on performance? Intuitively it shouldn’t add much to the feature being pulled from the frames.
3. Can you clarify the experiment “Oracle experiment to study flow-field” discussed between line 338 and 350? The initial part of the paragraph seems to imply the use of GT meshes and predicted flow, while the last part the use of predicted meshes and GT flow. Which one is the correct interpretation? Please also make this more clear in the text.
4. How does the flow predicted by the model compare to frame to frame discrete scene-flow? Could the two be interchanged or does the formulation need continuous fields? 

### Suggestions
* I would add some of the qualitative comparison to OFlow directly in the main paper and not only in the supplementary.
* Typo: line 550 supplementary: “helpss” -> “helps”

----------------------------------------------------------------------------------------------------------------------------------------------------------------

### Post Rebuttal Update
I would like to thank the authors for replying to my questions. I'm keeping my original score for this paper and would suggest acceptance.

limitations_and_societal_impact: The authors showed some of the limitations of the current work as well as provided insights on what may cause these limitations. There is no discussion on potential societal impact but the work doesn't raise any flag at the current stage since it's mainly targetd at being evaluated in very controlled environment (so far only on synthetic datasets).
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a pixel-aligned implicit function together with a pixel-aligned implicit flow field to address reconstruction of non-rigid objects in video.

The authors make several smart and well-motivated architectural choices, specifically around aggregating pixel-aligned features over time via a transformer, which, combined with strong results, make this paper a significant contribution.
main_review: I argue for acceptance of this paper. 

Exposition. I find the paper expertly written, with the method section as well as the architecture figure being crystal clear. The paper is well structured by bold paragraphs and a step-by-step description of the inference procedure.

Contributions. The contributions are clear: To the best of my knowledge, this is the first paper that tackles prior-based, class-agnostic reconstruction of non-rigidly moving objects from 4D video. The results look quite reasonable for such a first attempt, and outperform previously proposed baselines. Algorithmically, contributions are solid. The extension of the pixel-aligned feature idea to non-rigidly deformable objects is non-obvious - the key insights here are to transport a query point with the flow field in the canonical coordinate space, and subsequently transform into world coordinates, to finally project them into context images. The resulting context set is then elegantly aggregated via a transformer that aggregates features over time. This is a rather complex system, as it also includes the NeuralODE solver for the flow field, and betrays considerable engineering skill.

Related work. I was satisfied with the discussion of related work.

Limitations. I was glad to see the authors included failure cases.

I rank this paper in the top 50% of accepted papers. The only reason why I don't rank it higher is that while considerable, the algorithmic contributions aren't _fundamental_, in that they are (expertly!) executed combinations of previously known methods. In particular, pixel-aligned implicits, flow fields, and template implicits are well-known and explored tools. While I think this will encourage follow-up work, it seems that the methods proposed in this work are mostly specific to this application.

Altogether: Fantastic work!
limitations_and_societal_impact: The authors could point to prior discussions of the potential negative societal impacts of neural rendering and 3D reconstruction, which will allow easy re-targeting, but this is not a strong point.

I find that limitations are addressed well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes FREDO that aims to predict complete and temporally coherent implicit 3D shapes from an RGBD video given tracked instance masks and camera parameters. The model is trained and evaluated on a synthetic dataset SAIL-VOS with 3D ground-truth. It shows improvement over prior methods such as Occupancy flow. 
main_review: Significance and originality
- The problem of dynamic scene reconstruction from monocular RGBD videos has been studied in the past [1,2]. Further discussion and comparison on related work is needed. Specifically, [1,2] use a canonical space and implicit surface representation to aggregate depth scans, and are class-agnostic.
- The proposed solution seem to have the potential to reconstruct surfaces that are entirely occluded in a video, due to its data-driven nature. This could be a significant difference from the prior work of [1,2]. However, I don't see much discussion on its ability to handle entirely occluded surfaces.


Clarity
- For the method section,  I'm able to follow the specific architecture being proposed. However, I find the intuitions of design choices missing. For instance, I don't understand the role of image features. If depth are already used as inputs, why do we need to aggregate image features to predict 3D shapes? 
- In terms of approach, can the authors clarify the difference between FREDO and Oflow? It seems the significant difference is the way of using image features.
- How much degree of visibility does FREDO aim to address? Does it work for objects that are mostly occluded in all frames, or an object that only appear in one frame? 

Experiment
- I like the evaluation on the synthetic 3D dataset, which provides 3D groudtruth for a wide variety of objects in a realistic environment. 
- However, the comparison are not convincing. For one, I found the comparison with PiFUHD to be unfair. PiFUHD takes RGB as input but FREDO takes RGBD. Second, it's not clear how is OFlow adopted to the task of RGBD reconstruction. 
- The experiments are done on synthetic dataset, SAIL-VOS, but not on real data. It would be convincing if it shows generalization ability to real data, such as Youtube-YOS or DAVIS, which provide tracked masks. Can the authors provide such results or discuss what are the challenges of applying it to real data? I'd also suggest  to include comparison with RGBD fusion baselines on existing datasets [1,2] 

Minor
- l35: per-frame segmentation would not be enough. There is another level of difficulty in associating segmentation across frames that is usually handled by object-tracking or video-object segmentation [3].
- An experiment that ablates the effect of temporal aggregation and single-frame prediction would be helpful to understand the approach.

[1] Newcombe, Richard A., Dieter Fox, and Steven M. Seitz. "DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.

[2] Bozic, Aljaz, et al. "Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

[3] Perazzi, Federico, et al. "A benchmark dataset and evaluation methodology for video object segmentation." CVPR. 2016.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Three expert reviewers were initially positive about the paper, whereas a fourth reviewer was borderline negative. After rebuttal the positive reviewers became more positive, especially with additional results on real-world data. The more negative reviewer had requested these as well but did not interact in the reviewing process further.

Overall this paper seems provides a positive step in trying to tackle the very difficult problem of spatio-temporal class-agnostic object reconstruction from video so I am recommending acceptance.
Decision: Accept (Poster)