Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: **Problem**: This paper addresses the problem on predicting amodal segmentation masks of objects using full sequences of videos depicting the object in different states of occlusion. 

**Solution**: In this work, a novel model is proposed to address this problem which is trained in a self-supervised manner. For each frame, the proposed model predicts an amodal mask and an estimated motion of the object. These outputs are used jointly to estimate the mask in the subsequent frame. The paper proposes a novel objective that provides supervision using the estimated mask on the subsequent frame and the known modal mask on the subsequent frame. 



strengths_and_weaknesses: ## Strengths

### Writing
The text of the paper is very well organize and clearly written. The motivation and inspiration for the work have been explained in great detail. To the best of my knowledge, the related work section thoroughly covers the diversity of relevant literature and discusses the relationship to the proposed work. The details of the proposed model and training methodology has been concisely explained. 

### Novelty 
The architecture of the proposed model is very intuitive. A CNN is used to encode the modal mask, motion and image patch in each frame. An LSTM is used to capture the spatio-temporal relationships. And finally, an auto-encoder is used to predict the amodel mask using the modal mask and the LSTM features. To the best of my knowledge, this is a novel architecture designed specifically for the task of amodal object segmentation. 

One of the key novel contributions of this work is the self-supervised objective that takes advantage of modal masks as supervision to estimate amodal masks. I believe this contribution might be of interest to researchers in this domain and possibly could be extended to other domains. 

### Results

The results demonstrate that the proposed model can outperform existing work and produce better amodal segmentations. The experimental evaluation is very thorough and covers multiple datasets for this task. Furthermore, the paper presents results using test-time adaptation to perform inference on samples outside of the training distribution. These results demonstrate that the proposed method is well-suited for test-time adaptation and demonstrates improved generalization compared to existing work. 


## Weaknesses

Overall there are no major weaknesses of the proposed approach. However, some of the design choices are not entirely explained. I have listed these questions in the Questions section below.
questions: Some design choices of the proposed method are not entirely explained or intuitive:
- As explained in Line 182-185, usage of the LSTM based model leads to the "cold start problem" *i.e.* amodal predictions on the first few frames are going to be poor due to lack of prior video data. While a workaround has been proposed, wouldn't a more intuitive choice be a transformer-like architecture or even a bidirectional LSTM?
- Why is $\delta V$ necessary? It seems like $V$ and $I$ for each frame should be sufficient to estimate the amodal masks. It would be interesting to see this ablation. 
- In addition to the previous question, is the image patch also really necessary to perform this inference? In theory, it should be possible to complete masks given long enough/diverse enough sequences of masks only. 
limitations: The limitations have not been thoroughly discussed. It would be useful to see at least a few examples of failure cases.
The societal impact also has not been discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors mainly consider the amodal segmentation task for moving object in a video. hey propose a self- supervision strategy and a test-time adaptation pipeline for it. It achieves SOTA results on the FISHBOWL and KINS-Video-Car benchmark.
strengths_and_weaknesses: This paper is well written and easy to follow. The method's novelty is based on the intuition that the occluded part of an object can be explained away if that part is visible in other frames. The core idea makes sense and is novel in some, even in most cases. But according to the Theoretical Analysis on The Loss Design of SaVos in the supplementary material, ‘… with the following assumptions, we claim that the region covered, … 3) each pixel of this object is visible in at least one frame. …’ the assumption might be too strong and it is easy to point out the opposite examples, such as Fig. 3 and the video in the supplementary material. 
questions: My main concern is could the authors ease the strong assumption if would still continue in this direction. Maybe the authors could provide the total number of the cases under the strong assumption. Or could provide one option for the rest cases? 
limitations: The paper misses on some important baseline and comparisons in Structured generative model space, that do AMODAL RGB prediction and segmentation on videos, such as PARTS Kabra et al, Scalor (https://arxiv.org/abs/1910.02384), Flow Capsules (https://arxiv.org/abs/2011.13920) and SQAIR (https://arxiv.org/abs/1806.01794).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for amodal video object segmentation that does not rely on amodal segmentation annotations during training. Starting with image, optical flow and conventional segmentation as input, the method estimates an amodal mask and amodal motion for each instance and propagates this mask through time using the estimated flow. The learning signal comes from consistency terms with the input mask and flow through time.
strengths_and_weaknesses: Strengths
The paper has a clear motivation for learning amodal segmentation without amodal supervision. The writing is clear and the explanations are comprehensive.

The method itself is intuitive, by using the assumption that cumulatively, over the course of a video most of the object will be visible.

The results show good performance as well as the importance of test-time fine-tuning and generalization to unseen objects and sequences.


Weaknesses
Positioning and comparisons
The paper is missing a discussion and positioning relative to many recent works in the area:
-Amodal segmentation
Nguyen, Khoi, and Sinisa Todorovic. "A weakly supervised amodal segmenter with boundary uncertainty estimation." ICCV 2021, performs image-based amodal segmentation.
Ling, Huan, et al. "Variational amodal object completion." NeurIPS 2020, does amodal completion per object instance in single frames.
Zhou, Qiang, et al. "Human De-occlusion: Invisible Perception and Recovery for Humans." CVPR 2021, learns occlusion for people.
Zheng, Chuanxia, et al. "Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition.", IJCV 2021
-Weakly supervised instance segmentation
Arun, Aditya, C. V. Jawahar, and M. Pawan Kumar. "Weakly supervised instance segmentation by learning annotation consistent instances." ECCV 2020
The setup is quite similar to weakly supervised amodal segmentation - of course here applied to videos instead of images - so it would make sense to also classify this work as weakly supervised (uses segmentation and flow input) instead of unsupervised. 

Influence of input accuracy
As the method uses optical flow and instance segmentation input, it would be interesting to understand the dependency of the method on the accuracy of that input. Instead of using the GT on synthetic datasets, one could utilize a sota (supervised) method to estimate flow and instance segmentation and compare the results to GT inputs. As the paper is making a case for _unsupervised_ amodal segmentation it could be achieved through the use of unsupervised instance segmentation and unsupervised optical flow methods, however likely at the cost of performance as these methods fall behind their supervised counterparts. 

questions: Questions
The questions follow the weaknesses above.
1. How does the paper position itself compared to the related work above? This includes a potential discussion on unsupervised vs. weakly supervised methods.

2. What is the influence of the accuracy of the input optical flow and segmentation on the  performance of the method?

3. It is somewhat difficult to attribute the performance gains over [42] as the method uses a different architecture as well as a different formulation. What influence do the proposed architecture changes have on the final performance?

4. In table 1, why is the supervised baseline called “supervised oracle”? It is useful to compare to a supervised baseline, but what makes it an oracle in addition?

limitations: Some limitations are discussed in the appendix. It could be helpful to expand them with actual examples from the results.
Societal impact is discussed only very briefly: one line in the appendix. This could be expanded considerably. It is important to discuss that these models often hallucinate segmentations of parts of objects that are never actually visible and thus cannot be trusted.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a self-supervised method (SaVos) for amodal segmentation that tracks shapes of partially occluded objects in video frames to learn the complete (amodal) shape. Given visible modal segmentation masks, the method computes amodal masks and objects’ motion by relying on both spatiotemporal and type priors. The method generalizes well by applying one-shot test-time adaptation given that the focus is on the spatiotemporal prior in order to retrieve the output. The type prior is however useful in scenarios where some parts of the objects are never visible.
strengths_and_weaknesses: Strengths

+ The paper is very well written paper. The explanations are complete and make the paper is easy to follow.
+ Amodal segmentation is an arduous task and the key ideas to solve it are very intuitive and relevant.
+ Although the method is a combination of well known techniques, it is valuable in the the sense that there are improvements compared to the presented SotA even after a drastic distribution shift.
+ The connection to human vision in the first paragraph of the introduction is inspiring and informative.

Weaknesses

- The related work is not adequately mentioned or compared with. It should at least include [A] Ling et al. (NeurIPS 2020) and [B] Sun et al. (CVPR 2022). In the experiments, the baseline used to compare the results should be Ling et al. in addition to Zhan et al. (CVPR 2020), since it is a newer and better performing model. Moreover, similarly to what has been done in Sun et al., it would be more informative to present results as a function of the different occlusion levels, which would enable studying the effect of increasing amount of partial occlusion.
- In the experiments, the authors do not compare to any method that uses temporal information, which gives the proposed model an important advantage over related work. The authors state in lines 41-42 “traditional optical flow […] require[s] explicit visual correspondence across pixels, which would be impossible due to occlusions”. However, I actually think that aggregating optical flow across frames into a complete object shape would be the proper baseline for the proposed mode (also including the input provided, i.e. assuming rigid objects, slowly moving object, ground-truth bounding box and visible segmentation)?
- The proposed method makes very strong assumptions regarding the supervision at training time. In particular, the model assumes that the object is perfectly discovered, including ground-truth bounding box, instance label and visible segmentation mask. Therefore, referencing this approach as self-supervised is somehow misleading. Moreover, these assumptions are rather unrealistic in real world scenarios, therefore the experiments should study how sensitive the proposed method when those inputs are predicted, e.g. using a Faster-RCNN model. 
- The authors state in lines 275-276 “occluded part only occupies a small portion of the entire full mask, thus mean-IoU on the occluded part is a better indicator on the model performance”: This does not mention that this indicator is also significantly more sensitive to noise. Results should include standard deviation estimates to ensure that the uncertainties are considered when comparing results.
- It would be worthwhile to see how does this method generalizes to additional categories i.e. see how much the one-shot test-time adaptation to a new domain “unlearn” previous learned type priors, e.g. performing training on FISHBOWL, adaptation on KINS-Video-Car and re-evaluation on FISHBOWL would give enlightening clues about the generalization capacity of the method.
- The authors state in the Supplementary material lines 140-141: “this makes it easy to squeeze out type information since type is a concise representation”: Were there any experiments performed to support this statement? It would be interesting to underline the possibility to apply classification or clustering to see if it is possible to extract type information as it was mentioned.
- On what grounds are these statements based on? “reasonably learned” (l. 8-9), “as long as the deformation can be reasonably learned” (l. 56), “This module also learns reasonable deformation over time” (l. 216) 
- The paper mentions that the SAIL-VOS dataset is not used because it “has frequent camera switches” (l.79). It would still be interesting to study the performance of the model on that dataset. How long does a video sequence need to be for the proposed model to perform well compared to related work?

Minor:

- Concerning the alpha channel in the bi-direction prediction, does it help only at the beginning when the cold start problem is present or is it also useful at any point after? An analysis of the weights distribution of this channel to further detail its influence would be a great added value.
- Reference to figures, equations and sections (especially in sec. 3.1) are inconsistent and sometimes missing (l. 175 and l.178). Please make sure to precise what you are referring to in order to ensure clarity.
- The paper mentions that a limitation of related works is that they need to perform object discovery (l.  87.). At that location in the text it should be mentioned that this work assumes that the object is perfectly discovered, including ground-truth bounding box, instance label and visible segmentation mask.

============================ POST REBUTTAL==========================
After reading the other reviews and the rebuttal I increase my score to 6 and vote for accepting the paper. I very much appreciate the detailed comments of the authors which addressed my concerns and provide important clarifications regarding the points that I misunderstood. I think this is a solid work with moderate contribution. Remaining weaknesses to me are that it was mostly studied on rather simple video datasets and does not generalize to image collections such as COCO-A or OccludedPASCAL3D+ that are standard evaluation benchmarks in related works, due to the necessity for video data.
questions: -
limitations: The paper does not address the limitaitons of the method.
review_title: null

Meta-review: metareview: The paper develops a system for amodal object segmentation in video, trained in a self-supervised manner by requiring consistency between amodal and modal masks, as well as amodal masks estimated for a frame and those temporally propagated according to estimated object motion.  After rebuttal and discussion, all reviewers favor accepting the paper, citing novelty of the approach and convincing results.  The Area Chair agrees with the reviewer consensus.
confidence: Certain
Decision: Accept