Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: **Problem**: The paper addresses the problem of discovering and segmenting multiple foreground objects in videos without using supervision. 

**Solution**: The proposed solution involves training on synthetic data and only leveraging optical flow (not RGB) to facilitate easier sim2real generalization. The paper introduces a novel model architecture which tasks as input a sequence of optical flow frames and produces $K$ amodal segmentation masks each associated with an estimated depth ordering. These amodal segmentation maps can be combined using the ordering information to form the final estimated multi-object segmentation map. The architecture involves a transformer based model inspired from DETR which uses K learned queries to produce the K unique outputs.

 
strengths_and_weaknesses: ## Strengths

### Writing
The writing of the paper is very clear and easy to follow. The paper is organized very well to facilitate easier understanding of the details of the model and proposed data generation pipeline. The details of the proposed model is presented in a concise manner covering most necessary aspects. The paper includes detailed discussions of the pros and cons of most design choices. These discussions make it much easier to follow and verify the claims. 

### Proposed Model 
The paper proposes a novel architecture that is interesting and might be applicable to a wider range of domains. The model makes several interesting design choices:
- The choice of leveraging optical flow only from synthetic data is not novel and has been used in other domains. However, the application to the problem of multi-object segmentation is unique. 
- The model goes beyond segmenting objects in individual frames and produces *amodal* segmentation maps for each frame. This is an interesting design choice that seems to be very effective (see more about this in weakness below). 
- The core of the model is heavily inspired from DETR, but the ideas of using this architecture to estimate layer depth and amodal  masks is still novel and interesting. In addition to modifying and adopting the objective of DETR for estimating the masks, the paper introduces a layer ordering loss to accurately estimate the depth order of each object. 

### Synthetic Data Generation
The data generation pipeline proposed in this work is novel. The utility of this data beyond the task of multi-object segmentation is unclear. However, I believe this pipeline could be adopted by other researchers in this domain. 

### Evaluation
The experimental evaluation in this work exhaustively covers standard benchmarks and the necessary ablative studies to verify the claims.
Across two tasks (single object and multi-object segmentation), the proposed model outperforms existing unsupervised learning methods. The ablative studies show the benefits of using amodal segmentation maps as the intermediate output which is a key design choice of the proposed model.

## Weakness

### Supervision and Comparisons
- The paper claims to be an **unsupervised** method for video object segmentation. I'm not sure if this is true based on the conventional usage of the term unsupervised. The proposed method uses synthetic supervision. This can obviously be corrected in the text. However, the bigger issue is the comparisons to existing work. If this work were to be categorized as a supervised learning method, the single object results are only as good as other supervised learning methods and in the multi-object case there is no comparison to supervised learning methods.
- Since the proposed method uses synthetic supervision, it is also *somewhat* unfair to compare to a model trained with real-world human supervision. But I think it is at least important to demonstrate the benefit of synthetic data *i.e.* scalability. Since generating synthetic data is not expensive, if the proposed model can scale in performance with volume of synthetic data and outperform supervised methods that use limited supervision, that would be a compelling result for adopting the proposed model. 

### Amodal Evaluation
One of the interesting aspects of the proposed model is the choice of producing amodal maps as the intermediate step. However, the evaluation of this output is limited to the ablation of comparing to a model using modal maps. It would have been interesting to see how well this model performs on the amodal segmentation task. 
See the following for evaluation protocol:
>Zhan, Xiaohang, et al. "Self-supervised scene de-occlusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

> Xiao, Yuting, et al. "Amodal segmentation based on visible region segmentation and shape prior." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 4. 2021.

questions: - Please discuss the question on unsupervised vs synthetic supervised presented in the Weakness above which is my only major concern.
limitations: Yes, the limitations have been discussion fairly well. 
There is no discussion on the societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a new model for amodal segmentation of videos. Given optical flow,
a Transformer-based neural network predicts an amodal (i.e., unoccluded) segmentation
mask and depth for each object in the input frame. The model is trained using ground
truth supervision on a synthetic dataset created by the authors. When evaluated on
common video instance segmentation benchmarks, the model shows promising results without
requiring further training. The proposed model clearly outperforms previous unsupervised
segmentation methods and even outperform some supervised methods trained on the
respective benchmarks.
strengths_and_weaknesses: Video object segmentation is an active area of research. Predicting layered amodal segmentation masks that can be combined into a modal
segmentation is an uncommon but very natural approach. With this approach, the model is tasked to not
only learn about visible object parts but complete objects. Occlusions arise naturally
when masks are combined. Compared to the more common approach of only reasoning about
visible object fragments this may result in a more useful object representation.

The main weakness of the paper in my view is the comparison with other models. The
proposed model is trained supervisedly on synthetic data and then transferred without
supervision. None of the other models is trained in this way, but either completely
unsupervised or trained supervisedly on the respective benchmark. The performance
improvements on the benchmark are therefore not necessarily due to the proposed model
architecture, but might as well arise due to the different data seen during training.

A fair comparison would be to also train the other methods on the synthetic dataset and
evaluate on the benchmarks. The key question then would be whether the transfer
performance of the proposed model is better than that of previous models, which would be
especially interesting for other models using optical flow.
questions: What is your notion of an "unsupervised" model? It is stated that the proposed model
"achieve[s] state-of-the-art unsupervised segmentation performance". The model however
is trained with ground truth supervision, so in my view the model cannot be termed
unsupervised and the comparison to truly unsupervised methods is misleading. Instead, it
should be made clearer that the proposed method does "unsupervised transfer".

I cannot completely follow the computation of the model segmentation:
- The first accumulated opacity layer $\alpha^0$ should probably be all zeros. As it is
  stated now, it follows that $\alpha^1=2\cdot \hat{M}^0$.
- The modal segmentation masks are added to obtain the opacity layers. This however
  means that in the case of occlusions it might be $\alpha^k > 0$, leading to the
  amodal mask being multiplied with a negative weight. Did you take any means to adress
  this problem?
limitations: One limitation of the model arises due to using optical flow as primary input, which
makes it impossible to segment non-moving objects. This limitation is discussed openly
by the authors and subjected to future work, what I believe is justified.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper considers the problem of segmeting multiple moving objects in a scene by taking optical flow as inputs. The solution is composed of two key ideas: First, the inputs are optical flow, so they can employ simulated videos for training, and the learned model is able to generalize to real videos. Second, a depth-oredered layered representation is used to handle mutual occlusion. Experimental results show improvements upon methods with non-layered representation on single/multiple object segmentation datasets.
strengths_and_weaknesses: Originality:

Good. The key innovations are three fold: (1) object-centric representation; (2) layered representation; (3) amodel representation. Although all three techniques have already been explored in previous literature, there is no such an attempt before to combine them for video object segmentation. Furthermore, most relevant works experiment on simulated datasets, while in this work all experiments are conducted on real world videos, which is much more difficult and more convincing. 

Quality:

Good. The proposed method is easy to understand and shows good performance on all dataset considered, surpassing the most relevant method, Motion Grouping, by a significant margin.

Clarity:

Good but can be improved. In general the paper reads smoothly and is easy to follow. But, perhaps due to limited space, some important details are defered to the supplementary material, which from my point of view, would be better to be placed in the main text. For example, test-time adaptation seems bring large improvements and is important to the final performance, it would be clearer to put relevant details in the main text.

Significance:

The sutdied topic is of great importance. Object-centric, layered repersentation shows promise to become the next generation of vision paradigm.
questions: There might be some hard cases for layered representation. In some cases, objects have mutual occlusion with each other, say a part of object A occludes object B, meanwhile a part of object B occludes object A. Can the proposed method handle such cases? If not, is there any solutions to such cases under the layered representation framework?
limitations: The authors have discussed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a U-net for generating a motion segmentation of a given motion field. In particular the method assumes T snapshots of the field and generates T amodal segmentation masks. The novelty of the paper lies in combining the U-net with a Transformer, i.e. the Transformer receives the latent embeddings of the U-Net encoder and generates output embeddings from which the segmentations masks and an ordering of these masks is inferred. The authors follow the idea of layered motion and assume that each motion segment has unique depth by that the ordering is interpreted as a depth ordering. The model is limited to a fixed size of layers (queries), in this paper three layers. The method is then trained on new synthetic flow data without any manual labelling and applied to a variety of existing datasets and also to new data.
strengths_and_weaknesses: The paper is very well written, very well structured. The methodological approach sounds appropriate and the presentation of the results is well done. I am not working on this specific problem but I believe the authors that the proposed neural network model is novel and not published before. It shows how this generative problem can be elegantly solved by combining a U-Net for the task of segmentation together with a Transformer for the task of inferring spatiotemporal correlations to guide the segmentation. The evaluation seems properly done with partially very good results, although some statements go to far, e.g. in line 227 where it is argued “…large margin …”, although improvements are not that large e.g.for FBMS-59.
I believe that it has been shown that motion segmentation is basically learnable from synthetic data only. If so, one of the co-reviewers more expert than I will definitely refer to this. 
What I miss in the paper is a clear distinction between motion segmentation and occlusion reasoning (2.1D sketch). As the model can only infer motion boundaries, it might miss some of the occlusion boundaries especially for surfaces with interior occlusion boundaries. It is also not clear to the reader how the model behaves for corrupted flow inputs or for more than three motions, e.g. also for a moving camera. Finally, the reader might also be interested to understand why longer sequences give better results or why object boundaries can help improving the layer ordering (line 223).
Monet is also a new method to infer occlusion regions and motion boundaries (H. Kim et al., BMVC’21). I suggest to clarify this in the paper. I also suggest to add X. Zhan et al.’s, CVPR’20 paper to the related work that shows a nice way to compute amodal maps.
questions: How would your method behave in the case of multiple persons walking frontoparallel in the same direction?
Is your method influences by the choice of the optical flow method?
You assume three layers / three motion segments. How did you come to this assumption of three?
What is your idea how to handle this dilemma between appearance and flow for camouflaged cases in your method (line 238)?
limitations: Yes.
review_title: null

Meta-review: metareview: This paper uses synthetic data to train a CNN + transformer architecture for amodal object segmentation from optical flow input.  The model architecture can be viewed as an adaptation of DETR [12] to a different task.  Reviewer ratings lean positive, although there are concerns about experimental validation, as the combination of training regime (using synthetic data) and input modality (optical flow) does not match that of other methods tested on the same datasets; the proposed OCLR system outperforms self-supervised methods, but falls behind the state-of-the-art systems trained on real data, while using different training resources than either class.  The author response partially alleviates this ambiguity, with an additional ablation study comparing to an optical flow based Mask R-CNN model trained on synthetic data.
confidence: Certain
Decision: Accept