Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors create a large new video dataset for evaluating instance segmentation models in the presence of occlusion.  The authors empirically demonstrate that occlusion is an important problem by showing the substantial performance degradation of many state-of-the-art models in the presence of occlusion.  This hypothesis has been supported by other recent work in image classification (e.g., Compositional neural networks Kortylewski et al., 2020), however, this work still adds substantial value by evaluating many state-of-the-art models, and also studying this problem in the context of video instance segmentation.   Although this is an important problem, as far as I'm aware there are no benchmarks for occlusion, or even datasets that have labeled real-world occlusions that could support the evaluation of models in the presence of occlusion.  In summary, I think this is an important problem and this benchmark represents a major contribution to support research towards solving this problem. 
strengths: 1. The authors create a large dataset of videos with annotations to support instance segmentation
2. The authors annotate occlusions in their dataset to enable evaluation of models in the presence of varying degrees of occlusion
3. The authors provide strong empirical results to support the hypothesis that occlusion is indeed an important problem to study.  
4. The authors have already solicited solutions for their benchmark, and they discuss some of the best-performing models.  
weaknesses: 1. The authors show strong empirical evidence that the performance of contemporary instance segmentation models degrades rapidly in the presence of occlusion.  However, the authors use this to claim (e.g., in Section 5) that more work needs to be done to solve occlusion.  This assertion assumes however that better performance could be achieved with a more sophisticated model: how do we know this is true though?  And if we can achieve better performance in the presence of occlusion, then how much?  At what point can we say that the problem is approximately solved? 

While I largely agree with the authors' assertion, it seems to me that it still lacks strong empirical or theoretical evidence.  This is because occlusion does degrade model performance in an irrecoverable way; occlusion reduces the information available to a recognition model (computer or human) to make a decision.  Therefore, how do we know when we achieve the best possible performance, given the occlusion present?   It would be really helpful if the authors could discus this issue, and/or provide some estimate of the "upper bound" achievable performance using a human annotator (for example). 


2. It would be helpful if the authors could briefly discuss existing work on image recognition in the presence of occlusion (e.g., Compositional neural networks Kortylewski et al., 2020).  I think this would provide historical context for readers, and strengthen the case that occlusion is indeed a problem for recognition models. 

   
correctness: The material appears to be correct. 
clarity: The paper is relatively well-written. 
relation to prior work: The related work is clear.  
additional feedback: None.
review_title: Important problem and major contribution

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a video dataset focusing on segmenting occluded objects. This is an interesting problem as it tackles a more realistic setup where objects get occluded and disoccluded during the video instead of being clearly in view throughout the video. The authors clearly present the rationale behind their design choices and their approach towards creating this dataset. Finally, the authors compare the performance of several models on their benchmark and provide some analysis of the results and model ablations for one of the approaches. 

strengths: - The dataset is very well thought out with the rationale behind different design choices clearly outlined. 
- The splits and metrics chosen to analyze the models make a lot of sense and allow the reader to gain some insight into where the model might be failing or succeeding.
- The visualizations on the websites look very nice with good reidentification of the same object across the video. 
- I like noting not only how the dataset is currently used, but also other ways in which it could be used with little modification. 

weaknesses: 1. While Table 2 lists OVIS numbers, there is very little discussion of how those methods do on prior datasets with the exception of a brief comment in L239. I think including the results on those prior methods would be very useful here.

2. One of the main claims in the paper is that prior datasets do not adequately capture the complexity of the real world. While the authors show the performance of models on their dataset, this confounds two issues: dataset set complexity and dataset size. A lot of recent work on image-based models show that more data with weaker annotation can often lead to comparable performance to less data with stronger supervision/annotation. I think cross-dataset generalization experiments could really shed a lot of light on what is going on here. Do models train on YouTube-Vis do well on OVIS and vice-versa? This would help clarify both the challenge of evaluating occluded scenes as well as the potential benefit of training on occluded scenes. 

3. The authors do a great job on discussing the different models and trying to explain the reasons behind the differences in performance. However, the three representative submissions discussed in Sec 4.3 have different visual backbones, modules, and visual augmentations. I think this weakness of any analysis regarding what caused the models to improve. An ablation study similar to the one shown in Table 3 would be helpful to explain the approaches proposed by Team Ali2500 and Team L1-Minghan. 

4. There is little to no discussion of such issues despite L394 indicating otherwise.

5. While 3 models are trained and averaged to get a more representative performance, there aren’t any error bars reported or variance in performance across those runs despite L409 stating otherwise.

correctness: Overall, the dataset was constructed well and the design choices make sense with their rationale clearly outlined. One issue I had was with the definition and presentation of the mBOR metric.

**Systematic issue with mBOR metric:**
If I understood the mBOR metric correctly, then I think it has a major confounder when comparing to the previous datasets. 
The authors use the bounding boxes to get a pseudo-measure of object overlap and occlusion. I found this metric very interesting and creative, but it seems to hinge on the assumption that potential occluders are annotated. It’s unclear if this assumption applies equally strongly across datasets. For example, Figure 5 row b shows the bear being occluded by the tree.  This occlusion would not be counted by the mBOR metric since the tree is not annotated. While this issue applies to both YouTube VIS and OVIS, it seems from the datasets descriptions that it will systematically affect YouTube VIS more due to the number of annotated objects per frame since it’s unclear how well that correlates with number of actual objects in the frame. I understand that quantifying this is difficult, but I think the authors should discuss it in the paper to either refute my concern in case it’s correct or acknowledge that this metric will be unfair to prior works with less objects/frame annotated. 

clarity: Overall, the paper is well-written and was easy to read and comprehend. There are a few exceptions that I list below:
- It’s unclear how the average recall metrics were computed. My guess is that the subscript refers to the classes, but I would suggest clarifying it explicitly in the text. 
- Table 3 was a bit unclear. Based on the text [L255-269], it seems that Ach w/o SWA is equivalent to MaskTrack R-CNN + LSS + Swin-L, is this correct? If so, I think rewriting the first column descriptions would make this figure much clearer. If not, it would be great to clarify the relation between the top 3 rows and the bottom 2. This is particularly important since the jump from row 3 to row 4 is the second largest jump for most columns and the highest in high occlusion. 
- Why are the results for L1-Minghan on the test set missing? 

relation to prior work: Work is well-positioned with respect to prior work. 

additional feedback: **Questions to authors** 

I had two discussion questions regarding the paper that were not weaknesses or issues, but rather simply questions to the authors: 

1. In L129, the authors state that they remove videos in which most objects are standing still without moving. I am curious if the dataset still contains many non moving objects. One visualization shows the existence of subject objects (bike in third row of the visualization). I am curious about this since it could result in a bias against correctly segmenting objects when they are not moving. 
2. In L143, it is noted that videos range from 3 to 6 fps. Does that mean that some videos were only captured/released at 15 fps? Do you observe any impact on segmentation performance based on the annotation granularity. 

---------------

**Minor issues/Presentation suggestions**

Those points are minor issues or presentation suggestions and did not impact the rating of the paper. The authors do not need to respond to them, but I would encourage them to consider them. Comments in (rough) order of appearance in paper. 

- L13: typo, based not basing
- L19-22: the word our seems unnecessary here. 
- Figure 1: while showing animations is great, I noticed that they do not show up at all in printed documents. I would suggest using something that is more accessible in the paper and reserving animations for online viewing. 
- L147: severe no server.
- L154-156: future tense used when I think the present tense should be used. 
- Table 2: AP_{HO} goes over the line. 
- Table 2 and L229: I think it would be good to include the results on the VIS dataset as well instead of simply mentioning the difference in text. 
- L255: typo, based not basing. 
- Figure 5: I think describing the failure mode for each row in the caption or in the figure would help make this figure easier to understand. It looks good now, but I think adding a succinct summary of L270-278 here would make it much better and more self-contained.
review_title:  Very useful and well-thought out benchmark with some minor issues in analysis and benchmarking

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work presents a video instance segmentation dataset that focuses on scenes with occlusions. The dataset has 901 videos containing 5,223 instances of 25 categories. Several recent state-of-the-art models for VIS and VOS are evaluated on this dataset, and the results demonstrate that the proposed dataset is challenging and difficult.
strengths: - A meaningful dataset that focuses on occluded video instances, which can help to better evaluate the performance of VIS/VOS models.
- A comprehensive analysis of the proposed dataset is presented.
- Several recent SOTA models are tested and compared on the proposed dataset, which is helpful for future research in this field.


weaknesses: - The size of the proposed dataset is small comparing to existing datasets like youtube-vis.
- VIS task and VOS task are somehow mixed in Tab.2. VOS models like FEELVOS require annotation in the first frame, while VIS models like MaskTracker don't need that. Hence it should be better to evaluate performance for the VOS task and the VIS task individually.
correctness: Yes. The dataset looks sound and reasonable.
clarity: Yes. The paper is well written and easy to follow.
relation to prior work: Yes.
additional feedback: No
review_title: A reasonable and meaningful dataset for video instance segmentation.

Meta-review: The paper focuses on instance segmentations in videos of scenes with severe occlusions. This is a very challenging task, and improving models' capabilities in such scenarios will lead to significant advances in general scene understanding. The authors proposed a dataset and an associated challenge for this task. All the reviewers appreciated the usefulness of the dataset and the depth of the provided analysis. The authors updated the manuscript to take into account reviewers' recommendations.
Decision: Accept