Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a video instance segmentation method for simultaneously segmenting and tracking objects in video sequences. The proposed per-frame method first uses the query-based MaskFormer [27] to segment objects frame by frame, and associate object segments in different frame by matching the queries. The proposed method achieves SOTA performance on three VIS datasets. Additionally, the paper demonstrates competitive results of the proposed method when 1%, 5% and 10% of the training data is available. 
strengths_and_weaknesses: **Strengths**

The proposed approach takes the advantage of the design of the query-based image segmentation method and extend it to tracking object segments effectively. The proposed method achieves SoTA performance across three VIS datasets.

The proposed method only requires training the query-based image segmentation method. I appreciate the simplicity.

The paper is easy to follow.

****

**Weaknesses**

The key idea of the proposed approach is to use queries to associate object segments in different frame. In terms of originality, this is not entirely novel and have been used and studied in other related tasks, e.g., multiple object tracking (MOT). Importantly, the paper ignores literature review on related query-based works that also use queries for object association and tracking or related video tasks.
   
In the experiments, the authors show competitive results of their approach with little amount of training data. However, it’s unclear whether other baselines also have little reduction in the performance even with little amount of data. Note that data augmentation can and should be applied for those baselines, and it is still a fair comparison in terms of the amount of training data. 

The method highly depends on MaskFormer’s architecture design and its segmentation results.

questions: As mentioned in weaknesses above, what are the performance of baselines with 1%, 5%, 10% of the training data? Preferably data augmentation should be performed. Note that the clip-based baseline MaskFormer-VIS uses clips consisting of only 2 frames for training. Therefore, one can generate pairs of frames for training using data augmentation even in the 1% setting

What are the data augmentation strategies for training MaskFormer in the proposed approach, especially in the low data settings? Is the data augmentation performed in the low data settings different from the one with full data?

Why don’t  Table 4 and 5 have the standard deviation included? It seems hard to conclude which strategies are better as the APs for some of them are very similar (within 0.2 difference).

According to Ln 258, the temporal sampling strategy during training is important. What is the sampling strategy for the baselines? Do the authors try different sampling strategies and data augmentation on OVIS dataset except for those default ones on YouTube-VIS?

In Fig 4 column $t_1$ to $t_4$, why does the baseline Mask2Former-VIS miss to detect the second sheep from the right in the image? It doesn't make a lot of sense to me that MinVIS can detect that object but Mask2Former fails given that both of them are highly based on MaskFormer. Is it because Mask2Former-VIS uses a single query for an object in the video, and using only a single query cannot adapt to the drastic appearance change in this case? If it is the case, the choice of applying Mask2Former-VIS to clips with length of 30 (Ln 246) might not be optimal.
limitations: Yes, the authors addressed the limitations and potential negative societal impact in section 4.3 and the supplementary document.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a novel video instance segmentation framework called Min-Vis which leverages transformed based architecture to segment object instances in videos. The key contributions are:

* The paper demonstrates that good performance can be obtained for this problem even by using image level instance segmentation annotations alone. And instance tracking problem can be solved by conducting bipartite matching on object queries between adjacent frames without requiring any training.

* The paper also demonstrates that because of the simplicity of its design, it can even be trained with much smaller dataset size.

* The paper achieves state-of-the-art performance on multiple VIS benchmarks. 
strengths_and_weaknesses: Strengths:

* This is a solid paper with state-of-the-art results for the problem of video instance segmentation. The gains on YouTube-VIS 2021 and OVIS dataset are impressive. The paper also presents good ablation studies. Looking at the current leaderboard for YouTube-VIS dataset, the proposed method looks to be the best. 

* The proposed approach is fairly generic and can be extended to other tasks beyond instance segmentation. More experiments are needed but if the assumption that cosine similarity between object query embeddings is enough to associate instances between frames holds, this will be a natural method to apply on multi-object tracking problems as well. 

* The ability of being able to train video instance segmentation models using image only instance segmentation masks brings a significant practical advantage to this method compared to others. 

Weakness:
* More datasets like DAVIS benchmark and "Unidentified Video Objects" from Meta can be used for additional experiments. This will provide more diversity to the benchmarks on which the method is evaluated. 
* No analysis on model failures are presented in the paper, a brief discussion with some visual examples could provide more insights into where the current approach breaks down. 
* Visualization of object embeddings (Figure 3) should be presented for validation sets as well for better understanding. 
 
questions: More details on compute times for training and inference will be interesting to learn about. 
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new video instance segmentation (VIS) framework, MinVIS, which achieves state-of-the-art VIS performance with neither video-based architectures nor video-based training procedures. With this image-based framework, this work is more flexible on the requirements of training data. 
strengths_and_weaknesses: Strengths:
1) This paper achieves SOTA results on various video instance segmentation datasets. 
2) The method of this paper is simple but satisfied in performance.
3) This paper is well organized.

Weakness:
1) The technical contribution of this paper is a bit sufficient. The critical contribution of this paper is to utilize the query embedding from Mask2Former [27] and prove it is effective for instance association, but no more novel idea or method is proposed.

questions: As described in weakness.
limitations: No serious negative societal impact in this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes MinVIS, a minimal video instance segmentation system (VIS) that obtains SOTA results on a variety of VIS benchmarks without employing video-based architectures or video-based training. The proposed system operates in two stages: 1) first, an image-level instance segmentation is trained, 2) afterward, the frame-level instance associations across frames are built using a bipartite matching algorithm. The authors demonstrate that despite not using any video-related information, their proposed framework achieves highly competitive results in comparison to video-based VIS solutions. 
strengths_and_weaknesses: Strengths:
+ The authors tackle a challenging and highly impactful problem of video instance segmentation (VIS).
+ The proposed system is very simple, which would provide a convenient baseline for other researchers to build on.
+ The experiments are thorough and convincing, i.e., the proposed method achieves state-of-the-art results on multiple VIS benchmarks.
+ The paper is easy to read and understand.
+ Additional experiments showing strong performance with a limited amount of labeled VIS data.
+ The proposed method can be used in an online setting.

Weaknesses:
- Overall, I enjoyed reading the paper. The proposed approach is simple, yet highly effective. My only concern is that the proposed approach feels somewhat incremental, i.e., the technical contributions are quite limited. To be more precise, the proposed framework is largely built on Mask2Former, which was also originally developed for images. The bipartite matching algorithm to associate instances across frames has been widely used both in the image-only settings and in the video-related settings in the past. The only substantial contribution that I could spot is in the design of a prediction head, i.e., imposing the constraint to have Q convolve with the whole feature map F_{−1}. It's a small, yet important design choice that simplifies instance segmentation prediction. 
- In my view, the authors could discuss the limitations of not incorporating any video-based architectures/training procedures in more depth. Video provides lots of extra information that the authors are ignoring in this case. While I appreciate the simplicity and effectiveness of their solution, I think it would also be useful to have a more detailed discussion how their framework could be improved while incorporating video cues. Section 4.3 does not provide a good discussion on that. 
- The authors should carefully proofread their draft. There are some typos. 
questions: Questions:
1. Could the authors briefly summarize their TECHNICAL contributions, and also present a detailed comparison with the Mask2Former system, and how their proposed approach differs from this prior work.
2. I was quite puzzled by the results obtained using video-based training in 4.3. It's quite surprising to see the performance drop, and the justification that this happens due to occlusions doesn't seem very convincing to me. This is based on my own past experiences, where such pairwise matching-based training was highly successful for tracking. It would be useful if the authors could visualize the embeddings resulting from such training as they did in Figure 3 of the original draft. 
3. It would also be useful if the authors could incorporate some efficiency metrics. I imagine that the proposed approach is more efficient than prior video-based systems. It would be good to highlight such advantages. 
limitations: In my view, the authors could discuss the limitations of not incorporating any video-based architectures/training procedures in more depth. Video provides lots of extra information that the authors are ignoring in this case. While I appreciate the simplicity and effectiveness of their solution, I think it would also be useful to have a more detailed discussion how their framework could be improved while incorporating video cues. Section 4.3 does not provide a good discussion on that.
review_title: null

Meta-review: metareview: All four reviewers are positive about this work. Reviewers appreciate the clear writing, simple yet highly effective idea, and strong experimental validation on three Video Instance Segmentation datasets. The authors responses further clarified and sufficiently addressed the concerns from the reviewers. The AC reads the reviews, the rebuttal, and agree with the reviewers to recommend acceptance.
confidence: Certain
Decision: Accept