Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - This paper considers the problem of video instance segmentation through associating the object-centric representation from frame-wise detector.

- The proposed idea uses Transformer (local attentions) to associate the frame-wise object predictions.

- The idea is validated on different VIS benchmarks, e.g. YTB-VIS2019, 2021, and OVIS, showing good results.
strengths_and_weaknesses: - strengths 
    - the paper writing is good, easy to understand.
    - the motivation of building on pre-trained frame wise object detector is good, which enables the model to handle long and high-resolution videos with customer-level GPUs. 
    - very good results have been obtained.

- weakness
    - in Table 2, what is the results for Mask2Former on OVIS, they have released model, so I believe it's possible to benchmark these numbers.
    - the analysis can be done more thoroughly, for example, it would be interesting to see, what if the frame-wise object detector fails on some intermediate frames, how much VITA can recover it.
    - not much to say on the weakness.

I generally like this paper, for its reasonable motivation, simple implementation, and good results.
questions: - Numbers in Table2 can be filled in.

-  in Table 6, for frozen detector, what benchmark this is ? VIS-2019 ? please put it in caption. 

- if only 20 classes overlap, the recall remains quite high, especially on Swin-L, and this is only AR_10, I imagine AR_100 would be even closer, as you are using 100 queries, so in this case, why the gap on AP is quite large ? for example, 63 vs. 53.4.

- how much VITA can recover the failed prediction from frame-wise predictions ?
limitations: - There is broader impact, but no limitation section has been provided.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors applied Mask2former to generate frame-wise object tokens as a pre-processing of a video.  Afterward, the sliding attention mechanism from Swin Transformer is applied to aggregate all frame-wise understanding for  Video Instance Segmentation. The model trains with the losses from DETR and MaskTrackRCNN. The ensemble achieves a state-of-the-art result on Youtube-VIS 
and OVIS. This is the first offline method applied to the long video dataset OVIS.
strengths_and_weaknesses: Strengths:
1. The method performs state-of-the-art offline video instance segmentation on two challenging datasets. 
2. This is the first time an offline method can run an evaluation on the long video dataset OVIS. 
3. The experiments and ablations are sufficient. The writing is good and easy to understand. The method is described well with interpretable figures. 

Weaknesses: 
1. The primary weakness of this paper is the lack of novelty. It is an ensemble of two prior works, i.e., Mask2Former, and Swin 
Transformer. Both of the prior methods have significant performance in their respective objectives. Hence the original contribution of this work is limited.
2. The authors claim to fit long videos in a single GPU, enabling a long-range offline VIS method. But the Mask2Former features are extracted beforehand. Only the object representation from this prior work is processed in a single GPU. So the claim of fitting 
long-range video is misleading.
3. There is no qualitative analysis of the model's performance. It is unclear how or why the modules improve performance in some instances of the problem.
questions: 1. How were the object tokens calculated during the evaluation phase?
2. Is one GPU enough to fit the video into Mask2Former and the proposed method?
3. Why did the method improve performance for the challenging dataset OVIS?
4. What is the FPS of the method compared to prior work?
limitations: 1. The authors claim to have written the limitation in section 5, but it is not the case.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a memory efficient offline video instance segmentation method named VITA. It utilizes object tokens in each frame as condensed frame features which also embed object-level information. On the video level, it collects all the object tokens and use video queries to generate object sequences. The object sequences are multiplied with the frame features to generated object masks similar to MaskFormer work.
strengths_and_weaknesses: Strength:
-	The proposed VITA framework achieved state-of-the-art performance on YouTubeVIS and OVIS.
-	Compared to previous transformer-based methods, the proposed method effectively reduces running memory with the image features – object tokens – object sequences pipeline. It is able to run hundreds of frames in a single forward process while existing methods can only accommodate a few dozens.
-	The proposed method has faster convergence compared to existing baseline such as Mask2Former-VIS (Figure 5). Making it an appealing choice for model development.
Weaknesses:
-	Some missing references:
CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation AAAI 2021

questions: Other questions:
-	Any potential of adapting this method to online VIS? 
-	How to determine the number of N_f in the model? Does it have an impact on the performance?

limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a novel paradigm for offline video instance segmentation tasks through explicitly modeling object-oriented queries. Specifically, this paper uses an image detector to distill object-specific contexts into object tokens, then accomplishes video-level understanding by associating frame-level object tokens with an extra object encoder. Experiments on YouTuve-VIS-2019, YouTuve-VIS-2021, and OVIS demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
+ Using object tokens to directly model spatio-temporal is novel and efficient.
+ This paper is well written and easy to follow. In addition, the authors also provide the code to reproduce.
+ Experiments on the VIS benchmark demonstrate the effectiveness of the proposed method.
+ Long-range video understanding is a crucial topic in video object detection, multi-object tracking, and video instance segmentation. This paper provides a new perspective for such areas.

Weakness:
+ Since this paper uses mask2former as image instance segmentation, thus it is not fair to compare with the previous SOTA method SeqFormer. I would like to see the results using image instance segmentation as the same as SeqFormer.
+ Using all image queries may be redundant. So, I suggest abandoning background queries and only using foreground queries for such video instance segmentation tasks. The number of different frames may not be equal. Thus, an alternative way is to use part of object queries to ensure each frame has an equal number of queries.
+ Why not provide YouTube-VIS-2022 experimental results?

questions: In general, I think this paper is good with clear writing, a well-explained approach, and good experimental results. The only concern is an unfair comparison with related works.
limitations: There are no particular limitations in general. Please see the weakness above.
review_title: null

Meta-review: metareview: All four reviewers are positive about this work (with three Accept and one Weak Accept). All reviewers appreciate the clear writing, solid results, and the idea of using local attentions in transformers to associate object token extracted at each frame. During the discussion phase, the authors further clarified some of the questions and present additional results (e.g., limitations and ablation on frozen/finetuned detector). After reading the reviews and the rebuttal, the AC agrees with the reviewers that this is a solid work with strong results on video instance segmentation. The AC thus recommends to accept.
confidence: Certain
Decision: Accept