MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
Authors: De-An Huang
Zhiding Yu
Anima Anandkumar
Venue: NeurIPS 2022 Accept (2022)
Keywords: video instance segmentation
query-based transformers

Abstract: We propose MinVIS, a minimal video instance segmentation (VIS) framework that achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures. By only training a query-based image instance segmentation model, MinVIS outperforms the previous best result on the challenging Occluded VIS dataset by over 10% AP. Since MinVIS treats frames in training videos as independent images, we can drastically sub-sample the annotated frames in training videos without any modifications. With only 1% of labeled frames, MinVIS outperforms or is comparable to fully-supervised state-of-the-art approaches on YouTube-VIS 2019/2021. Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without any manually designed heuristics. MinVIS thus has the following inference pipeline: we first apply the trained query-based image instance segmentation to video frames independently. The segmented instances are then tracked by bipartite matching of the corresponding queries. This inference is done in an online fashion and does not need to process the whole video at once. MinVIS thus has the practical advantages of reducing both the labeling costs and the memory requirements, while not sacrificing the VIS performance.

MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training
De-An Huang NVIDIA deahuangånvidia.co com
Zhiding Yu NVIDIA zhidingyenvidia.com
Anima Anandkumar Caltech, NVIDIA anima@caltech. edu
Abstract
We propose MinVIS, minimal video instance segmentation (VIS) framework that achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures. By only training query-based image instance segmen- tation model, MinVIS outperforms the previous best result on the challenging Occluded VIS dataset by over 10% AP. Since Min treats frames in training videos as independent images, we can drastically sub-sample the annotated frames in training videos without any modifications. With only 1% of labeled frames, Min- VIS outperforms or is comparable to fully-supervised state-of- state-of-thhear approaches on YouTube-VIL 2019/2021. Our key observation is that queries trained be dis- criminative between intra-frame object instances are temporally consistent and can used to track instances without any manually designed heuristics. Min VIS thus has the following inference pipeline: we first apply the trained query-based image instance segmentation to video frames independently. The segmented instances are then tracked by bipartite matching of the corresponding queries. This inference is done an online fashion and does not need to process the whole video at once. Min VIS thus has the practical advantages of reducing both the labeling costs and the memory requirements, while not sacrificing the VIS performance. Code is available at: https: :etps://ggt //tp:/ggttbtbbtt Wooom//iaab///nl/i
Introduction
Video instance segmentation (VIS) aims to simultaneously detect, segment, and track object instances in videos [1]. The requirement to accurately track object instances through an entire video makes VIS much more challenging than image instance segmentation. Most of the early approaches for VIS build on image instance segmentation models, and process videos on a per-frame basis [1, 2]. The segmented object instances for each frame then matched temporally with post-processing step. This post-processing step often involves manually designed heuristics that do not generalize well to challenging scenarios like occlusions and large appearance deformations.
Recent VIS works address this issue by taking a per-clip approach, where the spatial-temporal volume of video is processed as whole to directly predict the nnatial-eempraal mask for each object instance [3-5]. Many of these end-to-end VIS approaches are built upon the recent advances of Transformers for end-to-end object detection [6]. Given learned embeddings called queries, Transformers process the queries jointly with the input video using cross-attention. so that each the processed queries can be used predict the spatial-temports mask for object instance in the video.
While these per-elip methods have led to considerable improvements for VIS, using attention to process the whole video, especially longer ones, requires large memory and computation. It is also not straightforward to adapt per-clip methods from offline to online processing to reduce the computational requirements. This limits their practical application, and maintaining the effectiveness of these per-elip methods while improving their efficiency remains an active research direction [7, 8].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Image Encoder
Transformer
LTA
Decoder
Do
Opeer
Image--bste Manchinglos loss
Chject
No Chject
Image Encoder
Transformer Decoder
Licober
Decaader
Quaomies
Oucputt
Groundirutht =1
Inpuct +
Quar Outoutt +:1
(a) Independent Image Instance Segmentation Training
(b) Query Matching for VIS Inference
Figure I: (a) Min trains query-based image instance segmentation model (Image Encoder + Transformer Decoder) using each frame independently. (b) During inference, the trained image instance segmentation model used for video instance segmentation by bipartite matching of query embeddings across frames. Min does not require further manually designed heuristics for tracking.
Another limitation for existing VIS methods is the requirement on annotation. Annotating object instance masks for each video frame prohibitively expensive at seale. While there have been works that alleviate this annotation requirement through weak supervision or image-based annotation, there still a significant performance gap compared to state-of-the-an thrrt sulyy-super methods [9, 10].
Our Approach. We simultancously address both of the aforementioned challenges of computational and labeling costs by showing that we can achieve state-of-the-art VIS performance by only training query-based image instance segmentation model. During inference, Min first applies the query-based image instance segmentation video frames independently. The segmented instances are then associated by bipartite matching of the corresponding queries. MinVIS processes each frame independently in an online fashion and does not need to process the whole video at once. MinVIS does not use any video-based training procedure and thus does not need annotations for all the frames in video. Our contributions are summarized below:
1. We show that video-based architecture and training are not required for competitive VIS perfor- mances. MIn outperforms previous state-of-the-art on YouTube-VVl VIS 2019 and 2021 datasets by 1% and 3% AP while only training an image instance segmentation model.
2. We show that image instance segmentation models capable of segmenting occluded instances are also well suited track occluded instances in videos our framework. Min outperforms its per-clip counterpart by over 13% AP on the challenging Occluded VIS (OVIS) dataset, which is over 10% improvement compared to the previous best performance on the dataset.
Our image-based based approach allows us to significantly sub-sample the required segmentation annotations in training without any change to the model. With only 1% of labeled frames, VIS VIS outperforms or comparable to sully-sueer tfate-of-the-art approaches on all three datasets.
Our key observation is that queries trained to be discriminative between intra-frame object instances are temporally consistent and can be used to track instances without being trained with video-based loss functions. MIn achieves this by requiring its image instance segmentation model to generate masks by convolving query embeddings with features of the whole input image, including regions of other object instances. A query is thus trained to only have high responses corresponding instance. Other query embeddings should instead have low responses on these features because instance masks are non-overlap This design encourages the query embeddings for different instances in frame to be separatad. On the other hand, the query embeddings that segment the same instance from two consecutive frames still need to be similar enough since the instance's image features to be convoluted do not change drastically between frames. This leads to temporally consistent query embeddings for tracking without the need of video-based training.
features of its
Min VIS thus has the following design for inference: We first apply query-based based image instance segmentation model on video frames independently. The segmented instances are then associated between frames by bipartite matching of the corresponding query embeddings. This post-processing step does not need any additional heuristics based on mask overlaps or classification scores in previous works [1, 11]. This because query embeddings already contain these information to track the instances. An overview of Min' training and inference is shown in Figure 1.

---

Since video frames are treated as independent images to train MinVIS, there is no requirement to annotate all the frames in video for training. This allows us to significantly sub-sample and reduce the annotation without any change to our model. We find that on YouTube- VIS 201 19/20221 datasets [1], where there are less variations between video frames, using only 1% of labeled frames leads to less than 3% drop in AP for MinVIS.
We further evaluate Min on the Occluded VIS (OVIS) dataset [12]. One common critique of per-frame approaches is that their tracking heuristics based on mask overlaps would not work when there are heavy occlusions. This is not the case for Min as we do not use any manually designed heuristics. We show that our query- methhing approach generalizes to occluded scenarios. Min VIS with Swin Transformers backbone [13] achieves 39.4% AP on OVIS, which is over 10% improvement from the previous best result on the dataset [14]. We further show that our image-based strategy leads to easier and better learning on OVIS. Min VIS outperforms its per-clip counterpart by over 13% AP.
2 Related Work
Video Instance Segmentation. Per-frame approaches for VIS process each frame independently and later track instances by post-processing. MaskTrack R-CNN [1] adds a tracking head to Mask R-CNN [15] for VIS. MaskProp [16] instead adds a mask propagation head to propagate object instance masks. Cross VIS [2] uses crossover learning to improve instance representation across video frames. QueryTrack [11] adds a contrastive tracking head to QueryInst [17] for instance association. Concurrent work IDOL [18] shows that per-frame models can still outperform per-clip models. Our approach also builds on image instance segmentation models, but unlike previous approaches, we need neither additional parameters nor additional losses to apply to VIS. Our query embeddings from image instance segmentation can directly be used for tracking without video-based training.
Recent per-clip approaches build on the success of Detection Transformer (DETR) [6]. VisTR [4] adopts the query-based approach of DETR to VIS, and there has been several follow-up works, such as Mask2Former- VIS [3] and SeqFormer [5]. One limitation of these approaches is the need to process the whole video at once. IFC [7] reduces the overhead of temporal message passing by using memory tokens. TeV [8] uses a parameter-shared self attention to efficiently model temporal contexts. We also use a query-based approach, but instead of using cross-and cuss-atttnntinn to process the whole video, we process each frame independently while not losing VIS performance. Our use of queries associate instances is also related other works that build on DETR for tracking in related fields. For example, MOTR [19] and TrackFormer [20] use identity preserving track queries for multi-object tracking (MOT).
Reducing Supervision for Video Instance Segmentation. Annotating instance masks for each video frame can be prohibitively expensive. Compared to video object segmentation [21-23] and image instance segmentation [24-27], there have been less works on reducing supervision for VIS [28]. FlowIRN [10] extends IRN [24] with motion and temporal consistency cues to have weakly- supervised VIS framework that only requires classification labels. SOLO-Track [9] learns to track instances without video annotations. It uses instance contrastive learning on SOLO [29] to learn grid cell embeddings for instance tracking. We make the same observation that disciminating between instances within frames beneficial or even sufficient for instance tracking. However, unlike our query-based association, the grid cell embeddings still need threshold-based post-processing and additional loss functions to better handle birth and death of objects.
3 Method
MinVIS is a minimal VIS framework that does not require video-based training and thus can be easily applied to real-world applications that only have sparse image instance segmentation annotations. VIn is a two stage approach: (1) image instance segmentation on each frame independently, (2) associating instances between frames by matching queries. We will first discuss the image instance segmentation architecture in VIS We will then discuss the temporal association of object instances. Finally, we will discuss training and reducing supervision for VIn.
3

---

0g: Skateboard
M
Initial Query ?
Transformer
Decoder
0
Head
0g:
Mask Annotation
Sparse
Oy: Person
Image Encoder
Training
Video
Input Frame
Features
Masks
X
X3
X
X5
X
X7
x
x
(a) Architectural Constraints for Image Instance Segmentation
(a) Reducing Annotation for VIS
Figure 2: (a) MinVIS's main architectural constraint is to require the segmentation masks M be generated by convolving the query embeddings Q with the final feature map F-1- This makes the query embeddings discriminative between each instances. (h) MinVIS's image-based approach allows direct annotation subsampling of training videos without any modification to the model.
3.1 Image Instance Segmentation Architecture for VIS
MinVIS builds on the query-based transformer architectures for detection and segmentation [6, 30, 31], which has the following main components: (1) Image Encoder that learn to extract features from input images. (2) Transformer Decoder that processes the outputs of Image Encoder to iteratively update the query embeddings. (3) Prediction Heads that use the final query embeddings to predict desired outputs (e.g., segmentation masks, class labels). The queries play an important role for the success of such end-to-end pipeline for set prediction with unknown number of outputs. The number of queries are selected as the maximum number of output instanes of the model. During inference, subset of queries predict @ outputs dynamically adjust the number of valid outputs.
An overview of MinVIS's image instance segmentation is shown in Figure 2(a). Given an image X E R.H.W the Image Encoder & extracts a of features F E(X) from the image. F (Fo F-1} is a sequence of multi-scale feature maps F, E HW,CH F-1 denotes the final output of E. The N initial query embeddings Q E N.C are learnable parameters, where N is a large enough number of outputs. The Transformer Decoder T then take both F and Q to iteratively obtain Q = T(F Q), Q E NNN.C. While most recent works focus the design of T to better process F for Q, MISVIS's architectural constraints are on the Prediction Heads. There are two outputs for each instance: classification and segmentation mask. The classification scores O = C(Q). for K classes are the output of Classification Head C, and Q should contain the class information for each instance. For segmentation masks M IR N.W. Min VIS requires that M be generated by convolving the query embeddings Q with the final feature map F-1. The shape for F_ is thus H, W, C. We have M a{Q F-1), where a()) is the sigmoid function.
R.N
The constraint to have Q convolve with the whole feature map F-1 is important for MInVIS. Consider two queries Q; and Q2 that corresponds to distinct object instances and thus non-overlapping masks. This formulation ensures that Q₂ should only have high inner products with features in F-1 that are covered by the mask of instance i. Since the instance masks non-overlapping, Qj should instead have low inner products with these features. This implicitly constrains the query embeddings to be discriminative between each other. On the other hand, if we apply this pipeline to two consecutive frames X and X Then Q should still have higher inner product with Q compared to O This is because Q and 2,+1 are also discriminative between each other, while Of and Q both need to have high inner products with features of the same instance, which do not change drastically between consecutive frames. We visualize our learned query embeddings in Figure 3. Each plot is for a video. Query embeddings belonging to the same instance (from different frames) have the same color. These embeddings are already grouped by instances without any video-based training. Further details are Section 4.2.
While not all image instance segmentation models satisfy our architectural constraints (e.g., ROI- based architectures), we believe these are rather flexible designs that are compatible with various query-based instance segmentation models. We use Mask2Former [30] in this work. The Image Encoder includes both the backbone and the pixel decoder of Former We also find that having culle-comnected layers to further process Q before convolution is beneficial to the performance.

---

YouTube VIS 2019
YouTube 2021
Occluded vis
Figure 3: Visualizing learned query embeddings with only image-based based training. Each plot is for a video, and query embeddings of the same instance (from different frames) have the same color. Query embeddings already grouped into clusters by instance without any video-based training.
3.2 Tracking by Query Matching
MinVIS is a per-frame two-stage approach, which requires a post-processing step to temporally associate instances. This post-processing often involves heuristics like mask overlaps, which does not generalize well to scenarios with heavy occlusions. Unlike previous two-stage approaches, MinVIS associate instances solely based on the query embeddings Q. Given
consecutive frames Xt and X We have Q' [FF, Q), where F E(X²). and similarly for Q++1, Q is the query embedding for instance i. Tracking in MinVIS is done by the assignment of applying the Hungarian algorithm on score matrix S, where Siş = cos(() 1 cos(.,) is the cosine similarity.
This approach is less affected by occlusions because each instance is represented by query that does not have a spatial extent. In addition, we do not need heuristics to handle the birth and death object instances in this framework. Since the number of queries is larger than the number of instances, there are queries that produce empty masks. The death an object instance happens when its embedding matched to such null query. On the other hand, the birth of instance is correctly handled if the matched query embeddings have been null before the actual birth the object instance.
3.3 Training with Less Supervision for VIS
Since the matching process does not need training, only the image instance segmentation model needs be trained. There are two outputs of the model: classification scores O R. N.K and segmentation masks M E H. for N queries, K object classes, and image size H. W. We can process the groundtruth video instances to groundtruth image instances O" E RR.K and M E BL...W where L is the number of groundtruth instances (N >> L) and Oj is a one-hot vector of groundtruth class. Given loss function L(O,, M,, Oj, .Ma) between predicted instance i and groundtruth instance we first use bipartite matching to find the assignments between predicted and groundtruth instances that minimize the overall loss function, and train those matched predictions with the loss function.
More specifically, there are two terms in the loss function: Lets and Lmask- We use cross entropy loss for Lets and binary cross entropy plus dice loss [32] for Lmask as in previous works [30]. Both terms are purely image-based. The groudtruth video instances are first processed instances for each frame independently. Therefore, even if there are only sparse frames labeled with instance, we can still train our model with the annotated frames. This provide a straightforward way to reduce the supervision for VIS. Figure 2(b) shows the annotation sub-samme sampling to reduce supervision.
5

---

4 Experiments
Datasets. We evaluate MinVIS on three datasets: YouTube-VIS 2019/2021 [1] and Occluded VIS (OVIS) [12]. The YouTube- VIS datasets contain 40 object classes. YouTube-Vin 2019 contains 2238/302/343 videos for uraiiing/valddaiiontesinngg while YouTube- 2021 expands the dataset to 2985/421/453 videos for traiiingvalidatiionnestigg and includes higher quality annotations. OVIS has 25 object classes and contains 607/140/154 for rraining/valiaation/een of videos is smaller, OVIS has more objects per frame, and the videos are also longer. This leads to more annotated instance masks compared the YouTube-VIS datasets. In addition, OVIS also has much higher Bounding-bos Occlusion Rate (0.22 v.s. 0.06/0.07) compared to the YouTube- VIS datasets, which indicates heavier occlusions between object instances.
While the number
Metrics. We follow previous works and use Average Precision (AP) and Average Recall (AR) as (IoU) thresholds from 50% to 95% with 5% increment. The reported AP and AR are first computed for each object class and then averaged over all classes. All three datasets have public evaluation servers.
evaluation metries [1]. AP is computed based on 10 intersection-over-uiiit
Baselines. We focus on results using ResNet50 and Swin-L backbones. ResNet50 is still the most widely used backbone for VIS, while Swin-L gives the best performances. Not all methods report both backbones on all three datasets. We include results that are available. For YouTube- VIS datasets, we include recent state-of-the art results from SeqFormer [5], VeT [8]. and Musk2Former- VIS [3]. These are all Transformer- based per-clip approaches as this paradigm has been recently dominating the field. On the other, out of of these methods, only VeT is applied to OVIS. Therefore, we further compare to CMaskTrack R-CNN [12], CrossVIS [2], and STC [33]. These are all methods that allow online processing. Even Te VIT uses near online inference for OVIS [34]. This is because OVIS has longer videos that would lead to out-of-memory for most of the per-elip approaches.
Out of all the baselines, Mask2Former- VIS [3] is the most related to MinVIS, as MinVIS is built on Mask2Former in this work. Mask22ormer--is VIS thus can be seen as the per-clip version of ours and is an important baseline for comparison. Therefore, we further apply Mask2Former- VIS on the OVIS dataset. Due to memory constraints, the videos in OVIS are first split into clips of length 30. We use the same post-processing as Min to merge the outputs from these clips.
Implementation Details. Unless otherwise noted, our hyper-parameters follow Former VIS [3]. All models are pre-trained with COCO instance segmentation [35]. For OVIS, we use the same hyper-parameter as YouTube VIS 2019 except training for 10k iterations instead of 6k. For training losses, the weights are 5.0 for fmaak and 2.0 for Lota- All results of MinVIS are averaged over 3 random seeds. We sub-sample training to X% by uniformly sampling frames in the video. We set minimum of frame per video. Since YouTube-VIS datasets often have videos less than hundred frames. Our I % results are better seen as one frame per video results for YouTube-VIS.
4.1 Main Results
YouTube-VIK 2019. The results for YouTube-VIS 2019 are shown in Table 1. MinVIS achieves highest AP and most other metrics for both ResNet-50 and Swin-L backbones. SeqFormer shows that it is beneficial to jointly train with images from COCO [35] that contain YouTube VIS categories (+C80k in table). TeVIT proposes messenger shift transformer (MsgShifT) that are as efficient as ResNet backbones, while improving the VIS performances. Our ResNet-50 results match or outperform their results without further modifications. Compared to the state-of-the-an Mask2Former- VIS, which can be seen as the per-elip approach to apply Mask2Former to VIS, Min consistently outperforms by around 1% for both backbones. Min with X% means sampling the annotated frames for each video in training. Since there are less temporal variations for videos in You Yobeube-Vo VIS 2019, Min VIS with 1% of training frames only reduces AP by 2.6%. This significantly reduces the annotation effort while not sacrificing much performance.
On this more challenging dataset, the performance improvements for MIn increase compared to YouTube- -IS 2019. Without better backbone like ViT and additional training data like SeqFormer, our ResNet- 50 results outperform by large margin for all metrics. This is the also case for Swin-L.. Min outperforms previous state-of-the-art M by 2.7%. By using only I % of training frames, Min VIS's AP decrease by only 2.4%, which means that our 1% result still outperforms
YouTube--ISS 2021. The results for YouTube- VIS 2021 are shown in Table 2.

---

Table I: YouTube-VIL VIS 2019 results. C80k indicates joint training with COCO images that have YouTube- VIS categories. MinVIS with X% means sub-sampling the annotated frames training.
Method
Backbone
Training
AP
AP 50
AP76
AR
AR
VeV [8]
R50
Full
42.1
67.8
44.8
41.3
49,4
TeViT [8]
MsgShifT
Full
46.6
71.3
51.6
44.9
54.3
SeqFormer [5]
R50
Full
45.1
66.9
50.5
45.6
54.6
SeqFormer [5]
R50
Full+C8Ok
47.4
69.8
51.8
45.5
54.8
Mask2Former VIS [3]
R50
Full
46.4
68.0
50.0
MinVIS
R50
Full
47.4
69.0
52.1
45.7
55.7
TeV [8]
Swin-L
Full
56.8
80.6
63.1
52.0
63.3
SeqFormer [5]
Swin-L
Full+C80k
59.3
82.1
66.4
51.7
64.4
Mask2Former- VIS [3]
Swin-L
Full
60.4
84.4
67.0
MinVIS
Swin-L
Full
61.6
83.3
68.6
54.8
66.6
MinVIS
Swin-L 1%
59.0
81.6
64.7
54.0
64.0
MInVIS
Swin-L 5%
59.3
81.4
65.8
53.8
64.1
MinVIS
Swin-L. 10%
61.0
83.0
67.7
54.6
66.1
Table 2: YouTube-VEs VIS 2021 Results. MinVIS's performance improvement increases on the more challenging YouTube-VVF VIS 2021. Our 1% results already outperform previous state-of-the-art.
Method
Backbone
Training
AP
AP 50
AP75
AR
AR 10
TeV [8]
MsgShifT
Full
37.9
61.2
42.1
35.1
44.6
SeqFormer [5]
R50
Full+C80k
40.5
62.4
43.7
36.1
48.1
Mask2Former-VV VIS [3]
R50
Full
40.6
60.9
41.8
MinVIS
R50
Full
44.2
66.0
48.1
39.2
51.7
SeqFormer [5]
Swin-L
Full+C8Ok
51.8
74.6
58.2
42.8
58.1
Mask2Former-Ve VIS [3]
Swin-L
Full
52.6
76.4
57.2
MinVIS
Swin-L
Full
55.3
76.6
62.0
45.9
60.8
MinVIS
Swin-L
52.9
74.9
58.9
44.7
58.3
MInVIS
Swin-L 5%
54.3
76.3
60.1
45.4
59.5
MinVIS
Swin-L 10%
54.9
76.3
61.9
45.3
60.1
previous state-of-the art. We also see that on YouTube- VIS datasets, reducing the annotations by 10x does not significantly affect the performances (-0.6% AP for 2019 and -0.4% AP for 2021).
Occluded VIS (OVIS). The results for OVIS are shown in Table Mask2Former- VIS denotes application of Mask2Former-VV VIS to OVIS. Since videos in OVIS can have up hundreds of frames, we apply Mask2Former- VIS to non-overlapping sliding windows of length 30. The outputs from these clips then merged by post-processine Min is online method and does not need modification to apply to OVIS. MinVIS shows significant improvement compared to existing works OVIS. With ResNet-50 backbone, MinVIS outperforms previous state-of-the-an VeV with MsgShifT backbone by 7.6% AP. With Swin-L backbone, Min VIS outperforms previous best result MaskTrack +SCN by 10.5% AP, which the winner of the 1st OVIS Challenge. Their key observation that sampling frames that are far apart in OVIS leads to drastically different features and makes hard to train MaskTrack R-CNN, This is in contrast to YouTube-VIS datasets, which the videos are shorter and there are less temporal variations within the video. We observe the same phenomenon when training Mask2Former-VV VIS However, the limited sampling reference frame strategy of Mask R R-CNN*+SMV fully end-to-end loss instead of an explicit tracking loss to learn temporal association, which makes the learning even harder in OVIS. On the other hand, Min VIS is image-based based and does not need to worry about the temporal sampling strategy to train the model. This contrary to common belief that per-fran fermframe approaches are worse for scenarios with heavy occlusions. Instead, our image-based approach leads easier and better learning on OVIS. We show that an image instance segmentation model that can segment occluded instances in each frame is also good at associating such instances
still does not work in this case. Mask2Former- VIS* uses
7

---

Table 3: OVIS Results. MinVIS significantly outperform existing approaches on OVIS. Our image- based framework leads to easier and better learning on this dataset with heavy occlusions.
Method
Backbone
Training
AP
AP 60
AP 75
AR1
AR
TeViT [8]
MsgShifT
Full
17.4
34.9
15.0
11.2
21.8
Cross VIS [2]
R50
Full
14.9
32.7
12.1
10.3
19.8
CMask Track R-CNN [12]
R50
Full
15.4
33.9
13.1
9.3
20.0
STC [33]
R50
Full
15.5
33.5
13.4
11.0
20.8
Mask2Former- VIS
R50
Full
17.3
37.3
15.1
10.5
23.5
MinVIS
R50
Full
25.0
45.5
24.0
13.9
29.7
MaskTrack R-CNN* [14]
Swin-L
Full
28.9
56.3
26.8
13.5
34.0
Mask2Former-Vis VIS*
Swin-L
Full
25.8
46.5
24.4
13.7
32.2
MinVIS
Swin-L
Full
39.4
61.5
41.3
18.1
43.3
MinVIS
Swin-L
1%
31.7
54.9
31.3
16.3
36.1
MinVIS
Swin-L
5%
35.7
60.1
35,8
17.3
39.9
MinVIS
Swin-L
10%
37.2
60.7
38.0
17.3
41.1
Mask2 orse2
Heurisite
MinVis
Figure 4: Qualitative results on OVIS. Min VIS stably tracks all the sheep in the video. Using mask overlap based heuristics instead leads to multiple identity switches in tracking. Mask2Former VIS uses per-clip training that is difficult to optimize on the challenging OVIS dataset.
across frames. Figure 4 shows qualitative results. MinVIS stably tracks all the sheep in the video. Using mask overlap based heuristics instead of query matching leads multiple identity switches in tracking. Mask2Former- VIS does not have good segmentation masks because its training is interfered by heavy occlusions and large appearance deformations between frames OVIS.
Figure 5 shows additional qualitative results on failure cases of Min on the OVIS dataset. As discussed in Section 3.2, MinVIS does not heuristies handle the birth and death of object instances. The death of an object instance is correctly handled if its query is matched to a query in the next frame that produces an empty mask. Despite its simplicity and effectiveness, the drawback of this approach is that there is nothing stopping the model from matching the disappearing query query with non-empty mask. From to to t in the top row of Figure 5, the fish in the lower left leaves the frame, Min VIS associates it to a mask covering the tail of nearby fish. From f to ts. when the fish in the upper left leaves the frame, Min VIS again associates it to mask covering the head of nearby fish. Since the associated masks are non-empty, Min VIS fails to correctly handle the death of these instances. On the other hand, when the two dogs in bottom row of Figure 5 are covered in (2, MinVIS correctly associates their queries to empty masks. MinVIS further correctly handles the object births I3. However, MinVIS is limited by the segmentation of the image segmentation model, which fails to segment the close-up person.

---

4 5
Figure 5: Failure cases of MinVIS on OVIS. When an object instance disappear from a video, Min can fail by associating its query embedding wrong mask without overlap (top). This is because we do not use mask overlap heuristics in our work. On the other hand, we are also limited by the image instance segmentation model, which might not work well on close-up objects (bottom).
Table Comparison of post-processing. Heuristics based on mask over laps lead to significant AP drop on OVIS. Our query matching approach has simpler design without loss of performance.
Method
Dataset
AP
APs0
AP75
AR1
AR10
heuristics only
YouTube-Vii VIS 2019
58.2
79.2
64.1
51.3
63.6
heuristics query query only
YouTube- VIS 2019
61.3
82.8
68.7
54.3
66.3
YouTube- VIS 2019
61.6
83.3
68.6
54.8
66.6
heuristics only heuristics query query only
YouTube- VIS 2021
52.7
75.3
57.3
44.4
58.3
YouTube- VIS 2021
55.1
76.2
61.9
46.0
60.7
YouTube- Miabe--VIS/a 2021
55.3
76.6
62.0
45.9
60.8
heuristics only heuristics query query only
Occluded VIS
31.7
56.0
31.3
15.8
35.8
Occluded VIS
39.1
62.5
40.8
17.7
43.4
Occluded VIS
39.4
61.5
41.3
18.1
43.3
4.2 Analyzing Query Matching
The success of MinVIS depends whether query matching is good for tracking instances. We conduct ablation studies by comparing it to manually designed heuristics. We use the bipartite matching heuristics in Section 3.3 for tracking by treating instances in the last frame groundtruth. The results are Table Using heuristics lead to around 3% AP drop on both YouTube- VIS 2019 and 2021. It leads to more significant drop OVIS (7.7%) due to heavier occlusions. We also combine query matching and heuristics with equal weights, which has mixed results. Our query only approach is simpler and more generalizable without loss of performance.
We visualize the learned query embeddings by t-SNE [36] Figure Each plot is for a video in the training set. We visualize the training set to see the effect of an image only objective (to segment instances in an image) on query embeddings across different frames. Query embeddings of the same instance have the same color. We obtain the instance IDs for queries by bipartite matching its outputs to groundtruth instances, which have consistent IDs across frames. Without any video-based tracking objective, query embeddings of the same instances are already grouped into distinct clusters, even for the OVIS dataset. This supports our design of only using based objectives. In Appendix ?? we further visualize query embeddings vidcos not used in training.
4.3 Effect of Viseo- Training
While we have shown that MinVIS VIS achieves state-of the--hhe-rt VIS performance without rideo-based training, it is interesting to see how we can leverage video annotation when available. We use the video annotation to supervise our matching as in previous per-frame works [1, 2, 11]. Given two sampled frames, we use hinge loss ensure that the correct associations of queries have the highest inner products compared to that of other queries between the two frames [1 1]. The results

---

Table 5: Results for adding supervision to query matching, The supervision can provide dataset dependent benefit if the temporal hyper- pyper-parameters are selected properly.
Method
Dataset
AP
APoo 50
AP75
AR1
AR10
MinVIs
YouTube-VIS 2019
61.6
83.3
68.6
54.8
66.6
+ Supervised Matching
YouTube-VIS 2019
61.0
82.1
67.6
54.3
66.1
+ Limited Range
YouTube- VIS 2019
60.7
82.5
67.0
54.1
65.5
MinVIS
YouTube- VIS 2021
55.3
76.6
62.0
45.9
60.8
+ Supervised Matching
YouTube-VIS 2021
54.4
75.7
60.6
45.5
59.5
+ Limited Range
YouTube-VEL 2021
55.2
77.0
61.5
45,4
60.1
MinVIS
Occluded VIS
39.4
61.5
41.3
18.1
43.3
+ Supervised Matching
Occluded VIS
38.7
61.2
39.6
17.9
42.4
+ Limited Range
Occluded VIS
39.6
63.2
41.0
18.2
43.0
in Table 5. The "'upervvsed Matching" rows mean directly applying the matching supervision to the original frame sampling process. In our case, this means that the two sampled frames might be separated up to 20 frames. As pointed out in previous works, frames that are far separated increase the training difficulty and can hurt model performances especially with occlusions [14]. We thus also consider the "Limited Range" training to only sample consecutive frames for supervised matching, as we only need to match consecutive frames. From the results, directly applying "Supervised all three datasets. Adding "Limited Range" recovers most of
Matching" hurt performances the performances for You TouTube-WV VIS 2021 and OVIS. On OVIS, it even marginally outperforms the original VIS. However, this improvement does rely the dataset dependent sampling range. We believe it is possible and important to use video-based training to further improve Min although this would take away MinVIS's practical advantages of only needing sparse annotations and having simple training pipeline. Appendix ?? discusses further limitations of not using video information in training.
5 Conclusion
We show that a purely image-based training procedure can lead to competitive performances for VIS. Our key finding is that instance tracking naturally emerges in query-based image instance segmentation models with proper architectural constraints. In addition to improving state-of-the-ar approaches YouTube-VIi 2019 2019/2021, image-based objective reduces the learning difficulty and leads to better performances. MinVIS only requires sparse frame annotations, which makes it much more applicable to real-worke scenarios. We believe a promising direction to extend MinVIS is to explore ways to better leverage the video frames that are not annotated further improve our performances with sub-sampled annotations.
show that this is particularly beneficial for OVIS. The
References
[1] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. 1, 6,
9
[2] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV, 2021. 1,
3, 3,6,8 8,9
[3] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10766
2021. 1, 3, 7.
[4] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2021. 3
[5] Junfeng Wu, Yi Jiang, Wenqing Zhang, Xiang Bai, and Song Bai. Seqformer: frustratingly simple model for video instance segmentation. arXiv preprint 2 2021. 1,3, 7
10

---

[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3.4
[7] Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim. Video instance segmentation using inter-frame communication transformers. NeurIPS, 2021. 3,3
[8] Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR, 2022.
1, 7,
[9] Yang Fu, Sifei Liu, Umar lqbal, Shalini De Mello, Humphrey Shi, and Jan Kautz. Learning to track instances without video annotations, In CVPR, 2021. 2,
[10] Qing Liu, Vignesh Ramanathan, Dhruv Mahajan, Alan Yuille, and Zhenheng Yang. Weakly supervised instance segmentation for videos with temporal mask consistency. In CVPR, 2021.
2, 3
[1I] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Ying Shan, Bin Feng, and Wenyu Liu. Tracking instances queries. arXiv preprint arXh::2106. //966, 2021. 2, 3,9
[12] Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip Torr, and Song Bai. Occluded video instance segmentation: A benchmark. arXiv preprint arXiv:2
2021. 3, 6,
[13] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 3
[14] Zhuang Li, Leilei Cao, and Hongbin Wang. Limited sampling reference frame for masktrack
r-cnn. In ICCVW, 2021. 3, 8, 10
[15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICVV, 2017.
[16] Gedas Bertasius and Lorenzo Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR, 2020.
[17] Yuxin Fang, Shusheng Yang. Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries. In ICCV, 2021. 3,4
[18] Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. In ECCV, 2022. 3
[19] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In ECCV, 2022. 3
[20] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In CVPR, 2022. 3
[21] Xiankai Lu, Wenguan Wang, Jianbing Shen, Yu-Wing Tai, David Crandall, and Steven CH Hoi. Learning video object segmentation from unlabeled videos. In CVPR, 2020. 3
[22] Paul Voigtlaender, Lishu Luo, Chun Yuan, Yong Jiang, and Bastian Leibe. Reducing the annotation effort for video object segmentation datasets. In WACV, 2021.
[23] Yanchao Yang. Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In CVPR, 2021. 3
[24] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmen- tation with inter-pixel relations. CVPR, 2019.
[25] Shiyi Lan, Zhiding Yu, Christopher Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry S Davis, and Anima Anandkumar, Discobox: Weakly supervised instance segmentation and semantic correspondence from box supervision. In ICCV, 2021.

---

[26] Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: high-perffrmance instance segmentation with box annotations. CVPR, 2021.
[27] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning to segment objects without annotations. In CVPR, 2022.
[28] Wenguan Wang, Tianfei Zhou, Fatih Porikli, David Crandall, and Luc Van Gool. A survey on deep learning technique for video segmentation. arXiv preprint arXiv:2107.0/ 2021. 3
[29] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV, 2020.
[30] Bowen Cheng, Ishan Mísra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. mask transformer for universal image segmentation. In CVPR, 2022. 4, 5
[31] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[32] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016. 5
[33] Zhengkai Jiang, Zhangxuan Gu, Jinlong Peng, Hang Zhou, Liang Liu, Yabiao Wang. Ying Tai, Chengjie Wang, and Liqing Zhang. Ste: Spatio-temporal contrastive learning for video instance segmentation. arXiv preprint arXiv:2202.03747 2022. 6, 8
[34] Ali Athar, Sabarinath Mahadevan, Aljosa Osep. Laura Leal-Taixé, and Bastian Leibe. Stem-seg: Spatio-temporal embeddings for instance segmentation in videos. In ECCV, 2020.
[35] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6
[36] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.3 and the appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the
appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See link in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Tables with error bars are in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
12

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4. All three datasets public and cited.
(b) Did you mention the license of the assets? [Yes] See the appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'te ro using/curating? [Yes] See the appendix.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See the appendix.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sg6Z1sLqoK4", "reviewer": "Reviewer_zDMq", "summary": "The paper proposes MinVIS, a minimal video instance segmentation system (VIS) that obtains SOTA results on a variety of VIS benchmarks without employing video-based architectures or video-based training. The proposed system operates in two stages: 1) first, an image-level instance segmentation is trained, 2) afterward, the frame-level instance associations across frames are built using a bipartite matching algorithm. The authors demonstrate that despite not using any video-related information, their proposed framework achieves highly competitive results in comparison to video-based VIS solutions. ", "questions": "Questions:\n1. Could the authors briefly summarize their TECHNICAL contributions, and also present a detailed comparison with the Mask2Former system, and how their proposed approach differs from this prior work.\n2. I was quite puzzled by the results obtained using video-based training in 4.3. It's quite surprising to see the performance drop, and the justification that this happens due to occlusions doesn't seem very convincing to me. This is based on my own past experiences, where such pairwise matching-based training was highly successful for tracking. It would be useful if the authors could visualize the embeddings resulting from such training as they did in Figure 3 of the original draft. \n3. It would also be useful if the authors could incorporate some efficiency metrics. I imagine that the proposed approach is more efficient than prior video-based systems. It would be good to highlight such advantages. ", "limitations": "In my view, the authors could discuss the limitations of not incorporating any video-based architectures/training procedures in more depth. Video provides lots of extra information that the authors are ignoring in this case. While I appreciate the simplicity and effectiveness of their solution, I think it would also be useful to have a more detailed discussion how their framework could be improved while incorporating video cues. Section 4.3 does not provide a good discussion on that. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The authors tackle a challenging and highly impactful problem of video instance segmentation (VIS).\n+ The proposed system is very simple, which would provide a convenient baseline for other researchers to build on.\n+ The experiments are thorough and convincing, i.e., the proposed method achieves state-of-the-art results on multiple VIS benchmarks.\n+ The paper is easy to read and understand.\n+ Additional experiments showing strong performance with a limited amount of labeled VIS data.\n+ The proposed method can be used in an online setting.\n\nWeaknesses:\n- Overall, I enjoyed reading the paper. The proposed approach is simple, yet highly effective. My only concern is that the proposed approach feels somewhat incremental, i.e., the technical contributions are quite limited. To be more precise, the proposed framework is largely built on Mask2Former, which was also originally developed for images. The bipartite matching algorithm to associate instances across frames has been widely used both in the image-only settings and in the video-related settings in the past. The only substantial contribution that I could spot is in the design of a prediction head, i.e., imposing the constraint to have Q convolve with the whole feature map F_{−1}. It's a small, yet important design choice that simplifies instance segmentation prediction. \n- In my view, the authors could discuss the limitations of not incorporating any video-based architectures/training procedures in more depth. Video provides lots of extra information that the authors are ignoring in this case. While I appreciate the simplicity and effectiveness of their solution, I think it would also be useful to have a more detailed discussion how their framework could be improved while incorporating video cues. Section 4.3 does not provide a good discussion on that. \n- The authors should carefully proofread their draft. There are some typos. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "rHwY6DOquAQ", "reviewer": "Reviewer_o2a1", "summary": "This paper proposes a new video instance segmentation (VIS) framework, MinVIS, which achieves state-of-the-art VIS performance with neither video-based architectures nor video-based training procedures. With this image-based framework, this work is more flexible on the requirements of training data. ", "questions": "As described in weakness.", "limitations": "No serious negative societal impact in this work.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) This paper achieves SOTA results on various video instance segmentation datasets. \n2) The method of this paper is simple but satisfied in performance.\n3) This paper is well organized.\n\nWeakness:\n1) The technical contribution of this paper is a bit sufficient. The critical contribution of this paper is to utilize the query embedding from Mask2Former [27] and prove it is effective for instance association, but no more novel idea or method is proposed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZYHxOlV4Wdt", "reviewer": "Reviewer_tb44", "summary": "The paper presents a video instance segmentation method for simultaneously segmenting and tracking objects in video sequences. The proposed per-frame method first uses the query-based MaskFormer [27] to segment objects frame by frame, and associate object segments in different frame by matching the queries. The proposed method achieves SOTA performance on three VIS datasets. Additionally, the paper demonstrates competitive results of the proposed method when 1%, 5% and 10% of the training data is available. ", "questions": "As mentioned in weaknesses above, what are the performance of baselines with 1%, 5%, 10% of the training data? Preferably data augmentation should be performed. Note that the clip-based baseline MaskFormer-VIS uses clips consisting of only 2 frames for training. Therefore, one can generate pairs of frames for training using data augmentation even in the 1% setting\n\nWhat are the data augmentation strategies for training MaskFormer in the proposed approach, especially in the low data settings? Is the data augmentation performed in the low data settings different from the one with full data?\n\nWhy don’t  Table 4 and 5 have the standard deviation included? It seems hard to conclude which strategies are better as the APs for some of them are very similar (within 0.2 difference).\n\nAccording to Ln 258, the temporal sampling strategy during training is important. What is the sampling strategy for the baselines? Do the authors try different sampling strategies and data augmentation on OVIS dataset except for those default ones on YouTube-VIS?\n\nIn Fig 4 column $t_1$ to $t_4$, why does the baseline Mask2Former-VIS miss to detect the second sheep from the right in the image? It doesn't make a lot of sense to me that MinVIS can detect that object but Mask2Former fails given that both of them are highly based on MaskFormer. Is it because Mask2Former-VIS uses a single query for an object in the video, and using only a single query cannot adapt to the drastic appearance change in this case? If it is the case, the choice of applying Mask2Former-VIS to clips with length of 30 (Ln 246) might not be optimal.", "limitations": "Yes, the authors addressed the limitations and potential negative societal impact in section 4.3 and the supplementary document.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed approach takes the advantage of the design of the query-based image segmentation method and extend it to tracking object segments effectively. The proposed method achieves SoTA performance across three VIS datasets.\n\nThe proposed method only requires training the query-based image segmentation method. I appreciate the simplicity.\n\nThe paper is easy to follow.\n\n****\n\n**Weaknesses**\n\nThe key idea of the proposed approach is to use queries to associate object segments in different frame. In terms of originality, this is not entirely novel and have been used and studied in other related tasks, e.g., multiple object tracking (MOT). Importantly, the paper ignores literature review on related query-based works that also use queries for object association and tracking or related video tasks.\n   \nIn the experiments, the authors show competitive results of their approach with little amount of training data. However, it’s unclear whether other baselines also have little reduction in the performance even with little amount of data. Note that data augmentation can and should be applied for those baselines, and it is still a fair comparison in terms of the amount of training data. \n\nThe method highly depends on MaskFormer’s architecture design and its segmentation results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9Gq-S1dWsPn", "reviewer": "Reviewer_2Rax", "summary": "The paper proposes a novel video instance segmentation framework called Min-Vis which leverages transformed based architecture to segment object instances in videos. The key contributions are:\n\n* The paper demonstrates that good performance can be obtained for this problem even by using image level instance segmentation annotations alone. And instance tracking problem can be solved by conducting bipartite matching on object queries between adjacent frames without requiring any training.\n\n* The paper also demonstrates that because of the simplicity of its design, it can even be trained with much smaller dataset size.\n\n* The paper achieves state-of-the-art performance on multiple VIS benchmarks. ", "questions": "More details on compute times for training and inference will be interesting to learn about. ", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* This is a solid paper with state-of-the-art results for the problem of video instance segmentation. The gains on YouTube-VIS 2021 and OVIS dataset are impressive. The paper also presents good ablation studies. Looking at the current leaderboard for YouTube-VIS dataset, the proposed method looks to be the best. \n\n* The proposed approach is fairly generic and can be extended to other tasks beyond instance segmentation. More experiments are needed but if the assumption that cosine similarity between object query embeddings is enough to associate instances between frames holds, this will be a natural method to apply on multi-object tracking problems as well. \n\n* The ability of being able to train video instance segmentation models using image only instance segmentation masks brings a significant practical advantage to this method compared to others. \n\nWeakness:\n* More datasets like DAVIS benchmark and \"Unidentified Video Objects\" from Meta can be used for additional experiments. This will provide more diversity to the benchmarks on which the method is evaluated. \n* No analysis on model failures are presented in the paper, a brief discussion with some visual examples could provide more insights into where the current approach breaks down. \n* Visualization of object embeddings (Figure 3) should be presented for validation sets as well for better understanding. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept