VITA: Video Instance Segmentation via Object Token Association
Authors: Miran Heo
Sukjun Hwang
Seoung Wug Oh
Joon-Young Lee
Seon Joo Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: video
instance segmentation
video instance segmentation
tracking
transformers

Abstract: We introduce a novel paradigm for offline Video Instance Segmentation (VIS), based on the hypothesis that explicit object-oriented information can be a strong clue for understanding the context of the entire sequence. To this end, we propose VITA, a simple structure built on top of an off-the-shelf Transformer-based image instance segmentation model. Specifically, we use an image object detector as a means of distilling object-specific contexts into object tokens. VITA accomplishes video-level understanding by associating frame-level object tokens without using spatio-temporal backbone features. By effectively building relationships between objects using the condensed information, VITA achieves the state-of-the-art on VIS benchmarks with a ResNet-50 backbone: 49.8 AP, 45.7 AP on YouTube-VIS 2019 & 2021, and 19.6 AP on OVIS. Moreover, thanks to its object token-based structure that is disjoint from the backbone features, VITA shows several practical advantages that previous offline VIS methods have not explored - handling long and high-resolution videos with a common GPU, and freezing a frame-level detector trained on image domain. Code is available at the link.

VITA: Video Instance Segmentation via Object Token Association
Miran Heo" Yonsei University
Sukjun Hwang" Yonsei University
Scoung Wug Oh Adobe Research
Joon-Young Lee Adobe Research
Scon Joo Kim Yonsei University
{miran, 8j. hwang, seonjookiim@@oonsi..ac
{seoh, jolee)
Abstract
We introduce novel paradigm for offline Video Instance Segmentation (VIS), based on the hypothesis that explicit object-oriented information can be strong clue for understanding the context of the entire sequence. To this end, we propose VITA, simple structure built top of an off-the-shelf Transformer-basee image instance segmentation model. Specifically, we use an image object detector as a means of distilling object-specific contexts into object tokens. VITA accomplishes video-level understanding by associating frame-level object tokens without using spatio-temporal backbone features. By effectively building relationships between objects using the condensed information, VITA achieves the state-of offtthe-rrt on VIS benchmarks with ResNet-50 backbone: 49.8 AP, 45.7 AP YouTube-VIS VIS 2019 & 2021, and 19.6 AP on OVIS. Moreover, thanks to its object boken-based structure that disjoint from the backbone features, VITA shows several practical advantages that previous offline VIS methods have not explored handling long and high- resolution videos with common GPU, and freezing frame-level detector trained on image domain. Code is available at ltttt:::/// /stps://gitaan thtbb.t..ttttttt.e/gge/ thub. com/ suk unhwang/
VITA
1 Introduction
The goal of Video Instance Segmentation (VIS) is to predict both mask trajectories and categories of each object belonging to set of predefined categories. Numerous studies have attained the
goal a variety of ways, but a notable innovation in terms of accuracy has been achieved by
Transformer-basee the first attempt to design an end-to-ent model that jointly predicts object trajectories with their
[27] architcctures. Extending DETR [5] to the video domain, VisTR [X] made
corresponding segmentation masks. By adopting this paradigm, subsequent studies [
also tackle the problem a complete-offlin manner: video-in and video-out.
The key message from the follow-up approaches [15
is to effectively design core interac- that improve the accuracy in various tasks by localizing the attention scope of Transformer layers, the subsequent VIS methods suggest bounding the attention scope the encoder [15
30
6
34
tions between frames. In parallel with recent studies 25
34 or the decoder [30] Specifically, they decompose the global attention by iteratively mixing two phases: intra-frame attention and fnter-frame communication. Interestingly, the temporal interactions between frames are commonly achieved with only small number of tokens, e.g., memory tokens [5 A4), messenger tokens
[44, and instance
queries 30 As a result, the question arises: "what information important to understand a video?"
Both authors contributed equally this work.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Tv
Franc
Delector
Poed.
Enn.
Pred w/ Assockazon
Thnn Erac
Prai of Assccaaion
Association
Frma
Trrs,
Prime
Detectos
Prod
O00
OOO
Dosse Festures
Object Queties
(a) Tracking-by -y-teection
(b) Existing Offline Methods
(c) VITA (Ours)
Figure (a) Early-stage VIS methods divide the problem into two components, detection and association. (b) To alleviate the context-limited structure, complete-ofiling methods jointly track and segment instances an end-to-end manner by employing dense other hand, our VITA is new paradigm that directly leverages object queries for offline VIS.
features. (c) On the
In this paper, we introduce Video Instance Segmentation via Object Token Association (VITA), a new offline VIS paradigm which suggests that a video can be effectively understood from collection
(Fig- D (b)) localize objects in multiple frames by iteratively referring to dense apatio-tempora backbone features. However, such methods show difficulties in handling long sequences as the myriad of dense reference features hinders the Transformer layers from retrieving relevant information. With the motivation to devise an effective method for the long-range understanding, we obtain clues from the traditional tracking- by- deyeecettoon paradigm (Fig. [ and make two hypotheses: 1) an image object detector can fully embody the context of an object into a feature vector (or a token); and 2) a video can be represented by the relationship between the objects.
of object-centric tokens. Existing offline methods 28 15
In this regard, VITA aims to parse an input video from the collection of object tokens without the necessity of referencing dense spatio-temporal backbone (Fig. (c)). Given the compactness of the token representation, VITA can collect the object tokens over the whole video and directly analyzes the collection using Transformer layers. This unique design enables the complete-ofilee inference (i.e., video-in and video-out) even for extremely long videos. This also facilitates building relationships between every detected object and successfully achieves global video understanding. As result, VITA achieves state-of-ther ofrte-of-the-att performance on various VIS benchmarks.
We evaluate VITA on three popular VIS benchmarks, YouTube- VIS 2019 & 2021 32 and OVIS 24 With ResNet-50 [ backbone, VITA achieves the new state-of-the-an of 49.8 AP 45.7 AP on YouTube- 2019 & 2021, and 19.6 AP on OVIS. Above all, VITA outperforms the previous best approaches by 5.1 AP for YouTube- VIS 2021, which contains more complicated and long sequences than YouTube- VIS 2019. VITA is the first offline method that presents the results on OVIS benchmark that consists long videos (the longest video has 292 frames) using a single 12GB GPU.
In addition to the performance, the design of VITA have several practical advantages over the previous offline VIS methods. can handle long and high-resolution videos does not require heuristics for associating clip-level results. VITA can process 1392 frames at once regardless of video resolution using a single 12GB GPU which is 11 times longer than IFC 1S Moreover, VITA can be trained on top of parameter-frozen image object detector without sacrificing the performance much. This property is especially useful for the applications that cannot afford to store separated image and video instance segmentation models. VITA takes only 6% additional parameters to extend the Swin-I. detector.
Related Works
Online VIS approaches first predict individual tracklets within a local range window consisting of a single or a few frames, After obtaining results from adjacent windows, they associate individual tracklets of same identities by hand-crafted or learnable matching algorithm. MaskTrack R- CNN [2 sets the groundwork for VIS research by proposing a simple tracking branch added on two-stage image instance segmentation model (I3]. The methods 4 33 21l that follow the tracking- beakiingby--eer deteccioo paradigm (Fig. || (a)) measure the similarities between per-frame predictions, then employ association algorithm.
To deploy temporal context from multiple frames, per-clip methods 1 2 design an architecture of predicting tracklets within local window and stitching the tracklets sequentially in near-

---

Pisel Decader
Mask
Class
Trans Enc.
Object Eneader
Object Decoder
Input Frames
Queries
Object Tokens
Queries
Frame-level Detector
VITA
Figure 2: VITA takes only mask features and frame queries that independently decoded by the frame-level detector for entire video sequence. By directly constructing temporal interactions between frame queries that encapsulate rich object-aware knowledge in spatial scenes, VITA yields mask trajectories with corresponding categories in an end-to-end manner.
online manner. Propagation-basee based methods 10 previous information stored in memories to facilitate online applications. EfficientVIS [29] introduces correspondence learning between adjacent tracklet features and successfully runs cascaded manner which eliminates the hand-crafted tracklet association.
devise paradigm that conjugates rich
Offline VIS architectures are proposed with the motivation of predicting mask trajectories through a whole video sequence at once. VisTR (28) successfully extends DETR [5] to the VIS domain, introducing a new paradigm of jointly tracking and segmenting instances. However, its dense self- attention over the spatio-temporal inputs leads to explosive computations and memories. With motivation of relaxing the heavy computation of VisTR, IFC [1S] adopts memory tokens to the Transformer encoder and decodes clip-level object queries. By setting the frame-level encoder to be independent and adopting the decoder of IFC, Mask2Former- VIS mance on benchmarks by taking the advantage its proposes backbone that efficiently exchanges temporal information internally based Vision [ instead of the trame-wise CNN backbone. SeqFormer [30 decomposes the decoder to be came-indeeenders
6 records considerable perfor- representation [7]. TeViT [34]
Transformers
while building communication between different frames using instance queries that are used for frame-wise detection. All these studies achieve promising performance by referring dense backbone features (Fig. (b)). On the other hand, VITA suggests new offline VIS paradigm that directly interprets a video from the collection of object tokens (Fig. (c)).
Global trackers that aim associate frame-level predictions across entire sequence as whole are studied the Multiple Object Tracking (MOT) community. Conventional approaches formulate the problem graph optimization interpreting each detection node and considering the edges as possible connections between the nodes 35 26 8. Different from existing methods, GTR 36 introduces Transformer-based architecture that receives queries, then explicitly searches for the predictions with the same identities. Similarly, a recent method [16] proposes a set classifier that classifies the category of each tracklet by globally aggregating information from multiple frames.
3 Method
In this section, we first give a brief overview of Mask2Former 77 a frame-level detector for VITA. Then, introduce the architecture of our proposed VITA, which is built top of Mask2Former. Finally, we describe how VITA handles extremely long videos in complete-offinee manner.
3.1 Frame-level Detector
In this paper, we adopt Mask2Former 7/ for the frame-level detector which directly localizes instances using masks without the necessity of bounding boxes. Following the set prediction mechanism of DETR 5 the frame-level detector parse an input image H x W using N, object queries, which we call frame queries (j E RC*N,)) throughout this paper. Having the spatially encoded features to decoded by the frame queries through Transformer decoder, each object in the image gets represented C-dimensional vector. Then, the frame queries used for both classifying and

---

segmenting their matched objects where the predictions are also used for auxiliary supervision for VITA. Specifically, the frame-level detector generates two features for the frame-level predictions: 1) dynamic 1 X convolutional weight from the frame queries; 2) per-pixel embeddings M IRCx $ * from the pixel decoder, where S is the stride of the feature map. Finally, the detector segments objects by applying a simple dot product between the two embeddings.
3.2 VITA
We now propose the novel end-to-end video instance segmentation method VITA, which can be largely divided into three phases (Fig. 2). First, VITA operates on top of the frame-level detector [ complete trame-independent manner, no inter-computation between frames is involved. Then, the frame queries that hold object-centric information are collected throughout the whole video and they embed video-level information by building communications between different frames using Object Encoder. Finally, Object Decoder aggregates information from the frame queries to video queries, which are eventually used for predicting categories and masks of objects in videos at once.
Input of VITA. Given an input video of I frames, the frame-level detector executes frame-by-free frame as previously explained. Among number of intermediate embeddings that are generated by the detector, the only features that are used by VITA are 1) frame queries {f') E RCXTxN, which hold object-centric information; and 2) per-pixel embeddings (M')T Jt=1 RCXTx pixel decoder.
from the
Object Encoder: After the frame-wise detector dis- tills the object-wise context into the frame queries, Object Encoder aims to build temporal communica- tion by employing self-attention along the temporal axis. First, Object Encoder gathers frame queries from all frames and converts them to object tokens through linear layer. However, naive self-attention over the whole T object tokens is not applicable when processing long videos due the quadratic computational overhead of Transformers. Inspired by Swin Transformer [ we adopt window-based self-attention layers that shift along the temporal di-
Window Attention
Shifled Window Attention
T
Figure 3: Illustration of an Object Encoder layer. Blocks with dashed line are local win- dows, and O indicates object token.
mension. As illustrated in Fig. 3 Object Encoder
initially partitions object tokens { | to the temporal axis with local windows of size W without an overlap. By alternatively shifting the windows, object tokens from different frames can exchange object-wise information which allows VITA to both effectively and efficiently handle long sequences.
Object Decoder and Output heads. Two limitations of previous offline VIS methods are the ineffectiveness in handling dynamic scenes and the inability of processing long vidcos. For example, such methods obtain high accuracy when dealing with static and short videos (YouTube- VIS 2019 (32)), but struggle to track objects executes end-to-end on benchmarks with dynamic and long videos (YouTube- VIS 2021 62 and OVIS (44). Both limitations are mainly caused by the decoder, which parses object contexts directly from dense spatio-temporal features. As recent suggest, typical Transformer decoders show difficulties in retrieving relevant information from global context. In the video domain, the number of backhone features being referred to proportionally increases with the number of frames. Therefore, when handling extremely long videos, the countless reference tokens result in both imprecise information retrieval and intractable peak memories.
studies 1 25 37
For the solution to the problem, we suggest Object Decoder which extracts information from the object tokens, not the spatio-temporal backbone features. Implicitly embedding the context of objects, object tokens can provide sufficient rnstance-specife information without the interference of dense backbone features. Specifically, we employ N,, trainable video queries v E CCN.. to decode object- wise information from all object tokens { that are collected from all T frames. Receiving much condensed input over naively taking dense spatio-temporal features, Object Decoder effectively captures video contexts and aggregates relevant information into the video queries. As a result, Object

---

Decoder shows fast convergence speed while achieving high accuracy. Furthermore, the compact input greatly saves memories, thus facilitates processing long and high-resolution videos.
{(ps, m,) using two output heads similar to IFC [5]; the class head and the mask head. The class head a single linear classifier, which directly predicts class probabilities p E RNNxx((+ of each video query, where K is the number of categories including auxiliary label "no object" (2). The mask head dynamically generates mask embeddings W, E RCx* No per a video query, which corresponds to the tracklet of an instance all frames. Finally, the predicted mask logits RR N.xxXxx can be obtained from a matrix multiplication between wy and {M')]
From the decoded video queries v, VITA returns final predictions
3.3 Clip-wise losses
Instance matching. We search for optimal pair indices between the predictions from VITA and G. ground-truth to remove post- processing heuristies such NMS. First, we calculate costs from all possible pairs using the cost function of Mask2Former 7 with simple extension of mask-related costs to the temporal axis [[5]. Then, from N x G, costs of pairs, we follow DETR [5] and use Hun- garian algorithm [8] for the optimal match- ing shown in Fig. (b).
(6) Prame-level
(b) VITA Muiching
Delector Matching
Input
(c) Similarity Loss
Similarity loss. Inspired by the initial VIS approach (MaskTrack R-CNN [32) where the similarity loss is adopted to track in- stances at different frames, we train video queries and frame queries to be clustered in the latent space by their identities. As shown in Fig. 4 (a), our adopted frame-level detec- tor 7 also searches for paired indices be-
Figure 4: Similarity loss. and indicate video query and frame query, respectively. Same color rep- resents same GT instance ID.
tween Ny frame-wise predictions and G32 ground-truth objects at each 1th frame. The frame queries and the video queries that are matched to ground-truths get collected and embed the collection through a linear layer. Then, we measure the similarity of all possible pairs using a simple matrix multiplication. Finally, shown in Fig. D (c), binary cross entropy is used to compute Lsim between the predicted similarities and the ground-truth where annotated to 1 for pairs of equal identities and 0 for vice-versa.
Total loss. We attach the proposed module VITA top of the frame-level detector, and all components of the model get trained end-to-end. Note that not only video-level outputs from VITA used for the loss computation, but also per-frame outputs from the frame-level detector get involved. Specifically, we use Ly from ? to calculate loss from the per-frame outputs to frame-wise ground-truth. Extending the loss function of 7 the temporal axis as similar to [15,, from VITA to calculate the video-level loss C.. Finally, we integrate all losses together as follows: fiotal XyGy XyLs + Aaim Lism-
outputs
4 Experiments
4.1 Datasets
You YouTube-VIS 2019. YouTube- VIS 2019 32 is the first dataset proposed for VIS and contains 40 semantic categories. Mostly originated from Video Object Segmentation (VOS) datasets, the VIS benchmark has a small number of unique instances (average 1.7 per video for the train set) and the categories of instances appearing in the same video are different in general. Also, the average length of videos in the valid set short (27.4 frames), which enables existing complete-offinee approaches to load whole video and infer the benchmark once.
5

---

Table 1: Comparisons on YouTube- VIS 2019.
Method
Backbone
AP
APso
AP7s
AR1
AR10
MaskTrack R-CNN 22
ResNet-50
30.3
51.1
32.6
31.0
35.5
Mask R-CNN
32
ResNet-101
31.8
53.0
33.6
33.2
37.6
(Near) Online
Cross
33
ResNet-50
36.3
56.8
38.9
35.6
40.7
CrossVI5
+5
ResNet-
36.6
57.3
39.7
36.0
42.0
PCAN 177
ResNet-50
36.1
54.9
39.4
36.3
41.6
PCAN 177
ResNet-101
37.6
57.2
41.3
37.2
43,9
EfficientVIS
24
ResNet-50
37.9
59.7
43.0
40.3
46.6
EfficientVIS
ResNet-101
39.8
61.8
44.7
42.1
49.8
VISOLO
ResNet-50
38.6
56.3
43.7
35.7
42.5
VisTR
ResNet-50
35.6
56.8
37.0
35.2
40.2
VisTR 28
ResNet-101
38.6
61.3
42.3
37.6
44.2
IFC 15
ResNet-50
41.2
65.1
44.6
42.3
49.6
IFC
[5
ResNet-101
42.6
66.6
46.3
43.5
51.4
TeVIT
MsgShifT
46.6
71.3
51.6
44.9
54.3
SeqFormer
60
ResNet-50
47.4
69.8
51.8
45.5
54.8
Offiiee
SeqFormer
50
ResNet-101
49.0
71.1
55.7
46.8
56.9
SeqFormer
20
Swin-L
59.3
82.1
66.4
51.7
64.4
Mask2Former- VIS
ResNet-50
46.4
68.0
50.0
Mask
ResNet-101
49.2
72.8
54.2
Mask2Former-VIS
Swin-L
60.4
84.4
67.0
ResNet-50
49.8
72.6
54.5
49.4
61.0
VITA (Ours)
ResNet-101
51.9
75.4
57.0
49.6
59.1
Swin-L
63.0
86.9
67.9
56.3
68.1
YouTube-VIS 2021. In order to address more difficult scenarios, additional videos included in YouTube- -VSub2- (794 videos for training and 129 videos for validation). In particular, a greater number of objects with confusing trajectories has been added (average 3.4 per video for the additional videos in the train set). However, the average length the additional validation videos still 39.7 frames, which is not significantly increased compared to YouTube- VIS 2019.
Under the same definition as YouTube-VV VIS. OVIS 2 specifically aims tackle objects with heavy occlusions that are belonging to 25 semantic categories. In addition to the heavily occluded situation, OVIS has three challenging characteristies that are distinct from the YouTube- VIS datasets. First, although it has fewer categories than YouTube-VVIS much more instances appear in a single video (average 5.9 per video for the train set). Second, the instances with the same categories the same video have almost similar appearances, thus approaches that rely heavily on visual cues often struggle to predict accurate trajectories. Finally, the average length videos for the valid set is 62.7 frames (the longest video has 292 frames) which is much longer than that of YouTube- VIS. Therefore, not only do previous approaches show relatively low accuracy, but all cxisting complete-offine VIS methods are not feasible to infer OVIS without hand-crafted association algorithms.
OVIS.
4.2 Implementation Details
Our method is implemented on top of detectron2 31 All hypae-parameters regarding the frame- level detector are equal the defaults of Mask2Former 77) The total loss Liotal is balanced with A
Xy. and Agim where 1.0, 1.0, and 0.5, respectively. By default, Object Encoder is composed of three layers with the window size W
and Object Decoder employs six layers with N 100 video
queries. Having VITA built on top of Mask2Former, we first train our model on the COCO 20 dataset following Former Then, we train our method on the VIS datasets 32 24
simultaneously
with pseudo videos generated from images 20 following the details of SeqFormer
30
During
inference, each frame is resized shorter edge size 360 and 448 pixels when using ResNet [4 and Swin
P2 backbones, respectively. Note that all reported scores in main results and ablation

---

Table Comparisons with ResNet-50 backbone on YouTube- VIS 2021 and OVIS. f indicates using MsgShifT backbone. I indicates using Swin-L 22 backbone.
TubTube- VIS 2021
OVIS
Method
AP
AP
AP7s
AR
AR
AP
AP50
AP7s
AR1
AR10
MaskTrack R-CNN
32
28.6
48.9
29.6
26.5
33.8
10.8
25.3
8.5
7.9
14.9
CMaskTrack R-CNN
15.4
33.9
13.1
9.3
20.0
STMask [9
31.1
50.4
33.5
26.9
35.6
15.4
33.8
12.5
8.9
21.3
CrossVIS
34.2
54.4
37.9
30.4
38.2
14.9
32.7
12.1
10.3
19.8
IFC [15
35.2
55.9
37.7
32.6
42.9
VISOLO 12
36.9
54.7
40.2
30.6
40.9
15.3
31.0
13.8
11.1
21.7
TeVIT"
34
37.9
61.2
42.1
35.1
44.6
17.4
34.9
15.0
11.2
21.8
SeqFormer
30
40.5
62.4
43.7
36.1
48.1
Mask2Former- VIS
40.6
60.9
41.8
VITA (Ours)
45.7
67.4
49.5
40.9
53.6
19.6
41.2
17.4
11.7
26.0
SeqFormer
30
51.8
74.6
58.2
42.8
58.1
Mask2Former-VIS'
52.6
76.4
VITA (Ours)*
57.5
80.6
61.0
47.7
62.6
27.7
51.9
24.9
14.9
33.0
studies the mean of five runs, and we use the standard ResNet-50 specified.
|E] for the backbone unless
4.3 Main Results
Using the popular VIS benchmarks - YouTube-Vii VIS 2019 2021 32 and OVIS
24
we compare
VTTA with state-of-the-art approaches following the standard evaluation metric 32
YouTube- VIS 2019. Tab. I shows the comparison YouTube-VIS 2019 dataset with backbones of
both CNN-based (ResNet-50 and 101 [14]) and Transformer-basee (Swin-L. (22)). Offline methods take two advantages over (near) online approaches: 1) they have greater receptive field to the temporal axis, and 2) they can avoid error propagation derived from hand-crafted association algorithms. As result, the tendency of offline methods with higher accuracy is clearly shown in the table. Among the competitive offline models, our VITA sets new state-of-the-an of 49.8 AP and 51.7 AP using CNN backbones, ResNet-50 and ResNet-101 respectively. In addition, with Swin-L backbone, VITA achieves 63.0 AP outperforming all existing VIS methods.
VIS 2021. We compare VITA with state-of-the-art methods on YouTube-VIS 2021 benchmark in Tab. Note that the longest video in the valid has 84 frames, thus previous
offline methods 15 6 can infer videos at once with GPUs with large memories. Above all, VITA
achieves the highest accuracy, 45.7 AP, which outperforms the previous state-of-the-are approach K with a huge margin of 5.1 AP. Considering the accuracy gap on YouTube- VIS 2019, the results demonstrate that VITA can effectively handle tricky scenarios, e.g., numerous unique instances with confusing trajectories. We hypothesize that the object-orienter design of VITA is more effective than typical dense Transformer decoders in addressing such challenging scenes.
demonstrate the competitiveness of VITA on the challenging OVIS benchmark. Due to the considerable lengths of videos the longest video has 292 frames existing offline approaches (28 [5 34 30 6 cannot process OVIS benchmark in their original design: video-in and video-out, To the best of our knowledge, VITA is the first complete-offling approach to evaluate on OVIS valid set. Thanks to its object token-based structure which is disjoint from backbone features, VITA can process the benchmark without any hand-crafted association algorithm. Moreover, VITA sets new state-of-the-art performance 19.6 AP, demonstrating the potential of the complete-offine pipeline in long and complicated scenes.
OVIS. In Tab.
7

---

Table 3: Impact of local windows of varying sizes in Object Encoder.
Table Maximum number of frames that can be processed at once using a single Titan XP.
W
AP
APsu
APTs
AR1
AR10
Max Frames
Method
360 x 640
1280
49.4
72.2
54.4
48.6
60.9
VRsTR 28
46
12
6
49.8
72.6
54.5
49.4
61.0
IFC 15
123
38
12
50.0
73.0
54.7
49.0
60.8
Mask2Former-Ve
81
20
All
50.1
72.4
54.7
49.0
60.6
W
2677
VITA (Ours)
W =
1392
Table 5: Use of different heuristic association algorithms OVIS valid set.
W 12
741
Length
Algorithm
AP
AP50
AP75
Table 6: Pruning tokens by different ratios r.
36
Greedy
18.8
39.4
17.1
AP
AP75
AR1
AR
Hungarian
18.4
38.9
16.3
1.0
49.8
72.6
54.5
49.4
61.0
Greedy
18.8
39.0
17.1
48
0.75
49.7
72.5
54.4
48.7
61.0
Hungarian
19.1
39.1
17.4
0.5
48.9
72.1
52.0
48.3
60.9
All
None
19.6
41.2
17.4
0.25
48.1
71.6
51.6
47.4
59.8
4.4
Ablation Studies
We provide series ablation studies using a ResNet-50 [4] backbone. All experiments are conducted on YouTube-VVIS 2019 32 valid set except for Tab. with OVIS 24 valid set.
Attention window size. Tab. shows the performance of VITA with varying sizes of shifted attention window W in Object Encoder during inference. The larger the window, the greater the receptive field for the temporal axis in Object Encoder. The results suggest that larger window sizes utilize information from multiple frames, which helps Object Encoder understand the context of objects in videos. We set W = =6 considering a trade-off between performance and inference scalability.
Maximum number of frames. In Tab. calculate the maximum number of frames that VITA can handle with respect to the various window sizes W, and compare it with existing complete-offine VIS methods. To take into account the general environment, all results computed using single 12GB Titan XP GPU. As shown in results, existing methods have limitations in processing long videos video-in and video-out manner. Clearly, the bottleneck of VisTR 28 is the encoder, where the full spatio-temporal self-attention leads to a tremendous memory usage. IFC [15] alleviates the computation of VisTR 28 achieving higher number of input frames. However, IFC makes use of typical Transformer decoder that visits all dense spatio-temporal features. Therefore, IFC cannot infer the OVIS 24 benchmark at once which contains video of 292 frames. The problem gets aggravated in Mask2Former: VIS [ as the scope of the decoder is extended to multiple feature levels [77]. On the other hand, VITA presents considerable frame numbers that can be inferred completely offline. Furthermore, VITA is independent from input frame resolutions as each frame gets summarized into compact object tokens. With input resolution of 360 x 640 and W 6, the maximum length of sequence that VITA is able to process in complete-offlin is about / x longer than IFC [1
Heuristic clip association. Tab. 5 shows the results on OVIS valid set of splitting a video into shorter clips and associating clip-wise predictions through heuristic matching. The length of the clip is set to be less than the average length of videos of OVIS valid set (62.7). Then, we associate outputs from different clips using mask loU score the matching cost. We test with two matching algorithms: Greedy and Hungarian. As shown in Tab, 5 VITA demonstrates the best performance on the complete-offline inference that use all the video frames at once.
In Tab. 6 we investigate the effects of removing redundant frame queries. From collection of frame queries, VTTA understands the overall context of the given clip. only a small portion of the collection is matched to foreground objects, the number of total input frame queries
Pruning Tokens.

---

50-
40-
Table 7: Results on YouTube- VIS 2019 with freezing detector pretrained on COCO.
Mask AP
VITA w/ sim
Backbone
Freeze
AP
AP50
AP75
AR1
AR
20
VITA wio sim
Mask2Fonmer-Ve VIS!
ResNet-50
49.8
72.6
54.5
49.4
61.0
Mask2Forme-
v
40.9
61.9
44.6
43.1
53.1
51.9
75.4
57.0
49.6
59.1
ResNet-101
20
60
43.2
64.4
48.7
46.1
55.9
Total Iterations
63.0
86.9
67.9
56.3
68.1
Figure 5: Train speed comparison with Swin-L Mask2Forme-- VIS [6]. it indicates the same training setup with VITA.
53.4
75.9
58.7
51.9
64.3
can be reduced. First, for each frame, we sort frame queries in ascending order by the "no object" (2) probability. Then, we keep only top y queries from the sorted list where is the ratio, and discard the rest. The accuracy with respect to the ratio is as shown in Tab.
By setting the ratio 0.75, the accuracy of VITA shows only a marginal degradation in the accuracy (-0.1 AP). This signifies that VITA focuses more on the foreground contexts that are embedded in the frame queries. Meanwhile, the quadratic computation in Clip Encoder be alleviated, VTTA can process a much greater number of frames; using the ratio r = 0.75, the maximum frame number increases from 1392 (Tab. 2635.
Convergence speed and Similarity loss. Fig. validates our claim of the faster convergence speed and the effectiveness of the proposed Similarity loss. For a fair comparison, we report average scores and standard deviations of five runs, each trained without pseudo videos, same as Mask2Former- VIS (). Thanks to its object-centric design, VITA shows faster convergence than Mask2Former- VIS. Furthermore, the use of Similarity loss leads to an additional accuracy gain of 1.8 AP. The results demonstrate that the loss mitigates the discrepancies between the embeddings of equal identities, leading to better performance.
Frozen frame-level detector. In Tab. we demonstrate the performance of VITA where the frame-level detector is completely frozen. Specifically, while VITA gets trained on YouTube-Vi VIS 2019, the frame-level detector 7 does not get updated from pretrained weights on COCO [20] Note that among 40 categories in YouTube-VIS 2019 dataset, only 20 categories overlap with the categories COCO. Interestingly, though the frame-level detector remains completely frozen, VITA achieves compelling results with various backbones. As shown in Tab. and Tab. 7 VITA presents a huge practicality as it surpasses all online approaches on top of the ResNet-50 backbone. This strategy be beneficial in various scenarios: 1) when the accuracy of image instance segmentation should be kept while extending the network to the video domain, and 2) when having limited time and GPUs to train models. The strategy can be especially useful mobile applications that have scarce storage for keeping two separate network parameters for image instance segmentation and video instance segmentation. With additional 6% parameters, VITA successfully extends the frozen Swin-L based frame-level detector to the video domain and it achieves great accuracy.
We also provide brief discussion of our understanding for the large gap in AP. Compared to COCO, we observe that YouTube-VIS VIS dataset is annotated with only a few salient objects as foregrounds. Having weights of the frame-level detector frozen COCO, the detector cannot adapt to the YubTube- VIS domain and it embeds and interprets more objects in scenes as foregrounds. Therefore, VITA outputs more predictions as foreground category even if such predictions are not labeled as ground- truths in YouTube- VIS. As a result, it leads to lower average precision as it comes out with more false positive predictions. On the contrary, the more false positive predictions only slightly affect AR.
Qualitative Results. We provide some visualizations of the predictions from VITA and frame-level detector Fig. The qualitative results show that VITA leads better video instance segmentation qualities compared to the frame-level detector. Specifically, the frame-level detector mistakenly interprets to recognize either category or mask of instances that have been largely occluded, while method successfully recovers it by leveraging the temporal information.

---

I'aunee Levee-
VITA
Frame level
VIIA
Figure 6: Visualization of predictions from the frame-level detector and VITA. Instances with the same identity are displayed in the same color.
Limitations
VITA has achieved high performance in the complete-ofllins paradigm while dramatically improving the number of input frames that can be processed at once. However, there are two major limitations for the ultimate long video understanding. First, the current architecture still has limitations in processing an infinite number of frames. addition, since object tokens do not explicitly utilize temporal information, they may have difficulties in identifying complex behaviors that span over very long sequences. We believe that devising explicit designs address these issues will be a promising future direction.
Conclusion
In this paper, we proposed VITA for offline Video Instance Segmentation. VITA simple model built on top of the off-the-shelf image instance segmentation model 77 Unlike existing offline methods, VITA directly leverages object queries decoded by independent frame-level detectors. We demonstrated that deploying object-oriented information is not only effective in improving performance, but also has robust practicality for processing long and high-resolution videos setting state-of-the-art on popular VIS benchmarks, e.g. YouTuube VIS-2019 & 2021 and OVIS. Moreover, since VTTA is designed to absorb spatial knowledge purely from image object detector, shows fast convergence and demonstrates competitive performance even if trained on frozen detectors. We hope that our method extends the scope of offline VIS research beyond benchmarks real-world applications.
Acknowledgements
This work has partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF2022R AEF2002RIA2220000 A2C2004509) and by Institute of Information communi- cations Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-001 13, Developing Sustainable Collaborative Multi- modal Lifelong Learning Framework), and Artificial Intelligence Graduate School Program under Grant 2020-0-01361.
Broader Impact
VITA designed for the VIS task and focuses on processing long and high-resolution videos in an end-to-end manner while achieving the state-of-the-an performance. We hope that VITA can have positive impact on many industrial areas such as video editing applications. We would like to note that research on VIS must be aware of potential misuse that violates personal privacy.
Licenses COCO 20), YouTube- VIS 32 OVIS 24), and detectron2 International, CC BY 4.0, CC BY-NC-SA 4.0, and Apache-2.0, respectively.
Attribution 4.0
10

---

References
[1] Ali Athar, Sabarinath Mahadevan, Aljoša OSep, Laura Leal-Taixe and Bastianan Leibe. Stem- seg: Spatio-temporal embeddings for instance segmentation in videos. In ECCV, 2020,
[2] Gedas Bertasius and Lorenzo Torresani.
Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR, 2020.
[3] Guillem Brasó and Laura Acix-T Learning neural solver for multiple object tracking, In CVPR, 2020.
[4] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang. and Ling Shao. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV, 2020.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[6] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. arXiv preprint 12.
2021.
[7] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
[8] Peng Dai, Renliang Weng, Wongun Choi, Changshui Zhang, Zhangping He, and Wei Ding. Learning a proposal classifier for multiple object tracking. In CVPR, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[10] Yang Fu, Linjie Yang, Ding Liu, Thomas S Huang, and Humphrey Shi. Compfeat: Comprehen-
sive feature aggregation for video instance segmentation. In AAAI, 2021.
[11] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co co-attention. In ICCV, 2021.
[12] Su Ho Han, Sukjun Hwang, Seoung Wug Oh, Yeonchool Park, Hyunwoo Kim, Min-Jung Kim, and Seon Joo Kim. Visolo: Grid-based space-time aggregation for efficient online video instance segmentation. In CVPR, 2022.
[13] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[15] Sukjun Hwang, Miran Hco, Scoung Wug Oh, and Seon Joo Kim. Video instance segmentation using inter-frame communication transformers. In NeurlPS, 2021.
[16] Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim. Cannot see the forest for the trees: Aggregating multiple viewpoints to better classify objects in videos. In CVPR, 2022.
[17] Lei Ke, Xia Li, Martin Danelljan, Yu- Wing Tai, Chi-Keung Tang, and Fisher Yu. Prototypical aross-attention networks for multiple object tracking and segmentation. In NeurlPS, 2021.
[18] Harold W Kuhn. The hungarian method for the assignment problem. In Naval research logistics quarterly, 1955.
[19] Minghan Li, Shuai Li, Lida Li, and Lei Zhang. Spatial feature calibration and temporal fusion for effective one-stage video instance segmentation. In CVPR, 2021.
[20] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
11

---

[21] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingje Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[23] Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation. arXiv preprint arXiv:2102.01558, 2021.
[24] Jiyang Qi. Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: Dataset and icev 2021 challenge. In NeurIPS Track on Datasets and Benchmarks, 2021.
[25] Zhiqing Sun, Shengeao Cao, Yiming Yang, and Kris Kitani, Rethinking transformer-based set prediction for object detection. In ICCV, 2021.
[26] Siyu Tang, Mykhaylo Andriluka, Bjoemn Andres, and Bernt Schiele. Multiple people tracking by lifted multicut and person re-identification In CVPR, 2017.
[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[28] Yuging Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2020.
[29] Jialian Wu, Sudhir Yarram, Hui Liang, Tian Lan, Junsong Yuan, Jayan Eledath, and Gerard Medioni. Efficient video instance segmentation via tracklet query and proposal. In CVPR, 2022.
[30] Junfeng Wu, Yi Jiang, Wenqing Zhang, Xiang Bai, and Song Bai. Seqformer: a frustratingly simple model for video instance segmentation. In ECCV, 2022.
[31] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. 2019.
[32] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019.
[33] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV, 2021.
[34] Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR, 2022.
[35] Li Zhang, Yuan Li, and Ramakant Nevatia. Global data association for multi-object tracking using network flows. In CVPR, 2008
[36] Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Phillip Krähenbühl. Global tracking transform- ers. In CVPR, 2022.
[37] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
12


Reviews:
[{"review_id": "rBa5PpN3JyR", "reviewer": "Reviewer_56An", "summary": "- This paper considers the problem of video instance segmentation through associating the object-centric representation from frame-wise detector.\n\n- The proposed idea uses Transformer (local attentions) to associate the frame-wise object predictions.\n\n- The idea is validated on different VIS benchmarks, e.g. YTB-VIS2019, 2021, and OVIS, showing good results.", "questions": "- Numbers in Table2 can be filled in.\n\n-  in Table 6, for frozen detector, what benchmark this is ? VIS-2019 ? please put it in caption. \n\n- if only 20 classes overlap, the recall remains quite high, especially on Swin-L, and this is only AR_10, I imagine AR_100 would be even closer, as you are using 100 queries, so in this case, why the gap on AP is quite large ? for example, 63 vs. 53.4.\n\n- how much VITA can recover the failed prediction from frame-wise predictions ?", "limitations": "- There is broader impact, but no limitation section has been provided.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- strengths \n    - the paper writing is good, easy to understand.\n    - the motivation of building on pre-trained frame wise object detector is good, which enables the model to handle long and high-resolution videos with customer-level GPUs. \n    - very good results have been obtained.\n\n- weakness\n    - in Table 2, what is the results for Mask2Former on OVIS, they have released model, so I believe it's possible to benchmark these numbers.\n    - the analysis can be done more thoroughly, for example, it would be interesting to see, what if the frame-wise object detector fails on some intermediate frames, how much VITA can recover it.\n    - not much to say on the weakness.\n\nI generally like this paper, for its reasonable motivation, simple implementation, and good results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "qgiwLeLGL4p", "reviewer": "Reviewer_4omf", "summary": "This paper presents a memory efficient offline video instance segmentation method named VITA. It utilizes object tokens in each frame as condensed frame features which also embed object-level information. On the video level, it collects all the object tokens and use video queries to generate object sequences. The object sequences are multiplied with the frame features to generated object masks similar to MaskFormer work.", "questions": "Other questions:\n-\tAny potential of adapting this method to online VIS? \n-\tHow to determine the number of N_f in the model? Does it have an impact on the performance?\n", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n-\tThe proposed VITA framework achieved state-of-the-art performance on YouTubeVIS and OVIS.\n-\tCompared to previous transformer-based methods, the proposed method effectively reduces running memory with the image features – object tokens – object sequences pipeline. It is able to run hundreds of frames in a single forward process while existing methods can only accommodate a few dozens.\n-\tThe proposed method has faster convergence compared to existing baseline such as Mask2Former-VIS (Figure 5). Making it an appealing choice for model development.\nWeaknesses:\n-\tSome missing references:\nCompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation AAAI 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W8wZLk1D312", "reviewer": "Reviewer_dieT", "summary": "The authors applied Mask2former to generate frame-wise object tokens as a pre-processing of a video.  Afterward, the sliding attention mechanism from Swin Transformer is applied to aggregate all frame-wise understanding for  Video Instance Segmentation. The model trains with the losses from DETR and MaskTrackRCNN. The ensemble achieves a state-of-the-art result on Youtube-VIS \nand OVIS. This is the first offline method applied to the long video dataset OVIS.", "questions": "1. How were the object tokens calculated during the evaluation phase?\n2. Is one GPU enough to fit the video into Mask2Former and the proposed method?\n3. Why did the method improve performance for the challenging dataset OVIS?\n4. What is the FPS of the method compared to prior work?", "limitations": "1. The authors claim to have written the limitation in section 5, but it is not the case.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The method performs state-of-the-art offline video instance segmentation on two challenging datasets. \n2. This is the first time an offline method can run an evaluation on the long video dataset OVIS. \n3. The experiments and ablations are sufficient. The writing is good and easy to understand. The method is described well with interpretable figures. \n\nWeaknesses: \n1. The primary weakness of this paper is the lack of novelty. It is an ensemble of two prior works, i.e., Mask2Former, and Swin \nTransformer. Both of the prior methods have significant performance in their respective objectives. Hence the original contribution of this work is limited.\n2. The authors claim to fit long videos in a single GPU, enabling a long-range offline VIS method. But the Mask2Former features are extracted beforehand. Only the object representation from this prior work is processed in a single GPU. So the claim of fitting \nlong-range video is misleading.\n3. There is no qualitative analysis of the model's performance. It is unclear how or why the modules improve performance in some instances of the problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LT3DgPfS7CL", "reviewer": "Reviewer_KReH", "summary": "This paper introduces a novel paradigm for offline video instance segmentation tasks through explicitly modeling object-oriented queries. Specifically, this paper uses an image detector to distill object-specific contexts into object tokens, then accomplishes video-level understanding by associating frame-level object tokens with an extra object encoder. Experiments on YouTuve-VIS-2019, YouTuve-VIS-2021, and OVIS demonstrate the effectiveness of the proposed method.", "questions": "In general, I think this paper is good with clear writing, a well-explained approach, and good experimental results. The only concern is an unfair comparison with related works.", "limitations": "There are no particular limitations in general. Please see the weakness above.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Using object tokens to directly model spatio-temporal is novel and efficient.\n+ This paper is well written and easy to follow. In addition, the authors also provide the code to reproduce.\n+ Experiments on the VIS benchmark demonstrate the effectiveness of the proposed method.\n+ Long-range video understanding is a crucial topic in video object detection, multi-object tracking, and video instance segmentation. This paper provides a new perspective for such areas.\n\nWeakness:\n+ Since this paper uses mask2former as image instance segmentation, thus it is not fair to compare with the previous SOTA method SeqFormer. I would like to see the results using image instance segmentation as the same as SeqFormer.\n+ Using all image queries may be redundant. So, I suggest abandoning background queries and only using foreground queries for such video instance segmentation tasks. The number of different frames may not be equal. Thus, an alternative way is to use part of object queries to ensure each frame has an equal number of queries.\n+ Why not provide YouTube-VIS-2022 experimental results?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept