SyncVIS: Synchronized Video Instance Segmentation
Authors: rongkun Zheng
Lu Qi
Xi Chen
Yi Wang
Kun Wang
Yu Qiao
Hengshuang Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: video instance segmentation
video-frame synchronization
Primary area: machine_vision
TL;DR: We propose to conduct synchronized modeling explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings

Abstract: Recent DETR-based methods have advanced the development of Video Instance Segmentation (VIS) through transformers' efficiency and capability in modeling spatial and temporal information. Despite harvesting remarkable progress, existing works follow asynchronous designs, which model video sequences via either video-level queries only or adopting query-sensitive cascade structures, resulting in difficulties when handling complex and challenging video scenarios. In this work, we analyze the cause of this phenomenon and the limitations of the current solutions, and propose to conduct synchronized modeling via a new framework named SyncVIS. Specifically, SyncVIS explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings: a synchronized video-frame modeling paradigm and a synchronized embedding optimization strategy. The former attempts to promote the mutual learning of frame- and video-level embeddings with each other and the latter divides large video sequences into small clips for easier optimization. Extensive experimental evaluations are conducted on the challenging YouTube-VIS 2019 & 2021 & 2022, and OVIS benchmarks, and SyncVIS achieves state-of-the-art results, which demonstrates the effectiveness and generality of the proposed approach. The code is available at https://github.com/rkzheng99/SyncVIS.

Syne VIS: Synchronized Video Instance Segmentation
Rongkun Zheng'
Lu Qi
Xi Chen' Yi Wang²
Hengshuang Zhao The University of Hong Kong Universsity of California, Merced Shanghai Artificial Intelligence Laboratory "Sense Research f f=rk228connect, [szhao@cs] hku.hk
Kun Wang*
Yu Qiao
Abstract
Recent DETR-based methods have advanced the development of Video Instance Segmentation (VIS) through transformers' efficiency and capability in modeling spatial and temporal information. Despite harvesting remarkable progress, existing works follow asynchronous designs, which model video sequences via either video-level queries only or adopting query-sensitive cascade structures, resulting in difficulties when handling complex and challenging video scenarios. In this work, we analyze the cause of this phenomenon and the limitations of the current solutions, and propose to conduct synchronized modeling via new framework named VIS. Specifically, Sync explicitly introduces video-level query embeddings and designs two key modules to synchronize video-level query with frame-level query embeddings: synchronized video-frame modeling paradigm and synchronized embedding optimization strategy. The former attempts promote the mutual learning of frame- and vidco-level embeddings with each other and the latter divides large video sequences into small clips for easier optimization. Extensive experimental evaluations are conducted on the challenging YouTTube- VIS 2019 & 2021 & 2022, and OVIS benchmarks, and Sync VIS achieves state-of-the- art results, which demonstrates the effectiveness and generality of the proposed approach. The code is available
Introduction
Video Instance Segmentation (VIS) is fundamental while challenging vision task that aims detect, segment, and track object instances inside videos based on a set of predefined object categories at the same time. With the prosperous video media, VIS has attracted various attention due to numerous vital applications in areas such as video understanding, video editing, autonomous driving, etc.
Benefiting from favorable long-range modeling among frames, query-based offline VIS methods [6. 31, 15, 29, 17, 36] like Mask2Former- VIS [6], and SeqFormer [31] begin to dominate the VIS. Inspired by the object detection method DETR [5], they learn group of queries that track and segment potential instances simultaneously across the multiple frames of a video. On the other hand, online VIS approaches like IDOL [32] also exploit the temporal consistency of query embeddings and associate instances via linking the corresponding query embeddings frame by frame. Albeit the success gained by those methods, we find they barely capitalize multi-frame inputs. practice, the Mask2Former- Vis [6] would significantly perform worse if more input frames are given during training (evidenced in Fig. 3). This is paradoxical our common sense that more frames could facilitate deep learning models obtaining more motion information of instances.
For this problem, many researchers [15, 31, 17] point out that the video-level queries are vitally hard to track the instances well if receiving many frames in training. That is because the trajectory
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trans.
Pred.
Optim
Trans. Dec.
Pred.
Sync.
Dec.
Optim.
Synchronized Frame & Video Querles
Input Video Frame Queries
Video Quertes
Fig. 1. Comparison of video instance segmentation paradigms. Previous methods (left part) like VITA [15] adopt asynchronous query-sensitive sensitive structures model instance appearances and trajectories. Our model (right part) employs frame and video emheddings in query-robust synchronous manner, and they synchronize with each other through the transformer decoder to generate the refined video-level query embeddings the prediction. Also, we employ a synchronized embedding optimization strategy 'Sync. Optim." instcad of the classic optimization approach.
complexity will increase in polynomials along with the number of frames. Therefore, state-of-the-ae methods like SeqFormer [31] and VITA [15] usually decouple the trajectory into spatial and temporal dimensions, which are modeled by frame-level and video-level queries, respectively. Specifically, they utilize the frame-level queries segment each frame independently and then associate these frame-level queries with video-level queries, which are responsible for the final video-level prediction. The well-trained frame-level queries guarantee the quality in the spatial dimension and thus decrease the burden of video queries. However, we argue that two issues remain in these asynchronous designs (as illustrated at the left of Fig. 1). First, with the asynchronous structure, the wellness of video-level queries heavily relies on the learning of former frame-level queries, inside which some motion information may be lost because it is an image encoding stage (rather than video encoding), which leads to the sensitivity of queries to the learning quality of former stages. Second, previous works have not solved the bipartite matching among more frames (rather than single frame), and thus the optimization complexity of trajectories remains exorbitant. Both two issues block the further development of query-based methods for video instance segmentation.
To this end, we propose to model video and frame queries synchronously with a new framework named VIS to address the above-mentioned issues. Built upon DETR-style structures [6. 32], our SIS has two key components: the synchronized video-fram fideo-frame modeling paradigm and the synchronized embedding optimization strategy. Both designs put effort into unifying the frame- and video-level predictions in synchronization. The synchronized video-frame modeling paradigm makes frame- and video-level embeddings interact with each other in a query-robust parallel manner, rather than query- serritsen cascade structure. Then the synchronized embedding optimization strategy adds a video-level buffer state to generate more tractable intermediate bipartite matching optimization compared with only frame-level losses. Fig. I demonstrates the schematic difference between the asynchronous srate-ofthheare method and our synchronous approach. Our model schematically simple but practically more effective, with exquisite designs as follows.
In the synchronized video-frame modeling paradigm, we employ frame and video-level embeddings in the transformer decoder to model object segmentation and tracking synchronously. Specifically, frame-level embeddings are assigned each sampled frame, and responsible for modeling the appearance of instances, and video-level embeddings are a set of shared instance queries for all sampled frames, which are used to characterize the general motion (In the DETR- architecture, when video queries are associated with features across time via the decoder, they can effectively model instance-level motion through the cascade structure. Mask2Former- VIS, the use of video queries alone enables the capture of instance motion). Frame-level embeddings are kept on each frame to attend to instances locally. In each decoder layer, the video-level embeddings are aggregated to refine frame-level embeddings on the corresponding frame. The refined frame-level embeddings, in turn, are aggregated into video-level embeddings. By repeating this synchronization in decoder layers, VIS incorporates the semantics and movement of instances in each frame. In the synchronized embedding optimization strategy, we focus more on video-level bipartite matching. Concretely, we decouple the input video into several clips to synchronize video and frame, and the total number of clips is related to the combinatorial number. Then, we calculate each clip loss independently by video-level bipartite matching, so that video embeddings
maintain their association ability.
We evaluate our Sync on four popular VIS benchmarks, including YouTube- VIS 2019 & 2021 2022 [34], and OVIS- 2022 [27]. The experiments show the effectiveness of our method with signifi-

---

cant improvement over the current state-of-the methods VITA [15], DVIS [38], and CTVIS [37]. Our contributions are as follows:
We analyze the limitations of existing video instance segmentation methods and propose a framework named SyneVIS with synchronized video-frame modeling. It can well character-
ize instances' trajectories under complex and challenging video scenarios.
We develop two critical modules: a synchronized video-frame modeling paradigm and synchronized embedding optimization strategy. The former adopts synchronized paradigm to alleviate error accumulation in cascade structures. The latter divides large video sequences
into small clips for easier optimization.
We conduct extensive experimental evaluations on challenging VIS benchmarks, including YouTube- VIS 2019 & 2021 &2022, and OVIS 2021, and the achieved state-of-the-ae results demonstrate the effectiveness and generality of the proposed approach.
VIS
2
Related Works
Online video instance segmentation. Most online VIS methods adopt the tracking-by-detectionn paradigm, integrating a tracking branch into image instance segmentation models. These methods predict detection and segmentation within local range using a few frames and associate these outputs using matching algorithms. MaskTrack R R-CNN [34] incorporates a tracking branch to Mask R-CNN [12]. Many subsequent approaches [4, 35, 21], follow this pipeline, measuring the similarities between frame-level predictions and associating them with different matching modules. Cross VIS [35] uses the instance feature in the current frame pixel-wisely localize the same instance in another frame. MIn [16] implements a query-based image instance segmentation model [7] on individual frames and associate query embeddings via bipartite matching.
Contrarily, some previous works [9, 19, 10, 14], draw inspiration from Video Object Segmenta- tion [25], Multi-Object Tracking [8, 24, 41, 2, 26, 39], and Multi-Object Tracking and Segmenta- tion [28]. GenVIS [14] adopts novel target label assignment strategy and builds instance prototype memory in query-based sequential learning. IDOL [32], based on Deformable- DETR [42], introduces a contrastive learning head that acquires discriminative instance embeddings for association [1 I].
CTVIS [37] improved upon IDOL by constructing consistent paradigm for both training and inference. However, online VIS methods usually adopt frame-level query and ignore the video-level associations across non-adjacent frames, which is problematic when handling complex long videos.
Offiine video instance segmentation. Offline methods predict instance masks and trajectories through the whole video in one step using the whole video as input. STEm-Seg [1] proposes
single-stage model which learns and clusters the spatio-temporal embeddings. MaskProp [3] and
Propose-Reduce [19] improve association and mask quality by mask propagation. Efficient-VIS [30]
uses a tracklet query paired with a tracklet proposal to represent object instances. VisTR [29]
successfully adapts DETR [5] to VIS, using instance queries model the whole video. IFC [17] proposes inter-frame communication transformers, using memory tokens to model associations across frames. By adapting 2askeFormer [7] 3D spatio-temporal features, Mask22ormee-VIS. [6]
becomes the state-of-thh-aa by exploiting its mask-oriented representation. TeVIT [36] introduces new approach based transformers instead of the CNN backbone and associates temporal information efficiently. SeqFormer [31] decomposes the shared instance queries into frame-level box ones and utilizes video-level instance queries to relate different frames. Recently, VITA [15] uses object tokens to represent the whole video and employs video queries to decode semantics from object tokens. TMT-VIS [40] manages to jointly train multiple datasets to improve performance via different taxonomy information. However, these methods typically implement only video query or utilize asynchronous structures, and the final query-sensitiin approaches have difficulties dealing with complex scenarios.
3 Method
Video instance segmentation can be formulated into a set prediction problem, which can be addressed by a DETR [5] style framework like Mask2Former [7]. We first revisit the Mask2Former-VIS [6], one of the baselines that our method is built on. Then we propose a synchronized transformer framework
named Sync VIS to address challenging video scenarios, with its two key designs.
3

---

Synchromizdd Video Frame Moddligg
Synchesnired Enzhesking Opaiiiction
Appeodec
Dowrs
Aguroaiaka
Rack-
Frame Pestures
Inpet
Fig. 2. Overview of the proposed synchronous video-frame modeling framework VIn The developed syn- chronized video-frame modeling paradigm enables video-level embeddings and frame-level ones to synchronize with cach other each stage of the decoder. SyncVIS also suggests a new synchronized embedding optimization strategy. As shown in the right part, SIn decouples the input video frames into several sub-clips and feeds each cub-clip into the mask and classification head. By applying these modules, SIS incorporate both semanties and movement of instances in each frame in synchronous manner for superior characterizing ability.
3.1 Revisiting Mask2Former
Mask2Former [6, 7] is universal Transformer-based framework for image video instance seg- mentation. Given an input sample, Mask2Former adopts a Transformer-bet based decoder, which first learns N number of C-dimensional queries Q R NCC to generate embeddings E E x then predicts N segmentation masks based on the generated embeddings, where the 2nd, 3rd and 4th dimensions of E correspond to temporal T, height H, and width W dimensions respectively. Here, we note that the transformer decoder is a nine-layer structure, where 1th layer cascades a masked cross-attention licx. a self-attention A and a ferd-forward network FFN!. For fraee-level Mask2Former, E is expanded along spatial dimensions W and H to the shape of N x 1 H W x C. Alternatively, for video-level Mask2Former,
is with a shape N x X x x W x C, and the combination of temporal T and spatial dimensions W and H enables Mask2Former to utilize the shared embeddings to represent the same visual instances across different frames consistently. Finally, E is utilized for instance-level classification pe and pixel-level mask prediction pmm..
Analysis. Although Mask2Former- VIS [6] has achieved impressive results, it exhibits notable performance degradation when dealing with complex videos. For instance, observe decrease in average precision (AP) of 1.5% when number of input frames increases to ten. This observation counter-intuitive as we expect models to improve their performance with increased number of training frames. We hypothesize that this decline in performance stems from the insufficiency of stand-alone vidco queries for effective long-range video modeling. In the case of challenging long- range video sequences, there is a need model more instances and their corresponding movements using video-level queries. This unexpected scenario suggests that there is significant demand for distinct sets of queries that can effectively characterize both the object categories and movement trajectories in video sequences.
3.2 Overall Architecture
The VIS new framework designed to improve the representation of long video frame infor- mation and optimize system learning processes. It combines video-level and frame-level embeddings synchronously, which enhances the overall functionality of the framework. The framework is depicted in Fig. 2 and features two fundamental modules, i.c., synchronized video-frame modeling paradigm (Sec. 3.3) and a combinatorial embedding optimization strategy (Sec. 3.4).
3.3 Synchronized Video-Frame Modeling
Synchronized video-frame modeling is a strategy designed to avoid the sensitive cascading in previous methods and improve the synchrony between the frame-level embeddings X E TxCC and video-

---

level X. E RIXNXC. X focuses on every frame separately, while x, mainly interacts with the whole video features.
Based on the design of the transformer decoder, we concurrently introduce frame- and video-level embeddings to each layer. Here, the frame- and video-level embeddings are replicated for T and 1 times by learnable frame- and video-level embeddings at first when given a video with T' frames. Thus both the frame- and video-level embeddings pass the transformer decoder layer and two kinds of interaction operations for synchronous exchange and refinement, For each step, these two embeddings are updated as follows:
X FFN (hiA (hiaa F))),
(1)
where E {f, v} indicates the frame- or video-level embeddings and means the pyramid features extracted from the backbone. X is the embeddings processed by the (th transformer decoder layer. The hew (g. r) indicates the cross-attention with video-level query embedding q and frame-level reference embedding In our design, frame-level embeddings are assigned to each sampled frame, and responsible for modeling the appearance of instances, and video-level embeddings are a set of shared instance queries for all sampled frames, which are used to characterize the general motion (because they encode the position information of instances across frames, and thereby naturally contain the motion information).
Then, we feed the frame- and video-level embeddings into the proposed synchronous structure for mutual information exchange and refinement follows:
x!- hew (x!+) FEN" (x + (1-
hes (X/+) FFN" (x + (1 (1- x
(2)
(3)
where 'v-s' and 'f-s' mean that we only select top N₁ embeddings in key and value to interact with the query, while 'fv' and 'vf" indicate the refinement direction of the feedfoward network, from frame to video and video to frame. (set to 0.05) is the update momentum of video-level embeddings, because we presume that the aggregation of frame-level features should not change the general video-level embeddings significantly, and vice versa.
The motivation behind this approach is similar to that of the masked attention mechanism used in Mask2Former. The key difference lies in the dimension where the masking happens. In Mask2Former, the strategy is to mask out the background regions within the spatial dimension. On the other hand, our method works differently by masking out background embeddings within the key and value dimensions. This is done by selecting the top N₂. embeddings based on the confidence scores provided by the prediction head. Therefore, while both methods aim to reduce the influence of irrelevant background information, they do so in different ways: Mask2Former masks spatially, while our method targets key and value embeddings.
3.4 Synchronized Embedding Optimization
Video-level bipartite matching is a challenging memory-cottly problem that remains asynchronous: The matching approaches from previous VIS methods are adapted directly from DETR, so the complexity of matching increases with the number of frames in a video, as the instances are not restrained to a single frame, but could be any frame the video. Even though larger input frames can bring more trajectory information of instances for prediction, this presents a challenge due to the resulting trajectory complexity, which scales polynomically with the input. Conversely, when the input is insufficient, the system may lack the necessary information to function optimally. Such asynchrony is the motivation of our new optimization strategy.
Regarding this, we present the synchronized embedding optimization strategy using the divide-and- conquer: if we want to associate frame f; to frame ty and yet the time interval may be large, an effective approach is to find k, s.t < k < j. and associate & to t4 as well as ty to ty. When the model achieves better segmentation results on sub-clips, combining these local optimums and we can achieve a better matching. Therefore, when generating the output predictions, we would divide the predictions into several sub-clips, and optimize each sub-clips independently. This sub-clip like a video-level buffer to help synchronizing video-level and frame-level embeddings. By optimizing the local sub-sequence of the video, rather than the entire video sequence, if the target instance becomes occluded in certain frames, our optimizing strategy can adjust the features within the sub-sequence to adapt to this change, without being affected by the unoccluded frames. The size of sub-clips, Te, is variable across all VIS datasets: as for VIS datasets with fewer instances per video, such as
5

---

Youtube-VIL VIS 2019, T, is set to 3, while for OVIS, T, works best at 2 (discussed in Sec. 4.4). In order further reduce the complexity for better optimization. dividing into smaller sub-clips can accelerate the optimization. Also, keeping the size of two is able to maintain the temporal information. In this way, our video-level objective by could be divided into several clips follows:
Ly
Celip(i.))
OSGASTT
where T indicates the number of input frames, And the overall training loss & for our model can be formulated as:
L= Σ
G
Σ
Chee (Pki G
kE(v.))
kv(v.))
Σ
Edice (P GF) + Leantras:
(5)
kE(v.f)
where LF and L denote the cross-entropy loss for frame- and video-level classification. Similarly, Cpe, ..ve, Ciiie, and Cdiie denote the binary cross-entropy and dice loss for frame- and video-level mask prediction, respectively. Here P is the prediction, and G is the ground truth, and refers to classification while m refers to mask. Leontras represents the contrastive loss, which is applied in online settings (but not in offline) IDOL [32] does, where the previous frame is set as reference frame and the current frame is set as key frame.
3.5 Implementation Details
regarding the pixel and transformer decoder are the same as these of Mask2Former- VIS [6]. the synchronized video-frame modeling, we set the number of frame-level and video-level embeddings N to 100. To extract the key information, we set the N₂ 10. Following the design of Farke VIS [6], we first trained our model on COCO [20] before training VIS datasets. We use the AdamW [23] optimizer with a base learning rate 5e-4 on LaineLarge backbone in Youtube 2019 (we use different training iterations and learning rates for different datasets). During inference, each frame's shorter side is resized to 360 pixels for ResNet [13] and 448 pixels for Swin [22]. Most of our experiments are conducted on A100 GPUs (80G), and cuda 11.1, PyTorch 3.9 environment. The training time is approximately 1.5 days when training with the Swin-L backbone.
Our method is built on detectron2 [33].
Experiments
Datasets and metrics. YouTube- VIS dataset is large-scale video database for video instance segmentation. The dataset has seen three iterations, in 2019, 2021, and 2022, with each adding more challenges to the dataset [34]. The first iteration, YouTube- VIS 2019, contains 2.9k videos with an average duration of 4.61 seconds. The validation set has an average length of 27.4 frames per video and covers 40 predefined categories. The dataset was updated to YouTube-Vii VIS 2021 with longer videos with more complex trajectories. As a result, the validation videos' average length increased to 39.7 frames. The most recent update, YouTube-Ve VIS 2022, adds an additional 71 long videos to the validation set and 89 extra long videos to the test set.
OVIS dataset is another resource for video instance segmentation, particularly focusing on scenarios with severe occlusions between objects [27]. It consists of 25 object categories and 607 training videos. Despite a smaller number of training videos compared to the YouTube-VIE datasets, the OVIS videos are much longer, averaging 12.77 seconds each. OVIS emphasizes the complexity of the scenes and the severity of occlusions between objects.
4.1 Main Results
We compare VIS with satt--o-the-ass approaches which are with ResNet-50 and Swin-L back- bones on the YouTube- VIS 2019 & 2021 & 2022 [34] & OVIS 2021 [27] benchmarks. The results reported in Tables 1, 2 and 3.
You VIS 2019. Table shows the comparison on YouTube- VIS 2019. When applying our design to CTVIS, we discover that the forward passing of CTVIS is still asynchronous. While

---

Table 1. Results comparison on the YouTube-VIS 2019 and 2021 validation sets. We group the results by online offline methods, and then with ResNet-50 or Swin-I. backhone structures. Syne VTS is the model which we add our two designs based on CTVIS decoder and optimization, our module could seamlessly integrate with both online & offline approaches without bells and whistles. Our algorithm gets the best AP performance under all of the settings.
VITA. Typically, since our design is orthogonally designed for
YouTube- VIS
2019
YouTube- VIS
2021 ARR
Method
Backbone
AP
APso
AP7s
AR;
AP
AP5o
AP7:
1R
Online
CrossVIS [35]
ResNet-50
36.3
56.8
38,9
35.6
40.7
34.2
54.4
37.9
30,4
38.2
MaskTrack R-CNN [34]
ResNet-50
38.6
56.3
43.7
35.7
42.5
36.9
54.7
40.2
30.6
40.9
MinVIS [16]
ResNet-50
47.4
69.0
52.1
45.7
55.7
44.2
66.0
48.1
39.2
51.7
TCOVIS [18]
ResNet-50
52.3
73.5
57.6
49.8
60.2
49.5
71.2
53.8
41.3
55.9
IDOL [32]
ResNet-50
49.5
74.0
52.9
47.7
58.7
43.9
68.0
49.6
38.0
50,9
DVIS [38]
ResNet-50
51.2
73.8
57.1
47.2
59.3
46.4
68.4
49.6
39.7
53.5
CTVIS [37]
ResNet-50
55.1
59.1
51.9
63.2
50.1
73.7
54.7
41.8
59.5
SyncVIS
ResNet-50
57.9
81.3
60.8
53.1
64.4
51.9
74.3
56.3
43.0
60.4
MinVIS [16]
Swin-L
61.6
83.3
68.6
54.8
66.6
55.3
76.6
62.0
45.9
60.8
DVIS [38]
Swin-L
63,9
87.2
70.4
56.2
69.0
58.7
80.4
66.6
47.5
64.6
TCOVIS [18]
Swin-L
64.1
86.6
69.5
55.8
69.0
61.3
82.9
68.0
48.6
65.1
IDOL [32]
Swin-L
64.3
55.6
69.1
80.8
63.5
45.0
60.1
CTVIS [37]
Swin-L
65.6
87.7
72.2
56.5
70.4
84.0
68.8
48.0
65.8
SyncVIS
Swin-L
67.1
88.9
73.0
57.5
71.2
62.4
84.5
69.6
49.1
66.5
EfficienVVSS [30]
Re
37.9
59.7
43.0
40.3
46.6
34.0
$7.5
37.3
33.8
42.5
IFC [17]
ResNet-50
41.2
65.1
44.6
42.3
49.6
35.2
55.9
37.7
32.6
42,9
Mask2Fomer- [6]
ResNet-50
46.4
68.0
50.0
40.6
60.9
41.8
TeVIT [36]
MsgShifT
46.6
71.3
44.9
54.3
37.9
61.2
42.1
35.1
44.6
SeqFormer [31]
ResNet-50
47.4
69.8
51.8
45.5
54.8
40.5
62.4
43.7
36.1
48.1
Offine
VITA [15]
ResNet-50
49.8
72.6
54.5
49.4
61.0
45.7
67.4
49.5
40.9
53.6
DVIS [38]
ResNet-50
52.6
74.5
58.2
47.4
60.4
47.4
71.0
51.6
39.9
55.2
SyneVIS
ResNet-50
54.2
75.1
51.2
61.7
48.9
71.4
52.8
40.4
57.9
SeqFormer [31]
Swin-L
59.3
82.1
66.4
51.7
64.4
51.8
74.6
58.2
42.8
58.1
Mask2Former-VV VIS [6]
Swin-L
60.4
84.4
67.0
$2.6
76.4
57.2
VITA [15]
Swin-L
63.0
86.9
67.9
56.3
68.1
$7.5
80.6
61.0
47.7
62.6
DVIS [38]
Swin-L
64.9
87.0
72.7
56.5
69.3
60.1
82.0
67.4
47.7
65.7
SyneVIS
Swin-L
65.7
87.3
56.7
69.8
60.3
81.8
67.5
48.6
65.4
single frame produces the frame embedding, there is no explicit video-level embedding to interact with the frame-level instance embedding. In our design, we add set of video-level embeddings that gradually update with the frame-level embeddings, Our Sync VIS sets new state-of-the-ass results under all of the settings. Among the online approaches. Sync gets the highest performance of 57.9% AP and 67.1 % AP with ResNet-50 and Swin-L. backbones, which outperforms the previous best solution CTVIS [37] by 2.8 and 1.5 points, exceeds the top-ranking method DVIS [38] by 6.7 and 3.2 points, respectively. We list the model parameters and FPS of SeqFormer (220M/27.7), VITA (229M/22.8), and our VIS (245M/22.1). Our model performs notably better with similar model parameters and inference speed. The two designs in SyncVIS can also boost the performance of both offline and online VIS solutions and can set new records in both settings, demonstrating the effectiveness and importance of synchronous modeling.
You TUS 2021 & 2022. Table also compares the results on YouTube-VIL VIS 2021. Our method hits the new records on the two backbone settings. SIn achieves 51.9% AP and 62.4% AP with ResNet-50 and Swin-L backbones, respectively, outperforming the previous SOTA by 1.8 and 1.2 points, which further demonstrates the effectiveness of our approach. In Table 2, SIS exceeds the previous SOTA by 1.1 points, proving its potency in handling complex long video scenarios.
OVIS. Table 3 illustrates the competitiveness of Sync on the challenging OVIS dataset. SyncVIS also shows superior performance over other perforaeformance algorithms with 36.3% AP and 50.8% AP on ResNet-50 and Swin-L. backhones, outperforming the current strongest architecture DVIS [38] by 2.2 and 0.9 points, respectively. VIS harvests the highest performance all four datasets, further evidencing its effectiveness and generality.
4.2 Ablation Studies
We ablate our proposed components, which are conducted with ResNet-50 on YouTube- VIS 2019.
7

---

Table 2. Results comparison on the YouTube-VVR
Table 4. Experiments aggregating our design to various
2022 long vidcos.
popular VIS methods.
Method
AP
AP50
APrs
ARt
AR1o
Method
AP
Method
AP
MinVIS [16]
33.1
54.8
33.7
29.5
36.6
Swin-L.
VITA [15]
41.1
63.0
39.3
44.3
2arm2rr VIS [6]
45.1
VITA [15]
49.5
DVIS [38]
45.9
69.0
48,8
37.2
51.8
Synchromized Modllling
50.3
Synchremized Modeling
53.0
SIncVIS
47.0
69.4
48.6
38,9
52.4
Synchronized Optmiizttio
46.7
Synchronized Optimization
51.2
Both (SyncVIS)
51.5
Both (SyncVIS)
54.2
Table 3. Results comparison on the OVIS.
TMI-VIS
47.3
DVIS
52.6
Method
AP
APse
APrs
AR:
AR
Synchronized Modeling
51.1
+ Synchronized Modeling
$4.9
R-50
DVIS [38]
34.1
59.8
32.3
15.9
41.1
Synchronized Optimization
Both (Sync
48.7
51.9
Syachronized Optimization
Both (8yneV1S)
54.0
55.8
SyncViS
36.3
60.9
33.0
17.0
42.8
GeaVIS [14]
51.3
IDOL [32]
49.5
CTVIS [37]
46.9
71.5
47.5
19.1
52.1
Synchromized Modeling
Synchrenized Modeling
55.1
[38]
49.9
53.0
19.4
55.3
Synchmaized Optimization
52.7
Synchroeized Optimization
51.3
SIn
50.8
75.7
53.1
20.5
55.9
(SyncVIS)
Both (Syn:V1S)
56.5
54
Table 5. Ablation study synchronized video-frane modeling.
52
ID Frame Video-+Frame
AP
AP50
APrs
I
45.1
65.7
49.0
AP(%)
50
V
50.2
72.5
54.2
VIN
III
71.4
51.8
IV
51.5
73.2
55.9
48-
Table Ablation study on the structure and query selection of synchronized video-fraat frame modeling.
46
Method
AP
APso
APrs
2
10
Input Frames
Cosaade Structure Frume-Sovel Queries
Cascade Structure Vidco-leee Queries
46.7
68.2
50.3
Fig. 3. Ablation study on the complexity of video scenarios regarding the number of input frames T.
Cascade Stucture Both Queries
49.9
72.0
54.4
Synchronous
Both Queries (SyncV1S)
Complexity of video scenarios. Changing the complexity of video scenarios can check the ca- pabillity of VIS solutions. We define the complexity instances, which is calculated as the maximum combination of trajectories between frames. For example, if frame & has n instances while - frame has m. then the maximum complexity would be mnn and thus complexity is in polynomials with input frames, and we could use the frame number as an indicator of complexity. We examine the effect of different numbers of input frames in Fig. 3. We find the popular Mask2Former- VIS framework meets difficulties when dealing with complex videos, i... T works best for the model, and as T continually increases, the performance will degrade notably. In contrast, as we increase the input frames, our Sync improves gradually and achieves the best performance at 9. This evidences that our model is capable handling challenging scenarios and can well characterize the movement trajectories of video instances.
indicator of the movements of different
Key component designs. Table 4 demonstrates the effect of our component designs when combined with the prevalent VIS methods. By aggregating the synchronized video-frama modeling paradigm, Mask2Former- VIS achieves a huge gain of 5.2 points in AP performance. This is credited to the design of two levels of queries as well as their mutual interactions. The synchronized embedding optimization strategy further advances performance improvement across all VIS methods. Aggregating two designs could also boost VITA by 3.5 and 1.7 points in performance, respectively. Note that the gain of 7.0 points for IDOL is also contributed by changing its original backbone to Mask2Former. Foasee--based backbone. The extensive results in Table show that our new designs can introduce consistent improvements to various popular VIS methods, further indicating the effectiveness and generality.
4.3 Synchronized Video-Frame Modeling
Enhancement direction. In Table 5, we investigate the effect of the direction of the modeling paradigm, including synchronous bidirectional and asynchronous unidirectional ones. Unidirectional embedding enhancement can be divided into two types according to the output of the transformer decoder: utilize the frame-level embeddings to enhance the video-level ones, and the aggregation module consists of an FFN and cross-attention layer (denoted FFrame Vriueo+Video' )); ii) adopt the video-
8

---

level embeddings to update the frame-level ones, and feed the frame-level embeddings to prediction heads to generate the masks and instance classes independently (denoted as "Video->Frame")
In Table 5, we find that without embedding enhancement, the decrease in performance is conspicuous as up 6.4 points. With either unidirectional asynchronous embedding enhancement strategy, the result gets improved but is still not paired with the bidirectional synchronized video-frame modeling, This signifies several points: first, introducing frame-level embeddings to refine video-level embed- dings can increment the performance by adding more frame-level instance details, thus strengthening the representative ability of video-level embeddings. Second, video-Ievel embeddings contain more spatial-tempora temporal information, and utilizing video-level embeddings to predict segmentation results for the video can receive better results. Third, adopting synchronized video-frame modeling is better than unidirectional modeling. Even though adding frame-specific information to video-level embeddings can contribute to representing more instance details, building the mutual association and aggregation leads to a stronger representation ability to characterize the semantics and motions.
Modeling structure. We suppose the superiority of using a synchronous structure over a cascade one that the former avoids motion information loss and error accumulation. In Table 6, we evaluate these two structures. For the cascade structure, we use frame-level embeddings to extract information and associate image-level embeddings with video-level ones. The synchronized rideo-frams modeling and synchronized embedding optimization remain the same in cascade structure experiments. The synchronous structure gets 1.6 points higher AP performance than the cascade one, demonstrating the superior design of the proposed synchronous structure over the classical cascade structure.
Query selection. As shown in Table 6, utilizing only video-level queries performs better than only adopting frame-level ones. Frame-level queries segment each frame independently and focus less on the association across frames, which leads to lower performance. Our synchronous model, on the other hand, adopts both queries and achieves the best performance, validating the effectiveness of our synchronized video-frame modeling paradigm.
Aggregation strategy. Table 7 shows the results of different aggregation strategies in the synchro- nized video-frame modeling. In the 'Query Sim- ilarity', we select the most similar embeddings by computing the cosine similarity between video- level and frame-level embeddings. Note we com- pute similarities frame--y- bore-bb--rraeeee and concatenate the top N₂ embeddings together as input to the aggregation module. In the 'Mask Similarity', we get similarities of corresponding mask embeddings determine the most similar ones. We use class scores (i.e., *Class Prediction') )) to select key embeddings that work the best. Since some objects only appear in few frames, the most similar embeddings may represent the background in extreme cases, disturbing the useful information for discrimination. Both aggregation methods have such problems, and using mask similarity is even worse since masks insufficient encode motion fully, leading to ineffective similarity calculation.
Table 7. Ablation study on aggregation strategies.
Method
AP
AP50
APTs
AR1
18
Query Similarity
49.7
72.8
53.2
48.7
60.3
Musk Similarity
48.2
71.6
52.8
47.8
59.1
Class Prediction
51.5
73.2
55.9
49.5
60.4
Aggregation embedding size. Table shows the performance of SIS with varying numbers of embedding in the aggregation stage of the synchronized video-frame modeling paradigm. When selecting top N₂- = 10 embeddings to aggregate, the model performance reaches its best. When N% decreases, the aggregated key information contained in embeddings is not sufficient, the selected one may not encode the semantic information of all instances in the video, and therefore cause the drop in performance. Alternately, when N& gets larger than optimum, the redundant query features dilute the original information, which also leads to performance degradation.
4.4 Synchronized Embedding Optimization
Sub-clips size. Table 8 shows the results of SIncVIS with a varying T, of sub-clips. The larger the sizes of sub-clips are, the more complicated the optimization will be, and embeddings are less likely to capture the proper semantics and trajectories. When we set the size of sub-clips to 3, the model achieves its best performance. When T, decreases to the lowest, the problem of optimizing the whole video descends to optimizing each frame, weakening the model's ability to associate

---

Table 8. Ablation study on the aggregation embed- ding size N₂ of synchronized pidco-frams modeling paradigm and the sub-clip size T, of synchronized cm- bedding optimization strategy.
Table 9. Ablation study on syachronized embedding optimization strategy with ResNet-50 backbone.
Datasets
Method
[A
APso
AP7s
Mask2Former VIS
45.1
65.7
49.0
N1
AP
AP
APss
AR;
ARw
T,
APwe
APs
AR,
A
YouTube VIS 2019
+ Optimization
46.7
68.6
50.7
10
51.1
51.5
73.2
73.0
55.4
55.9
49.5
49.1
59.3
60.4
2
50.9
513
73.3
73.7
54.9
55.6
49.0
492
60.1
60.2
YouTube-VIS 2021
Mask VIS
+ Optimization
41.3
39.8
62.1
59,8
42.5
41.5
50
50.9
49.3
73.5
728
55.1
52.3
47.4
48.4
59.6
56.7
51.5
50.7
73.2
738
54.1
55.9
49.5
47.9
60.4
58.9
OVIS
Mask2Former-V
10.6
25.4
7.2
100
47.5
70.4
51.4
46.8
56.1
5
50.4
73.3
54.2
472
58.1
Optimizaaiion
12.3
27.1
9.2
frames temporally. When T, increases, though there is a gain in the performance when compared to undivided circumstances, the optimization is still more complex, making the training process hard to reach optimum. Learned embeddings are insufficient to capture all semantics for sub-clip, and therefore the performance is weaker than the optimal T, value. However, T, = 3 is the optimum for Youtube- VIS 2019 & 2021. For Youtube- VIS 2022 and OVIS, SIn performs best when Ta is 2, which is the smallest size to maintain temporal associations. We suppose, that for more complex scenarios, dividing into smaller sub-clips is beneficial for query embeddings to associate across frames and accelerate the optimization. In optimization strategy, our main goal is reduce the increasing optimization complexity as the input frame number grows. To realize this target, our strategy is to divide the video into several sub-clips that could make optimization easier while retaining the temporal motion information. Longer Sub-clips could provide the model with more temporal information, but their optimization complexity also rises polynomially. By optimizing sub-clips, models can better adapt to changes in the target instance within the video, particularly in cases of occlusion of many similar instances (In OVIS, most cases are videos with many similar instances, most of which are occluded in certain frames). By optimizing the local sub-sequence of the video, rather than the entire video sequence, if the target instance becomes occluded in certain frames, our optimizing strategy can adjust the features within the sub-sequence to adapt to this change, without being affected by the unoccluded frames.
The proposed optimization strategy is effective and general that can be adapted into various DETR-based approaches. In these frameworks, the optimization problem for long video sequences still exists. As in Table 9, when adding our optimization strategy to Mask2Former- VIS. we harvest notable performance gains on all three benchmarks. This demonstrates that the proposed optimization can be treated robust design suitable for different video scenarios.
Generality.
Conclusion
We have proposed SyncVIS for synchronized Video Instance Segmentation. Unlike the current VIS approaches that use asynchronous structures, SyncVIS utilizes a synchronized fadeo-frame modeling paradigm to encourage the synchronization between frame embeddings and video embeddings in a synchronous manner, which incorporate both semantics and movement of instances more effectively. Moreover, VIS develops a plug-and-use synchronized embedding optimization strategy during training, which reduces the complexity of bipartite matching in divide-andd-onquee approach. Based on these two designs, our Sync outperforms current methods and achieves SOTA on four challenging benchmarks. We hope that our method can provide valuable insights and motivate the future VIS research.
Broader impacts and limitations. SIS is designed to propose a new synchronized structure for VIS with promising performance. We hope this work can contribute to further applications in video-related tasks and real-life applications. However, even though model achieves promising results, it has a problem segmenting very crowded or heavily occluded scenarios, which is discussed in the supplementary.
Acknowledgement. This work is partially supported by the National Natural Science Foundation of China (No. 02201484), National Key R&D Program of China (No. 20222D0160100,, HKU Startup Fund, and HKU Seed Fund for Basic Research.
10

---

References
[1] Ali Athar, Sabarinath Mahadevan, Aljosa Osep, Laura Leal-Taixé, and Bastian Leibe. Stem-seg: Spatio-temporal embeddings for instance segmentation in videos. In ECCV, 2020. 3
[2] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In ICCV, 3
[3] Gedas Bertasius and Lorenzo Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR, 2020. 3
[4] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV, 2020. 3
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 1, 3
[6] Bowen Cheng, Anwesa Choudhuri, Ishan Mísra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. arXiv:2112.10764, 2021. 1, 2,
4, 6, 7, 15, 16
[7] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022. 3,
[8] Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé. Motchallenge: A benchmark for single-camera multiple target tracking. IJCV, 2021.
[9] Yang Fu, Linjie Yang, Ding Liu, Thomas S Huang, and Humphrey Shi. Compfeat: Comprehen- sive feature aggregation for video instance segmentation. In AAAI, 2021.
[10] Su Ho Han, Sukjun Hwang, Scoung Wug Oh, Yeonchool Park, Hyunwoo Kim, Min-Jung Kim, and Seon Joo Kim. Visolo: Grid-based space-time aggregation for efficient online video instance segmentation. In CVPR, 2022. 3
(11) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. ICCV, 2017.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Miran Heo, Sukjun Hwang, Jeongseok Hyun, Hanjung Kim, Scoung Wug Oh, Joon- Young Lee, and Seon Joo Kim. A generalized framework for video instance segmentation. In CVPR, 2023.
3,8
[15] Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim. Vita: Video instance segmentation via object token association. NeurlPS, 2022. 1, 3, 8, 15, 16
[16] De-An Huang. Zhiding Yu, and Anima Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. In NewrIPS, 2022. 7,8
[17] Sukjun Hwang, Miran Heo, Scoung Wug Oh, and Scon Joo Kim. Video instance segmentation using inter-frame communication transformers. In NeurlPS, 2021. 1,
[18] Junlong Li, Bingyao Yu, Yongming Rao, Jie Zhou, and Jiwen Lu. Tcovis: Temporally consistent online video instance segmentation. In ICCV, 2023. 7
[19] Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia. Video instance segmentation with propose-reduce paradigm. In ICCV, 2021.
[20] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014. 6
11

---

[21] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingje Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
6
[23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.0S01,,
2017. 6
[24] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. In CVPR, 2022.
[25] Seoung Wug Oh, Loon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, 2019.
[26] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In CVPR, 2021. 3
[27] Jiyang Qi Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: benchmark. IJCV, 2022. 2, 6, 14
[28] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In CVPR, 2019. 3
[29] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2021. 1,3
[30] Jialian Wu, Sudhir Yarram, Hui Liang, Tian Lan, Junsong Yuan, Jayan Eledath, and Gerard Medioni. Efficient video instance segmentation via tracklet query and proposal. In CVPR, 2022. 3.7
[31] Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In ECCV, 2022. 3.
[32] Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. In ECCV, 2022. 2, 3, 6, 7,8
[33] Yuxin Wu, Alexander Kirillov, Francisco Massa, Yen Lo, and Ross Girshick. Detectron2. https: /ttpt///g/tttt/t/et http://ccooooreeaccc//aeeettttto 2019. 6
[34] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. 2, 7,
14
[35] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV, 2021. 3,
[36] Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR, 2022. 3,7
[37] Kaining Ying. Qing Zhong, Weian Mao, Zhenhua Wang, Hao Chen, Lin Yuanbo Wu, Yifan Liu, Chengxiang Fan, Yunzhi Zhuge, and Chunhua Shen. Ctvis: Consistent training for online video instance segmentation. In ICCV, 2023. 3, 7.8
[38] Tao Zhang, Xingye Tian, Yu Wu, Shunping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. Dvis: Decoupled video instance segmentation framework. In ICCV, 2023. 3, 7,8
[39] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification in multiple object tracking. IJCV, 2021.
12

---

[40] Rongkun Zheng, Lu Qi, Xi Chen. Yi Wang, Kun Wang, Yu Qiao, and Hengshuang Zhao. Tit-vis: Taxonomy-awaa aware multi-dataset joint training for video instance segmentation. In NeurlPS, 2023. 3, 8
[41] Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl. Global tracking transform- ers. CVPR, 2022.
[42] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
13

---

Appendix
This appendix provides more details about the proposed VIn more qualitative visual compar- isons, and the codebase of our implementation. The content is organized as follows:
More ablation study experiments of the SInc
The qualitative visual comparisons between popular VIS methods and our Vinc
The codebase is contained in the link:
https: / github. com/ 99 SyneVIs
A Dataset Details
Here, we provide detailed overview of various VIS datasets in Table 10. Our extensive experimental evaluations are conducted on four challenging benchmarks, namely YouTube- VIS 2019, 2021, and 2022 [34], and OVIS [27]. YouTube-Vil VIS 2019 [34] was the first large-scale dataset designed for video instance segmentation, comprising 2.9K videos averaging 4.6ls in duration and 27.4 frames in validation videos. YouTube-VIS. 2021 [34] poses greater challenge with longer and more complex trajectory videos, averaging 39.7 frames validation videos. The OVIS [27] dataset is another challenging VIS dataset with 25 object categories, focusing complex scenes with significant object occlusions. Despite containing only 607 training videos, OVIS's vidcos last an average of 12.77s. Lastly, the most recent update, YouTube- VIS 2022, adds additional 71 long videos the validation set and 89 extra long videos to the test set.
B Additional Ablation Studies
Update momentum. In this part, we show the performance of different values of which the update momentum in the synchronized video-frame modeling module. When À equals zero, the whole synchronization between two levels of embeddings is collapsed, and thus a huge degradation in performance is shown in Table 11. As the grows larger than the optimum value, the synchronization can not bring further gain. Rather, the aggregation interferes with the updating of both levels of embeddings in the decoder, which leads to a less increase in performance. Noted that in this experiment, we base our approach on IDOL instead of Mask2Former or CTVIS.
Limitations. As for limitations, our model has a problem in segmenting very crowded or heavily occluded scenarios. Even though our model shows better performance in segmenting complex scenes with multiple instances and occlusions than previous approaches (as shown in visualizations in main paper and supplementary file), handling with extremely crowded scenes not our main focus. Our Sync VIS, on the other hand, aims to build consistent video modeling by synchronously implementing both video-level and frame-leved embeddings as well as synchronized optimizations. We provide visualizations in our github repo: https: /gi githhb......... com/ wwom/o/eeee rkzheng99 SyneVIs.
C Visualization
Visual comparisons of different VIS methods are illustrated in Fig. 4. Our proposed SyncVIS obtains accurate segmentation masks and captures occluded movement trajectories in challenging video scenarios, evidencing its effectiveness over traditional solutions. In the visualization comparisons between Mask2Former- VIS and our model, we select some cases under different scenarios, which include setting with multiple similar instances, setting with reappearance of instance, setting with different poses of instance, and settings with long video in Fig. 5, Fig. and Fig. 7. The high- quality segmentation results under these diverse circumstances and scenarios prove our model's robustness and generality in modeling both semantics and movements of objects. Also, we choose visualizations of implementing different levels of embeddings in Fig. The comparisons further prove the effectiveness of synchronized video-frame modeling.
14

---

Table 10.
Key statistics of popular VIS datasets. YTTVIS" is the acronym of Youtube-Vii VIS"
Table 11. Ablation study of À in synchronized video- frame modeling paradigm. The results are evaluated on the Youtube- VIS 2019 dataset.
YTVISI9 YTVIS21
OVIS
Videos
2883
3859
901
A
AP
APso
APrs
X
AP
APto
AP75
Categories
0.0
52.5
74.8
57.3
0.10
56.1
78.8
99.3
Instances
4883
8171
5223
0.01
54.9
77.6
58.7
0.12
53.7
78.3
58.9
Masks
131K
232K
296K
55.8
78.9
0.15
58.4
Masks Frame
0.05
56.5
79.5
59.8
0.20
54.3
77.4
$8.0
Object per Video
1.6
2.1
5.8
MGE
VIIA
SyaViss
Fig. 4. Visual comparison of our VIS with Mask2Former- VIS ('M2F) [6] and VITA [15]. SyneVIS shows impressive accuracy in long, complex scenarios where objects share similar appearances and have heavy occlusions.
M2F
SyncVsS
M2F
SynVVIS
Fig. 5. Qualitative comparisons with Fask2 VIS (abbreviated 'M2F') Youtube- VIS 2019. In this case, we want to further prove that SIS can better distinguish and capture instances with the same identities. In the first two rows, the person on the right not segmented by Mask2Former-V VIS, while in the last two rows, the cyclist from the back is not segmented by Mask2Former-VIS
15

---

M2F
SynVVIs
M2F
SynVVIs
Fig. 6. Qualitative comparisons with Mask2Former-V VIS (abbreviated 'M2F") Youtube- VIS 2019. In the first two rows, the person riding the motorcycle reappears the frame, which tests model's ability to connect instances across the temporal axis. Our model successfully connects instances across two frames and segments more precisely than Mask2Former VIS does (the motorcyclist's 's leg in the third frame), which demonstrates VIS' s temporal association ability. In the last rows, the person the skateboard is changing his poses across time. Our model successfully segments the person different poses, while Mask2Former- VIS fails segment this person's arm in first frame. This further illustrates robustness and generality of our model's temporal association ability, which credited to the synchronization.
MOP
VIIA
Fig. 7. Visual comparison of VIS with Mask Former- -IS (abbreviated 'M2F') [6] and VITA [15]. SIS shows impressive accuracy in long videos, while the previous methods have either low confidence (the confidence of the car blue masks the first row is 77% while in third row 98%) incomplete masks (the first frame in the second row).
16

---

Viden- leden-level Emboddings
Prame-level Emboediings
Video- level Emboldings & Frame-level Embeddingg
Fig. Qualitative comparisons with different designs of embeddings. Video-level embeddings are from set shared instance queries for all sampled frames. In the first row, video-level embeddings successfully capture most of the instances, but fail to mask the fish in the middle of the image. Frame-level embeddings are assigned cach sampled frame. In the second row, frame-level embeddings segment instances better than the first row, but fail maintain the trajectories fish in the bottom right. When synchronizing these two of embeddings, our model achieves better segmentation results even under such complex scenario.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: I summarize the contributions and the motivations in the abstract, and my experimental results support my claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations the end of the paper, discussing some failure cases.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose the codebase via an anonymous link to share our approach, and the hyper-parameten are shared.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We disclose the codebase via an anonymous link to share our approach, and the syper-parameters are shared.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /tee://npppp.. cc/
public/ guides CodeSubmi ss Lasoooolccy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
i ps cc/ cccpubll c/gu des / CodeeS Submi i sa onPol 1 ayy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We share these parameters and the details of the datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We get the results with three runs each.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include these information in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips cc httppbccc/tttccceiiii
Answer: [Yes]
Justification:
We conduct in conform with the ethic code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include it at the end of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No data or models have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite each used asset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' raw w. thcode com/ data setss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include them and make them well-documented.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Gptry3e1SR", "reviewer": "Reviewer_3JCg", "summary": "This paper proposes SyncVIS, an approach for Video Instance Segmentation (VIS), which tries to jointly model frame-level and video-level embeddings thus can capture both semantics and movement of instances. The new architecture design is intuitive and generic enough to be applied to various VIS models. Experiments are done on Youtube-VIS 2019, 2021, 2022, and OVIS which show that SyncVIS achieves state-of-the-art results on these benchmarks. Ablations are thorough and enough to understand the proposed approach. Written presentation is clear and easy to read.", "questions": "- Why the results of Mask2Former-VIS on OVIS dataset in Table 9 are too low?", "limitations": "The author(s) have some statements about their method's limitation and further provide more details in supplementary.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The newly proposed architecture design is intuitive and effective.\n- The new approach can be applied to most of existing VIS architectures and gives consistent improvements.\n- SyncVIS achieves state-of-the-art performance on current benchmarks.\n- Ablation experiments are solid and provide good insights about the newly proposed method.", "weaknesses": "- The paper may be benefit from having from qualitative results to highly what SyncVIS can improve from the base models which were trained on asynchronous manner in the main text if space allows.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In the beginning, we want to thank you for the detailed, insightful, and constructive comments. \n\n### Qualitative results\nIn the Appendix, we provide many illustrations comparing previous base models with our SyncVIS. We select some cases under different scenarios, which include settings with multiple similar instances, settings with reappearance of instance, settings with different poses of instance, and settings with long video (Fig. 4 - Fig. 7). In the Fig. 8 of the Appendix, we provide visualizations of implementing different designs of embeddings, which further illustrate our synchronous design is capable of better segmenting and tracking multiple similar instances, while the asynchronous design ends in ignoring some instances and segmenting objects incompletely. \n\n### Mask2Former-VIS on OVIS dataset performance\n\n- **Complexity of OVIS** \n\nOVIS is known for having more occluded and crowded scenes than the YoutubeVIS dataset (5.8 objects per video for OVIS while 1.7 for YoutubeVIS 2019), and thus poses more challenges to the model's temporal modeling capacity on multiple instances. \n- **Mask2Former-VIS** \n\nMask2Former-VIS, on the other hand, is a typical **offline** VIS method that models the whole video sequence with only video-level embeddings, which is extremely insufficient for handling occluded and crowded scenes. Even if we increase the input frame number to bring more temporal information, the modeling is still harder for Mask2Former-VIS. Even though we experimented with different hyperparameter settings for OVIS scenario, the results are still unsatisfying due to the innate restrictions of Mask2Former-VIS. The following-up work of Mask2Former-VIS, such as Seqformer, also suffers from the typical asynchronous design of offline methods, and its performance is also below 15 AP.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2JUR679BmI", "reviewer": "Reviewer_MsYp", "summary": "This paper concentrates on the video instance segmentation task. To address the problem of motion information loss when existing methods use frame queries, and the optimization issue of bipartite matching across multiple frames, the authors propose a synchronized video-frame modeling paradigm. This paradigm allows for interaction between frame queries and video queries and introduces a synchronized embedding optimization strategy in a video-level buffer, making multi-frame tracking optimization feasible. The effectiveness of the proposed method is verified on four VIS benchmarks, and detailed ablation studies are conducted.", "questions": "1. The proposed Synchronized Embedding Optimization for bipartite matching is it aimed at the bipartite matching between prediction and Ground Truth, or object query bipartite matchings during the tracking process (Like MinVIS)? This aspect is quite confusing for me. From lines L186-L193, it seems that the author aims to enhance bipartite matching between prediction and GTs. Yet, lines L194-L199 appear to discuss associating the tracking results of two frames.\n\n2. The conclusion in Sec4.4 seems counterintuitive. Longer Sub-clips could provide the model with more temporal information to help it model complex scenes and trajectories. This should be especially evident on OVIS, which includes many objects and complex inter-object motions, including occlusions and disappearances and reappearances. According to the paper's motivation, longer Sub-clips should perform better on OVIS, but the experimental conclusion states the optimal length on OVIS to be T_s = 2, which corresponds to a pair of frames, providing very limited temporal information. Doesn't this contradict the motivation or the conclusion?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed SyncVIS shows impressive performance across multiple VIS benchmarks.\n2. The method introduced can be adapted to enhance multiple existing VIS frameworks, demonstrating a high level of versatility.", "weaknesses": "1. Line 45 mentions that 'image encoding stage (rather than video encoding)' could lead to motion information loss; however, the proposed method still employs image encoding. Frame queries take video query for feature aggregation, the aggregation is merely an accumulation of multi-frame information, similar to the video query in SeqFormer, which can help achieve robustness, but how does it model object motion? This statement might be inappropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for taking the time to provide such detailed and constructive criticism. Your suggestions have been invaluable in strengthening our paper.\n\n### Object motion modeling\n\n- **Synchronous design for robust modeling**\n\nOur proposed SyncVIS employs image encoding as well as video encoding **in a synchronous manner** to model object motion while compensating for the potential \"motion information loss\" in the \"image encoding\" stage. In our design, frame-level embeddings are assigned to each sampled frame, and responsible for modeling the appearance of instances, and video-level embeddings are a set of shared instance queries for all sampled frames, which are used to characterize the general motion (because they encode the position information of instances across frames, and thereby naturally contain the motion information). \n\nSeqFormer has the observation that \"a stand-alone instance query suffices for capturing a time sequence of instances in a video.\" It decomposes the decoder to be frame-independent while building communication between different frames using instance queries, which also aim to model motion across frames. Nevertheless it follows an asynchronous manner, which is **less robust** in modeling temporal associations than our synchronous design because the aggregation is unidirectional instead of a mutual enhancement and the motion loss in image encoding is not compensated. As proved in YoutubeVIS 2019 & 2021, SyncVIS achieves 54.2 and 48.9 in the offline mode, while Seqformer is 47.4 and 40.5, which is much lower than our results.\n\n- **Experiment**\n\nAs shown in Table (also in the Table 6 of the main paper), by implementing a synchornous structure, SyncVIS outperforms the asynchronous unidirectional structure by 1.6 AP. This further proves that our design is more robust than the design in SeqFormer in modeling object motion.  \n\n\n| Method                                                         | $AP$ | $AP_{50}$ | $AP_{75}$ |\n|----------------------------------------------------------------|:--:|:-----:|:-----:|\n| Cascade Structure + Both Queries                               |49.9| 72.0  |  54.4 |\n| Synchronous Structure + Both Queries (SyncVIS)                 |51.5| 73.2  |  55.9 |\n\n### Synchronized Embedding Optimization\n\nIn the synchronized embedding optimization, we aim to enhance bipartite matching between prediction and GTs. L194-L199, the illustrations of $t_i$ and $t_j$ are trying to explain how the optimization strategy works in a divide-and-conquer way and illustrate our motivation of dividing the whole training sample videos into sub-clips. \n\n### Setting of $T_s$\n \nThe results do not contradict the conclusion. In optimization strategy, our main goal is to **reduce the increasing optimization complexity** as the input frame number grows (L. 188, L. 260). To realize this target, our strategy is to divide the video into several sub-clips that could make optimization easier while retaining the temporal motion information. Longer Sub-clips could provide the model with more temporal information, but their optimization complexity also rises polynomially. \n\nThe other important factor is the **dataset**. OVIS, compared to YoutubeVIS 2019, has more occluded and crowded scenes (5.8 objects per video for OVIS while 1.7 for YoutubeVIS 2019). Thus, the complexity of OVIS for each frame is three times that of the YoutubeVIS 2019. In order to further reduce the complexity for better optimization, dividing into smaller sub-clips can accelerate the optimization. Also, keeping the size of two is able to maintain the temporal information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4WJCR5Dh2z", "reviewer": "Reviewer_2sLS", "summary": "This paper focuses on improving synchronization between frame and video queries in video instance segmentation for better long-range video analysis. The authors propose encoding frame and video queries separately, then using confidence scores to select Nk queries. These queries are updated through mutual information exchange with momentum updates. To train each query, they introduce a synchronized optimization method using a divide-and-conquer approach. They also identify that video-level bipartite matching complexity increases with the number of frames, and address this by suggesting sub-clip level matching. Their technique, when applied to CTVIS and VITA, demonstrates enhanced performance in both online and offline settings compared to existing methods.", "questions": "- In Section 3.1, the author highlights the performance drop in Mask2Former-VIS when increasing the number of frames. CTVIS, on the other hand, shows improved performance with more frames and seems to handle temporal modeling well through its memory bank. If CTVIS were used as the baseline, why does the proposed method show improved performance and what specific aspects are enhanced?\n\n- TCOVIS aggregates the cost for each frame and matches GT with predictions at the video level globally. The proposed method matches at the sub-clip level, which is a contrasting approach. What are the respective strengths and weaknesses of each method? Additionally, why does the proposed method perform worse on long videos like YouTubeVIS-2022 compared to TCOVIS?\n\n- In L219, it is mentioned that inference with the Swin-L backbone was done at 480p, but the code indicates 448p. Which is correct? Furthermore, in L218, the learning rate is stated as 1e-4, but the code shows 5e-5. Also, the iteration and learning rate decay settings for YouTubeVIS-2019 and 2021 datasets seem inconsistent. How were the optimization settings determined?", "limitations": "The authors address the limitations of the proposed method in the appendix.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is comprehensively written, with clear explanations of its contributions and a detailed analysis of prior research. The proposed methods are innovative and seem well-founded.\n\n- Extensive experiments validate the effectiveness of the proposed methods, significantly bolstering the paper's credibility.", "weaknesses": "- The analysis primarily focuses on early work (Mask2Former-VIS), while the baselines used are CTIVS and VITA. Despite this, there is a lack of thorough analysis on these baselines.\n- The GT assignment method of Synchronized Embedding Optimization is not compared with existing methods such as TCOVIS. Additionally, although the authors aim for long-range video modeling, their performance on the long video dataset YouTube-VIS 2022 is lower than that of TCOVIS.\n- In the Checklist under Experiments Compute Resources, the authors answered \"yes\" but did not specify the equipment used.\n\n\nReference\n\nLi, Junlong, et al. \"Tcovis: Temporally consistent online video instance segmentation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough review and valuable input. Your feedback has been instrumental in helping us enhance the quality and clarity of our work.\n\n### Analysis of baseline method\n\nThe analysis of CTVIS and VITA are included in the Introduction and Related Works (L. 37, L. 104). We choose to analyze Mask2Former-VIS because it's a basic and representative VIS model that both VITA and CTVIS are built on, while VITA utilizes frame-level queries to segment each frame independently and then associate frame-level queries with video-level queries and CTVIS adopts the frame-level queries to build consistent contrastive items and memory banks. 1) **VITA**: Even though VITA adopts video-level queries and manages to associate them, we argue that this process is **sensitive** because the wellness of video-level queries **heavily relies on the learning of frame-level queries**. Since frame-level queries (decoded by frame features) lack global motion information, this asynchronous unidirectional aggregation will cause potential information loss to the output video-level queries. 2) **CTVIS**: CTVIS utilizes contrastive learning to strengthen the representations of frame-level embeddings and uses frame-level embeddings to maintain a memory bank to model temporal relations. However, it mainly focuses on building discriminative frame-level embeddings, and **hardly models the long-term spatio-temporal object representation**. That is, CTVIS **lacks an explicit synchronous association** between video- embeddings and frame-level embeddings. As proved in YoutubeVIS 2019 & 2021, SyncVIS achieves 57.9 and 51.9 in the online mode, while CTVIS is 55.1 and 50.1, which is lower than our results.\n\nWe will revise our paper in the final version.  \n\n\n### Comparison with TCOVIS\n\nIn TCOVIS, the global instance assignment aims to match predictions from all frames in the video clip with GTs. It considers all frames as a whole in searching for the optimal objective. Our method, on the other hand, divides all video frames into smaller sub-clips for matching with GTs. \n\n1) **TCOVIS:** By matching predictions across all frames, TCOVIS is capable of **associating the video frames from the beginning to the end** of a minute-long video in the Youtube-VIS 2022 validation dataset, but it may **overlook the fine-grained local details** that is crucial for distinguishing multiple similar instances. Thus, when handling multiple instances and movements in consecutive frames, TCOVIS fails to simultaneously track and segment many more instances in crowded and occluded scenes. On OVIS, TCOVIS is 46.7 while our SyncVIS is 50.8, outperforming by a large margin.\n2) **SyncVIS:** By dividing video into several sub-clips, our optimization strategy aims to **reduce the increasing optimization complexity** as the input frame number grows (L. 188, L. 260) because our synchronous modeling paradigm can model temporal association better with more input frames. To realize this target, our strategy is to divide the video into several sub-clips that could make optimization easier while retaining the temporal motion information. \n\n3) **YoutubeVIS 2022:** Our SyncVIS mainly focuses on solving challenging scenarios with multiple instances and occlusions, which is usually overlooked for previous asynchronous methods because of their query-sensitive designs. When handling minute-long videos, our video-level embeddings are insufficient for modeling the associations across such a huge amount of input frames. Therefore, the performance on Youtube-VIS 2022 is not the SOTA result. \n\n    However, our SyncVIS manages to find a balance between modeling long-range video and keeping high performance on crowded and occluded video scenes in shorter lengths. Even though TCOVIS has better results on Youtube-VIS 2022 validation dataset, its performance on another challenging VIS dataset, OVIS, is 4.1 points below our SyncVIS. TCOVIS manages to model the extra-long videos in Youtube-VIS 2022 validation dataset, but it somehow neglects the much more common cases, which are combinations of long videos and complex scenes with more instances and occlusions. \n    \nWe will revise this part to the final version of the paper.\n\n\n### Computation resources\n\nMost of our experiments are conducted on 4 A100 GPUs (80G), and on a cuda 11.1, PyTorch 3.9 environment. The training time is approximately 1.5 days when training with the Swin-L backbone. We will specify this in the final version of the paper.\n\n### Optimization setting\n\nSince the two datasets have different sizes, and that 2021 has more videos than 2019, we use different training iterations and learning rates for different datasets. in practice.  We will revise our paper in the final version to state this difference (As for the image size, the number should be 448, and the learning rate of the Swin-Large backbone on YoutubeVIS 2019 is 5e-4, and we will correct this in the final version). \n\n### Improvement upon CTVIS\n\nIt's true that building a memory bank contributes to the temporal modeling of instances. However, the forward passing of CTVIS is still asynchronous. While a single frame produces the frame embedding, there is **no explicit video-level embedding** to interact with the frame-level instance embedding. In our design, we add a set of video-level embeddings that gradually update with the frame-level embeddings. The explicit video-level embedding can directly provide long-range information and temporal information to the frame-level embedding, which enhances the quality of Hungarian matching and the contrastive learning module afterward.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBvAlEK0fJ", "reviewer": "Reviewer_hMrY", "summary": "The paper argues that existing methods for video instance segmentation use asynchronous designs, leading to difficulties in handling complex video scenarios. To address this problem, the paper proposes a SyncVIS method for synchronized modeling, achieving state-of-the-art results on several benchmarks.", "questions": "In addition to the Weaknesses,\n1. How does the method ensure that the video queries learn motion information?\n2. Ablation study on different values of $N_{k}$ (line 173) should be conducted.\n3. How is the efficiency of the method?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is generally clear to understand.\n2. The paper achieves good performance by synchronized modeling.\n3. Visualizations are provided to illustrate the effectiveness of the method.", "weaknesses": "1. The novelty of synchronized embedding optimization is limited.\n2. The paper did not compare with DVIS++. For example, using ResNet-50 and offline mode, DVIS++ outperforms SyncVIS by 2.5% on YouTube-VIS 2019. DVIS++: Improved Decoupled Framework for Universal Video Segmentation, arXiv, 2023.\n3. More implementation details like training steps should be reported.\n4. Compute resources, such as the type of GPU, are not reported in the paper, yet the authors answered YES to question 8 in the checklist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In the beginning, we want to thank you for the detailed, insightful, and constructive comments.\n\n### Novelty\n\nTo sum up, as other reviewers have mentioned, our method provides an architectural design that is **intuitive and effective** (Reviewer 3JCg) and **innovative** (Reviewer 2sLS), **demonstrating a high level of versatility** (Reviewer MsYp). Our SyncVIS explicitly introduces video-level query embeddings to synchronize video-level embeddings with frame-level query embeddings. The synchronized video-frame modeling paradigm promotes the mutual learning of frame- and video-level embeddings by selecting key information from both embeddings and updating gradually via the synchronous decoder structure. \n\n### DVIS++\n\nIn offline mode, our method is lower than DVIS++, but our **best** performance on ResNet-50 backbone is in **online** mode, which is higher than DVIS++, by adapting to CTVIS. As mentioned in our paper and credited by Reviewer MsYp, our method **demonstrates a high level of versatility** on many state-of-the-art VIS methods. DVIS++ is built on DVIS, which incorporates a denoising training strategy and contrastive learning paradigm as well as an improved DINOv2 backbone. These adjustments, however, are orthogonal to our improvements to the modeling of spatial-temporal representations as well as our optimization strategy. \n\nIn the referring tracker and the temporal refiner, with the addition of our video-level embeddings and our synchronous modeling structure, our method can still bring significant improvements (2.1 AP) by adapting to DVIS++ in offline mode. \n\n| Method                      | AP   |\n|-----------------------------|------|\n| DVIS++                      | 56.7 |\n| + Synchronized Modeling     | 58.0 |\n| + Synchronized Optimization | 57.6 |\n| + Both (SyncVIS)            | 58.8 |\n\n### Implementation details\nAs for the implementation details such as training steps, we list these parameters in the **configs** file in the link provided in the **Appendix**. For example for the Youtube-VIS 2019, as the batch size is set to 8, the max_iter is 140000, and we use AdamW optimizer with a base learning rate of 5e-4 and a weight decay of 0.05. The backbone multiplier is 0.1.\n\n### Computation resources\nMost of our experiments are conducted on 4 A100 GPUs (80G), and on a cuda 11.1, PyTorch 3.9 environment. The training time is approximately 1.5 days when training with the Swin-L backbone.\n\n### Motion information in video queries\nIn the DETR-style architecture, when video queries are **associated with features across time** via the decoder, they can effectively model instance-level motion through the cascade structure. In Mask2Former-VIS, the use of video queries alone enables the capture of instance motion. \n\nA similar finding has been reported in the Seqformer model, which notes that **\"a stand-alone instance query suffices for capturing a time sequence of instances in a video.\"** However, as the number of input frames increases, relying solely on video queries becomes insufficient for simultaneously tracking the movement of all instances. Such limitations motivate us to propose such synchronous designs to address the shortcomings in motion modeling.\n\n### Ablation of $N_k$\nThe ablation study of N_k is provided in Table 8 and the related analysis starts in L.324. The ablation results reveal that when selecting top $N_{k}=10$ embeddings to aggregate, the model performance reaches its optimum. When $N_{k}$ gets larger than optimum, the **redundant** query features will **dilute** the original compact information. On the other hand, getting too small will lead to the insufficiency of the injected information.\n\n### Efficiency\nThe relevant results are provided in L.242-243. We list the model parameters and FPS of SeqFormer (220M/27.7), VITA (229M/22.8), and our SyncVIS (245M/22.1). Our model performs **notably better** with comparable model parameters and inference speed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers are inclined towards acceptance, with scores ranging from 5 to 7. They have highlighted several areas for improvement that the authors should address carefully in the final version. These include adding clarifications and detailed explanations and all suggested additional experiments, and enhancing the presentation and organization of the paper to minimize confusion.
Decision: Accept (poster)