TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation
Authors: rongkun Zheng
Lu Qi
Xi Chen
Yi Wang
Kun Wang
Yu Qiao
Hengshuang Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: taxonomy-aware
multiple-datasets
video instance segementation

Abstract: Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increase with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomy. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all these benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our proposed approach. The code and trained models will be publicly available.

TMT-VIS: Taxonomy-aware Multi-dataset Joint
Training for Video Instance Segmentation
Rongkun Zheng'
Lu Qi
Xi Chen'
Yi Wang'
Hengshuang Zhao The University of Hong Kong University of California, Merced Shanghai Artificial Intelligence Laboratory "SemseTime Research { c2r222ccneect,, hszhaol cs) hku.hk
Kun Wang
Yu Qiao
Abstract
Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, mask precision increases with the data volume, simply utilizing multiple datasets will dilute the attention of models different taxonomies. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-awarr Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-Vii VIS 2019, You Tube- VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-ar records on all benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our approach. The code is available at
Introduction
Video Instance Segmentation (VIS) fundamental while challenging visual perception task that involves detecting, segmenting, and tracking object instances within videos based on a set of prede- fined object categories. The goal of VIS accurately separate the objects from the background and assign consistent instance IDs across frames, making it crucial task for various popular downstream applications such as autonomous driving, robot navigation, video editing, etc.
Benefiting from favorable modeling of visual tokens rather than modeling pixels with CNNs, query- based transformer methods [7, 44, 16, 42, 19, 49] like Mask2Former- VIS [7], SeqFormer [44], and VITA [16] begin to dominate the VIS task. The recent success of SAM [21] demonstrates signif- icant improvements in segmentation and detection performance through data scaling. However, mainstream VIS datasets are much smaller than widely used image detection datasets (e.g.. approxi- mately 3K videos in YouTube-VV VIS [47] vs. 120K images COCO [27]), leaving the potential of leveraging scaled video datasets for multi-frame video input modeling largely unexplored. Previous researches [23, 10] have proved that transformer structures, when compared with CNNs, have the advantage that they can better leverage the large volume of data, and this naturally raises question: can we alleviate this dilemma via training on large-scale datasets?
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Video (Dataeet
Encoder
Embodding
Decoder
Pred
(Dabaert)
Encoder
Embodding -+
Taxo.
Decoder
Pred
C&L.
Video
f
Language Embeddling
(Daraset
Teconomy Embediing
Past Works
Ours
Figure 1: Comparison of cattippledattsse training paradigms. Previous multiple dataset training methods focus on unifying the label space, and some models adopt language embeddings to interact with queries after the decoder for the final classification. In order to enhance the performance, our model (right part) leverages the taxonomic embedding refine queries through taxonomic compilation and injection (C&I) before the decoder.
Nonetheless, obtaining a single large-scale VIS dataset is cost-ineffectivy because of the growing need for manual labeling. A sounding alternative would be training multiple VIS datasets so as to increase data scale and enrich the taxonomy space. Still, simply combining all these datasets to train a joint model does lead good performance. As a result of the difference category space, though mask precision increases with the data volume, dataset biases might hinder models from generalization: simply utilizing multiple datasets will dilute the attention of models on different categories. Therefore, increasing the data scale and enriching label space while improving classification precision become huge challenge for researchers.
For this problem, we intend to investigate framework that can train a unified video instance segmentor on multiple datasets while incorporating taxonomy information boost both mask precision and classification accuracy. To this end, we propose Taxonomy-awan Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to jointly train multiple datasets on DETR-based methods, which can train multiple video instance segmentation datasets directly without the requirements manually filtering irrelevant taxonomies. Built upon classic DETR-based VIS methods like Mask2Former-VD [7], our TMT-VIS consists of two key components to tackle the problem: we first leverage a Taxonomy Compilation Module (TCM), based on pre-trained text encoder, to compile taxonomy information from the input video. Then in the Taxonomy Injection Module (TIM), the taxonomic embeddings generated the TCM are instilled to the visual queries in the transformer decoder through prior to the queries. An additional iaxonomy-awar matching loss is added to supervise the injection procedure. With this extra taxonomy information aggregation structure added before the transformer decoder, queries can better concentrate on the desired taxonomy in input videos. Fig. 1 demonstrates the design of our modules. By incorporating taxonomic guidance into the DETR-based model, TMT-VIS model is able to train and utilize multiple datasets effectively.
wwwtaattteeenaaaaeeeeeae module, and thus provide a taxonomic
To the best of our knowledge, TMT-VIS is the DETR-style framework that is able to jointly train multiple video instance segmentation datasets with such improvement. We evaluate our TMT-VIS on four popular VIS benchmarks, including YouTube- VIS 2019 and 2021 [47], OVIS [35], and UVO [39]. The extensive experimental evaluations show the effectiveness and generality of our method with significant improvement over the baselines. For example, compared with Mask2Former- VIS [7] with the ResNet-50) backbone, our TMT-VIS gets absolute AP improvements of 3.3%, 4.3%, 5.8%, and 3.5% the aforementioned challenging benchmarks, respectively. Compared with another high-performance solution VITA [16], our solution gets absolute AP improvements of 2.8%, 2.6%, 5.5%, and 3.1% respectively. Finally, the proposed TMT-VIS sets new state-of-the-art records on these popular and challenging benchmarks. Our main contributions can be summarized threefold:
We analyze the limitations of existing video instance segmentation methods and propose novel multiple-datase training algorithm named TMT-VIS, which can well utilize taxonomic guidance to train and utilize multiple video instance segmentation datasets effectively.
We develop two-stage module: Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM). The former adopts the CLIP text encoder to compile taxonomic information in videos, while the latter utilizes the taxonomic information in TCM to inject guidance into queries.
We conduct extensive experimental evaluations on four popular and challenging VIS benchmarks, including YouTube- VIS 2019, YouTube- VIS 2021, OVIS, and UVO, and the achieved significant improvements demonstrate the effectiveness and generality of the proposed approach.

---

2
Related Work
Video instance segmentation. The current VIS methods can be categorized into two: online and offline. Online VIS methods typically adopt
approach. They first predict object detection and segmentation results within several frames, and link the outputs with the same identities across frames using optimized matching algorithms, which can be either hand-crafted or learnable-based. MaskTrack R-CNN [47] is the baseline online VIS method, which is built by adding a simple tracking branch to Mask R-CNN [13]. Following works [5, 48, 28, 18] adopt the similar motivation, measuring the similarities between frame-level predictions and associating them through different matching module designs. Inspired by Video Object Segmentation [33] MMulti-Object Tracking (MOT) [9], and Multi-Object Tracking and Segmentation (MOTS) [37], recent VIS works such as [11, 26, 12] are proposed. While many MOT methods use multiple trackers [2. 34, 50], some more recent MOT works [53, 30] utilize trajectory queries to exploit inter-instance relationships. Motivated by such approaches, GenVIS [15] implements a novel target label assignment and instance prototype memory in query-based sequential learning. On the other hand, IDOL [45] introduces contrastive learning head for discriminative instance embeddings, based on Deformable-DETT [54].
In contrast to online methods, offline methods aim to predict masks and trajectories of instances in the entire video one step by feeding the entire video. STEm-Seg [1] utilizes a single-stage model that gradually updates and clusters nempool embeddings. MaskProp [3] and Propose- Reducc [26] improve mask quality and instance relations through mask propagation. VisTR [42] successfully adapts DETR [6] to the VIS task for the first time, which adopts instance queries to model the whole video. 2ask2Former- VIS [7], which is an adaptation of Mask2Former for 3D spatio-temporal features, has become the leading method due its mask-focused representation. Not long ago, VITA has emerged, as detailed in [16], which employs object tokens to model the entire video and uses video queries to decode object information from these tokens.
Multi-dataset training. It is widely acknowledged that leveraging large-scale data in visual recogni- tion tasks can notably improve performance [4, 31, 52, 51, 40, 36), but collecting annotated data is challenging to scale. In object detection, several recent studies have explored using multiple datasets to train single detector and improve feature representations. For example, BigDetection [4] merges several datasets with a hand-crafted label mapping in a unified label space. Detection Hub [31] unifies multiple datasets by adapting queries on category embeddings, considering each dataset's 's category distributions. UniDet [52] trains a unified detector with split classifiers and incorporates a cost for merging categories across datasets to automatically optimize for common taxonomy. OmDet [51] uses language-bar based framework to learn from multiple datasets, but its specific architecture limits the transferability of pre-trained models to other detectors. In video tasks such as action recog- nition, several works [32, 17, 38, 22, 41] propose to combine multiple video datasets for training. ViTyViT [24] further extends this approach by including image, video, and audio datasets using different sampling procedures. MultiTrain [23] utilizes informative representation regularizations and projection losses to learn robust and informative feature representations from multiple action recognition datasets. Despite achieving huge progress, previous language-basee works simply use language embeddings in the end to match with outputs in order to get class predictions, whereas our solution leverages the taxonomic embedding to refine queries before the decoder which helps queries to concentrate on desired taxonomy, resulting in a huge improvement performance.
The UNISEXT[255 is the first DETR- based method that jointly trains multiple VIS datasets. It simply utilizes the BERT language encoder generate language embeddings of categories from all video datasets, and fuses the information with visual embeddings through simple bi-directional cross-attention module. However, UNINEXT has no video-specific design, and it doesn't use the language embeddings to predict a set of possible set of categories, so the semantic information of VIS categories is simply aggregated without further operations.
3 Method
In this section, we will first introduce the overall architecture of the proposed framework Taxonomy- aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) in Sec. 3.1. Then, we will detail the exquisitely designed taxonomy aggregation module, including Taxonomy Compilation Module (TCM) in Sec. 3.2 and Taxonomy Injection Module in Sec. 3.3. Following this, we further
3

---

Zeconnzyy Complation
Injection
Ditzed 1 Taccaco
FFN
Sparts temgoni Adacii
Text Baceder
Ieteet-2 2 Tacotans
Au line
CreaeA
K
Predictons
Back- bone
Pixel Desidae
Frozen
Frome Feetures
Inpot Video
Figure 2: Overall framework of the proposed TMT-VIS method. Taxonomy Compilation Mod- ule (TCM) adopts the CLIP text encoder and spatio-temporal Adapter to generate video-specific modulated taxonomic embeddings. Taxonomy Injection Module (TIM) leverages the modulated embeddings to provide taxonomic guidance to visual queries in the decoder. An additional taxonomy- aware loss is added to supervise the compilation.
demonstrate the taxonomy-aware matching loss and total optimization target in Sec. 3.4. Finally, we give several implementation details of the framework in Sec. 3.5.
3.1 Overall architecture.
Built upon DETR [6] style framework Mask2Former [8], we inherit the formulation that treats video instance segmentation as set prediction problem. To enhance classification performance, we incorporate unified category information as an additional input:
E f(V, (ee)))))
(1)
where f stands for our method and fiex is pretrained text encoder. V is an input video represented image sequence, L is the label space (referred to taxonomy), and E is the embedding to predict N segmentation masks, Q referring to queries.
With the improved formulation, we introduce new framework TMT-VIS, which incorporates taxonomy information into queries to improve the classification ability of the model trained on multiple datasets. As shown in Fig. 2, the framework contains two modules: Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM). By interacting video features with adapted taxonomic embedding, we generate video-specific taxonomic prior. Then, we integrate the learned taxonomic embeddings with the learnable queries, enabling better semantic representation through a divide-and-cos andie--nndcooquer strategy. We will discuss these key modules, after providing brief overview of the Mask2Former used.
Revisiting Fask2Former. Mask2Former [7, 8] is universal Transformer-bast framework for image or video instance segmentation. Mask2Former implements a Transformer- based decoder, which recursively learns and refines N number of C-dimensional queries Q E Ig-N to generate embeddings E. Specifically, the transformer decoder a nine-layer structure, where layer contains masked cross-attention, self-attention, and
network in cascaded manner.
3.2 Taxonomy Compilation Module
For joint-training multiple datasets, the biggest challenge is label inconsistency, where taxonomy from different datasets differs. Given multiple annotated VIS datasets D = (D1, D2.... ..... ... Dr J. have the whole label space L = {L1, L2...., J. C,, we utilize the text encoder of the CLIP to generate taxonomy embedding & = (i 1, K where K is the total number of category in L. with E, Rd representing the text embedding of specific category, where is the dimension size of the embedding. When given a video with T frames as input, we have the video features F E HWW DxT extracted by the Mask2 Decoder, where and represent the height and weight of the feature map. To align the dimensions between text embeddings and video features, add Spatio-temporaa Adapter after the text encoder
In our Taxonomy Compilation Module (TCM),
Pixel

---

of the CLIP. Concretely, we use two FC layers and an activation layer in the middle, with the first layer projecting the original embedding to a lower dimension and the second one projecting to align with the dimension of video features. The adapted text embeddings El are sent into cross-attention and FFN operations. We gradually update the taxonomic embeddings layer by layer by aggregating context information from the input video features via cross-attention and FFN, in order to unearth the potential taxonomy contained in images. After this, we calculated the dot product between the different modulated taxonomic embeddings to predict the most relevant taxonomy in the given video according to the score S. Note that indicates the dot production.
The whole process of TCM can be formulated as follows:
33 Adapter(E),
(2)
8² = FFN(CrossAttn(E" ,)),
(3)
83 (Ln( F2,F)),
(4)
S Sigmoid (Linearr (astncarr(e))))))) )
(5)
Thus, we can update the taxonomy set by selecting the top Nr embeddings with higher scores in S. This suggests that there Nr corresponding categories that most likely contained in the input
video. The compiled taxonomic embeddings Ey by concatenating the Nir embeddings from E
3.3 Taxonomy Injection Module
In DETR- based methods, learnable queries in the decoder are usually zero-initialized, which is lack of taxonomy information. With the supervision of taxonomy priors, query features are more likely converge faster and predict the desired categories. Also, since utilizing multiple datasets will dilute the attention of models on different categories, the aggregation of taxonomic embeddings can help query features concentrate on the predicted subset of taxonomies. Thus, Taxonomy Injection Module (TIM), we attempt to incorporate the extracted embeddings from TCM to the queries in the decoder structure. The process of TIM can be formulated follows:
X 1 = FEN(CrosAAtt((X-II1..))
(6)
X
(7)
where X, represents the query features of the 1 layer of the transformer decoder, / [1, L-
By aggregating E9 into query features through cross-attention and self-attention modules, we could have the refined query features with richer taxonomy information, and thus filtering irrelevant background taxonomy information contained in X₁. This strategy shares the same motivation with the masked attention in Mask2Former, which tries to mask out background regions in the spatial dimension, whereas our method alternatively masks out irrelevant taxonomies by using the modulated taxonomic embeddings Er.
3.4 Taxonomy-aware Matching Loss
Typically, DETR-like models view VIS as set prediction problem and they rely on global one-to-one assignment for bipartite matching, and the Hungarian algorithm is employed to find an optimal assignment with all categories. The classification loss based on the results of the matching. Inspired by this matching, as we utilize taxonomic embeddings as guidance to refine instance queries, we introduce an extra loss Ltaro to supervise the injection of taxonomic embeddings. Specifically, we add text-aware matching cost: the matching algorithm remains the same, which is gross-entrope loss, but the predicted class are not from the final prediction, but from the prediction right after the injection. By adding this supervision, we further guarantee the guidance provided by the compiled taxonomic embeddings successfully injected. Finally, we integrate losses together as follows:
L = Lmask + Aeta + Craso-
(8)
and we set des 2.0 and A = 0.5.
3.5 Implementation Details
Our method implemented on top of detectron2 [46]. Hyper-parameters regarding the pixel decoder and transformer decoder consistent with the settings of Mask2Former-VIL VIS [7]. In the Taxonomy
5

---

Table 1:
Results comparison on the YouTube- VIS 2019 and 2021 validation sets. We group the results by online or offline methods, and then with ResNet-50 or Swin-L backbone structures. TMT- VIS is the model we built upon Mask2Former- VIS, while TIS- in 'Online' and 'Offline' the model that we add our designs based on popular online and offline method, GenVIS and VITA. Our algorithm gets the highest AP performance compared with recent approaches.
YouTube- VIS
2019
YouTube- VIS 2021
Method
Backbone
AP
APs0
AP
AR
AR10
AP
APso
AP7s
AR₃
AR10
MaskTrack R-CNN [47]
ResNet-50
38.6
56.3
43.7
35.7
42.5
36.9
54.7
40.2
30.6
40.9
CrossVIS [48]
Re
36.3
56.8
38.9
35.6
40.7
34.2
54.4
37.9
30.4
38.2
MinVIS [18]
ResNet-50
47.4
69.0
52.1
45.7
55.7
44.2
66.0
48.1
39.2
51.7
IDOL [45]
ResNct-50
49.5
74.0
52.9
47.7
58.7
43.9
68.0
49.6
38.0
50.9
Online
GenVIS [15]
ResNet-50
51.3
72.0
57.8
49.5
60.0
46.3
67.0
50.2
40.6
53.2
TMT-VIS"
50
53.9
74.8
59.1
51.4
62.7
49.4
69.1
51.8
43.6
54.8
MinVIS [18]
Swin-L
61.6
83.3
68.6
54.8
66.6
55.3
76.6
62.0
45.9
60.8
IDOL [45]
Swin-L
64.3
87.5
71.0
55.6
69.1
56.1
80.8
63.5
45.0
60.1
GenVIS [15]
Swin-I.
63.8
85.7
68.5
56.3
68.4
60.1
80.9
66.5
49.1
64.7
TMT-VIS"
Swin-L
65.4
88.2
72.1
56.4
69.3
61.9
82.0
68.3
51.2
65.9
EfficientVIS [43]
ResNet-50
37.9
59.7
43,0
40.3
46.6
34.0
57.5
37.3
33.8
42.5
IFC [19]
ResNet-50
41.2
65.1
44.6
42.3
49.6
35.2
55.9
37.7
32.6
42.9
Musk2Former-
ResNet-50
46.4
68.0
50.0
40.6
60.9
41.8
TeViT [49]
MsgShifT
46.6
71.3
$1.6
44.9
54.3
37.9
61.2
42.1
35.1
44.6
SeqFormer [44]
Re
47.4
69.8
45.5
54.8
40.5
62.4
43.7
36.1
48.1
Offine
VITA [16]
ResNet-50
49.8
72.6
54.5
49.4
61.0
45.7
67.4
49.5
40.9
TMT-VIS
ResNet-50 50
49.7
73.4
53.9
49.2
60.7
44.9
66.1
48.5
39.8
52.1
TMT-VIS
ResNet-50
52.6
74.4
57.3
50.6
61.8
48.3
69.8
50.8
42.0
55.2
SeqFormer [44]
Swin-L
59.3
82.1
66.4
51.7
64.4
51.8
74.6
58.2
42.8
58.1
Mask2Former- VIS [7]
Swin-L
60.4
84.4
52.6
76.4
57.2
VITA [16]
Swin-L
63.0
86.9
57.5
61.0
47.7
62.6
TMT-VIS
Swin-I.
63.2
85.2
68.3
56.4
68.2
56.4
80.2
61.3
47.0
61.4
TMT-VIS
Swin-I.
64.9
86.1
69.7
57.2
69.3
59.3
81.0
63.7
48.9
63.5
Compilation Module, the size of the taxonomy embedding set Nr is set to 10, which matches the maximum instance number per video. Referring Mask2Former- VIS [7], initially train it the COCO [27] dataset before fine-tuning it on VIS datasets. During inference, we resize the shorter side of each frame to 360 pixels for ResNet [14] backbones and 480 pixels for Swin [29] backhones. In the inference part, our SOTA performance method utilizes the given vocabulary of the dataset. However, our method could still perform well with unified vocabulary, which is credited to the
utilization of the CLIP encoder.
4 Experiments
In this part, first give some details about the experimented benchmarks in Sec. 4.1. Then, report the main results in Sec. 4.2, which mainly demonstrates the superiority of the proposed approach over previous solutions. Furthermore, the exquisite design of the whole framework and related proposed modules. Finally, we illustrate some qualitative visualizations in Sec. 4.5 show the effectiveness.
give detailed ablation studies in Sec. 4.3 and Sec. 4.4 validate
4.1 Datasets
We conduct extensive experimental evaluations on four popular and challenging benchmarks, includ- ing YouTube-VVi 2019 and 2021 [47], OVIS [35], and UVO [39]. Key statistics of these datasets are presented in the Appendix. YouTube-VIS 2019 [47] is the first large-scale dataset for video instance segmentation, with 2.9K videos averaging 4.61s ls in duration and 27.4 frames in validation videos.
YouTube- VIS 2021 [47] includes more challenging longer videos with more complex trajectories, resulting in an average of 39.7 frames in validation videos. OVIS [35] dataset is another challenging VIS dataset, offering 25 object categories and focusing on complex scenes with significant occlusions between objects. While only containing 607 training videos, its videos are much longer, lasting

---

Table 2: Results comparison the UVO and OVIS validation sets. We group the results by online or offlin methods, and then with ResNet-50 or Swin-L backbone structures. TMT-VIS is the model built upon Mask2Former- VIS, while TMT-VIST in 'Online' and 'Offline' are the model that we add our designs based on popular online and offline method, GenVIS and VITA. Our algorithm gets the highest AP performance compared with recent approaches.
OVIS
UVO
Method
Backbone
AP
AP50
AP75
AR1
1R
AP
AP
AP7s
MaskTrack R-CNN [47]
ResNet-50
10.8
25.3
8.5
7.9
14.9
9.3
20.9
8.2
CrossVIS [48]
ResNet-50
14.9
32.7
12.1
10.3
19.8
MinVIS [18]
ResNet-50
25.0
45.5
24.0
13.9
29.7
IDOL [45]
ResNet-50
30.2
51.3
30.0
15.0
37.5
16.8
28.1
17.3
Online
Gen VIS [15]
ResNet-50
35.8
60.8
36.2
16.3
39.6
TMT-VIS
ResNet-50
38.4
62.1
39.5
17.3
41.5
Min [18]
Swin-L
39.4
61.5
41.3
18.1
43.3
IDOL [45]
Swin-L
42.6
65.7
45.2
49.6
GenVIS [15]
Swin-L
45.2
69.1
48.4
19.1
48.6
TMT-VIS
Swin-I.
46.9
71.0
48.9
18.8
52.0
IFC [19]
ResNet-50
13.1
27.8
11.6
9.4
23.9
Mask2Former- VIS [7]
ResNet-50
16.5
36.5
14.6
10.2
23.4
18.2
29.7
18.9
SeqFormer [44]
ResNet-50
15.1
31.9
13.8
10.4
27.1
VITA [16]
5e
19.6
41.2
17.4
11.7
26.0
18.9
30.6
19.8
Offlin
TMT-VIS
ResNet-50
22.8
43.6
21.7
13.1
28.3
21.2
32.9
22.1
TMT-VIS
ResNet-50
25.1
45.9
23.8
14.2
29.9
22.0
33.4
22.9
Formerossss [7]
Swin-I
23.1
45.4
21.8
13.3
29.2
27.3
42.0
27.2
VITA [16]
Swin-L
27.7
51.9
24.9
14.9
33.0
TMT-VIS
Swin-L
28.0
50.9
27.6
14.4
34.1
29.0
42.8
29.5
TMT-VIS
Swin-L
32.5
55.1
32.0
15.4
38.3
29.9
43.6
30.1
12.77s on average. UVO [39] dataset has 81 object categories (80 shared with COCO [27]. plus an extra "other" category for non-COCO instances). UVO, subset of 1.2K Kinetics-400 [20] videos, contains 503 videos densely annotated at 30fps, and provides segmentation masks exhaustively for all object instances.
4.2 Main Results
We compare TMT-VIS with state-of-the-ar approaches which are with ResNet-50 and Swin-L backbones on the YouTube-VVi VIS 2019 and 2021 [47], OVIS [35], and UVO [39] benchmarks. The results are reported in Tables and 2.
YouTube VIS 2019. Table shows the comparison on YouTube- VIS 2019 dataset. Our TMT-VIS sets new state-of-the-art results under all of the settings. Based Mask2Former-VID TMT-VIS gets 49.7% AP and 63.2% with ResNet-50 and Swin-I. backbones, respectively, outperforming the baseline by 3.3% and 2.8% absolutely. When the proposed designs are added onto VITA, TMT-VIS gets 52.6% AP and 64.9% AP with ResNet-50 and Swin-L backbones, respectively, outperforming the baseline by 2.8% and 1.9%. When GenVIS is added with our design, TMT-VIS gets 53.9% AP and 65.4% AP with ResNet-50 and Swin-L backbones, outperforming the baseline by 2.6% and 1.6%.. We list the model parameters of Mask2Former (216M), SeqFormer (220M), VITA (247M), and our TMT-VIS (255M), which shows that TMT- VIS performs notably better with similar model parameters. Also, we compared our TMT-VIS with UNINEXT [25] on YouTube-VISS 2019 dataset. With our delicate design of TCM & TIM, our method achieves higher performance of 64.9% AP with 4 VIS datasets on Swin-I. backbone, and the total training time approximately day 12 hours. On the other hand, UNINEXT is trained on video datasets and the total training time is

---

Table 3: Ablation study on training with multiple VIS datasets with Mask2Former-Vis (which is abbreviated as 'M2F") and TMT-VIS and their validation results on various VIS datasets, Experiments 1 7 report the results of training with M2F. Experiments 8 14 report the results of training with TMT-VIS.
ID
Method
YTVISerain OViSerain
UVOrraán
OVIS vl
UVO
M2F
46.4
2.3
1.9
2
M2F
5.2
16.5
3.6
3
M2F
4.4
2.5
18.2
4
M2F
47.3
17.4
4.9
5
M2F
46.2
3.7
19.0
6
M2F
7.1
16.6
18.7
7
M2F
V
7.1
16.6
18.7
8
TMT-VIS
47.3
7.2
6.5
9
TMT-VIS
10.5
17.8
8.0
10
TMT-VIS
8.6
18.8
11
TMT-VIS
48.8
20.9
10.1
12
TMT-VIS
47.0
20.4
13
TMT-VIS
20.2
14
TMT-VIS
49.7
22.8
21.2
Table 4: Ablation study on key component designs in TMT-VIS. The 'Multiple Datasets' setting refers to training on the three VIS datasets (Youtube-VIS 2019, OVIS, and UVO). Experiment I refers the Mask2Former-V VIS model. Note that experiments are evaluated on Youtube- VIS 2019, abbreviated as YTTVIS..
YTVIS
Multiple Datasets
ID TCM&TIM Taxonomy loss
AP
50
AP7s
APso
AP75
46.4
68.0
50.0
47.2
69.1
50.7
II
47.0
49.2
72.7
53.4
47.3
68.9
50.8
49.7
73.4
53.9
approximately 3 days, but its performance is 64.3% AP. Our method is proved to more effective and data-efficient.
YouTube-VIS 2021. Table also compares the results the YouTuuee-V VIS 2021 dataset. Specifically, based on Mask2Former-V/L TMT-VIS achieves 44.95 44.9% AP and 56.4% AP with ResNet-50 and Swin- backbones, respectively. Compared to the baseline, TMT-VIS boosts performance by 4.3 and 3,8 points. When the exquisite designs are added to VITA, TMT-VIS achieves 48.3% AP and 59.3% AP respectively, boosting baseline performance by 2.6 and 1.8 points. When plugging to GeS TMT-VIS achieves 49,4% AP and 61.9% AP respectively, boosting baseline performance by 3.1 % AP and 1.8% AP.
OVIS and UVO. Table 2 illustrates the competitiveness of TMT-VIS on the challenging OVIS dataset. Particularly, TMT-VIS achieves 25.1% AP and 32.5% AP with ResNet-50 and Swin-L backbones with VITA as the base model, and harvests 38.4% AP and 46.9% AP with GenVIS as the base model on these backbones. Table 2 shows the strong performance of TMT-VIS among offline methods on the challenging UVO dataset. These appealing and encouraging results further prove the effectiveness and generality of the proposed approach on multiple dataset training.
4.3 Ablation Studies
In the following, we conduct extensive ablation studies to check the properties of each component. We first show that our algorithm can improve model performance by large margin when trained on different combinations of datasets. Then we examine the detailed ablations of our design. Note that all of the TMT-VIS ablation experiments are conducted based on Fask2Former- VIS with the ResNet-50 backbone.
Multiple dataset joint-training problem. Even though mask precision increases with the data volume, due to the heterogeneity in category space, simply utilizing multiple datasets will dilute

---

Table 5: Ablation study sam- pling ratio of multiple datasets (The order of the datasets is YTVIS:UVO:OVIS).
Table 6: Ablation study on the size of taxonomic embedding set in TCM.
Table 7: Ablation study on aggre- gation strategy in TIM.
Size
AP
Size
AP
Method
AP
AP50
AP
Sampling Ratio|
AP
APso
AP7s
1
47.1
15
49.2
Add
47.6
71.8
50.3
2:1:0.75
48.6
71.0
52.4
2
47.5
20
48.5
Concatenation
47.4
71.0
51.1
1:1:0.5
49.4
72.8
53.2
5
49.3
50
47.4
Cross-attention
49.7
73.4
53,9
1:1:0.75
49.7
73.4
53.9
1:1:1
49.5
72.6
53.7
10
49.7
100
47.2
the attention of models on different categories. In Table 3, we carefully examine the performance of multiple datasets joint training on various datasets. We find that the popular Mask2Former- VIS framework meets difficulties in dealing with datasets with different taxonomy spaces. As the number of datasets increases, the performance may even decrease. In contrast, as we increase the number of datasets, the performance of our TMT-VIS improves gradually.
Key component designs. Table 4 demonstrates the effect of our component designs when combined with the prevalent Mask2Former- VIS. By adopting our algorithm, Mask2Former- VIS achieves a huge gain of 2.5 points in AP performance when both models are trained with three VIS datasets. Our extra loss also demonstrates a gain of 0.5 AP.
Datasets sampling ratio. The sampling ratio of different VIS datasets is crucial parameter. Diverse sampling ratios can cause model to focus on different categories, which can impact the attention and performance of the model. In this study, arrange the sampling ratio according the optimal number of iterations each dataset has trained individually. As shown in Table 5, 11:1:0.75" ratio yields the best results, which aligns with our previous training approaches a single dataset. The performance gap between different ratios validates that selecting an appropriate sampling ratio is important for ensuring that the model learns the relevant category information effectively and achieves optimal performance during multi-dataset training.
4.4 TCM and TIM
In Tables 6 and 7, we investigate the effect of different sizes of taxonomic embedding set as well as the aggregation strategy used in TCM and TIM.
Size of taxonomic embedding set. Table 6 displays the performance of TMT-VIS with varying numbers of output taxonomic embedding Ny in the TCM from 1 to 100. The results indicate that selected input embeddings to TIM.
the model performs best when Nr 10 embeddings When Nr is set to or 2, which is apparently less than the overall categories in input video, topk taxonomic embeddings miss the groundtruth classes. As Nr decreases, the aggregated taxonomic guidance contained in embeddings becomes insufficient, and the selected embeddings may not encode taxonomic semantics for all instances in the video, resulting in drop in performance. Conversely, when N1 gets larger than the optimum value, the redundant taxonomic embeddings dilute the category information, leading performance degradation. When using all embeddings, the improvement is trivial since TCM can not provide filtration to irrelevant classes, and the TIM module is simply injecting the information of the whole category space to queries.
A Aggregation strategy. Table 7 demonstrates the results of different aggregation strategies imple- mented in the TIM. There are multiple ways of aggregating taxonomic embeddings to queries in the transformer decoder: in the 'add' setting, taxonomic embeddings are expanded to match the dimensions of queries and addition is applied to the inflated embeddings; in the 'concatenation' setting, taxonomic embeddings are concatenated with queries provide additional semantic queries; in the "cross-attention setting, taxonomic embeddings are fed to the cross- attsnnttnnnion module as key and values to inject category semantics into queries. The results indicate that aggregating taxonomy guidance can improve multi-dataset training performance, and that cross-attention is found to be the most effective strategy for injecting taxonomic embeddings into instance queries. On the other hand, directly concatenating adding taxonomic embeddings does not improve significantly. This suggests that straightforward modifications to instance queries don't infuse taxonomy information into queries, and such superficial modifications will be refined after passing the cascades of the decoder.

---

M2R-VIS
TMI-VIS
M2FVVS
TMIVIS
Figure 3: Visual comparison of our model with Farme VIS (abbreviated as 'M2F.VIS'. Our TMT-VIS shows better precision in segmenting small instances, e.g., the swimmers in the middle of the image, and classifying similar instances, e.g., truck and sedan are similar categories, their appearances are similar but different in sizes.
4.5 Visualization
We also conduct qualitative evaluations and the visual comparison with Mask2Former- VIS. Results are illustrated in Fig. Our TMT- VIS demonstrates better capacity segmenting small instances and classifying instances with similar categories. The reason is that our TMT-VIS model is benefited from our taxonomy compilation module with multi-datased training.
5 Conclusion
In this paper, we propose TMT-VIS to address the dilemma in multiple dataset joint training in VIS. By developing two-stage module, Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM), our new algorithm able to train and utilize multiple datasets effectively by incorporating taxonomy guidance into the DETR-based model. Our proposed TMT-VIS harvests great performance improvements over the baselines and sets new state-of-the-ar records on multiple popular and challenging VIS datasets and benchmarks. We hope that valuable insights and motivate future VIS research.
approach provide
Limitations. The utilization and aggregation of taxonomic embeddings may need further investi- gations, and there might be more sophisticated ways of associating taxonomic embeddings with multi-level queries that may further improve the performance.
Broader impacts. TIS-VIS is designed for effectively training multiple VIS datasets which achieves promising performance. We hope that TMT-VI MIS-VIS can have positive impact on many industrial areas where dataset biases are severe. We would like note that VIS research should mind not violate personal privacy.
Acknowledgement. This work is partially supported by the National Natural Science Foundation of China (No. 62201484), National Key R&D Program of China (No. 2022ZD0160100), HKU Startup Fund, and HKU Seed Fund for Basic Research.
10

---

References
Ali Athar, Sabarinath Mahadevan, Aljosa Osep. Laura Leal-Taixé, and Bastian Leibe. Stem-seg: Spatio- temporal embeddings for instance segmentation in videos. ECCV, 2020.
[2] Philipp Bergmann, Tim Meinhardt, Laura Leal-Taixe. Tracking without bells and whistles. ICCV, 2019. 3
[3] Gedas Bertasius and Lorenzo Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR, 2020. 3
[4] Likun Cai, Zhi Zhang. Yi Zhu, Li Zhang, Mu Li, and Xiangyang Xue. Bigdetection: A large-scale benchmark for improved object detector pre-training. In CVPR, 2022. 3
[5] Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan. Yanwei Pang, and Ling Shao. Sipmask: Spatial information preservation fast image and video instance segmentation. In ECCV, 2020.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnacve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, 2020. 3,4
[7] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv:2 10764 2021. 3,
[8] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer universal image segmentation. CVPR, 2022. 14
[9] Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Tan Reid, Stefan Roth, and Laura Leal-Taixé. Motchallenge: A henchmark for single-camera multiple target tracking. IJCV, 2021.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 1
[11] Yang Fu, Linjie Yang, Ding Liu, Thomas S Huang, and Humphrey Shi. Compfeat: Comprehensive feature aggregation for video instance segmentation. In AAAI, 2021. 3
[12] Ho Han, Sukjun Hwang, Seoung Wug Oh, Yeonchool Park, Hyunwoo Kim, Min-Jung Kim, and Seon Joo Kim. Visolo: Grid-based space-time aggregation for efficient online video instance segmentation. In CVPR, 2022. 3
[13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 3
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6
[15] Miran Heo. Sukjun Hwang. Jeongseok Hyun, Hanjung Kim, Scoung Wug Oh. Joon- Young Lee, and Seon Joo Kim. A generalized framework for video instance segmentation. In CVPR, 2023. 6, 7
[16] Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon- Joon-Young Lee, and Seon Joo Kim. Vita: Video instance segmentation via object token association. In NeurlPS, 2022. 1, 2, 3, 6, 7, 14
[17] Ryota Hinami, Junwei Liang. Shin'ichi Satoh, and Alexander Hauptmann, Multimodal co-training for selecting good examples from webly labeled video. arXiv:1804400005, 2018. 3
[18] De-An Huang, Zhiding Yu. and Anima Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. In NeurIPS, 2022. 3, 6,
[19] Sukjun Hwang. Miran Heo, Scoung Wug Oh, and Seon Joo Kim. Video instance segmentation using inter- frame communication transformers. In NeurlPS, 2021. 1, 6,
[20] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjayanarasimhan. Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action vidco datasct. arXiv:1705.000 2017. 7, 14
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yer Yen Lo, al. Segment anything. In ICCV, 2023.
11

---

[22] KunChang Li. Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv:2305.06355, 2023. 3
[23] Junwei Liang, Enwei Zhang, Jun Zhang, and Chunhua Shen. Multi-dataset training of transformers for robust action recognition. In NeurIPS, 2022. 1.3
[24] Valerii Likhosherstov, Anurag Amab, Krzysztof Choromanski, Mario Lucie, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. TMLR, 2022.
[25] Fangjian Lin, Jianlong Yuan, Sitong Wu, Fan Wang, and Zhibin Wang. Uninext: Exploring a unified architecture for vision recognition. In ACMMM, 2023. 3,7
[26] Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia. Video instance segmentation with a propose-reduce paradigm. In ICCV, 2021. 3
[27] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 1, 6.7
[28] Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR, 2021. 3
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2CCV,
[30] Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. In CVPR, 2022. 3
[31] Lingchen Meng. Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang. Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In CVPR, 2023.
[32] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In CVPR, 2020. 3
[33] Scoung Wug Oh, Joon- Young Lee. Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, 2019.
[34] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In CVPR, 2021.
[35] Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Vuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benchmark. IJC,, 2022. 2, 6, 7, 14. 15
[36] Lu Qi, Jason Kuen, Weidong Guo, Jiuxiang Gu, Zhe Lin, Bo Du, Yu Xu, and Ming-IIsuan Yang. Aims: All- All-inclusive multi-level segmentation. In NeurlPS, 2023. 3
[37] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon I. Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In CVPR, 2019.
[38] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong. Yinan He, Yi Wang. Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking supplementary material. In CVPR, 2023. 3
[39] Weiyao Wang, Matt Feiszli, Heng Wang. and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In ICCV, 2021. 2, 6, 7, 14, 15
[40] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, et al. Intervvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv:2307.06042 2023. 3
[41] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing. Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv:2212.0319/ 2022. 3
[42] Yuqing Wang. Zhaoliang Xu. Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, 2021. 1,3
12

---

[43] Jialian Wu, Sudhir Yarram, Hui Liang, Tian Lan, Junsong Yuan, Jayan Eledath, and Gerard Medioni. Efficient video instance segmentation via tracklet query and proposal. In CVPR, 2022. 6
[44] Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In ECCV, 2022. 7.
[45] Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. In ECCV, 2022. 3, 6, 7, 14
[46] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick, Detectron2. httpa: //github.... com/ wwweel/esrseeaeeeeeeeeeetttooooe
2019. 5
[47] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. 2, 6, 14, 15
[48] Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online vidco instance segmentation. In ICCV, 2021. 3, 6,
[49] Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR, 2022. 1,6
[50] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairesss of detection and re-identification multiple object tracking. IJCV, 2021.
[51] Tiancheng Zhao, Peng Liu, Xiaopeng Lu, and Kyusong Lee. Omdet: Language-aware object detection with large-scale vision-language multi-datasct pre-training. arXiv:2209.0555000 2022. 3
[52] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In CVPR, 2022. 3
[53] Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähhenüühl. Global tracking transformers. In CVPR, 2022. 3
[54] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-cu to- and object detection. In ICLR, 2020. 3
13

---

Appendix
This appendix provides more details about the proposed TMT-VIS, further details of VIS datasets, more qualitative visual comparisons.
A
Dataset Details
Here, we provide a detailed overview of various VIS datasets in Table 8. Our extensive experimental evaluations conducted on four challenging benchmarks, namely YouTube-VIS. 2019 and 2021 [47], OVIS [35], and UVO [39]. YouTube- VIS 2019 [47] was the first large-scale dataset designed for video instance segmentation, comprising 2.9K videos averaging 4.61s in duration and 27.4 frames in validation videos. YouTube- VIS 2021 [47] poses a greater challenge with longer and more complex trajectory videos, averaging 39.7 frames in validation videos. The OVIS [35] dataset another challenging VIS dataset with 25 object categories, focusing on complex scenes with significant object occlusions. Despite containing only 607 training videos, OVIS's videos last an average of 12.77s. Lastly, the UVO [39] dataset consists of 1.2K Kinetics-400 [20] videos, densely annotated at 30fps. featuring 81 object categories, including an extra "other" category for non- non-COCO instances. provides exhaustive segmentation masks for all object instances in the 503 videos.
Among all categories in Youtube- VIS 2019, OVIS, and UVO, there overlapping categories between each dataset, and there are also different categories that share similar semantics. The detailed overlapping categories are marked in Table 9. Overall, Youtube-VIL 2021 and OVIS share more similar taxonomy space with Youtube- VIS 2019 than UVO, with a common category set of 34 out of 40 for Youtube-VIS 2021 and 22 of 25 for OVIS. Typically, when the taxonomy spaces of datasets are similar, training them jointly will have smaller dataset biases, which leads to a better result in performance. The characteristics of these datasets align with the improvement in performance when validating the joint-training models on various datasets: the increase is more significant on Youtube- VIS 2021 and OVIS than on UVO. Further details of some specific categories can be found in Table 11.
Table 8: Key statistics of popular VIS datasets. Note that UVO, the majority of the videos are for Video Object Segmentation, and only 503 videos annotated for the VIS task. 'YTVIS' is the acronym of 'Youtube- VIS".
YTVIS19
YTVIS21
OVIS
UVO
Videos
2883
3859
901
11228
Categories
40
25
81
Instances
4883
8171
5223
104898
Masks
131K
232K
296K
593K
Masks per Frame
1.7
2.0
4.7
12.3
Object per Video
1.6
2.1
5.8
9,3
B Additional Ablation Studies
In this section, we provide more experiments on our proposed methods, we will discuss the gener- ality and zero-shot properties of our training approach. and we will provide further details of the performance change in different taxonomies.
Generality property. The proposed new taxonomy-awar training strategy is an effective and general strategy that can be adapted into various base- approaches (both online & offline) and into various datasets. As in Table 10, when adding our TCM&TIM strategy to the popular VIS architecture Mask2Former- VIS [8], VITA [16], and IDOL [45], we harvest performance gains on all of the three challenging benchmarks. In OVIS, the increase can be up to 6.3% AP for Mask2Former-
As for popular IDOL, our strategy can also bring about an increase of 2.8% AP in performance. This demonstrates that the proposed taxonomy-aware module can be treated as alug-and-play design that can be used various DETR-based VIS methods (both online & offline) across different scenarios and all popular VIS benchmarks.
14

---

Per-category performance. In Table 11, we present the comparison in performance between Mask2Former- and our TMT-VIS on several specific taxonomies. When datasets other than YTVIS have no such taxonomy, training Mask2Former-Ve VIS on multiple datasets will end up decreas- ing the performance of such category, as shown 'Duck' case from Table 11. When applying our approach, we can obtain a performance improvement due to that taxonomy information of 'Duck' is compiled and injected to the instance queries. In other taxonomies, such as 'Person" which appears across all VIS datasets, the improvements are also significant.
Zero-shot property. Further, TMT-VIS also perform well zero-shot learning. We conducted the experiments on Youtube- VIS 2019 [47], OVIS [35]. and UVO [39] benchmarks. As exhibited in Table 12, TMT-VIS can be utilized transfer the knowledge of other VIS datasets another dataset with a significant increase of 4.5% AP and 3.8% AP respectively. The extra taxonomy information provided by our newly designed TCM & TIM improve the model's performance when dealing with unfamiliar taxonomies.
C Visualization
In the visualization comparisons between Mask2Former- -osoommee--iS and our model, we select some cases under different scenarios, which include setting with multiple similar instances, setting with fast- moving objects, and setting with different poses of instance.
In Fig. 4, we demonstrate videos with multiple similar instances, and TMT-VVS can both segment and track them more accurately than Mask2Former- VIS. The swimmers or the cyclists are all instances that belong 'person' category, and TMT-VIS shows better segmentation and tracking. Fig. we present example vidcos which have quick movements in camera's 's perspective and have instances with different poses. In the top two rows, the sedan and the truck have similar appearances, and our model can distinguish and segment them with higher confidence (90% over 70%). In the last two rows, our model successfully segments the person in different poses, while Mask2Former- VIS fails to segment this person's s arm in the first frame. However, the first two rows of Fig. 5 also show that our model still have the problem segmenting instances with heavy occlusions. This suggests that simply combining taxonomy information is insufficient of solving severely occluded scenes, and that more information should be aggregated to instance queries to make the model more robust in segmenting instances.
15

---

Table 9: Overlapping categories of multiple VIS datasets with Youtube- 2019 dataset." 'TTVIS is the acronym of 'Youtube- VIS'. As demonstrated in the table, YTVIS2021 and OVIS have a more similar taxonomy space, with 34 and 22 overlapping categories respectively.
YTVIS19 1111:9_44_ctterrress YTVIS21 _0_1400caaa _0_cceggori
OVIS Baris_2__categgre
UVO_81 110_81ccateeoriee
Overlapping Categories
34
22
19
Person
Giant_panda
Lizard
Parrot
Skateboard
Sedan
Ape
Dog
Snake
Monkey
Hand
Rabbit
Duck
Cat
Cow
Fish
Train
Horse
Turtle
Bear
Motorbike
Giraffe
Leopard
Fox
Deer
Owl
Surfboard
Airplanc
Truck
Zebra
Tiger
Ellephant
Snowboard
Boat
Shark
Mouse
Frog
Eagle
Earless
<
Tennis racket
16

---

Table 10: Ablation study on the generality property of TCM/TIM design with ResNet-50 backbone on multiple datasets.
Datascts
Method
AP
APso
AP7s
Mask2Former-Vin
46.4
68.0
50.0
+ TCM/TIM
49.7 (f 3.3)
73.4 (1 5.4)
53.9 (f 3.9)
VITA
49.8
72.6
54.5
YouTube- VIS 2019
+ TCM/TIM
52.6 ( 2.8)
74.4 (1 1.8)
57.6 (f 3.1)
IDOL
49.5
74.0
52.9
TCM/TIM
51.4 (1 1.9)
74.9 (1 0.9)
55.0 (f 2.1)
Mask2Former-VIS
40.6
60.9
41.8
+ TCM/TIM
44.9 (f 4.3)
66.1
(1 5.2)
48.5 (f 6.7)
VITA
45.7
67.4
49.5
YouTube-ViL VIS 2021
TCM/TIM
48.3 2.6)
69.8 (1 2.4)
50.8 (1 1.3)
IDOL
43,9
68.0
49.6
TCM/TIM
45.8 (f 1.9)
(1 1.2)
50.9 (f 1.3)
Mask2Former- VIS
16.5
36.5
14.6
+ TCM/TIM
22.8 (f 6.3)
43.6 (1 7.1)
21.7 (f 7.1)
VITA
19.6
41.2
17.4
OVIS
TCM/TIM
25.1 (1 5.5)
45.9 (1 4.7)
23.8 (f 6.4)
IDOL
30.2
51.3
30.0
TCM/TIM
33.0 (1 2.8)
55.7 (f 4.4)
33.2 (f 3.2)
Table 11: Comparisons between per-category performance of Mask2Former-VIS and TMT-VIS. "MDT" refers to 'Multiple Datasets Training', indicating whether the approach is trained on YTVIS, OVIS, and UVO. 'In Corresponding Dataset' is used demonstrate whether the category is contained in corresponding dataset.
In Corresponding Dataset
Categories
Methods
MDT
Test Set AP
YTVIS OVIS
UVO
Mask2Former- VIS TMT-VIS
YTVIS 57.2
YTVIS 57.9 (1 0.7)
Person
Mask2Former- VIS TMT-VIS
YTVIS 59.3
YTVIS 60.7 (1 1.4)
Mask2Former- VIS
YTVIS 41.6
TMT-VIS
YTVIS 42.4 (t 0.8)
Duck
Mask2Former-Ve VIS TMT-VIS
YTVIS 38.3
YTVIS 43.9 (1 5.6)
Mask2Former- -IS
YTVIS 24.7
TMT-VIS
YTVIS 26.4 (1 1.7)
Monkey
Mask2Former-Vis VIS TMT-VIS
YTVIS 25.6
YTVIS 29.1 3.5)
Mask2Former- VIS TMT-VIS Mask2Frrmer- VIS
YTVIS 8.9
YTVIS 11.8 (1 2.9)
Snowboard
YTVIS 10.0
TMT-VIS
/ YTVIS 14.5 (1 4.5)
17

---

Table 12: Zero-shot Performance of TMT- VIS with ResNet-50 backbone. The results demonstrate the zero ability of our proposed method. YouTube-VEL TuIb 2019 is abbreviated 'YTVIS'.
Train Set
Method
Test Set AP
AP5o
AP75
YTVIS OVIS UVO
Mask2Former- VIS
YTVIS 7.1
11.4
TMT-VIS
YTVIS 11.6 (1 4.5)
17.2 (1 5.8)
15.0 (1 6.7)
Mask2Former- VIS
OVIS 3.7
9.8
5.2
TMT- VIS
OVIS 7.5 3.8)
14.1 (1 4.3)
(1 3.3)
M2F-VIS
TMPVVS
M2FVIS
TMT-VIS
Figure 4: Visual comparison of our model with Mask2Former- VIS (abbreviated as 'M2F-VIS'). Our TMT- VIS shows better precision in segmenting and tracking small instances with the same taxonomy. such the giraffes from the first two rows the eyclists in the last two rows, and TMT-VE shows better performance).
18

---

M2RVIS
TMI-VIS
M2FVIS
TMI-VIS
Figure 5: Visual comparison of our model with Mask2Former-V VIS (abbreviated as 'M2F-VIS'). Our TMT-VIS shows better precision segmenting and tracking instances with occlusions. In the top two rows, TMT-VIS could successfully segment the shark hidden behind the bubbles as well as the diver in the middle, while M2F-VIS fails segment these instances. In the last rows, our model successfully segments the person in different poses, while M2F-VIS fails to segment this person's arm the first frame.
19


Reviews:
[{"review_id": "pwUKKNjv12", "reviewer": "Reviewer_5w4z", "summary": "In this paper, the authors propose a multi-dataset joint training method for the video instance segmentation task. They build on the MaskFormer-VIS model, introducing two additional modules: the Taxonomy Compilation Module and the Taxonomy Injection Module. And the proposed method shows compelling performance.", "questions": "Major concerns are raised in the weakness section.", "limitations": "This paper has discussed its limitations and societal impact in the main paper.", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "S1. This paper tackles an overlooked problem: multi-dataset joint training for the video instance segmentation task.", "weaknesses": "W1. A significant concern I have is with the explicit relevance and motivation of the proposed methodologies for the task of \"video\" instance segmentation. The paper currently lacks clear justification as to why these particular methods are designed and show impressive performance specifically for this task. It would benefit from a more in-depth exploration and justification of why these methodologies are uniquely suited for \"video\" instance segmentation.\n\nW2. The paper needs an overall improvement in writing quality. The related works section contains several inaccuracies that need rectification. To specify:\n  - L83: The claim that Mask R-CNN [13] and MinVIS [18] use a tracking branch is incorrect.\n  - L89: It is incorrect to label [47, 28] as MOTS works; they are, in fact, MOT methods.\n  - L90: The assertion that IDOL [39] is based on GenVIS [15] is inaccurate.\n\nAdditionally, there's a misleading terminology used for the proposed module on L296; it should be the Taxonomy Compilation Module (TCM), not the Taxonomy Extraction Module (TEM).\n\nW3. The paper seems to possess marginal novelty. In particular, as stated in L116-117, the proposed method leverages taxonomic embedding (as I understand, these are embeddings from the VLM), rather than language embeddings. The distinction of this work compared to previous multi-dataset joint training methods isn't clear. It would be advantageous if the authors could provide a more explicit elaboration on the unique novelty of their work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "At the beginning, we want to thank you for the detailed, insightful and constructive comments. \n\n### Uniqueness for \"video\" instance segmentation.\n\nWhen injecting the taxonomic embeddings, we add a spatio-temporal adapter to generate video-specific modulated taxonomic embeddings. This approach is not only parameter-efficient but also adapts the output text embeddings from text encoder to fit the scenarios of multiple frames. Thus, the modulated taxonomic embeddings are able to interact with the frame features across temporal dimensions through cross-attention and FFN operations. Since the difficulty of VIS tasks lies in segmenting while modeling the trajectories, which grows in polynomials along with the number of video frames, simply using queries from transformer decoder to gradually refine can be troublesome and ineffective. On the other hand, injecting modulated taxonomic embeddings as guidance can provide a prior for transformer queries, and because the modulated embeddings are filtered to possess the most possible categories’ semantic information, such injection can help queries to converge to the desired instances faster and model instances' trajectories more precisely.  \n\nWhen we downsample the YTVIS video clips to only 1 frame per video, the video instance segmentation has shrunk back to image instance segmentation.  we could notice that our design provides less improvement in such an image-level setting as with higher fps. The results are reasonable since the spatio-temporal adapter contracts to provide only spatial adjustment. With the backbone trained on COCO images, the strong segmentation backbone provides a great segmentation capacity in image-level, making the final results higher than training on the original YTVIS-19. This further illustrates that our design is specifically designed for \"video\" instance segmentation. \n\nTable 1-1. Experiments on different frames per video.\n| Method      | Testing Dataset | Frames Per Video | AP   |\n| ----------- | :-------------: | :--------------: | ---- |\n| Mask2Former | YTVIS-19         | 1                | 49.2 |\n| TMT-VIS     | YTVIS-19         | 1                | 53.5 |\n| Mask2Former | YTVIS-19         | ~30              | 46.4 |\n| TMT-VIS     | YTVIS-19         | ~30              | 52.6 |\n\n## Overall writing\n\nThanks for your careful suggestions about the misplacement of citations in related works as well as the terminologies of the `Taxonomy compilation module'. We will revise the whole paper thoroughly and with more carefulness.\n\nTo clarify some of the mistakes in our proposed version: MaskTrack R-CNN is the baseline method of online VIS method, which is built by embedding a simple tracking branch to Mask R-CNN. MinVIS implements a strong query-based image instance segmentation model (Mask2Former) on individual frames and associate query embeddings by bipartite matching. Based on Deformable-DETR, IDOL introduces a contrastive learning head that acquires discriminative instance embeddings for association. And as for the MOTS method mentioned in related works, the baseline method is TrackR-CNN. It extends the Mask R-CNN with three-dimensional convolution to combine contextual information and deploys association head to extract instance embedding for data association. Another important SOTA method is PointTrack, which performed the tracking-by-instance segmentation\nparadigm. It first obtains high-quality instance segmentation results with spatial embedding, and then extracts instance features from the\nsegmentation results. As for the misuse of terminologies, we want to again deeply apologize for the carelessness.\n\n### Novelty of TMT-VIS.\n\nA major challenge in multiple-dataset joint training is the heterogeneity of multiple datasets’ category space. As a result of the difference in category space, though mask precision increases with the data volume, dataset biases might hinder models from generalization: simply utilizing multiple datasets will dilute the attention of models on different categories. Therefore, increasing the data scale and enriching label space while improving classification precision become a huge challenge for researchers. This phenomenon was shown in our ablation study.  \n\nThe UNINEXT[1] is the first DETR-based method which jointly trains multiple VIS datasets. It simply utilizes BERT language encoder to generate language embeddings of categories from all video datasets, and fuse the information with visual embeddings through a simple bi-directional cross-attention module. However, UNINEXT has no video-specific design, and it doesn't use the language embeddings to predict a set of possible set of categories, so the semantic information of VIS categories are simply aggregated without further operations.  Our method, on the other hand, adjusts the language embeddings via a taxonomy compilation module which consists of a spatio-temporal adapter and a FFN network, as well as of a taxonomy injection module which fuses the taxonomy information into video queries with a multi-level cross-attention and self-attention layers. In the taxonomy compilation module, taxonomy embeddings are designed to interact with video features in order to unearth the potential taxonomy contained in each of the video frames. After this, we calculated the dot product between the different modulated taxonomic embeddings to predict the most relevant taxonomy in the given video. By compiling and injecting the possible taxonomic information to queries as guidance, queries  tend to converge to the desired instances faster and finally come up with better precision.\n\n\n### Reference\n[1] Yan B, Jiang Y, Wu J, et al. Universal instance perception as object discovery and retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15325-15336."}, {"review_id": "nlDgbnNzNQ", "reviewer": "Reviewer_wd7m", "summary": "This article addresses video instance segmentation from a perspective of multi-dataset joint training. The proposed method is based on DETR, while the main contribution is to inject label taxonomy into model training based on CLIP. The model is evaluated on several standard benchmarks. ", "questions": "Beyond the questions in [weaknesses], I have one last question: is the model applicable to open-world scenarios? or at least can it generalize to other datasets that not involve in model training?\n", "limitations": "Authors tried to discuss limitations; however, the discussion provided is rather elementary and does not offer crucial insights into the method. More in-depth analysis should be provided. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While multi-dataset training is not a new idea in such fields like object detection, the work is a pioneering study (as far as I know) to explore the idea in video instance segmentation. The paper is overall well written and easy to follow in most parts. The experiments are extensive. ", "weaknesses": "I would not claim these as weaknesses, but suggestions that might help improve the work. \n\nFirst, while it is good to see that the method shows improvements, I would suggest not characterizing the results as \"a such huge improvement\". On one hand, a 2%-3% improvement is not huge. On the other hand, the improvements just meet the expectation since more data are used for training. In addition to show quantitative results, it is essential to report some other metrics, such as training time, to render a more comprehensive comparison. \n\nSecond, as discussed in Sec. 2, there are other strategies for handling heterogeneous labels in multi-dataset joint training, and I am curious whether existing strategies can be directly used for video instance segmentation. If yes, how about the performance?\n\nThird, in terms of formulation,  E_\\infty is used in Sec 3.2, but does not appear in Eqs. 2-5. It is important to ensure consistency. Eq. 6 is not properly formulated as well; a same symbol is used for both input and output, i.e., $X_{l-1}$. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "At the beginning, we want to thank you for the detailed, insightful and constructive comments.\n\n### More metrics\n\nThanks for your careful suggestions. Firstly, it’s true that performance will improve as data volume increases, but our method successfully alleviates the heterogeneity in category space of multiple VIS datasets, which leads to a better improvement than simply training data together. As for the characterization of \"such a huge improvement\", we will carefully revise them in the final version. \n\nWe also tested the training time and model parameters with some current methods. We list the model parameters and FPS of SeqFormer[1] (220M/27.7), VITA[2] (247M/22.8), and our TMT-VIS (250M/21.4). Also, we compared our results with the previous multi-datasets joint training method UNINEXT[3], and the details are shown in Table 2-1. \n\nTable 2-1. Comparison with UNINEXT on YTVIS-19.\n\n| Method        | Datasets Number | Training time  | Backbone   | AP   |\n| ------------- | :-------------: | -------------- | :--------: | ---- |\n| UNINEXT       | 8               | 3 days         | ConvNext-L | 64.3 |\n| TMT-VIS(Ours) | 4               | 1 day 12 hours | Swin-L     | 64.9 |\n\n### Transfer multi-datasets joint training methods to VIS.\n\nWe transfer existing image-level strategies on handling heterogeneous labels in multi-dataset joint training to VIS tasks based on Mask2Former-VIS, and results are shown in the table. We integrate the popular multiple-dataset object detection method, UniDet[4], and compare it to our design. UniDet firstly trains a single partitioned detector on multiple datasets with shared backbone, dataset-specific outputs and loss and then unifies the outputs of the partitioned detector in a common taxonomy completely automatically. Another popular multiple-dataset object detection method is OmDet[5], but it imposes restrictions on its own specific architecture. As the table shows, UniDet has a lower performance than our design (3.7 AP lower), given that it trains in two steps.\n\nTable 2-2. Experiments on popular multi-datasets joint training methods.\n| Method        | Backbone  | AP   |\n| ------------- | --------- | ---- |\n| UniDet        | ResNet-50 | 48.9 |\n| TMT-VIS(Ours) | ResNet-50 | 52.6 |\n\n\n### Formulation.\nThanks again for your careful suggestions. $E_{\\infty}$ is actually corresponding to the $E_{1}$ in the Eqs.2-5, but we misuse the \\mathcal grammar to $E_{1}$ so it turns out to be $E_{\\infty}$. As for the X_{l-1}, in the final version we will add a ‘\\prime’ to distinguish between the input and output of the different layers of TCM.  We will revise the whole paper thoroughly and with more carefulness to ensure both consistency and properness of formulations.\n\n### Open-world scenarios or other video datasets\n\nWe update and reorganize Tab.3 in the original submitted paper, and the results are shown in the rebuttal pdf. The table contains the zero-shot performance of TMT-VIS. The results show the potentials of our model to open-world settings.\n\nWe also tested our method on VIPSeg[6] based on the popular Video K-Net[7], the results are shown in Table 2-3. VIPSeg is a large-scale dataset for video panoptic segmentation (VPS)[8], a task which aims to simultaneously predict object classes, bounding boxes, masks, instance id associations, and semantic segmentation in video frames. There are a total of 3,536 videos with 84,750 pixel-wise annotated frames in VIPSeg, which covers 232 scenarios with 124 categories, including 58 things’ classes and 66 stuff’s classes. Here, VPQ and STQ are both metrics from the VPS task. With our additional design, we compiled taxonomic information and injected them to thing and stuff kernels, and trained YTVIS and VIPSeg jointly, with the total training epochs set to 12. The results are shown below, with our design, the performance of Video K-Net increases with a 2.3% VPQ improvement and 2.6% STQ improvement. This further demonstrates the transferability of our design in in-the-wild scenarios.\n\nTable 2-3. Results on VIPSeg Dataset.\n| Method      | Multiple Dataset? | Backbone  | VPQ  | STQ  |\n| ----------- | :---------------: | --------- | ---- | ---- |\n| Video K-Net |                   | ResNet-50 | 26.1 | 33.1 |\n| Video K-Net | √                 | ResNet-50 | 27.6 | 34.8 |\n| TMT-VIS     | √                 | ResNet-50 | 28.4 | 35.7 |\n\n### Reference\n\n[1] Wu J, Jiang Y, Bai S, et al. Seqformer: Sequential transformer for video instance segmentation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 553-569.\n\n[2] Heo M, Hwang S, Oh S W, et al. Vita: Video instance segmentation via object token association[J]. Advances in Neural Information Processing Systems, 2022, 35: 23109-23120.\n\n[3] Yan B, Jiang Y, Wu J, et al. Universal instance perception as object discovery and retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15325-15336.\n\n[4] Zhou X, Koltun V, Krähenbühl P. Simple multi-dataset detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7571-7580.\n\n[5] Zhao T, Liu P, Lu X, et al. Omdet: Language-aware object detection with large-scale vision-language multi-dataset pre-training[J]. arXiv preprint arXiv:2209.05946, 2022.\n\n[6] Miao J, Wang X, Wu Y, et al. Large-scale video panoptic segmentation in the wild: A benchmark[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 21033-21043.\n\n[7] Li X, Zhang W, Pang J, et al. Video k-net: A simple, strong, and unified baseline for video segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 18847-18857.\n\n[8] Kim D, Woo S, Lee J Y, et al. Video panoptic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 9859-9868."}, {"review_id": "u5Y7tZq38A", "reviewer": "Reviewer_fxcA", "summary": "This paper works on multi-dataset training on video instance segmentation. The authors built on top of Mask2Former, and introduced two components to enable the model to work under different label sets, and take advantages of the given label sets. The two components are both ablated in experiments. The overall framework improved Mask2Former on three popular datasets and achieved state-of-the-art performance.", "questions": "Overall the paper works on an important task, proposes a valid model, and has great results. My concerns are mostly on presentation and clarity (see weekness). Please clarify them in the rebuttal.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem on training a video instance segmentation model on multiple datasets is important. This paper takes a good step in this direction. The overall framework makes sense to me.\n\n- The results are strong. The improvements over the Mask2Former and VITA baselines are significant, and are consistent under three datasets and two backbones. \n\n- The contributions are well ablated in Table 4 - 7.\n\n- The paper is well written and easy to follow.", "weaknesses": "- The most important Table, Table 3, is extremely hard to read. The numbers of different rows are on different datasets, making it hard to compare across rows. PLEASE split the columns by datasets. Please report numbers on all datasets, which should be feasible as the model used a CLIP classifier. We can remove AP50/ AP75 if space is a constraint. \n\n- Please clarify: does the model require a known vocabulary during inference? Can it test on in-the-wild images (e.g., using a unified vocabulary)?\n\n- [Optional] One other interesting aspect of multi-dataset training in VIS is the difference between the framerate/ image size across different datasets. If will be great if the authors can provide some discussion on this (if this is not trivial).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "At the beginning, we want to thank you for the detailed, insightful and constructive comments.\n### Reorganization of Table 3\n\nIn Table 5 of supplementary material, we have posted several experiments of zero-shot performance of our methods when compared with previous methods. In this part, we update these experiments and reorganize Table 3 in the submission file. The updated version is shown below:\n\nTable 3-1. Ablation study on training with multiple VIS datasets.\n\n|  ID  | Method  | YTVIS$_{train}$ | OVIS$_{train}$ | UVO$_{train}$  | YTVIS$_{val}$ | OVIS$_{val}$ | UVO$_{val}$  |\n| :--: | :-----: | :---: | :--: | :--: | :---: | :--: | :--: |\n|  1   |   M2F   |   √   |      |      | 46.4  | 2.3  | 1.9  |\n|  2   |   M2F      |       |  √   |      |  5.2  | 16.5 | 3.6  |\n|  3   |   M2F      |       |      |  √   |  4.4  | 2.5  | 18.2 |\n|  4   |   M2F      |   √   |  √   |      | 47.3  | 17.4 | 4.9  |\n|  5   |   M2F      |   √   |      |  √   | 46.2  | 3.7  | 19.0 |\n|  6   |   M2F      |       |  √   |  √   |  7.1  | 16.6 | 18.7 |\n|  7   |   M2F      |   √   |  √   |  √   | 47.2  | 17.2 | 19.3 |\n|  8   | TMT-VIS |   √   |      |      | 47.3  | 7.2  | 6.5  |\n|  9   | TMT-VIS        |       |  √   |      | 10.5  | 17.8 | 8.0  |\n|  10  |  TMT-VIS       |       |      |  √   | 10.1  | 8.6  | 18.8 |\n|  11  |  TMT-VIS      |   √   |  √   |      | 48.8  | 20.9 | 10.1 |\n|  12  |  TMT-VIS       |   √   |      |  √   | 47.0  | 10.3 | 20.4 |\n|  13  |  TMT-VIS       |       |  √   |  √   | 14.8  | 19.4 | 20.2 |\n|  14  |  TMT-VIS       |   √   |  √   |  √   | 49.7  | 22.8 | 21.2 |\n\n\n### Test on in-the-wild image\nThe tests on in-the-wild datasets can refer to the first answer. Unfortunately there are no other in-the-wild VIS datasets, and so we tested the performance on another in-the-wild dataset from a video-related task, which further proves the transferability of our design.\n\nWe tested our method on VIPSeg[1] based on the popular Video K-Net[2], the results are shown in Table. VIPSeg is a large-scale dataset for video panoptic segmentation (VPS)[3], a task which aims to simultaneously predict object classes, bounding boxes, masks, instance id associations, and semantic segmentation in video frames. There are a total of 3,536 videos with 84,750 pixel-wise annotated frames in VIPSeg, which covers 232 scenarios with 124 categories, including 58 things’ classes and 66 stuff’s classes. Here, VPQ and STQ are both metrics from the VPS task. With our additional design, we compiled taxonomic information and injected them to thing and stuff kernels, and trained YTVIS and VIPSeg jointly, with the total training epochs set to 12. The results are shown below, with our design, the performance of Video K-Net increases with a 2.3% VPQ improvement and 2.6% STQ improvement. This further demonstrates the transferability of our design in in-the-wild scenarios.\n\nTable 3-2. Results on VIPSeg Dataset.\n\n| Method      | Multiple Dataset? | Backbone  | VPQ  | STQ  |\n| ----------- | :---------------: | --------- | ---- | ---- |\n| Video K-Net |                   | ResNet-50 | 26.1 | 33.1 |\n| Video K-Net | √                 | ResNet-50 | 27.6 | 34.8 |\n| TMT-VIS     | √                 | ResNet-50 | 28.4 | 35.7 |\n\n### Vocabulary in inference\nIn the inference part, our SOTA performance method utilizes the given vocabulary of the dataset. However, our method could still perform well with a unified vocabulary, which is credit to the utilization of CLIP encoder, just as Table 3-1 shows.\n\n### Different frame rate across different datasets\n\nWe resample the VIS datasets to a different frames per second(fps) when training on multiple datasets, and the results are shown as follows. Worth mentioning, all VIS datasets are annotated in a fps of 6 (UVO is 30) . As we can see, as the fps decrease from 6 to 1, the performance significantly drops. This is likely to be the consequence of learning temporal relations. With a sparse annotation, the displacement of objects between annotated frames become more significant, and thus increase the difficulty for queries to both segment the desired instances and track their trajectories.\n\nTable 3-3. Experiments on different dataset fps.\n| Method      | Testing Dataset | Dataset FPS | AP   |\n| ----------- | :-------------: | :---------: | :--: |\n| Mask2Former | YTVIS-19         | 1           | 45.5 |\n| TMT-VIS     | YTVIS-19         | 1           | 50.8 |\n| Mask2Former | YTVIS-19         | 6           | 46.4 |\n| TMT-VIS     | YTVIS-19         | 6           | 52.6 |\n\n### Reference\n\n[1] Miao J, Wang X, Wu Y, et al. Large-scale video panoptic segmentation in the wild: A benchmark[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 21033-21043.\n\n[2] Li X, Zhang W, Pang J, et al. Video k-net: A simple, strong, and unified baseline for video segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 18847-18857.\n\n[3] Kim D, Woo S, Lee J Y, et al. Video panoptic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 9859-9868."}, {"review_id": "CmGrw19YIP", "reviewer": "Reviewer_VJ6M", "summary": "Due to the lack of large-scale datasets in the VIS task, the authors propose a multi-dataset joint training method. Due to the heterogeneity of the category spaces of different datasets, simply stacking datasets may lead to performance degradation. For this situation, the author designs a two-stage classification aggregation module, which first compiles the classification information of videos from different datasets, and then Aggregating these categorical information translates categorical priors into instance queries for better performance. The two-stage modules include a classification compilation module and a classification injection module. The former compiles classification information, and the latter utilizes TCM classification information to inject guidance for queries. Experiments prove the effectiveness of this method.", "questions": "1.In Taxonomy Compilation Module, what is the specific form or dimension of label space like? Is it a set of category names?\n2. Taxonomy-aware matching loss, mask loss, and cls loss - what are the specific formulas for these three losses?\n3.In Table 2, Can the effectiveness of the proposed method be verified as it has lower performance on OVIS dataset compared to both IDOL and GenVIS methods?\n4.In Experiment V in Table 3, when jointly training with the Youtube-VIS 2019 and UVO datasets, the Mask2Former-VIS method exhibits a decrease of 0.2 in AP compared to using only YTVIS, but it shows an improvement of 3.2 in AP50. However, the TMT-VIS method experiences a decrease of 0.3 in AP and 0.5 in AP50. Can we conclude that the model lacks robustness to differences between different datasets?", "limitations": "Please see the weakness.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main advantage of this paper is that the proposed joint training model for multiple datasets, which uses taxonomy information to mitigate the heterogeneity in the category space of different datasets, has achieved good results and can try to migrate this approach to other fields.", "weaknesses": "The paper lacks clear explanations on the description of the module and the loss function.\nIn the experimental comparison, the performance in the ovis dataset is not satisfactory.\nFurthermore, the generalization ability of this work on different datasets is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "At the beginning, we want to thank you for the detailed, insightful and constructive comments.\n\n### Label space in Taxonomy Compilation Module\nThe specific dimensions of label space are $K \\times D$, where $K$ refers to the total number of categories in datasets, and $D$ represents the hidden dimensions of outputs of text encoders. \n\n### Formula of taxonomy-aware matching loss, mask loss, and classification loss\n\nThe formula of different losses are shown below:\n\n$$\n\\mathcal{L} = \\sum\\mathcal{L}^{\\text{ce}}(\\mathbf{P^c}, \\mathbf{G^c}) + \\sum\\mathcal{L}_{inj}^{\\text{ce}}(\\mathbf{P^{c^{\\prime}}},\\mathbf{G^{c^{\\prime}}}) + \\sum \\mathcal{L}^{\\text{dice}}(\\mathbf{P^m}, \\mathbf{G^m}) + \\sum \\mathcal{L}^{\\text{bce}}(\\mathbf{P^m}, \\mathbf{G^m})\n$$\n\nwhere $\\mathcal{L}^{\\text{ce}}$ denotes the cross-entropy loss for classification, $\\mathcal{L}_{inj}^{\\text{ce}}$ denotes the extra ‘Taxonomy-aware matching loss’, which is also cross-entropy loss, except for the predictions and gts being different. The  $\\mathcal{L}^{\\text{dice}}$,$\\mathcal{L}^{\\text{bce}}$ denotes the dice loss and binary cross-entropy loss between mask predictions and matched ground truths, which are the same as the strategy in Mask2Former, where we sample different sets of $K=12544=122 \\times 122$ points for different pairs of prediction and ground truth using importance sampling. Here $\\mathbf{P}$ is the prediction, and $\\mathbf{G}$ is the ground truth, $\\mathbf{c}$ refer to classification while $\\mathbf{m}$ refer to mask, and $\\mathbf{c^{\\prime}}$ refers to classification right after the injection process. $\\mathbf{G^{c^{\\prime}}}$ refers to ground truth categories to the taxonomy represented by the predicted set of taxonomic embeddings. By adding this supervision, we further guarantee the guidance provided by the compiled taxonomic embeddings are successfully injected.\n\n### The effectiveness of TMT-VIS on OVIS\n\nOur method is built upon Mask2Former and VITA, which were the SOTA offline method. Also, our module can be added to SOTA method Gen-VIS \\& IDOL, which will also boost their performance significantly, as shown in Table.4-1. The results verify the effectiveness of our proposed method, which can be plug-and-play to both online and offline VIS solutions. Also, the parameters of our added design are not tuned, so that the performance could be even higher with further experiments. \n\nTable 4-1. Experiments on the effectiveness of TMT-VIS on OVIS.\n\n| Method | Multi-dataset? | w/o our design | Backbone  | AP   |\n| ------ | :--------------: | :------------: | --------- | :--: |\n| IDOL   |                   |                | ResNet-50 | 30.2 |\n| IDOL   | √                 |                | ResNet-50 | 32.1 |\n| IDOL   | √                 | √              | ResNet-50 | 33.6 |\n| GenVIS |                   |                | ResNet-50 | 35.8 |\n| GenVIS | √                 |                | ResNet-50 | 37.3 |\n| GenVIS | √                 | √              | ResNet-50 | 38.4 |\n\n### Robustness of our method across various datasets\n\nIt's true that there are degradation when jointly training UVO and YTVIS-19, but we argue that in most of the training settings TMT-VIS improves significantly better than Mask2Former (see the table in rebuttal pdf), and so this result can't indicate that our method lacks robustness. In the following we try to explain the reason why this may occur.\n\nThe drop in performance may be caused by the imbalance in scale among VIS datasets. YTVIS-19 takes advantage of an existing dataset called YouTube-VOS.  OVIS is collected specifically for video instance segmentation in occluded scenes. UVO, on the other hand, adopts videos from Kinetics-400, which are human-centric and contain diverse sets of human actions and human-object interactions, and UVO is densely annotated with many more instances.  The key statistics of these datasets are shown in the following table, which corresponds to the Table.1 of the supplementary file.  As illustrated in the table, the imbalance between these VIS datasets is significant. When jointly training YTVIS-19 with UVO, the model tends to converge to fit UVO, which leads to the degradation of performance on validation on YTVIS-19. When jointly training YTVIS-19 and UVO with OVIS, the imbalance is alleviated, and results in a notable performance improvement. \n\nIt’s worth noting that due to the scale of VIS datasets, especially YTVIS-19, the final results usually have a fluctuation of approximately 0.3AP, and this may also be the direct reason for the seemingly drop of our TMT-VIS compared with Mask2Former-VIS because in other training settings our TMT-VIS significantly outperforms Mask2Former-VIS. \n\nTable 4-2. Key statistics of multiple VIS datasets.\n\n|                  | YT19 | YT21 | OVIS | UVO    |\n| ---------------- | ---- | ---- | ---- | ------ |\n| Videos           | 2883 | 3859 | 901  | 11228  |\n| Categories       | 40   | 40   | 25   | 81     |\n| Instances        | 4883 | 8171 | 5223 | 104898 |\n| Masks            | 131k | 232k | 296k | 593k   |\n| Masks per Frame   | 1.7  | 2    | 4.7  | 12.3   |\n| Objects per Video | 1.6  | 2.1  | 5.8  | 9.3    |"}, {"review_id": "ebo6fklqtb", "reviewer": "Reviewer_k3N8", "summary": "The paper proposes TMT-VIS, combining multiple VIS datasets for a unified VIS model. Specifically, TMT-VIS introduces taxonomy embedding as prompts to make the model aware of taxonomy from different datasets. Experiments on YVIS-2019, YVIS-2021, OVIS, and UVO demonstrate the effectiveness of dataset unification.\n", "questions": "please see my listed questions in the weakness part. \n", "limitations": "The authors have adequately addressed their work's limitations and potential negative societal impact.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method is straightforward and easy to follow. The performance under joint training is good.\n+ Unifying multiple datasets into a single model is of great value.\n", "weaknesses": "- The authors claim that \"TMT-VIS is the first DETR-style framework that can jointly train multiple video instance segmentation datasets with such a huge improvement.\" However, previous work UNINEXT [1] also train multiple VIS dataset and achieves a performance gain with a DETR-style framework, which is missed in the related work.\n- Tab.3 is confusing. The authors are recommended to evaluate all three datasets in this table to show the performance gain under joint training. Furthermore, the authors claim that Mask2Former-VIS meets difficulties in dealing with datasets, decreasing performance under joint training. Does it indicate the ID-I and ID-V in Tab.3? It is worth noticing that TMT-VIS jointly trained on YTVIS and UVO also decreases compared to the model trained on a single dataset. In other words, the proposed TMT-VIS does not overcome performance drop under joint training. It is ambiguous whether the performance gain is caused by the proposed method boosting the performance on a single dataset or alleviating the conflict in multi-dataset training. \n- The ablation on the size of taxonomic embedding is not sufficient. For example, what's the performance using all embeddings? Does it have negative effects when the topk taxonomic embeddings miss the gt classes?\n- Please provide more details of the taxonomy-aware matching loss.\n\n[1] Universal Instance Perception as Object Discovery and Retrieval", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "At the beginning, we want to thank you for the detailed, insightful and constructive comments. \n\n### UNINEXT\nThanks for pointing out our wrong claim. We acknowledge that the UNINEXT is the first DETR-based method which jointly trains multiple VIS datasets. It simply utilizes BERT language encoder to generate language embeddings of categories from all video datasets, and fuse the information with visual embeddings through a simple bi-directional cross-attention module. \n\nHowever, we argue that we are the current SOTA method in jointly training multiple VIS datasets. Our method, on the other hand, adjusts the language embeddings via a taxonomy compilation module which consists of a spatio-temporal adapter and a FFN network, as well as of a taxonomy injection module which fuses the taxonomy information into video queries with a multi-level cross-attention and self-attention layers. Moreover, it achieves a lower performance of 64.3 AP (ours is 64.9 AP) with more training data on Youtube-VIS 2019. \n\nTable 5-1. Comparison with UNINEXT on YTVIS-19.\n| Method        | Datasets Number | Training time  | Backbone   | AP   |\n| ------------- | :-------------: | -------------- | ---------- | ---- |\n| UNINEXT       | 8               | 3 days         | ConvNext-L | 64.3 |\n| TMT-VIS(Ours) | 4               | 1 day 12 hours | Swin-L     | 64.9 |\n\n### Ablation study on the size of taxonomic embedding\n\nThanks for your careful suggestions about the size of taxonomic embedding. We conducted more ablation studies on this based on the ResNet-50 backbone, and the experiment is tested on YTVIS-19 dataset. As the results shown in Table 5-2, the size of taxonomic embedding is crucial to the performance. When the size is set to 1 or 2, which is apparently less than the overall categories in input video, topk taxonomic embeddings miss the gt classes. Compilation of such taxonomic information and injecting them into video queries provide no guidance to the convergence but diverge the attention of queries to irrelevant categories, and thus results in no improvement and even degradation in the final results. When using all embeddings, the improvement is trivial since TCM can not provide filtration to irrelevant classes, and the TIM module is simply injecting the information of the whole category space to queries.  \n\nTable 5-2. Ablation study on sizes of taxonomic embedding set in TCM.\n\n| Size | AP   | Size | AP   |\n| ---- | :--: | ---- | :--: |\n| 1    | 47.1 | 15   | 49.2 |\n| 2    | 47.5 | 20   | 48.5 |\n| 5    | 49.3 | 50   | 47.4 |\n| 10   | 49.7 | 100  | 47.2 |\n\n### The reorganization of Tab.3\n\nThanks for your advice on our table. In this part, we update these experiments and reorganize Tab.3, and the results are shown in the rebuttal pdf.\n\nWe present Tab.3 to demonstrate that due to the heterogeneity of category spaces of multiple VIS datasets, simply training on multiple datasets with an aggregated category space will not improve the performance significantly. The performance may even drop due to the huge imbalance of data, and that ID-I and ID-V in Tab.3 is an example in showing this phenomenon. \n\n### Performance degradation when joint training UVO and YTVIS\n\nYTVIS-19 takes advantage of an existing dataset called YouTube-VOS. OVIS is collected specifically for video instance segmentation in occluded scenes. UVO, on the other hand, adopts videos from Kinetics-400, which are human-centric and contain diverse sets of human actions and human-object interactions, and UVO is densely annotated with many more instances. The key statistics of these datasets are shown in the rebuttal pdf, which corresponds to the Tab.1 of the supplementary file.  As illustrated in the table, the imbalance between these VIS datasets is significant. When jointly training YTVIS-19 with UVO, the model tends to converge to fit UVO, which leads to the degradation of performance on validation on YTVIS-19. When jointly training YTVIS-19 and UVO with OVIS, the imbalance is alleviated, and results in a notable performance improvement. It’s worth noting that due to the scale of VIS datasets, especially YTVIS-19, the final results have a fluctuation of approximately 0.3AP, and this may also be the direct reason for the seemingly drop of our TMT-VIS compared with Mask2Former-VIS because in other training settings our TMT-VIS significantly outperforms Mask2Former-VIS. \n\n### More details of the taxonomy-aware matching loss\n\nThe formula of the taxonomy-aware matching loss is shown as follows:\n\n$$\n\\mathcal{L} = \\sum\\mathcal{L}^{\\text{ce}}(\\mathbf{P^c}, \\mathbf{G^c}) + \\sum\\mathcal{L}_{inj}^{\\text{ce}}(\\mathbf{P^{c^{\\prime}}},\\mathbf{G^{c^{\\prime}}}) + \\sum \\mathcal{L}^{\\text{dice}}(\\mathbf{P^m}, \\mathbf{G^m}) + \\sum \\mathcal{L}^{\\text{bce}}(\\mathbf{P^m}, \\mathbf{G^m})\n$$\n\nwhere $\\mathcal{L}^{\\text{ce}}$ denotes the cross-entropy loss for classification, $\\mathcal{L}_{inj}^{\\text{ce}}$ denotes the extra `Taxonomy-aware matching loss',which is also cross-entropy loss, except for the predictions and gts being different. The \n$\\mathcal{L}^{\\text{dice}}$,$\\mathcal{L}^{\\text{bce}}$ denotes the dice loss and binary cross-entropy loss between mask predictions and matched ground truths, which are the same as the strategy in Mask2Former, where we sample different sets of $K=12544=122 \\times 122$ points for different pairs of prediction and ground truth using importance sampling. Here $\\mathbf{P}$ is the prediction, and $\\mathbf{G}$ is the ground truth, $\\mathbf{c}$ refer to classification while $\\mathbf{m}$ refer to mask, and $\\mathbf{c^{\\prime}}$ refers to classification right after the injection process. $\\mathbf{G^{c^{\\prime}}}$ refers to ground truth categories to the taxonomy represented by the predicted set of taxonomic embeddings. By adding this supervision, we further guarantee the guidance provided by the compiled taxonomic embeddings."}]

Author rebuttal:
At the beginning, we want to thank all reviewers for the detailed, insightful and constructive comments. 

In the PDF file, we have attached the reorganized version of the ablation study on training with multiple VIS datasets, as well as the key statistics of these datasets. In this part, we want to address the novelty-related and the robustness-related concerns from reviewers.

### Novelty

A major challenge in multiple-dataset joint training is the heterogeneity of multiple datasets’ category space. As a result of the difference in category space, though mask precision increases with the data volume, dataset biases might hinder models from generalization: simply utilizing multiple datasets will dilute the attention of models on different categories. Therefore, increasing the data scale and enriching label space while improving classification precision become a huge challenge for researchers. This phenomenon was shown in our ablation study. 

A straightforward method of enhancing classification precision is to aggregate category-related language embeddings into the modelling process, and the aggregation targets are queries to be more specific in DETR-based models. Previous multiple-dataset joint training methods, such as UNINEXT[1], adopt this idea and manage to fuse category-guided prompt with video features. However, aggregating all category-guided prompts introduces irrelevant semantic information. We argue that the total instances in an input video are far less than the size of the whole label space, and so that by filtering out some irrelevant categories, the aggregation of language embeddings becomes more effective and efficient. Thus, in our TCM, taxonomy embeddings are designed to interact with video features in order to unearth the potential taxonomy contained in each of the video frames. After this, we calculated the dot product between the different modulated taxonomic embeddings to predict the most relevant taxonomy in the given video. By aggregating these most relevant compiled taxonomic embeddings into query features though cross-attention and self-attention modules, we could have refined query features with not only richer but also condensed taxonomy information, in contrast with previous methods, which further improves the performance. 


### Robustness of our method across various datasets

It's true that there are degradation when jointly training UVO and YTVIS-19, but we argue that in most of the training settings TMT-VIS improves significantly better than Mask2Former (see the table in rebuttal PDF), and so this result can't indicate that our method lacks robustness. In the following, we try to explain the reason why this may occur.

The drop in performance may be caused by the imbalance in scale among VIS datasets. YTVIS-19 takes advantage of an existing dataset called YouTube-VOS.  OVIS is collected specifically for video instance segmentation in occluded scenes. UVO, on the other hand, adopts videos from Kinetics-400, which are human-centric and contain diverse sets of human actions and human-object interactions, and UVO is densely annotated with many more instances.  The key statistics of these datasets are shown in the following table, which corresponds to the Table.1 of the supplementary file.  As illustrated in the table of key statistics of datasets, the imbalance between these VIS datasets is significant. When jointly training YTVIS-19 with UVO, the model's parameters converge to fit UVO, which leads to the degradation of performance on validation on YTVIS-19. When jointly training YTVIS-19 and UVO with OVIS, the imbalance is alleviated, and results in a notable performance improvement. 

It’s worth noting that due to the scale of VIS datasets, especially YTVIS-19, the final results usually have a fluctuation of approximately 0.3AP, and this may also be the direct reason for the seeming drop of our TMT-VIS compared with Mask2Former-VIS because in other training settings our TMT-VIS significantly outperforms Mask2Former-VIS. 

#### Reference
[1] Yan B, Jiang Y, Wu J, et al. Universal instance perception as object discovery and retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 15325-15336.

Meta-review: The AC and reviewers appreciate the authors rebuttal and discussion. All reviews, rebuttals and comments were carefully read and considered by the AC. All reviewers appreciated the idea of multi-dataset training but were not convinced that the methodology was specific to video. There were also concerns with the writing of the paper and missing experiments. The authors provided comprehensive rebuttals, filling in many of the gaps in the experimental section. 4 of 5 reviewers were positive in the end and thus the AC has decided to accept the paper. Please make the promised modification to the camera ready version of the paper.
Decision: Accept (poster)