Title: TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: taxonomy-aware

multiple-datasets

video instance segementation
Abstract: Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increase with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomy. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all these benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our proposed approach. The code and trained models will be publicly available.

TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation

Abstract

Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increases with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomies. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named T axonomy-aware M ulti-dataset Joint T raining for V ideo I nstance S egmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTubeVIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our approach. The code is available at https://github.com/rkzheng99/TMT-VIS.

1 Introduction

Video Instance Segmentation (VIS) is a fundamental while challenging visual perception task that involves detecting, segmenting, and tracking object instances within videos based on a set of predefined object categories. The goal of VIS is to accurately separate the objects from the background and assign consistent instance IDs across frames, making it a crucial task for various popular downstream applications such as autonomous driving, robot navigation, video editing, etc.

Benefiting from favorable modeling of visual tokens rather than modeling pixels with CNNs, querybased transformer methods [7, 44, 16, 42, 19, 49] like Mask2Former-VIS [7], SeqFormer [44], and VITA [16] begin to dominate the VIS task. The recent success of SAM [21] demonstrates significant improvements in segmentation and detection performance through data scaling. However, mainstream VIS datasets are much smaller than widely used image detection datasets (e.g., approximately 3K videos in YouTube-VIS [47] vs . 120K images in COCO [27]), leaving the potential of leveraging scaled video datasets for multi-frame video input modeling largely unexplored. Previous researches [23, 10] have proved that transformer structures, when compared with CNNs, have the advantage that they can better leverage the large volume of data, and this naturally raises a question: can we alleviate this dilemma via training on large-scale datasets?

Language

Embedding

Encoder

Pred

Video

(Dataset 1)

Embedding

Video

(Dataset 1)

Video

(Dataset N)

…

Past Works

Decoder

Encoder

Pred

Embedding

Ours

Taxo.

C&I.

Decoder

Taxonomy

Embedding

Nonetheless, obtaining a single large-scale VIS dataset is cost-ineffective because of the growing need for manual labeling. A sounding alternative would be training multiple VIS datasets so as to increase data scale and enrich the taxonomy space. Still, simply combining all these datasets to train a joint model does not lead to good performance. As a result of the difference in category space, though mask precision increases with the data volume, dataset biases might hinder models from generalization: simply utilizing multiple datasets will dilute the attention of models on different categories. Therefore, increasing the data scale and enriching label space while improving classification precision become a huge challenge for researchers.

For this problem, we intend to investigate a framework that can train a unified video instance segmentor on multiple datasets while incorporating taxonomy information to boost both mask precision and classification accuracy. To this end, we propose T axonomy-aware M ulti-dataset Joint T raining for V ideo I nstance S egmentation (TMT-VIS) to jointly train multiple datasets on DETR-based methods, which can train multiple video instance segmentation datasets directly without the requirements of manually filtering irrelevant taxonomies. Built upon classic DETR-based VIS methods like Mask2Former-VIS [7], our TMT-VIS consists of two key components to tackle the problem: we first leverage a Taxonomy Compilation Module (TCM), based on a pre-trained text encoder, to compile taxonomy information from the input video. Then in the Taxonomy Injection Module (TIM), the taxonomic embeddings generated in the TCM are instilled to the visual queries in the transformer decoder through a cross-attention-based module, and thus provide a taxonomic prior to the queries. An additional taxonomy-aware matching loss is added to supervise the injection procedure. With this extra taxonomy information aggregation structure added before the transformer decoder, queries can better concentrate on the desired taxonomy in input videos. Fig. 1 demonstrates the design of our modules. By incorporating taxonomic guidance into the DETR-based model, our TMT-VIS model is able to train and utilize multiple datasets effectively.

To the best of our knowledge, TMT-VIS is the DETR-style framework that is able to jointly train multiple video instance segmentation datasets with such improvement. We evaluate our TMT-VIS on four popular VIS benchmarks, including YouTube-VIS 2019 and 2021 [47], OVIS [35], and UVO [39]. The extensive experimental evaluations show the effectiveness and generality of our method with significant improvement over the baselines. For example, compared with Mask2FormerVIS [7] with the ResNet-50 backbone, our TMT-VIS gets absolute AP improvements of 3.3%, 4.3%, 5.8%, and 3.5% on the aforementioned challenging benchmarks, respectively. Compared with another high-performance solution VITA [16], our solution gets absolute AP improvements of 2.8%, 2.6%, 5.5%, and 3.1%, respectively. Finally, the proposed TMT-VIS sets new state-of-the-art records on these popular and challenging benchmarks. Our main contributions can be summarized threefold:

We analyze the limitations of existing video instance segmentation methods and propose a novel multiple-dataset training algorithm named TMT-VIS, which can well utilize taxonomic guidance to train and utilize multiple video instance segmentation datasets effectively.

We develop a two-stage module: Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM). The former adopts the CLIP text encoder to compile taxonomic information in videos, while the latter utilizes the taxonomic information in TCM to inject guidance into queries.

We conduct extensive experimental evaluations on four popular and challenging VIS benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO, and the achieved significant improvements demonstrate the effectiveness and generality of the proposed approach.

2 Related Work

Video instance segmentation. The current VIS methods can be categorized into two: online and offline. Online VIS methods typically adopt a tracking-by-detection approach. They first predict object detection and segmentation results within several frames, and link the outputs with the same identities across frames using optimized matching algorithms, which can be either hand-crafted or learnable-based. MaskTrack R-CNN [47] is the baseline online VIS method, which is built by adding a simple tracking branch to Mask R-CNN [13]. Following works [5, 48, 28, 18] adopt the similar motivation, measuring the similarities between frame-level predictions and associating them through different matching module designs. Inspired by Video Object Segmentation [33] ,Multi-Object Tracking (MOT) [9], and Multi-Object Tracking and Segmentation (MOTS) [37], recent VIS works such as [11, 26, 12] are proposed. While many MOT methods use multiple trackers [2, 34, 50], some more recent MOT works [53, 30] utilize trajectory queries to exploit inter-instance relationships. Motivated by such approaches, GenVIS [15] implements a novel target label assignment and instance prototype memory in query-based sequential learning. On the other hand, IDOL [45] introduces a contrastive learning head for discriminative instance embeddings, based on Deformable-DETR [54].

In contrast to online methods, offline methods aim to predict masks and trajectories of instances in the entire video in one step by feeding the entire video. STEm-Seg [1] utilizes a single-stage model that gradually updates and clusters spatio-temporal embeddings. MaskProp [3] and Propose-Reduce [26] improve mask quality and instance relations through mask propagation. VisTR [42] successfully adapts DETR [6] to the VIS task for the first time, which adopts instance queries to model the whole video. Mask2Former-VIS [7], which is an adaptation of Mask2Former for 3D spatio-temporal features, has become the leading method due to its mask-focused representation. Not long ago, VITA has emerged, as detailed in [16], which employs object tokens to model the entire video and uses video queries to decode object information from these tokens.

Multi-dataset training. It is widely acknowledged that leveraging large-scale data in visual recognition tasks can notably improve performance [4, 31, 52, 51, 40, 36], but collecting annotated data is challenging to scale. In object detection, several recent studies have explored using multiple datasets to train a single detector and improve feature representations. For example, BigDetection [4] merges several datasets with a hand-crafted label mapping in a unified label space. Detection Hub [31] unifies multiple datasets by adapting queries on category embeddings, considering each dataset's category distributions. UniDet [52] trains a unified detector with split classifiers and incorporates a cost for merging categories across datasets to automatically optimize for a common taxonomy. OmDet [51] uses a language-based framework to learn from multiple datasets, but its specific architecture limits the transferability of pre-trained models to other detectors. In video tasks such as action recognition, several works [32, 17, 38, 22, 41] propose to combine multiple video datasets for training. PolyViT [24] further extends this approach by including image, video, and audio datasets using different sampling procedures. MultiTrain [23] utilizes informative representation regularizations and projection losses to learn robust and informative feature representations from multiple action recognition datasets. Despite achieving huge progress, previous language-based works simply use language embeddings in the end to match with outputs in order to get class predictions, whereas our solution leverages the taxonomic embedding to refine queries before the decoder which helps queries to concentrate on desired taxonomy, resulting in a huge improvement in performance.

The UNINEXT[25] is the first DETR-based method that jointly trains multiple VIS datasets. It simply utilizes the BERT language encoder to generate language embeddings of categories from all video datasets, and fuses the information with visual embeddings through a simple bi-directional cross-attention module. However, UNINEXT has no video-specific design, and it doesn't use the language embeddings to predict a set of possible set of categories, so the semantic information of VIS categories is simply aggregated without further operations.

3 Method

In this section, we will first introduce the overall architecture of the proposed framework T axonomyaware M ulti-dataset Joint T raining for V ideo I nstance S egmentation (TMT-VIS) in Sec. 3.1. Then, we will detail the exquisitely designed taxonomy aggregation module, including Taxonomy Compilation Module (TCM) in Sec. 3.2 and Taxonomy Injection Module in Sec. 3.3. Following this, we further

𝑁𝑇

×

Modulated

Taxonomic Embeds

CLIP

Text Encoder

Spatio-temporal

Adapter

Dataset 1

Taxonomy

Dataset 2

Taxonomy

…

Decoder

Back-

bone

Input Video

Frame Features

FFN +

Cross-

Attention

Taxonomic

Embeds

Taxonomy Compilation  Module

Video  Queries

FFN

+

Self-Attention

+

Cross-Attention

Aggregation

Taxonomy Injection  Module

Taxonomy-aware

Matching Loss

L

×

Frozen

K

×

Predictions

demonstrate the taxonomy-aware matching loss and total optimization target in Sec. 3.4. Finally, we give several implementation details of the framework in Sec. 3.5.

3.1 Overall architecture.

Built upon a DETR [6] style framework Mask2Former [8], we inherit the formulation that treats video instance segmentation as a set prediction problem. To enhance classification performance, we incorporate unified category information as an additional input:

where f stands for our method and f text is a pretrained text encoder. V is an input video represented as an image sequence, L is the label space (referred to as taxonomy), and E is the embedding to predict N segmentation masks, Q referring to queries.

With the improved formulation, we introduce a new framework TMT-VIS, which incorporates taxonomy information into queries to improve the classification ability of the model trained on multiple datasets. As shown in Fig. 2, the framework contains two modules: Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM). By interacting video features with adapted taxonomic embedding, we generate video-specific taxonomic prior. Then, we integrate the learned taxonomic embeddings with the learnable queries, enabling better semantic representation through a divide-and-conquer strategy. We will discuss these key modules, after providing a brief overview of the Mask2Former used.

Revisiting Mask2Former. Mask2Former [7, 8] is a universal Transformer-based framework for image or video instance segmentation. Mask2Former implements a Transformer-based decoder, which recursively learns and refines N number of C -dimensional queries Q ∈ R N × C to generate embeddings E . Specifically, the transformer decoder is a nine-layer structure, where l th layer contains a masked cross-attention, a self-attention, and a feed-forward network in a cascaded manner.

3.2 Taxonomy Compilation Module

For joint-training multiple datasets, the biggest challenge is label inconsistency, where taxonomy from different datasets differs. Given multiple annotated VIS datasets D = {D 1 , D 2 , . . . , D n } , we have the whole label space L = {L 1 , L 2 , . . . , L n } . In our Taxonomy Compilation Module (TCM), we utilize the text encoder of the CLIP to generate taxonomy embedding E = {E i , | i = 1 , ..., K } , where K is the total number of category in L , with E i ∈ R d representing the text embedding of a specific category, where d is the dimension size of the embedding. When given a video with T frames as input, we have the video features F ∈ R HW × D × T extracted by the Mask2Former-VIS's Pixel Decoder, where H and W represent the height and weight of the feature map. To align the dimensions between text embeddings and video features, we add a Spatio-temporal Adapter after the text encoder

of the CLIP. Concretely, we use two FC layers and an activation layer in the middle, with the first layer projecting the original embedding to a lower dimension and the second one projecting to align with the dimension of video features. The adapted text embeddings E 1 are sent into cross-attention and FFN operations. We gradually update the taxonomic embeddings layer by layer by aggregating context information from the input video features via cross-attention and FFN, in order to unearth the potential taxonomy contained in images. After this, we calculated the dot product between the different modulated taxonomic embeddings to predict the most relevant taxonomy in the given video according to the score S . Note that · indicates the dot production.

The whole process of TCM can be formulated as follows:

Thus, we can update the taxonomy set by selecting the top N T embeddings with higher scores in S . This suggests that there are N T corresponding categories that are most likely contained in the input video. The compiled taxonomic embeddings E T by concatenating the N T embeddings from E 3 .

3.3 Taxonomy Injection Module

In DETR-based methods, learnable queries in the decoder are usually zero-initialized, which is lack of taxonomy information. With the supervision of taxonomy priors, query features are more likely to converge faster and predict the desired categories. Also, since utilizing multiple datasets will dilute the attention of models on different categories, the aggregation of taxonomic embeddings can help query features concentrate on the predicted subset of taxonomies. Thus, in Taxonomy Injection Module (TIM), we attempt to incorporate the extracted embeddings from TCM to the queries in the decoder structure. The process of TIM can be formulated as follows:

where X l represents the query features of the l th layer of the transformer decoder, l ∈ [1 , L ] , L = 9 .

By aggregating E T into query features through cross-attention and self-attention modules, we could have the refined query features with richer taxonomy information, and thus filtering irrelevant background taxonomy information contained in X l . This strategy shares the same motivation with the masked attention in Mask2Former, which tries to mask out background regions in the spatial dimension, whereas our method alternatively masks out irrelevant taxonomies by using the modulated taxonomic embeddings E T .

3.4 Taxonomy-aware Matching Loss

Typically, DETR-like models view VIS as a set prediction problem and they rely on global one-to-one assignment for bipartite matching, and the Hungarian algorithm is employed to find an optimal assignment with all categories. The classification loss is based on the results of the matching. Inspired by this matching, as we utilize taxonomic embeddings as guidance to refine instance queries, we introduce an extra loss L taxo to supervise the injection of taxonomic embeddings. Specifically, we add a text-aware matching cost: the matching algorithm remains the same, which is cross-entropy loss, but the predicted class are not from the final prediction, but from the prediction right after the injection. By adding this supervision, we further guarantee the guidance provided by the compiled taxonomic embeddings is successfully injected. Finally, we integrate all losses together as follows:

and we set λ cls = 2 . 0 and λ taxo = 0 . 5 .

3.5 Implementation Details

Our method is implemented on top of detectron2 [46]. Hyper-parameters regarding the pixel decoder and transformer decoder are consistent with the settings of Mask2Former-VIS [7]. In the Taxonomy

Compilation Module, the size of the taxonomy embedding set N T is set to 10, which matches the maximum instance number per video. Referring to Mask2Former-VIS [7], we initially train it on the COCO [27] dataset before fine-tuning it on VIS datasets. During inference, we resize the shorter side of each frame to 360 pixels for ResNet [14] backbones and 480 pixels for Swin [29] backbones. In the inference part, our SOTA performance method utilizes the given vocabulary of the dataset. However, our method could still perform well with a unified vocabulary, which is credited to the utilization of the CLIP encoder.

4 Experiments

In this part, we first give some details about the experimented benchmarks in Sec. 4.1. Then, we report the main results in Sec. 4.2, which mainly demonstrates the superiority of the proposed approach over previous solutions. Furthermore, we give detailed ablation studies in Sec. 4.3 and Sec. 4.4 to validate the exquisite design of the whole framework and related proposed modules. Finally, we illustrate some qualitative visualizations in Sec. 4.5 to show the effectiveness.

4.1 Datasets

We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019 and 2021 [47], OVIS [35], and UVO [39]. Key statistics of these datasets are presented in the Appendix. YouTube-VIS 2019 [47] is the first large-scale dataset for video instance segmentation, with 2.9K videos averaging 4.61s in duration and 27.4 frames in validation videos. YouTube-VIS 2021 [47] includes more challenging longer videos with more complex trajectories, resulting in an average of 39.7 frames in validation videos. OVIS [35] dataset is another challenging VIS dataset, offering 25 object categories and focusing on complex scenes with significant occlusions between objects. While only containing 607 training videos, its videos are much longer, lasting

12.77s on average. UVO [39] dataset has 81 object categories (80 shared with COCO [27], plus an extra 'other' category for non-COCO instances). UVO, a subset of 1.2K Kinetics-400 [20] videos, contains 503 videos densely annotated at 30fps, and provides segmentation masks exhaustively for all object instances.

4.2 Main Results

We compare TMT-VIS with state-of-the-art approaches which are with ResNet-50 and Swin-L backbones on the YouTube-VIS 2019 and 2021 [47], OVIS [35], and UVO [39] benchmarks. The results are reported in Tables 1 and 2.

YouTube-VIS 2019. Table 1 shows the comparison on YouTube-VIS 2019 dataset. Our TMT-VIS sets new state-of-the-art results under all of the settings. Based on Mask2Former-VIS, TMT-VIS gets 49.7% AP and 63.2% with ResNet-50 and Swin-L backbones, respectively, outperforming the baseline by 3.3% and 2.8% absolutely. When the proposed designs are added onto VITA, TMT-VIS gets 52.6% AP and 64.9% AP with ResNet-50 and Swin-L backbones, respectively, outperforming the baseline by 2.8% and 1.9%. When GenVIS is added with our design, TMT-VIS gets 53.9% AP and 65.4% AP with ResNet-50 and Swin-L backbones, outperforming the baseline by 2.6% and 1.6%.. We list the model parameters of Mask2Former (216M), SeqFormer (220M), VITA (247M), and our TMT-VIS (255M), which shows that TMT-VIS performs notably better with similar model parameters. Also, we compared our TMT-VIS with UNINEXT [25] on YouTube-VIS 2019 dataset. With our delicate design of TCM & TIM, our method achieves a higher performance of 64.9% AP with 4 VIS datasets on Swin-L backbone, and the total training time is approximately 1 day 12 hours. On the other hand, UNINEXT is trained on 8 video datasets and the total training time is

approximately 3 days, but its performance is 64.3% AP. Our method is proved to be more effective and data-efficient.

YouTube-VIS 2021. Table 1 also compares the results on the YouTube-VIS 2021 dataset. Specifically, based on Mask2Former-VIS, TMT-VIS achieves 44.9% AP and 56.4% AP with ResNet-50 and SwinL backbones, respectively. Compared to the baseline, TMT-VIS boosts performance by 4.3 and 3.8 points. When the exquisite designs are added to VITA, TMT-VIS achieves 48.3% AP and 59.3% AP respectively, boosting baseline performance by 2.6 and 1.8 points. When plugging to GenVIS, TMT-VIS achieves 49.4% AP and 61.9% AP respectively, boosting baseline performance by 3.1% AP and 1.8% AP.

OVIS and UVO. Table 2 illustrates the competitiveness of TMT-VIS on the challenging OVIS dataset. Particularly, TMT-VIS achieves 25.1% AP and 32.5% AP with ResNet-50 and Swin-L backbones with VITA as the base model, and harvests 38.4% AP and 46.9% AP with GenVIS as the base model on these backbones. Table 2 shows the strong performance of TMT-VIS among offline methods on the challenging UVO dataset. These appealing and encouraging results further prove the effectiveness and generality of the proposed approach on multiple dataset training.

4.3 Ablation Studies

In the following, we conduct extensive ablation studies to check the properties of each component. We first show that our algorithm can improve model performance by a large margin when trained on different combinations of datasets. Then we examine the detailed ablations of our design. Note that all of the TMT-VIS ablation experiments are conducted based on Mask2Former-VIS with the ResNet-50 backbone.

Multiple dataset joint-training problem. Even though mask precision increases with the data volume, due to the heterogeneity in category space, simply utilizing multiple datasets will dilute

the attention of models on different categories. In Table 3, we carefully examine the performance of multiple datasets joint training on various datasets. We find that the popular Mask2Former-VIS framework meets difficulties in dealing with datasets with different taxonomy spaces. As the number of datasets increases, the performance may even decrease. In contrast, as we increase the number of datasets, the performance of our TMT-VIS improves gradually.

Key component designs. Table 4 demonstrates the effect of our component designs when combined with the prevalent Mask2Former-VIS. By adopting our algorithm, Mask2Former-VIS achieves a huge gain of 2.5 points in AP performance when both models are trained with three VIS datasets. Our extra loss also demonstrates a gain of 0.5 AP.

Datasets sampling ratio. The sampling ratio of different VIS datasets is a crucial parameter. Diverse sampling ratios can cause model to focus on different categories, which can impact the attention and performance of the model. In this study, we arrange the sampling ratio according to the optimal number of iterations each dataset has trained individually. As shown in Table 5, '1:1:0.75' ratio yields the best results, which aligns with our previous training approaches on a single dataset. The performance gap between different ratios validates that selecting an appropriate sampling ratio is important for ensuring that the model learns the relevant category information effectively and achieves optimal performance during multi-dataset training.

4.4 TCMand TIM

In Tables 6 and 7, we investigate the effect of different sizes of taxonomic embedding set as well as the aggregation strategy used in TCM and TIM.

Size of taxonomic embedding set. Table 6 displays the performance of TMT-VIS with varying numbers of output taxonomic embedding N T in the TCM from 1 to 100. The results indicate that the model performs best when N T = 10 embeddings are selected as input embeddings to TIM. When N T is set to 1 or 2, which is apparently less than the overall categories in input video, topk taxonomic embeddings miss the groundtruth classes. As N T decreases, the aggregated taxonomic guidance contained in embeddings becomes insufficient, and the selected embeddings may not encode taxonomic semantics for all instances in the video, resulting in a drop in performance. Conversely, when N T gets larger than the optimum value, the redundant taxonomic embeddings dilute the category information, leading to performance degradation. When using all embeddings, the improvement is trivial since TCM can not provide filtration to irrelevant classes, and the TIM module is simply injecting the information of the whole category space to queries.

Aggregation strategy. Table 7 demonstrates the results of different aggregation strategies implemented in the TIM. There are multiple ways of aggregating taxonomic embeddings to queries in the transformer decoder: in the 'add' setting, taxonomic embeddings are expanded to match the dimensions of queries and addition is applied to the inflated embeddings; in the 'concatenation' setting, taxonomic embeddings are concatenated with queries to provide additional semantic queries; in the 'cross-attention' setting, taxonomic embeddings are fed to the cross-attention module as key and values to inject category semantics into queries. The results indicate that aggregating taxonomy guidance can improve multi-dataset training performance, and that cross-attention is found to be the most effective strategy for injecting taxonomic embeddings into instance queries. On the other hand, directly concatenating or adding taxonomic embeddings does not improve significantly. This suggests that straightforward modifications to instance queries don't infuse taxonomy information into queries, and such superficial modifications will be refined after passing the cascades of the decoder.

M2F-VIS

M2F-VIS

4.5 Visualization

We also conduct qualitative evaluations and the visual comparison with Mask2Former-VIS. Results are illustrated in Fig. 4. Our TMT-VIS demonstrates better capacity in segmenting small instances and classifying instances with similar categories. The reason is that our TMT-VIS model is benefited from our taxonomy compilation module with multi-dataset training.

5 Conclusion

In this paper, we propose TMT-VIS to address the dilemma in multiple dataset joint training in VIS. By developing a two-stage module, Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM), our new algorithm is able to train and utilize multiple datasets effectively by incorporating taxonomy guidance into the DETR-based model. Our proposed TMT-VIS harvests great performance improvements over the baselines and sets new state-of-the-art records on multiple popular and challenging VIS datasets and benchmarks. We hope that our new approach can provide valuable insights and motivate future VIS research.

Limitations. The utilization and aggregation of taxonomic embeddings may need further investigations, and there might be more sophisticated ways of associating taxonomic embeddings with multi-level queries that may further improve the performance.

Broader impacts. TMT-VIS is designed for effectively training multiple VIS datasets which achieves promising performance. We hope that TMT-VIS can have a positive impact on many industrial areas where dataset biases are severe. We would like to note that VIS research should mind not violate personal privacy.

Acknowledgement. This work is partially supported by the National Natural Science Foundation of China (No. 62201484), National Key R&D Program of China (No. 2022ZD0160100), HKU Startup Fund, and HKU Seed Fund for Basic Research.

References

Ali Athar, Sabarinath Mahadevan, Aljosa Osep, Laura Leal-Taixé, and Bastian Leibe. Stem-seg: Spatiotemporal embeddings for instance segmentation in videos. In ECCV , 2020. 3

Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In ICCV , 2019. 3

Gedas Bertasius and Lorenzo Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR , 2020. 3

Likun Cai, Zhi Zhang, Yi Zhu, Li Zhang, Mu Li, and Xiangyang Xue. Bigdetection: A large-scale benchmark for improved object detector pre-training. In CVPR , 2022. 3

Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV , 2020. 3

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020. 3, 4

Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexander G Schwing. Mask2former for video instance segmentation. arXiv:2112.10764 , 2021. 1, 2, 3, 4, 5, 6, 7

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In CVPR , 2022. 4, 14

Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé. Motchallenge: A benchmark for single-camera multiple target tracking. IJCV , 2021. 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. 1

Yang Fu, Linjie Yang, Ding Liu, Thomas S Huang, and Humphrey Shi. Compfeat: Comprehensive feature aggregation for video instance segmentation. In AAAI , 2021. 3

Su Ho Han, Sukjun Hwang, Seoung Wug Oh, Yeonchool Park, Hyunwoo Kim, Min-Jung Kim, and Seon Joo Kim. Visolo: Grid-based space-time aggregation for efficient online video instance segmentation. In CVPR , 2022. 3

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. 6

Miran Heo, Sukjun Hwang, Jeongseok Hyun, Hanjung Kim, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim. A generalized framework for video instance segmentation. In CVPR , 2023. 3, 6, 7

Miran Heo, Sukjun Hwang, Seoung Wug Oh, Joon-Young Lee, and Seon Joo Kim. Vita: Video instance segmentation via object token association. In NeurIPS , 2022. 1, 2, 3, 6, 7, 14

Ryota Hinami, Junwei Liang, Shin'ichi Satoh, and Alexander Hauptmann. Multimodal co-training for

selecting good examples from webly labeled video. arXiv:1804.06057 , 2018. 3

De-An Huang, Zhiding Yu, and Anima Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. In NeurIPS , 2022. 3, 6, 7

Sukjun Hwang, Miran Heo, Seoung Wug Oh, and Seon Joo Kim. Video instance segmentation using inter-frame communication transformers. In NeurIPS , 2021. 1, 6, 7

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv:1705.06950 , 2017. 7, 14

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV , 2023. 1

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv:2305.06355 , 2023. 3

Junwei Liang, Enwei Zhang, Jun Zhang, and Chunhua Shen. Multi-dataset training of transformers for robust action recognition. In NeurIPS , 2022. 1, 3

Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. TMLR , 2022. 3

Fangjian Lin, Jianlong Yuan, Sitong Wu, Fan Wang, and Zhibin Wang. Uninext: Exploring a unified architecture for vision recognition. In ACMMM , 2023. 3, 7

Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia. Video instance segmentation with a propose-reduce paradigm. In ICCV , 2021. 3

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 1, 6, 7

Dongfang Liu, Yiming Cui, Wenbo Tan, and Yingjie Chen. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR , 2021. 3

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV , 2021. 6

Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multiobject tracking with transformers. In CVPR , 2022. 3

Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In CVPR , 2023. 3

Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In CVPR , 2020. 3

Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV , 2019. 3

Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In CVPR , 2021. 3

Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benchmark. IJCV , 2022. 2, 6, 7, 14, 15

Lu Qi, Jason Kuen, Weidong Guo, Jiuxiang Gu, Zhe Lin, Bo Du, Yu Xu, and Ming-Hsuan Yang. Aims: All-inclusive multi-level segmentation. In NeurIPS , 2023. 3

Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In CVPR , 2019. 3

Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking supplementary material. In CVPR , 2023. 3

Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open-world segmentation. In ICCV , 2021. 2, 6, 7, 14, 15

Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv:2307.06942 , 2023. 3

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. arXiv:2212.03191 , 2022. 3

Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR , 2021. 1, 3

Jialian Wu, Sudhir Yarram, Hui Liang, Tian Lan, Junsong Yuan, Jayan Eledath, and Gerard Medioni. Efficient video instance segmentation via tracklet query and proposal. In CVPR , 2022. 6

Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, and Xiang Bai. Seqformer: Sequential transformer for video instance segmentation. In ECCV , 2022. 1, 6, 7

Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. In ECCV , 2022. 3, 6, 7, 14

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: //github.com/facebookresearch/detectron2 , 2019. 5

Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV , 2019. 1, 2, 3, 6, 7, 14, 15

Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Crossover learning for fast online video instance segmentation. In ICCV , 2021. 3, 6, 7

Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR , 2022. 1, 6

Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification in multiple object tracking. IJCV , 2021. 3

Tiancheng Zhao, Peng Liu, Xiaopeng Lu, and Kyusong Lee. Omdet: Language-aware object detection with large-scale vision-language multi-dataset pre-training. arXiv:2209.05946 , 2022. 3

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In CVPR , 2022. 3

Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl. Global tracking transformers. In CVPR , 2022. 3

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR , 2020. 3

Appendix

This appendix provides more details about the proposed TMT-VIS, further details of VIS datasets, more qualitative visual comparisons.

A Dataset Details

Here, we provide a detailed overview of various VIS datasets in Table 8. Our extensive experimental evaluations are conducted on four challenging benchmarks, namely YouTube-VIS 2019 and 2021 [47], OVIS [35], and UVO [39]. YouTube-VIS 2019 [47] was the first large-scale dataset designed for video instance segmentation, comprising 2.9K videos averaging 4.61s in duration and 27.4 frames in validation videos. YouTube-VIS 2021 [47] poses a greater challenge with longer and more complex trajectory videos, averaging 39.7 frames in validation videos. The OVIS [35] dataset is another challenging VIS dataset with 25 object categories, focusing on complex scenes with significant object occlusions. Despite containing only 607 training videos, OVIS's videos last an average of 12.77s. Lastly, the UVO [39] dataset consists of 1.2K Kinetics-400 [20] videos, densely annotated at 30fps, featuring 81 object categories, including an extra 'other' category for non-COCO instances. It provides exhaustive segmentation masks for all object instances in the 503 videos.

Among all categories in Youtube-VIS 2019, OVIS, and UVO, there are overlapping categories between each dataset, and there are also different categories that share similar semantics. The detailed overlapping categories are marked in Table 9. Overall, Youtube-VIS 2021 and OVIS share a more similar taxonomy space with Youtube-VIS 2019 than UVO, with a common category set of 34 out of 40 for Youtube-VIS 2021 and 22 out of 25 for OVIS. Typically, when the taxonomy spaces of datasets are similar, training them jointly will have smaller dataset biases, which leads to a better result in performance. The characteristics of these datasets align with the improvement in performance when validating the joint-training models on various datasets: the increase is more significant on Youtube-VIS 2021 and OVIS than on UVO. Further details of some specific categories can be found in Table 11.

B Additional Ablation Studies

In this section, we provide more experiments on our proposed methods, we will discuss the generality and zero-shot properties of our training approach, and we will provide further details of the performance change in different taxonomies.

Generality property. The proposed new taxonomy-aware training strategy is an effective and general strategy that can be adapted into various DETR-based approaches (both online & offline) and into various datasets. As in Table 10, when adding our TCM&TIM strategy to the popular VIS architecture Mask2Former-VIS [8], VITA [16], and IDOL [45], we harvest performance gains on all of the three challenging benchmarks. In OVIS, the increase can be up to 6.3% AP for Mask2Former-VIS. As for the popular IDOL, our strategy can also bring about an increase of 2.8% AP in performance. This demonstrates that the proposed taxonomy-aware module can be treated as a plug-and-play design that can be used in various DETR-based VIS methods (both online & offline) across different scenarios and all popular VIS benchmarks.

Per-category performance. In Table 11, we present the comparison in performance between Mask2Former-VIS and our TMT-VIS on several specific taxonomies. When datasets other than YTVIS have no such taxonomy, training Mask2Former-VIS on multiple datasets will end up decreasing the performance of such category, as shown in 'Duck' case from Table 11. When applying our approach, we can obtain a performance improvement due to that taxonomy information of 'Duck' is compiled and injected to the instance queries. In other taxonomies, such as 'Person' which appears across all VIS datasets, the improvements are also significant.

Zero-shot property. Further, our TMT-VIS can also perform well on zero-shot learning. We conducted the experiments on Youtube-VIS 2019 [47], OVIS [35], and UVO [39] benchmarks. As exhibited in Table 12, TMT-VIS can be utilized to transfer the knowledge of other VIS datasets to another dataset with a significant increase of 4.5% AP and 3.8% AP respectively. The extra taxonomy information provided by our newly designed TCM & TIM improve the model's performance when dealing with unfamiliar taxonomies.

C Visualization

In the visualization comparisons between Mask2Former-VIS and our model, we select some cases under different scenarios, which include setting with multiple similar instances, setting with fastmoving objects, and setting with different poses of instance.

In Fig. 4, we demonstrate videos with multiple similar instances, and TMT-VIS can both segment and track them more accurately than Mask2Former-VIS. The swimmers or the cyclists are all instances that belong to 'person' category, and TMT-VIS shows better segmentation and tracking. In Fig. 5, we present example videos which have quick movements in camera's perspective and have instances with different poses. In the top two rows, the sedan and the truck have similar appearances, and our model can distinguish and segment them with higher confidence (90% over 70%). In the last two rows, our model successfully segments the person in different poses, while Mask2Former-VIS fails to segment this person's arm in the first frame. However, the first two rows of Fig. 5 also show that our model still have the problem of segmenting instances with heavy occlusions. This suggests that simply combining taxonomy information is insufficient of solving severely occluded scenes, and that more information should be aggregated to instance queries to make the model more robust in segmenting instances.

M2F-VIS

M2F-VIS

M2F-VIS

M2F-VIS

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TMT-VIS, combining multiple VIS datasets for a unified VIS model. Specifically, TMT-VIS introduces taxonomy embedding as prompts to make the model aware of taxonomy from different datasets. Experiments on YVIS-2019, YVIS-2021, OVIS, and UVO demonstrate the effectiveness of dataset unification.\n\nSTRENGTHS:\n+ The proposed method is straightforward and easy to follow. The performance under joint training is good.\n+ Unifying multiple datasets into a single model is of great value.\n\nWEAKNESSES:\n- The authors claim that \"TMT-VIS is the first DETR-style framework that can jointly train multiple video instance segmentation datasets with such a huge improvement.\" However, previous work UNINEXT [1] also train multiple VIS dataset and achieves a performance gain with a DETR-style framework, which is missed in the related work.\n- Tab.3 is confusing. The authors are recommended to evaluate all three datasets in this table to show the performance gain under joint training. Furthermore, the authors claim that Mask2Former-VIS meets difficulties in dealing with datasets, decreasing performance under joint training. Does it indicate the ID-I and ID-V in Tab.3? It is worth noticing that TMT-VIS jointly trained on YTVIS and UVO also decreases compared to the model trained on a single dataset. In other words, the proposed TMT-VIS does not overcome performance drop under joint training. It is ambiguous whether the performance gain is caused by the proposed method boosting the performance on a single dataset or alleviating the conflict in multi-dataset training. \n- The ablation on the size of taxonomic embedding is not sufficient. For example, what's the performance using all embeddings? Does it have negative effects when the topk taxonomic embeddings miss the gt classes?\n- Please provide more details of the taxonomy-aware matching loss.\n\n[1] Universal Instance Perception as Object Discovery and Retrieval\n\nQUESTIONS:\nplease see my listed questions in the weakness part.\n\nLIMITATIONS:\nThe authors have adequately addressed their work's limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDue to the lack of large-scale datasets in the VIS task, the authors propose a multi-dataset joint training method. Due to the heterogeneity of the category spaces of different datasets, simply stacking datasets may lead to performance degradation. For this situation, the author designs a two-stage classification aggregation module, which first compiles the classification information of videos from different datasets, and then Aggregating these categorical information translates categorical priors into instance queries for better performance. The two-stage modules include a classification compilation module and a classification injection module. The former compiles classification information, and the latter utilizes TCM classification information to inject guidance for queries. Experiments prove the effectiveness of this method.\n\nSTRENGTHS:\nThe main advantage of this paper is that the proposed joint training model for multiple datasets, which uses taxonomy information to mitigate the heterogeneity in the category space of different datasets, has achieved good results and can try to migrate this approach to other fields.\n\nWEAKNESSES:\nThe paper lacks clear explanations on the description of the module and the loss function.\nIn the experimental comparison, the performance in the ovis dataset is not satisfactory.\nFurthermore, the generalization ability of this work on different datasets is limited.\n\nQUESTIONS:\n1.In Taxonomy Compilation Module, what is the specific form or dimension of label space like? Is it a set of category names?\n2. Taxonomy-aware matching loss, mask loss, and cls loss - what are the specific formulas for these three losses?\n3.In Table 2, Can the effectiveness of the proposed method be verified as it has lower performance on OVIS dataset compared to both IDOL and GenVIS methods?\n4.In Experiment V in Table 3, when jointly training with the Youtube-VIS 2019 and UVO datasets, the Mask2Former-VIS method exhibits a decrease of 0.2 in AP compared to using only YTVIS, but it shows an improvement of 3.2 in AP50. However, the TMT-VIS method experiences a decrease of 0.3 in AP and 0.5 in AP50. Can we conclude that the model lacks robustness to differences between different datasets?\n\nLIMITATIONS:\nPlease see the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on multi-dataset training on video instance segmentation. The authors built on top of Mask2Former, and introduced two components to enable the model to work under different label sets, and take advantages of the given label sets. The two components are both ablated in experiments. The overall framework improved Mask2Former on three popular datasets and achieved state-of-the-art performance.\n\nSTRENGTHS:\n- The problem on training a video instance segmentation model on multiple datasets is important. This paper takes a good step in this direction. The overall framework makes sense to me.\n\n- The results are strong. The improvements over the Mask2Former and VITA baselines are significant, and are consistent under three datasets and two backbones. \n\n- The contributions are well ablated in Table 4 - 7.\n\n- The paper is well written and easy to follow.\n\nWEAKNESSES:\n- The most important Table, Table 3, is extremely hard to read. The numbers of different rows are on different datasets, making it hard to compare across rows. PLEASE split the columns by datasets. Please report numbers on all datasets, which should be feasible as the model used a CLIP classifier. We can remove AP50/ AP75 if space is a constraint. \n\n- Please clarify: does the model require a known vocabulary during inference? Can it test on in-the-wild images (e.g., using a unified vocabulary)?\n\n- [Optional] One other interesting aspect of multi-dataset training in VIS is the difference between the framerate/ image size across different datasets. If will be great if the authors can provide some discussion on this (if this is not trivial).\n\nQUESTIONS:\nOverall the paper works on an important task, proposes a valid model, and has great results. My concerns are mostly on presentation and clarity (see weekness). Please clarify them in the rebuttal.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article addresses video instance segmentation from a perspective of multi-dataset joint training. The proposed method is based on DETR, while the main contribution is to inject label taxonomy into model training based on CLIP. The model is evaluated on several standard benchmarks.\n\nSTRENGTHS:\nWhile multi-dataset training is not a new idea in such fields like object detection, the work is a pioneering study (as far as I know) to explore the idea in video instance segmentation. The paper is overall well written and easy to follow in most parts. The experiments are extensive.\n\nWEAKNESSES:\nI would not claim these as weaknesses, but suggestions that might help improve the work. \n\nFirst, while it is good to see that the method shows improvements, I would suggest not characterizing the results as \"a such huge improvement\". On one hand, a 2%-3% improvement is not huge. On the other hand, the improvements just meet the expectation since more data are used for training. In addition to show quantitative results, it is essential to report some other metrics, such as training time, to render a more comprehensive comparison. \n\nSecond, as discussed in Sec. 2, there are other strategies for handling heterogeneous labels in multi-dataset joint training, and I am curious whether existing strategies can be directly used for video instance segmentation. If yes, how about the performance?\n\nThird, in terms of formulation,  E_\\infty is used in Sec 3.2, but does not appear in Eqs. 2-5. It is important to ensure consistency. Eq. 6 is not properly formulated as well; a same symbol is used for both input and output, i.e., $X_{l-1}$.\n\nQUESTIONS:\nBeyond the questions in [weaknesses], I have one last question: is the model applicable to open-world scenarios? or at least can it generalize to other datasets that not involve in model training?\n\nLIMITATIONS:\nAuthors tried to discuss limitations; however, the discussion provided is rather elementary and does not offer crucial insights into the method. More in-depth analysis should be provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a multi-dataset joint training method for the video instance segmentation task. They build on the MaskFormer-VIS model, introducing two additional modules: the Taxonomy Compilation Module and the Taxonomy Injection Module. And the proposed method shows compelling performance.\n\nSTRENGTHS:\nS1. This paper tackles an overlooked problem: multi-dataset joint training for the video instance segmentation task.\n\nWEAKNESSES:\nW1. A significant concern I have is with the explicit relevance and motivation of the proposed methodologies for the task of \"video\" instance segmentation. The paper currently lacks clear justification as to why these particular methods are designed and show impressive performance specifically for this task. It would benefit from a more in-depth exploration and justification of why these methodologies are uniquely suited for \"video\" instance segmentation.\n\nW2. The paper needs an overall improvement in writing quality. The related works section contains several inaccuracies that need rectification. To specify:\n  - L83: The claim that Mask R-CNN [13] and MinVIS [18] use a tracking branch is incorrect.\n  - L89: It is incorrect to label [47, 28] as MOTS works; they are, in fact, MOT methods.\n  - L90: The assertion that IDOL [39] is based on GenVIS [15] is inaccurate.\n\nAdditionally, there's a misleading terminology used for the proposed module on L296; it should be the Taxonomy Compilation Module (TCM), not the Taxonomy Extraction Module (TEM).\n\nW3. The paper seems to possess marginal novelty. In particular, as stated in L116-117, the proposed method leverages taxonomic embedding (as I understand, these are embeddings from the VLM), rather than language embeddings. The distinction of this work compared to previous multi-dataset joint training methods isn't clear. It would be advantageous if the authors could provide a more explicit elaboration on the unique novelty of their work.\n\nQUESTIONS:\nMajor concerns are raised in the weakness section.\n\nLIMITATIONS:\nThis paper has discussed its limitations and societal impact in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-09"}