SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rongkun Zheng $^{1}$  Lu Qi $^{2}$  Xi Chen $^{1}$  Yi Wang $^{3}$
Kun Wang<sup>4</sup> Yu Qiao<sup>3</sup> Hengshuang Zhao<sup>1*</sup>
<sup>1</sup>The University of Hong Kong <sup>2</sup>University of California, Merced
$^{3}$ Shanghai Artificial Intelligence Laboratory  $^{4}$ SenseTime Research
{zrk22@connect, hszhao@cs}.hku.hk
Training on large-scale datasets can boost the performance of video instance segmentation while the annotated datasets for VIS are hard to scale up due to the high labor cost. What we possess are numerous isolated filed-specific datasets, thus, it is appealing to jointly train models across the aggregation of datasets to enhance data volume and diversity. However, due to the heterogeneity in category space, as mask precision increases with the data volume, simply utilizing multiple datasets will dilute the attention of models on different taxonomies. Thus, increasing the data scale and enriching taxonomy space while improving classification precision is important. In this work, we analyze that providing extra taxonomy information can help models concentrate on specific taxonomy, and propose our model named Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to address this vital challenge. Specifically, we design a two-stage taxonomy aggregation module that first compiles taxonomy information from input videos and then aggregates these taxonomy priors into instance queries before the transformer decoder. We conduct extensive experimental evaluations on four popular and challenging benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO. Our model shows significant improvement over the baseline solutions, and sets new state-of-the-art records on all benchmarks. These appealing and encouraging results demonstrate the effectiveness and generality of our approach. The code is available at https://github.com/rkzheng99/TMT-VIS.
Video Instance Segmentation (VIS) is a fundamental while challenging visual perception task that involves detecting, segmenting, and tracking object instances within videos based on a set of predefined object categories. The goal of VIS is to accurately separate the objects from the background and assign consistent instance IDs across frames, making it a crucial task for various popular downstream applications such as autonomous driving, robot navigation, video editing, etc.
Benefiting from favorable modeling of visual tokens rather than modeling pixels with CNNs, query-based transformer methods [7, 44, 16, 42, 19, 49] like Mask2Former-VIS [7], SeqFormer [44], and VITA [16] begin to dominate the VIS task. The recent success of SAM [21] demonstrates significant improvements in segmentation and detection performance through data scaling. However, mainstream VIS datasets are much smaller than widely used image detection datasets (e.g., approximately 3K videos in YouTube-VIS [47] vs. 120K images in COCO [27]), leaving the potential of leveraging scaled video datasets for multi-frame video input modeling largely unexplored. Previous researches [23, 10] have proved that transformer structures, when compared with CNNs, have the advantage that they can better leverage the large volume of data, and this naturally raises a question: can we alleviate this dilemma via training on large-scale datasets?
Figure 1: Comparison of multiple-dataset training paradigms. Previous multiple dataset training methods focus on unifying the label space, and some models adopt language embeddings to interact with queries after the decoder for the final classification. In order to enhance the performance, our model (right part) leverages the taxonomic embedding to refine queries through taxonomic compilation and injection (C&I) before the decoder.
Nonetheless, obtaining a single large-scale VIS dataset is cost-ineffective because of the growing need for manual labeling. A sounding alternative would be training multiple VIS datasets so as to increase data scale and enrich the taxonomy space. Still, simply combining all these datasets to train a joint model does not lead to good performance. As a result of the difference in category space, though mask precision increases with the data volume, dataset biases might hinder models from generalization: simply utilizing multiple datasets will dilute the attention of models on different categories. Therefore, increasing the data scale and enriching label space while improving classification precision becomes a huge challenge for researchers.
For this problem, we intend to investigate a framework that can train a unified video instance segmentor on multiple datasets while incorporating taxonomy information to boost both mask precision and classification accuracy. To this end, we propose Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) to jointly train multiple datasets on DETR-based methods, which can train multiple video instance segmentation datasets directly without the requirements of manually filtering irrelevant taxonomies. Built upon classic DETR-based VIS methods like Mask2Former-VIS [7], our TMT-VIS consists of two key components to tackle the problem: we first leverage a Taxonomy Compilation Module (TCM), based on a pre-trained text encoder, to compile taxonomy information from the input video. Then in the Taxonomy Injection Module (TIM), the taxonomic embeddings generated in the TCM are instilled to the visual queries in the transformer decoder through a cross-attention-based module, and thus provide a taxonomic prior to the queries. An additional taxonomy-aware matching loss is added to supervise the injection procedure. With this extra taxonomy information aggregation structure added before the transformer decoder, queries can better concentrate on the desired taxonomy in input videos. Fig. 1 demonstrates the design of our modules. By incorporating taxonomic guidance into the DETR-based model, our TMT-VIS model is able to train and utilize multiple datasets effectively.
To the best of our knowledge, TMT-VIS is the DETR-style framework that is able to jointly train multiple video instance segmentation datasets with such improvement. We evaluate our TMT-VIS on four popular VIS benchmarks, including YouTube-VIS 2019 and 2021 [47], OVIS [35], and UVO [39]. The extensive experimental evaluations show the effectiveness and generality of our method with significant improvement over the baselines. For example, compared with Mask2Former-VIS [7] with the ResNet-50 backbone, our TMT-VIS gets absolute AP improvements of  $3.3\%$ ,  $4.3\%$ ,  $5.8\%$ , and  $3.5\%$  on the aforementioned challenging benchmarks, respectively. Compared with another high-performance solution VITA [16], our solution gets absolute AP improvements of  $2.8\%$ ,  $2.6\%$ ,  $5.5\%$ , and  $3.1\%$ , respectively. Finally, the proposed TMT-VIS sets new state-of-the-art records on these popular and challenging benchmarks. Our main contributions can be summarized threefold:
- We analyze the limitations of existing video instance segmentation methods and propose a novel multiple-dataset training algorithm named TMT-VIS, which can well utilize taxonomic guidance to train and utilize multiple video instance segmentation datasets effectively.
- We develop a two-stage module: Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM). The former adopts the CLIP text encoder to compile taxonomic information in videos, while the latter utilizes the taxonomic information in TCM to inject guidance into queries.
- We conduct extensive experimental evaluations on four popular and challenging VIS benchmarks, including YouTube-VIS 2019, YouTube-VIS 2021, OVIS, and UVO, and the achieved significant improvements demonstrate the effectiveness and generality of the proposed approach.
Video instance segmentation. The current VIS methods can be categorized into two: online and offline. Online VIS methods typically adopt a tracking-by-detection approach. They first predict object detection and segmentation results within several frames, and link the outputs with the same identities across frames using optimized matching algorithms, which can be either hand-crafted or learnable-based. MaskTrack R-CNN [47] is the baseline online VIS method, which is built by adding a simple tracking branch to Mask R-CNN [13]. Following works [5, 48, 28, 18] adopt the similar motivation, measuring the similarities between frame-level predictions and associating them through different matching module designs. Inspired by Video Object Segmentation [33], Multi-Object Tracking (MOT) [9], and Multi-Object Tracking and Segmentation (MOTS) [37], recent VIS works such as [11, 26, 12] are proposed. While many MOT methods use multiple trackers [2, 34, 50], some more recent MOT works [53, 30] utilize trajectory queries to exploit inter-instance relationships. Motivated by such approaches, GenVIS [15] implements a novel target label assignment and instance prototype memory in query-based sequential learning. On the other hand, IDOL [45] introduces a contrastive learning head for discriminative instance embeddings, based on Deformable-DETR [54].
In contrast to online methods, offline methods aim to predict masks and trajectories of instances in the entire video in one step by feeding the entire video. STEm-Seg [1] utilizes a single-stage model that gradually updates and clusters spatio-temporal embeddings. MaskProp [3] and Propose-Reduce [26] improve mask quality and instance relations through mask propagation. VisTR [42] successfully adapts DETR [6] to the VIS task for the first time, which adopts instance queries to model the whole video. Mask2Former-VIS [7], which is an adaptation of Mask2Former for 3D spatio-temporal features, has become the leading method due to its mask-focused representation. Not long ago, VITA has emerged, as detailed in [16], which employs object tokens to model the entire video and uses video queries to decode object information from these tokens.
Multi-dataset training. It is widely acknowledged that leveraging large-scale data in visual recognition tasks can notably improve performance [4, 31, 52, 51, 40, 36], but collecting annotated data is challenging to scale. In object detection, several recent studies have explored using multiple datasets to train a single detector and improve feature representations. For example, BigDetection [4] merges several datasets with a hand-crafted label mapping in a unified label space. Detection Hub [31] unifies multiple datasets by adapting queries on category embeddings, considering each dataset's category distributions. UniDet [52] trains a unified detector with split classifiers and incorporates a cost for merging categories across datasets to automatically optimize for a common taxonomy. OmDet [51] uses a language-based framework to learn from multiple datasets, but its specific architecture limits the transferability of pre-trained models to other detectors. In video tasks such as action recognition, several works [32, 17, 38, 22, 41] propose to combine multiple video datasets for training. PolyViT [24] further extends this approach by including image, video, and audio datasets using different sampling procedures. MultiTrain [23] utilizes informative representation regularizations and projection losses to learn robust and informative feature representations from multiple action recognition datasets. Despite achieving huge progress, previous language-based works simply use language embeddings in the end to match with outputs in order to get class predictions, whereas our solution leverages the taxonomic embedding to refine queries before the decoder which helps queries to concentrate on desired taxonomy, resulting in a huge improvement in performance.
The UNINEXT[25] is the first DETR-based method that jointly trains multiple VIS datasets. It simply utilizes the BERT language encoder to generate language embeddings of categories from all video datasets, and fuses the information with visual embeddings through a simple bi-directional cross-attention module. However, UNINEXT has no video-specific design, and it doesn't use the language embeddings to predict a set of possible set of categories, so the semantic information of VIS categories is simply aggregated without further operations.
In this section, we will first introduce the overall architecture of the proposed framework Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentation (TMT-VIS) in Sec. 3.1. Then, we will detail the exquisitely designed taxonomy aggregation module, including Taxonomy Compilation Module (TCM) in Sec. 3.2 and Taxonomy Injection Module in Sec. 3.3. Following this, we further
Figure 2: Overall framework of the proposed TMT-VIS method. Taxonomy Compilation Module (TCM) adopts the CLIP text encoder and spatio-temporal Adapter to generate video-specific modulated taxonomic embeddings. Taxonomy Injection Module (TIM) leverages the modulated embeddings to provide taxonomic guidance to visual queries in the decoder. An additional taxonomy-aware loss is added to supervise the compilation.
demonstrate the taxonomy-aware matching loss and total optimization target in Sec. 3.4. Finally, we give several implementation details of the framework in Sec. 3.5.
3.1 Overall architecture.
Built upon a DETR [6] style framework Mask2Former [8], we inherit the formulation that treats video instance segmentation as a set prediction problem. To enhance classification performance, we incorporate unified category information as an additional input:
$$
\mathbf {E} = f (\mathbf {V}, \mathbf {Q} | f _ {\text {t e x t}} (\mathbf {L})) \tag {1}
$$
where  $f$  stands for our method and  $f_{\mathrm{text}}$  is a pretrained text encoder.  $\mathbf{V}$  is an input video represented as an image sequence,  $\mathbf{L}$  is the label space (referred to as taxonomy), and  $\mathbf{E}$  is the embedding to predict  $N$  segmentation masks,  $\mathbf{Q}$  referring to queries.
With the improved formulation, we introduce a new framework TMT-VIS, which incorporates taxonomy information into queries to improve the classification ability of the model trained on multiple datasets. As shown in Fig. 2, the framework contains two modules: Taxonomy Compilation Module (TCM) and Taxonomy Injection Module (TIM). By interacting video features with adapted taxonomic embedding, we generate video-specific taxonomic prior. Then, we integrate the learned taxonomic embeddings with the learnable queries, enabling better semantic representation through a divide-and-conquer strategy. We will discuss these key modules, after providing a brief overview of the Mask2Former used.
Revisiting Mask2Former. Mask2Former [7, 8] is a universal Transformer-based framework for image or video instance segmentation. Mask2Former implements a Transformer-based decoder, which recursively learns and refines  $N$  number of  $C$ -dimensional queries  $\mathbf{Q} \in \mathbb{R}^{N \times C}$  to generate embeddings  $\mathbf{E}$ . Specifically, the transformer decoder is a nine-layer structure, where  $l^{\mathrm{th}}$  layer contains a masked cross-attention, a self-attention, and a feed-forward network in a cascaded manner.
3.2 Taxonomy Compilation Module
For joint-training multiple datasets, the biggest challenge is label inconsistency, where taxonomy from different datasets differs. Given multiple annotated VIS datasets  $\mathcal{D} = \{\mathcal{D}_1,\mathcal{D}_2,\dots ,\mathcal{D}_n\}$ , we have the whole label space  $\mathcal{L} = \{\mathcal{L}_1,\mathcal{L}_2,\ldots ,\mathcal{L}_n\}$ . In our Taxonomy Compilation Module (TCM), we utilize the text encoder of the CLIP to generate taxonomy embedding  $\mathcal{E} = \{\mathcal{E}_i,|i = 1,\dots,K\}$  where  $K$  is the total number of category in  $\mathcal{L}$ , with  $\mathcal{E}_i\in \mathbb{R}^d$  representing the text embedding of a specific category, where  $d$  is the dimension size of the embedding. When given a video with  $T$  frames as input, we have the video features  $\mathcal{F}\in \mathbb{R}^{HW\times D\times T}$  extracted by the Mask2Former-VIS's Pixel Decoder, where  $H$  and  $W$  represent the height and weight of the feature map. To align the dimensions between text embeddings and video features, we add a Spatio-temporal Adapter after the text encoder
of the CLIP. Concretely, we use two FC layers and an activation layer in the middle, with the first layer projecting the original embedding to a lower dimension and the second one projecting to align with the dimension of video features. The adapted text embeddings  $\mathcal{E}^1$  are sent into cross-attention and FFN operations. We gradually update the taxonomic embeddings layer by layer by aggregating context information from the input video features via cross-attention and FFN, in order to unearth the potential taxonomy contained in images. After this, we calculated the dot product between the different modulated taxonomic embeddings to predict the most relevant taxonomy in the given video according to the score  $S$ . Note that  $\cdot$  indicates the dot production.
The whole process of TCM can be formulated as follows:
$$
\mathcal {E} ^ {1} = \operatorname {A d a p t e r} (\mathcal {E}), \tag {2}
$$
$$
\mathcal {E} ^ {2} = \operatorname {F F N} (\operatorname {C r o s s A t t n} \left(\mathcal {E} ^ {1}, \mathcal {F}\right)), \tag {3}
$$
$$
\mathcal {E} ^ {3} = \operatorname {F F N} (\operatorname {C r o s s A t t n} \left(\mathcal {E} ^ {2}, \mathcal {F}\right)), \tag {4}
$$
$$
\mathcal {S} = \operatorname {S i g m o i d} \left(\operatorname {L i n e a r} \left(\mathcal {E} ^ {3}\right) \cdot \mathcal {E} ^ {1}\right). \tag {5}
$$
Thus, we can update the taxonomy set by selecting the top  $N_{T}$  embeddings with higher scores in  $\mathcal{S}$ . This suggests that there are  $N_{T}$  corresponding categories that are most likely contained in the input video. The compiled taxonomic embeddings  $\mathcal{E}_{\mathcal{T}}$  by concatenating the  $N_{T}$  embeddings from  $\mathcal{E}^3$ .
3.3 Taxonomy Injection Module
In DETR-based methods, learnable queries in the decoder are usually zero-initialized, which is lack of taxonomy information. With the supervision of taxonomy priors, query features are more likely to converge faster and predict the desired categories. Also, since utilizing multiple datasets will dilute the attention of models on different categories, the aggregation of taxonomic embeddings can help query features concentrate on the predicted subset of taxonomies. Thus, in Taxonomy Injection Module (TIM), we attempt to incorporate the extracted embeddings from TCM to the queries in the decoder structure. The process of TIM can be formulated as follows:
$$
\mathbf {X} _ {l - 1} ^ {\prime} = \operatorname {F F N} \left(\operatorname {C r o s s A t t n} \left(\mathbf {X} _ {l - 1}, \mathcal {E} _ {T}\right), \right. \tag {6}
$$
$$
\mathbf {X} _ {l} = \operatorname {F F N} \left(\operatorname {S e l f A t t n} \left(\operatorname {C r o s s A t t n} \left(\mathbf {X} _ {l - 1} ^ {\prime}, \mathcal {F}\right)\right)\right). \tag {7}
$$
where  $\mathbf{X}_l$  represents the query features of the  $l^{th}$  layer of the transformer decoder,  $l\in [1,L],L = 9$
By aggregating  $\mathcal{E}_T$  into query features through cross-attention and self-attention modules, we could have the refined query features with richer taxonomy information, and thus filtering irrelevant background taxonomy information contained in  $\mathbf{X}_l$ . This strategy shares the same motivation with the masked attention in Mask2Former, which tries to mask out background regions in the spatial dimension, whereas our method alternatively masks out irrelevant taxonomies by using the modulated taxonomic embeddings  $\mathcal{E}_T$ .
3.4 Taxonomy-aware Matching Loss
Typically, DETR-like models view VIS as a set prediction problem and they rely on global one-to-one assignment for bipartite matching, and the Hungarian algorithm is employed to find an optimal assignment with all categories. The classification loss is based on the results of the matching. Inspired by this matching, as we utilize taxonomic embeddings as guidance to refine instance queries, we introduce an extra loss  $\mathcal{L}_{taxo}$  to supervise the injection of taxonomic embeddings. Specifically, we add a text-aware matching cost: the matching algorithm remains the same, which is cross-entropy loss, but the predicted class are not from the final prediction, but from the prediction right after the injection. By adding this supervision, we further guarantee the guidance provided by the compiled taxonomic embeddings is successfully injected. Finally, we integrate all losses together as follows:
$$
\mathcal {L} = \mathcal {L} _ {\text {m a s k}} + \lambda_ {\text {c l s}} \mathcal {L} _ {\text {c l s}} + \lambda_ {\text {t a x o}} \mathcal {L} _ {\text {t a x o}}. \tag {8}
$$
and we set  $\lambda_{\mathrm{cls}} = 2.0$  and  $\lambda_{\mathrm{taxo}} = 0.5$
Our method is implemented on top of detectron2 [46]. Hyper-parameters regarding the pixel decoder and transformer decoder are consistent with the settings of Mask2Former-VIS [7]. In the Taxonomy
Table 1: Results comparison on the YouTube-VIS 2019 and 2021 validation sets. We group the results by online or offline methods, and then with ResNet-50 or Swin-L backbone structures. TMT-VIS is the model we built upon Mask2Former-VIS, while TMT-VIS† in 'Online' and 'Offline' are the model that we add our designs based on popular online and offline method, GenVIS and VITA. Our algorithm gets the highest AP performance compared with recent approaches.
Method	Backbone	YouTube-VIS 2019	YouTube-VIS 2021
AP	\(AP_{50}\)	\(AP_{75}\)	\(AR_1\)	\(AR_{10}\)	AP	\(AP_{50}\)	\(AP_{75}\)	\(AR_1\)	\(AR_{10}\)
Online	MaskTrack R-CNN [47]	ResNet-50	38.6	56.3	43.7	35.7	42.5	36.9	54.7	40.2	30.6	40.9
CrossVIS [48]	ResNet-50	36.3	56.8	38.9	35.6	40.7	34.2	54.4	37.9	30.4	38.2
MinVIS [18]	ResNet-50	47.4	69.0	52.1	45.7	55.7	44.2	66.0	48.1	39.2	51.7
IDOL [45]	ResNet-50	49.5	74.0	52.9	47.7	58.7	43.9	68.0	49.6	38.0	50.9
GenVIS [15]	ResNet-50	51.3	72.0	57.8	49.5	60.0	46.3	67.0	50.2	40.6	53.2
TMT-VIS†	ResNet-50	53.9	74.8	59.1	51.4	62.7	49.4	69.1	51.8	43.6	54.8
MinVIS [18]	Swin-L	61.6	83.3	68.6	54.8	66.6	55.3	76.6	62.0	45.9	60.8
IDOL [45]	Swin-L	64.3	87.5	71.0	55.6	69.1	56.1	80.8	63.5	45.0	60.1
GenVIS [15]	Swin-L	63.8	85.7	68.5	56.3	68.4	60.1	80.9	66.5	49.1	64.7
TMT-VIS†	Swin-L	65.4	88.2	72.1	56.4	69.3	61.9	82.0	68.3	51.2	65.9
Offline	EfficientVIS [43]	ResNet-50	37.9	59.7	43.0	40.3	46.6	34.0	57.5	37.3	33.8	42.5
IFC [19]	ResNet-50	41.2	65.1	44.6	42.3	49.6	35.2	55.9	37.7	32.6	42.9
Mask2Former-VIS [7]	ResNet-50	46.4	68.0	50.0	-	-	40.6	60.9	41.8	-	-
TeViT [49]	MsgShifT	46.6	71.3	51.6	44.9	54.3	37.9	61.2	42.1	35.1	44.6
SeqFormer [44]	ResNet-50	47.4	69.8	51.8	45.5	54.8	40.5	62.4	43.7	36.1	48.1
VITA [16]	ResNet-50	49.8	72.6	54.5	49.4	61.0	45.7	67.4	49.5	40.9	53.6
TMT-VIS	ResNet-50	49.7	73.4	53.9	49.2	60.7	44.9	66.1	48.5	39.8	52.1
TMT-VIS†	ResNet-50	52.6	74.4	57.3	50.6	61.8	48.3	69.8	50.8	42.0	55.2
SeqFormer [44]	Swin-L	59.3	82.1	66.4	51.7	64.4	51.8	74.6	58.2	42.8	58.1
Mask2Former-VIS [7]	Swin-L	60.4	84.4	67.0	-	-	52.6	76.4	57.2	-	-
VITA [16]	Swin-L	63.0	86.9	67.9	56.3	68.1	57.5	80.6	61.0	47.7	62.6
TMT-VIS	Swin-L	63.2	85.2	68.3	56.4	68.2	56.4	80.2	61.3	47.0	61.4
TMT-VIS†	Swin-L	64.9	86.1	69.7	57.2	69.3	59.3	81.0	63.7	48.9	63.5
Compilation Module, the size of the taxonomy embedding set  $N_{T}$  is set to 10, which matches the maximum instance number per video. Referring to Mask2Former-VIS [7], we initially train it on the COCO [27] dataset before fine-tuning it on VIS datasets. During inference, we resize the shorter side of each frame to 360 pixels for ResNet [14] backbones and 480 pixels for Swin [29] backbones. In the inference part, our SOTA performance method utilizes the given vocabulary of the dataset. However, our method could still perform well with a unified vocabulary, which is credited to the utilization of the CLIP encoder.