SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Backbone	Model	Fully-Supervised	Weakly-Supervised Transfer
ADE semantic mIoU	COCO panoptic PQ	ADE semantic → panoptic PQ	O365 box → instance mask AP
ResNet50	Separate	42.0	48.2	26.9	12.3
DaTaSeg	48.1 (+6.1)	49.0 (+0.8)	29.8 (+2.9)	14.3 (+2.0)
ViTDet-B	Separate	46.3	51.9	27.5	14.7
DaTaSeg	51.4 (+5.1)	52.8 (+0.9)	32.9 (+5.4)	16.1 (+1.4)
ViTDet-L	DaTaSeg	54.0	53.5	33.4	16.4
Table 1: Comparing DaTaSeg with separate dataset-specific models. We show results on both the training tasks (fully-supervised) and new tasks (weakly-supervised transfer). Our single DaTaSeg outperforms separately trained models on all datasets. They are trained under the same settings, so the performance gains come from knowledge in other datasets through our multi-dataset multi-task cotraining recipe. We observe: 1) DaTaSeg significantly improves tasks with limited data (ADE20k semantic); 2) DaTaSeg enables weakly-supervised knowledge transfer (ADE20k panoptic and O365 instance); 3) DaTaSeg scales well with backbones.
Training sets	Fully-Supervised	Weakly-Supervised Transfer
ADE semantic	COCO panoptic	O365bbox	ADE semantic mIoU	COCO panoptic PQ	ADE semantic → panoptic PQ	O365 box → instance mask AP
✓	42.0	8.4	26.7	1.0
✓	15.3	48.2	11.6	5.8
✓	11.0	12.4	5.8	12.3
✓	✓	46.3 (+4.3)	14.0	26.4 (-0.3)	15.0 (+2.7)
✓	✓	18.3	48.9 (+0.7)	12.3	15.2 (+2.9)
✓	✓	47.3 (+5.3)	49.0 (+0.8)	30.5 (+3.8)	5.6
✓	✓	✓	48.1 (+6.1)	49.0 (+0.8)	29.8 (+2.9)	14.3 (+2.0)
Table 2: Importance of training on multiple datasets. We train DaTaSeg on all combinations the three datasets. Experiments are done on ResNet50 under same settings. DaTaSeg scales with the number of training datasets.
except that we use Layer Norm [1] and GeLU [24] activation for training stability. For ViTDet, we use the ImageNet-1K MAE pretrained checkpoint [20] with the layerwise lr decay [39]; the pixel decoder is the simple feature pyramid inside ViTDet, whose outputs are upsampled and added together to get the high resolution feature map  $\mathcal{F}$ . The mask embedding head is a 3-layer MLP. The class embedding head contains a linear layer followed by ReLU. We use CLIP-L/14 [53] as the pretrained text encoder. We use 100 segment queries  $\mathcal{Q}_s$ , unless otherwise stated. See Appendix H for more details.
4.1 DaTaSeg improves over dataset-specific models
As an apples-to-apples comparison, we separately train a model on each dataset. We ensure that the models see the same number of images in cotraining and separately-training on each dataset. Table 1 (left) shows the results. DaTaSeg outperforms separately trained models on all datasets. Since we use exactly the same settings, we attribute the performance boost to our multi-dataset multi-task training recipe, which harnesses knowledge from multiple sources. Especially, DaTaSeg leads to a significant performance boost on ADE20k semantic:  $+6.1$  mIoU and  $+5.1$  mIoU with ResNet50 and ViTDet-B backbones, respectively. This proves our argument that cotraining on more data helps overcome the data limitation issue in smaller-scale segmentation datasets.
4.2 DaTaSeg enables weakly-supervised transfer
In Table 1 (right), we evaluate DaTaSeg on ADE20k panoptic and Objects365 instance segmentation tasks. Note we only have weak supervision (ADE20k semantic and Objects365 bbox) during training. Again, we compare to models separately trained on one dataset as our baselines. Specifically, we use the ADE semantic for ADE panoptic evaluation, and an Objects365 model trained only using weak box supervision for Objects365 instance evaluation, respectively.
By cotraining DaTaSeg in a multi-dataset multi-task fashion, we improve the ADE20k panoptic performance by  $+2.9$  PQ and  $+5.4$  PQ with ResNet50 and ViTDet-B backbones. Our design allows the panoptic segmentation knowledge to transfer from COCO to ADE20k. For reference, the fully-supervised performance is 37.7 PQ on ResNet50 and 42.3 PQ on ViTDet-B.
Method	Backbone	Training data	PC-59 mIoU	PC-459 mIoU	COCO mIoU	Cityscapes PQ
ODISE [68]	UNet+M2F	LAION+CLIP+COCO	55.3	13.8	52.4	23.9
MaskCLIP [13]	R-50	COCO pan+CLIP	45.9	10.0	-	-
OVSeg [40]	R-101c	COCO stuff+cap	53.3	11.0	-	-
Swin-B	55.7	12.4	-	-
OpenSeg [16]	R-101	COCO pan+cap	42.1	9.0	36.1	-
Eff-b7	COCO+Loc. Narr.	44.8	11.5	38.0	-
DaTaSeg	R-50	COCO panoptic	50.9	11.1	57.7	30.0
ViTDet-B	+ADE semantic	51.1	11.6	62.7	28.0
ViTDet-L	+O365 bbox	51.4	11.1	62.9	29.8
Table 3: Compare DaTaSeg to state-of-the-art open-vocabulary segmentation models. We apply DaTaSeg directly to other semantic or panoptic segmentation datasets without finetuning. We compare to ODISE [68], MaskCLIP [13], OVSeg [40], and OpenSeg [16] on their corresponding benchmarks. We only conduct system-level comparison due to differences in training data and backbones. DaTaSeg performs comparably on all semantic segmentation benchmarks and outperforms the recent work ODISE on Cityscapes. PC: Pascal Context. COCO: COCO is included in the training set for all models, and is not completely "open-vocaburally".
# queries	50	100	150
ADE semantic	mIoU	47.4	48.1	48.7
COCO panoptic	PQ	48.7	49.0	48.9
ADE panoptic†	PQ	30.7	29.8	29.1
O365 instance†	AP	13.4	14.3	15.8
Table 4: In general, the performance of DaTaSeg increases as we increase the number of segment queries. Besides, using only 50 queries already achieves good performance. Experiments are conducted using a ResNet50 backbone.  $\dagger$ : weakly-supervised tasks.
DaTaSeg	+D-S modules
ADE semantic	mIoU	48.1	48.1 (-0.0)
COCO panoptic	PQ	49.0	46.0 (-3.0)
ADE panoptic†	PQ	29.8	26.9 (-2.9)
O365 instance†	AP	14.3	10.9 (-3.4)
Table 5: Adding dataset-specific (D-S) modules hurts performance on almost all datasets. This shows the importance of sharing all parameters for better knowledge sharing, which particularly benefits the weakly-supervised tasks. Experiments are conducted on a ResNet50 backbone.
On Objects365, we only train with a weak box consistency loss  $\mathcal{L}_{proj}$ . The multi-dataset multitask training recipe improves the mask AP by  $+2.0$  AP and  $+1.4$  AP for the two backbones. The improvement is most likely from the instance segmentation knowledge in COCO panoptic.
4.3 DaTaSeg scales with size of backbones
Our DaTaSeg design is orthogonal to detailed network architectures, as long as it is able to output the universal representation (mask proposals with class predictions). This is supported by the consistent performance gains among ResNet50, ViTDet-B, and ViTDet-L backbones as shown in Table 1. It also scales well as we increase the sizes of the backbones. With ViTDet-L, DaTaSeg reaches 54.0 mIoU on ADE20k semantic and 53.5 PQ on COCO panoptic in a single model, without bells and whistles.
4.4 DaTaSeg scales with number of datasets
We study how DaTaSeg scales with the number of training datasets. We train DaTaSeg on all combinations of one, two, and three datasets among the ADE20k semantic, COCO panoptic, and Objects365 detection datasets, in order to conduct a comprehensive study, with a ResNet50 backbone. Table 2 presents the results. Looking at each column, we see that performance on each dataset generally improves as the number of training datasets increases, especially for ADE semantic.
Since DaTaSeg shares all parameters among all datasets and tasks, we can evaluate the cross-dataset transfer performance. We notice the model transfers to datasets that are not trained. e.g., A model trained on COCO panoptic and Objects365 detection achieves 18.3 mIoU on ADE semantic, which is comparable to open-vocabulary segmentation performance (LSeg+ [16, 36] achieves 18.0 mIoU on a ResNet101 backbone).
4.5 DaTaSeg enables open-vocabulary segmentation
A further advantage of our fully-shared architecture is that we have the ability to directly transfer to other segmentation datasets. We simply switch the text embedding classifier with the vocabularies
(d) Objects365 instance (weakly-supervised)
Figure 4: Qualitative results of DaTaSeg on all datasets & tasks. For every pair of images, the left is DaTaSeg prediction and the right is groundtruth. DaTaSeg succeeds on hard cases (e.g., transparent wine glasses on the top left) as well as weakly-supervised datasets. For object instances in panoptic and instance segmentation, the colors are not matched to the groundtruths. Black denotes ignored regions.
in a target dataset not used in training. Table 3 shows the results comparing DaTaSeg to several open-vocabulary works: ODISE [68] and MaskCLIP [13] are open-vocabulary panoptic segmentation methods and OVSeg [40] and OpenSeg [16] are open-vocabulary semantic segmentation approaches. Unlike these works, our method does not train on large-scale image-text data or use pretrained image-text models (except the pretrained text encoder), which puts our method at a disadvantage. Despite this disadvantage, DaTaSeg outperforms ODISE on the Cityscapes panoptic dataset. Note that there is a domain gap between our training sets and Cityscapes, which focuses on urban street scenes. DaTaSeg achieves comparable performance on PC-59 and PC-459. All methods in comparison train on COCO panoptic and ours has the best performance on COCO semantic. These results show that DaTaSeg enables open-vocabulary segmentation via our multi-dataset multi-task training approach.
4.6 DaTaSeg scales with number of queries
We perform ablation study on the number of segment queries  $\mathcal{Q}_s$  and show the results in Table 4. The performance improves as we increase the number of queries from 50 to 150, on almost all datasets. Besides, DaTaSeg achieves reasonably good performance with only 50 queries, which may benefit memory-limited application scenarios. We perform more ablation studies in Appendix D.
4.7 Adding dataset-specific modules hurts performance
In multi-dataset learning, prior works have introduced dataset-specific modules [47], in order to address inconsistencies across datasets [33, 61]. However, such a design may weaken knowledge sharing across different datasets and tasks, especially for weakly-supervised tasks that may rely more on other datasets and tasks to compensate for the weak supervision. To study this issue thoroughly, we carefully design several dataset-specific modules, including dataset-specific queries, dataset-specific heads, and dataset-specific prompts. These modules are lightweight by design so as to still encourage knowledge sharing through other shared parameters.
In Table 5, we evaluate the effectiveness of these dataset-specific modules by adding them to DaTaSeg, and keeping all other settings the same. Comparing with DaTaSeg that shares all parameters, results reveal that using dataset-specific modules hurts performance on all datasets except ADE20k semantic. Moreover, we see that the performance drops more on weakly-supervised tasks (ADE panoptic and O365 instance). This verifies that removing dataset-specific modules yields better knowledge sharing across datasets and tasks, and thus greatly improves performance. See Appendix E for more details.
4.8 Qualitative analysis
We show qualitative results (with ViTDet-L) in Fig. 4. DaTaSeg performs well in both fully and weakly supervised settings. We observe that localization quality is good in general, while classification is more error-prone. Nevertheless, DaTaSeg succeeds in challenging cases including transparent objects, occlusion, and crowd scenes.
The goal of our work is to train a single universal segmentation model on multiple datasets and multiple tasks (i.e., semantic, instance and panoptic segmentation). We present DaTaSeg, which leverages a universal segmentation representation, shared network parameters, and shared semantic embedding space for classification. DaTaSeg surpasses separately trained models, leading to significant gains on smaller datasets such as ADE20k semantic. It unlocks a new weakly-supervised transfer capability on datasets that were not explicitly trained for a particular task (e.g., ADE20k panoptic and Objects365 instance). DaTaSeg also directly transfer to more segmentation datasets and enables open-vocabulary segmentation. We believe that in the future, this multi-task multi-dataset setting will be the norm rather than the outlier as it currently is, and we see our paper as taking a step in this direction.
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450, 2016. 7
[2] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR, 2019. 6
[3] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. NeurIPS, 2019. 16
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 6
[5] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017. 3
[6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018. 1
[7] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In CVPR, 2020. 3
[8] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, 2022. 1, 3, 5, 17, 20, 21
[9] Bowen Cheng, Omkar Parkhi, and Alexander Kirillov. Pointly-supervised instance segmentation. In CVPR, 2022. 20
[10] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS, 2021. 1, 2, 3, 6, 17
[11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2, 6
[12] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV, 2015. 3
[13] Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskclip. ICML, 2023. 8, 9
[14] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. 15
Funding in direct support of this work: Google LLC. There is no additional revenues related to this work.
We propose a single framework for multi-dataset multi-task segmentation, which especially benefits memory-limited application scenarios, e.g., our work can make a single segmentation model that meets all kinds of segmentation needs on memory-limited mobile devices (semantic segmentation for photo editing, instance segmentation for object detection and image search, etc.), instead of using separate models for various segmentation tasks.
Moreover, our approach improves segmentation performance from a data-centric view: We leverage segmentation data from multiple sources to improve the overall segmentation performance, especially on datasets of smaller scales, e.g., ADE20k semantic. We also transfer knowledge from other sources to enable weakly-supervised segmentation. This is helpful on the scenario where segmentation data are very hard to collect, we can cotrain on similar data to help improve the performance, or even train on data with weaker supervision. e.g., if we want to train a segmentation model on rare wild animals, we can co-train a model on a small set of these kinds of animals, and a larger dataset of more common animals.
As for bias and fairness, our model learns from the training data, so it may inherit the bias and fairness issues existing in the training data. Nonetheless, our model trains on multiple sources, and the fairer datasets may compensate the bias and fairness issues in other datasets.
B More about our newly labeled Objects365 instance segmentation evaluation dataset
We randomly sampled 1,000 images from the Objects365 V2 validation set. Different from the "federated" LVIS dataset [18] (not all categories in each image are completely labeled), the bounding boxes in Objects365 are completely labeled for all categories and we follow this complete annotation in our instance masks. For each bounding box annotation, we generate a foreground/background annotation question for the raters. In the question, raters are asked to paint a semi-transparent instance mask on the original image, with the groundtruth bbox shown on the image as a guide and the groundtruth category displayed on the side. We do not crop the original image to the bbox region, but show the entire image to provide more context information, which is especially helpful for small objects. If the boundary is too blurry or too dark to annotate the instance mask, the raters can skip the question.
The annotation tool is a free painting tool, which allows the raters to freely draw the instance mask. We ask the raters to try to draw within the bbox, but if the object is obviously exceeding thebbox, then they can draw outside the_bbox. The size of the stroke is adjustable. Raters can zoom-in/out, draw straight lines, and fill the inside of a plotted boundary with a single click (which is quite useful for instance segmentation). Unlike polygons in COCO etc., the annotation tool allows us to save higher-resolution binary masks.
We inserted a total of 13,372 questions. Following the common practice in instance segmentation, we did not label the crowd instances, as they are skipped in both training and inference. In the end, we obtained 12,836 valid instance mask annotations. It took a total of 800.59 rater hours. On average, raters spent 3.74 minutes on each valid mask.
In Fig. 5, we show some statistics about our Objects365 instance segmentation dataset. The number of masks per category follows a relatively long-tailed distribution (Fig. 5(a)). The number of masks per image reveals that a large number of images have many instance annotations (Fig. 5(b)). And the statistics of the relative mask size is close to that of the LVIS datasets (Fig. 5(c)).
As visualized in Fig. 6, the mask annotations are very high-quality. The raters carefully handled small objects, thin structures, complicated shapes, occluded objects, etc.
(a) Number of annotated instance masks per category. It follows a relatively long tail distribution.
(b) Number of annotated instance masks per image. More than 500 images have more than 10 instances.
(c) Relative segmentation mask size (square root of the ratio of mask area to image area), compared with COCO, LVIS, and ADE20K. Our statistics are similar to LVIS.
Figure 5: Dataset statistics for our Objects365 instance segmentation evaluation set.
Figure 6: Visualization of our Objects365 instance segmentation dataset. We store binary masks, which has higher mask quality compared with polygons. The quality on small objects are good (first two images). Complex structures are also labeled delicately, e.g., the camels (1st image on 2nd row) and the decoration on the cake (1st image on 3rd row). For occlusions, raters carefully avoided the occluded regions as they are not a part of the object to annotate, e.g., the fruits in the basket are not included in the basket masks (last image on 1st row).
For the objects which do not have an instance mask since raters skipped it, it's possible that the model still predicts an instance mask for that object. We include the skipped annotations to make it optional to ignore them during evaluation, so as to avoid treating such cases as false positive.
We release this dataset. We believe this effort can provide the research community a good benchmark for instance segmentation evaluation, e.g., for evaluating weakly-supervised instance segmentation learned from Objects365 bounding boxes.
C Open-vocabulary semantic segmentation on PASCAL VOC
We evaluate DaTaSeg on the PASCAL VOC 2012 semantic segmentation dataset [14], which includes 20 object classes and a background class. We directly transfer our trained model to its validation set with 1,449 images without finetuning. We follow the setting in MaskCLIP+ [76] and ignore the background class during evaluation.
Method	Backbone	mIoU	mIoU-unseen	mIoU-seen
SPNet [65]	56.9	0.0	75.8
SPNet-C [65]	63.2	15.6	78.0
ZS3Net [3]	61.6	17.7	77.3
CaGNet [17]	DeepLabv2-ResNet101	65.5	26.6	78.4
STRICT [51]	70.9	35.6	82.7
MaskCLIP+ [76]	88.1	86.1	88.8
Fully-supervised	88.2	87.0	88.6
DaTaSeg	ResNet-50	89.4	87.7	89.9
ViTDet-B	92.8	90.9	93.4
ViTDet-L	94.7	94.3	94.9
Table 6: Open-vocabulary semantic segmentation on the PASCAL VOC 2012 dataset. DaTaSeg outperforms all other zero-shot/open-vocabulary methods, and even the fully-supervised baseline. Comparison methods are trained on 15 seen classes of PASCAL VOC and only 5 classes are unseen (mIoU-unseen). In contrast, DaTaSeg has never seen the PASCAL VOC dataset during training (it's trained on several other datasets). We attribute DaTaSeg's performance improvement to our multi-dataset multi-task training. Following [76], the background class is ignored. The numbers of the comparison methods are from [76]. SPNet-C stands for SPNet with calibration.
Dataset sampling ratio	Fully-supervised	Weakly-supervised transfer
ADE semantic mIoU	COCO panoptic PQ	ADE semantic → panoptic PQ	O365 box → instance mask AP
1:4:4	48.1	49.0	29.8	14.3
1:2:2	46.8	48.6	29.1	12.8
1:1:1	45.3	48.0	28.4	13.7
Table 7: Ablation study on dataset sampling ratio (ADE:COCO:O365). Evaluated on a ResNet50 backbone. Results show that our adopted sampling ratio (1:4:4) is better than the other sampling ratios.
We compare with other zero-shot and open-vocabulary segmentation methods in Table 6, which are trained on 15 seen classes with 5 classes held out during training as unseen classes (potted plant, sheep, sofa, train, and TV monitor). DaTaSeg has never seen the PASCAL VOC dataset during training, though it is trained on other segmentation datasets (COCO panoptic, ADE semantic, and Objects365 detection). DaTaSeg with a ResNet-50 backbone outperforms all these methods with DeepLabv2-ResNet101 backbones. Surprisingly, DaTaSeg even outperforms the fully-supervised counterpart. We attribute DaTaSeg's performance improvement to our multi-dataset multi-task training. We note this is not an apple-to-apple comparison, since the training sources are different. The main purpose of this comparison is to show how our "data-centric" DaTaSeg is positioned against zero-shot/open-vocabulary segmentation methods on the PASCAL VOC dataset, using a direct transfer manner.
We perform more ablation studies in this section.
Ablation study on dataset sampling ratio: Our dataset sampling strategy (Sec. 3.5) avoids the imbalance of dataset sampling by specifying the per-dataset sampling ratio. The proportion of training samples coming from each dataset in the whole training process is determined by that sampling ratio. In our main results, the sampling ratio is 1:4:4 for ADE:COCO:O365. We ablate the dataset sampling ratio in Table 7. Results show that our adopted sampling ratio is better than the other sampling ratios.
Ablation study on the portion of Objects365 training data: Readers may have the question about the trade-offs between the quality and quantity of the training samples when we introduce the weakly-supervised training data (Objects365). In Table 2, we show the results with and without cotraining on Objects365. The performance is generally better when cotraining on Objects365. In addition, we experiment with cotraining on different portions of the dataset: we cotrain DaTaSeg on the full COCO panoptic and ADE semantic datasets, and  $10\% / 25\% / 50\% / 100\%$  of O365 training data. We show the results in Table 8. Results show that when increasing the number of O365 weakly-supervised training samples, O365 performance increases, COCO panoptic performance is not affected, and ADE20k performance slightly decreases. Overall, the gains are larger than the losses.