Title: DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: universal segmentation

multi-task segmentation

multi-dataset segmentation

panoptic segmentation

semantic segmentation

instance segmentation

weakly-supervised segmentation
Abstract: Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg. We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and release it as a public evaluation benchmark on https://laoreja.github.io/dataseg.

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model

Abstract

Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multida taset multita sk seg mentation model: DaTaSeg. We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and release it as a public evaluation benchmark on https://laoreja.github.io/dataseg .

1 Introduction

Image segmentation is a core computer vision task with wide applications in photo editing, medical imaging, autonomous driving, and beyond. To suit different needs, various forms of segmentation tasks have arisen, the most popular ones being panoptic [29], semantic [23], and instance [19] segmentation. Prior works generally use specific model architectures tailored to each individual task [28, 45, 21, 6]. However, these segmentation tasks are closely related, as they can all be regarded as grouping pixels and assigning a semantic label to each group. In this paper, we address the following question: Can we leverage a diverse collection of segmentation datasets to co-train a single model for all segmentation tasks? A successful solution to this problem would leverage knowledge sharing among datasets, boosting model performance across the board, especially on smaller datasets.

Existing works on unified segmentation models either focus on a single architecture to handle multiple tasks [10, 8, 74, 25], but with separate weights for different datasets; or a single set of weights for

multiple datasets [27, 33, 77], but on the same task. By contrast, in this work, we aim to train a single model on multiple datasets for multiple tasks. Our key idea of unifying these segmentation tasks is to use a universal intermediate mask representation: a set of mask proposals ( i.e ., grouped pixels) with class labels [63, 10]. Different segmentation tasks can be realized by applying different merge and post-processing operations on this unified representation. This allows us to train our network on the same output space for different tasks. Furthermore, using this representation, we can exploit weak bounding box supervision for segmentation, which is far cheaper to collect than mask annotations.

To encourage knowledge sharing and transfer among the multiple segmentation sources, our network architecture shares the same set of model weights across all datasets and tasks. In addition, we utilize text embeddings as the category classifier, which maps class labels from different datasets into a shared semantic embedding space. This design further enhances knowledge sharing among categories with similar meanings in different datasets, e.g ., 'windowpane' and 'window-other'. Our approach can be contrasted with an alternative of using dataset-specific model components - we show in our experiments that our simpler, unified approach leads to improved performance and enables open-vocabulary segmentation via simply switching the text embedding classifier.

Putting these techniques together, we propose DaTaSeg as shown in Fig. 1, a universal segmentation model, together with a cotraining recipe for the multi-task and multi-dataset setting. We train DaTaSeg on the ADE20k semantic [75], COCO panoptic [29], and Objects365 detection [56] datasets. We show that DaTaSeg improves performance on all datasets comparing with training separately, and significantly benefits relatively small-scale datasets

Panoptic

Semantic

Detection Dataset (box)

Universal

Multi-Dataset Multi-Task

Segmentation Model

Panoptic Segmentation Task

Semantic Segmentation Task

Instance Segmentation Task

DaTaSeg

(ADE20k semantic), outperforming a model trained only on ADE20k semantic by +6.1 and +5.1 mIoU with ResNet50 [22] and ViTDet-B [39] backbones. The multi-dataset multi-task setting also allows us to seamlessly perform weakly-supervised segmentation by transferring knowledge from other fully supervised source datasets, which we demonstrate on ADE20k panoptic and Objects365 instance segmentation. DaTaSeg also directly transfers to other datasets not seen during training. It outperforms open-vocabulary panoptic segmentation methods on the Cityscapes dataset [11] and performs comparably with open-vocabulary semantic segmentation works on the Pascal Context dataset [49] .

To summarize our contributions, we present DaTaSeg, a single universal segmentation model on multiple segmentation tasks and datasets. DaTaSeg leverages knowledge from multiple sources to boost performance on all datasets. It seamlessly enables weakly-supervised segmentation, directly transfers to other datasets and is capable of open-vocabulary segmentation. As an additional contribution, we have annotated a subset of the Objects365 validation set with groundtruth instance masks and release it as an evaluation benchmark for instance segmentation.

2 Related Work

Muti-dataset training: Training on multiple datasets has become popular for developing robust computer vision models [64, 78, 33, 47, 61, 67]. For object detection, Wang et al . [64] train an object detector on 11 datasets in different domains and show improved robustness. UniDet [78] trains a unified detector on 4 large-scale detection datasets with an automatic label merging algorithm. DetectionHub [47] employs text embeddings to accommodate different vocabularies from multiple datasets, and features dataset-specific designs. For segmentation, MSeg [33] manually merges the vocabularies of 7 semantic segmentation datasets, and trains a unified model on all of them; however, the manual efforts are expensive and hard to scale. LMSeg [77] dynamically aligns segment queries with category embeddings. UniSeg [27] explores the label relation and conflicts between multiple

datasets in a learned way. Most of these works merge datasets using a unified vocabulary on a single task, while our work focuses on a more challenging problem: we merge datasets with different vocabularies and different tasks.

Unified segmentation model: Panoptic segmentation [29] unifies semantic and instance segmentation. Prior works [28, 66, 69, 7, 52] exploit separate modules for semantic segmentation [45, 5] and instance segmentation [19, 21], followed by another fusion module. Recently, the mask transformer framework proposed by MaX-DeepLab [63] directly predicts masks with class predictions, allowing end-to-end panoptic segmentation. MaskFormer [10] and K-Net [74] adopt a single transformer-based model for different segmentation tasks. Mask2Former [8] improves upon MaskFormer by proposing masked attention, while kMaX-DeepLab [71] develops k-means cross-attention. OneFormer [25] extends Mask2Former with a multi-task train-once design. All these works still train separate weights on different datasets. By contrast, we aim at a single unified model that can perform well across multiple segmentation tasks and datasets.

Weakly-supervised segmentation: Facing the issue of expensive segmentation mask annotations, many works have proposed to learn segmentation masks from cheaper forms of supervision [50, 12, 42]. In particular, box-supervised approaches are most related to our work, including BoxInst [59], Box2Mask [38], Cut-and-Paste [54] and MAL [34] for instance segmentation, Box2Seg [32] and BCM [58] for semantic segmentation, and DiscoBox [35] and SimpleDoesIt [26] for both semantic and instance segmentation. Despite the popularity of box-driven semantic and instance segmentation, fewer attempts have been made at weakly-supervised panoptic segmentation. Li et al . [37] employ a pretrained model to generate pseudo-ground-truth in advance, in addition to weak supervision from bounding boxes and image level labels. Shen et al . [57] add a handcrafted branch on top of semantic and instance segmentation models to generate panoptic proposals. Unlike these methods that require customized components, our approach can realize knowledge sharing among datasets with different forms of annotations in a more systematic and data-centric fashion, and seamlessly enable weakly-supervised segmentation.

3 Method

3.1 A universal segmentation representation

Segmentation aims to group pixels of the same concept together. We propose to use an intermediate and universal representation, namely mask proposals, for all segmentation tasks. A mask proposal is a binary foreground mask with a class prediction. We use this representation for its versatility: one mask proposal can represent a single instance, any number of instances, a region, or even a part of an object. They can overlap and can be combined to form higher-level segmentation outputs an instance ('thing') or a region of the same semantic class ('stuff'). Thus they are well suited for the different flavors of segmentation considered in this paper. We note that a similar concept has been used in existing works [63, 10, 8, 71], for specific segmentation tasks on a single dataset. We show this representation is especially beneficial in our multi-dataset setting, as different datasets define 'thing' and 'stuff' categories differently. For example, 'table' is a thing category in ADE20k panoptic dataset, but there is a 'table-merged' stuff category in COCO panoptic. In our framework, both 'thing' and 'stuff' categories are represented using this same representation, and we treat them differently in the next step.

3.2 Merging predictions for specific tasks

Notations: We introduce the notations we use in the equations. ̂ ( · ) denotes predictions. Let (ˆ z j , ̂ M j ) denote the j -th mask proposal ̂ M j with its class prediction ˆ z j , where ̂ M j ∈ R H × W is the (presigmoid) logits for the mask proposal and ˆ z j,c k ∈ R is the logit for class c k . We assume a fixed set of N mask proposals. C d,thing and C d,stuff are the set of thing and stuff categories in dataset d .

Merge operation ( MERGE ): Given the varying formats of groundtruth annotations for different segmentation tasks, we propose a MERGE operation to merge the mask proposals predicting the same category c k . We adopt a simple element-wise max operation to merge the mask proposal logits. In particular, the merged mask proposal logits at the ( p, q ) location for class c k is computed as:

'person' proposal

'person' proposal

'horse' proposal

'building' proposal

'building' proposal

'person' 1

'person' 2

'horse' 1

'building'

'person'

Hungarian

Matching

Panoptic Segmentation

Semantic Segmentation

'horse'

'building'

'person' 1

'person' 2

'horse' 1

Weakly-Supervised Instance Segmentation

Hungarian

Matching

Hungarian

Matching

Cross-Entropy Loss

Focal Loss

Dice Loss

Cross-Entropy Loss

Focal Loss

Dice Loss

Cross-Entropy Loss

Projection Loss (box GT)

Universal

Representation

We choose element-wise max so that applying MERGE on raw mask logits is equivalent to applying MERGE on the soft mask predictions post-sigmoid. We also choose to merge at the level of raw logits for numerical stability. If no proposal predicts class c k , then the corresponding merged mask M ( c k ) p,q is set to -100 to ensure it is close to 0 after sigmoid.

In addition to merging masks, we merge their corresponding class predictions by simply averaging the class prediction logits of the proposals predicting class c k :

✶ where glyph[epsilon1] is a small number to prevent division by zero.

How we apply the above merge operations depends on the given task, as we now describe:

Panoptic segmentation: To make the predicted mask proposals have the same format as the groundtruth, we apply MERGE to all 'stuff' categories c k ∈ C d,stuff .

Semantic segmentation: In contrast with panoptic segmentation, the semantic segmentation groundtruth for each 'thing' category is also a single mask ( i.e ., 'thing' categories are treated equivalently as 'stuff' categories). We thus apply MERGE to all predicted mask proposals to cover all thing and stuff categories C d,stuff ∪ C d,thing .

Instance segmentation: MERGE is not needed in this task, since the desired outputs are separate masks for separate instances. There are no 'stuff' categories in the vocabulary ( C d,stuff = ∅ ), and therefore no stuff proposals.

Training: During training, we apply the corresponding MERGE operations based on the segmentation task. We then use one-to-one Hungarian matching [31] to match the merged outputs with the groundtruth, and calculate the training losses. Detailed training losses are provided in Sec. 3.5.

Inference: At inference time, we apply the same MERGE operations to predicted mask proposals based on the given task. We post-process the mask proposals to obtain non-overlapping outputs for panoptic and semantic segmentation. For instance segmentation, we simply take the top mask proposals as final outputs, since overlaps are allowed. Please refer to supplementary for more details.

We illustrate how the MERGE operation works for different segmentation tasks in Fig. 2.

3.3 Weakly-supervised instance segmentation

Bounding boxes are much cheaper to annotate than instance masks, and thus detection datasets can have a much larger scale than instance segmentation datasets. In order to train on larger detection datasets with bounding box annotations only, as well as to demonstrate the versatility of our framework to handle various tasks, we propose to perform weak-bounding-box-supervised instance segmentation.

dataset 2 (C 2

classes)

input image

Backbone

Pixel Decoder

N mask proposals

N segment

queries

Transformer Decoder

feature map

Class Head

Mask Head

sky

horse

learnable background

person

sky

horse

dataset 1 (C 1  classes)

person

trained Text

Encoder

N class predictions

sky

horse

person

dataset-specific

shared

We adopt a simple projection loss L proj from [60], which measures consistency of vertical and horizontal projections of the predicted masks ̂ M j against groundtruth boxes b j :

where L dice is the dice loss [48]; b j is the groundtruth bounding box matched to j -th mask proposal ̂ M j ; Proj x/y denotes the projection operation along the x or y axis, which can be implemented by a max operation along the axis; and S ( · ) denotes the sigmoid function.

By itself, a box consistency loss such as Eqn. 3 is insufficient as a supervision signal for segmentation (e.g., Eqn. 3 is equally satisfied by predicting the bounding box of an object instead of its mask). Thus other works have resorted to additional, often more complex, loss terms (such as the pairwise affinity loss from [60]). However, by training on multiple datasets and multiple segmentation tasks, these handcrafted losses are not necessary, as our model can transfer knowledge gained from other fully-supervised segmentation tasks on other datasets.

3.4 Network architecture with knowledge sharing

Network architecture: We now describe the network architecture (similar to [8]) that predicts mask proposal and class prediction pairs (ˆ z j , ̂ M j ) . The input image first goes through a backbone , and we use a pixel decoder to 1) output multi-scale image features I multi , and 2) output a high-resolution feature map F that fuses information from the multi-scale image features. N mask proposal pairs are then generated from N learnable segment queries Q s : The segment queries are fed into a transformer decoder , which cross attends to the multi-scale image features I multi from the pixel decoder. The N decoder outputs are then passed to class embedding and mask embedding heads (both MLPs) to obtain N (mask embedding, class embedding) pairs. To extract final mask proposals and class predictions, we apply a dot product between the high-resolution feature map F and the mask embeddings to obtain N mask proposal logits ̂ M j . And to obtain class predictions, we compute dot products of the class embeddings against a per-category classifier w k for each category c k . Fig. 3 shows an overview.

Knowledge sharing: We bake knowledge sharing into our model in two ways. First, all network parameters are shared across all datasets, allowing knowledge sharing among different datasets and knowledge transfer among different tasks. Additionally, to share knowledge gathered from training samples of similar categories across datasets, e.g ., 'windowpane' in ADE20k and 'window-other' in COCO, we propose to map all category names into a shared semantic embedding space . We feed the category names into a frozen pre-trained text encoder to set the per-category classifier w k . i.e ., we let the fixed text embeddings serve as classifiers. This is similar to open-vocabulary

segmentation [16, 70, 72], but with a different purpose: our focus here is on knowledge sharing among different datasets. This automatic approach scales better than manually unifying label spaces in different datasets [33].

3.5 Co-training strategy

We adopt a simple co-training strategy: at each iteration, we randomly sample one dataset, then sample the entire batch for that iteration from the selected dataset. This can be contrasted with sampling from multiple datasets in one iteration. The main advantage for our strategy is that it is simple to implement, and allows more freedom to use different settings for different datasets, including applying distinct losses to various tasks. To account for various dataset scales, we use per-dataset sampling ratios, which randomly samples each dataset at a dataset-specific ratio during training, similar to [41].

For fully-supervised tasks, we employ direct mask supervision , which can be obtained from panoptic/semantic/instance segmentation groundtruth. To calculate the Hungarian matching cost and the training loss, we use a combination of focal binary cross-entropy loss L focal [44], and dice loss L dice [48], following [10]. Regarding weak bounding box supervision , we adopt the projection loss L proj introduced in Sec. 3.3 for the matching cost and training loss. In both cases of supervision, we use the negative of the class prediction probability p as the classification matching cost [4], and use cross-entropy loss L ce for classification training. The total training loss is:

The Hungarian matching cost is defined similarly:

Here, λ ∗ ,d and µ ∗ ,d are the weights for dataset d .

4 Experiments

Datasets and metrics: We train and evaluate DaTaSeg on COCO panoptic [29] and ADE20k semantic [75] using mask supervision, as well as Objects365-v2 [56] detection datasets using bounding box weak supervision. COCO panoptic is the most popular panoptic segmentation benchmark with 118,287 training images and 5,000 validation images. COCO has 80 thing categories and 53 stuff categories. ADE20k semantic is one of the most widely used semantic segmentation benchmarks with 150 categories, 20,210 training images, and 2,000 validation images.

For evaluation, besides the training datasets, we also evaluate on ADE20k panoptic , which uses the same validation images as ADE20k semantic but with panoptic annotations. The original 150 categories are divided into 100 thing categories and 50 stuff categories. Finally, we train on the Objects365-v2 detection (bounding boxes only) dataset, which has 365 categories and 1,662,292 training images. To evaluate the weakly-supervised instance segmentation results, we manually label a subset of 1,000 images from the Objects365 validation set. Our annotation pipeline involves 20 welltrained human annotators, following the protocol in [2], without using any automatic assistant tools. Please refer to Appendix B for more details. We release this Objects365 instance segmentation evaluation set as a public benchmark.

To compare DaTaSeg with the state-of-the-art methods, we in addition evaluate on popular openvocabulary segmentation benchmarks. Following OpenSeg [16], we evaluate on PASCAL Context datasets [49] with 5k val images. We use its two versions: 59 classes ( PC-59 ) and 459 classes ( PC-459 ). We also evaluate on the 500-image val set of Cityscapes panoptic dataset [11], which focuses on urban street scenes with 11 stuff categories and 8 thing categories.

We report results in standard evaluation metrics: panoptic quality ( PQ ), mean intersection-overunion ( mIoU ), and mean average precision ( AP ) for panoptic, semantic, and instance segmentation, respectively.

Implementation details: We experiment with ResNet [22] and ViTDet [39] backbones. For ResNet, we use an ImageNet-1K [55] pretrained checkpoint, with a backbone learning rate multiplier of 0.1 [4]; the pixel decoder consists of a transformer encoder [62] and a modified FPN [43] following [10],

except that we use Layer Norm [1] and GeLU [24] activation for training stability. For ViTDet, we use the ImageNet-1K MAE pretrained checkpoint [20] with the layerwise lr decay [39]; the pixel decoder is the simple feature pyramid inside ViTDet, whose outputs are upsampled and added together to get the high resolution feature map F . The mask embedding head is a 3-layer MLP. The class embedding head contains a linear layer followed by ReLU. We use CLIP-L/14 [53] as the pretrained text encoder. We use 100 segment queries Q s , unless otherwise stated. See Appendix H for more details.

4.1 DaTaSeg improves over dataset-specific models

As an apples-to-apples comparison, we separately train a model on each dataset. We ensure that the models see the same number of images in cotraining and separately-training on each dataset. Table 1 (left) shows the results. DaTaSeg outperforms separately trained models on all datasets. Since we use exactly the same settings, we attribute the performance boost to our multi-dataset multi-task training recipe, which harnesses knowledge from multiple sources. Especially, DaTaSeg leads to a significant performance boost on ADE20k semantic: +6.1 mIoU and +5.1 mIoU with ResNet50 and ViTDet-B backbones, respectively. This proves our argument that cotraining on more data helps overcome the data limitation issue in smaller-scale segmentation datasets.

4.2 DaTaSeg enables weakly-supervised transfer

In Table 1 (right), we evaluate DaTaSeg on ADE20k panoptic and Objects365 instance segmentation tasks. Note we only have weak supervision (ADE20k semantic and Objects365 bbox) during training. Again, we compare to models separately trained on one dataset as our baselines. Specifically, we use the ADE semantic for ADE panoptic evaluation, and an Objects365 model trained only using weak box supervision for Objects365 instance evaluation, respectively.

By cotraining DaTaSeg in a multi-dataset multi-task fashion, we improve the ADE20k panoptic performance by +2.9 PQ and +5.4 PQ with ResNet50 and ViTDet-B backbones. Our design allows the panoptic segmentation knowledge to transfer from COCO to ADE20k. For reference, the fully-supervised performance is 37.7 PQ on ResNet50 and 42.3 PQ on ViTDet-B.

On Objects365, we only train with a weak box consistency loss L proj . The multi-dataset multitask training recipe improves the mask AP by +2.0 AP and +1.4 AP for the two backbones. The improvement is most likely from the instance segmentation knowledge in COCO panoptic.

4.3 DaTaSeg scales with size of backbones

Our DaTaSeg design is orthogonal to detailed network architectures, as long as it is able to output the universal representation (mask proposals with class predictions). This is supported by the consistent performance gains among ResNet50, ViTDet-B, and ViTDet-L backbones as shown in Table 1. It also scales well as we increase the sizes of the backbones. With ViTDet-L, DaTaSeg reaches 54.0 mIoU on ADE20k semantic and 53.5 PQ on COCO panoptic in a single model, without bells and whistles.

4.4 DaTaSeg scales with number of datasets

We study how DaTaSeg scales with the number of training datasets. We train DaTaSeg on all combinations of one, two, and three datasets among the ADE20k semantic, COCO panoptic, and Objects365 detection datasets, in order to conduct a comprehensive study, with a ResNet50 backbone. Table 2 presents the results. Looking at each column, we see that performance on each dataset generally improves as the number of training datasets increases, especially for ADE semantic.

Since DaTaSeg shares all parameters among all datasets and tasks, we can evaluate the cross-dataset transfer performance. We notice the model transfers to datasets that are not trained. e.g ., A model trained on COCO panoptic and Objects365 detection achieves 18.3 mIoU on ADE semantic, which is comparable to open-vocabulary segmentation performance (LSeg+ [16, 36] achieves 18.0 mIoU on a ResNet101 backbone).

4.5 DaTaSeg enables open-vocabulary segmentation

A further advantage of our fully-shared architecture is that we have the ability to directly transfer to other segmentation datasets. We simply switch the text embedding classifier with the vocabularies

(a) COCO panoptic

(b) ADE semantic

(c) ADE20k panoptic (weakly-supervised)

(d) Objects365 instance (weakly-supervised)

in a target dataset not used in training. Table 3 shows the results comparing DaTaSeg to several open-vocabulary works: ODISE [68] and MaskCLIP [13] are open-vocabulary panoptic segmentation methods and OVSeg [40] and OpenSeg [16] are open-vocabulary semantic segmentation approaches. Unlike these works, our method does not train on large-scale image-text data or use pretrained image-text models (except the pretrained text encoder), which puts our method at a disadvantage. Despite this disadvantage, DaTaSeg outperforms ODISE on the Cityscapes panoptic dataset. Note that there is a domain gap between our training sets and Cityscapes, which focuses on urban street scenes. DaTaSeg achieves comparable performance on PC-59 and PC-459. All methods in comparison train on COCO panoptic and ours has the best performance on COCO semantic. These results show that DaTaSeg enables open-vocabulary segmentation via our multi-dataset multi-task training approach.

4.6 DaTaSeg scales with number of queries

We perform ablation study on the number of segment queries Q s and show the results in Table 4. The performance improves as we increase the number of queries from 50 to 150, on almost all datasets. Besides, DaTaSeg achieves reasonably good performance with only 50 queries, which may benefit memory-limited application scenarios. We perform more ablation studies in Appendix D.

4.7 Adding dataset-specific modules hurts performance

In multi-dataset learning, prior works have introduced dataset-specific modules [47], in order to address inconsistencies across datasets [33, 61]. However, such a design may weaken knowledge sharing across different datasets and tasks, especially for weakly-supervised tasks that may rely more on other datasets and tasks to compensate for the weak supervision. To study this issue thoroughly, we carefully design several dataset-specific modules, including dataset-specific queries, dataset-specific heads, and dataset-specific prompts. These modules are lightweight by design so as to still encourage knowledge sharing through other shared parameters.

In Table 5, we evaluate the effectiveness of these dataset-specific modules by adding them to DaTaSeg, and keeping all other settings the same. Comparing with DaTaSeg that shares all parameters, results reveal that using dataset-specific modules hurts performance on all datasets except ADE20k semantic. Moreover, we see that the performance drops more on weakly-supervised tasks (ADE panoptic and O365 instance). This verifies that removing dataset-specific modules yields better knowledge sharing across datasets and tasks, and thus greatly improves performance. See Appendix E for more details.

4.8 Qualitative analysis

We show qualitative results (with ViTDet-L) in Fig. 4. DaTaSeg performs well in both fully and weakly supervised settings. We observe that localization quality is good in general, while classification is more error-prone. Nevertheless, DaTaSeg succeeds in challenging cases including transparent objects, occlusion, and crowd scenes.

5 Conclusion

The goal of our work is to train a single universal segmentation model on multiple datasets and multiple tasks ( i.e ., semantic, instance and panoptic segmentation). We present DaTaSeg, which leverages a universal segmentation representation, shared network parameters, and shared semantic embedding space for classification. DaTaSeg surpasses separately trained models, leading to significant gains on smaller datasets such as ADE20k semantic. It unlocks a new weakly-supervised transfer capability on datasets that were not explicitly trained for a particular task ( e.g ., ADE20k panoptic and Objects365 instance). DaTaSeg also directly transfer to more segmentation datasets and enables open-vocabulary segmentation. We believe that in the future, this multi-task multi-dataset setting will be the norm rather than the outlier as it currently is, and we see our paper as taking a step in this direction.

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 , 2016. 7

Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR , 2019. 6

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. NeurIPS , 2019. 16

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020. 6

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI , 2017. 3

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV , 2018. 1

Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and LiangChieh Chen. Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In CVPR , 2020. 3

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In CVPR , 2022. 1, 3, 5, 17, 20, 21

Bowen Cheng, Omkar Parkhi, and Alexander Kirillov. Pointly-supervised instance segmentation. In CVPR , 2022. 20

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NeurIPS , 2021. 1, 2, 3, 6, 17

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, UweFranke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , 2016. 2, 6

Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV , 2015. 3

Zheng Ding, Jieke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskclip. ICML , 2023. 8, 9

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 15

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR , 2021. 21

Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In ECCV , 2022. 6, 8, 9

Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In ACM MM , 2020. 16

Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR , 2019. 14, 21

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV , 2014. 1, 3

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022. 7

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 1, 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. 2, 6

Xuming He, Richard S Zemel, and Miguel Á Carreira-Perpiñán. Multiscale conditional random fields for image labeling. In CVPR , 2004. 1

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv:1606.08415 , 2016. 7

Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR , 2023. 1, 3

Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In CVPR , 2017. 3

Dongwan Kim, Yi-Hsuan Tsai, Yumin Suh, Masoud Faraki, Sparsh Garg, Manmohan Chandraker, and Bohyung Han. Learning semantic segmentation from multiple datasets with label shifts. In ECCV , 2022. 2

Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR , 2019. 1, 3

Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR , 2019. 1, 2, 3, 6

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 , 2023. 19

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 1955. 4

Viveka Kulharia, Siddhartha Chandra, Amit Agrawal, Philip Torr, and Ambrish Tyagi. Box2seg: Attention weighted loss and discriminative feature learning for weakly supervised segmentation. In ECCV , 2020. 3

John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In CVPR , 2020. 2, 6, 9, 17

Shiyi Lan, Xitong Yang, Zhiding Yu, Zuxuan Wu, Jose M Alvarez, and Anima Anandkumar. Vision transformers are good mask auto-labelers. In CVPR , 2023. 3

Shiyi Lan, Zhiding Yu, Christopher Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry S Davis, and Anima Anandkumar. Discobox: Weakly supervised instance segmentation and semantic correspondence from box supervision. In ICCV , 2021. 3

Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. ICLR , 2022. 8

Qizhu Li, Anurag Arnab, and Philip HS Torr. Weakly-and semi-supervised panoptic segmentation. In ECCV , 2018. 3

Wentong Li, Wenyu Liu, Jianke Zhu, Miaomiao Cui, Risheng Yu, Xiansheng Hua, and Lei Zhang. Box2mask: Box-supervised instance segmentation via level-set evolution. arXiv:2212.01579 , 2022. 3

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV , 2022. 2, 6, 7

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. CVPR , 2023. 8, 9

Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv:2111.12993 , 2021. 6

Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR , 2016. 3

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR , 2017. 6, 20

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017. 6

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , 2015. 1, 3

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR , 2019. 20

Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. CVPR , 2023. 2, 9

Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV , 2016. 5, 6

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR , 2014. 2, 6

George Papandreou, Liang-Chieh Chen, Kevin P Murphy, and Alan L Yuille. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In ICCV , 2015. 3

Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look at self-training for zero-label semantic segmentation. In CVPRW , 2021. 16

Siyuan Qiao, Liang-Chieh Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. In CVPR , 2021. 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 7

Tal Remez, Jonathan Huang, and Matthew Brown. Learning to segment via cut-and-paste. In ECCV , 2018. 3

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV , 2015. 6

Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV , 2019. 2, 6

Yunhang Shen, Liujuan Cao, Zhiwei Chen, Feihong Lian, Baochang Zhang, Chi Su, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Toward joint thing-and-stuff mining for weakly supervised panoptic segmentation. In CVPR , 2021. 3

Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. In CVPR , 2019. 3

Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: High-performance instance segmentation with box annotations. In CVPR , 2021. 3

Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen. Boxinst: High-performance instance segmentation with box annotations. In CVPR , 2021. 5

Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. The missing link: Finding label relations across datasets. In ECCV , 2022. 2, 9, 17

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017. 6

Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR , 2021. 2, 3

Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In CVPR , 2019. 2

Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR , 2019. 16

Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR , 2019. 3

Hang Xu, Linpu Fang, Xiaodan Liang, Wenxiong Kang, and Zhenguo Li. Universal-rcnn: Universal object detector via transferable graph r-cnn. In AAAI , 2020. 2

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Openvocabulary panoptic segmentation with text-to-image diffusion models. CVPR , 2023. 8, 9

Tien-Ju Yang, Maxwell D Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, and Liang-Chieh Chen. Deeperlab: Single-shot image parser. arXiv:1902.05093 , 2019. 3

Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions die hard: Openvocabulary segmentation with single frozen convolutional clip. NeurIPS , 2023. 6

Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV , 2022. 3

Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1020-1031, 2023. 6

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICLR , 2018. 21

Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurIPS , 2021. 1, 3

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , 2017. 2, 6

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. ECCV , 2022. 15, 16

Qiang Zhou, Yuang Liu, Chaohui Yu, Jingliang Li, Zhibin Wang, and Fan Wang. Lmseg: Language-guided multi-dataset segmentation. ICLR , 2023. 2

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In CVPR , 2022. 2

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR , 2021. 20

Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In CVPR , 2023. 19

Appendix

Funding in direct support of this work: Google LLC. There is no additional revenues related to this work.

A Broader Impacts

We propose a single framework for multi-dataset multi-task segmentation, which especially benefits memory-limited application scenarios, e.g ., our work can make a single segmentation model that meets all kinds of segmentation needs on memory-limited mobile devices (semantic segmentation for photo editing, instance segmentation for object detection and image search, etc .), instead of using separate models for various segmentation tasks.

Moreover, our approach improves segmentation performance from a data-centric view: We leverage segmentation data from multiple sources to improve the overall segmentation performance, especially on datasets of smaller scales, e.g ., ADE20k semantic. We also transfer knowledge from other sources to enable weakly-supervised segmentation. This is helpful on the scenario where segmentation data are very hard to collect, we can cotrain on similar data to help improve the performance, or even train on data with weaker supervision. e.g ., if we want to train a segmentation model on rare wild animals, we can co-train a model on a small set of these kinds of animals, and a larger dataset of more common animals.

As for bias and fairness, our model learns from the training data, so it may inherit the bias and fairness issues existing in the training data. Nonetheless, our model trains on multiple sources, and the fairer datasets may compensate the bias and fairness issues in other datasets.

B More about our newly labeled Objects365 instance segmentation evaluation dataset

We randomly sampled 1,000 images from the Objects365 V2 validation set. Different from the 'federated' LVIS dataset [18] (not all categories in each image are completely labeled), the bounding boxes in Objects365 are completely labeled for all categories and we follow this complete annotation in our instance masks. For each bounding box annotation, we generate a foreground/background annotation question for the raters. In the question, raters are asked to paint a semi-transparent instance mask on the original image, with the groundtruth bbox shown on the image as a guide and the groundtruth category displayed on the side. We do not crop the original image to the bbox region, but show the entire image to provide more context information, which is especially helpful for small objects. If the boundary is too blurry or too dark to annotate the instance mask, the raters can skip the question.

The annotation tool is a free painting tool, which allows the raters to freely draw the instance mask. We ask the raters to try to draw within the bbox, but if the object is obviously exceeding the bbox, then they can draw outside the bbox. The size of the stroke is adjustable. Raters can zoom-in/out, draw straight lines, and fill the inside of a plotted boundary with a single click (which is quite useful for instance segmentation). Unlike polygons in COCO etc ., the annotation tool allows us to save higher-resolution binary masks.

We inserted a total of 13,372 questions. Following the common practice in instance segmentation, we did not label the crowd instances, as they are skipped in both training and inference. In the end, we obtained 12,836 valid instance mask annotations. It took a total of 800.59 rater hours. On average, raters spent 3.74 minutes on each valid mask.

In Fig. 5, we show some statistics about our Objects365 instance segmentation dataset. The number of masks per category follows a relatively long-tailed distribution (Fig. 5(a)). The number of masks per image reveals that a large number of images have many instance annotations (Fig. 5(b)). And the statistics of the relative mask size is close to that of the LVIS datasets (Fig. 5(c)).

As visualized in Fig. 6, the mask annotations are very high-quality. The raters carefully handled small objects, thin structures, complicated shapes, occluded objects, etc .

10 3

0

50

100

150

200

250

300

350

sorted category index

10 0

10 1

10 2

number of instances

(a) Number of annotated instance masks per category. It follows a rel- atively long tail distribution.

0

200

400

600

800

1000

sorted image index

10 0

10 1

10 2

number of instances

0.0

0.2

0.4

0.6

0.8

relative segmentation mask size

10

4

10

3

10

2

10

1

10 0

percent of instances

COCO

LVIS

ADE20K

Objects365

(c) Relative segmentation mask size (square root of the ratio of mask area to image area), compared with COCO, LVIS, and ADE20K. Our statistics are similar to LVIS.

区

For the objects which do not have an instance mask since raters skipped it, it's possible that the model still predicts an instance mask for that object. We include the skipped annotations to make it optional to ignore them during evaluation, so as to avoid treating such cases as false positive.

We release this dataset. We believe this effort can provide the research community a good benchmark for instance segmentation evaluation, e.g ., for evaluating weakly-supervised instance segmentation learned from Objects365 bounding boxes.

C Open-vocabulary semantic segmentation on PASCAL VOC

We evaluate DaTaSeg on the PASCAL VOC 2012 semantic segmentation dataset [14], which includes 20 object classes and a background class. We directly transfer our trained model to its validation set with 1,449 images without finetuning. We follow the setting in MaskCLIP+ [76] and ignore the background class during evaluation.

We compare with other zero-shot and open-vocabulary segmentation methods in Table 6, which are trained on 15 seen classes with 5 classes held out during training as unseen classes (potted plant, sheep, sofa, train, and TV monitor). DaTaSeg has never seen the PASCAL VOC dataset during training, though it is trained on other segmentation datasets (COCO panoptic, ADE semantic, and Objects365 detection). DaTaSeg with a ResNet-50 backbone outperforms all these methods with DeepLabv2-ResNet101 backbones. Surprisingly, DaTaSeg even outperforms the fully-supervised counterpart. We attribute DaTaSeg's performance improvement to our multi-dataset multi-task training. We note this is not an apple-to-apple comparison, since the training sources are different. The main purpose of this comparison is to show how our 'data-centric' DaTaSeg is positioned against zero-shot/open-vocabulary segmentation methods on the PASCAL VOC dataset, using a direct transfer manner.

D Ablation studies

We perform more ablation studies in this section.

Ablation study on dataset sampling ratio: Our dataset sampling strategy (Sec. 3.5) avoids the imbalance of dataset sampling by specifying the per-dataset sampling ratio. The proportion of training samples coming from each dataset in the whole training process is determined by that sampling ratio. In our main results, the sampling ratio is 1:4:4 for ADE:COCO:O365. We ablate the dataset sampling ratio in Table 7. Results show that our adopted sampling ratio is better than the other sampling ratios.

Ablation study on the portion of Objects365 training data: Readers may have the question about the trade-offs between the quality and quantity of the training samples when we introduce the weaklysupervised training data (Objects365). In Table 2, we show the results with and without cotraining on Objects365. The performance is generally better when cotraining on Objects365. In addition, we experiment with cotraining on different portions of the dataset: we cotrain DaTaSeg on the full COCO panoptic and ADE semantic datasets, and 10% / 25% / 50% / 100% of O365 training data. We show the results in Table 8. Results show that when increasing the number of O365 weakly-supervised training samples, O365 performance increases, COCO panoptic performance is not affected, and ADE20k performance slightly decreases. Overall, the gains are larger than the losses.

attention

mask

V

K

image

features

Q

Learnable Positional Embedding

Masked Cross-Attention

V

K

Q

Add & Norm

Self-Attention

Add & Norm

Feed Forward Network

Add & Norm

mask & class predictions

cross-attention weights

self-attention weights

segment queries

dataset-specific queries

Ablation study on projection loss L proj weights: Our settings of λ and µ (Table 13) closely follow Maskformer series [10, 8]. We add the the bounding-box projection loss L proj for Objects365. We run an ablation study on the weights ( λ proj and µ proj ) for the projection loss in Table 9. The results indicate that the 2nd setting (our final setting) has a better performance on all datasets. Finding the optimal weights for multiple losses is an interesting yet challenging problem, and we leave it for future work.

E Adding dataset-specific modules hurts performance

In this section, we first introduce our designed dataset-specific modules and then show more ablation studies on these modules.

As prior works have pointed out [33, 61], there are many inconsistencies across segmentation datasets, including labels having their own particular definition, being annotated using different protocols, etc . We explore if adding dataset-specific modules can improve overall performance by designing the following modules.

Dataset-specific queries: In our architecture design introduced in Sec. 3.4, an input image will not get any dataset-specific information until the text embedding classifier, which is almost at the final stage of the network. We thus propose to add a few dataset-specific queries Q d as an auxiliary input to the beginning of transformer decoder. Each training dataset d has its own Q d . The dataset-specific

dataset2 (C 2

classes)

input image

Backbone

Pixel Decoder

dataset-specific

queries

Transformer Decoder

feature map

Class Head

Mask Head

sky

horse

learnable background

person

sky

horse

dataset1 (C 1

classes)

person

trained Text

Encoder

dataset-specific

shared

N mask proposals

N class predictions

sky

horse

person

N segment

queries

queries are treated the same as regular segment queries Q s . The only difference is that we do not make mask and class predictions for the dataset-specific queries. Fig. 7 shows the details. In the cross-attention layer, Q d cross attends to the image features - the cross-attention weight is:

where I multi are the multi-scale image features in Sec. 3.4. W q , W k denote the linear projection weights; A denotes the attention mask.

In the self-attention layers, there are interactions for all pairs of Q d and Q s , allowing the segment queries to be aware of the dataset-specific information at the beginning of transformer decoder. This can be shown by the self-attention weight calculation:

Dataset-specific classification head and background classifier: We observe that dataset inconsistency tends to primarily be a classification issue, rather than localization issue. Hence, for all datasets we use the same set of parameters for localization (backbone, pixel decoder, transformer decoder, mask embedding head) to enhance knowledge sharing. For classification, we design some dataset-specific components: specifically, we use a dataset-specific MLP class embedding head as well as a learnable dataset-specific background classifier, since the definitions of background vary from dataset to dataset.

Overall dataset-specific architecture: The overall architecture equipped with all our datasetspecific modules is shown in Fig. 8. We design the dataset-specific modules to be light-weight, which allows us to save on memory costs. With this design of a mixture of a mostly shared network and light-weight dataset-specific modules, the model has the freedom to choose whether to leverage dataset-specific information, or to use the shared knowledge across datasets/tasks. However, one significant disadvantage of adding dataset-specific modules is that it's hard to decide which set of dataset-specific parameters to use when directly transferring to other datasets (Sec. 4.5), and thus it hurts the open-vocabulary capability .

Ablation studies on dataset-specific modules: We ablate on the number of dataset-specific queries Q d in Table 10. It shows that removing the dataset-specific queries achieves the best results on COCO panoptic and Objects365 by a large margin. It also achieves reasonable performance on other datasets. Using 30 queries hurts the performance on COCO panoptic and ADE semantic.

In Table 11, we compare using and not using dataset-specific classification modules (the classification embedding head and background classifier). Results indicate that removing dataset-specific classification improves performance on all datasets & tasks, especially the weakly-supervised tasks. We suspect it's because weakly-supervised tasks rely more on knowledge sharing.

System-level comparison between with and without dataset-specific modules: In the main paper, due to space limit, we only show results of adding all dataset-specific modules on a ResNet50 backbone. In Table 12, we show the same comparison on more backbones, i.e ., ResNet50, ViTDet-B, and ViTDet-L. The observation is consistent across all backbones: adding dataset-specific modules hurts performance on all datasets. The difference is more significant on backbones of smaller scales, and on COCO panoptic and Objects365 instance segmentation datasets.

F Comparison with related contemporary work

We discuss the differences between DaTaSeg and some contemporary related work as below.

Comparison with Segment Anything [30]: SAM [30] is a related work on multi-task universal segmentation model. There are several major differences between DaTaSeg and SAM. 1) The main use-case for SAM is prompt-based segmentation, i.e., the user inputs a point or box, and the model segments the object referred by the prompt. While we aim to segment all objects and stuff in a bottom-up way. 2) Besides, SAM built their own large-scale dataset with more than 1 billion masks (SA-1B) by designing a data engine consisting of model-assisted manual annotation and semi-automatic stage, which is very costly; while we only utilize multiple publicly available datasets and train a joint segmentation model to improve the performance. Hence, DaTaSeg and SAM's results are not comparable in terms of the training data. 3) Except for the explicitly trained text-to-mask task, SAM's outputs are class-agnostic binary masks (the SA-1B datasets are also class-agnostic), while our model outputs the class predictions together with the mask predictions.

Comparison with X-Decoder [80]: X-Decoder is a related work on multi-task segmentation. There are multiple differences between our work and X-Decoder, which makes it hard to have a strict comparison. We detail the differences below: 1) Task and Focus : The pre-training in X-Decoder includes panoptic segmentation, referring segmentation, and image-text pairs (image-text retrieval and image captioning), which has a focus on vision-language tasks. These tasks are very different from ours: we cotrain on three mainstream segmentation tasks (panoptic/semantic/instance) and exclusively focus on segmentation. 2) Training paradigm : We directly cotrain on multiple datasets using a shared set of parameters (single model), while the ADE and COCO results in X-Decoder Table 1 is task-specific transfer. That is, X-Decoder first pretrains on large-scale data and then fine-tunes on each target dataset using different sets of fine-tuned parameters. 3) Training data :

X-Decoder pretrains on COCO panoptic and referring segmentation and 4M image-text pairs with a long schedule, and we quote from their paper: 'all the pre-trained models are trained with 50 epochs of COCO data and roughly 45 epochs of 10 million image-text pairs'. Afterwards, they fine-tune the model on COCO and ADE20k. By contrast, we only train on COCO panoptic, ADE20k semantic, and Objects365 v2 detection with a total of 1.8M of training images, which is much fewer than X-Decoder. We also do not train on large amounts of text data. 4) Model architecture : X-Decoder adopts Mask2Former. Our model architecture is different as explained in Sec. G. More importantly, we don't have an online text encoder in our model architecture.

G Architecture change from Mask2Former

Our network architecture is based on Mask2Former [8]. However, due to hardware difference (TPU vs. GPU), we need to make modifications to Mask2Former architecture to run on our hardware. Unlike GPUs, TPUs require a static computation graph with fixed-shape data (otherwise, it recompiles for each computation graph change, and causes significant slowdown). Therefore, we drop all the TPU-unfriendly operations in Mask2Former. In particular, we change the multi-scale deformable attention transformer [79] pixel decoder to a plain FPN [43]. The performance comparison is shown in Table 4(e) in the Mask2Former paper (51.9 PQ of deformable-attention vs. 50.7 PQ of FPN on COCO panoptic). Also, we do not use the sampled point loss [9] in either the matching loss or the training loss calculation, and use the vanilla mask loss. The performance comparison is shown in Table 5 in the Mask2Former paper (51.9 PQ of point loss vs. 50.3 PQ of mask loss). Besides, we do not use varying input size during evaluation and use a fixed input size (first resize then pad).

To improve the training speed, we modify the deep supervision in the masked attention transformer decoder: for every decoder layer, we use the same matching indices computed from the outputs of the last decoder layer, but use a different prediction head. We do not use the mask predictions from the initial queries as attention masks, since the predictions are not made based on the input image.

H Additional implementation details

In addition to the implementation details described in Sec. 4, we provide more datails here.

Preprocessing. In order to do 1:1 Hungarian matching, we preprocess the groundtruth into the same format as the predictions described in Sec. 3.2. In panoptic segmentation , we transfer the groundtruth into a set of binary masks with class labels. For 'thing' categories, we group the pixels belonging to each instance into a separate groundtruth mask; for 'stuff' categories, we group all pixels in each stuff category as the groundtruth mask. Similarly, in semantic segmentation , we group all pixels in each category as the groundtruth mask. For instance segmentation , since we are using bounding box weak supervision, we do not need this preprocessing step. For all mask proposal groundtruth, we downsample it by 4 times, to save memory cost during training.

Training settings. We randomly scale the input image in the range of [0.1, 2.0] and then pad or crop it to 1024 × 1024 . For ADE20k dataset, since the image size is smaller than other datasets, we use a scaling range of [0.5, 2.0]. We use the AdamW optimizer [46] with a weight decay of 0.05. We

clip the gradients with a max norm of 0.1. The weight for the background class is set to 0.05 in L ce . The matching cost and loss weight settings for Eqn. 4,5 in the main paper are shown in Table 13. We use a dataset sampling ratio of 1:4:4 for ADE semantic, COCO panoptic, and Objects365 detection. We adopt a different learning rate multiplier for each dataset: We multiply the learning rate on ADE semantic, COCO panoptic, and Objects365 detection by 3, 5, 2, respectively. Since Objects365 detection dataset has a large vocabulary with imbalanced distribution, we apply repeat factor sampling with a frequency threshold t = 0 . 01 [18]. On ResNet50 backbones, we use a batch size of 384 and train 500k iterations, with a learning rate of 3e-5. We adopt the step learning rate schedule: We multiply the learning rate by 0 . 1 × at the 0.9 and 0.95 fractions of the total training iterations. On the ViTDet-B backbones, we train 600k iterations with a learning rate of 6e-5. On ViTDet-L, we use a batch size of 256 and train 540.5k iterations with a learning rate of 4e-5. For ResNet50, we train on 64 TPU v4 chips; for ViTDet backbones, we train on 128 TPU v4 chips. All evaluations are conducted on 4 V100 GPUs with a batch size of 8.

Postprocessing. We first apply the MERGE operation described in Sec. 3.2. We follow the postprocessing operations in [8] with some modifications: In panoptic segmentation, we use a confidence threshold of 0.85 to filter mask proposals for COCO panoptic, and set the threshold to 0.8 for ADE panoptic. For panoptic and instance segmentation, we filter out final segments whose area is smaller than 4. For instance segmentation, we return a maximum of 100 instances per image with a score threshold of 0.0. For all tasks, the final scores are the production of classification scores and localization scores (obtained from binary mask classification).

I Limitations

Our proposed method is not without limitations. Since we are the first work to explore universal multi-dataset multi-task segmentation models, we do not introduce the complexity for an efficient framework. There are several ways to improve the model efficiency, e.g ., calculating the mask loss on sampled points only [8], while we do not deploy this in our framework, since it's not our key contribution. Moreover, as shown in the experiment section (Table 1), our framework is orthogonal to the detailed network architectures as long as the network is able to output our universal segmentation representation.

For weakly-supervised panoptic segmentation on ADE20k panoptic, we notice that sometimes multiple instances of the same category are not separated in the prediction. In the weakly-supervised instance segmentation results on Objects365, there are some visual artifacts in the mask prediction. We believe using certain data augmentation techniques, e.g ., MixUp [73] or Copy-Paste [15], may farther enhance knowledge transfer between datasets of different tasks, and thus may help mitigate this issue. We leave it for future work to improve on these limitations.

J Additional qualitative results

We notice sometimes our predictions do not match the groundtruth, but it does not necessarily mean the predictions are not good. Fig. 9 shows two examples on COCO panoptic. On the left column, DaTaSeg does a better job in segmenting 'sky' and 'tree-merged' than the groundtruth. The classification of 'pavement-merged' is also better than 'dirt-merged, which is probably due to language ambiguity : the British meaning of 'pavement' is sidewalk, while the annotator may be more used to the North American usage of 'pavement'. On the right column, DaTaSeg predicts 'window-other', while the groundtruth is 'tree-merged' for the scene through the window, and both make sense due to label ambiguity . In addition, DaTaSeg is able to predict the 'mirror-stuff', 'shelf', 'light', 'bottle', and more 'book's, which are missing from the groundtruth.

We show qualitative results for the direct transfer experiments in Fig. 10,11,12,13, on PASCAL Context 59 (PC-59), PASCAL Context 459 (PC-459), COCO semantic, and Cityscapes panoptic

(a) DaTaSeg predictions

background

bottle

background

chair

STOP

truck

potted plant

bed

stop sign

book

blanket

bench

light

mirror-stuff

shelf

tree-merged

window-blind

sky-other-merged

window-other

cabinet-merged

pavement-merged

wall-other-merged

grass-merged

rug-merged

background

background

STOP

car

chair

truck

pottedplant

stop sign

bed

road

book

tree-merged

blanket

sky-other-merged

house

dirt-merged

tree-merged

(b) Groundtruths

datasets, respectively. These results serve as a supplementary material for Table 3 in the main paper. They show DaTaSeg directly transfer to other segmentation datasets unseen during training with high quality both in localization and classification. DaTaSeg performs well on large-vocabulary segmentation (PC-459), and handles hard cases well (thin structures, small objects, complicated scenes, occlusions, etc .).

aeroplane

aeroplane

sky

sky

tree

tree

aeroplane

ignore

car

aeroplane

grass

car

ground

grass

road

ground

sky

sign

tree

road

truck

sky

water

tree

bottle

ignore

chair

bottle

diningtable

chair

diningtable

pottedplant

pottedplant

book

book

cabinet

cabinet

floor

cloth

curtain

floor

tree

bedclothes

wall

wall

window

window

bicycle

bicycle

bottle

person

person

building

building

fence

flower

flower

grass

grass

sidewalk

sidewalk

sky

tree

tree

bowl

ignore

chair

bowl

cup

candle

floor

chair

flower

floor

fork

flower

frame

fork

knife

knife

plate

picture

pottedplant

plate

spoon

table

table

wall

wall

wineglass

bottle

ignore

chair

bottle

cup

cup

floor

food

food

fork

fork

knife

knife

patio

plate

plate

table

table

wineglass

wineglass

bench

bench

building

building

car

car

6op

6op

fence

fence

grass

ground

frame

person

grass

pole

ground

road

person

sidewalk

sky

pole

tree

sky

truck

tree

bicycle

bicycle

bottle

building

building

grass

grass

mountain

mountain

person

person

pole

pole

road

road

shoe

sky

tree

tree

bottle

cup

ignore

chair

pottedplant

bed

chair

三

book

blanket

flower

potted plant

light

mirror-stuff

bed

shelf

wall-wood

towel

book

window-blind

window-other

blanket

cabinet-merged

table-merged

floor-other-merged

house

paper-merged

wall-other-merged

rug-merged

tree-merged

person

boat

car

person

ignore

backpack

bird

boat

bird

cell phone

backpack

bridge

house

handbag

river

cell phone

sand

roof

bridge

water-other

gravel

fence-merged

tree-merged

river

ceiling-merged

tree-merged

sky-other-merged

sky-other-merged

pavement-merged

grass-merged

pavement-merged

dirt-merged

building-other-merged

wall-other-merged

rock-merged

wall-other-merged

rock-merged

tv

ignore

laptop

laptop

keyboard

cell phone

keyboard

light

light

window-blind

cabinet-merged

table-merged

table-merged

paper-merged

paper-merged

wall-other-merged

wall-other-merged

car

truck

ignore

banner

bridge

car

house

truck

tree-merged

fence-merged

road

sky-other-merged

tree-merged

pavement-merged

mountain-merged

grass-merged

sky-other-merged

dirt-merged

building-other-merged

building-other-merged

sports ball

person

ignore

wine glass

bottle

person

cup

wine glass

couch

dining table

dn3

remote

curtain

floor-wood

door-stuff

remote

light

floor-wood

stairs

window-other

window-other

tree-merged

fence-merged

table-merged

ceiling-merged

floor-other-merged

table-merged

paper-merged

food-other-merged

wall-other-merged

wall-other-merged

rug-merged

rug-merged

vegetation

sidewalk

bupig

ignore

fence

road

traffic sign

sidewalk

building

ignore

fence

road

pole

sky

vegetation

person

car

sky

e

bicycle

bicycle

bulldir

vegetatio

perse

ignore

road

sidewalk

building

fence

wall

traffic ight

traffic sign

pole

terrain

sky

person

bus

motorcycle

ignore

road

sidewalk

ignore

road

sidewalk

building

trafficlight

6upinq

wall

trafficlight

traffic sign

fence

pole

person

bus

car

vegetation

person

sky

car

motorcycle

motorcycle

bus

bicycle

bicycle

Peer reviews:
comments: SUMMARY:
This paper proposes the universal model for image segmentation using multi-dataset multi-task training. By using universal segmentation representations at the entity (thing or stuff) level, the paper use merge operation for different segmentation task. Experiments show the effectiveness of the proposed method.

STRENGTHS:
1, The paper is well-written and easy to understand.

2, The extensive experiments show the effectiveness of multi-dataset multi-task training.

WEAKNESSES:
1， Limited novelties. The main contribution of this paper is multi-dataset multi-task training. However, there is no obvious difference to OneFormer except for the multi-dataset training. To solve the label space conflict of multi-dataset training, the paper directly uses the language embeddings of the language model that has been widely explored.

2, Merging universal segmentation representations seems like following the fine-to-coarse pipeline. There is no obvious clue to prove whether this fine-to-coarse is better than the coarse-to-fine pipeline.

3, Weakly-supervised instance segmentation module also has limited novelties by just using the projection loss of BoxInst to add the larger dataset objects365.

QUESTIONS:
1, Could the author try the other sampling strategy in multi-dataset training? In the paper, it would lead to the imbalance of dataset sampling. 

2, The performance improvement on COCO mainly comes from using the objects365 dataset. Direct pretraining object365 and then finetuning to COCO could perform better than the proposed method in the paper.

LIMITATIONS:
This paper cannot solve the annotation inconsistency of multi-dataset training. For example, the `window' defined in COCO and ADE20K are different where COCO stuff but ADE20K thing.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

comments: SUMMARY:
This paper proposes DaTaSeg, a universal multi-dataset multi-task segmentation model. DaTaSeg uses a shared representation and different merge operations and post-processing for different tasks. Weak-supervision is employed for cheaper bounding box annotations and knowledge is sharing across different datasets with text embeddings from the same semantic embedding space and shared network parameters. A subset of the Objects365 validation set is annotated for instance segmentation. Experiments shows that DaTaSeg gets improved performance on dataset-specific models and enables weakly-supervised knowledge transfer. DaTaSeg also scales with the training dataset number and enables open-vocabulary segmentation.

STRENGTHS:
1) This paper is easy to follow.
2) The method proposed in this paper will be useful in the future as it achieves the segmentation task under the multi-task multi-dataset setting.

WEAKNESSES:
The ablation study in the paper is somewhat not sufficient. The hyperparameters λ in equation (4) and μ in equation (5) are not provided and there needs to be experiments on the impact of hyperparameters.

QUESTIONS:
This paper proposes the multi-dataset multi-task segmentation model. The hyperparameters λ and μ in the equations (4) and (5) are crucial for the training. Could you provide the experiments on the impact of hyperparameters on the model performance?The authors do not discuss Limitations in the paper.

LIMITATIONS:
The authors do not discuss Limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n[Tasks] This paper introduces DaTaSeg, a universal multi-dataset multi-task segmentation model that addresses the interconnections between panoptic, semantic, and instance segmentation tasks. \n\n[Methods] DaTaSeg utilizes a shared representation, consisting of mask proposals with class predictions, across all tasks. To handle task discrepancies, the model employs distinct merge operations and post-processing techniques tailored to each task. Additionally, DaTaSeg leverages weak supervision, enabling cost-effective bounding box annotations to enhance the segmentation model. To facilitate knowledge sharing across datasets, DaTaSeg utilizes text embeddings from the same semantic embedding space as classifiers and shares all network parameters among datasets. \n\n[Experiments] The model is trained on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg exhibits improved performance across all datasets, particularly for small-scale datasets, achieving a 54.0 mIoU on ADE semantic and a 53.5 PQ on COCO panoptic. Furthermore, DaTaSeg enables weakly-supervised knowledge transfer for ADE panoptic and Objects365 instance segmentation.\n\n[Results] Experimental results indicate that DaTaSeg scales effectively with the number of training datasets and facilitates open-vocabulary segmentation through direct transfer. \n\n[Dataset] Additionally, the authors have annotated an Objects365 instance segmentation set comprising 1,000 images, which will be released as a public benchmark.\n\nSTRENGTHS:\n1. The authors have conducted comprehensive experiments, showcasing the state-of-the-art performance achieved on multiple long-tailed recognition benchmarks. This highlights the robustness and effectiveness of their proposed method.\n\n2. The paper is skillfully organized, making it easy to follow. The logical structure and clear presentation enhance the reader's understanding of the research.\n\n3. Efficient Use of Supervision. While previous works have explored training universal models on multiple datasets and tasks, a notable strength of this paper is the effective utilization of weak bounding box supervision for segmentation. Compared to full mask annotations, weak bounding box supervision is a more cost-effective and practical solution. This approach makes the proposed method more accessible and applicable in real-world scenarios.\n\nWEAKNESSES:\n[Technical contributions on multi-task multi-task model.] One of the key contributions of this paper is the joint training of multiple datasets and multiple tasks within a unified framework. However, it should be noted that this aspect has already been explored by [1]. The referenced work employs a single set of parameters pre-trained for Semantic/Instance/Panoptic Segmentation, Referring Segmentation, Image Captioning, and Image-Text Retrieval tasks. Therefore, it might be necessary for the authors to clarify the differences between this work and X-Decoder. \n\n[Technical contributions on text embedding classifier.] The utilization of a text embedding classifier has also been explored in previous works, such as [2]. In this work, an image encoder is trained to encode pixel embeddings, while CLIP text embeddings are employed as the per-pixel classifier. The key ideas are quite similar, although there are some differences on how to better leverage the text embeddings. \n\n[Lack of comparisons with some published works and subpar performance compared to state-of-the-art methods.] The paper primarily compares the model's performance against methods designed or trained solely on a single task, referred to as \"Separate\" in the paper. However, it is worth noting that DaTaSeg benefits from training on a significantly larger sample size compared to these listed works. Consequently, the observed performance gains over the \"Separate\" models are not unexpected. Furthermore, there are published works, such as X-Decoder [1], that jointly train models on multiple datasets and tasks, yielding superior performance on many benchmarks compared to the reported results in this paper.\n\nOn ADE semantic (mIoU): X-Decoder achieves 58.1 compared to DaTaSeg's 54.0; on COCO Panoptic (PQ): X-Decoder outperforms DaTaSeg with 56.9 versus 53.5. In most benchmarks, DaTaSeg performs worse than X-Decoder.\n\n[1] \"Generalized decoding for pixel, image, and language.\" Zou, Xueyan, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai et al. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15116-15127. 2023.\n[2] \"Language-driven Semantic Segmentation.\" Boyi Li and Kilian Q Weinberger and Serge Belongie and Vladlen Koltun and Rene Ranftl. International Conference on Learning Representations. 2022\n\nQUESTIONS:\nMost of my questions are listed in the Weaknesses section, with my main concerns focused on the technical contributions and performance comparisons with some published works. However, I have a couple of minor questions:\n\n1. Will the code and model weights be made available to the public? It would be valuable to have access to these resources for further exploration and replication of the proposed method.\n\n2. Given that this model can be trained with weakly-supervised tasks, I'm curious if further improvements in performance can be achieved by training on a larger quantity of weakly-supervised samples. Are there any trade-offs between the quality and quantity of the samples when it comes to enhancing model performance?\n\nI believe addressing these questions would provide additional insights into the practicality and potential improvements of the proposed method.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes DaTaSeg, a general multi-dataset multi-task segmentation model. It utilizes shared representations and different pooling operations to perform panoramic, semantic and instance segmentation tasks. DaTaSeg benefits from weak supervision and knowledge transfer across datasets. It outperforms separate training on all datasets (especially smaller ones) and enables weakly supervised segmentation. The model also transfers well to unseen datasets and supports open vocabulary segmentation.

STRENGTHS:
The paper addresses the challenge of training a single model on multiple segmentation tasks and datasets by proposing DaTaSeg, which has the potential to save computational resources and streamline the development of segmentation models. It can also benefit from weak supervision by incorporating cheaper bounding box annotations.
The proposed DaTaSeg shows promising results in transferring to unseen datasets and enabling open-vocabulary segmentation.
The authors annotate a subset of the Objects365 dataset and plan to release it as a public benchmark for instance segmentation. This contributes to the research community by providing a standardized evaluation dataset.

WEAKNESSES:
The Segment Anything model (SAM) released in April  2023 should be included for comparison as it's also a universal segmentation model which has zero/few-shot capability.

QUESTIONS:
May need proofreading:

Line 35, "which are" -> "which is"
Line 38, "which map" -> "which maps"

LIMITATIONS:
According to the F section in Supplementary Material, the computational cost is quite high which may affect the reproducibility. Consider releasing the pre-trained models in various sizes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DaTaSeg, a universal multi-dataset multi-task segmentation model. It uses a shared representation for panoptic, semantic, and instance segmentation tasks, with different techniques to address task differences. Weak supervision and knowledge sharing are employed. DaTaSeg improves performance on various datasets, especially small-scale ones. It enables knowledge transfer and open-vocabulary segmentation. An Objects365 instance segmentation set of 1,000 images will be released as a benchmark.\n\nSTRENGTHS:\n1. It is noticed that training segmentation models on multiple datasets to obtain better results is difficult. It is more challenging to train segmentation models on multiple datasets for multiple tasks (instance segmentation, semantic segmentation, weakly supervised instance segmentation. The performance of DaTaSeg on COCO and ADE shows the effectiveness of the proposed approach\n2. The paper leverages box-level supervision to improve the segmentation performance.\n\nWEAKNESSES:\n1. [minor] The definition of mask proposal is commonly used in instance segmentation [1]. However, the description uses too many math terms to describe a simple concept that can be demonstrated by natural language, figures, and more compact math. This is not good for readers to understand.\n\n2. [major] The results of weakly supervised instance segmentation look not promising. In the previous state-of-the-art method, weakly supervised instance segmentation methods achieves 70-90% performance (mAP) as what their fully supervised versions do [2,3]. However, in the 2, the instance segmentation of Object365 looks not promising, only 10ish%. What is the upper-bound performance of O365 instance segmentation?\n\n3. [major] The approach introduces the unified mask representation without comparison or discussion with other mask representations. It is well-known that training multi-dataset or multi-tasks is not trivial, and it is better to show more training details and empirical design considerations in the paper. \n\n\n\n[1] run-length encoding: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L265\n[2] Lan, Shiyi, et al. \"Vision transformers are good mask auto-labelers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[3] Li, Wentong, et al. \"Box2Mask: Box-supervised Instance Segmentation via Level-set Evolution.\" arXiv preprint arXiv:2212.01579 (2022).\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}