DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
Authors: Xiuye Gu
Yin Cui
Jonathan Huang
Abdullah Rashwan
Xuan Yang
Xingyi Zhou
Golnaz Ghiasi
Weicheng Kuo
Huizhong Chen
Liang-Chieh Chen
David A Ross
Venue: NeurIPS 2023 poster (2023)
Keywords: universal segmentation
multi-task segmentation
multi-dataset segmentation
panoptic segmentation
semantic segmentation
instance segmentation
weakly-supervised segmentation
TL;DR: we propose to train a universal segmentation model, DaTaSeg, on multiple datasets to perform multiple segmentation tasks; it enhances segmentation performance by harnessing data from multiple sources.

Abstract: Observing the close relationship among panoptic, semantic and instance segmentation tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg. We use a shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge operations and post-processing for different tasks. We also leverage weak-supervision, allowing our segmentation model to benefit from cheaper bounding box annotations. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mIoU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and Objects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmentation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and release it as a public evaluation benchmark on https://laoreja.github.io/dataseg.

DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model
Xiuye Gu
Yin Cui+t
Jonathan Huang"
Abdullah Rashwan"
Xuan Yang"
Xingyi Zhou"
Golnaz Ghiasi
Weicheng Kuo
Huizhong Chen
Liang-Chich Chen
David Ross
Google Research
Abstract
Observing the close relationship among panoptic, semantic and instance segmenta- tion tasks, we propose to train a universal multi-dataset multi-task segmentation model: DaTaSeg. We use shared representation (mask proposals with class predictions) for all tasks. To tackle task discrepancy, we adopt different merge oper- ations and post-processing for different tasks. We also leverage veak-supervision, allowing our segmentation model to benefit from cheaper bounding box annota- tions. To share knowledge across datasets, we use text embeddings from the same semantic embedding space as classifiers and share all network parameters among datasets. We train DaTaSeg on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg improves performance on all datasets, especially small-scale datasets, achieving 54.0 mloU on ADE semantic and 53.5 PQ on COCO panoptic. DaTaaSeg also enables weakly-supervised knowledge transfer on ADE panoptic and 0bjects365 instance segmentation. Experiments show DaTaSeg scales with the number of training datasets and enables open-vocabulary segmen- tation through direct transfer. In addition, we annotate an Objects365 instance segmentation set of 1,000 images and release it as public evaluation benchmark on
.i github. io/ /ddataseg.
Introduction
Image segmentation a core computer vision task with wide applications photo editing, medical imaging, autonomous driving, and beyond. To suit different needs, various forms of segmentation tasks have arisen, the most popular ones being panoptic [29], semantic [23], and instance [19] segmentation. Prior works generally use specific model architectures tailored to each individual task [28, 45, 21, 6]. However, these segmentation tasks are closely related, as they can all be regarded grouping pixels and assigning semantic label to each group. In this paper, address the following question: Can we leverage a diverse collection of segmentation datasets to co-train a single model for all segmentation tasks? A successful solution to this problem would leverage knowledge sharing among datasets, boosting model performance across the board, especially on smaller datasets.
Existing works unified segmentation models either focus single architecture to handle multiple tasks [10, 8, 74, 25], but with separate weights for different datasets; or a single set weights for
Equul contribution. Correspondence to: Xiuye Gu (xiuyegu@google.c com). Work done while at Google. Now NVIDIA. Work done while at Google. Now at ByteDance.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

multiple datasets [27, 33, 77], but on the same task. By contrast, in this work, we aim to train single model on multiple datasets for multiple tasks. Our key idea of unifying these segmentation tasks is to use universal intermediate mask representation: set of mask proposals (i.e., grouped pixels) with class labels [63, 10]. Different segmentation tasks can be realized by applying different merge and post-processing operations on this unified representation. This allows us to train our network on the same output space for different tasks. Furthermore, using this representation, we can exploit weak bounding box supervision for segmentation, which is far cheaper to collect than mask annotations.
To encourage knowledge sharing and transfer among the multiple segmentation sources, our network archi- tecture shares the same set of model weights across all datasets and tasks. In addition, we utilize text em- beddings as the category classifier, which maps class labels from different datasets into a shared seman- tic embedding space. This design further enhances knowledge sharing among categories with similar meanings different datasets, e,g., 'windowpane" and 'window-other'. Our approach can be contrasted with an alternative of using datasct-specific model components we show in experiments that simpler, unified approach leads to improved perfor- mance and enables open- veee-vocabllrry segmentation via simply switching the text embedding classifier.
Input Image
Panapio Repmenation Dennve
Sognenteion Datsasi
Detectinn Detaue (ea))
DaTašeg
Universal Multi-Datsset Multi-Task Sigmentttion Model
Putting these techniques together, we propose DaTaSeg as shown in Fig. I, a universal segmen- tation model, together with cotraining recipe for the multi-task and multi-dataset setting. We train DaTaSeg the ADE20k semantic [75], COCO panoptic [29], and 0bjects365 detection [56] datasets. We show that DaTaSeg improves performance on all datasets comparing with training separately, and significantly benefits relatively small-scale datasets (ADE20k semantic), outperforming a model trained only on ADE20k semantic by +6.1 and +5.1 mIoU with ResNet50 [22] and ViTDet-B [39] backbones. The multi-dataset multi-task setting also allows us to seamlessly perform weakly-supervised segmentation by transferring knowledge from other fully supervised source datasets, which we demonstrate on ADE20k panoptic and Objects365 instance segmentation. DaTaSeg also directly transfers to other datasets not seen during training. It outperforms apen-vocabylary panoptic segmentation methods on the Cityscapes dataset [11] and performs comparably with open- open-vocabulary semantic segmentation works on the Pascal Context dataset [49].
Paropte Signentuion Tesk
8erra Bug Aegiiiitiin Tesk
Irsane Bogneenatoo Tesk
Figure I: Considering the similarities across var- ious segmentation tasks, and recognizing the po- tential for enhancing segmentation performance by harnessing data from multiple sources, we pro- pose to train a universal segmentation model, Da TaSeg, multiple datasets to perform mul- tiple segmentation tasks.
To summarize our contributions, we present DaTaSSeg, a single universal segmentation model on mul- tiple segmentation tasks and datasets. DaTaSeg leverages knowledge from multiple sources boost performance on all datasets. It seamlessly enables weakly-supervised segmentation, directly transfers to other datasets and is capable of open-vocabulary segmentation. As additional contribution, we have annotated a subset of the Objects365 validation set with groundtruth instance masks and release it an evaluation benchmark for instance segmentation.
2 Related Work
Training on multiple datasets has become popular for developing robust computer vision models [64, 78, 33, 47, 61, 67]. For object detection, Wang et al. [64] train an object detector on 11 datasets in different domains and show improved robustness. UniDet [78] trains a unified detector on large-scale detection datasets with an automatic label merging algorithm. DetectionHub [47] employs text embeddings to accommodate different vocabularies from multiple datasets, and features dataset-specific designs. For segmentation, MSeg [33] manually merges the vocabularies of 7 semantic segmentation datasets, and trains a unified model on all of them; however, the manual efforts are expensive and hard to scale. LMSeg [77] dynamically aligns segment queries with category embeddings. UniSeg [27] explores the label relation and conflicts between multiple
Muti-dataset training:

---

datasets in a learned way. Most of these works merge datasets using unified vocabulary on single task, while our work focuses on a more challenging problem: we merge datasets with different vocabularies and different tasks.
Unified segmentation model: Panoptic segmentation [29] unifies semantic and instance segmenta- tion. Prior works [28, 66, 69. 7, 52] exploit separate modules for semantic segmentation [45, 5] and instance segmentation [19. 21], followed by another fusion module. Recently, the mask transformer [63] directly predicts masks with class predictions, allowing end-to-end panoptic segmentation. MaskFormer [10] and K-Net [74] adopt a single transformer- based model for different segmentation tasks. Mask2Former [8] improves upon MaskFormer by proposing masked attention, while MMX--DeepLab [71] develops k-means cross-attention. OneFormer [25] extends 2armerrrmer with multi-task train-once design. All these works still train separate weights on different datasets. By contrast, we aim at a single unified model that can perform well across multiple segmentation tasks and datasets.
framework proposed by
Weakly-supervised segmentation: Facing the issue of expensive segmentation mask annotations, many works have proposed to learn segmentation masks from cheaper forms of supervision [50, 12, 42]. In particular,
supervised approaches are most related to our work, including BoxInst [59], Box2Mask [38], Cut-and-Paste [54] and MAL [34] for instance segmentation, Box2Seg [32] and BCM [58] for semantic segmentation, and DiscoBox [35] and SimpleDoesIt [26] for both semantic and instance segmentation. Despite the popularity of box-driven semantic and instance segmentation, fewer attempts have been made at weakly-superviiee panoptic segmentation. Li et al. [37] employ pretrained model to generate pscudo-ground -rrth in advance, in addition to weak supervision from bounding boxes and image level labels. Shen et al. [57] add a handcrafted branch on top of semantic and instance segmentation models to generate panoptic proposals. Unlike these methods that require customized components, our approach can realize knowledge sharing among datasets with different forms of annotations in more systematic and data-centric fashion, and seamlessly segmentation.
enable
3 Method
3.1 A universal segmentation representation
Segmentation aims to group pixels of the same concept together. We propose use an intermediate and universal representation, namely mask proposals, for all segmentation tasks. A mask proposal binary foreground mask with a class prediction. We use this representation for its versatility: one mask proposal can represent a single instance, any number of instances, region, or even a part of an object. They can overlap and can be combined to form higher-level segmentation outputs an instance ("thing") or region of the same semantic class ("stuff"). Thus they are well suited for the different flavors of segmentation considered in this paper. We note that a similar concept has been used in existing works [63, 10, 8, 71]. for specific segmentation tasks on a single dataset. We show this representation is especially beneficial in our multi-dataset setting, as different datasets define "thing" and "stuff" categories differently. For example, 'table' is thing category in ADE20k panoptic dataset, but there is a 'talle-erredd' stuff category in COCO panoptic. In our framework, both "thing" and "stuff" categories are represented using this same representation, and we treat them differently in the next step.
3.2 Merging predictions for specific tasks
Notations: We introduce the notations we use in the equations. (-) denotes predictions. Let (2j, M,) denote the j-th mask proposal M, with its class prediction ě,. where M, E #XW is the (pre- sigmoid) logits for the mask proposal and &n.c2 E IR is the logit for class cj. We assume a fixed set of N mask proposals. Ca,thing and Ca are the set of thing and stuff categories in dataset d.
Merge operation (MERGE): Given the varying formats of groundtruth annotations for different segmentation tasks, we propose MERGE operation to merge the mask proposals predicting the same category ej. We adopt a simple element-wist wise max operation to merge the mask proposal logits. In particular, the merged mask proposal logits at the (p. q) location for class e₁ is computed as:
M(ck)pa max (1 [argmmaxx()) (2,) = cx] M2n..).
(I)
3

---

your prpped
Toea JOB
yenor
Jenert
yous popped
Panagio Seginentatin
Uniianal
Representation
MERGE
Haggama
Creis Grecgy Leie
bawe popora
Semariic Segneriation
20b Loie
Aliy
Gritsyy Les
pencr
Suallp-Supoiied Insaanee Sepeeoitadon
Figure 2: The universal representation, MERGE operations, and losses for different segmentation tasks. We universal representation for all tasks: set of mask proposals with class predictions. Then adopt distinct MERGE operations based the same "stuff"" category. For semantic segmentation, both "thing" and "stuff" categories undergo merging. In instance segmentation, we do not perform MERGE and there are no "stuff" categories. During training, we use Hungarian matching, and upply different losses based on the supervision types.
segmentation task. For panoptic segmentation, we merge proposals predicting
We choose element -iise max so that applying MERGE on raw mask logits is equivalent to applying MERGE on the soft mask predictions post- sigmod.. We also choose merge at the level of raw logits for numerical stability. If no proposal predicts class ct, then the corresponding merged mask (C is set -100 to ensure it close to 0) after sigmoid.
In addition to merging masks, we merge their corresponding class predictions by simply averaging the class prediction logits the proposals predicting class cę:
E, (1(aggaaxx))) = czc] (g))
((cx)=
(argaaax)) cu)) +
where e is a small number to prevent division by zero.
How we apply the above merge operations depends on the given task, as we now describe:
Panoptic segmentation: make the predicted mask proposals have the same format as groundtruth, we apply MERGE to all "stuff" categories C₁ Cd.stuff-
Semantic segmentation: In contrast with panoptic segmentation, the semantic segmentation groundtruth for each "thing" category is also single mask (i.e., "thing" categories treated equivalently as "stuff" categories). We thus apply MERGE to all predicted mask proposals to cover all thing and stuff categories Cd,stuff U Ca,thing-
Instance segmentation: MERGE is not needed in this task, since the desired outputs are separate masks for separate instances. There are no "stuff" categories in the vocabulary (Cd,stuff 0), and therefore stuff proposals.
Training: During training, we apply the corresponding MERGE operations based on the segmentation task. We then use one-to-one Hungarian matching [31] to match the merged outputs with the groundtruth, and calculate the training losses. Detailed training losses are provided in Sec. 3.5.
Inference: At inference time, apply the based on the given task. We post-process the mask proposals to obtain non-overlapping outputs for panoptic and semantic segmentation. For instance segmentation, we simply take the top mask proposals as final outputs, since overlaps are allowed. Please refer to supplementary for more details.
MERGE operations predicted mask proposals
We illustrate how the MERGE operation works for different segmentation tasks in Fig. 2.
3.3 Weakly-supervised instance segmentation
Bounding boxes are much cheaper to annotate than instance masks, and thus detection datasets can have a much larger scale than instance segmentation datasets. In order to train larger detection datasets with bounding box annotations only, as well as to demonstrate the versatility of our framework to handle various tasks, propose to perform
bux-super instance segmentation.

---

shaeed
[e ciemany
Fruzen
00,+:) By
Tained
M
Encoder
Transformer Docodor
Dx
asgment queris
Mask
N'x
Rx
If W Byy feature
Plant Decznder
...
Figure 3: Overview of universal multi-dataset multi-task segmentation model (DaTaSegg. We feed V learnable segment queries Q, into transformer decoder. which cross cross-attent attends to multi-scale image features Tmoatt from the pixel decoder. The outputs serve as the universal representation for all tasks: N mask proposals Mj accompanied by class predictions 2,. promote knowledge sharing, we share all network parameters across different datasets and tasks. The only dataset-specific design is a classifier consisting of frozen text embeddings of categories specific to each dataset. Additionally, employ shared learnable background classifier.
We adopt a simple projection loss Crroj from [60], which measures consistency of vertical and horizontal projections of the predicted masks My against groundtruth boxes by:
(Mj, Laice (Proj,, (roj,)))) Ldice (Proi, (M,)) Proi, (b;)),
(b)),
where Ldice is the dice loss [48]; by is the groundtruth bounding box matched to j-th mask proposal My; Projz/, denotes the projection operation along the x or y axis, which can be implemented by max operation along the axis; and S(-) denotes the sigmoid function.
By itself, box consistency loss such as Eqn. 3 is insufficient as a supervision signal for segmentation (c.g., Eqn. 3 is equally satisfied by predicting the bounding box of an object instead of its mask). Thus other works have resorted to additional, often more complex, loss terms (such as the pairwise affinity loss from [60]). However, by training on multiple datasets and multiple segmentation tasks, these handcrafted losses are not necessary, as our model can transfer knowledge gained from other fully supevisue segmentation tasks on other datasets.
3.4 Network architecture with knowledge sharing
Network architecture: We now describe the network architecture (similar to [8]) that predicts mask
proposal and class prediction pairs (2,, M,). The input image first goes through a backbone, and we use pixel decoder to 1) output multi-scale image features Imuttii and 2) output a high-resolution feature map FF that fuses information from the multi-scale image features. N mask proposal pairs are then generated from N learnable segment queries Q.: The segment queries are fed into a transformer decoder, which cross attends to the multi-scale image features Iynutti from the pixel decoder. The N decoder outputs are then passed to class embedding and mask embedding heads (both MLPs) to obtain N (mask embedding, class embedding) pairs. To extract final mask proposals and class predictions, we apply a dot product between the high-resolution feature map F and the mask embeddings to obtain N mask proposal logits My. And to obtain class predictions, we compute dot products of the class embeddings against a per-category classifier wy for each category Ck. Fig. 3 shows an overview.
Knowledge sharing: We bake knowledge sharing into our model in two ways. First, all network parameters are shared across all datasets, allowing knowledge sharing among different datasets and knowledge transfer among different tasks. Additionally, to share knowledge gathered from training samples of similar categories across datasets, e.g., 'windowpane" in ADE20k and 'window-other" in COCO, we propose map all category names into shared semantic embedding space. We feed the category names into frozen pre-trained text encoder to set the per-category classifier wk. i.e., we let the fixed text embeddings serve classifiers. This is similar to open-vocabulary
5

---

segmentation [16, 70, 72], but with a different purpose: our focus here is on knowledge sharing among different datasets. This automatic approach scales better than manually unifying label spaces in different datasets [33].
3.5 Co-training strategy
We adopt simple co-training strategy: at each iteration, we randomly sample one dataset, then sample the entire batch for that iteration from the selected dataset. This can be contrasted with sampling from multiple datasets in one iteration. The main advantage for our strategy is that is simple to implement, and allows more freedom to use different settings for different datasets, including applying distinct losses to various tasks. To account for various dataset scales, we use per-dataset sampling ratios, which randomly samples each dataset at a datasct-specific ratio during training, similar to [41].
For fully-supervised tasks, we employ direct mask supervision, which can be obtained from panop- tic/semannic/instancce segmentation groundtruth. To calculate the Hungarian matching cost and the training loss, we use a combination of focal binary cross-entropy loss forat [44], and dice loss
Ldice [48], following [10]. Regarding weak bounding box supervision, we adopt the projection loss Epro; introduced in Sec. 3.3 for the matching cost and training loss. In both cases of supervision, we
use the negative of the class prediction probability p as the classification matching cost [4], and use cross-entropy loss Lce for classification training. The total training loss is:
La = dcee + af + ddice + aproj
(4)
The Hungarian matching cost is defined similarly:
C. = -Meed p P &f focal Adice + dfroj
(5)
Here, A and Ma,d are the weights for dataset d.
4 Experiments
Datasets and metrics: We train and evaluate DaTaSeg on COCO panoptic [29] and ADE20k semantic [75] using mask supervision, well Objects365-v2 [56] detection datasets using bounding box weak supervision. COCO panoptic is the most popular panoptic segmentation benchmark with 118,287 training images and 5,000 validation images. COCO has 80 thing categories and 53 stuff categories. ADE20k semantic is one of the most widely used semantic segmentation benchmarks with 150 categories, 20,210 training images, and 2,000 validation images.
For evaluation, besides the training datasets, we also evaluate on ADE20k panoptic, which uses the same validation images as ADE20k semantic but with panoptic annotations. The original 150 categories are divided into 100 thing categories and 50 stuff categories. Finally, we train on the Objects365-v2 detection (bounding boxes only) dataset, which has 365 categories and 1,662,292 training images. To evaluate the reakll-supervised instance segmentation results, we manually label subset of 1,000 images from the Objects365 validation set. Our annotation pipeline involves 20 well- trained human annotators, following the protocol in [2], without using any automatic assistant tools. Please refer to Appendix B for more details. We release this Objects365 instance segmentation evaluation set as a public benchmark.
To compare DaTaSeg with the state-of-the-art methods, we in addition evaluate on popular open- vocabulary segmentation benchmarks. Following OpenSeg [16], we evaluate on PASCAL Context datasets [49] with 5k val images. We use its two versions: 59 classes (PC-59) and 459 classes (PC-459). We also evaluate on the 500-image val set of Cityscapes panoptic dataset [11], which focuses urban street scenes with 11 stuff categories and 8 thing categories.
We report results in standard evaluation metrics: panoptic quality (PQ), mean intersection-over- union (mIoU), and mean average precision (AP) for panoptic, semantic, and instance segmentation, respectively.
We experiment with ResNet [22] and ViTDet [39] backbones. For ResNet, we use an ImageNet-1K K [55] pretrained checkpoint, with a backbone learning rate multiplier 0. [4]; the pixel decoder consists of transformer encoder [62] and modified FPN [43] following [10].
Implementation details:

---

Fully-Supervised
Weakly-Supervised Transfer
ADE
COCO
ADE
0365
Backbone
Model
semantic
panoptic
semantic + panoptic
box + instance
mioU
PQ
PQ
mask AP
Separate
42.0
48.2
26.9
12.3
ResNet50
DaTaSeg
48.1 (+6.1)
49.0 (+0.8)
29.8 (+2.9)
14.3 (+2.0)
Separate
46.3
51.9
27.5
14.7
ViTDet-B
Da
51.4 (+5.1)
52.8 (+0.9)
32.9 (+5.4)
16.1 (+1.4)
ViTDet-L
DaTaSeg 54.0
53.5
33.4
16.4
Table 1: Comparing DaTaSeg with separate dataset-specific models. We show results both the training tasks s and new tasks tweakly-supervise transfer). Our single DaTaSeg outperforms separately trained models on all datasets. They are trained under the same settings, so the performance gains come from knowledge in other datascts through our multi-dataset multi-task cotraining recipe. We observe: 1) DaTaSeg significantly improves tasks with limited data (ADE20k semantic); 2) DaTaSeg enables weakly- supkly-super knowledge transfer (ADE20k panoptic and 0365 instance); 3) DaTaSeg scales well with backbones.
Training sets COCO
Sally-Superviser
Weakly-Supervve Transfer
ADE
O365 bbox
ADE
COCO
ADE
O365
semantic
panoptic
semantic
panoptic
semantic - panoptic box + instance
mloU
PQ
PQ
mask AP
42.0
8.4
26.7
1.0
15.3
48.2
11.6
5.8
11.0
12.4
5.8
12.3
46.3 (44.3)
14.0
(-0.))
15.0 (+2.7)
18.3
48.9
(+0.7)
12.3
15.2 (+2.9)
47.3 (+5.3)
49.0 (+0.8)
30.5 (+3.8)
5.6
48.1 (+6.1)
49.0 (+0.8)
29.8 (+2.9)
14.3 (+2.0)
Table 2: Importance of training on multiple datasets. We train DaTaSeg on all combinations the three datascts.
Experiments are done on ResNet50 under same settings. DaTaSeg scales with the number of training datasets.
except that we use Layer Norm [1] and GeLU [24] activation for training stability. For ViTDet, we use the ImageNet-1K MAE pretrained checkpoint [20] with the layerwise Ir decay [39]] the pixel decoder is the simple feature pyramid inside ViTDet, whose outputs are upsampled and added together to get the high resolution feature map F. The mask embedding head a 3-layer MLP. The class embedding head contains a linear layer followed by ReLU, We use CLIP-L/14 [53] as the pretrained text encoder. We 100 segment queries 2., unless otherwise stated. See Appendix H for more details.
4.1 DaTaSeg improves over dataset-specific models
As an apples-to-aplle comparison, we separately train a model on each dataset. We ensure that the models see the same number of images in cotraining and separately (left) shows the results. DaTaSeg outperforms separately trained models on all datasets. Since we use exactly the same settings, we attribute the performance boost to our multi-datased multi-task training recipe, which harnesses knowledge from multiple sources, Especially. DaTaSeg leads to significant performance boost on ADE20k semantic: +6.1 mIoU and +5.1 mIoU with ResNet50 and ViTDet-B B backbones, respectively. This proves our argument that cotraining more data helps overcome the data limitation issue smaller-scale segmentation datasets.
on each dataset. Table I
4.2 Da DaSg enables weakly-supervised transfer
In Table (right), evaluate DaTaSeg ADE20k panoptic and Objects365 instance segmentation tasks. Note we only have weak supervision (ADE20K semantic and Objects365 bbox) during training. Again, we compare to models separately trained on one dataset our baselines. Specifically, use the ADE semantic for ADE panoptic evaluation, and an Objects365 model trained only using weak box supervision for Objects365 instance evaluation, respectively.
By cotraining DaTaSeeg in a multi-dataser multi-task fashion, we improve the ADE20k panoptic performance by +2.9 PQ and +5,4 PQ with ResNer50 and ViTDet-B backbones. Our design allows the panoptic segmentation knowledge to transfer from COCO to ADE20k. For reference, the fully-supervised performance is 37.7 PQ on ResNet50 and 42.3 PQ ViTDet-B.
7

---

PC-59
PC-459
COCO
Cityscapes
Method
Backbone
Training data
mloU
mloU
mloU
PQ
ODISE [68]
UN
LAION-CLIP&COCO
55.3
13.8
52.4
23.9
MaskCLIP [13]
R-50
COCO pan+CLIP
45.9
10.0
R-10Ie
53.3
11.0
OVSeg [40]
COCO stuff+cap
Swin-B
55.7
12.4
R-101
COCO pan+cap
42.1
9.0
36.1
OpenSeg [16]
Eff-b7
COCO+Loc. Narr.
44.8
11.5
38.0
R-50
COCO panoptic
50.9
11.1
57.7
30.0
DaTaSeg
ViTDet-B
+ADE semantic
51.1
11.6
62.7
28.0
ViTDet-L +0365 bbox
51.4
11.1
62.9
29.8
Table 3: Compare DaTaSeg to state-of-the-art open-vocabulary segmentation models. We apply DaTaSeg directly to other semantic panoptic segmentation datasets without finetuning. We compare to ODISE [68]. MaskCLIP [13]. OVSeg [40], and OpenSeg [16] their corresponding benchmarks. We only conduct system- level comparison due to differences
training data backbones. DaTaSeg performs comparably on all semantic segmentation benchmarks and outperforms the recent work ODISE on Cityscapes. PC: Pascal Context. COCO: COCO is included in the training set for all models, and is not completely oopen-vocabuarly..
queries
100
150
DalaSeg
+D-S modules
ADE semantic
mloU
47.4
48.1
48.7
ADE semantic
mloU
48.1
48.1 (-0.0)
COCO panoptic
PQ
48.7
49.0
48.9
COCO panoptic PQ
49.0
46.0 (-3.0)
ADE panoptic
PQ
30.7
29.8
29.1
ADE panoptic
PQ
29.8
26.9 (-2.9)
0365 instance
AP
13.4
14.3
15.88
0365 instance"
AP
14.3
10.9 (-3.4)
Table 4: In general, the performance of DaTaSeg increases as we increase the number of segment queries. Besides, using only 50 queries already achieves good performance. Exper- iments are conducted using ResNet50 backbone. : weakly-supervised tasks.
Table 5: Adding dataset-specific (D-S) modules hurts performance almost all datasets. This shows the importance of sharing all parameters for better knowledge sharing, which particularly benefits the weakly-supervisee tasks. Experiments are conducted on a ResNet50 backbone.
On Objects365, we only train with a weak box consistency loss Cyroj. The task training recipe improves the mask AP by +2.0 AP and +1.4 AP for the two backbones. The improvement is most likely from the instance segmentation knowledge COCO panoptic.
multi-
4.3
Da TaTaSe scales with size of backbones
Our DaTaSeg design orthogonal to detailed network architectures, as long it is able output the universal representation (mask proposals with class predictions). This is supported by the consistent performance gains among ResNet50, ViTDet-B, and ViTDet-L backbones as shown in Table 1. It also scales well as we increase the sizes of the backbones. With ViTDet-L, DaTaSeg reaches $4.0 mloU on ADE20k semantic and 53.5 PQ on COCO panoptic in single model, without bells and whistles.
4.4 Da TaSeg scales with number of datasets
We study how DaTaSeg scales with the number of training datasets. We train DaTaSeg on all combinations of one, two, and three datasets among the ADE20k semantic, COCO panoptic, and Objects365 detection datasets, in order to conduct a comprehensive study, with ResNet50 backbone. Table 2 presents the results. Looking at each column, we see that performance each dataset generally improves as the number of training datasets increases, especially for ADE semantic.
Since DaTaSee shares all parameters among all datasets and tasks, we can evaluate the cross-dataset transfer performance. We notice the model transfers to datasets that are not trained. e.g., A model trained on COCO panoptic and Objects365 detection achieves 18.3 mloU on ADE semantic, which comparable to open-vocabulary segmentation performance (LSeg+ [16, 36] achieves 18.0 mloU on a ResNet101 backbone).
4.5 Da TaSaSe enables open-vocabulary segmentation
further advantage of our fully-shared architecture is that we have the ability directly transfer to other segmentation datasets. We simply switch the text embedding classifier with the vocabularies

---

(A) COCO panopec
AD)2OO twesddy supervised)
Ubject3665 (wessly-superssee))
Figure 4: Qualitative results of DaTaSeg on all datasets & tasks, For every pair of images, the left is DaTaSeg prediction and the right groundtruth. DaTaSeg succeeds hard cases (e.g., transparent wine glasses the top left) as well as veakly-suupervise datasets. For object instances in panoptic and instance segmentation, the colors are not matched to the groundtruths. Black denotes ignored regions.
in a target dataset not used in training. Table 3 shows the results comparing DaTaSeg to several apen-vocabulary works: ODISE [68] and MaskCLIP [13] open- open-vocabulary panoptic segmentation methods and OVSeg [40] and OpenSeg [16] are open-vocabulary semantic segmentation approaches. Unlike these works, method does not train on large-scale image-text data or use pretrained image-text models (except the pretrained text encoder), which puts our method at disadvantage. Despite this disadvantage, DaTaSeg outperforms ODISE on the Cityscapes panoptic dataset. Note that there is a domain gap between our training sets and Cityscapes, which focuses on urban street scenes. DaTaSeg achieves comparable performance on PC-59 and PC-459, All methods in comparison train on COCO panoptic and ours has the best performance on COCO semantic. These results show that DaTaSeg enables open- vocabulary segmentation via our multi-datase multi-task training approach.
4.6 Da TaSeg scales with number queries
We perform ablation study on the number of segment queries Q, and show the results in Table The performance improves as we increase the number of queries from 50 to 150, on almost all datasets. Besides, DaTaSeg achieves reasonably good performance with only 50 queries, which may benefit memory-limited application scenarios. We perform more ablation studies in Appendix D.
4.7 Adding datasct-specific modules hurts performance
In multi-dataset learning, prior works have introduced dataset-specific modules [47], in order to address inconsistencies across datasets [33, 61]. However, such a design may weaken knowledge sharing across different datasets and tasks, especially for weakly-supervised tasks that may rely more on other datasets and tasks to compensate for the weak supervision. To study this issue thoroughly, we carefully design several dataset-specific modules, including dataset-specific queries, datasct-specific heads, and dataset-specific prompts. These modules are lightweight by design so as to still encourage knowledge sharing through other shared parameters.
In Table we evaluate the effectiveness of these dataset- dataset-specific modules by adding them DaTaSeg. and keeping all other settings the same. Comparing with DaTaSeg that shares all parameters, results reveal that using dataset-specific modules hurts performance on all datasets except ADE20k semantic. Moreover, see that the performance drops more weakly-supervised tasks (ADE panoptic and O365 instance). This verifies that removing dataset- modules yields better knowledge sharing across datasets and tasks, and thus greatly improves performance. See Appendix E for more details.

---

4.8
Qualitative analysis
We show qualitative results (with ViTDet-L) Fig. 4. DaTaSeg performs well both fully and weakly supervised settings. We observe that localization quality is good in general, while classification more error-prone. Nevertheless, DaTaSeg succeeds in challenging cases including transparent objects, occlusion, and crowd scenes.
5
Conclusion
The goal of our work is to train single universal segmentation model on multiple datasets and multi- ple tasks (i.e., semantic, instance and panoptic segmentation). We present DaTaSeg, which leverages a universal segmentation representation, shared network parameters, and shared semantic embedding space for classification. DaTaSeg surpasses separately trained models, leading significant gains smaller datasets such as ADE20k semantic. It unlocks a new weakly-superrisee transfer capability on datasets that were not explicitly trained for particular task (e.g., ADE20k panoptic and Objects365 instance). DaTaSeg also directly transfer to more segmentation datasets and enables open-vocabulary segmentation. We believe that in the future, this multi-task multi-dataset setting will be the norm rather than the outlier as it currently is, and we see our paper as taking a step in this direction.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:6007004450 2016.
[2] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR, 2019.
[3] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. NeurlPS, 2019. 16
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 6
[5] Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy. and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. TPAMI, 3
[6] Liang-Chied Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. ECCV, 2018. 1
Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang- Chich Chen. Panoptic-DeepL.at Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmenta- tion. In CVPR, 2020. 3
[8] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar, Masked- attention mask transformer for universal image segmentation. In CVPR, 2022. 1, 3, 17, 20, 21
[9] Bowen Cheng, Omkar Parkhi, and Alexander Kirillov. Pointly-supervised instance segmentation. In CVPR, 2022. 20
[10] Bowen Cheng, Alex Schwing, and Alexander Kirillov, Per-pixel classification is not all you need for semantic segmentation. NeurlPS, 2021. 1, 2, 6, 17
[11] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes datasct for semantic urhan scene understanding. In CVPR, 2016. 2,
[12] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV, 2015.
[13] Zheng Ding, Jicke Wang, and Zhuowen Tu. Open-vocabulary panoptic segmentation with maskelip. ICML, 2023. 8, 9
[14] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. 15
10

---

[15] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-paste strong data augmentation method for instance segmentation. In CVPR, 2021. 21
[16] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-wecahulary image segmentation with image-level lahels. In ECCV, 2022. 6, 8,9
[17] Zhangxuan Gu, Siyuan Zhou, Niu, Zihan Zhao, and Liqing Zhang. Context-award feature generation for zero-shot semantic segmentation. In ACM MM, 2020. 16
[18] Agrim Gupta. Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In CVPR, 2019. 14, 21
[19] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In ECCV, 2014. 1,3
[20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 7
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 1,3
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 2,
[23] Xuming He, Richard Zemel, and Miguel Á Carreira-Perpihee Multiscale conditional random fields for image laheling. In CVPR, 2004.
[24] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). wrXivv1606.084155, 2016. 7
[25] Jitesh Jain, Jiachen Li, Mang Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR, 2023. 1,3
[26] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schicle. Simple does it: Weakly supervised instance and semantic segmentation. In CVPR, 2017. 3
[27] Dongwan Kim, Yi-Hsuan Tsai, Yumin Suh, Masoud Faraki. Sparsh Garg. Manmohan Chandraker, and Bohyung Han. Learning semantic segmentation from multiple datasets with label shifts. In ECCV, 2022. 2
[28] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019. 1,3
[29] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019. 1, 2, 6,6
[30] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304. 02644,, 2023.
[31] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly. 1955. 4
[32] Viveka Kulharia, Siddhartha Chandra, Amit Agrawal, Philip Torr, and Ambrish Tyagi. Box2seg: Attention weighted loss and discriminative feature learning for weakly supervised segmentation. ECCV, 2020.
[33] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In CVPR, 2020. 2, 6, 9, 17
[34] Shiyi Lan, Xitong Yang. Zhiding Yu, Zuxuan Wu, Jose Alvarez, and Anima Anandkumar, Vision transformers are good mask auto-labelers. In CVPR, 2023.
[35] Shiyi Lan, Zhiding Yu, Christopher Choy, Subhashree Radhakrishnan, Guilin Liu, Yuke Zhu, Larry Davis, and Anima Anandkumar. Discobox: Weakly supervised instance segmentation and semantic correspondence from box supervision. In ICCV, 2021. 3
[36] Boyi Li, Kilian Q Weinberger, Serge Belongic, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. ICLR, 2022. 8
[37] Qizhu Li. Anurag Amnb, and Philip HS Torr. Weakly-and semi- coni-supervieed panoptic segmentation. In ECCV, 2018. 3

---

[38] Wentong Li, Wenyu Liu, Jianke Zhu. Miaomiao Cui, Risheng Yu, Xiansheng Hua, and Lei Zhang. Box2mask: Box-supervised instance segmentation via level-set evolution. urXi::22/2.01579, 2022.
[39] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. ECCV, 2022. 2, 6.7
[40] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Pcizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. CVPR, 2023. 8,9 9
[41] Valerii Likhosherstov, Anurag Arnah, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio.
arXiv:2111.1299) 2021. 6
[42] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Seribble-supervised convolttinaa networks for semantic segmentation. CVPR, 2016.
[43] Tsung-Y Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In CVPR, 2017. 6, 20
[44] Tsung- Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In ICCV, 2017. 6
[45] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR. 2015. 3.3
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019. 20
[47] Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengehen Liu, Jianfeng Wang. Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation language embedding. CVPR, 2023. 2,9
[48] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. 3DV, 2016. 5, 6
[49] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-W Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In CVPR, 2014. 2, 6
[50] George Papandreou, Liang-Chich Chen, Kevin Murphy, and Alan L Yuille. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In ICCV, 2015. 3
[51] Giuseppe Pastore, Fabio Cermelli, Yongqin Xian, Massimiliano Mancini, Zeynep Akata, and Barbara Caputo. A closer look self-training for zero-label semantic segmentation. In CVPRW, 2021. 16
[52] Siyuan Qiao, Liang-Chich Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. CVPR, 2021.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021.7 7
[54] Tal Remez, Jonathan Huang, and Matthew Brown, Learning to segment via cut-and-paste. In ECCV, 2018.
[55] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michacl S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
[56] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019. 2, 6
[57] Yunhang Shen. Liujuan Cao, Zhiwei Chen, Feihong Lian, Baochang Zhang, Chi Su, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Toward joint thing-and-stuff mining for weakly supervised panoptic segmentation. In CVPR, 2021. 3
[58] Chunfeng Song. Yan Huang, Wanli Ouyang, and Liang Wang. Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation. In CVPR, 2019. 3
12

---

[59] Zhi Tian, Chunhua Shen, Xinlong Wang. and Hao Chen, Boxinst: pirg--errformance instance segmentation with box annotations. In CVPR, 2021.
[60] Zhi Tian, Chunhua Shen, Xinlong Wang, and Hao Chen, Boxinst: High-performance instance segmentation with box annotations. In CVPR, 2021. 5
[61] Jasper Uijlings. Thomas Mensink, and Vittorio Ferrari. The missing link: Finding label relations across
datasets. In ECCV, 2022. 2,9, 17
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017. 6
[63] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021. 2,3
[64] Xudong Wang. Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. In CVPR, 2019. 2
[65] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, 2019. 16
[66]
Yuwen Xiong. Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. Upsnet: A unified panoptic segmentation network. In CVPR, 2019.
[67] Hang Xu, Limpu Fang, Xiandan Liang, Wenxiong Kang, and Zhenguo Li. Universal-renn: Universal object detector via transferable graph r-cnn. In AAAI, 2020. 2
[68] Jiarui Xu. Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. CVPR, 2023. 8,9
[69] Tien-Ju Yang, Maxwell D Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, and Liang-Chieh Chen. Deeperlab: Single-shot image parser. arXiv:190.00509,, 2019. 3
[70] Qihang Yu, Ju Hc, Xueqing Deng, Xiaohui Shen, and I. Liamg-Chich Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. NeurlPS, 2023.
[71] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022. 3
[72] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. simple framework for open- open-vocayulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020-1031, 2023.
[73] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ICIR, 2018. 21
[74] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurlPS, 2021. 1,3
[75] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k datasct. In CVPR, 2017. 2,
[76] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. ECCV, 2022. 15, 16
[77] Qiang Zhou, Yuang Liu. Chaohui Yu, Jingliang Li, Zhibin Wang, and Fan Wang. Lmseg: Language-guided dulti-d segmentation. ICLR, 2023. 2
[78] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Simple multi-datase detection. In CVPR, 2022.
[79] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li. Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021. 20
[80] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai. Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. CVPR, 2023. 19
13

---

Appendix
Funding in direct support of this work: Google LLC. There is additional revenues related this work.
A Broader Impacts
We propose a single framework for multi-dataset multi-task segmentation, which especially benefits memory- application scenarios, e.g., our work can make single segmentation model that meets all kinds of segmentation needs on memory- memory-limited mobile devices (semantic segmentation for photo editing, instance segmentation for object detection and image search, etc.), instead of using separate models for various segmentation tasks.
Moreover, our approach improves segmentation performance from data-centric view: We leverage segmentation data from multiple sources to improve the overall segmentation performance, especially on datasets of smaller scales, e.g., ADE20k semantic. We also transfer knowledge from other sources segmentation. This is helpful on the scenario where segmentation data
to enable weakly-supervise are very hard to collect, we can cotrain on similar data to help improve the performance, or even train on data with weaker supervision. e.g., if we want to train segmentation model on rare wild animals, we can co-train model on a small set of these kinds of animals, and larger dataset of more common animals.
As for bias and fairness, model learns from the training data, it may inherit the bias and fairness issues existing in the training data. Nonetheless, our model trains on multiple sources, and the fairer datasets may compensate the bias and fairness issues in other datasets.
B
More about our newly labeled Objects365 instance segmentation evaluation dataset
We randomly sampled 1,000 images from the Objects365 V2 validation set. Different from the "federated" LVIS dataset [18] (not all categories in each image are completely labeled), the bounding boxes in Objects365 are completely labeled for all categories and we follow this complete annotation in our instance masks. For each bounding box annotation, we generate a foreground/background annotation question for the raters. In the question, raters are asked to paint a semi- semi-transparent instance mask on the original image, with the groundtruth bbox shown on the image as guide and the groundtruth category displayed on the side. We do not crop the original image to the bbox region, but show the entire image to provide more context information, which is especially helpful for small objects. If the boundary is too blurry or too dark to annotate the instance mask, the raters can skip the question.
The annotation tool is a free painting tool, which allows the raters to freely draw the instance mask. We ask the raters to try to draw within the bbox, but if the object is obviously exceeding the bbox, then they can draw outside the bbox. The size of the stroke is adjustable. Raters can coom-in/out, draw straight lines, and fill the inside of plotted boundary with a single click (which quite useful for instance segmentation). Unlike polygons in COCO etc., the annotation tool allows us to save higher-resolution binary masks.
We inserted a total of 13,372 questions. Following the common practice in instance segmentation, we did not label the crowd instances, as they are skipped in both training and inference. In the end, we obtained 12,836 valid instance mask annotations. It took a total of 800.59 rater hours. On average, raters spent 3.74 minutes on each valid mask.
In Fig. 5, we show some statistics about our Objects365 instance segmentation dataset. The number of masks per category follows relatively long-tailed distribution (Fig. 5(a)). The number of masks per image reveals that a large number of images have many instance annotations (Fig. 5(b)). And the statisties of the relative mask size is close to that of the LVIS datasets (Fig. 5(c)).
As visualized Fig. 6, the mask annotations are very high-quality. The raters carefully handled small objects, thin structures, complicated shapes, occluded objects, etc.
14

---

serred dirggey Indis
162a4 image Bil
(a) Number of annotated instance masks category. It follows rel- masks image. More than 500 (square root of the ratio of mask atively long tail distribution.
(b) Number of annotated instance
(c) Relative segmentation mask size
images have more than 10 instances. area to image area), compared with
COCO, LVIS, and ADE20K. Our statistics are similar to LVIS. Figure 5: Dataset statisties for our Objects365 instance segmentation evaluation set.
Figure 6: Visualization of our Objects365 instance segmentation dataset. We store binary masks, which has higher mask quality compared with polygons. The quality small objects are good (first two images). Complex structures are also laheled delicately, e-g., the camels (1st image 2nd row) and the decoration image on 3rd row). For occlusions, raters carefully avoided the occluded regions as they are not a part object to annotate, e.g., the fruits in the basket are not included in the basket masks (last image on st row).
the cake (1st
For the objects which do not have an instance mask since raters skipped it's possible that the model still predicts an instance mask for that object. We include the skipped annotations to make it optional ignore them during evaluation, so as avoid treating such cases as false positive.
We release this dataset. We believe this effort can provide the research community a good benchmark for instance segmentation evaluation, e.g., for evaluating weakly-supervised instance segmentation leamned from Objects365 bounding boxes.
C Open-vocabulary semantic segmentation on PASCAL VOC
We evaluate DaTaSeg on the PASCAL VOC 2012 semantic segmentation dataset [14], which includes 20 object classes and a background class. We directly transfer our trained model to its validation set with 1,449 images without finetuning. We follow the setting in MaskCLIP+ [76] and ignore the background class during evaluation.
15

---

Method
Backbone
mloU
mloU-unseen
mloU-seen
SPNet [65]
56.9
0.0
75.8
SPNet-C [65]
63.2
15.6
78.0
7S3Net [3]
61.6
17.7
77.3
CaGNet [17]
Deepl.abv2-ResNeet00
65.5
26.6
78.4
STRICT [51]
70,9
35.6
82.7
MaskCLIP+ [76]
88.1
86.1
88.8
sully-supervised
88.2
87.0
88.6
ResNet-50
89.4
87.7
89.9
DaTaSeg
ViTDet-B 92.8
90.9
93.4
ViTDet-L
94.7
94.3
94,9
Table 6: Open-vocabulary semantic segmentation on the PASCAL VOC 2012 dataset. DaTaSeg outper- methods, and even the fully-supervised baseline. Comparison methods
Torms all other zero-shot/open-voalle are trained on 15 seen classes of PASCAL VOC and only 5 classes are unseen (mloU-unseen)). In contrast, DaTaSeg has never seen the PASCAL VOC dataset during training (it's trained several other datasets). We attribute DaTaSeg's performance improvement to our multi-datase multi task training. Following [76]. the background class is ignored, The numbers of comparison methods are from [76]. SPNet-C stands SPNet with calibration.
Fully-supervised
Seakly-supervised transfer
ADE
COCO
ADE
0363 box + instance
Dataset sampling ratio
semantic
panoptic
semantic - panoptic
mIoU
PQ
PQ
mask AP
1:4:4
48.1
49.0
29.8
14.3
1:2:2
46.8
48.6
12.8
1:1:1
45.3
48.0
28.4
13.7
Table 7: Ablation study on dataset sampling ratio (ADE:COCO:0365) Evaluated on a ResNet50 hackbone. Results show that our adopted sampling ratio (1:4:4) is better than the other sampling ratios.
We compare with other zero-shot and open-vocabulary segmentation methods in Table 6, which are trained on 15 seen classes with 5 classes held out during training as unseen classes (potted plant, sheep, sofa, train, and TV monitor). DaTaSeg has never seen the PASCAL VOC dataset during training, though it is trained other segmentation datasets (COCO panoptic, ADE semantic, and Objects365 detection). DaTaSeg with a ResNet-50 backbone outperforms all these methods with DeepLaby2-ResNetlle
backbones. Surprisingly. DaTaSeg even outperforms the fully-supervised counterpart. We attribute DaTaSeg's performance improvement to our multi-dataset multi-task training. We note this is not apple-to-apple comparison, since the training sources are different. The main purpose of this comparison is to show how our "data-centric" DaTaSeg is positioned against cooo-sho/open-oocabbee direct transfer manner.
segmentation methods on the PASCAL VOC dataset, using
D Ablation studies
We perform more ablation studies in this section.
Ablation study dataset sampling ratio: Our dataset sampling strategy (Sec. 3.5) avoids the imbalance dataset sampling by specifying the per-dataset sampling ratio. The proportion of training samples coming from each dataset in the whole training process determined by that sampling ratio. In our main results, the sampling ratio is 1:4:4 for ADE:COCO:0365, We ablate the dataset sampling ratio in Table 7. Results show that our adopted sampling ratio better than the other sampling ratios.
Ablation study on the portion of Objects365 training data: Readers may have the question about the trade-off's between the quality and quantity of the training samples when we introduce the weakly- supervised training data (Objects365). In Table 2, we show the results with and without cotraining on Objects365. The performance is generally better when cotraining on Objects365. In addition, we experiment with cotraining on different portions of the dataset: we cotrain DaTaSeg on the full COCO panoptic and ADE semantic datasets, and 10% 25% 50% 100% of 0365 training data. We show the results in Table 8. Results show that when increasing the number of O365 superissueervised training samples, 0365 performance increases, COCO panoptic performance is not affected, and ADE20k performance slightly decreases. Overall, the gains are larger than the losses.
16

---

Sully-super
Weakly-supervised transfer
ADE
COCO
ADE
0365
Portion of 0365 training data
semantic mloU
panoptic
semantic - panoptic
box + instance
PQ
PQ
mask AP
10%
48.5
48.7
30.0
10.4
25%
48.6
48.3
30.6
50%
47.7
48.5
29.8
13.1
100%
47.2
48.7
29.4
14.5
Table 8: Ablation study on the portion of 0365 training data. Evaluated on a ResNet50 backhone. When increasing the number of O365 weakly-supervised samples, O365 performance increases, COCO panoptic performance is affected, and ADE20K performance slightly decreases.
ADE semantic
COCO
ADE semantic panoptic hox - instance
Q366
Apros
Hproy
panoptic
1.0
45.5
48.3
28.4
12.5
0.5
47.2
48.7
29.4
14.5
Table Ablation study on the projection loss Cpros weights. Evaluated ResNet50 backbone. The results indicate that the 2nd setting (our final setting) has a better performance on all datasets.
masi & clase predictions
Lx
AADDE
В D
Network
Add Norm
Salf Self-Attantion
Vv
Suten
atter ticn
Missked Cross-Allendinn
mask
4V $
crote attention weights
AAA M
speoito
Q
Figure 7: Dataset-specific queries in the transformer decoder. use different dataset-specific queries Qs for each dataset d, which are treated the same as the segment queries Q., and involved in both cross-attention and self-attention. All datasct-specific queries are discarded before predicting masks and classes.
Ablation study on projection loss Lproj weights: Our settings of À and (Table 13) closely follow Maskformer series [10, 8). We add the the bounding-box projection loss Lproz for Objects365. We an ablation study indicate that the 2nd setting (our final setting) has a better performance on all datasets. Finding the optimal weights for multiple losses is interesting yet challenging problem, and we leave it for future work.
the weights (Arroj and μlrro) for the projection loss in Table 9, The results
E Adding dataset-specific modules hurts performance
In this section, we first introduce our designed dataset-specific modules and then show more ablation studies on these modules.
As prior works have pointed out [33, 61], there are many inconsistencies across segmentation datasets, including labels having their own particular definition, being annotated using different protocols, etc. We explore if adding dataset-specific modules can improve overall performance by designing the following modules.
Dateset-specific queries: our architecture design introduced in Sec. 3.4, an input image will not get any dataset-speciff information until the text embedding classifier, which is almost at the final stage of the network. We thus propose add few dataset-specific queries Qa as an auxiliary input to the beginning of transformer decoder. Each training dataset d has its own Qg. The dataset-specific
17

---

spconfc
sharld
bechgrourd
donssett (C cosses)
Frosenn Pree sained Text
(C) 1) dP
Errooder
home
persion
Tranafomar Dhocoder
Wimek pignoers
xx
Rx
Music Head
HxWxx
Ploal Deczzhe
...
Figure 8: Overview incorporating all our designed dataset-specific modules into our universal multi- dataset multi-task segmentation model (DaTaSeg). We design set of dataset-specifie queries fed into the transformer decoder, dataset--peciffe classificatin head, and pataset-specift background classifier. They are highlighted in orange. All these modules learaalle. Different datasets have separate parameters for these modules.
queries are treated the same regular segment queries Q.. The only difference is that we do not make mask and class predictions for the dataset-specific queries. Fig. shows the details. In the cross-attention layer, Qa cross attends to the image features the cross-attention weight is:
Wemss Jllien softmax + W, Qu 2..] W
(6)
where Immits are the multi-scale image features in Sec. 3.4. Wg. WE denote the linear projection weights; A denotes the attention mask.
In the self-attention layers, there are interactions for all pairs of Qa and Q. allowing the segment queries to be aware of the dataset-specific information at the beginning of transformer decoder. This can be shown by the self-attention weight calculation:
Whlt atm softmax (W,
Qa Q
[
2
w
Q.
e.O.
softmax (W9
w
e.
e e
Dstaset-specific classification head and background classifier: We observe that dataset incon- sistency tends to primarily be classification issue, rather than localization issue. Hence, for all datasets we use the same set of parameters for localization (backbone, pixel decoder, transformer decoder, mask embedding head) to enhance knowledge sharing. For classification, dataset-specific components: specifically, we use a dataset-specific MIP class embedding head as well as learnable dataset-specific background classifier, since the definitions of background vary from dataset to dataset.
design some
Overall dataset-specific architecture: The overall architecture equipped with all our dataset- specific modules is shown in Fig. 8. We design the dataset-Specific modules to be light-weight, which allows to save on memory costs. With this design of mixture of mostly shared network and light-weight dataset-specific modules, the model has the freedom to choose whether to leverage dataset-specific information, or to use the shared knowledge across datasets/tasks. However, one significant disadvantage of adding datasct-specific modules is that it's hard to decide which set of dataset-specific parameters use when directly transferring to other datasets (Sec. 4.5), and thus hurts the open-vocabulary capabiliry.
ablate on the number of dataset-specific queries Qa in Table 10. It shows that removing the dataset-specific queries achieves the best results on COCO panoptic and Objects365 by a large margin. It also achieves reasonable performance on other datasets. Using 30 queries hurts the performance COCO panoptic and ADE semantic.
Ablation studies on dataset-specific modules:
18

---

Fully-supervised
Weakly-supervised transfer
ADE
COCO
ADE
0365
# of dataset-specific queries
semantic
panoptic
semantic - panoptic
box - instance
mloU
PQ
PQ
mask AP
0
47.8
48.1
28.6
13.1
10
48.0
46.2
28.0
11.2
20
48.6
46.6
28.5
11.7
30
47.1
45,8
27.4
11.7
Table 10: Ablation study on the number of dataset-specific queries Qa. Evaluated on a ResNet50 backbone. Without dataset-specific queries, the performance on COCO panoptic and Objects365 instance segmentation is much better. Using large number of dataset-specific queries, e.g., 30, hurts performance on many datasets, since it weakens knowledge sharing among datasets tasks.
w/ dataset-specife classification
w/o datasct-specifie classification
ADE semantic
mIoU PQ
47.9
48.1 (+0.2)
COCO panoptic
48.5
49.0 (+0.5)
ADE panoptic!
PQ
28.7
29.8 (+1.1)
0365 instance
AP
12.9
14.3 (+1,4)
Table 11:
Adding -ppciic- classification modules hurt performance on almost all datasets, espe- cially the weakly-supervised transfer results!. The datasct-specific classification modules include the classifica- tion embedding head and the learnable background classifier. Evaluated on ResNet50 hackbone, without other datasct-specific modules.
In Table 11, compare using and not using dataset-specific classification modules (the classifi- cation embedding head and background classifier). Results indicate that removing dataset-specific classification improves performance on all datasets & tasks, especially the weakly-supervised tasks. We suspect it's because weakly-supervised tasks rely more on knowledge sharing.
System-level comparison between with and without dataset-specifie modules: In the main paper, due to space limit, we only show results of adding all dataset-specific modules on ResNet50 backbone. In Table 12, we show the same comparison on more backhones, i.e., ResNet50, ViTDet-B, and Vi. The observation is consistent across all backbones: adding dataset-specific modules hurts performance on all datasets. The difference is more significant on backbones of smaller scales, and on COCO panoptic and Objects365 instance segmentation datasets.
F Comparison with related contemporary work
We discuss the differences between DaTaSeg and some contemporary related work as below.
Comparison with Segment Anything [30]: SAM [30] is a related work on multi-task universal segmentation model. There are several major differences between DaTaSeg and SAM. 1) The main use-case for SAM prompt-based segmentation, i.e., the user inputs a point or box, and the model segments the object referred by the prompt. While we aim to segment all objects and stuff in a bottom-up way. 2) Besides, SAM built their own large-scale dataset with more than I billion masks (SA-IB) by designing a data engine consisting of model-assisted manual annotation and aumi-automatic stage, which is very costly; while we only utilize multiple publicly available datasets and train a joint segmentation model to improve the performance. Hence, DaTaSee and SAM's results are not comparable terms of the training data. 3) Except for the explicitly trained text-to-maa mask task, SAM's outputs are class aagnosicc binary masks (the SA-1B datasets also class-agnostic), while our model outputs the class predictions together with the mask predictions.
X-Decoder a related work on multi-task segmentation. There are multiple differences between our work and X-Decoder, which makes it hard to have a strict comparison. We detail the differences below: 1) Task and Focus: The pre-training in X-Decoder includes panoptic segmentation, referring segmentation, and image-text pairs (image-text retrieval and image captioning), which has a focus on dision-languae from ours: we cotrain on three mainstream segmentation tasks (panoptic/emmantie/nssii exclusively focus on segmentation. 2) Training paradigm: We directly cotrain on multiple datasets using shared set of parameters (single model), while the ADE and COCO results in X-Decoder Table task-specific transfer. That is, X-Decoder first pretrains on large-scale data and then fine-tunes on each target dataset using different sets of fine-tuned parameters. 3) Training data:
Comparison with X-Decoder [80]:
tasks. These tasks are very different
and
19

---

Fully-Supervised
Weakly-Supervissed Transfer
ADE
COCO
ADE
0365
Backhone
Model
semantic
panoptic
semantic - panoptic
box + instance
mloU.
PQ
PQ
mask AP
+D-S modules
48.1
46.0
26.9
10.9
ResNer50
DaTaSeg
48.1 (+0.0)
49.0 (+3.0)
29.8 (+2.9)
14.3 (+3.4)
+D-S modules
51.0
50.8
31.0
13.1
VITDet-B
Da DaTaSeg
51.4 (+0.4)
52.8 (+2.0)
32.9 (+1.9)
16.1 (+3.0)
+D-S modules
53.9
52.3
33.5
13.7
ViTDet-L
DaTaSeg
54.0 (+0.1)
53.5 (+1.2)
33.4 (-0.1)
16.4 (+2.7)
Table 12:
Comparing DaTasSeq with the alternative architecture adding all dataset-specific modules on various backbones. Results show that removing the dataset-specific modules improve performance on all datasets and all backbones. The gains are most significant on Objects365 instance and COCO panoptic segmentation (with datasct-specific modules, it cannot outperform the separately trained baseline on COCO panoptic).
X-Decoder pretrains on COCO panoptic and referring segmentation and 4M image-text pairs with a long schedule, and we quote from their paper: "all the pre-trained models are trained with 50 epochs of COCO data and roughly 45 epochs of 10 million image-text pairs". Afterwards, they fine-tune the model on COCO and ADE20K. By contrast, we only train on COCO panoptic, ADE20k semantic, and 0bjects365 v2 detection with a total of 1.8M of training images, which is much fewer than X- X-Decoder. We also do not train on large amounts of text data. 4) Model architecture: X-Decoder adopts Mask2Former. Our model architecture is different as explained in Sec. G. More importantly, we don't 't have online text encoder in our model architecture.
G Architecture change from Mask2Former
Our network architecture is based on Mask2Former [8]. However, due to hardware difference (TPU vs. GPU), we need to make modifications to Mask2Former architecture to run on our hardware. Unlike GPUs, TPUs require a static computation graph with fixed-shupe data (otherwise, it recompiles for each computation graph change, and causes significant slowdown). Therefore, we drop all the TPU-unfriendly operations in Mask2Former. In particular, we change the multi-scale deformable attention transformer [79] pixel decoder to a plain FPN [43]. The performance comparison is shown in Table 4(e) in the Mask2Former paper (51.9 PQ of deformable-attention vs. 50.7 PQ of FPN on COCO panoptic). Also, we do not use the sampled point loss [9] in either the matching loss or the training loss calculation, and use the vanilla mask loss. The performance comparison is shown in Table 5 in the Mask2Former paper (51.9 PQ of point loss vs. 50.3 PQ of mask loss). Besides, we do not use varying input size during evaluation and use fixed input size (first resize then pad).
To improve the training speed, we modify the deep supervision in the masked attention transformer decoder: for every decoder layer, the last decoder layer, but use a different prediction head. We do not use the mask predictions from the initial queries attention masks, since the predictions are not made based on the input image.
the same matching indices computed from the outputs of
H Additional implementation details
In addition to the implementation details described in Sec. 4, we provide more datails here.
Preprocessing. In order to do 1:1 Hungarian matching, we preprocess the groundtruth into the same format as the predictions described in Sec. 3.2. In panoptic segmentation, transfer the groundtruth into a set of binary masks with class labels. For "thing" categories, we group the pixels belonging to each instance into separate groundtruth mask; for "stuff" categories, we group all pixels in each stuff category as the groundtruth mask. Similarly, in semantic segmentation, we group all pixels in each category as the groundtruth mask. For instance segmentation, since we are using bounding box weak supervision, we do not need this preprocessing step. For all mask proposal groundtruth, we downsample it by 4 times, to save memory cost during training.
Training settings. We randomly scale the input image in the range of [0.1, 2.0] and then pad or crop to 1024 x 1024. For ADE20k dataset, since the image size is smaller than other datasets, we use scaling range of [0.5, 2.0]. We use the AdamW optimizer [46] with a weight decay of 0.05. We
20

---

piae
foccl
pdice
Mproj
Ace
Afacal
Adice
Aproj
ADE semantic COCO panoptic
20
0
20
5
0
1
0
0
20
5
0
0365 instance (box GT)
1
0
0
0.5
0
0
2
Table 13: The weights we use to compute the matching cost and total loss (Eqn. 4,5 in the main paper) for all training datasets. jt's are for the matching cost and X's are for the total training loss.
clip the gradients with a max norm of 0.1. The weight for the background class is set to 0.05 in Cee- The matching cost and loss weight settings for Eqn. 4,5 in the main paper are shown in Table 13. We use dataset sampling ratio of 1:4:4 for ADE semantic, COCO panoptic, and Objects365 detection. We adopt a different learning rate multiplier for each dataset: We multiply the learning rate on ADE semantic, COCO panoptic, and Objects365 detection by 3, 5, 2, respectively. Since Objects365 detection dataset has a large vocabulary with imbalanced distribution, we apply repeat factor sampling with a frequency threshold = 0.01 [18]. On ResNet50 backbones, we use a batch size of 384 and train 500k iterations, with learning rate of 3e-5. We adopt the step learning rate schedule: We multiply the learning rate by 0.1 x at the 0.9 and 0.95 fractions of the total training iterations. On the ViTDet-B backbones, we train 600k iterations with learning rate of 6.-5. On ViTDet-L, use a batch size of 256 and train 540.5k iterations with a learning rate of 4e-5. For ResNet50, we train on 64 TPU v4 chips; for ViTDet backbones, we train on 128 TPU v4 chips. All evaluations conducted on 4 V100 GPUs with a batch size of 8.
Postprocessing. We first apply the MERGE operation described in Sec. 3.2. We follow the postpro- cessing operations in [8] with some modifications: In panoptic segmentation, we use a confidence threshold of 0.85 to filter mask proposals for COCO panoptie, and set the threshold to 0.8 for ADE panoptic. For panoptic and instance segmentation, we filter out final segments whose area is smaller than 4, For instance segmentation, we return maximum of 100 instances per image with score threshold of 0.0. For all tasks, the final scores are the production of classification scores and localization scores (obtained from binary mask classification).
I Limitations
Our proposed method is not without limitations. Since we are the first work to explore universal multi-dataset multi-task segmentation models, we do not introduce the complexity for efficient framework. There are several ways to improve the model efficiency, e.g., calculating the mask loss on sampled points only [8], while we do not deploy this in framework, since it's not our key contribution. Moreover, as shown in the experiment section (Table I), our framework is orthogonal to the detailed network architectures long as the network is able output universal segmentation representation.
For weakly-supervised panoptic segmentation on ADE20k panoptic, we notice that sometimes multiple instances of the same category are not separated in the prediction. In the weakly-supervised instance segmentation results Objects365, there are some visual artifacts in the mask prediction. We believe using certain data augmentation techniques, e.g., Mix [73] or Copy-Paste [15], may farther enhance knowledge transfer between datasets of different tasks, and thus may help mitigate this issue. We leave it for future work to improve on these limitations.
J Additional qualitative results
We notice sometimes our predictions do not match the groundtruth, but it does not necessarily mean the predictions are not good. Fig. shows two examples on COCO panoptic. On the left column, DaTaSeg does a better job in segmenting 'sky' and 'tree-mereed' than the groundtruth. The classification of "pavement-merged"" is also better than 'dirt-merged, which is probably due to language ambignity: the British meaning of 'pavement" is sidewalk, while the annotator may be more used to the North American usage of 'pavement'. On the right column, DaTaSeg predicts 'window- while the groundtruth is 'rreemerreed' for the scene through the window, and both make sense due to label ambiguity. In addition, DaTaSeg is able to predict the 'tirro--stff"', 'shelf", 'light', 'bottle'. and more 'book's, which missing from the groundtruth.
We show qualitative results for the direct transfer experiments in Fig. 10,11,12,13, on PASCAL. Context 59 (PC-59), PASCAL Context 459 (PC-459), COCO semantic, and Cityscapes panoptic
21

---

background
bottle
background
chair
STOP
truck
potted plant
bed
stop sign
book
blanket
bench
light
road
nirror-tuuf
merged
window-blind
sky-atha--merge
window other
pavernent: merged
cabinet-merged
wall other merged
grass merged
rug-merged
(a) DaTaSeg predictions
background
background
STOP
truck
potted plant
stop sinn
bod
tree-merged
blanee
dy-ather-merre ged
house
dirt-morged
rraammargd
(b) Groundtruths
Figure 9: Examples COCO panoptic showing that sometimes DaTaSeg's predictions do not match the groundtruth, but does not necessarily "pavement-merged while groundtruth 'dirt-merged" (Probably because British meaning of 'pavement' sidewalk, while the annotator is more used to the North American meaning of "pavement'). Ours also segments 'tree' and 'sky' better. On the right, the definition for the scene through the window is ambiguous: Hindow-other (ours) 'tree-merged" (GT). The groundtruth misses to label several objects, while DaTaSeg recognizes objects, e.g., 'mirror-stuff'' 'light', 'shelf', 'bottle'. Legends for 'book' truncated due space limit.
they wrong. On the left, DaTaSeg's prediction
supplementary material for Table 3 in the main paper. They show DaTaSeg directly transfer to other segmentation datasets unseen during training with high quality both localization and classification. DaTaSeg performs well large-vocabulary segmentation (PC-459), and handles hard cases well (thin structures, small objects, complicated scenes, occlusions, elc.).
datasets, respectively. These results
22

---

aeroplane
sky
sky
tree
free
aeroplane
ignore
car
aeroplane
gr a5s
grass
road
ground
sky
sign
free
road
sky
bottle
Ignore
chair
bottle
diningtable
chair
diningtable
pottedplant
pottedplant
book
cabinet
cabinet
cloth
bedciothes
wall
wall
window
window
bicycle
bicycle
bottle
person
building
building
fence
flower
flower
grass
grass
sidewalk l
sidewalk
tree
Figure 10: Qualitative results of DaTaSeg directly transferring to PASCAL Context semantic dataset with 59 categories (PC-59). The results demonstrate DaTaSeg has good generalization ability. The top row shows DaTaSeg is able to segment small objects (aeroplane), and the last row indicates DaTaSeg segments fine structures (bicycles) well. For each row, the left is the input image, the middle is prediction, and the right groundtruth. With ViTDet-L backbone.
23

---

bowl
innre
cup
candle
floor
chair
flower
fork
flower
frame
fork
plate
picture
pottedplant
spoon
table
table
wall
wall
wineglass
bottle
ignore
chair
bottle
cup
floor
food
food
tork
fork
knife
knife
palio
plate
plate
table
wineglass
wineglass
bench
bench
building
building
car
dog
fence
dog
grass
fence
ground
frame
person
grass
pole
ground
road
person
sidewalk
pole
sky
tree
sky
truck
tree
bicycle
bicycle
bottle
building
building
grass
gress
mountain
mountainn
person
pole
pole
road
road
shoe
sky
sky
free
tree
Figure 11: Qualitative results of DaTaSeg directly transferring to PASCAL Context semantic dataset
with 459 categories (PC-459). The results demonstrate DaTaSeg has good generalization ability, and enables open-vocabulary segmentation. For each row, the left the input image, the middle is prediction, and the right groundtruth. With ViTDet-L backbone.

---

both
cidlvt vregen
Neo ctver ereeee
2107
Nis
ha randaug
casi
paper sopereeraa
grua
spern bel
ca
diinng
TETA
wood
addo sther
lng cranges
Sookeff verped
tabikrragg
vall-iie reiged
ni mragee
Figure 12: Qualitative results on COCO semantic dataset. DaTaSeg does high-quality semantic segmentation COCO. Note that DaTaSeg trains COCO panoptic. For each row, the left is the input image, the middle is our prediction, and the right groundtruth. Our model is with ViTDet-L backbone.
25

---

KL
Caro
soong
Figure Qualitative results DaTaSeg directly transferring Cityscapes panoptic dataset. Cityscapes focuses on street view, which is different from all our training datasets. The results demonstrate good generaliza- tion ability. For each row, the left is our prediction and the right is groundtruth. Our model is with ViTDet-L backbone.
26


Reviews:
[{"review_id": "9PdbMfnlfB", "reviewer": "Reviewer_EhPn", "summary": "This paper introduces DaTaSeg, a universal multi-dataset multi-task segmentation model. It uses a shared representation for panoptic, semantic, and instance segmentation tasks, with different techniques to address task differences. Weak supervision and knowledge sharing are employed. DaTaSeg improves performance on various datasets, especially small-scale ones. It enables knowledge transfer and open-vocabulary segmentation. An Objects365 instance segmentation set of 1,000 images will be released as a benchmark.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. It is noticed that training segmentation models on multiple datasets to obtain better results is difficult. It is more challenging to train segmentation models on multiple datasets for multiple tasks (instance segmentation, semantic segmentation, weakly supervised instance segmentation. The performance of DaTaSeg on COCO and ADE shows the effectiveness of the proposed approach\n2. The paper leverages box-level supervision to improve the segmentation performance. \n", "weaknesses": "1. [minor] The definition of mask proposal is commonly used in instance segmentation [1]. However, the description uses too many math terms to describe a simple concept that can be demonstrated by natural language, figures, and more compact math. This is not good for readers to understand.\n\n2. [major] The results of weakly supervised instance segmentation look not promising. In the previous state-of-the-art method, weakly supervised instance segmentation methods achieves 70-90% performance (mAP) as what their fully supervised versions do [2,3]. However, in the 2, the instance segmentation of Object365 looks not promising, only 10ish%. What is the upper-bound performance of O365 instance segmentation?\n\n3. [major] The approach introduces the unified mask representation without comparison or discussion with other mask representations. It is well-known that training multi-dataset or multi-tasks is not trivial, and it is better to show more training details and empirical design considerations in the paper. \n\n\n\n[1] run-length encoding: https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L265\n[2] Lan, Shiyi, et al. \"Vision transformers are good mask auto-labelers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[3] Li, Wentong, et al. \"Box2Mask: Box-supervised Instance Segmentation via Level-set Evolution.\" arXiv preprint arXiv:2212.01579 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions and the helpful feedback. We carefully address the comments below.\n\n>**W1: Mask proposal:**\n\n- We agree with the reviewer that the mask proposal is commonly used in instance segmentation. However, mask proposals are used differently in different segmentation tasks (e.g., one mask proposal corresponds to one instance mask in instance segmentation, but it may correspond to one amorphous stuff region in semantic or panoptic segmentation). Therefore, Sec. 3.2 is mainly about how we novelly apply *different* merge operations for *different* segmentation tasks. We briefly introduce the mask proposal representation in L121 in a math format, in order to use it in Eq. 1 and 2, which are our novelly proposed MERGE operation. We thank the reviewer for the suggestion and will improve the presentation.\n\n>**W2: Weakly supervised instance segmentation:**\n\nWe thank the reviewer for the question, and address it below.\n\n- First, the reason why the scores on O365 are generally low in absolute number is because: O365 has 365 categories with imbalanced distribution [A]; hence, it is more challenging than detection datasets on common categories, e.g., COCO.\n- Second, we also evaluate DaTaSeg's class-agnostic mask average recall (AR@100) on O365. DaTaSeg achieves **32.8** and **34.5** AR@100 using ResNet50 and ViTDet-B backbones, respectively, which is not low.\n- Third, since there is no O365 instance segmentation training set available (it is one of our contributions to annotate 1000 O365 images with instance segmentation annotation, used as a validation set for weakly supervised instance segmentation), we do not have a fully-supervised upper bound. That being said, we have also tried our best to find an **approximate** upper-bound performance: in Detic [A], their fully-supervised detector on Objects365 is 31.2 box AP with a Swin-B backbone, which additionally uses several techniques (e.g., Federated loss  [B] and repeated factor sampling [C]) to address the category imbalance issue. Since mask AP is generally several points lower than box AP (e.g., in Detectron2 Mask R-CNN benchmark, COCO box AP can be 5-6 points higher than mask AP), our performance on the comparable ViTDet-B backbone of 16.1 mask AP is reasonable.\n- Finally, we thank the reviewer for the pointers. We will include [2,3] in the related work section. Ours and [2,3] focus on different aspects. [2] first generates high-quality pseudo masks using groundtruth bounding boxes, and then trains an instance segmentation model. Its pipeline is more complicated than ours. [3] proposes several complex modules specifically for box-supervised instance segmentation, while our framework is simpler and tackles multiple segmentation tasks simultaneously.\n\n>**W3: Comparison with other mask representations:**\n\n- For the mask representation, we explain our motivation in L110-113. \n- Another commonly used mask representation is first getting the bounding box prediction, and then predicting the mask within the predicted bounding-box region, as in Mask R-CNN. However, this representation is not suitable for stuff categories, e.g., it is unnatural to get a single bounding box for multiple grass fields. As a result, some methods such as [D] additionally add a semantic segmentation branch for stuff categories. By contrast, in the mask representation stage, we did not treat thing and stuff categories differently. We explain the reason in L116 — different datasets have different definitions for things and stuff, e.g., the 'table' category; so treating thing and stuff mask representation separately is unsuitable. Instead, we apply different merge operations and postprocessing in the later stage.\n- We agree with the reviewer that *”it is better to show more training details”*. We have already provided more empirical design considerations in the supplementary Sec. D, E, and more implementation details in supplementary Sec. F.\n\n\n[A] Zhou, Xingyi, et al. \"Detecting twenty-thousand classes using image-level supervision.\" ECCV 2022.\n\n[B] Zhou, Xingyi, and Philipp Krähenbühl. \"Joint COCO and LVIS workshop at ECCV 2020: LVIS challenge track technical report: CenterNet2.\" 2020.\n\n[C] Gupta, Agrim, Piotr Dollar, and Ross Girshick. \"Lvis: A dataset for large vocabulary instance segmentation.\" CVPR 2019.\n\n[D] Kirillov, Alexander, et al. \"Panoptic feature pyramid networks.\" CVPR 2019."}, {"review_id": "gvc0FH3Xh7", "reviewer": "Reviewer_7Lho", "summary": "This paper proposes DaTaSeg, a general multi-dataset multi-task segmentation model. It utilizes shared representations and different pooling operations to perform panoramic, semantic and instance segmentation tasks. DaTaSeg benefits from weak supervision and knowledge transfer across datasets. It outperforms separate training on all datasets (especially smaller ones) and enables weakly supervised segmentation. The model also transfers well to unseen datasets and supports open vocabulary segmentation.", "questions": "May need proofreading:\n\nLine 35, \"which are\" -> \"which is\"\nLine 38, \"which map\" -> \"which maps\"", "limitations": "According to the F section in Supplementary Material, the computational cost is quite high which may affect the reproducibility. Consider releasing the pre-trained models in various sizes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper addresses the challenge of training a single model on multiple segmentation tasks and datasets by proposing DaTaSeg, which has the potential to save computational resources and streamline the development of segmentation models. It can also benefit from weak supervision by incorporating cheaper bounding box annotations.\nThe proposed DaTaSeg shows promising results in transferring to unseen datasets and enabling open-vocabulary segmentation.\nThe authors annotate a subset of the Objects365 dataset and plan to release it as a public benchmark for instance segmentation. This contributes to the research community by providing a standardized evaluation dataset.", "weaknesses": "The Segment Anything model (SAM) released in April  2023 should be included for comparison as it's also a universal segmentation model which has zero/few-shot capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions and the helpful feedback. We carefully address the comments below.\n\n>**Weaknesses: Comparison with SAM:**\n\nWe thank the reviewer for the suggestion. We are happy to include a comparison with the Segment Anything paper in the revision. We note that SAM is a great paper, but there are several major differences between DaTaSeg and SAM. \n  1. The main use-case for SAM is prompt-based segmentation, i.e., the user inputs a point or box, and the model segments the object referred by the prompt. While we aim to segment all objects and stuff in a bottom-up way. The only instance segmentation quantitative results reported in the SAM paper (Table 5) requires taking boxes from an external trained detector ViTDet.\n  2. Besides, SAM built their own large-scale dataset with more than 1 billion masks (SA-1B) by designing a data engine consisting of model-assisted manual annotation and semi-automatic stage, which is very costly; while we only utilize multiple publicly available datasets and train a joint segmentation model to improve the performance. Hence, our DaTaSeg and SAM are not comparable in terms of the training data.\n  3. Third, except for the explicitly trained text-to-mask task, SAM's outputs are class-agnostic binary masks (the SA-1B datasets are also class-agnostic), while our model outputs the class predictions together with the mask predictions.\n\n\n>**Questions: Proofreading:**\n\nThanks! We will fix these typos and do a thorough proofreading.\n\n\n>**Limitations: Computational cost and open-sourcing:**\n\n- We thank the reviewer for the suggestion. Indeed, we plan to open-source the code and release the models in various sizes, upon acceptance of the paper. \n- We use a longer training schedule because we are cotraining on the much larger Objects365 dataset (1,662,292 training images), which is 14 times larger than COCO. In order to balance the samples from different datasets, we train longer on COCO and ADE.\n- We have some explanation about the computational efficiency in supplementary Sec. G, including several techniques to improve it. We leave it for future work to further improve the efficiency.\n"}, {"review_id": "P84sfAl4BH", "reviewer": "Reviewer_t7xz", "summary": "[Tasks] This paper introduces DaTaSeg, a universal multi-dataset multi-task segmentation model that addresses the interconnections between panoptic, semantic, and instance segmentation tasks. \n\n[Methods] DaTaSeg utilizes a shared representation, consisting of mask proposals with class predictions, across all tasks. To handle task discrepancies, the model employs distinct merge operations and post-processing techniques tailored to each task. Additionally, DaTaSeg leverages weak supervision, enabling cost-effective bounding box annotations to enhance the segmentation model. To facilitate knowledge sharing across datasets, DaTaSeg utilizes text embeddings from the same semantic embedding space as classifiers and shares all network parameters among datasets. \n\n[Experiments] The model is trained on ADE semantic, COCO panoptic, and Objects365 detection datasets. DaTaSeg exhibits improved performance across all datasets, particularly for small-scale datasets, achieving a 54.0 mIoU on ADE semantic and a 53.5 PQ on COCO panoptic. Furthermore, DaTaSeg enables weakly-supervised knowledge transfer for ADE panoptic and Objects365 instance segmentation.\n\n[Results] Experimental results indicate that DaTaSeg scales effectively with the number of training datasets and facilitates open-vocabulary segmentation through direct transfer. \n\n[Dataset] Additionally, the authors have annotated an Objects365 instance segmentation set comprising 1,000 images, which will be released as a public benchmark.", "questions": "Most of my questions are listed in the Weaknesses section, with my main concerns focused on the technical contributions and performance comparisons with some published works. However, I have a couple of minor questions:\n\n1. Will the code and model weights be made available to the public? It would be valuable to have access to these resources for further exploration and replication of the proposed method.\n\n2. Given that this model can be trained with weakly-supervised tasks, I'm curious if further improvements in performance can be achieved by training on a larger quantity of weakly-supervised samples. Are there any trade-offs between the quality and quantity of the samples when it comes to enhancing model performance?\n\nI believe addressing these questions would provide additional insights into the practicality and potential improvements of the proposed method.", "limitations": "Yes, the authors adequately addressed the limitations", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors have conducted comprehensive experiments, showcasing the state-of-the-art performance achieved on multiple long-tailed recognition benchmarks. This highlights the robustness and effectiveness of their proposed method.\n\n2. The paper is skillfully organized, making it easy to follow. The logical structure and clear presentation enhance the reader's understanding of the research.\n\n3. Efficient Use of Supervision. While previous works have explored training universal models on multiple datasets and tasks, a notable strength of this paper is the effective utilization of weak bounding box supervision for segmentation. Compared to full mask annotations, weak bounding box supervision is a more cost-effective and practical solution. This approach makes the proposed method more accessible and applicable in real-world scenarios.", "weaknesses": "[Technical contributions on multi-task multi-task model.] One of the key contributions of this paper is the joint training of multiple datasets and multiple tasks within a unified framework. However, it should be noted that this aspect has already been explored by [1]. The referenced work employs a single set of parameters pre-trained for Semantic/Instance/Panoptic Segmentation, Referring Segmentation, Image Captioning, and Image-Text Retrieval tasks. Therefore, it might be necessary for the authors to clarify the differences between this work and X-Decoder. \n\n[Technical contributions on text embedding classifier.] The utilization of a text embedding classifier has also been explored in previous works, such as [2]. In this work, an image encoder is trained to encode pixel embeddings, while CLIP text embeddings are employed as the per-pixel classifier. The key ideas are quite similar, although there are some differences on how to better leverage the text embeddings. \n\n[Lack of comparisons with some published works and subpar performance compared to state-of-the-art methods.] The paper primarily compares the model's performance against methods designed or trained solely on a single task, referred to as \"Separate\" in the paper. However, it is worth noting that DaTaSeg benefits from training on a significantly larger sample size compared to these listed works. Consequently, the observed performance gains over the \"Separate\" models are not unexpected. Furthermore, there are published works, such as X-Decoder [1], that jointly train models on multiple datasets and tasks, yielding superior performance on many benchmarks compared to the reported results in this paper.\n\nOn ADE semantic (mIoU): X-Decoder achieves 58.1 compared to DaTaSeg's 54.0; on COCO Panoptic (PQ): X-Decoder outperforms DaTaSeg with 56.9 versus 53.5. In most benchmarks, DaTaSeg performs worse than X-Decoder.\n\n[1] \"Generalized decoding for pixel, image, and language.\" Zou, Xueyan, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai et al. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15116-15127. 2023.\n[2] \"Language-driven Semantic Segmentation.\" Boyi Li and Kilian Q Weinberger and Serge Belongie and Vladlen Koltun and Rene Ranftl. International Conference on Learning Representations. 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and helpful comments, and for recognizing multiple strengths of our submission. We carefully address the comments and questions below.\n\n>**W1: Technical contributions on multi-task multi-task model:**\n\n- We train a single universal segmentation model on multiple segmentation tasks and multiple segmentation datasets, in one stage. As reviewer EhPn pointed out, we explore a very challenging task. However, the Table 1 results in the X-Decoder paper is **\"task-specific transfer\"**, which follows the “first pre-training then fine-tuning” paradigm to achieve the best performance on each dataset. It also requires separate fine-tuned model weights for each dataset.\n- The joint pre-training in X-Decoder only includes panoptic segmentation, referring segmentation, and image-text pairs, which has a focus on vision-language tasks. These tasks are very different from ours: we cotrain on three mainstream segmentation tasks (semantic/instance/panoptic) and exclusively focus on segmentation. This difference also leads to significantly different training data.\n- Besides, we novelly propose to include a simple weakly-supervised instance segmentation training using bounding-box supervision to increase the training data, since mask supervision is expensive and thus limited in scale.\n- We'll cite X-Decoder and include the comparison in the revision. \n\n\n>**W2: Technical contributions on text embedding classifier:**\n\n- We thank the reviewer for the question. We note that we have never claimed that it is our major novelty to use a text embedding classifier. Instead, our novelty lies in using the text embedding classifier for knowledge sharing among different datasets (L186-188). On the other hand, open-vocabulary segmentation, like [2], mainly uses it to support arbitrary text queries. In L190, we have acknowledged the same technique is used in open-vocabulary segmentation, and discussed the difference.\n\n>**W3: Comparisons with published works:**\n\n- We appreciate the comments, and agree that X-Decoder is a great work. However, there are multiple differences between our work and X-Decoder, which makes it hard to have a strict comparison. We detail the differences below:\n  1. **Training paradigm:** As we pointed out in W1, we directly cotrain on multiple datasets using a shared set of parameters (single model), while the ADE and COCO results in X-Decoder Table 1 is **task-specific transfer**. That is, X-Decoder first pretrains on large-scale data and then fine-tunes on each target dataset using different sets of fine-tuned parameters.\n  2. **Training data:** X-Decoder pretrains on COCO panoptic and referring segmentation and 4M image-text pairs with a long schedule, and we quote from their paper: *\"all the pre-trained models are trained with 50 epochs of COCO data and roughly 45 epochs of 10 million image-text pairs\"*. Afterwards, they fine-tune the model on COCO and ADE20k. By contrast, we only train on COCO panoptic, ADE20k semantic, and weakly-supervised Objects365 v2 detection with a total of 1.8M of training images, which is much fewer than X-Decoder.\n  3. **Model architecture:** X-Decoder adopts Mask2Former. Our model architecture is different as explained in supplementary Sec. E.\n\n- Finally, we note that X-Decoder is a concurrent work that was published in CVPR 2023, June 2023, while the NeurIPS submission deadline was in May 2023.\n\n- Therefore, we think comparing with the \"separated\" baselines under the same setting is reasonable to show our proposed multi-dataset multi-task training scheme improves segmentation performance.\n\n>**Q1: Open-sourcing:**\n\n- We thank the reviewer for the question. Indeed, we will release the code and model weights to the public, upon acceptance of the submission.\n\n>**Q2: Trade-offs between the quality and quantity of the training samples:**\n\n- Thanks for the great suggestion. In Table 2 of the submission, we show the results with and without cotraining with Objects365. The performance is generally better when cotraining with O365.\n- In addition, we experimented with cotraining on different portions of the O365 dataset: we cotrain DaTaSeg on COCO panoptic, ADE semantic, and 10%/25%/50%/100% of O365 training data, on the ResNet50 backbone. We show the results in the table below:\n\n| Ratio of O365 training data | ADE semantic | COCO panoptic | ADE panoptic | O365 instance |\n|--------------------:|:------------:|:-------------:|:------------:|:-------------:|\n|                 10% |     48.5     |      48.7     |     30.0     |      10.4     |\n|                 25% |     48.6     |      48.3     |     30.6     |      12.0     |\n|                 50% |     47.7     |      48.5     |     29.8     |      13.1     |\n|                100% |     47.2     |      48.7     |     29.4     |      14.5     |\n\n- The results show that when increasing the number of O365 weakly-supervised samples, O365 performance increases, COCO panoptic performance is not affected, and ADE20k performance slightly decreases. Overall, the gains are larger than the losses. We will include this interesting analysis in the revision.\n"}, {"review_id": "1CPff2xw3b", "reviewer": "Reviewer_LhuR", "summary": "This paper proposes DaTaSeg, a universal multi-dataset multi-task segmentation model. DaTaSeg uses a shared representation and different merge operations and post-processing for different tasks. Weak-supervision is employed for cheaper bounding box annotations and knowledge is sharing across different datasets with text embeddings from the same semantic embedding space and shared network parameters. A subset of the Objects365 validation set is annotated for instance segmentation. Experiments shows that DaTaSeg gets improved performance on dataset-specific models and enables weakly-supervised knowledge transfer. DaTaSeg also scales with the training dataset number and enables open-vocabulary segmentation. ", "questions": "This paper proposes the multi-dataset multi-task segmentation model. The hyperparameters λ and μ in the equations (4) and (5) are crucial for the training. Could you provide the experiments on the impact of hyperparameters on the model performance?The authors do not discuss Limitations in the paper.", "limitations": "The authors do not discuss Limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper is easy to follow.\n2) The method proposed in this paper will be useful in the future as it achieves the segmentation task under the multi-task multi-dataset setting.\n", "weaknesses": "The ablation study in the paper is somewhat not sufficient. The hyperparameters λ in equation (4) and μ in equation (5) are not provided and there needs to be experiments on the impact of hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions and the helpful feedback. We carefully address the comments below.\n\n>**Weaknesses and Questions: Hyperparameters and ablation study:**\n\n- Our settings of $\\lambda$ and $\\mu$ closely follow Maskformer [A] and Mask2former [B], and we add the weights for the bounding-box projection loss $L_{proj}$ for the O365 dataset. \n- We did an ablation study on the weights for the projection loss $L_{proj}$, $\\lambda_{proj}$ and $\\mu_{proj}$, using ResNet50 backbone, as shown in the table below:\n\n| $\\lambda_{proj}$ | $\\mu_{proj}$ | ADE semantic | COCO panoptic | ADE panoptic | O365 instance |\n|-----------------:|-------------:|:------------:|:-------------:|:------------:|:-------------:|\n|              5.0 |          1.0 |     45.5     |      48.3     |     28.4     |      12.5     |\n|              2.0 |          0.5 |     47.2     |      48.7     |     29.4     |      14.5     |\n\n- The results indicate that the 2nd setting (our final setting) has a better performance on all datasets.\n- Finding the optimal weights for multiple losses is an interesting but challenging problem, and we leave it for future work.\n- We will add the above table, and the tables in our rebuttal to reviewer t7xz and 8rwi to ablation studies in the revision.\n\n>**Limitations:**\n\n- We have a \"Limitations\" section in supplementary Sec. G.\n\n\n[A] Cheng, Bowen, Alex Schwing, and Alexander Kirillov. \"Per-pixel classification is not all you need for semantic segmentation.\" NeurIPS 2021.\n\n[B] Cheng, Bowen, et al. \"Masked-attention mask transformer for universal image segmentation.\" CVPR 2022.\n"}, {"review_id": "zKtieJJWTd", "reviewer": "Reviewer_8rwi", "summary": "This paper proposes the universal model for image segmentation using multi-dataset multi-task training. By using universal segmentation representations at the entity (thing or stuff) level, the paper use merge operation for different segmentation task. Experiments show the effectiveness of the proposed method.", "questions": "1, Could the author try the other sampling strategy in multi-dataset training? In the paper, it would lead to the imbalance of dataset sampling. \n\n2, The performance improvement on COCO mainly comes from using the objects365 dataset. Direct pretraining object365 and then finetuning to COCO could perform better than the proposed method in the paper.", "limitations": "This paper cannot solve the annotation inconsistency of multi-dataset training. For example, the `window' defined in COCO and ADE20K are different where COCO stuff but ADE20K thing.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1, The paper is well-written and easy to understand.\n\n2, The extensive experiments show the effectiveness of multi-dataset multi-task training.", "weaknesses": "1， Limited novelties. The main contribution of this paper is multi-dataset multi-task training. However, there is no obvious difference to OneFormer except for the multi-dataset training. To solve the label space conflict of multi-dataset training, the paper directly uses the language embeddings of the language model that has been widely explored.\n\n2, Merging universal segmentation representations seems like following the fine-to-coarse pipeline. There is no obvious clue to prove whether this fine-to-coarse is better than the coarse-to-fine pipeline.\n\n3, Weakly-supervised instance segmentation module also has limited novelties by just using the projection loss of BoxInst to add the larger dataset objects365.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback.\n\n>**W1: Novelties and difference to OneFormer:**\n\nThere are multiple significant differences between DaTaSeg and OneFormer. Though we discussed the most significant difference in L27 and L90, We detail more differences below:\n  1. Multi-dataset training is challenging (Reviewer EhPn) and using the same set of parameters (i.e., a single model) for multiple datasets saves computational resources (Reviewer 7Lho). OneFormer doesn't support multi-dataset training while DaTaSeg does. We emphasize that it's non-trivial to develop a single model for multi-dataset training, while attaining good performance on all datasets. The final simple yet effective design is in fact optimized from another more complicated design, as explained in supplementary Sec. D.\n  2. With our multi-dataset training, we 1) leverage knowledge from multiple datasets, containing different types of annotations, to improve segmentation performance on all datasets; 2) enable weakly-supervised instance and panoptic segmentation through knowledge transfer; 3) directly enable open-vocabulary segmentation. All these points are missing from OneFormer.\n  3. There are significant differences between DaTaSeg's and OneFormer's multi-task training strategy: we handle different segmentation tasks by post-processing (our MERGE operation), and OneFormer handles different segmentation tasks by defining a task prompt and using task-specific queries. This means OneFormer would have different model outputs for different tasks, while our model always produces the same, universal mask representation. Advantages of our framework include enhancing knowledge sharing among tasks, and potential better generalization to more future segmentation tasks.\n  4. Different supervision: OneFormer only uses fully-supervised training; while DaTaSeg explores weaker supervision. DaTaSeg achieves weakly-supervised instance segmentation using only bbox annotation.\n  5. Different network architectures: Except from the above differences, OneFormer first obtains a set of text queries using a text encoder and then applies a contrastive learning between the text queries and the object queries. DaTaSeg does not use such text queries.\n\n>**W2: Fine-to-coarse pipeline:**\n\n* The fine-to-coarse pipeline of merging universal segmentation representation is straightforward, simple, and effective.\n* Designing a coarse-to-fine pipeline is challenging, since we need to decompose semantic segmentation into meaningful instance segmentation masks. \n* Our fine-to-coarse pipeline is non-ambiguous, while ambiguity is hard to eliminate in a coarse-to-fine pipeline.\n\n>**W3: Novelties in weakly-supervised instance segmentation module:**\n\n* Our novelty lies in the proposed multi-dataset multi-task universal segmentation framework. We aim to design a simple, yet effective network architecture, and the projection loss from BoxInst fits our goal well. \n* We propose to use a universal segmentation representation (Sec. 3.1) with a fully-shared network architecture (Sec. 3.4), and apply different losses to different types of segmentation tasks.\nWith the projection loss, we perform bounding-box-supervised instance segmentation *without any modification to the architecture*, which also performs well empirically.\n* To the best of our knowledge, we are the first to apply the projection loss in the multi-dataset multi-task segmentation setting.\n\n>**Q1: Sampling strategy in multi-dataset training:**\n\n* Our sampling strategy (L194-L199) avoids the imbalance of dataset sampling by specifying the per-dataset sampling ratio. The proportion of training samples coming from each dataset in the whole training process is determined by that sampling ratio.\n* In our main results, the sampling ratio is 1:4:4 for ADE:COCO:O365. We ablate the sampling ratio on a Resnet50 backbone and show the results in the table below.\n\n| Sampling ratio | ADE semantic | COCO panoptic | ADE panoptic | O365 instance |\n|:--------------:|:------------:|:-------------:|:------------:|:-------------:|\n|      1:4:4     |     48.1     |      49.0     |     29.8     |      14.3     |\n|      1:2:2     |     46.8     |      48.6     |     29.1     |      12.8     |\n|      1:1:1     |     45.3     |      48.0     |     28.4     |      13.7     |\n\n* Results show that our adopted sampling ratio is better than the other sampling ratios.\n\n>**Q2: \"Improvement on COCO mainly comes from Objects365\":**\n\n* In Table 2 of the submission, we have shown the results of training DaTaSeg on all combinations of the three datasets. With or without cotraining on Objects365, COCO performance is approximately the same. Hence, the performance improvement on COCO does not mainly come from using the Objects365 dataset.\n* We also experimented with training DaTaSeg from a checkpoint pretrained on O365 (3rd row in Table 2 in our submission), using ResNet50 backbone. The results are shown in the table below:\n| Pretrain data | ADE semantic | COCO panoptic | ADE panoptic | O365 instance |\n|--------------:|:------------:|:-------------:|:------------:|:-------------:|\n| IN+O365 |     47.6     |      48.7     |   28.8   |      13.6     |\n|      IN |     47.2     |      48.7     |     29.4     |      14.5     |\n* Results demonstrate that pretraining on O365 does not improve the performance for DaTaSeg. We hypothesize it's because weak bouding box pretraining doesn't help segmentation-only model, like DaTaSeg.\n\n>**Limitations: Annotation inconsistency of multi-dataset training:**\n\n* For stuff categories, we apply MERGE operation on mask predictions to obtain final predictions, during both training and inference. Therefore, our method can predict the window category as “stuff” category in COCO and “thing” category in ADE20k. \nThis inconsistency is one motivation behind our design (L117-119): We first adopt a universal segmentation representation for different tasks, and then treat them differently in merging and postprocessing."}]

Author rebuttal:

Meta-review: This paper proposes an approach to unify different segmentation datasets and different tasks like panoptic, semantic and instance segmentation tasks.  This is a challenging setting and the proposed DaTaSeg demonstrate promising results.  Concerns were raised regarding comparisons with alternative methods, implementation details and discussion with some up-to-date references.

The authors made good effort in  the rebuttal and clarified some technical details and added more experimental results. The AC recommends acceptance, and encourages the authors to revise the paper accordingly.
Decision: Accept (poster)