LMSeg: Language-guided Multi-dataset Segmentation
Authors: Qiang Zhou
Yuang Liu
Chaohui Yu
Jingliang Li
Zhibin Wang
Fan Wang
Venue: ICLR 2023 poster (2023)
Keywords: Segmentation
Multi-dataset
Vision-language

Abstract: It’s a meaningful and attractive topic to build a general and inclusive segmentation model that can recognize more categories in various scenarios. A straightforward way is to combine the existing fragmented segmentation datasets and train a multi-dataset network. However, there are two major issues with multi-dataset segmentation: (i) the inconsistent taxonomy demands manual reconciliation to construct a unified taxonomy; (ii) the inflexible one-hot common taxonomy causes time-consuming model retraining and defective supervision of unlabeled categories. In this paper, we investigate the multi-dataset segmentation and propose a scalable Language-guided Multi-dataset Segmentation framework, dubbed LMSeg, which supports both semantic and panoptic segmentation. Specifically, we introduce a pretrained text encoder to map the category names to a text embedding space as a unified taxonomy, instead of using inflexible one-hot label. The model dynamically aligns the segment queries with the category embeddings. Instead of relabeling each dataset with the unified taxonomy, a category-guided decoding module is designed to dynamically guide predictions to each dataset’s taxonomy. Furthermore, we adopt a dataset-aware augmentation strategy that assigns each dataset a specific image augmentation pipeline, which can suit the proper-
ties of images from different datasets. Extensive experiments demonstrate that our method achieves significant improvements on four segmentation datasets and three panoptic datasets, while the ablation study evaluates the effectiveness of each component.


Published as a conference paper at ICLR 2023
LMSEG: LANGUAGE-GUIDEI T SEGMENTATION
Qiang Zhou', Yuang Liu', Chaohui ,u', JingLiang Li', Zhibin Wang', Fan Wang Alibaba Group "Eatt China Normal University
3 Uniivessity of the Chinese Academy of Sciences
{ [jianchong. 2q, huakun. ych, zhibin waz, fan. (aln::/aalbbbbaiinn com f frank iu624 4 @gmail. com, Lijingliang20emaill ucas.ac.cm
ABSTRACT
It's meaningful and attractive topic to build general and inclusive segmentation model that can recognize more categories in various scenarios. A straightforward way is to combine the existing fragmented segmentation datasets and train multi- dataset network. However, there are two major issues with multi-dataset segmen- tation: (i) the inconsistent taxonomy demands manual reconciliation to construct a unified taxonomy; (ii) the inflexible one-hot common taxonomy causes time- consuming model retraining and defective supervision of unlabeled categories. In this paper, we investigate the multi-dataset segmentation and propose a scal- able Language-guided Multi-datasct Segmentation framework, dubbed LMSeg, which supports both semantic and panoptic segmentation. Specifically, we intro- duce pre-trained text encoder map the category names to text embedding space as a unified taxonomy, instead of using inflexible one-hot label. The model dynamically aligns the segment queries with the category embeddings. Instead of relabeling each dataset with the unified taxonomy, a category-guide ing module designed to dynamically guide predictions each dataset's taxon- omy. Furthermore, we adopt dataset-aware augmentation strategy that assigns each dataset a specific image augmentation pipeline, which can suit the properties of images from different datasets. Extensive experiments demonstrate that our method achieves significant improvements on four semantic and three panoptic segmentation datasets, and the ablation study evaluates the effectiveness of each component.
decod-
INTRODUCTION
Image Segmentation has been a longstanding challenge in computer vision and plays pivotal role wide variety of applications ranging from autonomous driving (Levinson et al., 2011] I: Maurer et al. 2016) to remote sensing image analysis (Ghassemian 2016). Building general and inclusive segmentation model is meaningful to real-world applications. However, due to the limitation of data collection and annotation cost, there are only fragmented segmentation datasets of various scenarios available, such as ADE20K (Zhou et al. (017), Cityscapes (Cordts et al.) 2016). COCO- stuff (Caesar al. 2018), etc. Meanwhile, most work of segmentation (Long et al. 2015 Chen iet al., 2018 Zheng et al. 2021) focus on single-dataset case, and overlook the generalization of the deep neural networks. Generally, for different data scenarios, new set of network weights are supposed to be trained. As compromise of expensive images and annotations for all scenarios, how construct multi-dataset segmentation model with the existing fragmented datasets attractive for supporting more scenarios.
The primary issue of multi-dataset learning is the inconsistent taxonomy, including category coin- cidence, ID conflict, naming differences, efc. For example, the category of "person" in ADE20k dataset are labeled as "person" and "rider" in Cityscapes dataset. As shown Figure Ia). Lambert et al. 2020) manually establish unified taxonomy with the one-hot label, relabel each dataset, and then train a segmentation model for all involved datasets, which is time-consuming and error-prone.
"Corresoonding author.

---

Published as a conference paper at ICLR 2023
Semantic & Panoptic segmentation
Head
Head-1
Head-2
Text embedding
Query embedding
Encoder- Decoder
Encoder- Decoder
Text Encoder
Encoder- Decoder
Decoder
Class
N queries
Image
Unified Taxonomy
D1
D2
D1
D2
DI
D2
(a) manual taxonomy
(b) multi-head
(c) language-guided (Ours)
Figure 1: Comparison of different multi-dataset segmentation approaches.
Moreover, the one-hot taxonomy is inflexible and unscalable. When extending the datasets or cat- egories, the unified taxonomy demands reconstruction and the model requires retraining. A group of advanced researches (Wang al.) 2022) utilizes multi-head architecture to train weight-shared secoder-decoder module and multiple dataset-specific headers, as shown in Figure [b). The multi- head approach is simple extension of traditional single-dataset learning, not convenient during inference. For example, to choose the appropriate segmentation head, which dataset the test image comes from needs to predefined or specified during inference.
To cope with these challenges, propose language-guided multi-dataset segmentation (LMSeg) framework that supports both semantic and panoptic segmentation tasks (Figure [[c).. On the one hand, in contrast to manual one-hot taxonomy, introduce mre-trained text encoder to automat- ically map the category identification to a unified representation, i.e., text embedding space. The image encoder extracts pixel-level features, while the query decoder bridges the text and image encoder and associates the text embeddings with the segment queries. Figure 2 depicts the core of text-driven laxonomy for segmentation. As can see that the embeddings of categories reflect the semantic relationship among the classes, which cannot be expressed by one-hot labels. Thus, the text-driven taxonomy can be extended infinitely without any manual reconstruction. On the other hand, instead of relabeling each dataset with unified taxonomy, we dynamically redirect the model's predictions each dataset's taxonomy. To this end, we introduce category- guided decoding (CGD) module to guide the model to predict involved labels for the specified taxonomy. In addition, the image properties of different datasets are various, such as resolution, style, ratio, etc. And, applying appropriate data augmentation strategy is necessary. Therefore, design dataset-aware augmentation (DAA) strategy to cope with this. nutshell, our contributions are four-fold:
We propose novel approach for multi-dataset semantic and panoptic segmentation, using text-query alignment to address the issue of taxonomy inconsistency.
To bridge the gap between cross-dataset predictions and per-dataset annotations, we de- sign category- guided decoding module to dynamically guide predictions to each dataset's taxonomy.
A dataset-aware augmentation strategy introduced to adapt the optimal preprocessing pipeline for different dataset properties.
The proposed method achieves significant improvements on four semantic and three panop- tic segmentation datasets.
2 RELATED WORK
2.1 SEMANTIC SEGMENTATION
As a dense prediction task, semantic segmentation plays a key role in high-level scene understand- ing. Since the pioneering work of fully convolutional networks (FCNs) (Long al.) 2015), pixel-
2

---

Published as a conference paper at ICLR 2023
COCO-stuff-10
Mapillary Vistas
Railroad
Rail Track
clothes
On Rails
cloth
Tuptop
keybourd
A AD220k
surtboard
apparel
snowboard
computer
skateboard
Text embedding space
Figure 2: t-SNE (van der Maaten & Hinton 2008) visualization of category embeddings for several semantic segmentation datasets using CLIP's text encoder for feature extraction. The text embedding space of CLIP is suitable unified taxonomy, with semantically similar categories holding closer text embeddings.
wise classification has become the dominant approach for deep learning- based semantic segmenta- tion. After FCNs, most semantic segmentation models focus on aggregating long-range context in the final feature map. These methods include atrous convolutions with different atrous rates (Chen jt al. 2018 2017), pooling operations with varying kernel sizes (Zhao et al.) (2017). 7), and variants of non-local blocks (Fu al. 2019; Yuan et al. 2021 Huang et al. 2019 Wang al. 2018). More recently, SETR (Zheng et al. 2021) and Segmenter (Strudel et al. 202 replace the traditional convolutional backbone with Vision Transformers (ViT) (Dosovitskiy et al. 2021 to capture long- range context from the first layer. Pixel-wise semantic segmentation methods are difficult to extend to instance-level segmentation. Some mask classification-basel methods (Carion et al. 2020 Wang et al. 202T) have recently emerged to unify semantic and instance segmentation tasks. The most representative work is MaskFormer (Cheng al., [02T 2022), which solves these two segmenta- tion tasks in unified manner. However, these segmentation methods typically follow the setting of single-dataset training and can maintain high accuracy on other datasets without finetuning.
2.2 MULTI-DATASEST TRAINING
Compared to single-dataset learning, recently multi-dataset learning has received increasing atten- tion in consequence
robustness and generalization. Perrett & Damen (2017) apply multi- dataset learning to action recognition during the pre-training stage. To settle the challenge of multi- dataset object detection, Zhao et al. (2020) propose a pseudo-labeling strategy and Yao et al. (2020) propose dataset-aware classification loss. To avoid manually building unified taxonomy. Zhou ctt al. (2022c) propose a formulation to automatically integrate the datasct-specific outputs of the partitioned detector into unified semantic taxonomy. For multi-dataset semantic segmentation, MSeg (Lambert et al.) 2020) manually creates a common taxonomy to unite segmentation datasets from multiple domains. (Shi et al.] (2021) first pre-trains the network on multiple datasets with a contrast loss and then fine-tunes it on specific datasets. Based on the multi-head architecture,
and
CDCL (Wang et al. 2022) proposes a dataset-aware block to capture the heterogeneous statisties of different datasets. Yin et al. (2022) utilize text embedding to improve the zero-shot performance on semantic segmentation and claim to adopt sentences instead of category names for better text embedding. In contrast to Yin et al. (2022), we focus on more general multi-dataset segmentation, including semantie and panoptic segmentation, and further propose solutions for the problem of incomplete annotation in multi-dataset training.
2.3 VISION-LANGUAGE PRE-TRAINI
Vision- LaniuaL Pre- fre-training (VLP) has achieved significant progress in the last few years, which aims jointly obtain a pair of image and text encoders that can be applied to numerous multi-modal tasks. CLIP (Rafford et al.) 2021) is a landmark work of VLP, which jointly trains the encoders on 400 million image-text pairs collected from the web. Following CL.P, many researchers attempt
3

---

Published as a conference paper at ICLR 2023
classffcation
building
wky
Text Encoder
anbedklings (K+])eC
prodiction
Xx(K (1)
Leamable
NXCC
Se segmenttztinn
Datnset-1
X6
N queris
Category Eaidgary-guided guideed Decoder
Dataset-N
Panapei segmentation
Dataset-1
MLP NKC
binury mask loss
: Dataset-N
Image Encoder
CxddoW
N hiitary mask peediction
decoder
NsNfoW
Iasgee
Post
Figure 3: LMSeg: language-guides multi-dataset segmentation framework.
transfer the CLIP model to downstream tasks. Zhou et al. 2002bb Gao al. 22021 Zhang et al. (2021) show that CLIP can help to achieve state-of- of-ec--f-the-att art performance on few-shot or even zero-shot classification tasks. DenseCLIP (Rao al. 2022) applies CLIP to dense prediction tasks via language-guided fine-tuning and achieves promising results. We claim that the text embedding space of CL.IP can regarded as a unified taxonomy, as shown in Figure and propose text-query alignment instead of softmax classifier for multi-dataset segmentation.
METHOD
3.1 OVERVIEW
In this section, we introduce the proposed LMSeg framework, new sagguage-guided multi-dataset segmentation framework, supporting both semantic and panoptic segmentation under multi-dataset learning. As shown F the inputs consist of an image and set of class names correspond- ing to the dataset to which the image belongs. The LMSeg is decomposed of encoder-decoder pixel feature extractor, a pre-trained text encoder, a Transformer decoder with category-guis ule and dataset-aware augmentation strategy. The image is first preprocessed using the proposed dataset-aware augmentation strategy, and then image features are extracted through the image en- coder and pixel decoder. The class names are mapped text embeddings by the pre-trained text encoder. The category-guide guidded decoding module bridges the text embeddings and the image features semantic and panoptic mask predictions. We detail each component of framework in the following.
mod-
3.2 TEXT ENCODER AND PIXEL FEATURE EXTRACTOR
The text encoder takes the class names of dataset input and outputs the corresponding text em- beddings e E IR 11xx for that dataset, where K₃ is the number of classes of the k-th dataset, and C, is the text embedding dimension. The implementation the text encoder follows CLIP (Rad- ford et al.) 2021). During the training phase, the parameters of the text encoder initialized with a pre-trained CLIP model and fixed without parameter update. The text encoder's 's output dimension C, is incompatible with the segment query embedding, and use linear adapter layer adjust the dimension of the text embeddings. It is worth noting that we can store the text embeddings of class names after training. That is, by reducing the overhead introduced by the text encoder, the inference time of LMSeg is barely increased compared to other segmentation frameworks (Cheng al. 202I). Existing work show that, text prompt (e.g., "a photo of [class name]") can improve the transfer performance of pre-trained CLIP models. Inspired by CoOp (Zhou al. 2022b), when gen- erating the text embedings & we utilize learnable prompt, namely "[v]ivv]e... [vl. name]", where L is the length of the learnable vectors and is set to 8 by default. Note that the learnable vectors are shared across all datasets.
The pixel feature extractor takes an image of size 3 H x W as input. The image encoder first converts the image into low-resolution image feature map F e RC*x xw' and then the pixel decoder gradually upsamples the feature map to generate per-pixel embeddings épixel E CCC.HxW
4

---

Published as a conference paper at ICLR 2023
Text guuded prodictions
X6
IFN
augmentation A
Test Cree- Alteniinn
Text
augmentation A
Image encoder
embeddings
Visuni-query Cross- Atleution
Imsage features
Atteution
augmentation A,
Aqueries
Batch of samples
Data augmicatation
Figure 4: Sategory-guided decoding (CGD).
Figure Dataset aware augmentation (DAA).
where C. is the embedding dimension. Finally, we obtain each binary mask prediction my E [0, 1] 1] via dot product between the i-th segment query embedding and the per-pixel embed- dings Episel- The image encoder can use arbitrary backbone models, not limited to the ResNet (He
al. 2016 as we use in this work.
3.3 TEXT-QUERY ALIGNMENT
In multi-dataset segmentation, establishing unified taxonomy is often essential step, which is usually cone-consuning and error- prror-prone, especially as the number categories increases. this subsection, we introduce the proposed text-query alignment, which uses the text embedding unified taxonomy representation without manually construction.
Specifically, remove the static classifier layer (usually connecco layer), which transforms the segment query embeddings into the final classification logits. Instead, we align the segment query embeddings Equery NC and the text embeddings εςet E (K++)) via a contrastive loss. Taking the samples of k-th dataset as example, we dot product the text embeddings and query embeddings to predict the classification logits p* (R-ex((e+1) for N segment queries:
Equery (Etee))
pa
(1)
where is the temperature parameter and is set to 0.07 by default. The Equery E N denotes N query embeddings with dimension C, which is the output the category-guided decoding (CGD) module Figure The e [(K++))xC represents the text embedding for the K2 categories of the k-th dataset (including one "background" category), and is the output of the text encoder module in Figure The dynamic classification logits pa are supervised with a contrastive objectives Let as:
N-1 Lu
-log exp(pf.+) E exp(p!.))
(2)
N
1=0
where the outer sum is over N segment queries. For the i-th query, the loss the log loss a (Kg + 1)-way softmax-based classifier that tries to classify the i-th query j+.
3.4 CATRGORY-GUIDDED DECODING MODULE
The text-query alignment does not entirely address the issue of taxonomy differences in multi- dataset segmentation. For example, the query embedding corresponding
human might be forced to align with the text embedding of "person" or "rider" in the Cityscapes dataset but only to "person" the ADE20K dataset. That is, for input images, the alignment targets of query embeddings are non-deterministic (varies with the dataset), which affects the performance of multi-dataset segmen- tation, as our experiments demonstrate. Instead of relabeling each dataset with unified taxonomy, propose simpler approach that dynamically redirects the model's predictions to each dataset's taxonomy. Through prediction redirection, we can arbitrarily specify the categories that the model the original annotations of each dataset.
needs to predict so that
5

---

Published as a conference paper at ICLR 2023
Tocal training cost
Backhone
Model
Pre
M.D.T
Per dataset mloo
Average mloU
ADE20K
Cityscapes
COCO-Stti Stuff
Mapillary Vistas
Muskfarmer
ImageNet
49.80
160k
R50
MuskFarmer
CLIP
39.75
77:30
38.50
47.37
50.73
LMScg (Ours)
CLIP
41.37
79.15
39.01
50.82
52.58
Maskforrmrr
ImageNett
51.47
320k
R50
MaskFarmer
CLIP
43.18
78.20
38.69
52.15
53.05
LMSeg (Ouns
CLIP
44.89
79.81
39.21
52.27
$4,04
MaskFormer MaskFormer
InoggeNet
52.63
640k
R50
CLIP
45.66
77.95
38.16
53.16
53.73
LMSeg (Ours)
CLIP
45.16
80.93
38.60
54,34
54.75
Table I: Semantic segmentation accuracy (mloU) compared with single-dataset training models. M.D.T denotes multi-datasct training.
We propose a category-guide decoding module to dynamically adapt to classes to be predicted by the model, as shown in Figure 4 The decoder module follows the standard architecture of the transformer, using multi-head self- and cross-attention mechanisms and an FFN module to transform N segment queries Equer. The self-attention to query embeddings enables the model to make global inferences for all masks using pairwise relationships between them. The cross-attention between query embeddings and image features F is able to use the entire image as context. Finally, the cross- attention between query embeddings 'query and text embeddings fiet guides the query embeddings to the classes corresponding to the input text embeddings.
We use 6 decoder modules with 100 segment queries by default. The N segment queries are initial- ized as zero vectors, each associated with a learnable positional encoding.
3.5 DATASET-AWARE AUGMENTATION
Different datasets usually have different characteristics (e.g., image resolution) and therefore differ- ent data augmentations are used for training in general. For example, in MaskFormer (Cheng et al.) 202T], a crop size of 512 1024 used for Cityscapes, while crop size of 512 512 for ADE20K. For multi datti- training, MSeg (Lambert et al. 2020) first upsamples the low-resolution training images with super-resolution model, then uses the same data augmentation to train all datasets. Un- like MSeg, we propose a dataset-aware augmentation strategy, shown in Figure []] which is simpler and more scalable. Specifically, for each training sample, we determine which dataset the sample comes from and choose the corresponding augmentation strategy A,. Furthermore, the dataset- aware augmentation strategy allows us to make fair comparisons with single-dataset training models while keeping the data augmentation for each dataset the same.
3.6 TOTAL TRAINING OBJECTIVE
The total training objective of LMSeg consists of a contrastive loss L. and two binary mask losses, focal loss Liccal (Lin al., 2017) and dice loss Ldice (Milletari et al. 2016), which are applied to the batch of samples from multiple datasets:
L B- No (Lel + Atocal Lrocal + Adice Laice)
(3) where M is the number of datasets. 2 is the number of training samples from the k-th dataset. The hyper-parameters Afecal and Adice are set to 20.0 and 1.0 by default. The weight for the "no object" (9) in the contrastive loss La is set to 0.1.
Before computing the loss function, we have figure out how to assign the ground truth targets to the model's 's predictions since LMSeg outputs N unordered predictions. We take one image from the k-th dataset as an example to illustrate how the ground truth is assigned during training. We denote 7 = {(c mn()) )i = 1,... .N } as the ground truth set of masks in the image, where ot is the label and m. represents the binary mask. LMSeg infers fixed-size set of N unordered predictions = {pp m² j/ 1, ... N for the input image. Assuming that N is larger than the number of target masks the image, we pad set ÿ with (no object) to the size of N. To find a bipartite matching between these two sets we search for a permutation of N elements o E G with the lowest cost:
= arg min
Ematch )o(s)>
(4)

---

Published as a conference paper at ICLR 2023
Per-dataset mloli
Total training
Backhone
Model
Pre-truin
Average mloU
ADE20K
Cityscapes
COCO-Stuff
Mapillary Vistas
Manual-axxooony
CLIP
40.24
79.37
38.36
46.70
51.16
160k
Multi-head
CLIP
79.29
38.
L.MSeg (Ours)
CLI
41.37
79.13
39.01
50.82
52.38
Manual-txxanomy
CLJP
43.46
80.23
39.01
51.58
53.56
320k
R50
Mulli-hest head
CLIP
44.30
80.06
38.57
52.13
53.76
I.See (Ours)
CLJIP
44.89
79.81
39.21
52.27
54,04
Manua-taxonomy
640k
R50
Multi head
CLIP
46.47
80.07
37.73
53.22
54.37
LMSeg (Ours)
CLIP
Table 2: Semantic segmentation accuracy (mloU) compared with multi-dataset models.
Per der-eat PQ (number of classes)
Training Setting
Backhome
Method
Pre train
Average
ADE20K-Parne Panoopiic
COCO-Panoptic
Mammal-taxemene
CLIIP
25,88
51.50
31,26
36.21
320%
R50
Multi head
CLIP
23,51
48.56
27.61
33.22
LMSeg (Ours)
CLIP
30.84
51.75
34.16
38,91
Manual-tan taxomomy
CLIP
28.65
$2.18
34.76
38.53
640%
R50
multi head
CLIP
28.66
50.05
31.27
36.66
LMSeg (Ours)
CLIP
34,20
55.28
37.72
42.40
Manual-takonony
CLIP
30.68
53.47
36.63
960k
R50
multi-head
CLIP
31.99
54.17
35.18
40.44
LMSeg (Ours)
CLIP
35.43
54.77
38.55
42,91
Table 3: Panoptic segmentation accuracy (PQ) compared with other multi-dataset training methods.
where the matching cost Lmanch (3h. (o)) takes into account the class prediction and the similarity of predicted and ground truth masks,
Lymonly; Jo(o)) = (-P
{ (c"(s) + Aexcal Lrocu (m,, ma(e)) + Aacee Laice (m, m)
C * Ø
(S)
+00,
During the training phase, we compute the optimal assignment ă for each sample in the batch and then accumulate the loss function of Equation over these samples.
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Datasets. For semantic segmentation, evaluate four public semantic segmentation datasets: ADE20K (Zhou et al., 2017) (150 classes, containing 20k images for training and 2k images for val- idation), COCO-Stufi Sttff-1fft--- 10K (Caesar et al.) 2018) (171 classes, containing 9k images for training and Ik images for testing), Cityscapes (Cordts et al. 2016 (19 classes, containing 2975 images for train- ing, 500 images for validation and 1525 images for testing), and Mapillary Vistas (Neuhold al.) 2017) (65 classes, containing 18k images for training, 2k images for validation and 5k images for testing). For panoptic segmentation, we use COCO- Panoptic (Lin al. 2014) (80 "things" and 53 "stuff" categories), ADE20K PPaokpii (Zhou et al. 2017) (100 "things" and 50 "stuff" categories) and Cityscapes- Panootic (Cordts al. 2016) (8 "things" and "stuff" categories).
Training setup. We Detectron2 (Wu al. 2019) to implement our LMSeg. Without specific instruction, dataset-aware augmentation is adopted and the same as MaskFormer (Cheng et al. 2021) 2018 learn-
for each dataset. We use AdamW (Loshchilov & Hutter 2019) and the poly (Chen ing rate schedule with an initial learning rate of le and weight decay of le Image encoders are initialized with pre-trained CLIP (Radford et al. 202T) weights. Following MaskFormer, learn- ing rate multiplier of 0.1 is applied to image encoders. Other common data augmentation strategies like random scale jittering, random horizontal flipping, random cropping and random color jittering are utilized. For the ADE20K dataset, we use crop size of 5 512x512. For the Cityscapes dataset, we use a crop size of 512x 1024. For the COCO- SOCCO-Stuff111 10k dataset, we use a crop size of 640 640. crop size of 1280 1280. All models A100 GPUs and a batch size of 16. Each image in the batch is randomly sampled from all datasets. For panoptic segmentation, follow exactly the same architecture, loss, and training procedure as
For the Mapillary Vistas dataset,
trained with 8

---

Published as a conference paper at ICLR 2023
use for semantic segmentation. The only difference is supervision: i.e., category region masks in semantic segmentation vs. object instance masks in panoptic segmentation. The data augmentation of each dataset follows MaskFormer, and also provide the detail of augmentation in the appendix.
4.2 RESULTS AND COMPARISONS
When conducting multi-dataset training, train single LMSeg model and report the model's per- formance on each dataset. While conducting single-dataset training, we train a separate MaskFormer model for each dataset, and the total training cost is cumulative over all datasets.
When comparing with multi-dataset training methods, we re-implement the "multi-head" and "manual-taxonommy" methods based MaskFormer for fair comparison. For the multi-head method denoted by "MaskFormer + Multi-head" in Table and Table3] we share the components of MaskFormer various datasets, except for the classification head. For the manual-taxonomy method denoted by "MaskFormer + Manual-taxonomy" in Table 2 and Table we manually con- struct the unified taxonomy. For simplicity, we only unite duplicate class names across datasets. Semantically similar classes such as "apparel" and "clothes" are not split or merged.
4.2.1 MULTI-DATASET SEMANTIC SEGMENTATION
Table [ depicts the comparison between LMSeg and single-dataset training models. Our LMSeg outperforms single-dataset training models in all settings. This experimental results show that com- pared with single-dataset training, multi-dataset training can obtain a single robust model capable of recognizing more categories and improving the erage performance on multiple datasets. For more intuitive comparison, we plot the average mloU w.r.t. total training cost for four datasets in Figure
55
Average mlat) on four datasets
54 54.32
34,04
54-
55,75
53,05
53.2
52.55
51 10.25
. MankFonnur single-dstastt training
LMSog milit-datase triining
50
160k
320k
480K
Table shows shows the comparison between LMSeg and other multi-daten dataset training methods. Experimental results show that LMSeg outperforms "multi-head and "manual-tare "aauual-txxooom"" under various settings.
Tranning iterations
Figure 6: Average mloU w.r.t total training cost for four semantic segmentation datasets. All models use CLIP as pre-trained weights.
4.2.2 MULTI-DATASET PANOPTIC SEGMENTATION
Considering the high training cost of panoptic segmentation, we only give comparison of LMSeg with other multi-dataset training methods. As shown in Table for the more challenging panoptic segmentation task, LMSeg outperforms "multi-head" and "manual-taxonomy" by large margin in various settings.
4.3 ABLATION STUDY
Weight Initialization. LMSeg learns to align text embedding and segmented query embedding in the same embedding space. This subsection shows that Vision-Languag pre-training (VIP) benefits LMSeg, as VLP helps initialize an aligned embedding space for vision and language. As shown in Table when the text encoder is initialized with CLIP, and the image encoder is initialized with ImageNet pre-training, the experimental results correspond to language pre-training initialization.
Pre-training weights
Per-datasel mloU
Average mloU
Text encoder
Image encoder
ADE20K
Cityscapes
COCO-Sti Stuff-
Mapillary Vistas
CLIP-R50
INT-R50 CLIP-R50
42.57
74.93
38.52 40.76
45.21 49.11
50.30 52.75
CLIP-R500
43.44
77.72
Table 4: Ablation experiments of VLP initialization in LMSeg. All models are trained with a batch size of 16 and total number of iterations of 320k.
8

---

52.91 Published as a conference paper at ICLR 2023
Per-dataset mloU
Order of cross attention
Average mloU
ADE20K
Cityscapes
COCO-Stuff
Mapillary Vistas
text-visual
43.69
77.91
40.48
49.75 50.57
52.95
visual-text
44.37
78.20
40.53
53.41
Table 5: Experiments on cross-attention order in the CGD module. All models are trained with batch size of 16 and total number of iterations of 320k.
Per-dataset mloU
CGD
DAA
Average mloU
ADE20K
Cityscapes
COCO-Stuff
Mapillary Vistas
43.44
77.72
40.76
49.76
44.37
78.20
40.53
50.57
53,41
44.89
79.81
39.21
52.27
54.04
Table Ablation experiments on proposed CGD and DAA. All models are trained on four semantic segmentation datasets with batch size of 16 and total number of iterations of 320k.
Per-dataset PQ
CGD
DAA
Average PQ
ADE20K-PPaoppe
ChyscapessPanoopicc COCO-Panoptic
25.77
48.35
34,38 33.74
29.03
30,24
49.69
37.88
30.84
51.75
34.16
38.91
Table 7: Ablation experiments proposed CGD and DAA. All models are trained on three panoptic datasets with a batch size of 16 and total number iterations of 320k.
When text and image encoders are both initialized with CLIP, the experimental results correspond vision-language pre-training initialization. Experimental results show that LMSeg with VLP initialization significantly outperforms language pre-training initialization, improving the average mloU on four datasets from 50.30 52.75.
Category-guided Decoding Module. There are two cross-attention layers in the category- guidedd decoding module. In this subsection, we study the effect of their order. "text-visual" indicates that the cross-attention module between segment query and text embedding is in front; otherwise "visual-text". As shown in T the experimental results show that "visual-text" performs better than "text-visual", "visual-text" is used by default.
Factor-by-factor Ablation. As shown in Table and Table 7 the proposed category-guided de- coding module increases the average mloU on four datasets from 52.91 to 53.41 and the average PQ on three datasets from 34.38 37.88, verifying the module's effectiveness. The proposed dataset- aware augmentation strategy adopts separate suitable data augmentation for each dataset, thereby improving the model's overall performance. The average mloU and average PQ increase from 53.41 to 54.04 and 37.88 to 38.91, respectively.
CONCLUSION
In this work, we propose language-guided multi-dataset segmentation framework that supports both semantic and panoptic segmentation. The issue of inconsistent taxonomy is addressed by uni- fied text-represented categories which not only reflect the implicit relation among classes but also are easily extended. To balance the cross-dataset predictions and per-dataset annotation, we introduce category-guidee decoding module to inject the semantic information each dataset. Additionally, dataset-aware augmentation strategy is adopted to allocate an approximate image preprocessing pipeline. We conduct experiments on four semantic and three panoptic segmentation datasets, and the proposed framework outperforms other single-dataset we will introduce more datasets and further study the zero-shot segmentation problem with comple- mentary and shared categories across datasets.
methods. In the future,

---

Published as a conference paper at ICLR 2023
REFERENCES
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, volume 12346, pp. 213-229. Springer, 2020.
Liang- Chiih Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 706.05587 2017.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI, 40(4):834-848, 2018.
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov, Per-pixel classification is not all you need for semantic segmentation. In NeurIPS, pp. 17864-17875, 2021.
Bowen Cheng, Ishan Misra, Alexander G. Schwing. Alexander Kinillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, pp. 3146-3154, 2019.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544, 2021.
Hassan Ghassemian. A review of remote sensing image fusion methods. Information Fusion, 32: 75-89, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Conet: Criss-cross attention for semantic segmentation. In ICCV, pp. 603-612, 2019.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In CVPR, June 2020.
Jesse Levinson, Jake Askeland. Jan Becker, Jennifer Dolson, David Held. Soeren Kammel, J Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et Towards fully autonomous driving: Sys- tems and algorithms. In IEEE T-IV, pp. 163-168, 2011.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. ECCV, volume 8693, pp. 740-755, 2014.
Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, pp. 2999-3007, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pp. 3431-3440, 2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2023
Markus Maurer, Christian Gerdes, Barbara Lenz, and Hermann Winner. Autonomous driving: technical, legal and social aspects. Springer Nature, 2016.
Fausto Milletari, Nassir Navab, and Seyed- Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural net- works for volumetric medical image segmentation. In 3DV, pp. 565-571, 2016.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulò, and Peter Kontschieder: The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, pp. 5000-5009, 2017.
Toby Perrett and Dima Damen. Recurrent assistance: Cross-dataset training of Istms kitchen tasks. In ICCV Workshop. pp. 1354-1362, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML. volume 139, pp. 174488763, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, pp. 18082-18091, 2022.
Bowen Shi, Xiaopeng Zhang, Haohang Xu, Wenrui Dai, Junni Zou, Hongkai Xiong, and Qi Tian. Multi-dataset pretraining: A unified model for semantic segmentation. arXiv preprint arXiv:2106.04121, 2021.
Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pp. 7242-7252, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(86):2579 2605, 2008
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan L. Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. CVPR, pp. 1463-5474, 2021.
Li Wang, Dong Li, Han Liu, Jinzhang Peng, Lu Tian, and Yi Shan. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. In AAAI, pp. 2487-2494, 2022.
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, pp. 7794-7803, 2018.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Yongqiang Yao, Yan Wang. Yu Guo, Jiaojiao Lin, Hongwei Qin, and Junjie Yan. Cross-dataset .04621,
training for class increasing object detection. arXiv preprint arXiv:2001 200.220, 2020.
Wei Yin, Yifan Liu, Chunhua Shen, Anton van den Hengel, and Baichuan Sun. The devil is in the labels: Semantic segmentation from sentences. arXiv preprint arXiv:2202.02002, 2022.
Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet:
Object context for semantic segmentation. IJCV, 129(8):2375222 2299,, 2021.
Renrui Zhang, Rongyao Fang. Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111 11.0930,, 2021.
Hengshuang Zhao. Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR, pp. 0230-6239, 2017.
Xiangyun Zhao, Samuel Schulter, Gaurav Sharma, Yi-Hsuan Tsai, Manmohan Chandraker, and Ying Wu. Object detection with a unified label space from multiple datasets. In ECCV, volume 12359, pp. 178-193, 2020.
11

---

Published as a conference paper at ICLR 2023
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H. Torr, and Li Zhang. Rethinking semantic segmentation from sequence-to-sequene perspective with transformers. In CVPR. pp. 6881-6890, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pp. 5122-5130, 2017.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IICV, 2022b.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In CVPR, 2022c.
12


Reviews:
[{"review_id": "g3vI8HxS0A", "reviewer": "Reviewer_wkpT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a language-guided framework for multi-dataset training. To solve the consistency of categories in each dataset, the paper encodes the categories in text embedding without manual reconciliation. Also, the paper adopts a dataset-aware augmentation strategy that assigns each dataset a specific image augmentation pipeline, which can suit the properties of images from different datasets.\nThe extensive experiments demonstrate that the proposed method achieves significant improvements on four semantic and three panoptic segmentation datasets.", "summary_of_the_review": "The proposed method gives new insight, using a text encoder to solve the inconsistency of the label system in the different datasets, in multi-dataset segmentation training. This is an interesting direction. In my view, the more experiments analysis should be added.", "strength_and_weaknesses": "Pros:\n\n1, The paper is well-written and easy to understand.\n2, The proposed method is simple but efficient.\n3, The experiments are sufficient.\n\nCons:\n\n1, In the table2 of semantic segmentation, the proposed method seems to improve marginally if has a long training time. However, the table3 of panoptic segmentation improves largely.  Could the author have more analysis? is it related to the improvement of instance segmentation?\n\n2, I note that the proposed method is benefited from both CLIP image encoder and text encoder. Thus I wonder if the main reason for the improvement of the proposed method is the usage of the CLIP model, especially the CLIP Image encoder.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J56Iozn4Jp5", "reviewer": "Reviewer_G1t4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve two problems of multi-dataset segmentation: (1), inconsistent taxonomy. (2), inflexible of one-hot taxonomy. They map the taxonomy into embedding space with pre-trained pre-trained text encoder. Then they proposed a category-guided decoding module is designed to dynamically guide predictions to each dataset’s taxonomy.\nExtensive experiments demonstrate that the proposed method achieves significant improvements on four semantic and three panoptic segmentation datasets\n", "summary_of_the_review": "Overall, this paper is interesting to me. However, using text embedding for multi-dataset segmentation has been explored before.  Reminded by other reviewer opinions (Reviewer DHCs), I find the real technical novelty is very limited (Category-guided decoding (CGD) and Dataset-aware augmentation (DAA). I lower down my ratings.", "strength_and_weaknesses": "Overall, this paper is good. \n\nStrength:\n\n\n1, The paper is well written and easy to follow. \n\n2, The idea of using unified text embedding to enhance the object queries is interesting and novel for DETR-like framework for multiple datasets segmentation tasks. Category-guided decoding (CGD) is simple yet effective. \n\n3, The extensive results show the effectiveness of proposed framework. \n\n\nWeakness:\n\n1, The idea of using text embedding for multi-dataset segmentation is not novel. The idea is proposed in the work “The devil is in the labels: Semantic segmentation from”(half year ago on arxiv. ) This paper adds learnable embedding (From COOP-IJCV-2022) and extra attention module. The author should not claim this is very huge contribution in introduction part. \n\n2, Missing Experiment results M-Seg dataset. It would be better for benchmarking results on this dataset. \n\n3, Missing ablation studies on CGD design. \n\n4, Missing parameter and GFlops analysis on proposed CGD. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5wo-9FIY8x", "reviewer": "Reviewer_6V2p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work extends a MaskFormer model to build LMSeg model, that learns to segment categories across multiple datasets using unified training without resorting to manual taxonomy. LMSeg is built on the fact that the text embeddings for semantically similar categories, as retrieved from CLIP text encoder, projects close to each other on euclidean space. Hence by learning to align the segment embeddings to the text embeddings, it is possible to learn to pool similar categories from multiple datasets without manual taxonomy mapping. Further, in order to dynamically guide the segment embeddings towards dataset-specific taxonomy, LMSeg proposes to use category-guided text-query cross-attention within the transformer decoder. This together with the dataset-aware augmentation strategy can train SoA multi-dataset model that outbeats single-dataset trained models (with similar total budget).", "summary_of_the_review": "LMSeg work makes a good read with clean experimental evaluation with few insights drawn during ablation. However, due to limited technical novelty I would place current version at Borderline acceptance. Adding extra evaluation and drawing better insights, would clearly add more value. ", "strength_and_weaknesses": "Strengths:\n1. Multi-dataset training for segmentation using a unified taxonomy is the first step towards open-set model. LMSeg automates this training without any -- additional parameters in the form of dataset-specific head, manual mapping and drop in per-dataset mIoU. Similar to other open-set vision model, LMSeg leverages language embeddings to build a unified model.\n2. Moreover, the proposed LMSeg is simple, elegant and easy to implement straight out of MaskFormer model. \n\nWeaknesses:\n1. The model is well engineered. However, I feel there is limited novelty. Multiple recent work has resorted to text-query alignment to build an open-set model for different problem settings (object detection, referring expression). Motivated by this development, LMSeg modifies and trains Maskformer like framework with similar losses. To me, the only novel contribution seems category-guided decoder design. \n2. Secondly, Table 2 & 6 suggests that the impact of LMSeg, CGD & DAA framework on semantic segmentation setting is very minimal. Hence the novel components only contributes to improvements in panoptic segmentation. Any thoughts on what causes this discrepancy ? \n\nHow about evaluating LMSeg in an open-world setting ? Or atleast, cross-category evaluation (i.e., train model by leaving out test categories) or datasets relevant to training data (such as ADE-Full) ? These results should strengthen the empirical evaluation. As for now the multi-dataset evaluation has very limited use case.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1mkUA0sp1rX", "reviewer": "Reviewer_DHCs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents LMSeg, which aims to train a model for image segmentation on multi-datasets. The authors claim that they resolve two major challenges in this paper:\n- 1 the domain gap or inconsistency between semantic segmentation and panoptic segmentation.\n- 2 the separated label space.\n\nExperiments show that the proposed method can achieve better results than the previous baseline.", "summary_of_the_review": "My concerns primarily lie in the novelty and efficacy of the proposed method. As discussed above, neither the story nor the performance of this paper impress me, I am thereby inclined to reject this paper.", "strength_and_weaknesses": "### Strength\n\n1. The paper is well written and easy to follow.\n\n2. I recognize the significance of the problem targeted in this paper. It should be feasible to identify instances in images of datasets labelled for semantic segmentation.\n\n### Weaknesses\n\n**1. The novelty is limited.** The main claim of the authors is that they propose *text-query alignment to address the issue of taxonomy inconsistency*. However, introducing language embeddings to build a unified label space for image segmentation has already been proven to be effective [1, 2, 3]. The Category-guided decoding (CGD) here is just another variant of LSeg [1] that exploits the text-pixel affinity to bridge the domain gap across different label spaces. Dataset-aware augmentation (DAA) here is a non-learnable strategy for data augmentation, which seems to be better mentioned in the section of implementation details instead of methodology.\n\n**2. The improvement is not strong enough.** The numbers shown in Table 2 are not impressive from my point of view. For example, the gain for the average mIOU across four datasets can only be shrunk to a negligible 0.2% for 640K. I focus more on the results trained with a longer schedule since MaskFormer is hard to optimize and originally it needs 300 epochs to train on COCO panoptic. The results look to be prettier when trained under a short schedule, however, this might be due to a better initialization as the encoder of the proposed method is first pre-trained by CLIP (as discussed in Table 4).\n\n### Reference\n\n[1] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. ICLR 2022\n\n[2] Xu, Mengde, et al. \"A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model.\" arXiv preprint arXiv:2112.14757 (2021).\n\n[3] Rao, Yongming, et al. \"Denseclip: Language-guided dense prediction with context-aware prompting.\" CVPR 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster