Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs
Authors: Rong Ma
Jie Chen
Xiangyang Xue
Jian Pu
Venue: NeurIPS 2024 poster (2024)
Keywords: Semantic Segmentation
Multi-dataset Training
Graph Neural Networks
Primary area: machine_vision
TL;DR: We propose a novel approach that leverages graph neural networks to automatically construct a unified label space for training semantic segmentation models across multiple datasets.

Abstract: Deep supervised models possess significant capability to assimilate extensive training data, thereby presenting an opportunity to enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. In this paper, we propose a novel approach to automatically construct a unified label space across multiple datasets using graph neural networks. This enables semantic segmentation models to be trained simultaneously on multiple datasets, resulting in performance improvements. Unlike existing methods, our approach facilitates seamless training without the need for additional manual reannotation or taxonomy reconciliation. This significantly enhances the efficiency and effectiveness of multi-dataset segmentation model training. The results demonstrate that our method significantly outperforms other multi-dataset training methods when trained on seven datasets simultaneously, and achieves state-of-the-art performance on the WildDash 2 benchmark. Our code can be found in https://github.com/Mrhonor/AutoUniSeg.

Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs
Rong Ma, Jie Chen, Xiangyang Xue, and Jian Put Fudan University
rma220m fudan. edu. en, {chenj19, xyxue, jianpuj edu.
Abstract
Deep supervised models possess significant capability to assimilate extensive train- ing data, thereby presenting an opportunity enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. this paper, we propose novel approach to automatically construct unified label space across multiple datasets using graph neural networks. This enables semantic segmentation models to be trained simultaneously multiple datasets, resulting in performance improve- ments. Unlike existing methods, our approach facilitates seamless training without the need for additional manual reannotation taxonomy reconciliation. This sig- nificantly enhances the efficiency and effectiveness of multi-dataset segmentation model training. The results demonstrate that method significantly outperforms other multi multi-dataset training methods when trained on seven datasets simultaneously, and achieves state-of-ther art performance the WildDash 2 benchmark. Our code can be found httpss//gthub.com///hhnooo wwt.:/ooooo///////aee/.o UmiSoe
Introduction
Recent advances computer vision [35 21 have shown the advantages of large datasets training robust visual models (2). However, for deep supervised visual models that rely annotated data, the collection of such extensive annotated datasets can be prohibitively costly [IT]. To address this expense and expand the data available for training, several efforts 4 23 30 focus on the challenges
of multi-datasct training, enabling the use of diverse datasets to train more robust and generalizable
models.
Models trained on multiple datasets must confront the challenge of reconciling conflicting annotation standards and label spaces. For example, the class road in the BDD dataset [46] can be further divided into several classes in the Mapillary dataset [3]: road, lane marking and crosswalk. Similarly, the Mapillary dataset labels both barrier and curb distinct classes, while in the IDD dataset [42), they are combined under the single label curb. These confficts impact the supervised learning of models, they may be incorrectly penalized for predicting finer-grained classes from other datasets.
Another challenge is the task of unifying diverse dataset labels to produce outputs in standardized format 29 224]. Several methods [TI 30 attempt to address this by concatenating the label spaces of all datasets and using language models to encode label names into a text embedding space. However, there approaches introduce redundancy and fail to handle issues where labels share names but differ in annotation granularity. Other methods involve manually constructing universal taxonomies 24 or relabeling 233, both of which are time-consuming and labor-intensive. Recent approaches aim to
"These authors contributed the work equally and should be regarded co-first authors. "Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Label Encoding
Automated Label Unification
Dalaset-specffcc Label Node
Dataset Label Mappings
Dataset Labels
Unified Label Node
Text Features
Graph
road
Prediction the Dataset Label Space
lane marking
Language Model
train
6** rails DatasetN
Networks
road
Unified
marking
Embedding
rails
Space
Segmentation Unfied Label Space
Prediction the Unified Label Space
$ Lee
Input Image
Segmentation Network
Ground Truth
Our method consists of three modules. The label encoding provides the semantic text features of the dataset labels. The GNNs learn the unified label embedding space and dataset label mappings based on the textual features and input images. The segmentation network leverages the unified label embedding space produce segmentation results in the unified label space.
Figure 1:
automatically construct universal taxonomies relations between only two datasets, involving time-consuming iterative training process.
but these methods typically identify inter-label
In this paper, propose novel approach leveraging Graph Neural Networks (GNNs) 20 to automatically construct unified label space, enabling segmentation model to be trained simultaneously on multiple datasets. In contrast previous approaches [1] 4 6 our approach eliminates the need for manual re-annotation or iterative training procedures to construct universal taxonomies, while methods in distinguishing categories with identical
also addressing the limitation of language-baus semantics. As depicted in Figure I we utilize language model to convert the dataset labels into text features. Then, apply GNNs learn the relationships and associations among these labels. The process creates unified label embedding space and dataset label mappings. The output head of the segmentation network incorporates the unified label embedding space to generate unified segmentation results within this unified space. The dataset label mappings are subsequently utilized align the unified segmentation results with the label spaces relevant each dataset. This enables the training of segmentation models and graph neural networks with dataset-speciftt annotations.
2 Related work
Multi-datasets Semantic Segmentation. In recent years, numerous studies [52] 29 focused on train- ing semantic segmentation models on multiple datasets. A simple approach involved incorporating dataset-specific modules in the model, such as -ppaset- output heads [28 or dataset-specifie
batch normalization layers 44, to produce predictions tailored to each dataset domain. While these methods effectively avoided issues of dataset label space conflicts, they offered limited applicabil- ity in real-world scenarios as they could not deliver unified predictions. MSeg [23] addressed the problem at the data level via manual re-annotation processes to resolve label conflicts. However, this approach was time-consuming. error-provt prone and not easily scalable. Recent methods employed manual [5] or automatic techniques [6] to construct universal taxonomies and establish label relationships between datasets label space and the unified label space. These methods, leveraging partial label learning approaches 48 12] enabled training with datasct-specific annotations and producing unified
predictions.
Construct Universal Taxonomies. Each dataset has its unique domain, necessitating the construction of universal taxonomies to enable the model to cover all domains. Several approaches [I] B0 attempted to concatenate dataset label spaces and differentiated similar classes by aligning text embedding encoded by language model. However, this approach struggled with classes that had the same name but different levels of annotation granularity, and direct concatenation of label spaces could lead to semantic conflicts. Other research 24 29] attempted to establish a unified label space

---

Training Datasets
Sampled Image from Dataset
Pixel Embedding
Dataset spedfic Prediction
Dataset Annotation
E Encoder Decoder
UniSegHead
Dataset
x w.x. I.
UniSegHead
Unified Prediction
Dataset-specific Prediction
Dataset Poxel Embedding
Projected Unified Label Embedding Space
Mapped to Dataset- specific Label Space
HXWXL
H HxWxL,
Label Relation Learning
Unified Label Embedding Space
Dataset-specffic Label Mappings
Dataset Label Description by
Dataset
road curb
Dataset Embedding
L Lw
from
Text Encoder
Dataset
Initialized Bipartite Graph Matrix
from dotasot
table
structre...
Wall
Features
Randomized Un Unilled Features
GNNs
Figure 2: Illustration of our method that training with dataset-speciff annotations through label mappings constructed by GNNs. We leverage a unified segmentation head (UniSegHead) to enable simultaneous training multiple datasets. In the UniSegHead, between pixel embedding and augmented unified node features output by the GNNs, resulting in predictions for the unified label space. We finally utilize the label mappings constructed by GNNs to map the unified predictions to dataset-specific prediction for training.
compute the matrix product
through the expertise of human annotators. Additionally, other studies 41 53 aimed to address this issue by developing automated mechanisms create universal taxonomies. However, these methods either could only construct unified label space between two datasets at time [6] could not handle complex class relationships (53]. Our approach stands out by automatically constructing universal taxonomies in single training session with multiple datasets, resulting in significant time savings compared methods that require multiple training iterations.
Graph Neural Networks demonstrated exceptional effectiveness in dealing with complex topo- logical data structures, as highlighted recent research 43). The applicability extended across various domains, including recommendation systems [18] knowledge graph construction [50], and skeletal action recognition
[54]. In the context of our problem, discovering label relations can be conceptualized as a link prediction task [49 (22). However, conventional approaches for graph link prediction [9] TO] are suitable for our model since we do not possess ground truth links. We use the values of the learnable adjacency matrix predictions of whether nodes are linked. Supervision of the linked predictions relys on the segmentation results of the segmentation network and the corresponding image annotations.
Proposed Method
The comprehensive framework depicted in Figure 2 We first define the unified representation of the multi-dataset label space. Utilizing this representation, build graph neural network to learn the unified label space. Finally, leveraging the unified label space, we train our segmentation network and the graph neural network using the dataset annotations.
3.1 Unification of Multi-dataset Label Space
Unified Label Space. Given K datasets with their respective label space (L1, Iz,.... ., Lx ) multi- datasct semantic segmentation requires model to predict within a consistent label space that

---

encompasses all dataset label spaces Lpred - Lg Each pixel must be assigned to a label in this unified label space. We define N unified label nodes A {ai, Q2y,..... ...,,..,,,, serving as the nodes in the graph neural networks. Their corresponding D-dimensional learnable embedding {x1, x2,..., xN represent the unified label embedding space. The number of unified label nodes N is often smaller than the total number of dataset classes [. because we aim to merge multiple identical classes into a single unified label. The image is first encoded into pixel embedding P by the segmentation network, which is then projected into the unified label embedding space to assign unified label to each pixel.
Label Mappings. We define a mapping from the unified label space to the dataset-specific label space M, AA- - Li, which is used to train the model with dataset-specific annotations. Mathematically, M, E {0, 1) 1x is boolean linear transformation. Each unified class a is at most linked to class e within a specific dataset i to prevent label conflicts: M,1 < 1. To handle different annotation granularities, we use label mappings to merge multiple unified label nodes representing fine-grained classes into one super-class c: M 1 2 1. For example, the curb from IDD can be simultancously mapped by unified label nodes represented curb and barrier. We use unified label nodes as input nodes for the GNNs, which learn the label mappings and unified label embedding space, thus enabling the automated unification of multi-dataset label spaces.
3.2 Learning Unified Label Space with Graph Neural Networks
Learning the label mappings between dataset label spaces and the unified label space can be viewed as bipartite graph matching problem. This makes graph neural networks well-suited address this issue. Below, we detail our approach of constructing GNNs for learning unified label space.
Input Nodes. To construct the input feature of dataset-specifie label nodes, illustrated in Figure 2 we used the dataset labels in the template "An image of <label> from the dataset <dataset> as plain text input and employed ChatGPT to complete the detailed description of each label. Then, we employ llama-2 [40] to encode these label descriptions and generate text features. To further distinguish nodes from different datasets, we introduce learnable dataset embedding for each dataset {d1, dz...., dx) }. The dataset embedding is combined with the text features to form the input features for dataset-specific label nodes:
Xi. (el.m) + di,
where Xi, is the input feature of the m-th label from dataset i, and fe f(( is the text feature of the label description 4im encoded by language model. The input features of unified label nodes are randomly initialized to the same dimension as the dataset-specific label node. To determine the appropriate number of unified label nodes, inspired by the approach in (53). we used cross- validation results across different datasets to identify the number of mergeable categories, which served as the initial selection for the unified label nodes. The specific algorithm can be found in Appendix B Together, -ataset-specific label nodes and unified label nodes constitute the input nodes of GNNs. During the training process, we maintain a constant number of nodes. After the training is completed, we will remove inactive unified label nodes, meaning those that were not assigned to any dataset-specific label.
Learnable Adjacency Matrix. To enable label mappings to be updated via gradient descent, we embed the label mappings a continuous, learnable graph adjacency matrix M.. Values in the adjacency matrix represent the weights of corresponding edges. Only the edges between unified label nodes and dataset-specifie nodes are learnable, while others are fixed to zero. We apply softmax operation to the edges connecting each unified label node with nodes of a particular dataset, ensuring that the sum of the edge weights w between each unified label node and nodes of this dataset equals one. The element at the intersection of the r-th row and the e-th column in the lower triangular portion of M, is formulated as:
M { EdteLorEd Bo
if |L] and c << |L]
otherwise.
Unified Label Embedding. The forward propagation of our graph model follows the GraphSAGE framwork 177 as formulated in Eqqation 3. w and X are the weight and feature of the k-th GNNs layer. The a indicates nonlinear activation function, implemented as the tanh in this work. The output

---

Table 1: Training and test datasets in our experiments.
Dataset Domain
Training and Validtion dalasets
Unseen lest datasets
Driving scene
CityScapes 13], Mapillary [33 BDD 46
IDD
42
WildDash 447), KITTI [6] Cam Vid 8.
Indoor scene
SUN RGBD 37
ScanNet-20 [4
Everyday objects
ADE20K ST. COCO
25
PASCAL VOC [T.] PASCAL Context
3
features of the unified label nodes from the final layer serve as the unified label embedding space,
x = [x1:X] X2..... [N]
x = - oo" [X |M,X**))
To obtain the dataset label mappings, we partition the adjacency matrix into submatrices, each corresponding to specific dataset. Each submatrix contains only the unified label nodes and the label nodes specific to that dataset. We compute the label mappings for each dataset based on the values in its corresponding submatrix, divided into the following two cases. During training, we will alternately train the segmentation network and the GNNs. When training the GNNs, we directly use the value of the learnable adjacency matrix to establish label mappings. thereby facilitating weight updates through gradient descent. When training the segmentation network, we utilize the unbalanced optimal transport algorithm to solve for the boolean label mappings that satisfies the many- many-to-one mapping constraints. This algorithm detailed in [Appendix Geffectively handles the conversion of the continuous adjacency matrix into discrete dataset label mappings required for training segmentation network.
3.3 Training a Universal Model with Dataset-specific Annotations
Training a universal model is divided into two steps. The first step involves training a robust encoder- decoder to provide the pixel embedding for each pixel position in the image, where similar objects should have similar features. The second step focuses on learning label mappings and unified label embedding space by GNNs. During the training phase, we alternate between these two steps, freezing one network while training another network. Both of these steps require supervised training using dataset-specific annotations. Here, we primarily focus on the training of GNNs, while further details of the training strategies are provided in the Appendix A
Training with Dataset-specific Annotations. During training, an image is randomly sampled from dataset i and fed into an segmentation network, which provides embedding for each pixel position where is D-dimensional vectors. We utilize a unified segmentation head
P = (PI, P2.................... ....... -2... }. (UniSegHead) to assign a dataset label to each pixel. In the UniSegHead, We first project the pixel embedding into the unified label embedding space by multiplying the pixel embedding by the output feature of unified label node X as shown in Equation 4 Then, to train the universal model with -pteset- annotations, we need to map the predictions in unified label space to dataset-specifer label space obtain the probabilities of dataset-specific classes. This achieved by computing per- per-pixel dataset-specift logits s by multiplying the dataset-specific label mappings M; at each pixel:
Ug X.,Ps,
Ss ug.
(5)
Finally, probabilities of dataset-specifit classes computed by per-pixel softmax operation over the logits 8. This allows us to use dataset-specific annotations to compute the pixel-wise loss function, train the network, and update the label mappings. We formulate the cross-entropy loss for a specific pixel to train the segmentation network:
[4,1 Σ
Loe Lyc 8)
Ye log(soottmax(s))),
where y is the pre-pixel annotation, L,| represents the total number of classes in the dataset i from which the image originates.
Orthogonality Loss. To achieve a conflict-free unified label space and avoid redundant unified label nodes that represent the same class or have overly similar features, we introduce soft constraints to promote orthogonality among the unified label node features, inspired by [39]. This orthogonality loss encourages unified label node embedding to be mutually orthogonal. It not only aligns the unified
5

---

Table 2: Multi-dataset performance compared with other methods.
Methods
Backbone
Venue
Label space'
CS
MPL
SUN
BDD
IDD
ADE
COCO
Mcan
MSeg [23
HRNet-W48
CVPR 20
MR
76.3
51.9
46.1
63.5
61.8
42.8'
48.6'
55.9
NII.+
4
SNp-RNI8
WACV
MC:
72.6
39.1
41.7
58.5
54.4
31.0
35.4
47.5
NLL+ [5]
SNp-DN161
UCC
MC
76.1
44.2
46.9
60.4
56.7*
35.6
39.3
51.3
Single dataset
HRNec-W48
77.0
30.2
43.9
66.8
34.5
38.0
50.4
Multi-SegHess
HRNet-WaB W48
DS
79.5
36.1
47.3
65.6
67.0
30.5
36.7
51,8
Auto univ. [
SNp-RN18
BMVC
Auto
72.7
35.8
42.3
59.6
55.2
30.7
35.6
47.4
Ours
HRNet-W48
Auto
80.7
43.7
47.5
65.5
68.6
42.0
46.7
56.4
Appreach constract label space. MC:Manullly Construct, Monualy Relabel, DS Dataset-specific Automaalttal Construct. MSeg train and evaluate on 43 of 65 class in Mapillary dataset, 117 of 150 class in ADE dataset, 122 of 133 class in COCO dataset.
These methods truined evaluated using classes recommended 26 classes.
while trained evaluatied using officially
Table Performance comparison with baselines on training and unseen datasets.
Trained dataset or label space
Mean results across training datasets
Mean results actoss unseen datasets
Single datasct
Multi-SegHead
Single datasct
Multi-SegHead
CS
23,9
30.9
31.4
37.9
MPL
30.6
36.1
36.9
41.0
SUN
12.9
22.0
16.7
28.9
BDD
24.8
29.1
30.4
35.7
IDD
26.0
34.0
28.9
36.4
ADE
28.5
40.3
37.2
49.8
COCO
30.5
38,8
45.1
53.2
Ours
56.4
56.9
label nodes with annotation standards for practical use but also enhances the diversity of the model and helps in finding better label mappings:
N
Corth = softmax log(softmax((xxx)))
Σ
The final loss function used to train the GNNs is represented as follows, with As as hyperparameters to adjust the weights of different loss functions:
L = Aicee + AzCorth.
(8)
4 Experiments
Datasets. We list the semantic segmentation datasets used for training and testing in Table Our training datasets cover wide range of scenarios, from indoor scenes to driving scenes. We also introduce corresponding test datasets, which are not used in the training process, for the respective scenes to evaluate generalization capability.
Implementaion Details. Our segmentation model is based on the HRNet-Wa W48 architecture (38 while the GNN model is a three-layer GraphSAGE [17]. We utilize the llama-2-7B model to encode label descriptions into dimendiimeii text features. These text features, augmented with dataset embedding of the same dimensionality, are then employed as node features input into the GraphS AGE. When forming a minibatch from multiple datasets, we evenly sample images per dataset within a batch for each GPU. For all images, We first apply random resizing with ratio ranging from 0.5 to 2, followed by a random crop operation to achieve final image size of 768 x 768 pixels. We use AdamW W optimizer (27) with warmup and polynomial learning rate decay, starting with a learning rate of 0.0001. We train our model for 300k iterations on four 80G A100 GPUs.
4.1 Comparison on Multiple Datasets
In Table 2 we present the accuracy of our methods and compare them to other approaches on the seven training datasets. We use mean Intersection over Union (mloU) to quantify the performance of models, a common metric used to evaluate the performance of segmentation models. Different methods adopt various approaches to construct their label spaces: Dutaset-Specific represents lack of unified label space, where the model outputs separate label space for each dataset. Manually

---

Image
Ground Truth
BDD Model
MPL Model
ADE Model
Our 7ds Model
ADE MPI
Figure 3: Visual comparisons with Single dataset model on different training datasets.
Relabel means manually re-annotating each image. Manually Construct means the label space is constructed through human expertise. Automaticulyy Construct includes methods where the unified label space is automatically constructed by the model. We also establish two baseline methods: Single dataset and Mulli-SegHHed.. Single datasel demonstrates the results of training on individual dataset only, while Mulii-SegHead trains on multiple datasets by using dataset-specific segmentation heads.
The results demonstrate that our method achieves the best average performance in multi-dataset training, while also achieving significant performance improvements on datasets with a large number of classes such as the ADE and COCO datasets. We attribute this to the construction of robust unified label space. Leveraging visual connections from the samples, our unified label space can discover label relationships beyond textual similarities. For instance, the visual appearance of the fireplace ADE is similar to the tunnel in Mapillary. Despite their different semantic meanings, our model merges these labels for prediction, as detailed in subsection 4.5) This approach saves model capacity and facilitates knowledge transfer across datasets for improved prediction.
In Table 3] as a supplement to Table 2| we compare our model with various models trained on single dataset, as well each segmentation head output of the Multi-SegHHead. Detailed data for each dataset can be found in the Appenddix D From the table, it can be observed that training with multiple datasets helps improve the model's generalization performance. However, the performance of different segmentation head outputs in Multi-SegHead model shows significant differences due to the lack of unified form output that performs well across all datasets. In contrast, our approach provides unified label space covering all datasets, resulting in a significant advantage in average performance. We also list the performance on the five unseen test datasets mentioned Table 1] To evaluate the performance of our model on unseen datasets, we first evaluate the model results on its training dataset. We search for the optimal label mappings based on the accuracy of label predictions. There updates to any model parameters except the label mappings in this process. This process can actually be done manually without any annotation information. The results indicate that our model exhibits better generalization performance. It can handle various scenarios and consistently achieve excellent performance on the test set. Compared to Multi Headi-SegHead model, our automatically constructed label space has advantages over dataset label spaces. The unified label space integrate the semantic information from multiple dataset label spaces.
Figure 3 presents the segmentation results on multiple datasets predicted in unified label space.
Compared to models trained only on Single dataset, our model successfully provides consistent predictions on all datasets. It's worth noting that in the BDD dataset, annotations are not provided for lane marking, crosswalk and manhole, which are only annotated in the Mapillary dataset. Our model successfully integrates the label space of the Mapillary dataset, thereby predicting these classes in the BDD dataset. More results are presented in Appendix
4.2 Results on WildDash 2 Benchmark.
WildDash 2 477 provides a benchmark for semantic segmentation, designed to test the robustness of algorithms in real-world driving scenarios. Due to the insufficient number of training samples
7

---

Table Performance comparison on WildDash benchmark.
Mela Avg mloU Class
Classic mloU Class
Classic ilaU Class
Classic mlaU
Classic iloU Cat.
Negative mloU class
Model
Venue
Trained
EMPS
31
DCV 21
32.2
35.7
24.4
63.8
56.0
20.4
MSeg
23
CVPR 20
X
35.2
38.7
35.4
65.1
50.7
24.7
SeunSeg 34
CVPR 19
X
37.9
41.2
37.2
63.1
58.1
30.5
UniSeg
19
ECCV 22
39.4
41.7
35.3
65.8
57.4
Ours
40.5
41.2
33.7
65.4
54.2
43.4
SNPRNIS2
arXiv 20
45.4
48.9
42.7
70.1
64.8
32.5
NLL+ 4
WACV 22
V
46.8
51.0
43.9
71.4
65.5
32.6
Uni NLL
IICV 24
46.9
$1.6
45.9
72.8
67.5
29.0
FAN 45
arXiv
47.5
50.8
74.2
67.5
34.4
MIX6D 26
arXiv 22
48.5
51.2
46.5
72.4
66.1
40.8
Ours
50.0
52.2
47.5
72.4
68.6
44,6
Table 5: Comparison of Different Methods of Construct Label Spaces.
Methods
CS
MPL
SUN
BDD
IDD
ADE
COCO
Mean
Direct concatenation
448
79.2
38.8
46.9
64.3
66.7
33.3
38.2
52.5
Clustered by text features
329
79.9
40.8
47.3
65.6
68.6
33.8
38.0
53.4
Without GNN training
231
79.8
39.7
47.7
66.1
68.1
37.8
45.3
54.9
Without GPT's label description
226
80.2
43.4
47.2
64.7
68.8
40.1
46.0
55.8
Our proposed method
217
80.7
43.7
47.5
65.5
68.6
42.0
46.7
56.4
provided by this dataset, the official recommendation is to use multiple datasets for training. Therefore, this benchmark well-suited to evaluate the effectiveness of multi-dataset training methods. The WildDash 2 dataset includes negative test cases to challenge the robustness of the model. These negative test cases mainly consist of unconventional driving scenarios, and even non-driving scenarios. Across all pixels within negative test images, robust model is expected to predict the void label for open-set classes and anomalous objects. The WildDash benchmark refers to the metric named Meta Avg mloU Class, which calculates the mean Intersection over Union for each class by weighting negative and positive test cases according to their occurrence in the benchmark dataset.
Table 4 presents the current results on the WildDash 2 leaderboard. We present results for zero-shot generalization using our 7ds model. To evaluate an unseen setting, we map the non-evaluated classes in the unified label space to a void label for both positive and negative test frames. To ensure fair comparison with other works, we also provide evaluation results for models trained using the training datasets from the Robust Vision Challenge 2022[ which include CityScapes, ADE20K, Vistas, VIPER (36), ScanNet, and WildDash 2. The results indicate that our method achieves fate-offttheart performance in both zero-shot and trained settings. Our method exhibits significant performance improvements compared to other methods on negative test cases. We attribute this to the robustness of model, which has been trained on diverse datasets, enabling it to perform well in unconventional scenarios. Our zero-shot model has been trained on a wider range of datasets compared to the trained model. Therefore, even without training the WildDash dataset, it achieves similar performance on negative test cases.
4.3 Ablation Study
To further explore the ability of our GNNs to construct a unified label space, we compared it with four alternative methods. The first method concatenates all dataset label spaces into single unified space L The second approach constructs unified label space by clustering text features based on cosine similarity using the DBSCAN [7] algorithm. In the third method, we train a segmentation network using an initial adjacency matrix, outlined (pppendix training. The final method involves an ablation experiment, removing the label description module to observe its impact. Experimental results, presented in Table 5] demonstrate that the label space constructed based on GNNs can better assist in the learning of segmentation models. Unlike the first approach, our method optimizes model capacity by focusing on label relationships rather than dataset recognition. Compared to the second approach, our method can differentiate between classes with identical names but differing levels of granularity. By leveraging label descriptions enrich semantic context, approach constructs more refined and functional label space.
Ly.
without incorporating GNN

---

Dolasets
VDC
Table 6: Performance on unseen
KPP
dataset.
ADE
Driving
Non-Driving
2V
SUN
Model
CV
KT
VOC
SN
General
71.1
65.5
69.2
40.6
Spocific
74.4
67.4
28.7
8.1
Figure 4: The composition of the training datasets.
4.4 Exploring the Impact Training Datasets
To explore the impact of training dataset selection on model performance, we train domain-speciie model focusing on road driving scenes and a domain domain-general model on more datasets, as shown in Figure We conduct four sets of comparisons to evaluate the performance across datasets that were trained on both models, trained on one model and not on the other, and not trained on either model. As shown in Table 6 and Appendix F. the domain-specific model can focus more on learning features specific to the particular scene, resulting in slightly better performance on both trained and unseen driving scene datasets compared to domain-genert model. On the other hand, domain- -eneeal model trained on more scenes and more data exhibit better generalization performance. Therefore, while does not lag too far behind in performance driving scene datasets, it demonstrates overwhelming advantages on other scene datasets.
4.5 Qualitative Results
Figure 5 presents the qualitative analysis results of the label space learned by our model. We compare the label space learned by our model with the label space constructed using text features. The class curb in the IDD dataset actually encompasses both the classes curb and barrier in Mapillary, whereas the constructed label space by text features cannot handle such subclass/superclas relations. In contrast, our GNNs learned label space splits the IDD curb into two classes for prediction, effectively handling such label relationships. Similar situations also include the class tunnel or bridge in IDD dataset. However, since the proportion of bridge pixels is orders of magnitude greater than that of tunnels (10$ pixels for bridge and 10 pixels for tunnel), a more reasonable approach is to merge it with the class bridge, similarities, the Mapillary tunnel has been merged with the ADE fireplace. This does not actually introduce any conflict because no dataset simultancously annotates both the tunnel and fireplace. The model will construct the label space in way that facilitates its learning process.
GNNs have done. Additionally, it is worth noting that due visual
Conclusion
We propose a novel approach that leverages graph neural networks to construct unified label space for training semantic segmentation models across multiple datasets. Our method addresses the challenge of label conflicts in multi-dataset semantic segmentation and demonstrates performance improvements across various datasets. The unified label space generated during training, generalizes well to unseen datasets, showcasing the effectiveness of our approach.
Broader Impact. Our work explores the use of graph neural networks to unify label spaces between datasets, providing new direction for achieving robust and efficient multi-dataset training. By cnabling semantic segmentation models to be trained on multiple datasets with a unified label space, method can potentially reduce human effort required for re-labeling images and facilitate the expansion training datasets. This can lead to the development of models that are more universally applicable across various datasets, benefiting wide range of applications.
Limitations. Although our approach does not require manual relabeling efforts, it still relies on fully annotated datasets for training, in contrast to yeakly-superviiee and unsupervised methods. We aim to explore ways to integrate these alternative methods in future research. Errors the fully automated construction of unified label space do present some safety risks for autonomous driving tasks. Therefore, we recommend introducing a manual review mechanism to address these issues. Additionally, using ChatGPT may generate inaccurate label descriptions, which could affect the

---

IBD Dataset: aurb
100 Delaset:
MPI Dataset
Constructed by
curb
MPL:
IDD:
MPL
carb
curb
barrier
Image
Learned
MPL:
Medel
I80 Dataset:
ADE treepacee
bridge
by text featurre
MPL:
IDD
MPL:
ADE:
Image
bridge
freeplcee
bridge
Ground Truth
Learned by GNNs
MPL: turnel
ADE: fireplace
ID or bridgee
MPL:
ADE bridge
bridge
Figure 5: Comparison of unified label space learned by GNNs with constructed by text features.
prediction of label relationships. Therefore, we aim improve the accuracy of label descriptions by incorporating label descriptions provided by official datasets as prompts.
Acknowledgments
This work was supported in part by NSFC Project (62176061), Shanghai Municipal Science and Technology Major Project (202IHZZZZX0103) The computations in this research were performed using the CFFF platform of Fudan University.
References
[1] Robust vision challenge. http://ww..robustvisioo.. 12-02.
Accessed: 2022-
[2] Muhammad Awais, Muzammal Nascer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Minng-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721, 2023.
[3] Petra Bevandić, Marin Oršić, Ivan Grubišic, Josip Šarić, and Siniša Šegvić. Multi-domain semantic segmentation with pyramidal fusion. arXiv preprint arXiv:2009.0 000:20009001366 2020.
[4] Petra Bevandić, Marin Oršić, Ivan Grubišíć, Josip Šarić, and Siniša Šegvić. Multi-domain semantic segmentation with overlapping labels. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 2615-2624, 2022.
[5] Petra Bevandić, Marin Oršić, Josip Šarić, Ivan Grubišić, and Siniša Šegvić. Weakly supervised training of universal visual concepts for multi-domain semantic segmentation. International Journal of Computer Vision, pages 1-23, 2024.
[6] Petra Bevandié and Siniša Segvić. Automatic universal taxonomies for multi-domain semantic segmentation. British Machine Vision Conference, 2022.
[7] FM Bi, WK Wang, and I. Chen. Dbscan: density-bant based spatial clustering of applications with noise. J. Nanjing Univ, 48(4):491-49) 2012.
[8] Gabriel Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters, 30(2):88-97, 2009.
[9] Lei Cai and Shuiwang multi-scale approach for graph link prediction. In Proceedings of the AAAI conference artificial intelligence, pages 3308-33155 2020.
10

---

[10] Lei Cai. Jundong Li, Jie Wang, and Shuiwang Ji. Line graph neural networks for link prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5133-5113, 2021.
[11] Yanbei Chen, Manchen Wang, Abhay Mittal, Zhenlin Xu, Paolo Favaro, Joseph Tighe, and Davide Modolo. Scaledet: A scalable multi-dataset object detector. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7288-7297, 2023.
[12] Jesús Cid-Sueiro. Proper losses for learning from partial labels. Advances in neural information processing systems, 25, 2012.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[14] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1828-5839, 2017.
[15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
[16] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1331-1277 2013.
[17] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[18] Junheng Hao, Tong Zhao, Jin Li, Xin Luna Dong, Christos Faloutsos, Yizhou Sun, and Wei Wang. P-companion: A principled framework for diversified complementary product recommen- dation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2517 2514,2524, 2020.
[19] Dongwan Kim, Yi-Hsuan Tsai, Yumin Suh, Masoud Faraki, Sparsh Garg, Manmohan Chan- draker, and Bohyung Han. Learning semantic segmentation from multiple datasets with label shifts. In European Conference on Computer Vision, pages 20-36. Springer, 2022.
[20]
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2016.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson,
Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv: 026443 2023.
[22] Ajay Kumar, Shashank Sheshar Singh, Kuldeep Singh, and Bhaskar Biswas. Link prediction techniques, applications, and performance: A survey, Physica A: Statistical Mechanics and its Applications, 553: 124289, 2020.
[23] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2879-2888, 2288, 2020.
[24] Feng Lin, Wenze Hu, Yaowei Wang, Yonghong Tian, Guangming Lu, Fanglin Chen, Yong Xu, and Xiaoyu Wang. Universal object detection with large vision model. International Journal of Computer Vision, 132(4): 2024.
[25] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[26] Yajie Liu, Pu Ge, Qingjie Liu, Shichao Fan, and Yunhong Wang. An empirical study on multi-domain robust semantic segmentation. arXiv preprint arXiv:2212.04221, 2022.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 101, 2017.
[28] Shota Masaki, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Multi- domain semantic-segmentttion using multi-head model. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2802-2807. IEEE, 2021.
11

---

[29] Panagiotis Meletis and Gijs Dubbelman. Training semantic segmentation on heterogeneous datasets. arXiv preprint arXiv:2301.06634, 2023.
[30] Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu. Jianfeng Wang. Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1 11402-11411, 2023.
[31] Rohit Mohan and Abhinav Valada. Efficientps: Efficient panoptic segmentation. International Journal of Computer Vision, 129(5):551- 2021.
[32] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whai Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 891-898, 2014.
[33] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapil- lary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990-4999, 2017.
[34] Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 8277-8286, 2019.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
[36] Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2232-2241, 2017.
[37] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[38] Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang. Wenyu Liu, and Jingdong Wang. High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.044
2019.
[39] Marco Toldo, Umberto Michieli, and Pietro Zanuttigh. Unsupervised domain adaptation in semantic segmentation via orthogonal and clustered embeddings. In Proceedings of the JEEE/CVF Winter conference on Applications of Computer Vision, pages 1358-1368, 2021.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922
2023.
[41] Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. The missing link: Finding label relations across datasets. In European Conference on Computer Vision, pages 540-556. Springer, 2022.
[42] Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and
CV Jawahar. Idd: A dataset for exploring problems of autonomous navigation in unconstrained environments. In 2019 IEEF Winter Conference on Applications of Computer Vision ( (WACV), pages 1753-1751. IEEE, 2019.
[43] Petar Veličković. Everything is connected: Graph neural networks. Current Opinion in Structural Biology, 79:102238, 2023.
[44] Li Wang, Dong Li, Han Liu, Jinzhang Peng, Lu Tian, and Yi Shan. Cross-dataset collabora- tive learning for semantic segmentation in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2487-2494, 2022.
[45] Junfei Xiao, Zhichao Xu, Shiyi Lan, Zhiding Yu, Alan Yuille, and Anima Anandkumar. 1st place solution of the robust vision challenge 2022 semantic segmentation track. arXiv preprint arXiv:2210. 12852, 2022.
[46] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, Trevor Darrell, et al. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint 1rXiv 2(5);, 2018
12

---

[47] Oliver Zendel, Matthias Schörghuber, Bernhard Rainer, Markus Murschitz, and Csaba Beleznai. Unifying panoptic segmentation for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21351-21360, 2022.
[48] Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-free partial label learning. IEEE Transactions Knowledge and Data Engineering, 29(10):11552216, 2017.
[49] Muhan Zhang and Yixin Chen. Link prediction based graph neural networks. Advances in neural information processing systems, 31, 2018.
[50] Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A comprehensive survey on automatic knowledge graph construction. arXiv preprint arXiv:2302.0500 2023.
[51] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[52] Qiang Zhou, Yuang Liu, Chaohui Yu, Jingliang Li, Zhibin Wang, and Fan Wang. Lmseg: Language-guided multi-dataset segmentation. In The Eleventh International Conference Learning Representations, 2022.
[53] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571-7580, 2022.
[54] Guangming Zhu, Liang Zhang, Hongsheng Li. Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Bennamoun. Topalogy-learnabl graph convolution for skeleton-basee action recognition. Pattern Recognition Letters, 135:286-22: 292, 2020.
13

---

A Training Strategy
The training process is divided into three distinct stages: Multi-SegHead Training Stage, Alternating GNNs Training Stage and Alternating SegNet Training Stage, as shown in Algorithhm I Initially, we commence with the Multi-SegHese SegHead Training Stage, we discard the multi-heads and initialize the UniSegHead for predicting class probabilities during subsequent training phases. Subsequently, we alternate between the Alternating GNNs Training Stage and Alternating SegNet Training Stage for a total of three cycles, each lasting 20k iterations. Notably, we prioritize the Alternating GNNs Training stage at the beginning of each cycle.
Training Stage for 100k iterations. After completing Multi-
Multi-SegHeadd Training Stage aims to equip the segmentation network with fundamental segmen- tation capabilities. By commencing with a Multi- Setteed model, we provide a robust starting point for the subsequent stages, enhancing both performance and training efficiency. During this stage, the network trained on multiple datasets using specific segmentation heads for each dataset. Each segmentation head comprises feature weights W, of dimension C' x [La], where [Ts] represents the number of classes of dataset i. Pixel embedding P is multiplied by the i-th feature weight to make predictions within the i-th dataset's label space. This facilitates the utilization of dataset-specific annotations and cross-entropy loss for network training.
sy, = Σ Be log(softmax((.pp)) (P.)...
Alternating GNNs Training Stage. Training SegNet and GNNs simultaneously is challenging and impractical because training the segmentation network requires a discrete label mapping, while training the GNNs necessitates a continuous and differentiable adjacency matrix. Hence, we alter- nately training the GNNs and the segmentation network. The Alternttig GNNs Training Stage focuses on optimizing a unified label space. During this stage, the GNNs are trained while keeping the segmentation network frozen. When mapping the prediction from the unified label space to the dataset label space, we directly multiply the adjacency matrix M.. We calculate the loss according to Equation 8
Alternating SegNet Training Stage. During this stage, the GNNs remain frozen while the seg- mentation network is trained. We will use the technique detailed in Appendix Cto convert the continuous adjacency matrix into a discrete label mappings that satisfies the constraints, and then use it to compute the cross-entropy loss function and train according to Equation 6) During the last Alternating SegNet training stage, we extend the training procedure to 100k iterations and drop the entire GNNs but retain the dataset label mappings (M, = 1, .... K}. The unified label embedding X, in this stage is no longer keep frozen and trained with the SegNet. In the latter part of this stage, we will use training dataset to evaluate the model's performance. Those links between the unified label space and dataset-specific label space which is not activated during the evaluation will be removed. After removing the unused connections, And thus, we obtain the final model. When applying the final model to predict in the wild, we could manually assign each unified label node class name accord to their prediction and semantics.
continue training with the new label mappings.
B Selection of Unified Label Node Quantity and Initialization of Adjacency Matrix
The appropriate quantity of unified label nodes is crucial for balancing model performance and computational expenses. An excessive number of unified label nodes not only increases computational costs but may also result in redundancy in both the model and semantic space, leading to situations where multiple unified label nodes represent similar semantic concepts. Conversely, insufficient number of unified label nodes can prevent the model from fully expressing the entire semantic space
UP L, making it difficult to learn certain classes in the datasets, thus impacting model performance. Inspired by (53], we determine the number unified label nodes using the following steps. We first enumerate all feasible schemes for merging dataset labels, which form the Cartesian product of all dataset label sets I. = I. x T.2 ... x L.k. Label merging does not require the participation of all datasets simultancously, L; can be an empty element, indicating no participation in label merging.
14

---

Algorithm The training pipeline of our model
Input: the number of datasets K, the Segmentation Network Seg.et, the Graph Neural Networks GNNs, Multi-head iters Ip. GNNs training iters I and SegNet training iters I
Stage Multi-head stage; iter 0
for multi-dataset. sampled mini-batch M do
if Stage == Multi- Multi-SeHHead Training:
Calculate Lm by [Equationn
Update Seg.Net to minimize L.n.
if iter++> 1p: Replace Multi-SegHead with UniSegHead; Stage GNNs training; iter
if Stage == GNNs training:
Calculate C, by [Equation
Update Ns to minimize Lg
if ffiter++> > g: Solve dataset label mappings by Algorithm 2 Stage Seg training; iter
if Stage == SegNet training:
Calculate Loe by EEquation 6
Update Net to minimize Lee
if ittr++ > Ig: Stage GNNs training; iter =
end switch end for
We evaluate the quality of label merging using the predefined loss function Lc, where merging classes with the same semantic meaning results in smaller losses. Additionally, we require that the merging satisfies the constraint: each dataset node has only one corresponding merged node. It is worth noting that this constraint is stronger than our label mapping constraint described in in/sbsecction 3.1 thus ensuring that our adjacency matrix is always valid. We formulate optimization objective function as:
minimizez
Σ Es. 1ET Σ
E (()=
Le
St (Stt 0o:59
+ +ALL|
(10)
subject to LET
Zt =
Ve:
where T represents the set of all feasible edges connecting dataset label nodes to unified label nodes, re E {0, 1} indicates whether edge t E is selected, with 2t = 1 indicating the presence of edge and 0 indicating its absence. S₁. denotes the dataset to which edge t belongs, L. represents the label space of Sk. and t(c) is the label connected by edge t. A| is the penalty term for the number of nodes, encouraging the optimizer to merge similar nodes to obtain more compact unified label space. Based on experimental insights and referencing paper [52], we selected the hyperparameter
0.5. Le (st Šk is primarily used to evaluate the quality of label merging, where 5 represents
the output results on class using the original dataset segmentation head, and 3% represents the output results on class using segmentation heads from other datasets participating in the merging.
Le
St
5kk
Σ IOU(S IoU (sc),
(11)
2.E8t
where, /oU is metric commonly used to measure the performance of semantic segmentation models. Therefore, we use this metric here to evaluate the assistance of merged nodes to the model. By using linear optimization solver solve the optimization objective [qquation 10, can obtain a unified label space. We use the number of labels in this label space as the quantity of our unified label nodes and initialize our adjacency matrix based on their label mapping relationships.
C Solving the dataset Label Mappings
Given the definition of the label mappings and the constraints outlined in pnussbbection 3.1 solving the dataset label mappings could be conceptualized as a weighted bipartite graph matching problem. The weight of each edge is determined by the values of the GNNs learnable adjacency matrix. The objective to maximize the sum of edge weights while ensuring that each node of the specific dataset is connected. The unbalanced optimal transport algorithm offers approximate solution to such
15

---

Algorithm 2 The procedure of solving the dataset label mappings
Input: Submatrix of adjacency matrix for each dataset (5 i ,... KJ, the classes of each dataset {c(9) ..., }}, the number of unified label node N
Output: Mapping matrices M ..... ((K)
Initialize a
N 1N11. g
cl x0 I,..,, K). k), == 0,
for i
K do
cli xN UOT(a, ,
# Assign the label to the node with the highest score
M j.4 arg max, m .... j.N
for j = 1 to cl do
# Find an unlinked Bataset-specifie node
if max M == 0 then
9, = SQ
# Find first multi- node, where sum (M >
j'. K' findFirtMMulMappee((;;; .M())
# Replace the corresponding label mappings
M (1)
.'- 0, M
end if end for
Σ k= (1)
g(0) = p,((1 + +11- - 30)
end for
problems, but it does guarantee adherence to the constraint of connecting each node in the specific dataset. Therefore, based on the optimal matching result obtained from the algorithm, we employ greedy strategy adjust the matching scheme for all unconnected nodes within the specific dataset.
Initially, for every unconnected dataset-speeific node, we sort all unified label nodes based on their matching preferences, from highest to lowest. Subsequently, and their associated dataset-specific nodes. In cases where the associated dataset-specific node also connected by another unified label node, adjust the edge so that this unified label node links the unconnected specific dataset node, following this process until all nodes are connected. The specific procedure is outlined in Alllorittmm 2]
examine these unified label nodes
D Comparison with Baselines on Multiple Datasets
Table lists the results of mutual evaluation among training datasets. The results indicate that individually-traiin models generally demonstrate good accuracy when tested on the same dataset, but perform poorly on other datasets. Our method can leverage knowledge from multiple datasets to improve performance on individual datasets. Table lists the results of the Multi Multi-SegHead model's predictions on different datasets using different segmentation head outputs. It can be observed that none of the segmentation head outputs could consistently provide accurate predictions across all datasets, indicating that their label spaces do not effectively cover all datasets. [Table 9and Table 10) list the performance of different models on unseen datasets. The results indicate that our method has better generalization performance unified label space we construct contains richer semantic information, enabling flexible adaptation to datasets from different scenarios.
unseen datasets compared to other methods. Additionally, the
16

---

Table Semantic segmentation accuracy (mloU) on training datasets compared with Single dataset
model.
Train Test
CS
MPI.
SUN
BDD
IDI
ADE
COCO
Mean
CS
77.0
7.2
3.7
43.4
30.4
2.6
2.8
23.9
MPL
65.6
30.2
4.8
54.5
48.4
4.7
5.9
30.6
SUN
13.0
3.4
43.9
12.4
9.5
5.1
3.0
12.9
BDD
58.9
12.1
3.6
62.4
30.9
2.7
2.7
24.8
IDD
50.9
10.4
4.3
43.5
66.8
3.1
2.8
26.0
ADE
42.7
11.5
35.5
34.7
27.5
34.5
12.8
28.5
COCO
45.5
14.0
29.1
42.0
30.1
14.8
38.0
30.5
Ours
80.7
43.7
47.5
65.5
68.6
42.0
46.7
56.4
Table 8: Semantic segmentation accuracy (mloU) training datasets compared with Multi-SegHead.
Label (esce
CS
MPL
SUN
BDD
IDD
ADE
COCO
Mean
CS
79.5
16.1
10.4
61.6
42.8
2.5
3.6
30.9
MPL
72.9
36.1
12.9
60.4
54.7
7.0
8.9
36.1
SUN
33.5
47.3
29.5
6.4
5.3
22.0
BDD
67.1
16.6
8.9
65.6
39.0
2.6
3.6
29.1
IDD
72.3
20.3
10.8
59.8
67.0
3.3
4.3
34.0
ADE
62.7
20.8
43.5
57.6
46.6
30.5
20.5
40.3
COCO
59.5
21.4
34.4
57.5
44.4
18.0
36.7
38.8
Ours
80.7
43.7
47.5
65.5
68.6
42.0
46.7
56.4
Table 9: Semantic segmentation accuracy (mloU) unseen datasets compared with Single dataset.
Train Test
KT
SN
CV
VOC
CT
Mean
CS
62.3
4.3
67.5
16.1
7.0
31.4
MPL
67.3
6.1
73.1
24.6
13.3
36,9
SUN
12.4
32.2
20.2
BDD
56.0
5.3
66.7
17.3
6.7
30.4
IDD
50.6
6.1
62.6
17.7
7.4
28.9
ADE
45.6
25.8
51.7
39,0
23.7
37.2
COCO
45.4
27.4
53,9
35.8
45.1
Ours
65.5
40.6
71.1
69.2
38.1
56.9
Table 10: Semantic segmentation accuracy (mloU) on unseen datasets compared with Multi-SegHead.
Label space)
KT
SN
CV
VOC
CT
Mcan
CS
59.7
14.6
72.3
29.0
9.3
37.9
MPL
63.7
18.3
72.8
33.6
16.4
SUN
27.0
42.9
34.9
29.9
10.0
28.9
BDD
61.0
15.7
71.2
21.9
8.7
35.7
IDD
56.8
16.3
69.0
29.8
10.3
36.4
ADE
56.4
40.8
65.1
56.2
30.7
49.8
COCO
55.9
37.6
64.6
69.6
53.2
Ours
65.5
40.6
71.1
69.2
38.1
56.9
17

---

Table 11: Performance on both trained
Table 12: Unseen domain -eniinal model vs.
datasets.
Trained domain-specifie model.
Driving scene dataset
Driving scene dataset
Model
Trained
CS
MPL
BDD
IDD
Model
Trained
WD2
General
80.7
43.7
65.5
68.6
General
x
40.3
Specific
82.2
45.7
68.8
71.4
Specific
50.2
Table 13: Trained Domain-geneere model vs.
Table 14: Performance both unseen datasets.
Unseen domain-specine model.
Driving
Non-Driving
Non- Driving scene dataset
Model General
Trained
CV
KT
VOC
SN
Model
Trained
SUN
ADE
COCO
x
71.1
65.5
69.2
40.6
General
V
47.5
42.0
46.7
Specific
x
74.4
67.4
28.7
Specific x
4.4
5.4
7.3
E
Exploring the Impact of Training Datasets
Tables 11 14 present performance comparison between the domain-general and domain-specific models across various training settings and datasets. Figure 6illustratess the segmentation results of these models on the different datasets.
Performance Comparison on Both Trained Datasets. From Table 11] it can concluded that the performance of the domain-specific model focusing driving scenes is higher all trained datasets compared to the domain-genen general model, despite the domain-genett model being trained on five times more samples than the domain-specific model. This is likely attributed the fact that the sppeific model needs to predict fewer classes and is less affected by label space conflicts. The model can therefore focus on learning features specific to the particular scene.
Performance Comparison: Unseen Domain-general Model vs. Trained Domain-specific Model. In Table 12] we selected driving scene dataset that the domain-specific model was trained while the domain-specific model was not trained on it, for comparison. As expected, the nomain-specific model demonstrates superior performance, and the domain-general model closely follows suit.
Performance Comparison: Trained Domain-general Model vs. Unseen Domain-specific Model. In Table 13] we selected several non-drin non-driving scene datasets that the domain general model was trained on. while the Noominn-speciff model was trained on it. The Nomain-sspeciff model fails to predict in non- non-driving scenes. Compared to Table 12] the generalization performance of the domain-specifee model in unseen scenes markedly trails behind the domain-general model. Although the domain-specins model may have encountered similar objects in driving scene datasets like wall, still struggles to predict these seen objects well non-driving scene datasets, as shown Figure
Performance Comparison on Both Unseen Datasets. In Table 14 we selected datasets that were unseen both models, including driving scene datasets and non-driving scene datasets. We can observe that the generalization performance of the domain-specific model is slightly better in driving scenes, but its performance in other scenes lags far behind that of the domain domain-general model. Overall, models trained on more scenes and more data tend to achieve better generalization performance.
18

---

CS
IDD
WD2
SUN
COCO
CV
Image
GT
General
Specifie
BDD
MPL
ADE
KT
SN
VOC
Image
GT
General
Specific
Figure 6: Visual comparisons of different training dataset models.
19

---

Visualization
Figure presents the visual comparisons of models trained on single dataset and our model predicted in universal label space. From the figure, it is evident that our model achieves consistently strong performance across all training datasets while integrating label spaces from different datasets. For example, it predicts class lane marking and crosswalk for the ADE and BDD datasets, and predicts class books for the SUN dataset. Figure 8/sows our universal predictions on test datasets. The results demonstrate that our method generalizes well across multiple unseen datasets from different domains. Figure shows the visual comparisons of different models on the WildDash benchmark.
CS
MPL
SUN
BDD
IDD
ADE
COCO
Image
GT
Ours
CS
MPL
SUN
BDD
IDD
ADEE
COCO
Figure Visual comparisons on training datasets.
20

---

KT
WD2
SN
CV
VOC
CT
Image
GT
Ours
Figure 8: Visual comparisons on unseen test datasets.
Image
GT
Our trained
model
MIX6D
FAN
Uni NLL+
Our unseen
model
UniSeg
Figure 9; Visual comparisons on WildDash 2 benchmark.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The final paragraph of our introduction (eectinn 1) provides comprehensive
summary of the paper's contributions, which align with the claims made in the abstract.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the third paragraph of our conclusion (section 5). we discuss the limita- tions of our work, including factors influencing performance, and computational efficiency considerations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [es]]
In section 3. Appendix A, we provide detailed implementation steps, ensuring that our experiments can be replicated. Additionally, sossection 4 we thoroughly describe the settings of our experiments, further facilitating reproducibility.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: We provide reproducible code along with detailed instructions, ensuring that the main experimental results can faithfully reproduced.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: section 4
we provide comprehensive details of our experimental setting.
7. Experiment Statistical Significance
Question: Does paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Training on numerous datasets is time-consuming, with each experiment taking over week. During our experiments, we observed consistent and stable performance in terms of mloU for model.
8. Experiments Compute Resources
22

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: In sessection we specify that our experiments were conducted on four 80G A100 GPUs.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of
Answer: [Yes]
Justification: We have read and adhered to the NeurIPS Code of Ethics conducting our research.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the first paragraph of our conclusion (section 5). we thoroughly discuss the broader impacts of our work.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our research utilizes openly available datasets for training, and our models primarily used for semantic segmentation tasks. Therefore, there are no such risks associated with our work.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: section 4] we clearly specify the assets used in our work, including datasets and models. We provide proper citations to the original papers and ensure that the licenses and terms of use are explicitly mentioned and respected.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We provide our code as asset and include comprehensive documentation alongside it, covering details such as training procedures, licensing, limitations, and any necessary consent obtained from individuals whose assets used.
Justification:
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing experiments research with human subjects.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve human subjects.
24


Reviews:
[{"review_id": "uHhWFXdB8h", "reviewer": "Reviewer_GmR6", "summary": "This paper presents a novel approach for automated label unification across multiple datasets in semantic segmentation using Graph Neural Networks (GNNs). The proposed method aims to create a unified label space that allows semantic segmentation models to be trained on multiple datasets simultaneously without the need for manual re-annotation or taxonomy reconciliation. The results demonstrate significant performance improvements over existing methods, achieving state-of-the-art performance on the WildDash 2 benchmark.", "questions": "Please see the weaknesses.", "limitations": "The paper is interesting and the experiments seem to prove the effectiveness. But I need more evidence to prove the robustness of the GNN results when using different datasets and models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Novel Approach: The use of GNNs to automatically construct a unified label space is innovative and addresses the challenge of conflicting label spaces in multi-dataset training.\n+ Performance: The method shows impressive performance improvements, achieving state-of-the-art results on the WildDash 2 benchmark and outperforming other multi-dataset training methods.\n+ Efficiency: The approach eliminates the need for manual reannotation and iterative training procedures, significantly enhancing the efficiency of multi-dataset segmentation model training.\n+ Robustness: The method demonstrates good generalization to unseen datasets, indicating its robustness and applicability across various scenarios.", "weaknesses": "This indeed seems a interesting problem. I have several questions:\n\n1. How do you select the unified label size N? Is it adaptive to different dataset labels, or preset hyper-parameters?\n2. Can you show more results of the bipartite graph generated the GNN? Like what labels is hard to link by text features, but linked by the GNN.\n3. Is the generated graph by GNN influenced by the number of datasets in training or the models? For example, if I use different models for the training, will the graph be the same? If I use two dataset in multi-dataset training, will the nodes of these two dataset labels have the same edges as in when I use three datasets?\n4. Can I apply the generated graph to other models' training without the need for doing GNN training again? If not, how much training cost will GNN bring?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank for your thoughtful questions. Below are our responses to the specific questions raised:\n\n**Q1: How do you select the unified label size N? Is it adaptive to different dataset labels, or preset hyper-parameters?**\n\nA: We apologize for not clearly explaining the selection of the unified label size N in the main text, which might have caused some confusion. In Supplementary Material B, we clarify that N is determined by solving Algorithm 2, rather than being a preset hyper-parameter. Specifically, we construct the label co-occurrence relationships through cross-validation of the multi-seghead model across different datasets. This automatic solving method allows us to adapt to different dataset labels and enhances scalability. We will add this explanation to Section 3.2 of the main text in the revised version of the paper.\n\n**Q2: Can you show more results of the bipartite graph generated by the GNN? Like what labels are hard to link by text features, but linked by the GNN.**\n\nA: Please refer to our global rebuttal for more examples. For instance, ADE: Signboard is annotated as both Traffic sign (front) and Traffic sign (back), while CS annotates Traffic sign (back) as an ignore category. We believe that such knowledge should be learned through samples.\n\n**Q3: Is the generated graph by GNN influenced by the number of datasets in training or the models? For example, if I use different models for the training, will the graph be the same? If I use two datasets in multi-dataset training, will the nodes of these two dataset labels have the same edges as when I use three datasets?**\n\nA: Yes, the selection of different models and datasets affects the constructed nodes. The GNN relies on the model’s segmentation predictions to learn the label mapping matrix. If the model cannot accurately recognize fine-grained categories, the GNN will not be able to construct the label mappings for these fine-grained categories. The influence of datasets on the generated graph is likely present. For instance, in 5ds training, IDD: Non-drivable fallback or rail track is merged with Terrain. However, in 7ds training, after removing WD2: Terrain and adding COCO: Railroad, IDD: Non-drivable fallback or rail track is merged with the fine-grained categories MPL: Rail Track and COCO: Railroad. We speculate this is because the inclusion of COCO: Railroad in 7ds training improves the model’s learning of fine-grained categories, affecting the prediction results for Non-drivable fallback or rail track in the IDD dataset, leading to different outcomes.\n\n**Q4: Can I apply the generated graph to other models' training without the need for doing GNN training again? If not, how much training cost will GNN bring?**\n\nA: Yes, similar to other multi-dataset training methods that apply manually designed unified label spaces to different models, our GNN model constructs a unified label space as a boolean label mapping matrix that can be separated and used for training other models. We will open-source the constructed unified label space along with the revised version of our code to facilitate training with other models.\n\nIf there are any further questions or concerns, please feel free to reach out to us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VWDOPYVVZv", "reviewer": "Reviewer_kdwE", "summary": "The paper presents an approach to automatically train a semantic segmentation network using multiple datasets with individually different class label policies. A unified label space emerges during training automatically without direct manual label mapping definitions by using a graph neural network (GNN) which is guided by textual descriptions of each label in unison with a multi-head segmentation head. The training alternates between GNN and segmentation network training to jointly optimize the label mapping for multiple dataset as well as the actual image segmentation task itself. Multiple experiments evaluate the approach to unify 7 mixed datasets (CS, MPL, SUN, BDD, IDD, ADE, COCO) as well as 5 road-scene datasets (CS, IDD, BDD, MPL, WD2). Performance is compared by both training single heads and using the Multi-Seg. Head. as well as benchmarking on WD2. Additional ablation experiments showcases the value of textual label descriptions for the unification process. \nIn summary this approach allows for a joint training with multiple mixed segmentation datasets without additional manual intervention.", "questions": "*) The creation of textual descriptions using ChatGPT is not described at all; which version of ChatGPT was used? How were per-dataset label spaces described as textual input? Were images/masks/masked images used with a multi-modal version of ChatGPT (e.g. GPT4V; GPT4o; ...). Using masks together with images as inputs for ChatGPT is non-trivial. L110 contains no details; wording in Figure 2 suggests a multi-modal model as \"image of\" is part of the dataset label text or simply the \"blind\" pure-text completion of prompts in the style \"An image of <label> from the dataset <dataset>....\". Please clarify; the text label descriptions are an important aspect of the paper's novelty!\n*) Where is the final resulting unified label mapping?  Figure 5 shows a small part but the full joined label set would give a good indication of generalization vs. specialization. \n*) Supplem. B. (target number of unified labels) is a crucial component as it directly steers the separation/merging of similar concepts (e.g. seperate \"Cityscapes_road\"/\"MPL_road\" labels vs. a unified \"road\" label). How is the hyperparameter Lambda|L| chosen and what number of labels did the experiments end up with?\n*) What pre-training was used for the initialization of the multi-heads?\n*) WD2 negative examples are a defined subset within the benchmarking dataset. However, the official rules of the benchmark state, that both regular/\"positive\" and negative frames/subsets have to be treated equally. The comparability within the benchmark is only held upright, as long as this rule is observed.  L246/L247 can be interpreted otherwise: \"We map non-evaluated classes to a void label\"; is this applied to both positive and negative frames alike?", "limitations": "*) Limitations relating to dataset bias could be mentioned.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "*) Generation of segmentation label space mappings to automatically create a unified label space.\n*) Great overall performance for each tested dataset\n*) Record benchmark results for Wilddash2\n*) Many experiments to showcase the approach on multiple driving-scene datasets as well as some more generic datasets.\n*) Supplemental contains the full training code to recreate the experiments", "weaknesses": "*) Some details are omitted (see Questions) making it hard to judge the extent/validity of some claims (e.g. robustness/generalization).\n*) Multi-head training and evaluations favor the accumulation of per-dataset bias. The cross-validations are a good start but there is no evaluation of \"new\"/out-of-distribution labels from one dataset within the other.  This amplifies dataset-specific training (each head is optimized to work for one specific dataset); thus training to \"solve\" datasets rather than the task at hand. L281 (tunnel == fireplace) is a great observation but also shows the weakness of the approach.\n*) \"Splitting\" of source labels is not supported; the multi-heads are trained in isolation so if a label in one dataset requires splitting in another dataset, the respective heads are isolated (one has the split, the other doesn't). During inference of an unseen image this can result in only a single merged representation instead of the more detailed split. This effect is underrepresented during evaluation as dataset bias further separates the heads to work in favor of the overlapping test data.\n*) Both WD1 and WD2 are used. WD2 supersedes WD1 in every way /(it also includes all WD1 frames with the extended label policy) and crucially also uses a unified label policy combining IDD, MPL, CS & WD1 (80 label cat.; the benchmark uses the reduced WD2_eval 25 cat policy). So contrary to L223, WD2 does contain the labels \"lane marking\", \"crosswalk\" and \"manhole\". WD2 has a different publication associated with it which is unmentioned: CVPR22 \"Unifying Panoptic Segmentation for Autonomous Driving\". The paper/experiments would be easier to understand with only WD2.\n*) For some figures/descriptions in the paper it is unclear which dataset combinations was used during training. Introduction of fixed names (e.g. 5ds vs. 7ds for 5/7 datasets; or explicitly mention \"domain-general\"/\"domain-specific\" wording) could help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful inquiries and for pointing out crucial elements of our methodology. Your feedback is instrumental in refining our paper and addressing any weaknesses. Below, we summarize your questions and provide detailed responses.\n\n**Q1: About the creation of textual descriptions using ChatGPT.**\n\nA: Thank you for pointing this out. We apologize for not providing specific details in our initial submission. In the revised version of the paper, we will clarify that we used ChatGPT 3.5 to generate label descriptions without any image input. Each dataset label category was formatted into the text input as “An image of <label> from the dataset <dataset>.” This prompt was used to encourage ChatGPT to complete the label descriptions. The resulting complete label descriptions were then inputted into a text encoder to generate the corresponding textual features for the label nodes.\n\n\n**Q2: Where is the final resulting unified label mapping?**\n\nA: Please refer to our global rebuttal for more examples. We plan to present our complete unified label mapping in the form of open-source code.\n\n\n**Q3: The importance of Supplement B regarding unified labels and the choice of hyperparameter Lambda |L|.**\n\nA: We appreciate you highlighting the importance of this aspect. Based on experimental insights and referencing paper [52], we selected the hyperparameter ($\\lambda$ = 0.5), as this setting provided a good initial value. In our experiments with seven training datasets (CS, MPL, SUN, BDD, IDD, ADE, COCO), we had a total of 448 label categories. With the selected parameter, the resulting unified label space \\(|L|\\) comprised 231 labels. We will include these details in the revised version of the paper.\n\n\n\n**Q4: What pre-training was used for the initialization of the multi-head models?**\n\nA: In the Multi-SegHead Training Stage section of Supplementary A, we describe the process of training our multi-head model. Specifically, we construct a separate segmentation head for each dataset, where each head contains only a linear layer to map the embedded feature channels to the specific label categories of that dataset. The training was conducted using four 80G A100 GPUs for 100K iterations. Image preprocessing and hyperparameter settings were consistent with those used in other training stages.\n\n\n**Q5: Treatment of WD2 negative examples in benchmarking datasets.**\n\nA: We apologize for the lack of clarity in our description, which may have caused confusion. We adhere strictly to the WD2 evaluation guidelines, applying uniform label mapping to all samples. This process is indeed applied to both the positive and negative frames. We will explicitly include this description in the revised version of the paper.\n\n---\n\n**Weakness 1: Multi-head training and dataset bias concerns.**\n\nA: Thank you for highlighting the issue of dataset bias in multi-head training. We acknowledge this limitation. After completing cross-validation, we discard the multi-heads and the model utilizes a unified label embedding space for predicting class probabilities during subsequent training phases, which should help mitigate biases introduced by the initial training setup. We believe adding more datasets may help alleviate out-of-distribution label concerns, as combining datasets may reduce errors from mislabeling during cross-validation. We also recognize the importance of distinct semantic gaps, such as between \"tunnel\" and \"fireplace\". We plan further research to leverage label descriptions from official datasets and utilize advanced large models to provide accurate label mappings while constraining the model’s output based on textual features to minimize incorrect connections between semantically different classes.\n\n\n**Weakness 2: Splitting of source labels with respect to multi-head training.**\n\nA: We apologize for any confusion regarding the isolation of multi-heads training. After the initial multi-seghead training stage, we discard the multi-head structure, and thus, only a single UniSeghead is maintained for predictions. During inference on unseen images, we solely output results from the unified label space, eliminating the complications related to the isolation of respective heads. This will be clarified in the revised version of the paper. Additionally, our visualization results (in Supplement F) show that the final model generates fine-grained segmentation results.\n\n\n**Weakness 3 & 4: The use of both WD1 and WD2, and unclear figures or descriptions.**\n\nA: Thank you for noting the potential confusion arising from using both WD1 and WD2. To clarify, our model did not utilize WD1 during training, and we plan to remove references to testing on WD1 in the final version of the paper for better understanding. We also appreciate your reminder regarding the citation of the WD2 paper, which we will include in the revised version of the paper. Additionally, thank you for pointing out the lack of model descriptions in our visual results that caused confusion. The L223 experiment in the paper refers to the 7ds training model, which does not include WD2, resulting in only MPL having *lane markings*, *crosswalks*, and *manholes*. We will add more detailed descriptions to the images (Fig. 3 and Fig. 5) and revise 'Ours' to 'Our 7ds model'.\"\n\n\nWe hope these responses provide clarity and address your concerns effectively. Thank you once again for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ge9HavrD1F", "reviewer": "Reviewer_M8bi", "summary": "This paper introduces a method to automatically match and unify different label spaces for semantic segmentation. This allows to train a single model on multiple, differently annotated datasets. The authors can show that this can yield benefits in overall model performance, also compared to other multi-label approaches. The method is the currently best result on the public WildDash2 benchmark.", "questions": "see above. The points that can hopefully easily be clarified by the authors are the question how visual information is used in the matching, what to do about GPT outputting wrong definitions, and why the method underperforms on more general datasets with larger label spaces.", "limitations": "-\tThe broader impact does not at all discuss potential societal impact of the work. While it is true that it reduces the re-labeling effort, the method still requires segmentation labels, which are the much higher cost compared to re-labeling existing segmentation labels. What should be rather discussed are the societal impact of deploying a method with an automatically generated label space, and what the implications of this are to safety and certification when trying to deploy this method e.g. to automated driving.\n- The discussed limitations are OK. It would be interesting of the authors could further comment on ways how to potentially get around the scaling problem of current requiring all datasets loaded on one node, and every dataset beeing sampled for every batch.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThis method is the current SOTA on the public learderboard of WildDash2, which means the proposed method holds against a very rigorous testing setup.\n-\tWith the current knowledge that data scale is one of the key ingredients in well-generalizing models, progress on multi-dataset-training has high potential for impact. For example, the current leader of all ScanNet 3D segmentation leaderboards is also a method based on multi-dataset-training. \n-\tThe presentation is very clear and structured for the most parts of the paper.", "weaknesses": "-\tIt is currently unclear to me based on what information the graph connectivity is learned. How the method is described, the name of the annotated label is the only information that is put into the nodes, and all matching is based on that plus a hallucinated description from GPT. However, the authors mention in line 199-200 that the method would leverage visual similarity between annotations. It is not clear to me how this is facilitated.\n-\tAll results are based on an architecture from 2019. It is lucky that WildDash2 has no stronger competitors in the leaderboard, but it remains unclear whether the method would yield the same improvements for a more competitive base architecture (e.g., mIoU of 77% on Cityscapes is a very bad baseline performance in Table 2. The achieved “improved” 80.7% are on-par with DeepLabv3, which was SOTA in 2018).\n-\tThe method underperforms on more general indoor/outdoor datasets. Both on ADE and COCO, MSeg is better (even when predicting less classes, and in contrast to the bolded numbers, which are somehow ignoring MSeg)\n-\tI don’t think it is well discussed how hallucinating label descriptions through ChatGPT can introduce mistakes into the methodology. I don’t know about all of the datasets, but for example Cityscapes provides their own descriptions of the labels, which are the descriptions that also label workers use to label the data (see https://www.cityscapes-dataset.com/dataset-overview/#labeling-policy ). LLMs that are simply prompted with the label name can easily make up wrong descriptions that are not calibrated to the label policy of the annotation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for highlighting the areas that require clarification in our paper. We appreciate your insights, which allow us to improve our manuscript. Below, We summarize each of your questions and provide detailed responses.\n\n**Q1: Clarification on how graph connectivity is learned and visual similarity is utilized.**\n\nA: Thank you for your question. The node features in our approach only contain textual characteristics. The visual information, is provided by the segmentation model during training. Specifically, the graph neural network outputs embeddings and adjacency matrix that are utilized in the segmentation network's inference process. For the image training samples, the segmentation network encodes visual embedding features. The GNN makes predictions by classifying and mapping these visual features to labels. The loss function is computed using formulas (4-6), and the loss value is backpropagated through the GNN model to update the parameters.\n\n\n**Q2: Concerns about using an older architecture and its competitive performance.**\n\nA: We appreciate your observations regarding the architecture used in our studies. To ensure a fair comparison, we selected this architecture based on the configuration outlined in Mseg [23]. Training on seven datasets is indeed time-consuming (approximately one week on four 80G A100 GPUs), which limits our capability to evaluate more competitive architectures at this stage. However, our results (Table 11) from five datasets show potential for improvement, as we've enhanced Cityscapes' mIoU to 82.2%. This indicates that there is room for further optimization in our method. We will explore more competitive base architectures, such as transformer-based models, in our future research.\n\n\n**Q3: Performance evaluation relative to the MSeg approach on indoor/outdoor datasets.**\n\nA: Thank you for raising this question. It's worth noting that MSeg combines categories and overlooks some difficult classes, which simplifies the learning task. Many of these difficult classes correspond to IoU values lower than the overall mean IoU (e.g., for ADE, mIoU for omitted/merged classes is 37.5 vs. all categories at 42.0; for COCO, it's 37.1 vs. 46.7). MSeg's merged categories also take into account label alignment across different datasets, which eases the learning process and thus may not represent a fair comparison to our method. We have provided examples of merged categories and their IoUs to help illustrate this point:\n\n|MSeg Label| ADE Label| IoU|\n|---|--|--|\n| **table**| coffee table| 55.9|\n|| table| 50.6|\n| **terrain**| land| 0.4|\n|| earth| 31.1|\n|| field| 22.1|\n|| grass| 64.7|\n|| sand| 36.4|\n|| dirt track| 6.4|\n| **mountain_hill**| mountain| 56.2|\n|| hill| 8.9|\n| **car**| car| 84.3|\n|| van| 35.9|\n| **stairs**| stairs| 28.5|\n|| stairway| 28.9|\n| **railing_banister**| railing| 26.4|\n|| bannister| 10.0||\n| **unlabeled**| computer| 66.3|\n|| signboard| 36.3|\n|| monitor| 45.5|\n|| crt screen| 5.0|\n|| screen| 63.9|\n|| canopy| 8.8|\n|| plant| 44.0|\n|| bar| 23.8|\n|| step| 11.7|\n\n| MSeg Label| COCO Label| IoU|\n| --| --|--|\n| **building**| house| 30.0|\n|| roof| 52.7|\n|| building-other-merged| 46.6|\n| **floor**| floor-wood| 36.0|\n|| floor-other-merged| 47.8|\n| **table**| table-merged| 38.3|\n|| dining table| 75.8|\n| **terrain**| grass-merged| 37.6|\n|| sand| 14.5|\n|| dirt-merged| 22.5|\n| **wall**| wall-brick| 24.9|\n|| wall-stone| 54.9|\n|| wall-tile| 30.3|\n|| wall-wood| 16.0|\n|| wall-other-merged| 43.1|\n| **vegetation**| flower| 10.9|\n|| tree-merged| 49.5|\n\n\n**Q4: Potential issues with hallucinated label descriptions through ChatGPT.**\n\nA: Thank you for your invaluable input regarding the potential inaccuracies in generated label descriptions. We agree that using official label descriptions from datasets like Cityscapes greatly enhances the reliability of our methodology. Given that some datasets may lack corresponding label descriptions or have inconsistent language styles, this can pose challenges for the model. Therefore, in the future, we plan to use the label descriptions provided by official datasets as prompts, employing GPT models to generate more accurate and stylistically consistent label descriptions to help the model learn better. This adjustment may help improve the model's learning process by reducing the risks associated with hallucinated descriptions.\n\n\n**Limitation: Discussion on societal impacts and scalability challenges.**\n\nA: Thank you for the addition to the limitations section. Indeed, compared to unsupervised or weakly supervised methods, we still require complete annotated data. However, our approach leverages the available annotation information in existing datasets to reduce the reliance on annotated data. Errors in the fully automated construction of a unified label space do present some safety risks for autonomous driving tasks. Therefore, we also recommend introducing a manual review mechanism to address these issues. Current automatic label generation methods may introduce significant safety risks, so we recommend incorporating a manual review mechanism for generated labels to ensure accuracy and mitigate these concerns.\n\nRegarding scalability, we concur that it's not necessary to load all datasets on a single node. Instead, as long as the computations for weight gradient updates include representations from all datasets, we can distribute this across multiple nodes, allowing more efficient processing. This approach can help facilitate scaling while avoiding bottlenecks in performance. We will include additional descriptions of these limitations in the final version of the paper, particularly focusing on safety considerations for autonomous driving scenarios.\n\nWe hope that these responses provide the clarity you were seeking and address your concerns adequately. We sincerely appreciate your constructive feedback, which is invaluable in improving our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OnCJlUnF4Q", "reviewer": "Reviewer_dye9", "summary": "This paper introduces a method using GNN to automatically construct a unified label space across multiple datasets, addressing the issue of label space conflicts in multi-dataset semantic segmentation. The method eliminates the need for manual re-annotation or iterative training, significantly enhancing the efficiency and effectiveness of model training. Experiments show that this method has certain effectiveness.", "questions": "Please see Weaknesses.", "limitations": "Limitations are discussed in the Conclusion.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation behind this approach is clear, and automatically constructing a unified label space across multiple datasets in multi-dataset semantic segmentation is meaningful.\n2. The authors use numerous illustrations and provide a detailed description of the experimental parameters, making it easy to follow and reproduce.", "weaknesses": "1. This paper proposes a new method, but in my view, it is merely a collection of tricks. For example, $d_i$ is introduced in Equation (1), but the authors neither explain its role through ablation experiments nor from a methodological perspective. It is recommended that the authors provide a more detailed explanation of the motivation for each component of the method.\n2. Although the motivation for this work is multi-dataset semantic segmentation, the experimental comparison methods do not include the latest methods from the multi-dataset semantic segmentation community, such as [1,2], etc.\n3. This paper only uses HRNet-W48 as the backbone. The authors are encouraged to further explore the scalability of the proposed method on transformer-based networks.\n4. From Table 5, it appears that the improvement in results due to label description is not significant. However, using GPT to complete this step inevitably incurs a substantial consumption of time and resources, which raises questions about whether it is worth it. Additionally, the authors should discuss the impact of the proposed method on time and space costs.\n\n[1] Gu X, Cui Y, Huang J, et al. Dataseg: Taming a universal multi-dataset multi-task segmentation model. NeurIPS 2023.\n\n[2] Wang L, Li D, Liu H, et al. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. AAAI 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to review our work. Below, we summarize each of your questions and provide detailed responses.\n\n**Q1 & Q2: Concerns about the lack of detailed explanation and the omission of the latest methods**\n\nA: We respectfully disagree with your assessment. Our approach is fundamentally different from existing methods and two methods you mention. While existing methods require manual re-labeling (e.g., MSeg [23]), expert knowledge (e.g., NLL+ [4]), or are limited to only two datasets (e.g., Auto Univ. [6]), our method **automatically** constructs **a unified label space** across multiple datasets without human intervention or iterative processes. Furthermore, we are the first to leverage Graph Neural Networks for this task.\n\nThe papers you mentioned do not construct a unified label space. Instead, they utilize different segmentation heads (with different weights) for evaluation across various datasets. For instance, [1] employs a text encoder to encode label categories into corresponding embedding spaces for each dataset, predicting within dataset-specific embedding spaces. This method also struggles with handling categories that share the same name but have different annotation standards across datasets (e.g., IDD \"curb\" vs. MPL \"curb\"). This challenge is precisely why we introduced $d_i$ in Equation (1), allowing nodes with the same name from different datasets to obtain distinct node features, thereby differentiating these nodes.\n\nOn the other hand, [2] uses dataset-specific batch normalization and heads, which can be heavily influenced by dataset biases. In real-world inference scenarios, the outputs from different segmentation heads may conflict, complicating practical application. Our method, however, consistently predicts across different datasets using a unified label space (with the same weights), employing different boolean label mapping matrices solely for performance evaluation. In practical inference, this label mapping is unnecessary. Therefore, methods [1-2] require prior knowledge of the target label space for predictions, which does not align with our task setup and is not suitable for comparison.\n\n\n**Q3: Suggestion to explore scalability on transformer-based networks.**\n\nA: Thank you for your valuable suggestion. We acknowledge the lack of experiments exploring the effectiveness of our method across different models. Training on seven datasets requires approximately one week on four 80G A100 GPUs, which limits our ability to provide results using transformer-based networks at this time. However, we plan to investigate the scalability of our method with transformer-based architectures in future work.\n\n\n**Q4: Concerns about the significance of improvements from label descriptions and resource consumption.**\n\nA: We appreciate your feedback regarding the use of GPT for generating text descriptions. It is important to clarify that the process of generating these descriptions using the text encoder is not conducted in real-time; it involves a one-time inference step. The time cost for this step is less than one minute, and the space requirement for preserving text features is only 3.5MB, which is negligible compared to the overall training costs. Furthermore, after training, the GNN component is discarded, meaning it does not introduce any additional overhead during the inference phase.\n\n\nWe hope these clarifications address your concerns effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0PR5BF82N", "reviewer": "Reviewer_pNQc", "summary": "The paper proposes a multi-dataset training approach that works on top of a unified output taxonomy mapped onto individual dataset-specific taxonomies. First, a multi-head model is trained. Then, a unified taxonomy is automatically constructed by merging semantically identical classes, determined based on segmentation performance on individual classes when a connection is made. After that, GNNs are used to further refine the mapping from the individual dataset-specific classes to the unified taxonomy.\n\nThe input nodes of the GNN are dataset-specific labels and unified classes. Dataset-specific labels are first enriched with a textual description provided by ChatGPT, which is then encoded with LLaMA-2 and combined with a trainable dataset embedding to create node input features. The addition of the trainable dataset embedding enables the model to learn the different meanings of classes with the same name in different datasets.\n\nFinally, the training process alternates between fine-tuning the segmentation network and refining the mappings with the GNN.", "questions": "1. What is the size of the created taxonomy in different tables (e.g. in Table 5)? Furthermore, Table 5 could also include the performance after initializing the adjacency matrix, without additional GNN training.\n2. How does the quality of the created taxonomy compare to the manual taxonomy provided by [5]? Are all relationships recovered? For example, IDD:tunnel is not connected to MPL:tunnel in Figure 5. \n3. How are unseen datasets mapped to the unified taxonomy in Table 3? Is it done manually or automatically?\n4. Is there any advantage to using the formulation in Eq. 6 as opposed to NLL+ [4]?\n5. How much does the trained taxonomy differ from the initialized taxonomy? Do all unified classes \"survive\" after training, or do some become obsolete?\n6. What exactly is the zero-shot model on WildDash 2 in Table 4?\n7. How long does it take to initialize the adjacency matrix? What exactly is the difference compared to [52]? Are there similarities to [5], such as pairwise merging?", "limitations": "1. Focus on individual benchmark performance, rather than the process and the quality of the produced taxonomy.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Well-written and clear.\n2. Achieved state-of-the-art results on the WildDash 2 benchmark, demonstrating improvements over previous work.\n3. Introduces a novel approach for label unification.", "weaknesses": "1. The paper primarily focuses on individual benchmark performance rather than the quality of the recovered taxonomy. In practical scenarios, it's crucial to reason within a unified label space. Mistakes such as equating MPL:tunnel to ADE:fireplace could mean the difference between advancing or halting in a real-world scenario.\n2. The model achieving state-of-the-art performance on WildDash is trained exclusively on driving datasets, unlike other methods submitted that were trained on the full RVC collection (including ADE20K and COCO), potentially impacting performance in road driving contexts.\n3. There is significant variation in training design choices in Table 2, making it challenging to isolate the impact of each individual choice (e.g., backbone, selection of dataset-specific classes, etc.).\n4. The initialization of the adjacency matrix appears to be a crucial step for model performance, yet it is not discussed in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful comments regarding our manuscript. Below are our responses to your questions and concerns.\n\n**W1: Focus on benchmark performance vs. taxonomy quality** \n\nWe acknowledge the importance of reasoning within a unified label space. Our focus on multi-dataset training aims to automatically integrate labels across datasets. Evaluation on these datasets is a straightforward approach. To assess the quality of our unified label space, we conducted an indirect evaluation using WD2, demonstrating its effectiveness in unseen scenarios.\n\n**W2: Impact of training exclusively on driving datasets** \n\nWe recognize the potential concern regarding dataset bias. It's worth noting that other methods on the WD2 leaderboard also use datasets beyond RVC. In the revised version, we will include a model trained on the full RVC collection for a more comprehensive comparison.\n\n**W3: Difficulty isolating individual training design choices** \n\nWe apologize for the lack of clarity in Table 2. Due to long training times and limited access to open-source methods, comprehensive comparative experiments are challenging. Our primary comparison highlights the significant improvement our model offers over the single dataset and multi-SegHead baselines.\n\n**W4: Lack of discussion on adjacency matrix initialization** \n\nWe regret this omission and have conducted ongoing experiments to address it. Results indicate that our proposed initialization method (Algorithm 2) provides a slight improvement over randomized initialization on a 3ds setting, suggesting a relatively minor impact on overall performance.\n\n| Method|\\|L\\||CS|SUN|CV|Mean|\n|--|--|--|--|--|--|\n|Randomized adjacency matrix|54| 78.0|43.1|82.4|67.8\n|Without GNN training|Pending|Pending| Pending|Pending|Pending\n|Ours| 54|78.4|43.3|82.6|68.1\n\n---\n\n**Q1: Size of the created taxonomy in different tables and performance after initializing the adjacency matrix.**\n\nThank you for your suggestion. We agree that including the initialization of the adjacency matrix in Table 5 would enhance reader understanding. However, due to time constraints, we cannot currently provide results for all seven datasets. These findings will be included in the revised version of the paper. Comparative experiments in a smaller setting were conducted, as addressed in our response to Weakness 4. Below is a summary of label space sizes for the methods, which we will incorporate into the revised version.\n\n| Variants | \\|L\\| |\n|--|--|\n|1 | 448 |\n|2 | 329 |\n|3 | 226 |\n|Ours | 217 |\n\n\n**Q2: Comparison of created taxonomy with the manual taxonomy.**\n\nTo compare our constructed taxonomy with the manual taxonomy provided by [5], we adopted [5] as the ground truth standard, assessing whether categories from all datasets were appropriately linked. Categories were categorized into Merged, Single, and Split classes, each evaluated based on Correct, Partially Correct, and Wrong Classifications. We introduced *Wrong but reasonable* to accommodate justified connections that were nonetheless classified as wrong. For instance, while the Cityscapes dataset marked the Traffic sign (back) as ignored, [5] connected it based on expert knowledge.\n\nThe summary of our 7ds learned taxonomy is as follows:\n\n| |Correct|Partial Correct|Wrong|Total Num|Wrong but reasonable |\n|--|--|--|--|--|--|\n|Merged|137|108|47|292|17 |\n|Single|38|12|16|66|6 |\n|Split|1|3|86|90|23 |\n|Total Num|176|123|149|448|46 |\n\nAn initial taxonomy calculated using Algorithm 2 yields:\n\n| |Correct|Partial Correct|Wrong|Total Num|Wrong but reasonable |\n|--|--|--|--|--|--|\n|Merged|131|115|46|292|6 |\n|Single|44|7|15|66|3 |\n|Split|0|0|90|90|20 |\n|Total Num|175|122|151|448|29 |\n\nNotably, our constructed unified label space contains only 217 categories, representing a 6% reduction from the 231 categories formed by Algorithm 2’s initial adjacency matrix, while achieving nearly consistent quality in taxonomy recovery.\n\n**Q3: Mapping of unseen datasets to the unified taxonomy.**\n\nAs indicated in Lines 212-214 of our paper, the mapping process is conducted automatically. We identify the optimal mapping by comparing the unified label categories predicted by our model on the training set of the unseen datasets against the ground truth labels.\n\n**Q4: Advantages of using the formulation in Eq. 6 compared to NLL+ [4].**\n\nWe regret that because [4] is not open-sourced, our reproduced version of NLL+ does not converge under our training framework, prompting us to utilize Eq. 6 for training instead.\n\n\n**Q5: Differences between trained taxonomy and initialized taxonomy.**\n\nFor the differences in the label space, please refer to Q2 and the global rebuttal. The initial unified label space had 231 categories, but some became obsolete during training, resulting in 217 categories in the final 7ds-trained model. Our paper (L487-480) explains the methods to eliminate inactive connections. Specifically, during the final training phase, we evaluate the model in the training dataset to remove inactive connections and unlinked nodes.\n\n**Q6: Explanation of the zero-shot model on WildDash 2.**\n\nThe zero-shot model on WildDash 2, referenced in Table 4, is the model trained using our 7 datasets (i.e., the \"ours\" model in Table 2). We evaluated this model against the WildDash 2 dataset without any additional fine-tuning.\n\n\n**Q7: Time required to initialize the adjacency matrix and comparison with [52].**\n\nUsing four 80G A100 GPUs, initializing the adjacency matrix takes approximately two days to train the Multi-SegHead model, followed by half a day of cross-evaluation across multiple datasets. Obtaining the initial adjacency matrix using Algorithm 2 takes around one hour. In comparison to [52], we modified the cost calculation to utilize the IoU metric. This approach is only applicable to category merging and shares similarities with the partial merge method in [5].\n\nWe hope this clarifies your questions and appreciate your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their valuable comments and suggestions. The automatic construction of a unified label space using GNNs is the main innovation and contribution of our method. However, due to the limitations of our presentation format, we apologize for not being able to provide a comprehensive demonstration. We plan to open-source the constructed unified label space along with our code. In the supplementary material, we also present more examples of the generated graphs.

Our supplementary material is divided into four parts:

1. **Fig. 1**: Shows well-constructed examples across the 7 datasets.
2. **Fig. 2**: Illustrates examples of the label space initialized using Algorithm 2 and subsequently refined through GNN training.
3. **Fig. 3**: Displays examples of some incorrectly constructed links.
4. **Fig. 4**: Shows the label space constructed on 5 datasets.

Since our method constructs the unified label space through sample-based learning, the constructed label space depends on the samples included in the datasets. For instance, in **Fig. 2**, Traffic sign (back) is labeled as an ignored category in CS, BDD, and IDD, making it difficult to learn a link with MPL: Traffic sign (back). Additionally, the model's learning relies on visual features. For example, COCO: Window-other often includes vegetation outside the window, leading to an incorrect link in **Fig. 3**. Similarly, the tunnel == fireplace issue mentioned in the paper arises because these categories have similar visual features. The model merges them into one category to save space for predictions. However, these categories have significant semantic differences. We believe that using label descriptions provided by official datasets to generate more accurate text features, combined with incorporating knowledge from large models, can help constrain the model’s label links and reduce incorrect semantic links. This is an area worth further research.

We thank all the reviewers for your insightful questions and feedback. If there are any more issues or concerns to discuss, we welcome further inquiries and are happy to provide additional clarifications.

Meta-review: This manuscript addresses the learning of a unified class space given several datasets and their particular labeling conventions. Such approaches need to recover the incidence matrices M_d that link universal classes with the corresponding labels of the dataset d. Different than in the previous work, the proposed method recovers soft incidence by a GNN that operates upon the bipartite graph that associates textual label embeddings with the universal visual vectors. The GNN aligns the universal predictions of a frozen feature extractor with the labels and encourages orthogonality of the universal visual vectors.

The manuscript has received in-depth reviews and widely divergent ratings. The diversity of viewpoints reflects the complexity of this multifaceted problem. The reviewers commend the joint recovery of universal class embeddings and dataset-to-universal incidence, as well as state of the art performance on the WildDash dataset. The criticism can be subdivided into low-level and high-level issues. The low-level issues involve limited impact of textual label embeddings due to relying on chatgpt descriptions, theoretically unfounded loss (log-softmax-sum vs log-sum-softmax), missing comparison with previous work [5], and outdated baselines. Most of these issues could be addressed in a straight-forward manner. 

High-level criticism addresses vagueness of the chosen performance metric (mIoU). Ultimately, we can not say whether the performance is great due to better universal mappings or more aligned training setup and more GPU RAM. It appears that an additional performance metric (eg AP [40]) would be helpful in this regard as well as validation of the GNN capacity and more entries in Table 5 (eg independently recovering the incidence and the universal vectors, without using GNN).

Ultimately, I agree with M8bi. This paper can be both rejected due to insufficient technical correctness, and accepted due to significance of the research problem.
Decision: Accept (poster)