Title: Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs
Venue: NeurIPS 2024
Accepted: yes
Keywords: Semantic Segmentation

Multi-dataset Training

Graph Neural Networks
Abstract: Deep supervised models possess significant capability to assimilate extensive training data, thereby presenting an opportunity to enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. In this paper, we propose a novel approach to automatically construct a unified label space across multiple datasets using graph neural networks. This enables semantic segmentation models to be trained simultaneously on multiple datasets, resulting in performance improvements. Unlike existing methods, our approach facilitates seamless training without the need for additional manual reannotation or taxonomy reconciliation. This significantly enhances the efficiency and effectiveness of multi-dataset segmentation model training. The results demonstrate that our method significantly outperforms other multi-dataset training methods when trained on seven datasets simultaneously, and achieves state-of-the-art performance on the WildDash 2 benchmark. Our code can be found in https://github.com/Mrhonor/AutoUniSeg.

Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs

Abstract

Deep supervised models possess significant capability to assimilate extensive training data, thereby presenting an opportunity to enhance model performance through training on multiple datasets. However, conflicts arising from different label spaces among datasets may adversely affect model performance. In this paper, we propose a novel approach to automatically construct a unified label space across multiple datasets using graph neural networks. This enables semantic segmentation models to be trained simultaneously on multiple datasets, resulting in performance improvements. Unlike existing methods, our approach facilitates seamless training without the need for additional manual reannotation or taxonomy reconciliation. This significantly enhances the efficiency and effectiveness of multi-dataset segmentation model training. The results demonstrate that our method significantly outperforms other multi-dataset training methods when trained on seven datasets simultaneously, and achieves state-of-the-art performance on the WildDash 2 benchmark. Our code can be found in https://github.com/Mrhonor/AutoUniSeg.

1 Introduction

Recent advances in computer vision [35, 21] have shown the advantages of large datasets in training robust visual models [2]. However, for deep supervised visual models that rely on annotated data, the collection of such extensive annotated datasets can be prohibitively costly [11]. To address this expense and expand the data available for training, several efforts [4, 23, 30] focus on the challenges of multi-dataset training, enabling the use of diverse datasets to train more robust and generalizable models.

Models trained on multiple datasets must confront the challenge of reconciling conflicting annotation standards and label spaces. For example, the class road in the BDD dataset [46] can be further divided into several classes in the Mapillary dataset [33]: road , lane marking and crosswalk . Similarly, the Mapillary dataset labels both barrier and curb as distinct classes, while in the IDD dataset [42], they are combined under the single label curb . These conflicts impact the supervised learning of models, as they may be incorrectly penalized for predicting finer-grained classes from other datasets.

Another challenge is the task of unifying diverse dataset labels to produce outputs in a standardized format [29, 24]. Several methods [11, 30] attempt to address this by concatenating the label spaces of all datasets and using language models to encode label names into a text embedding space. However, there approaches introduce redundancy and fail to handle issues where labels share names but differ in annotation granularity. Other methods involve manually constructing universal taxonomies [4, 24] or relabeling [23], both of which are time-consuming and labor-intensive. Recent approaches aim to

Language

Model

Text

Features

Graph

Neural

Networks

Prediction in the

Prediction in the

…

on rails

road

train

road

lane marking

curb

barrier

train

road

lane marking

on rails

…

Dataset 1

Unified Label

Embedding

Space

Automated Label Unification

Dataset

Label

Mappings

Segmentation Network

Segm

e

ntation

in

Unfied

Label

Space

Dataset

Label

s

…

Ground Truth

DatasetN

ce

automatically construct universal taxonomies [6, 41], but these methods typically identify inter-label relations between only two datasets, involving a time-consuming iterative training process.

In this paper, we propose a novel approach leveraging Graph Neural Networks (GNNs) [20] to automatically construct unified label space, enabling segmentation model to be trained simultaneously on multiple datasets. In contrast to previous approaches [11, 4, 6], our approach eliminates the need for manual re-annotation or iterative training procedures to construct universal taxonomies, while also addressing the limitation of language-based methods in distinguishing categories with identical semantics. As depicted in Figure 1, we utilize a language model to convert the dataset labels into text features. Then, we apply GNNs to learn the relationships and associations among these labels. The process creates a unified label embedding space and dataset label mappings. The output head of the segmentation network incorporates the unified label embedding space to generate unified segmentation results within this unified space. The dataset label mappings are subsequently utilized to align the unified segmentation results with the label spaces relevant to each dataset. This enables the training of segmentation models and graph neural networks with dataset-specific annotations.

2 Related work

Multi-datasets Semantic Segmentation. In recent years, numerous studies [52, 29] focused on training semantic segmentation models on multiple datasets. A simple approach involved incorporating dataset-specific modules in the model, such as dataset-specific output heads [28] or dataset-specific batch normalization layers [44], to produce predictions tailored to each dataset domain. While these methods effectively avoided issues of dataset label space conflicts, they offered limited applicability in real-world scenarios as they could not deliver unified predictions. MSeg [23] addressed the problem at the data level via manual re-annotation processes to resolve label conflicts. However, this approach was time-consuming, error-prone and not easily scalable. Recent methods employed manual [5] or automatic techniques [6] to construct universal taxonomies and establish label relationships between datasets label space and the unified label space. These methods, leveraging partial label learning approaches [48, 12], enabled training with dataset-specific annotations and producing unified predictions.

Construct Universal Taxonomies. Each dataset has its unique domain, necessitating the construction of universal taxonomies to enable the model to cover all domains. Several approaches [11, 30] attempted to concatenate dataset label spaces and differentiated similar classes by aligning text embedding encoded by language model. However, this approach struggled with classes that had the same name but different levels of annotation granularity, and direct concatenation of label spaces could lead to semantic conflicts. Other research [24, 29] attempted to establish a unified label space

Text

Features

Dataset

Embedding

Label Relation Learning

Encoder-

Decoder

UniSegHead

Embedding

Unified Label

Dataset-specific

…

Dataset N

Dataset 1

Training Datasets

Dataset-specific

Prediction

Sampled Image

from Dataset

𝑖

Dataset

Annotation

Dataset-specific

Prediction

Unified

Prediction

Projected to Unified

Mapped to Dataset-

Randomized Unified

Node Features

Initialized Bipartite

Graph Matrix

UniSegHead

❄ 🔥

𝐻

×

𝑊

×

𝐿𝑖

Dataset Label

Description by GPT

Text

Encoder

Text

Encoder

·

Road

:  An

image of

road

from

dataset 1

.

Road is a paved

Surface...

·

Wall

:  An

image of

wall

from

dataset N

.

Wall is a solid

structure...

·

Road

:  An

image of

road

from

dataset 1

.

Road is a paved

Surface...

·

Wall

:  An

image of

wall

from

dataset N

.

Wall is a solid

structure...

…

𝐶

×

𝐿

…

…

wall

table

Dataset N

...

road

curb

Dataset 1

...

…

wall

table

Dataset N

...

road

curb

Dataset 1

...

…

𝐻

×

𝑊

×

𝐶

𝐻

×

𝑊

×

𝐶

Dataset     Pixel

Embedding

ℒ𝑐𝑒

𝐻

×

𝑊

×

𝐿

𝑖

𝑖

GNNs

𝐻

×

𝑊

×

𝐿𝑖

𝐿

×

𝐿

1

𝐿

×

𝐿

1

𝐿

×

𝐿𝑁

❄

🔥

Trainable Parameters

𝐻

×

𝑊

×

𝐿𝑖

through the expertise of human annotators. Additionally, other studies [41, 53] aimed to address this issue by developing automated mechanisms to create universal taxonomies. However, these methods either could only construct a unified label space between two datasets at a time [6] or could not handle complex class relationships [53]. Our approach stands out by automatically constructing universal taxonomies in a single training session with multiple datasets, resulting in significant time savings compared to methods that require multiple training iterations.

Graph Neural Networks demonstrated exceptional effectiveness in dealing with complex topological data structures, as highlighted in recent research [43]. The applicability extended across various domains, including recommendation systems [18], knowledge graph construction [50], and skeletal action recognition [54]. In the context of our problem, discovering label relations can be conceptualized as a link prediction task [49, 22]. However, conventional approaches for graph link prediction [9, 10] are not suitable for our model since we do not possess ground truth links. We use the values of the learnable adjacency matrix as predictions of whether nodes are linked. Supervision of the linked predictions relys on the segmentation results of the segmentation network and the corresponding image annotations.

3 Proposed Method

The comprehensive framework is depicted in Figure 2. We first define the unified representation of the multi-dataset label space. Utilizing this representation, we build a graph neural network to learn the unified label space. Finally, leveraging the unified label space, we train our segmentation network and the graph neural network using the dataset annotations.

3.1 Unification of Multi-dataset Label Space

Unified Label Space. Given K datasets with their respective label space { L 1 , L 2 , . . . , L K } , multidataset semantic segmentation requires a model to predict within a consistent label space that

encompasses all dataset label spaces L pred = ⋃ K i =1 L i . Each pixel must be assigned to a label in this unified label space. We define N unified label nodes A = { α 1 , α 2 , . . . , α N } , serving as the nodes in the graph neural networks. Their corresponding D -dimensional learnable embedding { x 1 , x 2 , . . . , x N } represent the unified label embedding space. The number of unified label nodes N is often smaller than the total number of dataset classes | L | , because we aim to merge multiple identical classes into a single unified label. The image is first encoded into pixel embedding P by the segmentation network, which is then projected into the unified label embedding space to assign a unified label to each pixel.

Label Mappings. We define a mapping from the unified label space to the dataset-specific label space M i : A → L i , which is used to train the model with dataset-specific annotations. Mathematically, M i ∈ { 0 , 1 } N ×| L i | is a boolean linear transformation. Each unified class α is at most linked to a class c within a specific dataset i to prevent label conflicts: M α 1 ≤ 1 . To handle different annotation granularities, we use label mappings to merge multiple unified label nodes representing fine-grained classes into one super-class c : M ⊤ c 1 ≥ 1 . For example, the curb from IDD can be simultaneously mapped by unified label nodes represented curb and barrier . We use unified label nodes as input nodes for the GNNs, which learn the label mappings and unified label embedding space, thus enabling the automated unification of multi-dataset label spaces.

3.2 Learning Unified Label Space with Graph Neural Networks

Learning the label mappings between dataset label spaces and the unified label space can be viewed as a bipartite graph matching problem. This makes graph neural networks well-suited to address this issue. Below, we detail our approach of constructing GNNs for learning a unified label space.

Input Nodes. To construct the input feature of dataset-specific label nodes, as illustrated in Figure 2, we used the dataset labels in the template "An image of <label> from the dataset <dataset>" as plain text input and employed ChatGPT to complete the detailed description of each label. Then, we employ llama-2 [40] to encode these label descriptions and generate text features. To further distinguish nodes from different datasets, we introduce learnable dataset embedding for each dataset { d 1 , d 2 , . . . , d K } . The dataset embedding is combined with the text features to form the input features for dataset-specific label nodes:

where x i,m is the input feature of the m -th label from dataset i , and f t ( l i,m ) is the text feature of the label description l i,m encoded by language model. The input features of unified label nodes are randomly initialized to the same dimension as the dataset-specific label node. To determine the appropriate number of unified label nodes, inspired by the approach in [53], we used crossvalidation results across different datasets to identify the number of mergeable categories, which served as the initial selection for the unified label nodes. The specific algorithm can be found in Appendix B. Together, dataset-specific label nodes and unified label nodes constitute the input nodes of GNNs. During the training process, we maintain a constant number of nodes. After the training is completed, we will remove inactive unified label nodes, meaning those that were not assigned to any dataset-specific label.

Learnable Adjacency Matrix. To enable label mappings to be updated via gradient descent, we embed the label mappings as a continuous, learnable graph adjacency matrix M a . Values in the adjacency matrix represent the weights of corresponding edges. Only the edges between unified label nodes and dataset-specific nodes are learnable, while others are fixed to zero. We apply a softmax operation to the edges connecting each unified label node with nodes of a particular dataset, ensuring that the sum of the edge weights w between each unified label node and nodes of this dataset equals one. The element at the intersection of the r -th row and the c -th column in the lower triangular portion of M a is formulated as:

Unified Label Embedding. The forward propagation of our graph model follows the GraphSAGE framwork [17] as formulated in Equation 3. W k and X k are the weight and feature of the k -th GNNs layer. The σ indicates nonlinear activation function, implemented as the tanh in this work. The output

features of the unified label nodes from the final layer serve as the unified label embedding space,

To obtain the dataset label mappings, we partition the adjacency matrix into submatrices, each corresponding to a specific dataset. Each submatrix contains only the unified label nodes and the label nodes specific to that dataset. We compute the label mappings for each dataset based on the values in its corresponding submatrix, divided into the following two cases. During training, we will alternately train the segmentation network and the GNNs. When training the GNNs, we directly use the value of the learnable adjacency matrix to establish label mappings, thereby facilitating weight updates through gradient descent. When training the segmentation network, we utilize the unbalanced optimal transport algorithm to solve for the boolean label mappings that satisfies the many-to-one mapping constraints. This algorithm detailed in Appendix C effectively handles the conversion of the continuous adjacency matrix into a discrete dataset label mappings required for training segmentation network.

3.3 Training a Universal Model with Dataset-specific Annotations

Training a universal model is divided into two steps. The first step involves training a robust encoderdecoder to provide the pixel embedding for each pixel position in the image, where similar objects should have similar features. The second step focuses on learning label mappings and unified label embedding space by GNNs. During the training phase, we alternate between these two steps, freezing one network while training another network. Both of these steps require supervised training using dataset-specific annotations. Here, we primarily focus on the training of GNNs, while further details of the training strategies are provided in the Appendix A.

Training with Dataset-specific Annotations. During training, an image is randomly sampled from dataset i and fed into an segmentation network, which provides embedding for each pixel position P = { p 1 , p 2 , . . . , p j } , where p k is D -dimensional vectors. We utilize a unified segmentation head (UniSegHead) to assign a dataset label to each pixel. In the UniSegHead, We first project the pixel embedding into the unified label embedding space by multiplying the pixel embedding by the output feature of unified label node X u , as shown in Equation 4. Then, to train the universal model with dataset-specific annotations, we need to map the predictions in unified label space to dataset-specific label space to obtain the probabilities of dataset-specific classes. This is achieved by computing per-pixel dataset-specific logits s by multiplying the dataset-specific label mappings M i at each pixel:

Finally, probabilities of dataset-specific classes are computed by per-pixel softmax operation over the logits s . This allows us to use dataset-specific annotations to compute the pixel-wise loss function, train the network, and update the label mappings. We formulate the cross-entropy loss for a specific pixel to train the segmentation network:

where y is the pre-pixel annotation, | L i | represents the total number of classes in the dataset i from which the image originates.

Orthogonality Loss. To achieve a conflict-free unified label space and avoid redundant unified label nodes that represent the same class or have overly similar features, we introduce soft constraints to promote orthogonality among the unified label node features, inspired by [39]. This orthogonality loss encourages unified label node embedding to be mutually orthogonal. It not only aligns the unified

1 Approach to construct label space. MC:Manually Construct, MR:Manually Relabel, DS:Dataset-specific, Auto:Automatically Construct.

2 MSeg train and evaluate on 43 of 65 class in Mapillary dataset, 117 of 150 class in ADE dataset, 122 of 133 class in COCO dataset.

label nodes with annotation standards for practical use but also enhances the diversity of the model and helps in finding a better label mappings:

The final loss function used to train the GNNs is represented as follows, with λ s as hyperparameters to adjust the weights of different loss functions:

4 Experiments

Datasets. We list the semantic segmentation datasets used for training and testing in Table 1. Our training datasets cover a wide range of scenarios, from indoor scenes to driving scenes. We also introduce corresponding test datasets, which are not used in the training process, for the respective scenes to evaluate our generalization capability.

Implementaion Details. Our segmentation model is based on the HRNet-W48 architecture [38], while the GNN model is a three-layer GraphSAGE [17]. We utilize the llama-2-7B model to encode label descriptions into 4096-dimensional text features. These text features, augmented with dataset embedding of the same dimensionality, are then employed as node features input into the GraphSAGE. When forming a minibatch from multiple datasets, we evenly sample 3 images per dataset within a batch for each GPU. For all images, We first apply random resizing with a ratio ranging from 0.5 to 2, followed by a random crop operation to achieve a final image size of 768 × 768 pixels. We use AdamW optimizer [27] with warmup and polynomial learning rate decay, starting with a learning rate of 0.0001. We train our model for 300k iterations on four 80G A100 GPUs.

4.1 Comparison on Multiple Datasets

In Table 2, we present the accuracy of our methods and compare them to other approaches on the seven training datasets. We use mean Intersection over Union (mIoU) to quantify the performance of models, a common metric used to evaluate the performance of segmentation models. Different methods adopt various approaches to construct their label spaces: Dataset-Specific represents a lack of a unified label space, where the model outputs a separate label space for each dataset. Manually

Image

Ground Truth

BDD Model

ADE Model

Our 7ds Model

BDD

MPL

ADE

Relabel means manually re-annotating each image. Manually Construct means the label space is constructed through human expertise. Automatically Construct includes methods where the unified label space is automatically constructed by the model. We also establish two baseline methods: Single dataset and Multi-SegHead . Single dataset demonstrates the results of training on individual dataset only, while Multi-SegHead trains on multiple datasets by using dataset-specific segmentation heads.

The results demonstrate that our method achieves the best average performance in multi-dataset training, while also achieving significant performance improvements on datasets with a large number of classes such as the ADE and COCO datasets. We attribute this to the construction of a robust unified label space. Leveraging visual connections from the samples, our unified label space can discover label relationships beyond textual similarities. For instance, the visual appearance of the fireplace in ADE is similar to the tunnel in Mapillary. Despite their different semantic meanings, our model merges these labels for prediction, as detailed in subsection 4.5. This approach saves model capacity and facilitates knowledge transfer across datasets for improved prediction.

In Table 3, as a supplement to Table 2, we compare our model with various models trained on single dataset, as well as each segmentation head output of the Multi-SegHead . Detailed data for each dataset can be found in the Appendix D. From the table, it can be observed that training with multiple datasets helps improve the model's generalization performance. However, the performance of different segmentation head outputs in Multi-SegHead model shows significant differences due to the lack of a unified form of output that performs well across all datasets. In contrast, our approach provides a unified label space covering all datasets, resulting in a significant advantage in average performance. We also list the performance on the five unseen test datasets mentioned in Table 1. To evaluate the performance of our model on unseen datasets, we first evaluate the model results on its training dataset. We search for the optimal label mappings based on the accuracy of label predictions. There are no updates to any model parameters except the label mappings in this process. This process can actually be done manually without any annotation information. The results indicate that our model exhibits better generalization performance. It can handle various scenarios and consistently achieve excellent performance on the test set. Compared to Multi-SegHead model , our automatically constructed label space has advantages over dataset label spaces. The unified label space can integrate the semantic information from multiple dataset label spaces.

Figure 3 presents the segmentation results on multiple datasets predicted in unified label space. Compared to models trained only on Single dataset , our model successfully provides consistent predictions on all datasets. It's worth noting that in the BDD dataset, annotations are not provided for lane marking , crosswalk and manhole , which are only annotated in the Mapillary dataset. Our model successfully integrates the label space of the Mapillary dataset, thereby predicting these classes in the BDD dataset. More results are presented in Appendix F.

4.2 Results on WildDash 2 Benchmark.

WildDash 2 [47] provides a benchmark for semantic segmentation, designed to test the robustness of algorithms in real-world driving scenarios. Due to the insufficient number of training samples

provided by this dataset, the official recommendation is to use multiple datasets for training. Therefore, this benchmark is well-suited to evaluate the effectiveness of multi-dataset training methods. The WildDash 2 dataset includes negative test cases to challenge the robustness of the model. These negative test cases mainly consist of unconventional driving scenarios, and even non-driving scenarios. Across all pixels within negative test images, a robust model is expected to predict the void label for open-set classes and anomalous objects. The WildDash 2 benchmark refers to the metric named Meta Avg mIoU Class, which calculates the mean Intersection over Union for each class by weighting negative and positive test cases according to their occurrence in the benchmark dataset.

Table 4 presents the current results on the WildDash 2 leaderboard. We present results for zero-shot generalization using our 7ds model. To evaluate in an unseen setting, we map the non-evaluated classes in the unified label space to a void label for both positive and negative test frames. To ensure a fair comparison with other works, we also provide evaluation results for models trained using the training datasets from the Robust Vision Challenge 2022[1], which include CityScapes, ADE20K, Vistas, VIPER [36], ScanNet, and WildDash 2. The results indicate that our method achieves state-of-the-art performance in both zero-shot and trained settings. Our method exhibits significant performance improvements compared to other methods on negative test cases. We attribute this to the robustness of our model, which has been trained on diverse datasets, enabling it to perform well in unconventional scenarios. Our zero-shot model has been trained on a wider range of datasets compared to the trained model. Therefore, even without training on the WildDash 2 dataset, it achieves similar performance on negative test cases.

4.3 Ablation Study

To further explore the ability of our GNNs to construct a unified label space, we compared it with four alternative methods. The first method concatenates all dataset label spaces into a single unified space ⋃ D i =1 L i . The second approach constructs a unified label space by clustering text features based on cosine similarity using the DBSCAN [7] algorithm. In the third method, we train a segmentation network using an initial adjacency matrix, as outlined in Appendix B, without incorporating GNN training. The final method involves an ablation experiment, removing the label description module to observe its impact. Experimental results, presented in Table 5, demonstrate that the label space constructed based on GNNs can better assist in the learning of segmentation models. Unlike the first approach, our method optimizes model capacity by focusing on label relationships rather than dataset recognition. Compared to the second approach, our method can differentiate between classes with identical names but differing levels of granularity. By leveraging label descriptions to enrich semantic context, our approach constructs a more refined and functional label space.

Domain-general

model

Domain-specific

model

ADE

COCO

SUN

CS

MPL

BDD

IDD

WD2

KT

VOC

SN

Datasets

4.4 Exploring the Impact of Training Datasets

To explore the impact of training dataset selection on model performance, we train a domain-specific model focusing on road driving scenes and a domain-general model on more datasets, as shown in Figure 4. We conduct four sets of comparisons to evaluate the performance across datasets that were trained on both models, trained on one model and not on the other, and not trained on either model. As shown in Table 6 and Appendix E, the domain-specific model can focus more on learning features specific to the particular scene, resulting in slightly better performance on both trained and unseen driving scene datasets compared to domain-general model. On the other hand, domain-general model trained on more scenes and more data exhibit better generalization performance. Therefore, while it does not lag too far behind in performance on driving scene datasets, it demonstrates overwhelming advantages on other scene datasets.

4.5 Qualitative Results

Figure 5 presents the qualitative analysis results of the label space learned by our model. We compare the label space learned by our model with the label space constructed using text features. The class curb in the IDD dataset actually encompasses both the classes curb and barrier in Mapillary, whereas the constructed label space by text features cannot handle such subclass/superclass relations. In contrast, our GNNs learned label space splits the IDD curb into two classes for prediction, effectively handling such label relationships. Similar situations also include the class tunnel or bridge in the IDD dataset. However, since the proportion of bridge pixels is orders of magnitude greater than that of tunnels ( 10 8 pixels for bridge and 10 4 pixels for tunnel ), a more reasonable approach is to merge it with the class bridge , as our GNNs have done. Additionally, it is worth noting that due to visual similarities, the Mapillary tunnel has been merged with the ADE fireplace . This does not actually introduce any conflict because no dataset simultaneously annotates both the tunnel and fireplace . The model will construct the label space in a way that facilitates its learning process.

5 Conclusion

We propose a novel approach that leverages graph neural networks to construct a unified label space for training semantic segmentation models across multiple datasets. Our method addresses the challenge of label conflicts in multi-dataset semantic segmentation and demonstrates performance improvements across various datasets. The unified label space generated during training, generalizes well to unseen datasets, showcasing the effectiveness of our approach.

Broader Impact. Our work explores the use of graph neural networks to unify label spaces between datasets, providing a new direction for achieving robust and efficient multi-dataset training. By enabling semantic segmentation models to be trained on multiple datasets with a unified label space, our method can potentially reduce human effort required for re-labeling images and facilitate the expansion of training datasets. This can lead to the development of models that are more universally applicable across various datasets, benefiting a wide range of applications.

Limitations. Although our approach does not require manual relabeling efforts, it still relies on fully annotated datasets for training, in contrast to weakly-supervised and unsupervised methods. We aim to explore ways to integrate these alternative methods in future research. Errors in the fully automated construction of a unified label space do present some safety risks for autonomous driving tasks. Therefore, we recommend introducing a manual review mechanism to address these issues. Additionally, using ChatGPT may generate inaccurate label descriptions, which could affect the

IDD Dataset:

curb

Ground Truth

Our 7ds Model

Image

IDD Dataset:

curb

barrier

curb

MPL:

barrier

IDD:

curb

MPL:

curb

4

Learned by GNNs

7

MPL:

barrier

IDD:

curb

MPL:

curb

4

Learned by GNNs

7

tunnel

IDD Dataset:

tunnel or bridge

MPL:

barrier

IDD:

curb

MPL:

curb

4

Constructed by text features

7

MPL:

barrier

IDD:

curb

MPL:

curb

4

Constructed by text features

7

Ground Truth

Our 7ds Model

Image

bridge

ADE Dataset:

fireplace

ADE:

fireplace

IDD:tunnel or

bridge

MPL:

tunnel

6

Constructed by text features

137

76

MPL:

bridge

ADE:

bridge

ADE:

fireplace

IDD:tunnel or

bridge

MPL:

tunnel

6

Constructed by text features

137

76

MPL:

bridge

ADE:

bridge

ADE:

fireplace

IDD:tunnel or

bridge

MPL:

tunnel

137

76

MPL:

bridge

ADE:

bridge

ADE:

fireplace

IDD:tunnel or

bridge

MPL:

tunnel

137

76

MPL:

bridge

ADE:

bridge

Learned by GNNs

ADE:

fireplace

IDD:tunnel or

bridge

MPL:

tunnel

137

76

MPL:

bridge

ADE:

bridge

Learned by GNNs

Bridge

Guard

curb

Barrier

Curb

1

EOT

66

100

tunnel or bridge

Vall

Tunnel

TrafficSign（Fro

traficsign

lleror caravan orvehicle fallback

137

.10

83

7

10676

51

129

prediction of label relationships. Therefore, we aim to improve the accuracy of label descriptions by incorporating label descriptions provided by official datasets as prompts.

6 Acknowledgments

This work was supported in part by NSFC Project (62176061), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0103). The computations in this research were performed using the CFFF platform of Fudan University.

References

Robust vision challenge. http://www.robustvision.net/index.php . Accessed: 202212-02.

Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721 , 2023.

Petra Bevandi´ c, Marin Orši´ c, Ivan Grubiši´ c, Josip Šari´ c, and Siniša Šegvi´ c. Multi-domain semantic segmentation with pyramidal fusion. arXiv preprint arXiv:2009.01636 , 2020.

Petra Bevandi´ c, Marin Orši´ c, Ivan Grubiši´ c, Josip Šari´ c, and Siniša Šegvi´ c. Multi-domain semantic segmentation with overlapping labels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2615-2624, 2022.

Petra Bevandi´ c, Marin Orši´ c, Josip Šari´ c, Ivan Grubiši´ c, and Siniša Šegvi´ c. Weakly supervised training of universal visual concepts for multi-domain semantic segmentation. International Journal of Computer Vision , pages 1-23, 2024.

Petra Bevandi´ c and Siniša Šegvi´ c. Automatic universal taxonomies for multi-domain semantic segmentation. British Machine Vision Conference , 2022.

FM Bi, WK Wang, and L Chen. Dbscan: density-based spatial clustering of applications with noise. J. Nanjing Univ , 48(4):491-498, 2012.

Gabriel J Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes in video: A high-definition ground truth database. Pattern Recognition Letters , 30(2):88-97, 2009.

Lei Cai and Shuiwang Ji. A multi-scale approach for graph link prediction. In Proceedings of the AAAI conference on artificial intelligence , pages 3308-3315, 2020.

Lei Cai, Jundong Li, Jie Wang, and Shuiwang Ji. Line graph neural networks for link prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(9):5103-5113, 2021.

Yanbei Chen, Manchen Wang, Abhay Mittal, Zhenlin Xu, Paolo Favaro, Joseph Tighe, and Davide Modolo. Scaledet: A scalable multi-dataset object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7288-7297, 2023.

Jesús Cid-Sueiro. Proper losses for learning from partial labels. Advances in neural information processing systems , 25, 2012.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-338, 2010.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Junheng Hao, Tong Zhao, Jin Li, Xin Luna Dong, Christos Faloutsos, Yizhou Sun, and Wei Wang. P-companion: A principled framework for diversified complementary product recommendation. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 2517-2524, 2020.

Dongwan Kim, Yi-Hsuan Tsai, Yumin Suh, Masoud Faraki, Sparsh Garg, Manmohan Chandraker, and Bohyung Han. Learning semantic segmentation from multiple datasets with label shifts. In European Conference on Computer Vision , pages 20-36. Springer, 2022.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2016.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Ajay Kumar, Shashank Sheshar Singh, Kuldeep Singh, and Bhaskar Biswas. Link prediction techniques, applications, and performance: A survey. Physica A: Statistical Mechanics and its Applications , 553:124289, 2020.

John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi-domain semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2879-2888, 2020.

Feng Lin, Wenze Hu, Yaowei Wang, Yonghong Tian, Guangming Lu, Fanglin Chen, Yong Xu, and Xiaoyu Wang. Universal object detection with large vision model. International Journal of Computer Vision , 132(4):1258-1276, 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Yajie Liu, Pu Ge, Qingjie Liu, Shichao Fan, and Yunhong Wang. An empirical study on multi-domain robust semantic segmentation. arXiv preprint arXiv:2212.04221 , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Shota Masaki, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Multidomain semantic-segmentation using multi-head model. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) , pages 2802-2807. IEEE, 2021.

Panagiotis Meletis and Gijs Dubbelman. Training semantic segmentation on heterogeneous datasets. arXiv preprint arXiv:2301.07634 , 2023.

Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11402-11411, 2023.

Rohit Mohan and Abhinav Valada. Efficientps: Efficient panoptic segmentation. International Journal of Computer Vision , 129(5):1551-1579, 2021.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 891-898, 2014.

Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision , pages 4990-4999, 2017.

Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, and Peter Kontschieder. Seamless scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8277-8286, 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017 , pages 2232-2241, 2017.

Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 567-576, 2015.

Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang. High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.04514 , 2019.

Marco Toldo, Umberto Michieli, and Pietro Zanuttigh. Unsupervised domain adaptation in semantic segmentation via orthogonal and clustered embeddings. In Proceedings of the IEEE/CVF Winter conference on Applications of Computer Vision , pages 1358-1368, 2021.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. The missing link: Finding label relations across datasets. In European Conference on Computer Vision , pages 540-556. Springer, 2022.

Girish Varma, Anbumani Subramanian, Anoop Namboodiri, Manmohan Chandraker, and CV Jawahar. Idd: A dataset for exploring problems of autonomous navigation in unconstrained environments. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1743-1751. IEEE, 2019.

Petar Veliˇ ckovi´ c. Everything is connected: Graph neural networks. Current Opinion in Structural Biology , 79:102538, 2023.

Li Wang, Dong Li, Han Liu, Jinzhang Peng, Lu Tian, and Yi Shan. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2487-2494, 2022.

Junfei Xiao, Zhichao Xu, Shiyi Lan, Zhiding Yu, Alan Yuille, and Anima Anandkumar. 1st place solution of the robust vision challenge 2022 semantic segmentation track. arXiv preprint arXiv:2210.12852 , 2022.

Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, Trevor Darrell, et al. Bdd100k: A diverse driving video database with scalable annotation tooling. arXiv preprint arXiv:1805.04687 , 2(5):6, 2018.

Oliver Zendel, Matthias Schörghuber, Bernhard Rainer, Markus Murschitz, and Csaba Beleznai. Unifying panoptic segmentation for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21351-21360, 2022.

Min-Ling Zhang, Fei Yu, and Cai-Zhi Tang. Disambiguation-free partial label learning. IEEE Transactions on Knowledge and Data Engineering , 29(10):2155-2167, 2017.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems , 31, 2018.

Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A comprehensive survey on automatic knowledge graph construction. arXiv preprint arXiv:2302.05019 , 2023.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

Qiang Zhou, Yuang Liu, Chaohui Yu, Jingliang Li, Zhibin Wang, and Fan Wang. Lmseg: Language-guided multi-dataset segmentation. In The Eleventh International Conference on Learning Representations , 2022.

Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7571-7580, 2022.

Guangming Zhu, Liang Zhang, Hongsheng Li, Peiyi Shen, Syed Afaq Ali Shah, and Mohammed Bennamoun. Topology-learnable graph convolution for skeleton-based action recognition. Pattern Recognition Letters , 135:286-292, 2020.

A Training Strategy

The training process is divided into three distinct stages: Multi-SegHead Training Stage, Alternating GNNs Training Stage and Alternating SegNet Training Stage, as shown in Algorithm 1. Initially, we commence with the Multi-SegHead Training Stage for 100k iterations. After completing MultiSegHead Training Stage, we discard the multi-heads and initialize the UniSegHead for predicting class probabilities during subsequent training phases. Subsequently, we alternate between the Alternating GNNs Training Stage and Alternating SegNet Training Stage for a total of three cycles, each lasting 20k iterations. Notably, we prioritize the Alternating GNNs Training stage at the beginning of each cycle.

Multi-SegHead Training Stage aims to equip the segmentation network with fundamental segmentation capabilities. By commencing with a Multi-SegHead model, we provide a robust starting point for the subsequent stages, enhancing both performance and training efficiency. During this stage, the network is trained on multiple datasets using specific segmentation heads for each dataset. Each segmentation head comprises feature weights W i of dimension C ×| L i | , where | L i | represents the number of classes of dataset i . Pixel embedding P is multiplied by the i -th feature weight to make predictions within the i -th dataset's label space. This facilitates the utilization of dataset-specific annotations and cross-entropy loss for network training.

Alternating GNNs Training Stage. Training SegNet and GNNs simultaneously is challenging and impractical because training the segmentation network requires a discrete label mapping, while training the GNNs necessitates a continuous and differentiable adjacency matrix. Hence, we alternately training the GNNs and the segmentation network. The Alternating GNNs Training Stage focuses on optimizing a unified label space. During this stage, the GNNs are trained while keeping the segmentation network frozen. When mapping the prediction from the unified label space to the dataset label space, we directly multiply the adjacency matrix M a . We calculate the loss according to Equation 8.

Alternating SegNet Training Stage. During this stage, the GNNs remain frozen while the segmentation network is trained. We will use the technique detailed in Appendix C to convert the continuous adjacency matrix into a discrete label mappings that satisfies the constraints, and then use it to compute the cross-entropy loss function and train according to Equation 6. During the last Alternating SegNet training stage, we extend the training procedure to 100k iterations and drop the entire GNNs but retain the dataset label mappings { M i , i = 1 , ..., K } . The unified label embedding X u in this stage is no longer keep frozen and trained with the SegNet. In the latter part of this stage, we will use training dataset to evaluate the model's performance. Those links between the unified label space and dataset-specific label space which is not activated during the evaluation will be removed. After removing the unused connections, we continue training with the new label mappings. And thus, we obtain the final model. When applying the final model to predict in the wild, we could manually assign each unified label node a class name accord to their prediction and semantics.

B Selection of Unified Label Node Quantity and Initialization of Adjacency Matrix

The appropriate quantity of unified label nodes is crucial for balancing model performance and computational expenses. An excessive number of unified label nodes not only increases computational costs but may also result in redundancy in both the model and semantic space, leading to situations where multiple unified label nodes represent similar semantic concepts. Conversely, an insufficient number of unified label nodes can prevent the model from fully expressing the entire semantic space ⋃ D i =1 L i , making it difficult to learn certain classes in the datasets, thus impacting model performance. Inspired by [53], we determine the number of unified label nodes using the following steps. We first enumerate all feasible schemes for merging dataset labels, which form the Cartesian product of all dataset label sets L = L 1 × L 2 × . . . × L K . Label merging does not require the participation of all datasets simultaneously, so L i can be an empty element, indicating no participation in label merging.

Algorithm 1 The training pipeline of our model

Input: the number of datasets K , the Segmentation Network SegNet , the Graph Neural Networks GNNs , Multi-head iters I p , GNNs training iters I g and SegNet training iters I s

We evaluate the quality of label merging using the predefined loss function L c , where merging classes with the same semantic meaning results in smaller losses. Additionally, we require that the merging satisfies the constraint: each dataset node has only one corresponding merged node. It is worth noting that this constraint is stronger than our label mapping constraint described in subsection 3.1, thus ensuring that our adjacency matrix is always valid. We formulate optimization objective function as:

where T represents the set of all feasible edges connecting dataset label nodes to unified label nodes, x t ∈ { 0 , 1 } indicates whether edge t ∈ T is selected, with x t = 1 indicating the presence of edge t and 0 indicating its absence. S k denotes the dataset to which edge t belongs, L k represents the label space of S k , and t ( c ) is the label connected by edge t . λ | L | is the penalty term for the number of nodes, encouraging the optimizer to merge similar nodes to obtain a more compact unified label space. Based on experimental insights and referencing paper [52], we selected the hyperparameter λ = 0 . 5 . L c ( S k c , ˜ S k c ) is primarily used to evaluate the quality of label merging, where S k c represents the output results on class c using the original dataset segmentation head, and ˜ S k c represents the output results on class c using segmentation heads from other datasets participating in the merging.

where, IoU is a metric commonly used to measure the performance of semantic segmentation models. Therefore, we use this metric here to evaluate the assistance of merged nodes to the model. By using a linear optimization solver to solve the optimization objective Equation 10, we can obtain a unified label space. We use the number of labels in this label space as the quantity of our unified label nodes and initialize our adjacency matrix based on their label mapping relationships.

C Solving the dataset Label Mappings

Given the definition of the label mappings and the constraints outlined in subsection 3.1, solving the dataset label mappings could be conceptualized as a weighted bipartite graph matching problem. The weight of each edge is determined by the values of the GNNs learnable adjacency matrix. The objective is to maximize the sum of edge weights while ensuring that each node of the specific dataset is connected. The unbalanced optimal transport algorithm offers an approximate solution to such

Algorithm 2 The procedure of solving the dataset label mappings

Input: Submatrix of adjacency matrix for each dataset { S ( i ) , i = 1 , ..., K } , the classes of each dataset { c ( i ) , i = 1 , ..., K } , the number of unified label node N

problems, but it does not guarantee adherence to the constraint of connecting each node in the specific dataset. Therefore, based on the optimal matching result obtained from the algorithm, we employ a greedy strategy to adjust the matching scheme for all unconnected nodes within the specific dataset.

Initially, for every unconnected dataset-specific node, we sort all unified label nodes based on their matching preferences, from highest to lowest. Subsequently, we examine these unified label nodes and their associated dataset-specific nodes. In cases where the associated dataset-specific node is also connected by another unified label node, we adjust the edge so that this unified label node links to the unconnected specific dataset node, following this process until all nodes are connected. The specific procedure is outlined in Algorithm 2.

D Comparison with Baselines on Multiple Datasets

Table 7 lists the results of mutual evaluation among training datasets. The results indicate that individually-trained models generally demonstrate good accuracy when tested on the same dataset, but perform poorly on other datasets. Our method can leverage knowledge from multiple datasets to improve performance on individual datasets. Table 8 lists the results of the Multi-SegHead model's predictions on different datasets using different segmentation head outputs. It can be observed that none of the segmentation head outputs could consistently provide accurate predictions across all datasets, indicating that their label spaces do not effectively cover all datasets. Table 9 and Table 10 list the performance of different models on unseen datasets. The results indicate that our method has better generalization performance on unseen datasets compared to other methods. Additionally, the unified label space we construct contains richer semantic information, enabling flexible adaptation to datasets from different scenarios.

E Exploring the Impact of Training Datasets

Tables 11 to 14 present a performance comparison between the domain-general and domain-specific models across various training settings and datasets. Figure 6 illustrates the segmentation results of these models on the different datasets.

Performance Comparison on Both Trained Datasets. From Table 11, it can be concluded that the performance of the domain-specific model focusing on driving scenes is higher on all trained datasets compared to the domain-general model, despite the domain-general model being trained on five times more samples than the domain-specific model. This is likely attributed to the fact that the domain-specific model needs to predict fewer classes and is less affected by label space conflicts. The model can therefore focus on learning features specific to the particular scene.

Performance Comparison: Unseen Domain-general Model vs. Trained Domain-specific Model. In Table 12, we selected a driving scene dataset that the domain-specific model was trained on, while the domain-specific model was not trained on it, for comparison. As expected, the domain-specific model demonstrates superior performance, and the domain-general model closely follows suit.

Performance Comparison: Trained Domain-general Model vs. Unseen Domain-specific Model. In Table 13, we selected several non-driving scene datasets that the domain-general model was trained on, while the domain-specific model was not trained on it. The domain-specific model fails to predict in non-driving scenes. Compared to Table 12, the generalization performance of the domain-specific model in unseen scenes markedly trails behind the domain-general model. Although the domain-specific model may have encountered similar objects in driving scene datasets like wall , it still struggles to predict these seen objects well in non-driving scene datasets, as shown in Figure 6.

Performance Comparison on Both Unseen Datasets. In Table 14, we selected datasets that were unseen to both models, including driving scene datasets and non-driving scene datasets. We can observe that the generalization performance of the domain-specific model is slightly better in driving scenes, but its performance in other scenes lags far behind that of the domain-general model. Overall, models trained on more scenes and more data tend to achieve better generalization performance.

CS

IDD

WD2

SUN

COCO

Image

GT

General

Specific

BDD

MPL

ADE

KT

SN

VOC

Image

GT

General

Specific

F Visualization

Figure 7 presents the visual comparisons of models trained on single dataset and our model predicted in universal label space. From the figure, it is evident that our model achieves consistently strong performance across all training datasets while integrating label spaces from different datasets. For example, it predicts class lane marking and crosswalk for the ADE and BDD datasets, and predicts class books for the SUN dataset. Figure 8 shows our universal predictions on test datasets. The results demonstrate that our method generalizes well across multiple unseen datasets from different domains. Figure 9 shows the visual comparisons of different models on the WildDash 2 benchmark.

CS

MPL

SUN

BDD

IDD

ADE

COCO

Image

GT

Ours

CS

MPL

SUN

BDD

IDD

ADE

COCO

KT

WD2

SN

VOC

CT

Image

GT

Ours

READ

petonPe

GT

Image

Our trained

model

MIX6D

FAN

Uni NLL+

Our unseen

model

UniSeg

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The final paragraph of our introduction (section 1) provides a comprehensive summary of the paper's contributions, which align with the claims made in the abstract.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the third paragraph of our conclusion (section 5), we discuss the limitations of our work, including factors influencing performance, and computational efficiency considerations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In section 3, Appendix A, we provide detailed implementation steps, ensuring that our experiments can be replicated. Additionally, in section 4, we thoroughly describe the settings of our experiments, further facilitating reproducibility.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide reproducible code along with detailed instructions, ensuring that the main experimental results can be faithfully reproduced.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In section 4, we provide comprehensive details of our experimental setting.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Training on numerous datasets is time-consuming, with each experiment taking over a week. During our experiments, we observed consistent and stable performance in terms of mIoU for our model.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In section 4, we specify that our experiments were conducted on four 80G A100 GPUs.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and adhered to the NeurIPS Code of Ethics in conducting our research.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the first paragraph of our conclusion (section 5), we thoroughly discuss the broader impacts of our work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research utilizes openly available datasets for training, and our models are primarily used for semantic segmentation tasks. Therefore, there are no such risks associated with our work.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In section 4, we clearly specify the assets used in our work, including datasets and models. We provide proper citations to the original papers and ensure that the licenses and terms of use are explicitly mentioned and respected.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide our code as an asset and include comprehensive documentation alongside it, covering details such as training procedures, licensing, limitations, and any necessary consent obtained from individuals whose assets are used.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing experiments or research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a multi-dataset training approach that works on top of a unified output taxonomy mapped onto individual dataset-specific taxonomies. First, a multi-head model is trained. Then, a unified taxonomy is automatically constructed by merging semantically identical classes, determined based on segmentation performance on individual classes when a connection is made. After that, GNNs are used to further refine the mapping from the individual dataset-specific classes to the unified taxonomy.\n\nThe input nodes of the GNN are dataset-specific labels and unified classes. Dataset-specific labels are first enriched with a textual description provided by ChatGPT, which is then encoded with LLaMA-2 and combined with a trainable dataset embedding to create node input features. The addition of the trainable dataset embedding enables the model to learn the different meanings of classes with the same name in different datasets.\n\nFinally, the training process alternates between fine-tuning the segmentation network and refining the mappings with the GNN.\n\nSTRENGTHS:\n1. Well-written and clear.\n2. Achieved state-of-the-art results on the WildDash 2 benchmark, demonstrating improvements over previous work.\n3. Introduces a novel approach for label unification.\n\nWEAKNESSES:\n1. The paper primarily focuses on individual benchmark performance rather than the quality of the recovered taxonomy. In practical scenarios, it's crucial to reason within a unified label space. Mistakes such as equating MPL:tunnel to ADE:fireplace could mean the difference between advancing or halting in a real-world scenario.\n2. The model achieving state-of-the-art performance on WildDash is trained exclusively on driving datasets, unlike other methods submitted that were trained on the full RVC collection (including ADE20K and COCO), potentially impacting performance in road driving contexts.\n3. There is significant variation in training design choices in Table 2, making it challenging to isolate the impact of each individual choice (e.g., backbone, selection of dataset-specific classes, etc.).\n4. The initialization of the adjacency matrix appears to be a crucial step for model performance, yet it is not discussed in the main paper.\n\nQUESTIONS:\n1. What is the size of the created taxonomy in different tables (e.g. in Table 5)? Furthermore, Table 5 could also include the performance after initializing the adjacency matrix, without additional GNN training.\n2. How does the quality of the created taxonomy compare to the manual taxonomy provided by [5]? Are all relationships recovered? For example, IDD:tunnel is not connected to MPL:tunnel in Figure 5. \n3. How are unseen datasets mapped to the unified taxonomy in Table 3? Is it done manually or automatically?\n4. Is there any advantage to using the formulation in Eq. 6 as opposed to NLL+ [4]?\n5. How much does the trained taxonomy differ from the initialized taxonomy? Do all unified classes \"survive\" after training, or do some become obsolete?\n6. What exactly is the zero-shot model on WildDash 2 in Table 4?\n7. How long does it take to initialize the adjacency matrix? What exactly is the difference compared to [52]? Are there similarities to [5], such as pairwise merging?\n\nLIMITATIONS:\n1. Focus on individual benchmark performance, rather than the process and the quality of the produced taxonomy.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method using GNN to automatically construct a unified label space across multiple datasets, addressing the issue of label space conflicts in multi-dataset semantic segmentation. The method eliminates the need for manual re-annotation or iterative training, significantly enhancing the efficiency and effectiveness of model training. Experiments show that this method has certain effectiveness.\n\nSTRENGTHS:\n1. The motivation behind this approach is clear, and automatically constructing a unified label space across multiple datasets in multi-dataset semantic segmentation is meaningful.\n2. The authors use numerous illustrations and provide a detailed description of the experimental parameters, making it easy to follow and reproduce.\n\nWEAKNESSES:\n1. This paper proposes a new method, but in my view, it is merely a collection of tricks. For example, $d_i$ is introduced in Equation (1), but the authors neither explain its role through ablation experiments nor from a methodological perspective. It is recommended that the authors provide a more detailed explanation of the motivation for each component of the method.\n2. Although the motivation for this work is multi-dataset semantic segmentation, the experimental comparison methods do not include the latest methods from the multi-dataset semantic segmentation community, such as [1,2], etc.\n3. This paper only uses HRNet-W48 as the backbone. The authors are encouraged to further explore the scalability of the proposed method on transformer-based networks.\n4. From Table 5, it appears that the improvement in results due to label description is not significant. However, using GPT to complete this step inevitably incurs a substantial consumption of time and resources, which raises questions about whether it is worth it. Additionally, the authors should discuss the impact of the proposed method on time and space costs.\n\n[1] Gu X, Cui Y, Huang J, et al. Dataseg: Taming a universal multi-dataset multi-task segmentation model. NeurIPS 2023.\n\n[2] Wang L, Li D, Liu H, et al. Cross-dataset collaborative learning for semantic segmentation in autonomous driving. AAAI 2022.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nLimitations are discussed in the Conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to automatically match and unify different label spaces for semantic segmentation. This allows to train a single model on multiple, differently annotated datasets. The authors can show that this can yield benefits in overall model performance, also compared to other multi-label approaches. The method is the currently best result on the public WildDash2 benchmark.\n\nSTRENGTHS:\n-\tThis method is the current SOTA on the public learderboard of WildDash2, which means the proposed method holds against a very rigorous testing setup.\n-\tWith the current knowledge that data scale is one of the key ingredients in well-generalizing models, progress on multi-dataset-training has high potential for impact. For example, the current leader of all ScanNet 3D segmentation leaderboards is also a method based on multi-dataset-training. \n-\tThe presentation is very clear and structured for the most parts of the paper.\n\nWEAKNESSES:\n-\tIt is currently unclear to me based on what information the graph connectivity is learned. How the method is described, the name of the annotated label is the only information that is put into the nodes, and all matching is based on that plus a hallucinated description from GPT. However, the authors mention in line 199-200 that the method would leverage visual similarity between annotations. It is not clear to me how this is facilitated.\n-\tAll results are based on an architecture from 2019. It is lucky that WildDash2 has no stronger competitors in the leaderboard, but it remains unclear whether the method would yield the same improvements for a more competitive base architecture (e.g., mIoU of 77% on Cityscapes is a very bad baseline performance in Table 2. The achieved “improved” 80.7% are on-par with DeepLabv3, which was SOTA in 2018).\n-\tThe method underperforms on more general indoor/outdoor datasets. Both on ADE and COCO, MSeg is better (even when predicting less classes, and in contrast to the bolded numbers, which are somehow ignoring MSeg)\n-\tI don’t think it is well discussed how hallucinating label descriptions through ChatGPT can introduce mistakes into the methodology. I don’t know about all of the datasets, but for example Cityscapes provides their own descriptions of the labels, which are the descriptions that also label workers use to label the data (see https://www.cityscapes-dataset.com/dataset-overview/#labeling-policy ). LLMs that are simply prompted with the label name can easily make up wrong descriptions that are not calibrated to the label policy of the annotation.\n\nQUESTIONS:\nsee above. The points that can hopefully easily be clarified by the authors are the question how visual information is used in the matching, what to do about GPT outputting wrong definitions, and why the method underperforms on more general datasets with larger label spaces.\n\nLIMITATIONS:\n-\tThe broader impact does not at all discuss potential societal impact of the work. While it is true that it reduces the re-labeling effort, the method still requires segmentation labels, which are the much higher cost compared to re-labeling existing segmentation labels. What should be rather discussed are the societal impact of deploying a method with an automatically generated label space, and what the implications of this are to safety and certification when trying to deploy this method e.g. to automated driving.\n- The discussed limitations are OK. It would be interesting of the authors could further comment on ways how to potentially get around the scaling problem of current requiring all datasets loaded on one node, and every dataset beeing sampled for every batch.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach to automatically train a semantic segmentation network using multiple datasets with individually different class label policies. A unified label space emerges during training automatically without direct manual label mapping definitions by using a graph neural network (GNN) which is guided by textual descriptions of each label in unison with a multi-head segmentation head. The training alternates between GNN and segmentation network training to jointly optimize the label mapping for multiple dataset as well as the actual image segmentation task itself. Multiple experiments evaluate the approach to unify 7 mixed datasets (CS, MPL, SUN, BDD, IDD, ADE, COCO) as well as 5 road-scene datasets (CS, IDD, BDD, MPL, WD2). Performance is compared by both training single heads and using the Multi-Seg. Head. as well as benchmarking on WD2. Additional ablation experiments showcases the value of textual label descriptions for the unification process. \nIn summary this approach allows for a joint training with multiple mixed segmentation datasets without additional manual intervention.\n\nSTRENGTHS:\n*) Generation of segmentation label space mappings to automatically create a unified label space.\n*) Great overall performance for each tested dataset\n*) Record benchmark results for Wilddash2\n*) Many experiments to showcase the approach on multiple driving-scene datasets as well as some more generic datasets.\n*) Supplemental contains the full training code to recreate the experiments\n\nWEAKNESSES:\n*) Some details are omitted (see Questions) making it hard to judge the extent/validity of some claims (e.g. robustness/generalization).\n*) Multi-head training and evaluations favor the accumulation of per-dataset bias. The cross-validations are a good start but there is no evaluation of \"new\"/out-of-distribution labels from one dataset within the other.  This amplifies dataset-specific training (each head is optimized to work for one specific dataset); thus training to \"solve\" datasets rather than the task at hand. L281 (tunnel == fireplace) is a great observation but also shows the weakness of the approach.\n*) \"Splitting\" of source labels is not supported; the multi-heads are trained in isolation so if a label in one dataset requires splitting in another dataset, the respective heads are isolated (one has the split, the other doesn't). During inference of an unseen image this can result in only a single merged representation instead of the more detailed split. This effect is underrepresented during evaluation as dataset bias further separates the heads to work in favor of the overlapping test data.\n*) Both WD1 and WD2 are used. WD2 supersedes WD1 in every way /(it also includes all WD1 frames with the extended label policy) and crucially also uses a unified label policy combining IDD, MPL, CS & WD1 (80 label cat.; the benchmark uses the reduced WD2_eval 25 cat policy). So contrary to L223, WD2 does contain the labels \"lane marking\", \"crosswalk\" and \"manhole\". WD2 has a different publication associated with it which is unmentioned: CVPR22 \"Unifying Panoptic Segmentation for Autonomous Driving\". The paper/experiments would be easier to understand with only WD2.\n*) For some figures/descriptions in the paper it is unclear which dataset combinations was used during training. Introduction of fixed names (e.g. 5ds vs. 7ds for 5/7 datasets; or explicitly mention \"domain-general\"/\"domain-specific\" wording) could help.\n\nQUESTIONS:\n*) The creation of textual descriptions using ChatGPT is not described at all; which version of ChatGPT was used? How were per-dataset label spaces described as textual input? Were images/masks/masked images used with a multi-modal version of ChatGPT (e.g. GPT4V; GPT4o; ...). Using masks together with images as inputs for ChatGPT is non-trivial. L110 contains no details; wording in Figure 2 suggests a multi-modal model as \"image of\" is part of the dataset label text or simply the \"blind\" pure-text completion of prompts in the style \"An image of <label> from the dataset <dataset>....\". Please clarify; the text label descriptions are an important aspect of the paper's novelty!\n*) Where is the final resulting unified label mapping?  Figure 5 shows a small part but the full joined label set would give a good indication of generalization vs. specialization. \n*) Supplem. B. (target number of unified labels) is a crucial component as it directly steers the separation/merging of similar concepts (e.g. seperate \"Cityscapes_road\"/\"MPL_road\" labels vs. a unified \"road\" label). How is the hyperparameter Lambda|L| chosen and what number of labels did the experiments end up with?\n*) What pre-training was used for the initialization of the multi-heads?\n*) WD2 negative examples are a defined subset within the benchmarking dataset. However, the official rules of the benchmark state, that both regular/\"positive\" and negative frames/subsets have to be treated equally. The comparability within the benchmark is only held upright, as long as this rule is observed.  L246/L247 can be interpreted otherwise: \"We map non-evaluated classes to a void label\"; is this applied to both positive and negative frames alike?\n\nLIMITATIONS:\n*) Limitations relating to dataset bias could be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach for automated label unification across multiple datasets in semantic segmentation using Graph Neural Networks (GNNs). The proposed method aims to create a unified label space that allows semantic segmentation models to be trained on multiple datasets simultaneously without the need for manual re-annotation or taxonomy reconciliation. The results demonstrate significant performance improvements over existing methods, achieving state-of-the-art performance on the WildDash 2 benchmark.\n\nSTRENGTHS:\n+ Novel Approach: The use of GNNs to automatically construct a unified label space is innovative and addresses the challenge of conflicting label spaces in multi-dataset training.\n+ Performance: The method shows impressive performance improvements, achieving state-of-the-art results on the WildDash 2 benchmark and outperforming other multi-dataset training methods.\n+ Efficiency: The approach eliminates the need for manual reannotation and iterative training procedures, significantly enhancing the efficiency of multi-dataset segmentation model training.\n+ Robustness: The method demonstrates good generalization to unseen datasets, indicating its robustness and applicability across various scenarios.\n\nWEAKNESSES:\nThis indeed seems a interesting problem. I have several questions:\n\n1. How do you select the unified label size N? Is it adaptive to different dataset labels, or preset hyper-parameters?\n2. Can you show more results of the bipartite graph generated the GNN? Like what labels is hard to link by text features, but linked by the GNN.\n3. Is the generated graph by GNN influenced by the number of datasets in training or the models? For example, if I use different models for the training, will the graph be the same? If I use two dataset in multi-dataset training, will the nodes of these two dataset labels have the same edges as in when I use three datasets?\n4. Can I apply the generated graph to other models' training without the need for doing GNN training again? If not, how much training cost will GNN bring?\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nThe paper is interesting and the experiments seem to prove the effectiveness. But I need more evidence to prove the robustness of the GNN results when using different datasets and models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}