Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation
Authors: Ruihao Xia
Yu Liang
Peng-Tao Jiang
Hao Zhang
Bo Li
Yang Tang
Pan Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain adaption
Semantic segmentation
Multi-modality
Primary area: machine_vision

Abstract: Despite their success, unsupervised domain adaptation methods for semantic segmentation primarily focus on adaptation between image domains and do not utilize other abundant visual modalities like depth, infrared and event. This limitation hinders their performance and restricts their application in real-world multimodal scenarios. To address this issue, we propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. Specifically, MADM comprises two key complementary components to tackle major challenges. First, due to the large modality gap, using one modal data to generate pseudo labels for another modality suffers from a significant drop in accuracy. To address this, MADM designs diffusion-based pseudo-label generation which adds latent noise to stabilize pseudo-labels and enhance label accuracy. Second, to overcome the limitations of latent low-resolution features in diffusion models, MADM introduces the label palette and latent regression which converts one-hot encoded labels into the RGB form by palette and regresses them in the latent space, thus ensuring the pre-trained decoder for up-sampling to obtain fine-grained features. Extensive experimental results demonstrate that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities. We open-source our code and models at https://github.com/XiaRho/MADM.

Unsupervised Modality Adaptation with Text-to-lmage Diffusion Models for Semantic Segmentation
Ruíhao Xia 1.2 Yu Liang" Peng-TTo Jiang" HHo Zhang²
Bo Li Yang Tang ,3, Pan Zhou"
East China University of Science and Technology, vvivo Mobile Communication Co., Ltd
Peng Cheng Laboratory, *Singapore Management University
Abstract
Despite their success, unsupervised domain adaptation methods for semantic seg- mentation primarily focus on adaptation between image domains and do not utilize other abundant visual modalities like depth, infrared and event. This limitation hinders their performance and restricts their application in real-world multimodal scenarios. To address this issue, we propose Modality Adaptation with text-to- image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained extensive image-text pairs enhance the model's cross-modality capabilities. Specifically, MADM comprises two key complementary components tackle major challenges. First, due to the large modality gap, using one modal data to generate pseudo labels for another modal- ity suffers from a significant drop in accuracy. To address this, MADM designs diffusion-bassee based pseudo- labels and enhance label accuracy. Second, overcome the limitations of latent low-resolution features in diffusion models, MADM introduces the label palette and latent regression which converts one-hot encoded labels into the RGB form by palette and regresses them in the latent space, thus ensuring the pre-trained decoder for up-sampling obtain fine-grained features. Extensive experimental results demonstrate that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities. We open-source our code and models at https: // /////gttubb/ com com/Xiie aRho/ /MADM.
generation which adds latent noise to stabilize pseudo-
Introduction
Unsupervised Domain Adaptation for Semantic Segmentation (UDASS) involves a source domain with image-label pairs and target domain with only unlabeled samples [9-11], and has achieved promising segmentation results in the image modality. Currently, most existing UDASS methods are restricted to transferring knowledge between similar image domains, such from virtual scene [2, 12] to real scene [13], or from daytime scene [13] to nighttime scene [3, 4J. However, these approaches do account for the wide range of visual modalities present in real-world scenarios, such as depth, infrared, and event modalities, which are valuable in nighttime perception [14-16] but often lack sufficient and high-quality labels for supervising segmentation training. Hence, in this paper, we are particularly interested in extending UDASS to Unsupervised Modality Adaptation for Semantic Segmentation (UMASS) across different visual modalities. i.c., the adaptation of a model from labeled source image modality to unlabeled target modality.
Differences across images arise from the objects, lighting, camera parameters, etc, while there fundamental disparities in imaging principles across modalities that lead to greater variability. This
Work was done during interning vivo. Conresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Image
Target Modalities
Comparison
Small Domain Gaps
0.46
3.40
Synhetic:
Source Modality
Depth:
0.58
0.43
0.57
0.44
Nighttime: 0.77
Images: 0.74
Infrared: 0.75
Large Modality Gaps
Snow: 0.79
Event:
Rein
MADM (Ours)
Label
Figure I: (1) On the left, we leverage the multi-modality model ImageBind [1] to quamtify the simi- larity of images and modalities across datasets, i.e., GTAS-Synthetie [2], Dark Zurich-Nighttime [3],
ACDC-Snow [4], DELIVER-Dee -Epttttthh [5], FMB-Infrared [6]. and DSEC-Event [7]. Specifically, we randomly select 500 samples from each dataset, and compute the average cosine similarity of the
output vectors within the dataset (right side of the text) and between the datasets (on the arrows). (2) On the right, compare the quantitative results with the state-of-the-art (SoTA) method Rein [8] on three different modalities.
is illustrated by Figure which shows that the similarity among various image domains tends to be higher than between different modalities. These significant modality discrepancies poses significant challenges to existing UDASS methods [17, 18] on multimodal segmentation due to their limited pre-trained knowledge. Specifically, the backbone [19] used in current SoTA UDASS methods is pre-trained on the ImageNet- K dataset [20] which contains one million images categorized into 1,000 distinct classes, providing the network with foundational level of semantic understanding. While this backbone achieves promising results in UDASS, its insufficient pre-trained knowledge limits generalization to other visual modalities.
To address this issue, inspired by Text-to-lmage Diffusion Models (TIDMs) [21], which are trained with internet-scale image-text pairs [22], we recognize that extensive pre-training data unifies samples with different distributions but similar semantic properties through texts, significantly enhancing the model's ' semantic understanding and generalization. Although TIDMs are not trained on other visual modalities, their large-scale samples and unification through texts enable them to adapt to broader distribution of domains. Also, the extensive pretraining provides TIDMs with robust understanding of high-level visual concepts, enabling their application to various domains, such as [25]. This strong prior motivates us to utilize TIDMs robust backbone for solving UMA. Therefore, Adaptation with text-to-image Diffusion Models, which takes full advantage of the generalization of pre-trained diffusion models and facilitates robust adaptation for accurate semantic segmentation in other visual modalities. Specifically, TIDM used to extract robust features for segmentation and is trained in self-training manner [17] which takes unlabeled target samples input and generates pseudo-labels for training TIDM itself. Building upon TIDMs [21] ]] and self-training [17], our MADM incorporates two innovative components: Diffusion-based Pseudo-Label Generation (DPLG) and Label Palette and Latent Regression (LPLR), which address the challenges unstable useudo-labeling and lack of fine-graine' features extraction, respectively.
semantic matching [23], depth estimation [24], and 3D awareness
present MADM: Modality
First, significant modality discrepancies hinder robust and high-quality pseudo-label generation, which is crucial for further training and enhancing the model. Directly using these unstable labels to train the model can lead to serious biases in the target modality. Thus, we propose DPLG which adds latent noise to target samples before generating pscudo-labels where the noise level gradually

---

decreases as training stabilizes. These pseudo-labels then supervise the noise-free free target modality predictions. The mechanism behind DPLG leverages the denoising property of diffusion models, making the target modality more consistent with pre-trained inputs, thus improving accuracy. Unlike previous supervised diffusion- semantic segmentation methods [26-28] which adopt single- step forward and remove the diffusion process, we find that proper diffusion process can stabilize the generation of pseudo-labels and adapt more successfully to the target modality.
Second, TIDMs [21] encode images into the latent space using a pre-trained Variational AutoEncoder (VAE) for diffusion and denoising, and then up-sample the latent output to the original resolution using the VAE decoder. When adopting TIDMs as backbone, the resolution of the features is too low, resulting in loss of details. To address this, we propose LPLR to convert pixel-level classification into regression in RGB form, utillizigg the up-sampling capability of the pre-trained VAF decoder a recyeling manner. Specifically, we use palette to convert one-hot encoded labels into RGB form and encode these RGB labels into latent representations. Then, the model is trained with a regression loss fit the latent labels, obtaining high-resolution fine-gres graiigd features via the VAE decoder. Different from previous diffusion- baassd semantic segmentation methods [26-28] that directly extract multi-scale features from the denoising UNet network, we take inspiration from depth estimation with TIDMs [24] and propose LPLR to extract high-resolution features. Our contributions are summarized as follows:
We propose MADM, extending traditional UDASS UMASS with a pre-trained TIDM backbone for generalizing across various visual modalities.
We design Diffusion- Pseudo-Labed Generation (DPLG) to provide more robust pseudo- labels by adding annealed latent noise to target samples for stable modality adaptation.
We introduce Label Palette and Latent Regression (LPLR) to convert semantic segmentation into regression for learning details, thereby repeatedly utilizing pre-trained VAE decoders for high-resolution feature extraction.
We demonstrate the effectiveness of our MADM through extensive experiments on three different modalities: Image [13] - Depth [5], Infrared [6], and Event [7].
2 Related Works
2.1 Unsupervised Domain Adaptation Semantic Segmentation
UDASS can be broadly categorized into two primary methods: adversarial and self-training methods. Adversarial approaches aim to align the distributions of the source and target domains at the level of images [29, 30] or features [31, 32] outputs [33, 34], thereby facilitating the transfer of knowledge. On the other hand, self- teff-training methods [9, 35] operate on the paradigm of pseudo- labeling, where the model's predictions on the target domain act ground truth during training, iteratively refining the segmentation. Recently, the field has witnessed the development from CNN-based methods [30] to Transformer-based methods [17, 18, 36), leveraging the self-attention mechanism to capture long-range dependencies and enhance cross-domain feature representation.
However, most of these methods have predominantly focused on adaptation between different image domains, such as from synthetic [2, 12] to real-world images or across varying environmental conditions [4]. They will fail when adapting to other visual modalities, such depth, infrared, or event, which have distinct data distribution. Thus, we propose MADM to address the limitation of UDASS for adapting other unexplored visual modalities. Besides, different from multi/cross- modality domain adaptation [37, 38] which has paired two modalities in both domains, our modalities different in source and target.
2.2 Text-to-Image Diffusion Models
Diffusion denoising probabilistic models [39] have set new benchmarks in the quality and controlla- bility of generative tasks. These models are distinguished by their two-phase paradigm: the diffusion process that progressively adds noise to the sample, and the denoising process that learns to denoise the corrupted sample by network. Utilizing the MSE loss between the residual noise and the predic- tion as training objective, diffusion models have demonstrated greater training stability compared to generative adversarial networks [40] and VAEs [41]. To achieve high-quality controllable image
3

---

Label Pulette and Latent Regression (LPLR)
VAE Decoder
Latent RGB Label
Finelune
Frozee
reg s/t
Palette
VAE Encoder
Source Label
One-Hor Label ys/9c
RGB Label
Latent Feat.
Oxfz
I
Prediction Part
}
VAB Encoder
Multi-Scut Scale Feat. fatt
Segmentation Hend
Source Image X,
Target Depth
Denoising UNet
EMA Update
2p
Teacher Network
Latent Noise
Diffusion-based Pseudo Label Generation (DPLG)
Pscudo Label
Figure 2: Our framework is divided into three parts. Self-Training: Supervised loss in the source modality L, and pseudo-labeled loss L; in the target modality are used to train the network. (2) Diffusion- Pscudo-Label Generation (DPLG): In the early stage of training, we add noise on the latent representation to stabilize the pseudo-label generation.
(3) Label Palette and Latent Regression (LPLR)): The one-hot encoded labels y./ / are converted to RGB form by palette and then encoded to the latent space to supervise the UNet output 0s/t-
generation with reduced computational demands, Rombach et al. [21] proposed the latent diffusion model that leverages cross-attention layers and confines the diffusion process to a low-resolution latent space, which has emerged as a widely recognized TIDM.
In recent advancements beyond the generative tasks, the exceptional semantic comprehension capabil- ities of TIDMs have been harnessed to enhance performance in many downstream applications. Xu et al. [27] presented a novel framework that integrates pretrained TIDM with discriminative model to address the challenges of vocebulaay segmentation. Similarly. Zhao et al. [26] demonstrated the versatility of TIDMs by fine-tuning deal with various visual perception tasks, including semantic segmentation, referring image segmentation, and depth estimation. Gong et al. [28] introduced innovative scene prompts and prompt randomization strategy on TIDMs, achieving new milestones in domain generalization and test-time domain adaptation. Their works highlight the potential of
TIDMs to generalize across diverse domains and adapt to new ones with minimal additional training. It's worth noting that the aforementioned methods necessitate only single-stey step forward pass through the denoising UNet, significantly streamlining the inference process.
The successful and diverse applications of TIDMs inspire our exploration into the generalization of TIDMs more challenging visual modalities. However, we observe that the latent diffusion property within TIDMs leads to the lower-resolution feature extraction. To address this limitation, we propose the LPLR that converts semantic segmentation into latent regression to obtain fine- graigra features.
3 Method
3.1 Overview
In UMASS, given the labeled source RGB modality {(x,, ys)} and the unlabeled target modality {(x,), our objective is to train network which accepts input and outputs the corresponding semantic segmentation results pr. As shown in Sec. 1, primary challenge this task stems from the significant disparities between the two modalities. To address this challenge,
propose to
leverage the TIDM [21] as our backbone which tre-tr on a vast array of image-text pairs [22] to enhance its generalization and can robustly extract features across modalities. Next, to overcome

---

the inaccurate pseudo-labels due to large modality gaps, propose Diffusion unffusion-based Pseudo- Generation (DPLG). which stabilizes pscudo-label generation by injecting noise to the target modality, Moreover, TIDMs can only extract low-resolution features within the latent space, leading to a loss of semantic detail. To address this, we propose the Label Palette and Latent Regression (L.PLR), which transforms pixel-wise classification into the regression, thereby allowing us to harness the fine-grained features upsampled by the pre-trained VAE Decoder. Our framework is illustrated in Figure 2. Next, we will introduce the our proposed DPLG and LPLR in detail.
3.2 Diffusion-based Pseudo-Label Generation
As shown in Figure in MADM. we employ the TIDM perform single-step diffusion to extract multi-scale features from the intermediate output of the denoising UNet, following the approach in [26, 27]. First, samples from source and target modalities Is; 2z are encoded to the latent representation 2x/t E(xa/t) by the pretrained VAE encoder &. Without any additional noise, we feed them to the denoising UNet and obtain multi-scale features faʃt UNet(z,jee c), where e is a learnable conditioned embedding instead of a textual description. Then. these features are subsequently fed into segmentation head to generate the semantic segmentation prediction Ps/t Seg(fs/e).
Our training method is anchored on the self-training DAFormer [17] prevalent in UDASS. The training objective is composite of supervised loss from the source modality and pseudo-labeled loss from the target modality. For the labeled source modality, we use a cross-entropy loss between the prediction ps and the ground truth labels ys:
C. = CCE
For the unlabeled target modality, adopt student-teacher paradigm in self-training [17]. Here, the existing network acts as the student, and through the Exponential Moving Average (EMA), we derive teacher network [42]. The teacher network generates bseudo-labels ÿt. which then supervise the student network's 's output on A(rz), where A(-) represents the strong data augmentation [43].
However, the data distribution varies greatly between modalities. The teacher network is unable to provide accurate pseudo-labels for self-training, resulting in an unstable modality adaptation to target modality. We observe that more robust pseudo-labels can be generated by injecting appropriate Pscudo-Label Generation (DPLG). The proposed DPLG exploits the denoising property (perception of noise) of diffusion models to improve the robustness of semantic understanding on target samples.
noise in the latent space and therefore propose the Diffusion-basee
Specifically, given target sample It, we first encode it into the latent representation 2; -(rr) and then apply diffusion process that adds noise to 2 to obtain noisy latent representation:
O + Van NOO, k B max - -177).
Here, āg is a predetermined schedule that con- trols the amount of noise added at each step [39]. l is the diffusion step that controls the propor- tion of noise based on the initial diffusion step 3 and noise addition period y, and the current iteration count.
Training time
This noisy representation =/ is then used to gener- ate pseudo-labels ÿr Seg( (UNe c)). In the pre-training of TIDMs [21], the objective is to estimate noise from latent inputs containing vari- ous noise levels. By injecting noise into the latent code, we effectively simulate this noisy distribu- tion. This simulation aligns the latent space more closely with the data distribution encountered during the pre-training phase. Such alignment fosters more robust and accurate semantic inter- pretation, which, in turn, enhances the quality of the pscudo labels generated. This shares similar
Pseuda Figure 3: We visualize the pscudo-labels for event modality at the iter- ation 1250, 1750, and 2250. The introduction of DPLG effectively improves the quality of pseudo-labels.
Event
5

---

spirits in other applications of diffusion models, such as the text-to-3D [44] where injecting extra noise into data can improve the denoising quality of image and yields better pseudo labels. As shown in Figure 3, our DPLG can generate more accurate pseudo-labels compared to the noise-free addition. By strategically incorporating noise in the pseudo-label generation, DPLGG enhances the model's adaptability to the target domain and mitigates the bias of semantic understanding, Then, the pseudo-labeled loss formulated as:
Li = LCC LPE Ut)
Here, q is the confidence value calculated by the softmax probability of Pt [17, 18, 43]. The consistency regularization 2 between the teacher and student networks promotes adaptation to the target modality. encourages the student network match the teacher's predictions, even under the perturbations introduced by strong data augmentation.
3.3 Label Palette and Latent Regression
TIDMs compress the sample into latent space with 8x down-sampling factor, which leads serious loss of semantic detail. Specifically, for the input sample with resolution of 512 x 512, reduced to 64 x 64 after embedded via E. Then, within the denoising UNet decoder, multi-scale features are extracted fs/t after the 5th, 8th, and Ith blocks: 64 x 64, 32 x 32, and 16 x 16.
For diffusion-based depth estimation [24], leveraging the VAE decoder D to upsample the denoised latent representation back to the original resolution is natural fit, which recovers the one-ggaieed scene details. However, the above method is not applicable to semantic segmentation due to the inherent difference between regression and classification. Therefore, to address the problem of semantic detail loss, we propose the Label Palette and Latent Regression (LPLR) module which converts semantic segmentation into regression and utilizes the VAE Decoder 'D to obtain high- resolution semantic features.
Initially, the one-hot encoded labels y, and Ør are transformed into a perceptually meaningful RGB space with pre-defined palette. These RGB representations then encoded back into the latent space and supervise the UNet's output Ox/t = UNet(z e) in a regression form:
1 ((aaletteyy//be) - o.
With this supervision, we are able to utilize the VAE decoder obtain a high- righuutsoottion semantic regression feature ((0,)e). This feature, combined with the multi-scale features, is then fed to the segmentation head Ps/t = Seg( ((o,/t)).
By employing LPLR, effectively convert the semantic segmentation into regression problem that can be tackled by the VAE decoder's 's upsampling capabilities, which retain the fine-grained details necessary for accurate segmentation. Finally, the total training objective is sum of these losses:
L = L. + Li + (2 + 4re
Experiments
4.1 Implementation Details
In our work, we utilize the Stable Diffusion v1-4 model [21], which has been pre-trained on the LAION-5B [22] dataset, as our TIDM. For the segmentation head, we instantiate it with the de- coder from DAFormer [17]. We train our MADM for 10k iterations with a batch size of and an image resolution of 512x512. The optimization is instantiated with AdamW [45] with learning rate of 5e-6. For hyperparameters 8, 7. and Arey in DPLG and LPLR, we set them to 1,800,60,100//880000 1,0000,,0.1000///101 11tp..10///8// (//(8000,500,000 10.0) for depth/infrared/event modalities, respectively. In our [13] dataset as the source modality and the DELIVER- Depth [5], FMB-Infrared [6]. and DSEC-Event [7] datasets as the target modalities. Since the semantic classes these datasets are not identical, we merge some semantically similar classes during training. Experiments conducted on NVIDIA H800 GPU, occupying about 57G memory.
experiments, we adopt the Cityscapes-Imanl

---

Table Semantic segmentation results evaluated with MloU (%) on three modalities. Bold numbers
are the best, underscored second best.
(a) Cityscapes [13]
DELIVER Depth [5].
Method
Sky
Build.
Fence Person
S.walk
Veg.
Vehi.
Wall
Tr.S.
MIoU (avg)
DAFormer [17]
82.28
43.35
11.82
56.03
13.90
80.10
15.44
60.08
72.67
0.18
44,20
43.64
MIC [18]
85.10
77.78
7.30
33.41
77.04
27.24
67.07
57.25
43.92
PiPa [46]
76.90
79.65
15.61
60.21
18.76
77.71
35.30
59.76
84.54
0.00
31.04
49.04
Rein [8]
92.00
78.78
27.75
43.88
32.34
78.81
27.50
58.06
76.45
0.34
36.68
50.23
MADM
95,52
86.70
12.48
41,88
18.99
93.97
54.12
67.12
84.29
0.00
33.34
$3.49
(b) Cityscapes [13] -> FMB-Infrared [6].
Method
Sky
Build.
Person
Pole
Road S. S.walk
Veg.
Vehi.
TrS..
MloU (avg)
DAFormer [17]
36.97
66.78
51.42
18.91
41.23
28.81
43.88
69.44
12.71
41.13
PiPa [46]
25.42
71.60
63.62
16.40
39.53
31.64
45.21
70.25
41.38
45.01
MIC [18]
38.11
71.63
57.89
17.59
40.68
33.93
49.49
70.26 29.85
45,49
Rein [8]
84.07
72.84
67.10
26.40
85.92
30.50
72.61
84.51
21.95
60.65
MADM
88.79
71.52
70.51
22.30
89.08
19.88
69.83
77.10
51.08
62.23
(c) Cityscapes [13]
DSEC-Event [7].
Method
Sky
Build.
Fence
Person
Pole
Road
S.walk
Veg.
Vehi.
Wall
Tr..
MIoU (avg)
DAFormer [17]
81.14
51.43
1,15
0.03
10.59
72.49
26.45
61.14
39.79
0.00
24.84
33.55
PiPa [46]
91.38
76.30
6.41
0.71
18.15
83.97
33.22
77.88
55.61
0.00
32.49
43.28
MIC [18]
92.36
79.20
6.69
32.80
19.30
79.75
31.46
68.17
58.35
0.01
39.30
46.13
Rein [8]
85.40
73.34
32.28
18.71
90.64
53.88
75.42
79.44
12.77
39.13
51.86
MADM
92.60
78.21
26.51
29.08
22.78
92.20
62.90
81.70
75.11
23.92
34.43
56.31
4.2 Datasets Setting
Cityscapes [13] the source dataset in our experiments, which constitutes real-world collection of street-view images captured across 50 distinct urban environments. The dataset is split into 2.975 training images and 500 validation images with resolution of 2048 1024. It provides comprehensive semantic labeling at the pixel-level with 19 distinct semantic classes.
Cityscapes-Image.
DELIVER-Depth DELIVER [5] is synthetic dataset containing five environmental conditions created by the CARLA simulator [47]. The dataset contains 25 semantic classes and 3,983/2,005/1,89 samples for traiiiggvaliddaiionnnsstn with resolution of 1024x 1024.
FMB-Infrared. FMB [6] is an urban street dataset with 1,500 RGB-IIfrared pairs at a resolution of 600 with 14 semantic classes. It contains wide range of real driving scenes under different lighting and weather conditions.
DSen-Event, DSEC [7] is a stereo event camera dataset for driving scenarios. Driving data are recorded for 3.193 seconds diverse illumination conditions and urban/rural environments. Event data have resolution of 640xx80 with 11 semantic classes and we aggregate them into the edge form in a recurrent manner [48].
4.3 Comparison with State of the Art Methods
Table presents the comparison with existing SoTA methods DAFormer [17], MIC [18], PiPa [46], and Rein [8] across three modalities: Depth, Infrared, and Event. The comparison is based on the Mean Intersection over Union (MIoU) over all classes, standard measure of segmentation accuracy,
Our MADM demonstrates strong performance, achieving the MIoU of 53.49%, 62.23%, and 56.31 (6.31% % on the depth, infrared, and event modalities, respectively. It showcases a significant improvement over the SoTA method Rein [8] by +3.26%, +1.58% +4.45%, which underscores the robustness and effectiveness of MADM in handling the other visual modalities. Also, it worth noting that the self-training loss in our MADM built upon DAFormer [17], exceeding it average of +17.9%
Figure offers a intuitive comparison of the semantic segmentation results. MIC [18] leverages the SegFormer backbone [19] that is pre-trained on the ImageNet- 1k dataset [20], enabling it to capture
7

---

Input
MIC
Rein
MADM (Ours)
Label
Figure 4: Qualitative semantic segmentation results generated by SoTA methods MIC [18], Rein [8], and proposed MADM on three modalities.
more details within scenes, such as "pole". However, it exhibits weaker modality understanding. leading to frequent mis-segmentation, such as incorrectly classifying the "sky" as the "road". In contrast, Rein [8] is built upon the DINOv2 backbone that is pre-trained on extensive, curated datasets without explicit supervision [49]. This results in improved semantic understanding of modalities compared to MIC [18]. Nonetheless, Rein still encounters issues with mis-segmentation and instances of under- sellerr-emmenttiion.
Our MADM stands for its exceptional ability output precise segmentation results that closely mirror the ground truth. The incorporation of TIDM significantly enhances the generalization of our approach, providing an enhanced comprehension diverse visual modalities and substantially mitigating the mis-segmentation.
4.4 Ablation Studies
Table 2: Ablation of DPLG and LPLR in depth, infrared, and event modalities.
Table presents the complete ablation studies that quantify the performance gains achieved by incorporating our proposed DPLG and LPLR into the baseline. The "Baseline" column in- dicates the performance of the MADM model without DPLG and LPLR. It serves refer-
Modality
Baseline
DPLG
LPLR
MADM
Depth
50.61
51.65
$2.91
$3.17=0.26
Infrared
56.28
61.86
58.75
62.14-0.18
Event
52.27
52.84
53.05
56.12=0020
Average
53.05
+2.40
+1.85
+4.09

---

Depth
k =
k= 10
k 25
k 50
k == 100
200
Figure 5: At the 1,250th iteration, we present a visual analysis of diffusion step k in DPLG.
point but achieves performance on par with the SoTA methods in Table 1, which demonstrates the strong generalization TIDMs.
(I) When DGLP is employed, the MIoU is improved by average of +2.40%. highlighting the effectiveness of generating robust pseudo-labels. Especially the infrared modality, achieves +5.58% relative improvement over the baseline. The quantitative results of the pseudo-labels enhancement are shown in Figure 3. (2) The application LPLR contributes to an average gain of +1.85%, emphasizing the importance of high-resolution features for segmentation tasks. (3) By employing both DGLP and LPLR, we observe significant enhancement in +4.09% over the baseline, which underscores the synergistic benefits of combining robust pseudo-labels generation with fine-grained feature extraction.
4.5 Diffusion-basee Pseudo-Label Generation
We analyze the pivotal roles of 3 and y in our proposed DPLG, particularly within the depth modality. These parameters control the diffusion step on 2t, which is central to the stability and quality of pseudo-labels.
Table 3: Ablation of 3 and y in DPLG on the depth modality.
2,000
5,000
8,000
Average
Table 3 provides detailed presentation the impact of and y. For instance, when is set to 5,000, an increase in 3 from to 60 leads to noticeable improvement performance, with the model achieving its peak score of 53.49%. However, further increasing 3 to 80 results a decline in performance, indicating the existence of an optimal balance between these parameters.
51.62
52.46
52.52
52.20
60
51.30
53.49
52.55
52.45
80
52.01
52.85
48.63
51.16
Average
51.64
52.93
51.23
In Figure 5, we offer an illustration of how the diffusion step influences the generation of pseudo- labels. With noise-free addition (k = 0), the model encounters difficulties in accurately segmenting the "car" and "person" classes. Upon introducing moderate quantity of noise (k- 10 50), the segmentation is noticeably enhanced, yielding more robust segmentation. Conversely, excessive amount noise (k 200) leads significant degradation in segmentation.
4.6 Label Palette and Latent Regression
Table analyzes the loss weight Xreg within the proposed LPLR, which regulates the con- tribution of the regression losses. A minimal Areg of 1.0 and 3.0 yields MloU of 53.31 % and 54.40%, respectively, indicating the initial bene- fits of incorporating regression losses. Increasing Areg to 10.0 achieves the optimal MIoU of 56.31% signifying the most effective balance between the segmentation and regression losses.
Table 4: Ablation of Areg in LPLRR on event modality.
Xreg
3.0
5.0
10.0
15.0
MIoU
53.31
54.40
55.84
56.31
55.31
In Figure 6, we offer an illustration of the impact of LPLR. It can be seen that the utilization of LPLR results in more fine-grained segmenta- tion, e.g., "person" and "vegetation" in the left yellow box, "road" and "sidewalk" in the right yellow box, which greatly improves the perfor- mance of our MADM.
LPLK
Figure Qualitative our proposed LPLRR The prediction with LPIRR shows more accurate fine-grained segmentation.
modality

---

4.7 Benefits of MADM in Nighttime Datasets
As mentioned in Section 1, we indicate that other visual modalities present in real-world scenarios are valuable in nighttime perception. In this section, experiments are conducted on the infrared modality to prove this. The FMB-Infrared dataset [6] includes both image and infrared modalities on daytime and nighttime scenes. We adapt from cityscapes [13] with daytime RGB images to the nighttime image modality and infrared modality by our proposed MADA, respectively. Figure and Table show that the infrared modality has a clear advantage in the "Person" class due to obvious thermal differences and good suppression of light interference.
Table 5: Semantic segmentation results of RGB and infrared modalities evaluated with MloU (%) on FMB dataset [6].
Method
Sky
Build.
Person
Pole
Road
S. S.wlk
Veg.
Vehi.
Tr.S.
MIoU (avg)
RGB Infrared
88.85
68.14 82.40
64.79
25.80
89.09
32.43
70.32
84.13
7.27
58.98
87.94
82.69
21.50
76.21
26.50
76.61
83.80
16.69
61.59
RGB
RGB Output
Infrared
Infrared Output
Label
Figure 7: Visualization of daytime RGB images in Cityscapes dataset [13] ~> nighttime RGB and Infrared modalities in FMB dataset [6]
5 Conclusion
In this paper, we present MADM. With the powerful generalization of TIDMs, we extend domain adaptation to modality adaptation, aiming to segment other unexplored visual modalities in the real-world. Mcanwhile, we propose DPLG and LPLR solve the problems of pseudo-labeling instability and low-resolution features extraction within TIDMs. We hope our method can motivate further research on visual modalities other than RGB images. Limitations: However, despite using only a single-step forward for the diffusion model, the computation far exceeds existing UDASS networks. Future work could focus on distilling knowledge of TIDMs into lightweight models when adapting. Broader Impacts: Our work pushes the boundary of semantic segmentation for other visual modalities, which will benefit several applications like multimodal fusion. To the best of ability, MADM has little no negative social impact.
10

---

Acknowledgements
This work was supported by National Natural Science Foundation of China (Basic Science Center Program: 61988101), National Natural Science Foundation of China (62233005, 62293502), Funda- mental Research Funds for the Central Universities(2222224) 7006). the Programme of Introducing Talents of Discipline to Universities (the 111 Project) under Grant B17017. Pan Zhou was supported by the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier grants (project ID: 33-SIS-SMU-O22 and AA-SI-SMUU-000))
References
[1] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3110-151990 2023, 2
[2] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pages 102-118. Springer, 2016. 2,
[3] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertainty- aware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 7374-7385 2019. 1,2
[4] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. ACDC: The adverse conditions dataset with corresoondecces for semantic driving Conference on Computer Vision, pages 10765- 10775, 2021. 1,
understanding. In Proceedings of the IEEE/CVF International
[5] Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang. Simon Reiß, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Delivering arbitrary-moaal semantic segmentation. In Proceedings of the IEEL/CVF Conference Computer Vision and Puttern Recognition, pages 1 1136-1147 2023. 2,3, 6,
[6] Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan. Multi-interactive feature learning and full-time multi-modality benchmark for image fusson and segmentation. In Proceedings of the TEEE/CVF International Conference on Computer Vision, pages 8115-8124, 2023. 2, 3, 6, 1010
[7] Mathias Gehrig, Willem Aarents, Daniel Gchrig, and Davide Scaramuzza. DSEC: A sterco event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947 4954, 2021. 2, 3, 6, 7, 14
[8] Zhixiang Wei, Lin Chen, Yi Jin, Xiaoxiao Ma, Tianle Liu, Pengyang Ling, Ben Wang, Huaian Chen, and Jinjin Zheng. Stronger, fewer, & superior: Harnessing vision foundation models for domain generalized semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024. 2, 7, 8. 15
[9] Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. Category anchor-guider unsupervised domain adaptation for semantic segmentation. Advances in Neural Information Processing Systems, 32, 2019. 1,3
[10] Quanliang Wu Huajun Liu. Unsupervised domain adaptation for semantic segmentation using depth distribution. Advances in Newral Information Processing Systems, 35: 35:14374 14387, 2022.
[11] Guoliang Kang, Yunchao Wei, Yi Yang. Yueting Zhuang, and Alexander Hauptmann. Pixel-level cycle as- sociation: A new perspective for domain adaptive semantic segmentation. Advances in Neural Information Processing Systems, 33:3569-3580, 2020.
[12] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images semantic segmentation of urban scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3234-3243, 2016. 1,
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rchfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiele. The cityscapes dataset for semantic urban seene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3213-3223,
2016. 1, 3, 6, 7,
[14] Zhiwei Zhang, Zhizhong Zhang, Qian Yu. Ran Yi. Yuan Xie, and Lizhuang Ma. Lidar-camera panoptic segmentation via geometry consisteennistentttte and semantic- aware alignment. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3662-3671, 2023.

---

[15] Johan Vertens, Jannik Zürn, and Wolfram Burgard. Heatnet: Bridging the day-night domain gap in semantic segmentation with thermal images. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 8461-8468, 2020.
[16] Ruihao Xia, Chaoqiang Zhao. Meng Zheng, Ziyan Wu, Qiyu Sun, and Yang Tang. CMDA: Cross- domain adaptation for nighttime semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 011572-21588 2011,22000 2023.
[17] Lukas Hoyer, Dengxin Dai, and Luc Van Gool. DAFormer: Improving network architectures and training strategies for comain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1924-9935, 2022. 2, 3,5, 6, 7, 14,
[18] Lukas Hoyer. Dengxin Dai, Haoran Wang, and Luc Van Gool. MIC: Masked image consistency for context-enhanced domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1721-11732, 2023. 2,3, 6, 7,8, 15
[19] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers, Advances in Neural Information Processing Systems, 34:12077- 12090, 2021. 2,
[20] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNett large scale visual recognition challenge. International Journal of Computer Vision, 115:211 1 1-252, 2015. 2,
[21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 2, 3, 4, 5,
[22] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022. 2. 4, 6
[23] Xinghui Li, Jingyi Lu, Kai Han, and Victor Adrian Prisacariu. SD4Match: Learning to prompt stable diffusion model for semantic matching. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27558-27568, 2024.
[24] Bingxin Ke, Anton Ohukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, 2024. 3,3,
[25] Mohamed El Banani, Amit Raj. Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21795-21806, 2024. 2
[26] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5729-5733 2023. 3,4,5
[27] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang. and Shalini De Mello. Open- vocabulary panoptic segmentation with to-tae diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2955-2966, 2023. 4,5
[28] Rui Gong, Martin Danelljan, Han Sun, Julio Delgado Mangas, and Luc Van Gool. Prompting diffuusion representations for cross-domain semantic segmentation. arXiv preprint arXiv:2307.02/ 2023. 3,
[29] Suhyeon Lee, Junhyuk Hyun, Hongje Seong, and Euntai Kim. Unsupervised domain adaptation for semantic segmentation by content transfer. In Proceedings of the AAAI conferencce on Artificial Intelligence, volume 35, pages 8306-882 00008835.2200 2021.
[30] Yunan Liu, Shanshan Zhang, Yang Li, and Jian Yang. Learning to adapt via latent domains for adaptive semantic segmentation. Advances in Neural Information Processing Systems, 34:1167-1178, 2021. 3
[31] Yuhua Chen, Wen Li, and Luc Van Gool. Road: Reality oriented adaptation for semantic segmentation of urban scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7892 7901, 2018.
12

---

[32] Chaoqi Chen, Weiping Xie. Wenbing Huang, Yu Rong, Xinghao Ding, Yue Huang, Tingyang Xu, and Junzhou Huang. Progressive feature alignment for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 627-636 2019. 3
[33] Matteo Biasetton, Umberto Michieli, Gianluca Agresti, and Pietro Zanuttigh. Unsupervised domain adaptation for semantic segmentation of urban scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops. 2019. 3
[34] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming- Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEE Conference on Computer Vision and Pattern Recognition, pages 7472-7481, 2018. 3
[35] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European Conference on Computer Vision, pages 289-208900 305, 2018.
[36] Lukas Hoyer, Dengxin Dai, and Luc Van Gool. HRDA: Context-aware dighrresolution -adapive semantic segmentation. In European Conference Computer Vision, pages 372-391. Springer, 2022. 3
[37] Duo Peng, Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo. Sparse-to-dent feature matching: Intra and inter domain cross modal learning in domain adaptation for 3D semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7108 7117, 2021. 3
[38] Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang, Xiang Yu, Stan Sclaroff, Kate Sacnko, and Manmohan Chandraker. Learning mross-modal contrastive features for video domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 35618-13627, 2021."
[39] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-68511 2020. 3,5 5
[40] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations. 2019.
[41] Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013. 3
[42] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi supervisd deep learning results. Advances in Neural Information Processing Systems, 30, 2017. 5
[43] Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. DACS:
Domain adaptation via cross-domain mixed sampling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1379- 1389, 2021. 5,
[44] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D
diffusion. arXiv preprint arXiv:2209. 2488,88 2022. 6
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05100.200 2017. 6
[46] Mu Chen, Zhedong Zheng, Yi Yang, and Tat-Seng Chua. Pipa: Pixel-and patch-wise self-supervised learning for domain adaptative semantic segmentation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1905 1914, 2023. 15
[47] Alexey Dosovitskiy, German Ros. Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Conference on Robot Learning, pages 1-16. PMLR, 2017."
[48] Henri Rebecq, René Ranftl, Vladlen Koltun, and Davide Scaramuzza. Events-to viveoo Bringing modern computer vision event cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3857-3866, 2019.
[49] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacidin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07119 2023. 8
13

---

A Appendix
A.1 Visualization of LPLR
We visualize LPLR under different iteration steps in Figure 8. "Regression" and "Classification" in Figure denote the output of the VAE decoder and segmentation head, respectively. Our proposed LPLR leverages the up-sampling capability of a pre-trained VAE decoder in recycling manner. As model converges, the regression results transform from blurry to progressively clearer states, presenting more details compared to the classification results. This assists the segmentation head in producing more accurate semantic segmentation results.
Training time
Source RGB
Regression
Classification
Target Event
Regression
Classification
Figure 8: Visualization of the output of VAE decoder (Regression) and segmentation head (Classifi- cation).
A.2
Influence of Different Data Volumes
Table 6: Influence different data volumes testing the DSEC dataset [7].
Method
Baseline-10094
MADM-10%
MADM-25%
MADM-50%
MADM-100%
MloU
52.27
53.21
53.69
54.55
56.31
We train our method with 10%, 25%, and 50% of the total target samples in the event modality. Here, the "Baseline- 100%% column indicates the performance of the MADM model without DPLG and LPLR and trained on the whole target samples. The results Table indicate that our proposed MADM consistently outperforms the baseline across all tested data volumes. Additionally, our MADM robust and effective even when the dataset size relatively small.
A.3 Parameters and Costs
Table 7 presents detailed comparison of training timeper iteration, number of iterations, total training time, parameters, and performance across various methods in the DSEC event modality [7], including our MADM and its distilled variant.
While MADM does exhibit higher training time per iteration, the advanced visual prior derived from TIDMs necessitates fewer iteration for adaptation, presenting minimum total training time. Moreover, MADM achieves a substantial performance improvement, with an MIoU 57.34% surpassing other methods. Recognizing the trade-off in parameter count, we have leveraged our MADM model as a teacher to perform secondary self-training. This approach has enabled us to distill the knowledge embedded in MADM into more compact DAFormer model [17], MADM
14

---

Table 7: Comparison of parameters and costs.
Training/tter. (seconds)
Total training (hours)
Params (million)
Method
Iteration
MloU
DAFormer [17]
0.36
40k
4.0
85
33.55
PiPa [46]
1.12
60k
18.7
85
43.28
MIC [18]
0.48
40k
5.3
85
46.13
Rein [8]
1.25
40k
13,9
328
51,86
MADM
1.38
10k
3.8
949
56.31
MADM (Distilled)
0.46
10k
1.3
85
54.03
(Distilled), which retains highMIoU of 54.03% while significantly reducing parameters to 85M and only increasing the training time by 1.3 hours. Our distilled model demonstrates that it is possible maintain high performance with reduced computational costs, addressing the concerns raised regarding the parameters and efficiency of MADM.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of the paper provide clear and concise overview the research's main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We detail our work in the Methods section and describe implementation details in the Experiments section.
Justification:
Guidelines: The paper thoroughly details all the necessary components for reproducing the main experimental results.
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code as soon as our work is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wttps.cc/ppblccc//eccccccpcppppp geicc//gueee...e CoodeSummii agionPoiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars with overall run with given experimental conditions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eurrpp.ee/ee cc wctppbcic//aeeccccccccccccccclct htwp:/tthhicoeeeeeeeeeeeeeeeee
Answer: [Yes]
Justification:
Our research conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss broader impacts our work conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credite the creator and cite the original paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "fSk6c9oCsO", "reviewer": "Reviewer_x1By", "summary": "This paper proposes an interesting task that adapting image segmentation knowledge to other input modalities, such as depth, infra, and event. This is beneficial for applications at nighttime.", "questions": "1. Since the training dataset of TIDMs mainly consists of RGB images, why can TIDMs robustly extract features across modalities?\n2. What's the motivation behind the single-step diffusion operation (Line 146)?\n3. Why does injecting noise into the latent code produce more accurate pseudo labels?\n4. As stated in Line 28, taking depth or other modalities is valuable in nighttime perception. But there are no experiments showing this. For example, given a nighttime dataset, comparing the performance with the input of RGB images or depth images (adapted by MADM).", "limitations": "The limitations are briefly stated in Sec. 5.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The task is promising for applications at nighttime.\n2. The label palette is novel and can be generalized to more tasks.", "weaknesses": "1. The presentation needs to be improved. \n   - The meaning of \"Unsupervised Modality Adaptation\" is unclear from the introduction section.\n   - Although the authors try to illustrate why they use a pretrained Text2Image Diffusion Model (TIDMs), and why they propose DPLG and LPLR, some motivations are not well supported by experiments or other accepted papers. For example, in Lines 45-47, \"Although TIDMs are not trained on other visual modalities, their large-scale samples and unification through texts enable them to adapt to a broader distribution of domains.\" What is the meaning of \"a broader distribution of domains\"? Which prior is provided by TIDMs that results in this convenience?\n\n2. Lacking discussions on the motivation of critical components. For example, see the third question below. \n\n3. Experiments do not well support the potential applications (see question 4).\n\nTypos:\n1. Line 134, Sec.1 --> Fig. 1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**To Reviewer x1By**\n\nThank you for the insightful and positive comments. In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which further helps to solve the current issues.\n\n>**Q1**: *The meaning of \"Unsupervised Modality Adaptation\" is unclear from the introduction section.*\n\n**A1**: Thanks. We apologize for any confusion caused by the unclear explanation. To clarify, \"Unsupervised Modality Adaptation\" refers to the adaptation of a model from a labeled source image modality to an unlabeled target modality, such as depth, infrared, or event data. This concept was initially explained in lines 132-134 of the manuscript. To further address your concerns, we will provide a more detailed explanation when we first introduce this concept in the revision.\n\n>**Q2**: *Some motivations are not well supported by experiments or other accepted papers. What is the meaning of \"a broader distribution of domains\"? Which prior is provided by TIDMs that results in this convenience?*\n\n**A2**: Thanks. The phrase \"a broader distribution of domains\" refers to the model’s adaptability to different visual modalities beyond its primary design of generating RGB images from text. The prior we refer to is derived from the large-scale pretraining data on which TIDMs are trained. This extensive pretraining provides TIDMs with a robust understanding of high-level visual concepts, enabling their application to various domains, such as semantic matching [1], depth estimation [2], and 3D awareness [3]. As demonstrated in Table 2 of our manuscript, our baseline method performs comparably with SoTA techniques, highlighting the potential of leveraging this advanced visual prior for the Unsupervised Modality Adaptation (UMA) problem.\n\nHowever, two significant challenges remain. First, significant modality discrepancies hinder robust and high-quality pseudo-label generation during self-training. Second, TIDMs extract features in the 8x downsampling latent space which limit the acquisition of high-resolution features. To address these issues, we propose LPLR and DPLG. These methods are designed to mitigate the identified problems, as detailed in our manuscript.\n\nIn the revised version, we will include the above discussion to enhance the motivations. \n\n[1] SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching. In CVPR, 2024.\n\n[2] Repurposing Diffusion-based Image Generators for Monocular Depth Estimation. In CVPR, 2024.\n\n[3] Probing the 3D Awareness of Visual Foundation models. In CVPR, 2024.\n\n>**Q3**: *Why can TIDMs robustly extract features across modalities?*\n\n**A3**: Thanks. Through extensive pre-training on diverse and large-scale datasets, TIDMs have demonstrated a remarkable capacity to generate various objects with distinct attributes, such as cars of different types and colors. While TIDMs are not explicitly trained on other modalities, they can capture the fundamental characteristics of objects, such as shape and spatial orientation (e.g., a car's position on the road). These characteristics remain consistent across different modalities, enabling TIDMs to achieve high-level visual intelligence. As a result, TIDMs can identify and extract essential characteristics of objects even when encountering modalities that were not part of their pre-training data. This ability to generalize across modalities showcases the strength and versatility of TIDMs on feature extraction in various tasks.\n\n>**Q4**: *What's the motivation behind the single-step diffusion operation?*\n\n**A4**: Thanks. As evidenced by the experimental results in [1], a single diffusion step effectively removes noise for dense visual prediction. Additionally, single-step diffusion significantly reduces inference costs compared to multi-step diffusion. Considering these factors, we adopt the single-step diffusion operation, following the approach in [2, 3].\n\n[1] DDP: Diffusion Model for Dense Visual Prediction. In ICCV, 2023. \n\n[2] Unleashing Text-to-Image Diffusion Models for Visual Perception. In CVPR, 2023. \n\n[3] Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. In CVPR, 2023.\n\n>**Q5**: *Why does injecting noise into the latent code produce more accurate pseudo labels?*\n\n**A5**: In the pre-training of TIDMs, the objective is to estimate noise from latent inputs containing various noise levels. By injecting noise into the latent code, we effectively simulate this noisy distribution. This simulation aligns the latent space more closely with the data distribution encountered during the pre-training phase. Such alignment fosters a more robust and accurate semantic interpretation, which, in turn, enhances the quality of the pseudo labels generated. This shares similar spirits in other applications of diffusion models, such as the text-to-3D [1] where injecting extra noise into data can improve the denoising quality of image and yields better pseudo labels.\n\n[1] DreamFusion: Text-to-3D using 2D Diffusion. In ICLR, 2023.  \n\n>**Q6**: *Given a nighttime dataset, comparing the performance with the input of RGB images or depth images (adapted by MADM).*\n\n**A6**: The FMB-Infrared dataset includes both image and infrared modalities on daytime and nighttime scenes. We adapt from cityscapes with daytime RGB images to the nighttime image modality and infrared modality by our proposed MADA, respectively. The following table and Figure 1 in the attached pdf show that the infrared modality has a clear advantage in the \"Person\" class due to obvious thermal differences and a good suppression of light interference. We will include these results into the revision.\n\n|Modality|Sky|Build.|Person|Pole|Road|S.walk|Veg.|Vehi.|Tr.S.|MIoU(avg)|\n|-|-|-|-|-|-|-|-|-|-|-|\n|RGB|**88.85**|68.14|64.79|**25.80**|**89.09**|**32.43**|70.32|**84.13**|7.27|58.98|\n|Infrared|87.94|**82.40**|**82.69**|21.50|76.21|26.50|**76.61**|83.80|**16.69**|**61.59**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nbPKlyTrSD", "reviewer": "Reviewer_uNLv", "summary": "This paper proposes Modality Adaptation with text-to-image Diffusion Models (MADM). MADM leverages pre-trained text-to-image diffusion models to enhance cross-modality capabilities, comprising two main components: diffusion-based pseudo-label generation to improve label accuracy and a label palette with latent regression to ensure fine-grained features. Experimental results show MADM achieves SOTA performance across various modalities.", "questions": "N/A", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper is well-written.\n- Label Palette and Latent Regression are well-designed.\n- The ablation studies show the effectiveness of the method.", "weaknesses": "- The use of text-to-image Diffusion Models introduces higher training and inference costs, as well as an increase in model parameters. The authors need to discuss the fairness of these costs compared to other methods.\n\n- Overall, using text-to-image Diffusion Models for semantic segmentation is not novel, as many works have applied diffusion models for dense prediction tasks, and pseudo-label generation is also commonly used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**To Reviewer uNLv**\n\nThank you for the insightful and positive comments. In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which further helps to solve the current issues.\n\n>**Q1**: *Discuss the fairness of the costs compared to other methods.*\n\n**A1**: We appreciate your feedback on the computational costs of our proposed MADM. The following table presents a detailed comparison of training time per iteration, number of iterations, total training time, parameters, and performance across various methods in the event modality, including our MADM and its distilled variant. \n\nWhile MADM does exhibit a higher training time per iteration, the advanced visual prior derived from TIDMs necessitates fewer iteration for adaptation, presenting a minimum total training time. Moreover, MADM achieves a substantial performance improvement, with an MIoU of 57.34\\%, surpassing other methods. Recognizing the trade-off in parameter count, we have leveraged our MADM model as a teacher to perform a secondary self-training. This approach has enabled us to distill the knowledge embedded in MADM into a more compact DAFormer model, MADM (Distilled), which retains a high MIoU of 54.03\\% while significantly reducing parameters to 85M and only increasing the training time by 1.3 hours.\n\nOur distilled model demonstrates that it is possible to maintain high performance with reduced computational costs, addressing the concerns raised regarding the parameters and efficiency of MADM.\n\n|Method|Training time/Iter. (seconds)|Iteration|Total training time (hours)|Params (million)|MIoU|\n|-|-|-|-|-|-|\n|DAFormer|0.36|40k|4.0|85|33.55|\n|PiPa|1.12|60k|18.7|85|43.28|\n|MIC|0.48|40k|5.3|85|46.13|\n|Rein|1.25|40k|13.9|328|51.86|\n|MADM|1.38|10k|3.8|949|56.31|\n|MADM (Distilled)|0.46|10k|1.3|85|54.03|\n\n>**Q2**: *using text-to-image Diffusion Models for semantic segmentation is not novel, as many works have applied diffusion models for dense prediction tasks, and pseudo-label generation is also commonly used.*\n\n**A2**: Thanks. We address your concerns from two main perspectives.\n\n**1) Extension of TIDMs to UMA**: For the first time, this work uniquely extends TIDMs to the UMA task across a broader range of visual modalities, introducing a novel perspective to this domain. While TIDMs have been used in various dense prediction tasks, their application has predominantly been limited to the RGB modality. Applying TIDMs directly to our UMA problem encounters significant issues, i.e., unstable and inaccurate pseudo-labeling and the lack of fine-grained feature extraction, as illustrated in Figure 3 and Table 2. These challenges have not been explored, despite their significance. Our work pioneers the extension of TIDMs to unsupervised semantic segmentation in other visual modalities, paving a new path for TIDMs in the UMA problem.\n\n**2) Novel Techniques of DPLG and LPLR**: Our DPLG and LPLR techniques are novel, and effectively address or alleviate two severe issues that have not been explored before, yielding significant improvements over previous approaches.\n\nLPLR addresses the lack of fine-grained features in the UMA task, which has been entirely ignored by previous approaches. Latent-diffusion models pre-trained solely on continuous RGB images cannot handle discrete segmentation labels well. So previous methods [1, 2] discard the VAE decoder and use an additional classifier for features from the low-resolution latent space. However, the spatial size of the low-resolution space is much smaller than the original input size, leading to significant loss of spatial details crucial for semantic segmentation tasks. Our LPLR converts discrete labels into the RGB format, allowing them to be regressed by the VAE decoder to achieve high-resolution features and finer detail. Table 3 shows that LPLR achieves a significant +1.85\\% average improvement across all modalities over previous methods.\n\nDPLG addresses the issue of noisy and unstable pseudo-label generation in previous approaches like [3]. The significant distribution gap between images and other modalities results in noisy and inaccurate vanilla pseudo-labels generated by self-training methods as shown in Figure 5 and Table 3. DPLG utilizes the unique task property by injecting a certain amount of noise into the target modality for accurate pseudo-label generation. This injection aligns the latent space more closely with the data distribution encountered during the pre-training phase, fostering more robust and accurate semantic interpretation and pseudo-label generation. Figure 5 and Table 2 demonstrate that DPLG improves vanilla pseudo-label generation by a significant +3.39\\% on the infrared dataset.\n\nWe appreciate your thoughtful comments and will include these clarifications and results in the revision.\n\n[1] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, Ping Luo. DDP: Diffusion Model for Dense Visual Prediction. In ICCV, 2023. \n\n[2] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, Jiwen Lu. Unleashing Text-to-Image Diffusion Models for Visual Perception. In CVPR, 2023. \n\n[3] Lukas Hoyer, Dengxin Dai, Luc Van Gool. DAFormer: Improving Network Architectures and Training Strategies for Domain-adaptive Semantic Segmentation. In CVPR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TBxLRHSiUW", "reviewer": "Reviewer_4TqZ", "summary": "This paper introduced text-to-image diffusion models to enhance the generalization across different modalities. The proposed MADM includes two key components: Label Palette and Latent Regression (LPLR) and Diffusion-based Pseudo-Label Generation (DPLG). This method alleviates issues related to pseudo-labeling instability and low-resolution features extraction within TIDMs. Experimental results show that MADM achieves state-of-the-art results across three different modalities.", "questions": "I like the idea of utilizing  text-to-image diffusion models to enhance the generalization across different modalities. However, the significant time consumption and absence of experiments testing different data volumes lead me to downgrade the recommendation.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic is interesting and valuable.By leveraging the powerful pre-trained Text-to-Image Diffusion Models, the method effectively reduces discrepancies across different modalities (e.g., image, depth, infrared, and event). \n2. The writing style is concise and easy to understand, and the paper is logically clear and well-organized.", "weaknesses": "1. The proposed modules designed of the model is commonly used in generative fields, such as DPLG which converts labels into latent space and DPLG which utilizes the extracted features from TIDM to generate pseudo-labels, thus lacking novelty. \n2. The approach is time-consuming, and there is no complexity analysis in Table 1. \n3. The paper does not discuss the method's performance with different data volumes. The size of the dataset often significantly impacts performance improvement. It is recommended to test with varying data volumes (e.g., from 500 to 10,000) across different modalities to validate the enhancement and discuss the results, which would be beneficial for practical applications. \n4. Minor issues: It should be Table 2 in Lines 260 and 267, Page 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**To Reviewer 4TqZ**\n\nThank you for the insightful and positive comments. In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which further helps to solve the current issues.\n\n>**Q1**: *The proposed modules lack novelty.*\n\n**A1**: Thanks. We address your concerns from two main perspectives.\n\n**1) Extension of TIDMs to UMA**: For the first time, this work uniquely extends TIDMs to the UMA task across a broader range of visual modalities, introducing a novel perspective to this domain. While TIDMs have been used in various dense prediction tasks, their application has predominantly been limited to the RGB modality. Applying TIDMs directly to our UMA problem encounters significant issues, i.e., unstable and inaccurate pseudo-labeling and the lack of fine-grained feature extraction, as illustrated in Figure 3 and Table 2. These challenges have not been explored, despite their significance. Our work pioneers the extension of TIDMs to unsupervised semantic segmentation in other visual modalities, paving a new path for TIDMs in the UMA problem.\n\n**2) Novel Techniques of DPLG and LPLR**: Our DPLG and LPLR techniques are novel, and effectively address or alleviate two severe issues that have not been explored before, yielding significant improvements over previous approaches.\n\nLPLR addresses the lack of fine-grained features in the UMA task, which has been entirely ignored by previous approaches. Latent-diffusion models pre-trained solely on continuous RGB images cannot handle discrete segmentation labels well. So previous methods [1, 2] discard the VAE decoder and use an additional classifier for features from the low-resolution latent space. However, the spatial size of the low-resolution space is much smaller than the original input size, leading to significant loss of spatial details crucial for semantic segmentation tasks. Our LPLR converts discrete labels into the RGB format, allowing them to be regressed by the VAE decoder to achieve high-resolution features and finer detail. Table 3 shows that LPLR achieves a significant +1.85% average improvement across all modalities over previous methods.\n\nDPLG addresses the issue of noisy and unstable pseudo-label generation in previous approaches like [3]. The significant distribution gap between images and other modalities results in noisy and inaccurate vanilla pseudo-labels generated by self-training methods as shown in Figure 5 and Table 3. DPLG utilizes the unique task property by injecting a certain amount of noise into the target modality for accurate pseudo-label generation. This injection aligns the latent space more closely with the data distribution encountered during the pre-training phase, fostering more robust and accurate semantic interpretation and pseudo-label generation. Figure 5 and Table 2 demonstrate that DPLG improves vanilla pseudo-label generation by a significant +3.39% on the infrared dataset.\n\nWe appreciate your thoughtful comments and will include these clarifications and results in the revision.\n\n[1] DDP: Diffusion Model for Dense Visual Prediction. In ICCV, 2023. \n\n[2] Unleashing Text-to-Image Diffusion Models for Visual Perception. In CVPR, 2023. \n\n[3] DAFormer: Improving Network Architectures and Training Strategies for Domain-adaptive Semantic Segmentation. In CVPR, 2022.\n\n>**Q2**: *The approach is time-consuming.*\n\n**A2**: We appreciate your feedback on the computational costs of our proposed MADM. The following table presents a detailed comparison of training time per iteration, number of iterations, total training time, parameters, and performance across various methods in the event modality, including our MADM and its distilled variant. \n\nWhile MADM does exhibit a higher training time per iteration, the advanced visual prior derived from TIDMs necessitates fewer iteration for adaptation, presenting a minimum total training time. Moreover, MADM achieves a substantial performance improvement, with an MIoU of 57.34\\%, surpassing other methods. Recognizing the trade-off in parameter count, we have leveraged our MADM model as a teacher to perform a secondary self-training. This approach has enabled us to distill the knowledge embedded in MADM into a more compact DAFormer model, MADM (Distilled), which retains a high MIoU of 54.03\\% while significantly reducing parameters to 85M and only increasing the training time by 1.3 hours.\n\nOur distilled model demonstrates that it is possible to maintain high performance with reduced computational costs, addressing the concerns raised regarding the parameters and efficiency of MADM.\n\n|Method|Training time/Iter. (seconds)|Iteration|Total training time (hours)|Params (million)|MIoU|\n|-|-|-|-|-|-|\n|DAFormer|0.36|40k|4.0|85|33.55|\n|PiPa|1.12|60k|18.7|85|43.28|\n|MIC|0.48|40k|5.3|85|46.13|\n|Rein|1.25|40k|13.9|328|51.86|\n|MADM|1.38|10k|3.8|949|56.31|\n|MADM (Distilled)|0.46|10k|1.3|85|54.03|\n\n>**Q3**: *The paper does not discuss the performance with different data volumes. It is recommended to test with varying data volumes to validate the enhancement and discuss the results.*\n\n**A3**: Thanks. Per your suggestion, we train our method with 10%, 25%, and 50% of the total target samples in the event modality. Here, the \"Baseline-100%\" column indicates the performance of the MADM model without DPLG and LPLR and trained on the whole target samples. The results in the following table indicate that our proposed MADM consistently outperforms the baseline across all tested data volumes. Additionally, our MADM is robust and effective even when the dataset size is relatively small. We will include these results into revision.\n\n|Method|Baseline-100%|MADM-10%|MADM-25%|MADM-50%|MADM-100%|\n|-|-|-|-|-|-|\n|MIoU|52.27|53.21|53.69|54.55|56.31|\n\n>**Q4**: *Minor issues: It should be Table 2 in Lines 260 and 267, Page 8.*\n\n**A4**: Thanks for your carefully proofread. We will fix this typo in revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8HZVMqzei", "reviewer": "Reviewer_Qg6E", "summary": "The authors extend the traditional image domain adaptation to modality adaptation for unsupervised semantic segmentation in real-world multimodal scenarios. They use text-to-image diffusion model with strong generalization capabilities and proposed Diffusion-based Pseudo-Label Generation (DPLG) and Label Palette and Latent Regression (LPLR) to correct the pseudo-labels and obtain high-resolution features. SOTA performance on depth, infrared, and event modalities proves the effectiveness of the method.", "questions": "1.\tAdd the LPLR visualization results and analyze them.\n2.\tDetails of the framework implementation need to be supplemented.", "limitations": "1.\tAdd the LPLR visualization results and analyze them.\n2.\tDetails of the framework implementation need to be supplemented.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThe paper is well-organized and easy to understand.\n2.\tFor the first time, the authors extend the adaptation between image domains to the adaptation between modalities.\n3.\tThe proposed MADM significantly outperform existing methods in the adaptation of three different modalities.\n4. \tFigures 3 and 5 clearly demonstrate the effect of DPLG. They visualize the influence of noise-adding on the generation of pseudo-labels.", "weaknesses": "1.\tThe paper lacks the visualization results obtained by feeding potential features into the VAE Decoder. These results will provide a better understanding of LPLR.\n2.\tThe specific layers of the three multiscale features extracted from the UNet decoder are not clearly stated in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**To Reviewer Qg6E**\n\nThank you for the insightful and very positive comments. In the following, we provide our point-by-point response and hope our response helps address your concerns. We also look forward to the subsequent discussion which further helps to solve the current issues.\n\n>**Q1**: *Add the LPLR visualization results and analyze them.*\n\n**A1**: Thank you for your suggestion. We have visualized LPLR under different iteration steps in Figure 2 of the attached PDF. \"Regression\" and \"Classification\" in Figure 2 denote the output of the VAE decoder and segmentation head, respectively. Our proposed LPLR leverages the up-sampling capability of a pre-trained VAE decoder in a recycling manner. As the model converges, the regression results transform from blurry to progressively clearer states, presenting more details compared to the classification results. This assists the segmentation head in producing more accurate semantic segmentation results. We will include these results in the revision.\n\n>**Q2**: *The specific layers of the three multiscale features extracted from the UNet decoder are not clearly stated in the paper.*\n\n**A2**: Thanks for your carefully proofread. We extract the multiscale features from the denoising UNet decoder at the outputs of the 5th, 8th, and 11th blocks. We will clarify them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all of the four reviewers for the valuable and constructive feedback provided on our manuscript. Your insights have been instrumental in enhancing the quality and clarity of our work.

In the attached PDF, we include additional visualizations as requested by Reviewers x1By and Qg6E, which we believe will clarify our experimental findings.

Thank you for your valuable time and insights. We are open to further discussion and look forward to your continued guidance.

Meta-review: Dear authors,

This draft has received 1 Accept, 1 Weak Accept, and 2 borderline. One of the reviewer has expressed. concerns regarding After carefully reviewing the comments, the draft is being accepted for publication. Authors are encouraged to update the draft according to the suggestions and concerns mentioned by the reviewers.

For example, 
1) As concern raised by x1By, "Although TIDMs are not trained on other visual modalities, their large-scale samples and unification through texts enable them to adapt to a broader distribution of domains" ----Needs proper citation or experimental verification. Reviewer x1By  further raised the concern that SD4Match (and others) referenced in rebuttal do not properly answer the question regarding TIDM never trained on other modalities e.g. Depth can still adapt to Broader Distribution of Domain. We will encourage authors to answer this question properly in the final draft. 

2) Qg6E's Q1 should at least be in supplementary and Q2 should be adjusted in the main draft. 

3) Improve the draft to clarify novelty (4TqZ) and concerns raised by other reviewers. 

4) Authors have indicated the Code will be release,  we are taking this as that both the Training and Evaluation code will be released (along with the experimental settings used in the paper), when the paper gets accepted. This is vital for the community to reproduce the results and do the comparisons. 

Congratulations.

regards
AC
Decision: Accept (poster)