OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
Authors: Bo-Wen Yin
Jiao-Long Cao
Xuying Zhang
Yuming Chen
Ming-Ming Cheng
Qibin Hou
Venue: NeurIPS 2025 poster ()
Keywords: Semantic Segmentation
Multi-modal
Representation Learning
Primary area: applications

Abstract: Recent research on representation learning has proved the merits of multi-modal clues for robust semantic segmentation. Nevertheless, a flexible pretrain-and-finetune pipeline for multiple visual modalities remains unexplored. In this paper, we propose a novel multi-modal learning framework, termed OmniSegmentor. It has two key innovations: 1) Based on ImageNet, we assemble a large-scale dataset for multi-modal pretraining, called OmniSegmentor, which contains five popular visual modalities; 2) We provide an efficient pretraining manner to endow the model with the capacity to encode different modality information in the OmniSegmentor. For the first time, we introduce a universal multi-modal pretraining framework that consistently amplifies the model's perceptual capabilities across various scenarios, regardless of the arbitrary combination of the involved modalities. Remarkably, our OmniSegmentor achieves new state-of-the-art records on a wide range of multi-modal semantic segmentation datasets, including NYU Depthv2, EventScape, MFNet, DeLiVER, SUNRGBD, and KITTI-360. Data, model checkpoints, and source code will be made publicly available: https://github.com/VCIP-RGBD/DFormer.

OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation
Bo-Wen Yin Jiao-Long Cao 1.2 Xuying Zhang 1.2, Yuming Chen 1.2
Ming-Ming Cheng! Qibin Hou" 1:21
NKIARI, Shenzhen Futian
VVCIP, College of Computer Science, Nankai University 'Corresponding author
bowenyin@mail. nankai edu. cn
Evest
Figure I: Visualizations of our assembled ImageNeXt dataset. Built upon ImageNet [44], widely used large-scale RGB classification dataset, IeageNeXt composed of five popular visual modalities for each sample, including RGB, Depth, LiDAR, Thermal, and Event.
Abstract
Recent research on representation learning has proved the merits of multi-modal clues for robust semantic segmentation. Nevertheless, flexible pretrain-and- finetune pipeline for multiple visual modalities remains unexplored. In this paper, we propose a novel multi-modal learning framework, termed OmniSegmentor. It has two key innovations: 1) Based on ImageNet, we assemble a large-scale dataset for multi-modal modal pretraining, called ImageNeXt, which contains five popular visual modalities; 2) We provide an efficient pretraining manner to endow the model with the capacity to encode different modality information the ImageNeXt. For the first time, we introduce a universal multi-moda, pretraining framework that consis- tently amplifies the model's perceptual capabilities across various scenarios, regard- Om-
less of the arbitrary combination of the involved modalities. Remarkably, niSegmentor achieves new state-of-the-ar records on a wide range of multi-modal semantic segmentation datasets. including NYU Depthv2, EventScape, MFNet, DeLiVER, SUNRGBD, and KITTI-360. Data, model checkpoints, and source code will be made publicly available: https://gitub..com//CC Dttpr/ttiw.ccmmccccRGGGDDDDDmmm
Introduction
With the widespread use of modular sensors, multi-mode data for semantic segmentation is becoming more and more accessible. The knowledge perceived from multi-mon modal data can achieve more robust
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

scene understanding, facilitating multi-modal learning research on a series of vision tasks. However, existing works [71, 72, 10, 58] usually employ RGB pretrained or randomly initialized weights to process different modalities, leading to mismatched encoding of the data [3]. Recent work DFormer [66] attempts to solve this issue using new pretraining manner on the modality-specifff scenes, L.e. RGB-D. Considering the current trend of fusing more and more modalities [6. 72, 58), it would be of great interest to explore a flexible and efficient pretrain-and- finetfintuue framework for multi-modal data, which was seldom researched before.
To construct such a flexible and efficient framework, several prominent problems should be considered. The foremost challenge is that muuti--mdal pretrain- ing requires a large-scale dataset containing vari- ety of visual modalities. Although some existing datasets [18, 24, 34, 72, 48, 49] can partially satisfy this requirement, they either concentrate on a spe- cific modality besides RGB images have a limited scale of training samples, making them unsuitable for multi-modal pretraining. In addition, when the types of visual modalities increase, how to efficiently perform multi-modal pretraining and how to flexibly deploy the pretrained weights to downstream tasks with different types of visual modalities are still open questions.
Bi-modal
CMNN6X
DYUDegth
HREPser
457.6
MPNet
$669
KJTTI-660
45.0
RGB-E
55.6
meaisaatte (RGEDAD
RGB-E
RGB-D
66.7
DELIVER (RGB-D-EA))
ROB-D
67A
66
RGB-D-F
65.5
RGB-D-L
RGB-D-E-L
Taking the above analysis into account, in this paper, we attempt to construct a flexible and ef- ficient pretrain-and-fineturr framework for multi- modal semantic segmentation, named OmniSegmen- tor. Firstly, we need to address the issue of lacking large-scale
Figure 2: Performance comparisons between our OmniSegmentor and recent statc-of-the-art meth- ods (e.g. HRFuser [6] and CMNcXt [72]) on var- ious multi-moda semantic segmentation bench- marks. 'D, E, L. T are abbreviations for depth, event, LiDAR, and the thermal modalities.
training data. Some meth- ods [66, 3, 63] discover that synthetic data can com- pensate for this deficiency and improve the capacity of the model. For instance, DFormer [66] and DepthTrack [63] utilize synthetic depth data perform multi-modal pretraining, thereby avoiding the mismatch between RGB pretrained models and RGB- D data and bringing significant improvement in RGB-D segmentation and tracking, respectively. Inspired by these works, we build a large-scale multi-modal dataset with synthetic data, called ImageNeXt, as shown in Fig. 1, to address the data issue and make the joint multi-modal pretraining feasible. This dataset is built upon ImageNet [44] and supplements each RGB image with four additional visual modalities, i.e. depth, thermal. LiDAR, and event. We empirically found that the assembled dataset can help the model learn strong visual representations during pretraining.
Given the large-scale multi-modal data, the next challenge is to present efficient method for multiple modalities. However, our experiments reveal that simultaneously pretraining a unified model on all the modalities not only imposes considerable computational burdens but also leads to optimization difficulties. To better accommodate the multi-modal data, we design a novel pretrain- and-finetune pipeline that can achieve efficient pretraining and flexible finetuning. To be specific, during pretraining, instead of simultaneously inputting all types of modality data at each iteration, we propose feeding the RGB data and randomly selected other modality data into the model and conducting feature alignment. This simple strategy enables the model to efficiently absorb the patterns from different modality data, thus avoiding the mismatch problem between pretraining on RGB and finetuming on multi-modal data. Moreover, the training efficacy can be largely improved. For finetuning on downstream tasks, the weights corresponding to the supplementary modality in the pretrained model are used to initialize the weights for each supplementary modality. This approach allows each modality to processed separately within each building block, thereby providing diverse informative features from different types of modality data for semantic segmentation. By adding a lightweight decoder head to the top of the ImageNeXt pretrained model, OmniSegmentor can generate high-quality predictions for different multi-modal segmentation tasks.
To the best of our knowledge, we are the first construct flexible pretrain-and-fintue pipeline for semantic segmentation with increasing visual modalities, i.e., OmniSegmentor, composed of the ImageNeXt, well-designed pretraining and finetuning method. Extensive experiment results
2

---

demonstrate the effectiveness of OmniSegmentor on the benchmarks of a wide range of multi- modal semantic segmentation tasks, including NYU Depthv2 [48]. EventScape [18], MFNet [24], DeLiVER [72], SUNRGBD [49], and KITTI-360 [34]. As shown in Fig. 2, our OmniSegmentor achieves new state-of-the-an records across all settings on all benchmarks. We hope that this work will provide new insights for multi-modal representation learning and set new baselines for multi-modal semantic segmentation.
Related Work
2.1 Multi-Modal Semantic Segmentation
Recently, significant advancements in semantic segmentation have been made as the rise of deep learning technologies, typified by CNNs [38, 26, 27, 33] and Transformers [39, 54, 15, 5]. However, most methods still struggle to cope with real-world scenes, as they only focus on processing RGB images [74, 51, 23, 36, 65], which lack sufficient information from other visual modalities, like LiDAR and depth. Multi-modal semantic segmentation has been explored by harvesting comple- mentary information from supplementary modalities, such as depth [71, 9], thermal [73, 61, 47], LiDAR [64], and event [2, 70]. A series of methods are proposed to utilize the characteristics within other modalities for more robust semantic segmentation. CMX [71] addresses multi-modal segmentation through multi-level cross cross-modal interactions, including channel and token exchanges. CMNeXt [72] introduces universal multi-modal semantic segmentation framework with arbitrary modal complements. However, most existing relevant methods [58, 71, 72] employ RGB pretrained or randomly initialized weights to process the supplementary modalities, which may not fully extract the specific characteristics of each modality. To address this issue, DFormer [66] proposes pretrain the encoder with RGB-D data to better leverage depth cues and alleviate the mismatch problem between pretraining and finetuning. Its significant improvement in both efficiency and effectiveness also emphasizes the importance of solving the mismatched encoding. However, DFormer is modality- specific (RGB-D) DD and is difficult to be applied other modalities. Beyond the aforementioned works, we aim to provide flexible and efficient pretrain-and -ad---nnndtnnnnn framework that can efficiently perform multi-modal pretraining and flexibly deploy the pretrained weights to various downstream tasks.
2.2 Multi-Modal Representation Learning
Multi-modal representation learning endows models with the capacity to establish the relations among the specific information from multiple signal sources. The learned transferable representations can yield remarkable performance across various downstream tasks, as demonstrated in previous works [43, 67, 32]. Existing multi-modal learning methods encompass a large number of modalities, including image-text [8, 11, 37, 43, 75, 60), text-video [1], image-depth [20, 3], and image-text- audio [77, 21], etc. Structurally, these methods can be categorized into two types. The first type of method adopts separate encoders. They exploit multiple encoders to independently project the inputs of different modalities into a common space and minimize the distance between/among them or perform feature fusion. For instance, CLIP [43] employs two individual encoders to encode the image-text pairs and align them via contrastive learning. The second type of method adopts unified encoders to encode different modalities individually or multiple modalities jointly. Typically, Omnivore [20] and cectatrranffrmmr [77] are able to process different modalities separately, while DFormer [66] and MultiMAE [3] can simultaneously deal with two visual modalities, i.e RGB and depth. However, the former cannot establish connections between different modalities, and the latter is limited to specific kinds of multi-modal data. An important reason that limits the development of multi-modal representation learning the lack of a targe-scale modil dataset. For example, multi-modal datasets SUNRGBD [49] contains 10,325 RGB-D data, and KITTI-360 [34] has 61,280 RGB-L data, which are relatively small-scale and limited to specific multi-modal data, i.e. RGB-L or RGB-D. Taking the above analysis into account, in this paper, we provide a large-scale dataset and novel pretrain-and-fin ann-finetune framework, enabling supervised pretraining on five types of modal data and flexible finetuning on downstream tasks.
3

---

3 ImageNeXt Dataset
Building upon the ImageNet dataset, the assembled ImageNeXt is large-scale dataset for multi- modal representation learning. To the best of our knowledge, it is the first attempt to cover as many popular visual modalities as possible, including RGB, depth, thermal, LiDAR, and event. Unless otherwise specified, the ImageNet dataset in this paper refers to the original ImageNet- I [44]. As the ImageNet dataset, the sample numbers of the training set and the validation set of ImageNeXt are 1.2M and 50K, respectively. We will describe some details of each visual modality data.
RGB. RGB images are the foundational visual modality in computer vision research. It contains in- formation about objects' color, texture, shape, surroundings, etc. The RGB images in our ImageNeXt come from the ImageNet dataset [44], which is one of the most popular large-scale image datasets so far.
Depth. Depth maps provide 3D geometry information about range, position, and object contours. Combining RGB and depth enhances the ability to distinguish objects with similar colors and textures, especially when they occupy different spatial locations [71, 72j. Following DFormer [66], we employ popular depth estimation method, i.e., Omnidata [16], to produce depth maps for all the images in our ImageNeXt.
Event. Event data offers numerous advantages, including a high dynamic range, excellent temporal resolution, and immunity to motion blur. These qualities are crucial in dynamic scenarios with motion-related information, such as driving and flying scenes. The N-ImageNet [29] dataset acquires event data from an event camera that observes monitor-displayed images from ImageNet. We follow this work and employ the samples in N-ImageNet as the event data for the ImageNeXt.
LiDAR. LiDAR cameras can furnish dependable and precise spatial-depti information about the physical environment. Following the recent methods like CMX [71] and DeLiVER [72], we adopt the widely-used pseudo-LiDAR generation method [56] to generate the LiDAR data based on our synthetic depth maps of ImageNet. To maintain consistency between the LiDAR data and the RGB images in terms of representation, we adhere to the approach used in [83], which involves transforming LiDAR data into a format resembling range-view image.
The thermal sensor can detect temperature differences on the surface of objects, making very suitable for finding thermally concealed objects or detecting temperature anomalies. It does not rely on visible light, but rather the infrared radiation emitted by objects. According to our investigation, there is no method for thermal image estimation. Thus, we train a thermal estimation model, which imitates the depth estimation method adabins [4], on four RGB-T datasets VT821 [55], VT1000 [52], VT5000 [53], and FLIR [22]. Then we use it to generate the thermal data.
Thermal.
OmniSegmentor
4.1 Efficient Multi-Modal Pretraining
Multi-moda pretraining needs align different modal features and build interaction among them, making it challenging to optimize and time-consuming. Existing works [58, 71, 28] mostly attempt to finetune the RGB pretrained backbone for the multi-modal scenes, as shown on the left of Fig. 3. However, the pretrained backbone for downstream task finetuning is often trained on RGB images, which is inconsistent with the multi-modal input data during finetuning. This may cause representation distribution shifts in that the multi-modal data is not considered during pretraining, and the RGB pretrained backbone may not effectively extract the special information within the supplementary modalities. We aim to explore multi-modal pretraining manner to alleviate this issue by leveraging the proposed ImageNeXt dataset.
Given the ImageNeXt dataset where each sample has five modalities and a classification label, a straightforward way implement mudti-modal pretraining is to perform the classification optimization on all modalities simultaneously, as shown in the middle part of Fig. 3. This paradigm is also adopted in [76, 59], which uses sppelit encoders to process multi-modal images. Under this setting, each visual modality needs to be encoded independently, and the interaction will be performed between the RGB images and each supplementary modality. However, such a pretraining method yields considerable computational cost, greatly decreasing the pretraining efficiency. More

---

Smallsreess Pretratinn
Plostuning
Imsgehck Pretraining
Fiastuning (Duza))
RGB Cody
Bralitng Blees
All the Modalttes
Bullingg Biock
Sardanattay Moddlies
Basl Bjock
Frozehin
Kodditee Stared
Modallyy Spette
Bankenly
Eacadeg
intidio
indiaioo
Modalit Spectfc
3CE
Mocaltp-Speclle Fanding
Siddaity Specit:
Exodig
Bodc Visio
2100ding
AI de Nodiiis
Salling Black
All Modslites
Bullan Blck
Sapphenedary Xidddine
Interned
Figure 3: Illustration for different pretraining manners. The corresponding finetuning manner is also included. Left: RGB-only pretraining, Middle: simultaneous pretraining on all the modalities; Right: ImageNeXt pretraining of our OmniSegmentor. We omit the classification and segmentation heads for simplicity. The classification accuracy of the three manners is calculated with RGB input, all the modalities input, and the average on the RGB and each supplementary modality settings, respectively.
importantly, observe that the above joint pretraining manner makes the optimization process difficult. As shown in Fig. 4, the training curve cannot converge well, and the Top-1 accuracy on ImageNet greatly decreases compared pretraining on only RGB images. This issue also exists the downstream multi-modal segmentation tasks as shown in Fig. 6.
To alleviate the above issue and meanwhile improve pretraining efficiency, we propose an efficient multi-modal pretraining manner, called ImageNeXt pretraining. Instead of optimizing the model with all the modality data simultancously, RGB images and a randomly selected supplemen- tary modality
Training Loss
Only RLB
Ours
Simu tanetus
method takes
input. This paradigm is inspired by [20, 77, 21]. which uses single encoder to en- code different modalities. Considering that process- ing RGB images is the primary factor affecting 2D semantic segmentation accuracy [3. 25, 66), we argue that assigning less computational load to supplemen- tary modalities compared RGB can lead to better performance and computation trade-offs. To imple- ment this, we adopt an existing popular architecture, DFormer [66], which is originally designed for pre- training on RGB-D data. DFormer performs the si- multaneous fusion of RGB and depth features from global and local views, as shown the left part of Fig. 5. Meanwhile, it uses base module to preserve the diverse appearance information within the RGB features. We find that such block design can adapt well to our pretraining manner, though it is originally designed for RGB-D data.
50
100
150
250
300
Top-I Accuracy
80
60
Ours
Only RGB
5
250
Epochs
Pretrain
Param
Flops
Top-1 (%)
Time (h)
RGB
39.0M
14.7G
81.4
69.5
Simul
48.7M
21.8G
79.9
180.5
Ours
39.0M
14.7G
83.0
78.9
Figure 4: Curves of different pretraining man-
ners on ImageNeXt. 'RGB': pretraining RGB and removing the modality fusion op- eration; 'Simul': simultaneously pretraining on all the modalities; 'Ours': our ImageNeXt pretraining.
Our pretraining strategy offers the following advantages. First, each supplementary modality partic- ipates in the pretraining process. This makes the interaction between the RGB images and all the supplementary modality data efficient and hence can avoid the negative influence of other modalities on the representations of RGB images as much as possible. We empirically found that this strategy also improves multi-modal encoding efficacy during finetuning on downstream tasks. Besides, the training process can be largely sped up. Particularly, compared to pretraining with all modality data as input, our strategy brings 3.1 % gains Top-1 precision shows consistent improvements in various downstream tasks in Fig. 6.
5

---

Xna
DOwer Lode Depto REB
Global
PEDD
Global
Local
Deght
Addioo Molooe&&&Noon &elen
Loodl
Liter
X
Enest
Buiding B Block for Pretraining
Building Block for Finetuning
Figure 5: Building block of our OmniSegmentor. During pretraining. fusion modules aggregate the RGB features, and the features of the chosen modality, and the separate MLPs encode the features of different modalities. During finetuning. the sum of the features of supplementary modalities is fused with RGB features, and the features of different modalities are encoded separately by different MLPs.
4.2 Flexible Multi-Modal Finetuning
Given the pretrained model as described above, how to apply it to downstream tasks with multiple modality data as input is also important. The pretraining architecture the left part Fig. 5 is only suitable for processing data with an RGB image and supplementary modality and is difficult to use for multiple supplementary modalities. Here, we present a flexible finetuning strategy and explain how to load the pretrained weights to initialize the model for downstream tasks.
During pretraining. the architecture adopts the modality-shared encoding for different supplementary modalities to efficiently absorb the patterns. Differently, during finetuning, we need to utilize all the provided modalities to perform robust semantic segmentation. In this situation, modality-specifff encoding can better extract the unique characteristics within each supplementary modality, enabling the model to focus on different perspectives of the given scene. To achieve this, we use separate stem layers and MLPs for different modalities to implement mosality-specific encoding, and the resulting features for different supplementary modalities are aggregated and then utilized to enhance RGB features, shown in the right part of Fig. 5. Compared the pretraining process, the model for finetuning has extra stem layers and MLPs to extract the characteristics within different supplementary modalities. The extra stem layers and MLPs are initialized by the pretrained stem layer and MLPs for the supplementary modality, while the other modules directly load the pretrained weights. We will describe the pipeline of OmniSegmentor for the finetuning in the following.
Given the RGB image and supplementary modalities, we first use different stem layers separately process the input modalities. Then, the resulting features of different modalities are fed into the hierarchical encoder to encode multi-scale features. In each block, we first adopt an addition operation followed by layer normalization to aggregate the information of all the supplementary modalities, and the aggregated feature denoted as X "999 The number of supplementary modalities can be arbitrary. Here, empirically found that more sophisticated fusion module will not bring further improvement compared to the above simple fusion operation, and the details are shown in Tab. 3. Then we fuse the RGB feature Xrab and the aggregated feature Xe09 generate the enhanced feature Xenh as follows as DFormer. After the fusion of modality clues, we add the Xenh to the features of each modality the output for each modality. The resulting features for each modality are processed by separate MLPs and then sent to the next block. For the decoder during finetuning. following DFormer, we adopt the Ham head [19] the decoder. As result, the OmniSegmentor able to serve as an encoder for multi-modal segmentation tasks that input different modalities.
Experiments
5.1 Experiment Setup
To validate the effectiveness of our OmniSegmentor, we conduct extensive experiments on six popular
multi-modal segmentation datasets, including NYU Depthv2 [48], SUNRGBD [49], MFNet [24],
KITTI-360 [34], EventScape [18], and DeLiVER [72]. The experiments are conducted on NVIDIA A40 GPUs. The models are optimized using the cross-entropy loss function and the AdamW [30] method, where the learning rate is initialized to 6e-5 and scheduled by the poly strategy. The images
augmented by random resize with ratio of 0.5 to 1.75, random horizontal flipping, and random

---

Table 1: Results on multimodal semantic segmentation datasets. 'D, E. L, T are abbreviations for depth, event, LiDAR and thermal modalities, respectively. Follwoing [71, 72, 66], we adopt multi-scale inference (a), (b), and single-scale inference in (c)-(f)
(a) NYUDepth V2 (RGB-D) D) [48]. (b) SUNRGBD (RGB-D) [49].
(d) KITTI- 360 (RGB-L) L) [34].
Method
Backbone
mloU (%)
Method
Backbone
mloU (%)
Method
Backhone
mioli (%)
3DGNN [42]
VGG-16
43,1
CEN(ST]
ResNes-101
50
HRFuser [6]
HRPhemen-T
CFN [35]
RelineNet-152
47.7
TokenFusion[SS
M2T-B2
50,3
[S3]
SalaNeet
54.5
ACNot [28]
Omnivore [20)
ResNer-50
Swin-T
49.7
48.3
PGDENe([s]]
CEN[57]
TokenFusio[]sS]
ResNet-152 52
ResNet:34
MiT-B3
51.0
51.1
$1.0
TokenFisium [58]
CMX
Trand user [4]]
KegNetY
MiF--2
MIT.8
64.1
$4.6
56.5
RD3-152 [40]
ResNet-152
50.1
MultMAB
VIT-B
CMNcXI [72]
MIT
05.3
ESANe(155]
ResNet-34
50.3
FRNa(R2]
ResNet-34
DRomer [66]
Divemae
66.3
EMSANNe[46]
R4
51.0
CMNeXI [72]
McT-84
51.9
OmnSSgmetorr
MiF82
67.8
SGNNet
ResNet-101
$1.1
CMX[71]
MM
52.1
OrmiSepmetor
Dieemesd
DFormer [66]
ShapeConw [7]
RcsNext-101
Diormer:T
51.3
51.1
CMX [7]]
DFermer [66]
Diamser-L
MTT-B5
52.4
52.5
(c) EventScape (RGB- D-B) [18).
CEN [57]
ResNet-101 101
51.7
OmniSepmente
ReaNet-101
51.7
Mathol
Modal
Backhone
milo
NANet [69]
ResNet-101
52.3
OmmiSegmente
MiT-82
52.0
SA SA-Giate [10]
ResNet-101
52.4
OmmiSegmentor
DRomee--
HRFusss [6]
RGB-E
HRRoencsT
CEN [57]
Omnivvore [20]
ResNet-152
Swin-S
52.5
52.7
(c) MFNet (RGB-T) [24].
CMX
CM
RGB E
RGN-H
MIT
Mrli-12
62.1
61.9
InkenPasion[[58) S8]
MiT-B2
53.3
Method
Backbone
mlol (%)
CMX
RGB-E
M/T-B4
64.3
DFormer [66]
Onanivore [20]
FRSed821
[8
TokenFusion [58]
DFonma-S
ResNet:34
RecNet-34
MIT-B3
Swin-B
54.2
53.6
53.4
$4.0
53.7
HuseSeg [50]
ACNa [28)
ABMDRNe [75]
[ASNar
PAP
DesseNes-161
ResNes-152
ResNat-18
ReaNes-51
18
46.3
54.8
CMX
CMS
HRFwar [5]
OonSSgmence
Omniegmmmner
OmtiSemmenx
RGB-D
RGB
RGB-D
RGB-E
RGBFB
RGB-B
Hilfoemr--
ReeNer [0]
DRumand,
M/T-B2
MIT-B2
MIT B2
64.5
65.1
613
59.9
CMX [71)
MalriMAE
DFormee [66]
DForme-B
MiT-B2
ViT-B
56.0
54.4
55.6
[1ANe((19]
GMNe [50)
MFTNes [79]
ResNet: 152
ReiNes-150 152
RexNee-50 50
58.3
57.3
51.3
CMX [1]]
QuuiSeguenn
OuniSgmantie
RGB.D
RGB-D
ResNec
Mr1-18
MIT-B2
CMX [71]
MiT-B4
56.3
DoDDLeNet11]]
ResNet 101
57.5
OmnSegmeene
RGB D
Dlomee I.
66.2
CMX [7]]
CMNeXt
MiT-B4
MIT-B5
56.9
56.9
Difermer
CMX [71]
Divemee
MiT-B2
MiT-B4
58.2
59.6
HRFwer [5]
CMX
RGB D
RGB-D-E
HRIsemer T
M:T-82
63.0
60.3
DFFommer [66]
DFomme-L
57.2
CMNeXI
MiT-B4
CMNeX1U72
RGI-D-E
Mr1-112
639
OmmiSegnmeetor
ResNet-101
54.1
Smanieegmencr
CMX
R.B-D.E
MIT
65.0
OmmiSegmentor
MIT-B2
56.8
MIT-B2
60.5
OmmiSggmanee
RGB-D-E
RcNNa-101 101
62.8
OmniSeggentor
DFormer-I.
57.6
DFoemee-
69.6
RGBI-DE
M11-11
OuniSegment
RCB
DFunner4.
( DeLiVER (RGB-D-F-LI) [72]. RGB-X' means the corresponding input modalities.
Method
Backbone
RGB-E
RGB-D
RGB-L
RGB-D.E
RGB-E.L
RGB-D.L
RGB-D.E-L
HRFasee [6]
HRFormer-T T [63]
49.7
51.9
50.3
51.8
50.7
52.5
53.0
CMX [7]]
MiT-B2 [62]
57.6
63.9
CMNeXi [72]
MTT-B2 [62]
57.5
63.6
58.0
64.4
58.9
65.5
66.3
OmmiSegmenter
MiT-B2
58.4
64.9
59.0
65.7
60.1
67.0
67.5
OmniSegmenter
DFommelL
58.6
61.7
59.2
65.9
60.4
67.2
68.0
crop. More details, e-g., pretraining settings, are in supplementary materials. Following DFormer [66]. we adopts the light decoder head [19] by default. More experimental details are in the supplementary materials.
5.2 Comparison with Other Methods
Tab. shows the comparisons of our OmniSegmentor against the recent state-of-the-ar methods. In the following, we illustrate the results in two settings, i.e., RGB with a single supplementary modality and RGB with multiple supplementary modalities.
Single Supplementary Modality. We first conduct experiments on four bi-modal segmentation datasets. As shown in Tabs. (a-dd), our OmniSegmentor achieves new state-of-the-art records across all the four benchmarks. For the RGB-D segmentation benchmarks, our OmniSegmentor achieves 57.6% on NYUDepth [48] and 32.6% on SUNRGBD [49], even better than the recent strong RGBD-specific pretrained methods, i.e. DFormer [66], Omnivore [20] and MultiMAE [3]. For RGB-T segmentation on MFNet, our OmniSegmentor surpasses the recent SOTA CMNeXt (MiT-B4) by 0.7% mloU. In addition, the performance of our OmniSegmentor achieves 69.2% on KITTI-360, which exceeds previous cutting-edge methods by a large margin (nearly +4%). Tab. 6 shows the parameters and Flops of our OnniSegmentor and the recent SOTA methods. As can be seen, our OmniSegmentor has lower computational cost compared to other methods but receives better results. Following [71, 72, 66), we adopt multi-scale inference in RGB-D semantic segmentation benchmarks, as shown in Tab. 1(a) and Tab. 1(b).
7

---

Table 2: Different modality settings within our ImageNeXt pretraining. Note that the pretraining duration is 100 epochs in this experiment. We mark the significantly dropped performance in bold.
pretraining modalities Depth Event LiDAR Thermal
NYU V2 RGB-D
MFNet RGB-T
KITTI RGB-L
EventScape RGB-E
EventScape RGB-D-E
Index
RGB
54.3
57.6
64.6
61.8
63.8
2
v
52.2
57.5
64.6
61.6
61.9
3
54.2
57.6
64.5
60.5
62.9
v
54.3
57.7
61.2
61.9
63.7
r
d
54.3
56.4
64.8
62.1
63.8
6
v
54.4
56.4
61.4
62.1
64.0
54.6
56.2
61.3
60.5
63.1
50.9
55.6
60.1
58.7
59.7
Multiple Supplementary Modalities. Then, we carry out studies on two segmentation datasets with more visual modalities. As shown in Tab. ((c), our OmniSegmmntor surpasses all the other methods on the RGB-E and RGB-D of the EventScape dataset. Moreover, compared to CMNeXt, the advantage of OmniSegmentor is further enlarged from +0.7% (RGB-E), +1.4% (RGB-D), to +2.6% (RGB-D-E). Similarly, as shown in Tab. (f), our OmniSegmentor shows significant and consistent improvements across all seven settings of the input modalities the DeLiVER dataset.
Furthermore, the ImageNeXt pretraining brings consistent improvements across multi-modal segmentation benchmarks for different backbones, including ResNet-101, MiT-B2, and DFormer-L. For example, OmniSegmentor with the MiT-B2 backbone significantly exceeds other methods with the same backbone and even achieves better results than other methods with the MiT-B4 backbone.
5.3 Analysis on ImageNeXt Pretraining
Pretraining manners. To explain the necessity of our OmniSegmentor pretraining, we compare it with RGB-only pretraining and simultaneous multi-men modal pretraining. Note that the modalities of the input data and the model structure are the same for the finetuning setting.
We adopt NYU Depthv2 [48], MFNet [24], KITTI- 360 [34] and EventScape [18] to conduct this exper- iment. The results are shown in Fig. 6. The effi- cient pretraining of OmniSegmentor brings signif- icant improvements in all benchmarks, e.g., 2.4% on MFNet and 5. 5.1% KITTI-360. Specifically, on the EventScape benchmark, the improvement in- creased from 3.3% to 4.2% on the RGB-D - and RGB- D-E settings, illustrating that the improvement of the OmniSegmentor pretraining increases as the number of modalities increases. demonstrates the multi- modal pretraining endows the model capacity to en- code various modalities while the RGB-only pre- trained model may not fully use other modalities. These experiments indicate that the multi-modal rep- resentation capacity learned at the pretraining stage of OmniSegmentor crucial for robust semantic seg- mentation.
Performance of Different Pretraining Manners
NY U Depthv2 RGB-D
54.4
RGB only
54.2 (-02%)
Simultaneous
57.6 (+32%)
Ours
MFNet RGB-T
$7.7 (-03%)
60.6 (42.9%)
KITTI-360
64.1
RGB-L
63.8 (45.%)
EventScape RGB-D
62.9
62.5 (40.%)
66.2 (483%)
EventScape RGB-D-E
63.4
63.1 (-0.3)
67.6 (+42%)
(l)U(%)
Figure 6: Comparison of the performance for different pretraining manners. 'RGB only': RGB-only pretraining: 'Simultaneous': si- multaneously pretraining all modalities; 'Ours': our ImageNeXt pretraining, Note that the input modalities when finetuning are the same for all the models.
Ablation study on the pretraining modalities. For more insights into our ImageNeXt pretraining, we take off different modalities during pretraining and then finetune to verify whether the improvement
is direct from the modal data used in pretrain- ing. We finetune the models that are pretrained on different modalities for modli--mddal semantic segmentation tasks.
As shown Tab. 2, first, from Row to Row 5, we take off one the supplementary modalities in our ImageNeXt pretraining. It clear that the missing modality during pretraining leads to

---

Table 3: Different fusion operations under different pretraining manners on EventSeape (RGB-D-E). B). The pretraining duration of the model is 100 epochs.
Table 4: Effect of separate MLPs and shared MLP to encode supplementary modalities on the EventScape (RGB-D-E).
Fusion Operation
RGB-only
ImageNeXi
Modality Encoding
Params
Flops
mloU
Simple fusion SQ-Hub [72]
59.7
63.8
Shared MI.P
39.0M
68.9G
66.7 67.6
61.0
63.7
Separate MLPs
41.9M
68.9G
significant performance drop in the settings that contain this modality. For example, in Row 3, the missing event modality during ImageNeXt pretraining leads to a significant performance drop in the RGB-E and RGB-D-E segmentation settings on the EventScape [18] dataset. This phenomenon demonstrates that the mismatched encoding may have an influence on the information extraction of the supplementary modalities.
From Row 6 to Row 8, we further take off more modalities. When adopting the ImageNeXt pretrain- ing in RGB-D-E modalities, the performance of RGB-D-E semantic segmentation on EventScape is improved by 0.2%. Similarly, ImageNeXt pretraining on the RGB-D data brings 0.3% improvement on RGB-D segmentation on NYU Depth V2. Compared with the RGB-D pretraining settings, Ima- geNeXt brings significant improvements on all the other settings without sacrificing the performance of the RGB-D scenes. These results illustrate that common weight for the supplementary modalities is sufficient to learn the information pattern within them during pretraining.
Moreover, observe that ImageNeXt pretraining in RGB-D D modalities also benefits segmentation tasks with RGB and other supplementary modalities, such as the effect in Row 7 and Row 8. Nevertheless, the improvement is relatively limited compared to the ImageNeXt pretraining with the corresponding modality. Similar phenomena also appear in Tab. 2. Even with part of the types of supplementary modality, the ImageNeXt pretraining can still help the model build the connection between the RGB and supplementary clues. For example, in Row 6 of Tab. 2, ImageNeXt pretraining without thermal also brings improvement on RGB-T segmentation compared to the RGB pretraining, i.., from 55.6 to 56.4 on MFNet. Meanwhile, ImageNeXt pretraining with thermal modality (Row I) can further improve the RGB-T segmentation results, ie.. from 56.4 57.6 on MFNet. These results illustrate that ImageNeXt pretraining with all the supplementary modalities is necessary.
Discussion on the modalities fusion operation. In Tab. 3, we compare the adopted simple fusion operation in our model with the self-query hub in the DeLiVER [72]. Under RGB-only pretrain, sophisticated fusion operation brings a significant improvement compared to the simple fusion operation, but it has no effect under the ImageNeXt pretrain. We hypothesize that the SQ-Hub may help alleviate the optimization difficulties in selecting information from different modalities from the random initialization weight, but our model can align the features of different modalities during the ImageNeXt pretraining thus alleviating the optimization difficulties in the finetuning.
Separate MLPs or Shared MLP. In our OmniSegmentor, we adopt separate MLPs to encode the unique characteristics within different supplementary modalities. In Tab. 4, we use the shared MLP to encode different supplementary modalities as comparison to the separate ones. As can be seen, separate MI.Ps can bring significant improvement with a small increase in parameters. We hypothesize that specific parameters help extract the unique characteristics of each supplementary modality and achieve more robust segmentation results.
Conclusions and Future Work
In this paper, we propose a flexible framework for multi-modal segmentation, which is composed of the ImageNeXt dataset and the geerain-anddfinetn method. To the best of our knowledge, OmniSegmentor is the first framework to endow the model with the capacity to jointly encode more than three types of multi-modal data during pretraining. Benefiting from the modality selection mechanisms, OmniSegmentor can be applied for different multi-modal scenes, presenting robust segmentation across all the multi multi-modal scenes.
In the experimental part, we have conducted extensive experiments on various multi-modal segmen- tation benchmarks. However, these existing benchmarks are limited in scope, as they cover only a subset of the five major visual modalities, and some rely on synthetic data generated by simulation tools. In the future, we will attempt to gather more comprehensive multi-modal data from the real pretraining manners perform mudti- learning.
world and explore mnsupervised/seffsuperre

---

Acknowledgments This work was partially funded by NSFC (No. 62495061, 62276145), Na- tional Key Research and Development Project of China (No. 2024YFE0100700), the Science and Technology Support Program of Tianjin, China (No. 23IC7DJC01150), Shenzhen Science and Technology Program (JCYJ20240813) 14237048), and the Fundamental Research Funds for the Central Universities (Nankai University) under Grant 070-63253220.
References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurlPS, 2021.
[2] Inigo Alonso and Ana C Murillo. Ev-segnet: Semantic segmentation for event-based cameras. In CVPRW, 2019. 3
[3] Roman Bachmann. David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi molti-modal multi-task masked autoencoders. In ECCV, 2022. 5,
[4] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In CVPR, 2021. 4
[5] Bowen, Yin, Xuying Zhang, Li Liu, Ming-Ming Cheng, Yongxiang Liu, and Qibin Hou. Camouflaged object detection with adaptive partition and background retrieval. Intermational Journal of Computer Vision (IJCV), 133(7):4877-485) 2025.
[6] Tim Bredermann, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. HRFuser: A multi-resolution sensor fusion architecture for 2D object detection. In /TSC, 2023. 7.7
[7] Jinming Cao, Hanchao Leng. Dani Lischinski, Danny Cohen-Or, Changhe Tu, and Yangyan Li. ShapeConv: Shape-aware convolutional layer for indoor RGB-D -D semantic segmentation. In ICCV, 2021. 7
[8] Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Learning aligned cross-modal representations from weakly aligned data. In CVPR, 2016. 3
Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang, and Ming-Ming Cheng. Spatial information guided convolution for real-time RGBD semantic segmentation. T7P, 20:2313-2324, 2021. 3,7
[10] Xiaokang Chen et al. Bi-directional cross-modality feature propagation with separation-andaaarreaaiio gate for RGB-D semantic segmentation. ECCV, 2020. 2,
[11] Yen-Chur Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
[12] Marius Cordts et al. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 21
[13] Fuqin Deng et al. FEANet: Feature-enhanced attention network for RGB-thermal call-ime semantic segmentation. In IROS, 2021. 7
[14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CoRL, 2017. 21
[15] Alexey Dosovitskiy et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021. 3
[16] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task miv-level vision datasets from seans. In ICCV, 2021.
[17] Oriel Frigo, Lucien Martin-Gaffe, and Catherine Wacongne. Doodlenet: Double deeplab enhanced feature fusion for thermal-color semantic segmentation. In CVPR, 2022.
[18] Daniel Gehrig, Michelle Rüegg, Mathias Gehrig, Javier Tidalgo-Carrió, and Davide Scaramuzza. Combin- ing events and frames using recurrent asynchronous multimodal networks for monocular depth prediction. RA-L, 6(2):2822 2829, 2021. 2, 3, 6, 7, 8, 21
[19] Zhengyang Geng, Meng- Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin. Is attention better than matrix decomposition? NewrIPS, 2021. 6,7
[20] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In CVPR, 2022. 5.7
10

---

[21] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023. 3,5
[22] F. A. Group. Flir thermal dataset for algorithm training. In httt:://wwww//hccooommmmmmddddds form/., 2018.
[23] Meng- Hao Guo, Cheng-Zo Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. NeurlPS, 2022. 3,21
[24] Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. MFNct: Towards real-time semantic segmentation for autonomous vehicles with multi-ppectral scenes, In IROS, 2017. 2, 6, 7, 8, 21
[25] Zhiwei Hao, Zhongyu Xiao, Yong Luo, Jianyuan Guo. Jing Wang. Li Shen, and Han Hu. Primkd: Primary modality guided multimodal fusion for rgb-d semantic segmentation. In ACM MM. 2024. 5
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 3, 21
[27] Qibin Hou, Cheng-Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style convnet for visual recognition. arXiv preprint arXiv:2211.11945 2022.
[28] Xinxin Hu. Kailun Yang. Lei Fei, and Kaiwei Wang. ACNet: Attention based network to exploit complementary features for RGBD semantic segmentation. In ICIP, 2019. 4.7
[29] Junho Kim, Jachyeok Bac, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. In ICCV, 2021. 4
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 6, 21
[31] Gongyang Li, Yike Wang, Zhi Liu, Xinpeng Zhang, and Dan Zeng. Rgb-t semantic segmentation with location, activation, and sharpening. IEEE TCSVT, 33(3):123--225,, 2022.
[32] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[33] Weijie Li. Wei Yang. Yuenan Hou, Li Liu, Yongxiang Liu, and Xiang Li. Saratr-x: Toward building foundation model for sar target recognition. IEEE Transactions on Image Processing, 34(1))869-888) 884, 2025. 3
[34] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-600: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE TPAMI, 45(3):322--3310, 2022, 2, 3, 6, 7, 7,8,2 21
[35] Di Lin, Guangyong Chen, Daniel Cohen-Or, Pheng-Ann Heng, and Hui Huang. Cascaded feature network for semantic segmentation RGB-D images. In ICCV, 2017.
[36] Li Liu and Paul Fieguth. Texture classification from random features. IEEE Transactions Pattern Analysis and Machine Intelligence, 346))557--886 2012. 3
[37] Li Liu, Shuzhou Sun, Shuaifeng Zhi, Fan Shi, Zhen Liu, Janne Heikkilä, and Yongxiang Liu. A causal adjustment module for dehiasing scene graph generation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 4705):402444043, 2025.
[38] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 31
[39] Ze Liu al. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[40] Seong- Jin Park, Ki- Ki-Sang Hong, and Seungyong Lee, RDFNet: RGB-D multi-leved residual feature fusion for indoor semantic segmentation. In ICCV, 2017. 7
[41] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR, 2021. 7
[42] Xiaojuan Qi. Renjie Liao, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. 3D graph neural networks for RGBD semantic segmentation. In ICCV, 2017. 7
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.

---

[44] Olga Russakovsky et al. ImageNet large scale visual recognition challenge. WCV, 115(3):2111122)) 2015.
1, 2,24 4
[45] Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michad Gross, Effi- cient rgb-d semantic segmentation for indoor scene analysis. In ICRA, 2021. 7
[46] Daniel Seichter, Söhnke Benedikt Fischedick, Mona Köhler, and Morst-Michacl GroB. Efficient multi-task rgh-d scene analysis for indoor environments. In IJCNN, 2022." 7
[47] Shreyas S. Shivakumar, Ncil Rodrigues, Alex Zhou, Ian D. Miller, Vijay Kumar, and Camillo J. Taylor. PST900: RGB-thermal calibration, dataset and segmentation network. In ICRA, 2020. 3
[48] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012. 2, 3, 6, 7, 8, 21
[49] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. SUN RGB-D. D: A RGB-D seene understanding benchmark suite. In CVPR, 2015. 3, 6, 7, 21
[50] Yuxiang Sun, Weixun Zuo Peng Yun, Hengli Wang, and Ming Liu. FuseSeg: Semantic segmentation of urban sccnes based on RGB and thermal data fusion. T-ASE, 18(3):1000-101, 2021 7
[51] Bowen, Yin, Xuying Zhang. Deng-Ping Fan, Shaohui Jiao, Ming-Ming Cheng, Luc Van Gool, and Qibin Hou. Camoformer: Masked separable attention for camouflaged object detection. JEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 46(12):10362- 103774 2024. 3
[52] Zhengzheng Tu, Tian Xia, Chenglong Li, Xiaoxiao Wang. Yan Ma, and Jin Tang. Rgb-t image saliency detection via collaborative graph learnin.. IEEE TMM, 22(1):160 167-17,,,,11,,,,1,1,11 2019. 4
[53] Zhengzheng Tu, Yan Ma, Zhun Li, Chenglong Li, Jieming Xu, and Yongtao Liu. Rgbt salient object detection: A large-scale datasct and benchmark. IEEE TMM, 2022. 4
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017. 3
[55] Guizhao Wang, Chenglong Li, Yunpeng Ma, Aihua Zheng, Jin Tang, and Bin Luo. Rgb-t saliency detection benchmark: Dataset, baselines, analysis and a novel approach. In IGTA, 2018, 4
[56] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Camphell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019. 4
[57] Yikai Wang. Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. NewrIPS, 2020. 7
[58] Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In CVPR, 2022. 2,
[59] Shicai Wei, Chunbo Luo, and Yang Luo. Mmanet: Margin- aware distillation and modality- regular- ization for incomplete multimodal lcarning. In CVPR, 2023.
[60] Mingrui Wa, Xuying Zhang, Xiaoshuai Sun, Yiyi Zhou, Chao Chen, Jiaxin Gu, Xing Sun, and Rongrong
Ji. Difnet: Boosting visual information flow for image captioning. In CVPR, 2022.
[61] Wei Wu, Tao Chu, and Qiong Liu. Complementarit-awver cross-modal feature fusion network for rgb-t semantic segmentation. PR, 131:108881, 2022.
[62] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021. 7,21
[63] Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng. Alei Leonardis, and Joni-Kristian Kämäräänen. Depth- track: Unveiling the power of rghd tracking. In ICCV, 2021.
[64] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang. Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. In ECCV, 2022. 3
[65] Bowen Yin and Zheng Lin. Exploring salient object detection with adder neural networks. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2025. 3
[66] Bowen Yin, Xuying Zhang, Zhongyu Li, Li Liu, Ming-Ming Cheng, and Qibin Hou, Dformer: Rethinking rgbd representation learning for semantic segmentation. In ICLR, 2024. 2, 3, 4, 4, 71
12

---

[67] Bo- Wen Yin, Jiao-Long Cao, Ming-Ming Cheng, and Qibin Hou. Dformerv2: Geometry self-attention for rgbd semantic segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19345-19355, 2025.
[68] Yuhui Yuan al. HRFormer: High-resolution transformer for dense prediction. In NeurlPS, 2021. 7
[69] Guodong Zhang, Jing-Hao Xue, Pengwei Xie, Sifan Yang, and Guijin Wang. Non-local aggregation for RGB-D semantic segmentation. SPL, 28:658-662, 2021.7
[70] Jiaming Zhang, Kailun Yang, and Rainer Stiefelhagen. Issafe: Improving semantic segmentation in accidents by fusing event-based data. In IROS, 2021.
[71] Jiaming Zhang, Huayao Liu, Kailun Yang. Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen. Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers. T-I7S, 2023. 2, 3, 4, 7,
[72] Jiaming Zhang, Ruiping Liu, Hao Shi, Kailun Yang, Simon ReiB, Kunyu Peng, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Delivering arbitrary-modal semantic segmentation. In CVPR, 2023. 4, 6,
9,21
[73] Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, and Jungong Han. ABMDRNet: weighte bi-directional modality difference reduction network for RGB-T semantic segmentation. In CVPR, 2021. 3,7
[74] Shi-Chen Zhang, Yunheng Li, Yu-Huan Wu, Qibin Hou, and Ming-Ming Cheng. Revisiting efficient semantic segmentation: Learning offsets for better spatial and class feature alignment. In ICCV, 2025. 3
[75] Xuying Zhang. Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In CVPR, 2021. 3
[76] Yao Zhang, Nanjun He, Jiawei Yang, Yuexiang Li, Dong Wei, Yawen Huang, Yang Zhang, Zhiqiang He, and Yefeng Zheng. mmformer: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In MICCAI, 2022. 4
[77] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning, arXiv preprint arXiv:2307.10880 10802, 2023, 3,5
[78] Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang. Patter-afflnitive propagation across depth, surface normal and semantic segmentation. In CVPR, 2019.
[79] Heng Zhou, Chunna Tian, Zhenxi Zhang, Qizheng Huo, Yongqiang Xie, and Zhongho Li. Multispectral fusion transformer network for rgb thermaamal urban scene semantic segmentation. IEEE GRSL, 19:1-5, 2022.
[80] Wujie Zhou, Jinfu Liu, Jingsheng Lei, Lu Yu, and Jeng-Neng Hwang. GMNet: Graded-feature multilabel- learning network for RGB-thermal urban scene semantic segmentation. TIP, 30:7790-7802, 2021." 7
[81] Wujie Zhou, Enquan Yang, Jingsheng Lei, Jian Wan, and Lu Yu. Pgdenet: Progressive guided fusion and depth enhancement network for rgb-d indoor scene parsing, IEEE TMM, 2022.
[82] Wujie Zhou, Enquan Yang, Jingsheng Lei, and Lu Yu. Frnet: Feature reconstruction network for rgb-d indoor scene parsing. JSTSP, 16(4))677-687, 2022.
[83] Zhuangwei Zhuang, Rong Li, Kui Jia, Qicheng Wang, Yuanqing Li, and Mingkui Tan. Perception-aware multi-sensor fusion for 3d lidar semantic segmentation. In ICCV, 2021. 4,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims we made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide both implementation details and training datasets Sec. 5 and Sec. A.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will release of code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt / ssisssslicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide the detailed experiment setup in Sec. 5 and Sec. A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the convention in prior works and report the performance the standard benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the demand of compute resources in Sec. and Sec. A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neett httppp::///uuipp///
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited the creators or original owners of assets (e.g., code, data, models), used in the paper and conformed the license and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the details of dataset/code/mmodl along with experiments setup, license, limitations in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / /ttp///ersp////// theurips.cc/oonfeeee eeenneer///eee22//////// /LLM/LLM for what should or should be described.
20

---

A Experiment Details
A.1 RGBX benchmarks
To validate the effectiveness of our OnniSegmentor, we conduct extensive experiments on six popular multi-modal segmentation datasets. The statistics of the datasets and the corresponding training strat- egy of our model are shown in Tab. 5. We conduct extensive experiments with our OmniSegmentor on different multi-modal segmentation datasets and briefly introduce them in the following. NYU D (RGB-D) [48] contains 1, 1,449 RGB-D images with a size of 640x480 which is divided into 795 training and 654 test images with annotations for 40 categories. SUNRGBD (RGB-D) [49] includes 10,335 samples with 530x 730 resolution. There are 37 semantic categories. Following [66], we randomly crop and resize the input to 480x480 during training. MFNet (RGB-T) [24] is a
multi-spectral RGB-T image dataset, which has 1,569 images. 784/392/393 samples are used for training/validatiionneess respectively, annotated in 8 classes at the resolution of 640x480. KITTI- 360 [34] is a suburban driving dataset that contains 19 classes as same as the Cityscapes dataset [12]. EventScape [18] was originally designed for using RGB and event data to conduct depth estimation. It has pixel annotations for semantic segmentation well. Following CMX [71], we select one frame from every 30 frames, obtaining 4,077/749 for training and evaluation to maintain data diversity from the original sequences generated by the CARLA simulator [14]. DeLiVER [72] is large-scale multi- modal segmentation dataset, which is also generated by the CARLA simulator. This dataset contains 7,885 front-view samples divided into 3,983 / 2,005 1,897 for training / validation / test, respectively. each of which contains two types of annotations (i.e., semantic and instance segmentation labels).
Table 5: Statistics of the used multimodal segmentation datasets and the corresponding training settings used in the proposed method.
Darasets
NYU Depih [48]
SUNRGBD [49]
MPNet [24]
KITTI-360 [34]
EventScape [18]
Del DeLIVER [72]
Medalities
RGB-D 795/ / 654 /-
RGB-D
RGB-T
RGB-L
RGB-D-E
RGB-D-E-L
Train/val/est split
5285 / 55500
1568 //2 393
49,004
12,276
4,077 774 /- 12
3983 2005 / 1897
Classes
40
37
19
25
Input size
640x480
480xx400
640xx400
x408x376
512x256
1024x1024
Batch size
8
16
8
16
4
8
Epochs
500
300
40 6e-
100
200
Base It
6e-5
Se-5
6e-5
6e-5
6e-5
Optimizer
AdamW
Adam
Adam
Adam 0.01
Adam
Adam
Weight decay
0.01
0.01
Lr schedule
Linear decny 0.35
Linear decny 0.20
Linear deany 0.40
Linear decay 0.30
Linear decny 0.20
Linear decay 0.30
Stochastic depth
A.2 Experiment Setup
Pretraining details. The multi-modal features from the last stage are flattened along the spatial dimension and fed into the linear projection obtain the category probabilities, which are used to calculate the classification loss, i.e., the standard cross-entropy loss. To verify the universality and robstness of our methods on different architectures, we adopt DFormer-L [66], MiT-B2 [62], and ResNet-101 [26] as backbone and perform ImageNe pretraining with them. In the experiments, unless otherwise specified, the OmniSegmentor uses the DFormer-L backbone. ImageNeXt pre- training adopt the same hyperparameters as DFormer-L. Following the commonly used pretraining durations [38, 62, 23, 66), OmniSegmentor is pretrained for 300 epochs. We use Adam [30] with learning rate le-3 and weight decay 5e-2 as our optimizer, and the batch size is set to 1024. We adopt the same data augmentation strategies as DFormer [66], i.e., data augmentation strategies related to color, e.g., auto contrast, are only used for RGB images, while other common strategies are simultaneously performed on all the modalities, e.g., random rotation.
Finetuning details. The experiments are conducted on NVIDIA A40 GPUs. The models are optimized using the cross-entropy loss function and the AdamW [30] method, where the learning rate is scheduled by the poly strategy. The images are augmented by random resizing with a ratio of 0.5 to 1.75, random horizontal flipping, and random cropping. Tab. 5 presents the detailed training settings for different segmentation datasets. In Tab. 1, we compare our OmniSeggmntor with other SOTA methods on all multi-modal benchmarks. The segmentation of RGB and multiple supplementary modalities an emerging field. In EventScape (RGB-D-E) and DeLiVER (RGB-D. D- E-L), the results of some methods are missing. For example, CMNeXt [72] lacks the results on the
21

---

Table 6: Comparisons of our OmniSegmentor with other SOTA methods the number of parameters and Flops. The calculation is based on the same code, and these methods are evaluated when keeping the inference settings in the corresponding papers. The inference time, i.e., frames per second (FPS), is calculated on a single NVIDIA A40 GPU. When calculating FLOPs, the input size is set to x80X
Methods
Ours
CMNeXt-B4
CMNeXi-B2
CMX-B5
CMX-B4
TokenFusion
MultiMAE
HRFuser
Omnivore
Purams
39.0M
119.6M
58.8M
181.MM
139.9M
45.9M
95.2M
30.5M
95.7M
FLOPs
65.7G
131.9G
62.9G
167.8G
134.3G
94.4G
267.9G
223.0G
109.3G
FPS
28.0
13.7
27.2
10.1
13.0
12.6
20.0
18.4
10.5
RGB-R-L setting. To make the comparison more comprehensive, we implement these methods the missing settings based on their official code.
22


Reviews:
[{"review_id": "TofcnZus34", "reviewer": "Reviewer_A5r8", "summary": "The paper presents **OmniSegmentor**, a novel framework for multimodal semantic segmentation. The proposed framework comprises two main components: (i) a new large-scale dataset, **ImageNeXt**, which includes RGB images along with four additional modalities, thermal, depth, LiDAR, and event data; and (ii) an elegant training method that enables efficient learning across multiple modalities and facilitates easy fine-tuning for downstream tasks.\n\nThe proposed method achieves state-of-the-art performance on several multimodal semantic segmentation benchmarks, including **NYU Depth v2**, **EventScape**, **MFNet**, **DeLiVER**, **SUN RGB-D**, and **KITTI-360**.", "questions": "See the questions in the Strengths And Weaknesses section.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n* The paper introduces ImageNeXt, the first large-scale multimodal dataset that includes thermal, depth, LiDAR, and event data, alongside RGB images sourced from ImageNet. The dataset is constructed using synthetic data generated through multiple techniques, enabling the authors to extract all required modalities from RGB images.\n\n* A novel pretrain-and-finetune pipeline is proposed, offering an efficient pretraining strategy combined with flexible fine-tuning. Specifically, to maximize efficiency during pretraining, the model is fed only the RGB image at each iteration, along with a randomly selected supplemental modality. This strategy significantly reduces training time, by more than a factor of two, compared to training on all modalities simultaneously, while maintaining the same number of parameters and FLOPs as RGB-only training.\n\n* The proposed pipeline enables straightforward fine-tuning on downstream tasks by simply loading the pretrained weights and appending lightweight modules to capture the specific characteristics of each modality.\n\n* OmniSegmentor achieves state-of-the-art results across all evaluated multimodal semantic segmentation benchmarks.\n\n* The paper is well-written and easy to follow. The authors provide extensive experimental results to thoroughly validate the proposed approach.\n\n---\n\n**Weaknesses**\n\n* **Concern about modality selection randomness.** While not a critical flaw, this point could benefit from further clarification. During pretraining, a supplemental modality is randomly selected at each iteration. Have you considered alternative, more deterministic selection strategies, such as a round-robin schedule (e.g., depth → thermal → LiDAR → event → [repeat])? Exploring such strategies might further enhance the robustness and effectiveness of the proposed pretraining approach.", "was_revised": true, "final_justification": "After the rebuttal, all of my initial concerns were successfully addressed, so I chose to maintain my original positive rating.", "rebuttal": "Thanks for your effort for reviewing our paper and giving some kind suggestions. We are happy to see your positive comments on writing, experiments and method. We hope the following responses could solve your concerns.\n\n**About modality selection randomness.**\nThank you for the valuable comment regarding the modality selection strategy during pretraining. \nThe ImageNeXt pretraining involves randomly sampling one supplementary modality at each iteration. \nFollowing your suggestion, we conduct experiments with a more deterministic round-robin schedule (e.g., depth → thermal → LiDAR → event → ...), and compare its effectiveness with our random selection strategy.\nAs shown in the table below, we observe that they have similar performance across all evaluated multi-modal tasks. The results indicate that our pretraining method is robust to the choice of modality scheduling.\n\n| Sample Manner | NYU V2 (RGB-D) | MFNet (RGB-T) | KITTI (RGB-L) | EventScape (RGB-E) | EventScape (RGB-D-E) |\n|----|---------|----------------|----------------|----------------|---------------------|\n|  random sampling  | 57.6           | 60.6           | 69.2           | 65.0                | 67.6                   |\n| deterministic round-robin | 57.8       | 60.5           | 68.8           | 65.5            | 67.4               |\n\n\nWe hypothesize this robustness stems from the fact that such scheduling differences effectively correspond to variations in the gradient update order, which become less significant as training iterations accumulate. In other words, given sufficient training iterations, both random sampling and fixed round-robin schedules tend to converge to similar solutions in practice.\n\nWe sincerely thank the reviewer for this constructive suggestion, which provides additional insights and makes this study more in-depth."}, {"review_id": "fqi9hnePNA", "reviewer": "Reviewer_HYgf", "summary": "This paper presents OmniSegmentor, a multi-modal semantic segmentation framework trained on a large-scale synthetic dataset, ImageNeXt, covering five visual modalities. The model extends DFormer to support arbitrary modality combinations and achieves strong results on several benchmarks.\nHowever, all non-RGB modalities are synthesized from RGB using public tools, limiting cross-modal diversity. The model introduces minimal architectural innovation, and performance gains from large-scale pretraining are marginal, raising concerns about the practical value and novelty of the work.", "questions": "Table2 doesn't seem to include all the modality combinations. Has the paper systematically explored all combinations of RGB with a single additional modality (e.g., RGB+LiDAR, RGB+Thermal, RGB+Event)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\nAssemble a large-scale dataset and introduce a framework that achieves good results on multimodal semantic segmentation datasets.\n\n## Weaknesses\n\n- Limited Dataset Contribution\n\nImageNeXt introduces multiple modalities, but all supplementary ones (depth, LiDAR, thermal, event) are *synthesized from RGB* using off-the-shelf tools like Omnidata and pseudo-LiDAR. This essentially reduces multimodal learning to **an advanced form of RGB data augmentation**, rather than capturing genuine cross-modal relationships from independent sensors. Since all modalities stem from the same RGB source, they exhibit strong inter-modal correlation and offer limited diversity. The dataset is easily reproducible using public methods, which diminishes its originality and contribution. \n\n- Minimal Gains from Large-Scale Pretraining\n\nAs the authors mentioned, ImageNeXt is a large-scale dataset (1.2M samples) comprising five visual modalities, designed to be distinguishable from existing datasets. However, despite pretraining on three additional visual modalities (L/T/E) compared to its baseline (DFormer, which uses only RGB-D pairs), the improvements are marginal (+0.4% in Table 1.a and +0.3% in Table 1.b), raising concerns about the effectiveness and efficiency of large-scale synthetic data generation.", "was_revised": true, "final_justification": "The author solves my concern. Thus I keep my original score.", "rebuttal": "Thanks for your time for reviewing our paper and giving some insightful suggestions. We hope the following responses could solve your concerns.\n\n**1.About the Dataset Contribution.**\nThank you for your thoughtful comments regarding the nature of the supplemental modalities in our ImageNeXt dataset.\n\nWhile it is true that the additional modalities are synthesized from RGB using established tools, we emphasize that each modality retains its unique structural and statistical characteristics. Pretraining on these synthesized data enables the model to effectively learn and capture modality-specific representations.\nAs shown in Table 2 of the main paper, pretraining on the synthesized ImageNeXt significantly improves performance on real-world downstream tasks, indicating that our synthetic modalities successfully capture modality-specific patterns essential for effective representation learning.\n\nFurthermore, the synthetic nature of our dataset is a deliberate design choice. It enables a scalable and reproducible paradigm for generating diverse multi-modal pretraining data, where multi-modal signals can be constructed directly from RGB images without requiring physical sensors. This allows the framework to be extended to construct even larger-scale pretraining data. We believe this aspect of ImageNeXt offers a practical and impactful contribution to the community, potentially serving as a foundation for future large-scale multi-modal dataset construction.\n\nWe will highlight these clarifications more explicitly in the revised version. Thank you again for raising this important point.\n\n\n**2.About the Performance Gains on RGB-D benchmarks.**\n\nThank you for the feedback. We would like to clarify the positioning and broader implications of our pretraining paradigm.\n\nFirst, it is worth noting that the second-best method on RGB-D benchmarks, DFormer, also leverages large-scale ImageNet (1.2M RGB samples) for pretraining, and it is specifically designed for RGB-D segmentation tasks. Its pretraining is conducted on RGB-D pairs and tailored to extract complementary features from depth and RGB inputs for that specific scenario.\n\nIn contrast, our approach is designed to be more general and modality-agnostic. It incorporates additional modalities beyond RGB-D, and establishes a unified pretraining strategy applicable to a wider range of multi-modal tasks. Despite this generality, our model still shows improvements even on the RGB-D tasks where DFormer is highly specialized, as shown in Tables 1(a) and 1(b). We believe this underscores the robustness of our pretraining approach.\n\nMore importantly, the value of our method lies not only in performance improvements on any single task, but in its transferability and broad applicability across modalities and architectures. The learned representations from ImageNeXt can serve as strong initialization across a spectrum of multi-modal scenarios, which we further demonstrate in downstream evaluations beyond RGB-D.\n\nWe will make this broader perspective more explicit in the revised version. Thank you again for the helpful suggestions.\n\n\n**3.About the Table 2.**\n\nDue to space limitations, we did not include all combinations of RGB with a single supplementary modality in Table 2. Following your suggestion, we have conducted and now provide the complete results for RGB with a single additional modality in line 7-10 of the table below. It can be observed that pretraining RGB together with any additional modality consistently yields significant improvements in downstream tasks involving that modality.\n\n\nTable 2. Different modality settings within our ImageNeXt pretraining. Note that the pretraining\nduration is 100 epochs in this experiment. We mark the significantly dropped performance in **bold**.\n| Index | RGB | Depth | Event | LiDAR | Thermal | NYU V2 (RGB-D) | MFNet (RGB-T) | KITTI (RGB-L) | EventScape (RGB-E) | EventScape (RGB-D-E) |\n|-------|-----|-------|-------|--------|---------|----------------|----------------|----------------|---------------------|------------------------|\n| 1     | ✓   | ✓     | ✓     | ✓      | ✓       | 54.3           | 57.6           | 64.6           | 61.8                | 63.8                   |\n| 2     | ✓   |    |    ✓     | ✓      | ✓       | **52.2**       | 57.5           | 64.6           | 61.6  | **61.9**               |\n| 3     | ✓   | ✓     |       |    ✓     | ✓       | 54.2           | 57.6           | 64.5           | **60.5**               | **62.9**                   |\n| 4     | ✓   | ✓     |    ✓     |    |  ✓    | 54.3           | 57.7           | **61.2**       | 61.9                | 63.7                   |\n| 5     | ✓   | ✓     |   ✓      |    ✓     |      | 54.3           | **56.4**       | 64.8           | 62.1                | 63.8                   |\n| 6     | ✓   | ✓     | ✓     |       |         | 54.4           | **56.4**       | **61.4**       | 62.1                | 64.0                   |\n| 7     | ✓   |     |       |    | ✓         | **51.8** | 57.7     | **61.1**       | **60.4**            | **60.8**               |\n| 8     | ✓   |     |       |  ✓   |         | **52.0**  | **55.9**       | 64.8       | **60.7**            | **61.1**               |\n| 9     | ✓   |   |    ✓      |    |         | **51.6**           | **56.3** | **61.0**       |61.9  | **62.4**               |\n| 10     | ✓   | ✓     |       |    |         | 54.6           | **56.2**       | **61.3**       | **60.5**            | **63.1**               |\n| 11     | ✓   |   |       |       |         | **50.9**       | **55.6**       | **60.1**       | **58.7**            | **59.7**               |\n\nWe will update the Table 2 of the main paper to incorporate these more comprehensive experiment results. We appreciate your suggestion, which has helped us improve the completeness and transparency of our experimental evaluation."}, {"review_id": "j48hBZs1Uo", "reviewer": "Reviewer_Eebt", "summary": "The paper presents a novel multi-modal dataset based on ImageNet, containing different synthetic visual modalities beside RGB. Together with the dataset, an efficient and effective pertaining strategy is shown, where and RGB Tower is augmented with a second encoder fed with a random additional modality during pretraining, reducing the mismatch across different visual modalities. The pretrained multimodal encoder is then fine-tuned for downstream tasks, potentially leveraging all modalities together.", "questions": "How does DFormer work? Which architecture does it leverage?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n- the paper is easy to follow;\n- a new large scale multimodal dataset is presented;\n- the new pretraining strategy is efficient and yields better results wrt naive pretraining with all modalities;\n- the proposed model outperforms competitors across a wide set of benchmarks.\n\nWeaknesses:\n- being the paper largely based on DFormer, this model should be at least briefly described from an architectural and training point of view;\n- the gain over competitors is very limited in certain experiments (e.g. Table 1 a and b);\n- the use of synthetic data may hinder real world applications, although the effect seems to be limited when fine-tuning.", "was_revised": true, "final_justification": "The dataset contribution may be relevant to the field, plus training strategy and method are pretty interesting.", "rebuttal": "Thanks for your effort for reviewing our paper and giving valuable suggestions. We are happy to see your positive comments on writing, experiments and method. We hope the following responses could solve your concerns.\n\n**1. Details about DFormer.**\n\nThank you for your suggestion. We have briefly described DFormer's architecture and training strategy in the main text (lines 193–203), but we agree that a clearer exposition would improve the completeness of the paper.\n\nTo clarify, DFormer is a unified RGB-D pretraining framework that learns transferable representations directly from image-depth pairs. Architecturally, it adopts a hierarchical encoder-decoder structure built on customized RGB-D blocks, which integrate RGB and depth features via two core modules: Global Awareness Attention (GAA) for global semantic fusion and Local Enhancement Attention (LEA) for local detail refinement. For pretraining, DFormer uses synthetic depth maps derived from ImageNet-1K and is trained with a classification objective. During finetuning, only RGB features are used by a lightweight decoder to perform segmentation tasks.\n\nWhile DFormer has demonstrated the effectiveness of multi-modal feature fusion in RGB-D pretraining, the core contribution of our work goes beyond validating multi-modal pretraining effectiveness. Our OmniSegmentor proposes a general and extensible multi-modal pretraining framework that addresses several critical limitations of prior work:\n(1) Modality Extensibility: Our framework supports arbitrary modality combinations (e.g., RGB-depth-thermal-event-LiDAR and RGB-depth-thermal), whereas DFormer is limited to RGB-D inputs.\n(2) Unified Pretraining Framework: Instead of designing modality-specific fusion modules, we propose a unified pretraining strategy that dynamically adapts to varying modality availability and combinations during training.\n(3) Dataset Innovation: We construct a large-scale synthetic dataset, ImageNeXt, which covers five modalities and provides a scalable and reproducible infrastructure for future multi-modal pretraining research.\n\nWe will revise the paper to include a more explicit description of DFormer's architecture and training pipeline, along with a discussion highlighting the key distinctions and innovations introduced by our proposed approach. We appreciate your constructive feedback, which will help clarify our contribution and position it more precisely in relation to prior work.\n\n\n**2. About the Performance Gains in RGB-D Scenes.**\n\nThank you for your valuable feedback. \nWhile the gains in Tables 1(a) and 1(b) may appear modest, it is important to note that the second-best method, DFormer, is specifically designed for RGB-D segmentation and also utilizes large-scale ImageNet (1.2M RGB images) with RGB-D pairs for pretraining. In contrast, our method adopts a more general and modality-agnostic design, incorporating additional modalities beyond RGB-D and enabling a unified pretraining strategy applicable to a wide range of multi-modal tasks.\n\nDespite this broader design, our model still achieves consistent improvements even on tasks where DFormer is highly specialized, highlighting the robustness of our approach. More significantly, our method demonstrates strong transferability across modalities and architectures, which we further validate on diverse downstream benchmarks.\n\nWe will make this broader perspective more explicit in the revised version. Thank you again for your insightful comment.\n\n\n**3. Synthetic Data may Hinder Real World Applications.**\n\nThank you for your insightful comment. In this paper, the synthetic multi-modal data is used for large-scale pretraining to equip the model with capability to process diverse multi-modal data. This is followed by fine-tuning on downstream datasets, which enables the model to adapt to different multi-modal scenarios.\n\nAs shown in our experiments, this strategy leads to consistent improvements across multiple real-world benchmarks. For example, NYU DepthV2 and SUN-RGBD represent indoor scenes, while KITTI-360 and MFNet capture outdoor environments. These results demonstrate that our pretraining paradigm effectively benefit the real-world scene perception, despite being pretrained on synthetic data.\n\nWe appreciate the reviewer’s concern and fully agree that exploring real data sources is a meaningful direction. In future work, we plan to incorporate more real-world multi-modal data and evaluate our method's robustness on previously unseen scenarios."}, {"review_id": "to3Xnb91af", "reviewer": "Reviewer_TR9e", "summary": "OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation proposed ImageNeXt as assembled large-scale dataset for multi-modal training on various segmentation benchmarks including RGB, depth, event, LiDAR, and the thermal modalities. Based on an earlier work of DFormer, modality-specific encoding modules enable to train and evaluate the method on different datasets with different inputs. OmniSegmentor sets state-of-the-art results on NYU-D_v2, MFNet, KITTI-360, DeLiVER, and EventScape datasets.", "questions": "Is there any specific challenge in scaling the approach to more modalities or different domains?", "limitations": "The limitations are noted in the text. \n\nThe real-world generalization and dependence on backbone choice could be explored in more depth.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes new multi-modal training benchmark that supports setting state-of-the-art results on various datasets.\n\nThe paper provides a user-friendly and flexible training framework for integrating multiple modalities, which can benefit real-world applications.\n\nStrong empirical results are reported on several standard multi-modal segmentation benchmarks.\n\nThe release of data, code, and checkpoints will facilitate reproducibility and further research.\n\nWeaknesses & Suggestions:\n\nThe approach primarily builds on existing models, with the main contribution being the training pipeline; architectural or methodological novelty is slightly limited.\n\nTypo errors in the text, such as “Follwoing” [in Table 1], can be corrected.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We are happy to see your positive comments on experiments and method. We will release our data, code, and checkpoints once the paper is made public. We hope the following responses could solve your concerns.\n\n**1. About the Contribution.**\n\nThank you for your thoughtful comment. We understand the concern regarding architectural or methodological novelty, and we appreciate the opportunity to clarify the primary focus of our work.\n\nCurrently, there is no effective pretraining strategy capable of learning representations for such a diverse set of modalities. DFormer, for instance, was designed for two-modality (RGB-D) scenarios and cannot be directly applied to arbitrary multi-modal settings. As illustrated in Figure 4 of the main text, directly combining and optimizing all modalities simultaneously, in a manner similar to DFormer, introduces significant optimization challenges due to the complexity and heterogeneity of the data.\n\nTo address this issue, we propose an efficient multi-modal pretraining strategy, termed ImageNeXt pretraining. Instead of optimizing the model with all modality data at once, our method takes RGB images and a randomly selected supplementary modality as input at each iteration. This design reduces optimization difficulty while maintaining the ability to learn transferable cross-modal representations.\n\nWhile our approach builds upon existing backbone models such as DFormer, our main contribution lies in designing a general and scalable training paradigm for multi-modal pretraining. \nOur unified pretraining pipeline enables the model to learn from multi-modal data and supports flexible fine-tuning under arbitrary modality settings. This strategy enhances the model’s adaptability and generalization in real-world multi-modal tasks.\n\nAlthough DFormer serves as a backbone in some experiments, we also demonstrate that the same pretraining paradigm can be effectively applied to other architectures such as ResNet-101, suggesting that our method is not limited to a particular design but offers broader applicability.\n\nWe will revise the manuscript to better emphasize this motivation and the scope of our contribution. Thank you again for your valuable feedback, which has helped us improve the clarity of our work.\n\n\n\n**2. About Real-world Generalization and Backbone Choice.**\n\nThank you for your valuable feedback. We agree that exploring both real-world generalization and the dependence on backbone choice is important for fully understanding the effectiveness of our approach.\n\nRegarding real-world generalization, our pretraining strategy leverages synthetic multi-modal data to equip the model with the capacity to handle diverse modality combinations. We then fine-tune the model on real-world downstream datasets, covering a wide range of practical scenarios—e.g., NYU DepthV2 and SUN RGB-D for indoor environments, and KITTI-360 and MFNet for outdoor scenes. As demonstrated in our results, the pretrained models consistently improve performance across these real-world benchmarks, indicating good transferability.\n\nWe appreciate the suggestion and agree that further validation in real deployment scenarios would be beneficial. In future work, we plan to collect and evaluate on additional real-world multi-modal datasets to more thoroughly assess robustness in unseen conditions.\n\nIn terms of backbone dependence, our method is designed to be generally applicable and not tied to a specific architecture. In the current version, we have already evaluated our training paradigm with multiple popular backbones, including DFormer, ResNet, and MiT (SegFormer), as shown in Table 1. Across these backbones, our method brings consistent performance improvements. We will further expand our ablation studies to include more backbones and explicitly highlight these results to better demonstrate the generality of our approach.\n\nWe will conduct these additional experiments and incorporate them into the manuscript, which we believe will make the paper more convincing and solid. Thank you again for your helpful suggestions.\n\n\n\n**3. About Scaling to More Modalities and Domains.**\n\nThank you for raising this important question. \nOur approach is explicitly designed to scale to additional modalities and domains without architectural or procedural changes. In the ImageNeXt pretraining paradigm, all supplementary modalities are treated equally, without requiring any modality-specific architectural customization. When new modalities are introduced, we can simply expand the modality pool and perform random sampling across modalities during pretraining. This ensures that the model is exposed to diverse cross-modal combinations and learns to handle new modalities in a unified manner.\n\nFurthermore, we find encouraging evidence that the representations learned through ImageNeXt pretraining generalize beyond the specific modalities involved in training. As shown in Table 9 of the supplementary material, ImageNeXt pretraining significantly boosts performance on RGB + Polarization semantic segmentation, even though the polarization modality was not present during pretraining. We hypothesize that the multi-modal fusion patterns and generalizable representations learned during pretraining enable effective transfer to new modality pairs, demonstrating strong scalability and domain generalization capabilities.\n\nWe will clarify this point and include the corresponding analysis in the revised paper to better highlight the extensibility of our approach. Thank you again for the insightful question.\n\nThank you for pointing out the typographical issues. We will carefully proofread and polish the writing to correct such errors and improve the overall quality of the paper."}]

Author rebuttal:

Meta-review: The reviewers were in unanimous agreement regarding the acceptance of this paper in their initial reviews. The author's rebuttal effectively addressed all remaining concerns, maintaining the positive consensus. Overall, the paper is well above the NeurIPS standard. Therefore, the AC recommends acceptance.
Decision: Accept (poster)