UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation
Authors: Xiaoqi Zhao
Youwei Pang
Chenyang Yu
Lihe Zhang
Huchuan Lu
Shijian Lu
Georges El Fakhri
Xiaofeng Liu
Venue: NeurIPS 2025 poster ()
Keywords: Incomplete Modalities
Self-supervised Learning
Hierarchical Compensation
Primary area: deep_learning

Abstract: Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance. While existing methods address training-inference modality gaps via specialized per-combination models, they introduce  high deployment costs by requiring exhaustive model subsets and model-modality matching. In this work, we propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC). Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels. 
First, we adopt modality reconstruction with the hybrid shuffled-masking augmentation, encouraging the model to learn the intrinsic modality characteristics and generate meaningful representations for missing modalities through cross-modal fusion.  
Next, modality-invariant contrastive learning implicitly compensates the feature space distance among incomplete-complete modality pairs.  Furthermore, the proposed lightweight reverse attention adapter explicitly compensates for the weak perceptual semantics in the frozen encoder. Last, UniMRSeg is fine-tuned under the hybrid consistency constraint to ensure stable prediction under all modality combinations without large performance fluctuations. Without bells and whistles, UniMRSeg significantly outperforms the state-of-the-art methods under diverse missing modality scenarios on MRI-based brain tumor segmentation, RGB-D semantic segmentation, RGB-D/T salient object segmentation.  The code will be released at \url{https://github.com/Xiaoqi-Zhao-DLUT/UniMRSeg}.

UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation
Xiaoqi Zhao'
Youwei Pang
Chenyang Yu² Georges El Fakhri'
Lihe Zhang'
Huchuan Lu
Shijian Lu Yale University, USA
Xiaofeng Liu'
2 Nanyang Technological University, Singapore
Dalian University of Technology, China xiaoqi. zhao@yale.....oooooooo.. edu
Abstract
Multi-modal image segmentation faces real-world deployment challenges from incomplete/corapted modalities degrading performance. While existing methods address training-inferent modality gaps via specialized per-combination mod- els, they introduce high deployment costs by requiring exhaustive model subsets and modellmy matching. In this work, we propose unified modality-relaa segmentation network (UniMRSeg) through hierarchical self-supervised compen- sation (HSSC). Our approach hierarchically bridges representation gaps between complete and incomplete modalities across imput, feature and output levels. First, augmentation, we adopt modality reconstruction with the hybrid characteristics and generate encouraging the model learn the intrinsic modality meaningful representations for missing modalities through cross-modal fusion. Next, modality-invarian contrastive learning implicitly compensates the feature space distance among incomplete-comples modality pairs. Furthermore, the pro- posed lightweight reverse attention adapter explicitly compensates for the weak perceptual semantics in the frozen encoder. Last, UniMRSeg is fine-tuned under the hybrid consistency constraint to ensure stable prediction under all modality combinations without large performance fluctuations. Without bells and whistles, UniMRSeg significantly outperforms the state-of oftte--f-thhaatt methods under diverse missing modality scenarios on MRI-base brain tumor segmentation, RGB-D se- mantic segmentation, RGB-D/T salient object segmentation. The code will be released at htthttpp:///gg h/ggtttubb... com Xiaoqi Scha--LLU ALUT/UUnT /UniMRSeg
Introduction
Visual multi-modal image segmentation has become a cornerstone in critical applications such as autonomous driving 19, medical diagnostics (23, and robotics [67], where complementary
visual cues (e.g., RGB-D, MRI sequences) improve scene understanding. Advanced hybrid CNN-
Transformer 14 46
39
79
68, 40). global- attention (64] 66 81
82
84
78
75 41) multi-
dynamic convolution [35 77 76 techniques have achieved remarkable success under idcalized settings with complete modalities. However, real-world scenarios often suffer from incomplete modality inputs due to sensor failures, low-quality data and clinical constraints. For example, although ideal to use four complementary MRI sion recovery (Flair), contrast-enhanced T1- T1-weighted (Tice), T1 T1-weighted (T1), and T2- 12-weighted (T2) for brain tumor diagnosis, variations in scanning protocols and patient conditions may limit the ability obtain all MRI scans.
scale 38
36
37
inver-
Conresoonding Authors: Youwei Pang and Lihe Zhang
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ARGB
Dupel
UnidiSSag
Salient Object Segmandatiin
RdB/Dayth RGB Depth
RGB
RGS/Thermal
Semandic Stgmontation
Brain Tumar
RGB
Depeli
Fay Thurts Th Tiorts To Poarts ro to Male
Sigmentation
MRL-hased Brain Tamer Seginentation
KGBB-D Salient Object Segmentatioa
RGB-T Salient Object Separentzzion
RCiB-D Semante Separeitizion
03
52-
50-
60
70
80-
Figure 1: Topp UniMRSeg has unified framework and parameters within each segmentation task to handle 21 modality combinations (RGB- IRGB-Depth: 3, RGB-Thermal 3, MRI: 15). Bottom: Box plots compare UniMRSeg with existing methods across four benchmarks, displaying average performance (red dots) and standard deviation (error bars).
Recent research on addressing missing modalities falls into two main challenges. Firstly, most meth- 58
72 80 focus on designing adaptable cross-modal interaction to amalgamate multi-modal features while preserving generalizable architectures for single-modal scenarios. However, during inference, diverse modality combinations require separate model parameters 771 58 or independent encoder parameters 669 74,, which not only increase resource consumption practical deployment but also necessitate additional manual or automatic modality classification prerequisite. Although some works [25] S leverage the knowledge distillation from complete to incomplete modalities, they still demand multiple models for each modality subset, complicating clinical deployment. Secondly, modality reconstruction-bue
ods 71
25] 85] aim to predict missing modality inputs to align features during training and inference. Since segmentation requires precise spatial features and boundary information, the pre-trained reconstruction model prioritizes global feature compression, resulting in insufficient feature representation. Therefore. is difficult directly inherit the ability to reduce the modality gap obtained by input-level [80 25 or feature-level [69 S5
methods [80 69
reconstruction for downstream tasks (i.e., image segmentation). In particular, cascading low-quality reconstruction predictions (25 as input to the segmentation network will increase error propagation and degrade performance.
In this paper, we propose a unified modality-relax segmentation frame- work (UniMRSeg). As shown in Fig. [ UniMRSeg shares 100% of its parameters across all possible modal- ity input combinations in given segmentation task. Since complete modality inputs typically yield the best prediction, our goal is to ensure that UniMRSeg, after training, can ap- proach complete modality represen- tation quality during inference with arbitrary modality inputs. To achieve this, we fully exploit the power of self-supervision | in representation learning and propose a hierarchical compensation mechanism that operates at the input level, feature level, and output level, shown in Fig. 2 First, we adopt cross-modal reconstruction as pretext task. Unlike previous methods 80 that either discard complete modal- ities employ partial masking strategies, our approach simultaneously applies global and local masking mechanisms. This dual masking design enables the model to capture both ine-grained local patterns and holistic semantic representations from intra intra-modality and cross- mods-it interactions.
Ilierarciicll Compensation
Input Level
Feature Level
3
Output Level
TurTa
TDP TOMFLAR
Figure 2:
Illustration of the hierarchical compensation.

---

Furthermore, we introduce channel-wise modality shuffling operation that deliberately breaks the correspondence between imput and reconstructed modalities. This operation implicitly formulates a modality classification task, compelling the model to disentangle modality-speciffe characteristics while eliminating dependence on modality category priors during inference. Next, we leverage contrastive learning to finish the feature-level compensation. Specifically, we construct complete and randomly missing modalities from the same sample as positive pairs, and those from other samples negative pairs. To enhance the inheritance of representation for downstream segmentation tasks 33 7 we jointly optimize the spatial distance metric and object segmentation at this stage to guide feature clustering in a direction that is beneficial for the results of the segmentation. Consid- ering the inevitable prediction errors in the aforementioned pixel-level reconstruction and feature contrast learning, they may still limit the boundaries of the compensation mechanism. Inspired by adapter-based methods [26 61 we design a lightweight reverse attention adapter to explicitly compensate for weak perceptual semantics in the frozen encoder, while embedding 31) Swin Transformer [28 52 to capture high-response mutual attention patterns across modalities. By adding feature-level consistency constraints, we ensure that the adapter is aware of the partial representation defects inherent in any missing modal combination. Last, UniMRSeg is fine-tuned by enforcing segmentation consistency constraints, and the knowledge presented by the complete modality the output will be distilled to all missing modality combinations as supervision information. Through the hierarchical three-level compensation strategy. UniMRSeg achieves optimal average performance, the highest full- mol--modlltty performance, and minimal performance variance, as shown in Fig.
Our main contributions be summarized as follows:
We propose a unified framework, i.e., UniMRSeg, with one set of parameters adapting to varying modality- scenarios for general multi-modal segmentation.
We integrate pixel-level modality reconstruction, feature-level contrastive learning, and prediction- level label distillation to construct hierarchical annotation-free compensation mechanism, breaking through the previous isolated self-supervised research paradigm.
Benefiting from the reverse attention adapter, UniMRSeg can explicitly obtain the compensation about difficult regions with weak perception of the complete modality, aligning missing and complete modality representations.
Extensive comparisons conducted on brain tumor, salient object and semantic segmentation tasks in MRI (15 combinations), RGB-D (3 combinations), and RGB-T (3 combinations) modalities across 2D and 3D images within medical and natural scenes, show that our method consistently achieves the best performance in all individual modality combinations while attaining superior average accuracy with minimal standard deviation.
2 Related Works
2.1 Incomplete Multi-modal Image Segmentation
There are three popular research patterns to handle missing modalities situations. I) Generalizable Architectures. Some methods [58] 71 73 aim to fully integrate multi-modal features while min- imizing structural modifications in single-modal scenarios. CMX (7]) introduces cross-modal interaction attention mechanism that combines both the CNN and Transformer. Tokenfusion (58] focuses efficient token-based fusion, pruning multiple single-modal Transformer and repurposing the pruned units for multi-modal fusion. II) Projection-basee Methods. Hetero [1 and HeMIS [T5 perform arithmetic operations (e.g., averaging) in the projected space to obtain the final segmentation result. SFusion [30] proposes a self seff-attention-based fusion block, where extracted features from available modalities are projected as tokens and processed through a self-attentin
layer to capture cross-modal relationships. III) Reconstruction Strategies. M3AE (25) refines the segmentation network by reusing the reconstructed modality images inputs for fine-tuning. MaskMentor (85 and SSLSOD [80) treat modality reconstruction as pre-training representation task, where missing and complete modalities share weights for joint training. Zeng et al. [69] enforce consistency between missing modality feature reconstruction and complete modality features at the token level. Different from them, we aim to achieve a simple yet efficient architecture while completing the comprehensive compensation of multi-modal and multi-combbnatorial representations with unified parameters by constructing multi-granularity self-supervised tasks that are not limited to pixel/token-leve recon- struction.
3

---

2.2 Self-supervised Learning
Self-super learning (SSL) has emerged as powerful paradigm for learning representations without relying on manual annotations. Three principal SSL, mechanisms dominate current research: 1) Mask-based SSL involves training models to predict missing masked parts of an input. Early image inpainting works [42 700 evolved into modern masked autoencoders [16] 60] using high-ratio
masking and asymmetric encoder-decoder architectures to various visual tasks, including image segmentation, depth estimation, and low-level image restoration. II) Contrastive learning encourages models to bring similar data points closer while distancing dissimilar ones. SimCLR [4] establishes data augmentation principles with momentum encoders, while MoCo series [17 6 address negative sample scarcity through dynamic queues. CLIP 44 leverages cross-modal contrastive learning to embed images and text into a unified semantic space, enabling zero-shot transfer capabilities and providing universal representation foundation for multi-modal tasks like medical image retrieval and caption generation. III) Knowledge distillation [99] transfers supervision via teacher-student paradigms. In image segmentation, Chen et al. [49 propose normalize the activation map of each channel to obtain a soft probability map. For cross-modal scenarios, Gupta et al. [13] transfer supervision from labeled RGB images to unlabeled depth and optical flow images. Existing studies usually develop these paradigms in isolation. We hope to take the incomplete multi-modal image segmentation task as an opportunity to effectively combine these three different levels of self- self-supervision techniques and demonstrate their synergy.
3 Approach
Preliminaries. In this section, we adopt the MRI-based brain tumor segmentation with the T1, Tlc T2 and Flair modalities the targeted task to describe the proposed multi-stage learning framework. Let E g xx xT' be the input modality-speciffc sequence with '' slices for the model, where 1, H. and W are the channel, height, and width of the slice. The final model generates the segmentation tensor P E IR x x1' where each channel corresponds to specific class and indicates the probability of each spatial-temporal location being assigned that class.
Multi-stage Learning Framework. Our framework aims to bridge the performance gap between complete and incomplete multi-modal inputs through three-stage progressive learning, specifically designed to empower flexible handling of diverse missing modality scenarios while maintaining segmentation robustness. The overall pipeline is shown in Fig. The basic models in all stages follow a unified 3D U-Net-style 45) encoder-decoder structure and embed a 3D ASPP [ with dilation rates of [1, 6. 12, 18] into the high-level feature. The following sections elaborate on the technical specifics of each learning stage.
3.1 Multi-granular Modality Reconstruction
This stage emphasizes enhancing the representation capabilities under diverse potential input-side missing modality scenarios. Existing work [16] has demonstrated that effective data perturbations facilitate learning more robust representations. To encourage the model to mine implicit contextual relationships across modalities, this stage integrates three strategies, including modality dropout, modality shuffle and spatial masking, to enable multi-granular information reconstruction.
Data Perturbation. For the complete multi-modal input, we first apply the random modality dropout strategy to randomly discard some modalities with a 50% probability and generate the output. And it also preserves at least one modality, thus retaining the fundamental information for overall reconstruction. Furthermore, mitigates the model's reliance on fixed modality order and further decouples modality-agnostic repre- sentations from their inherent sequential dependencies in existing paradigms the following spatial masking strategy randomly masks a portion of the input data, thus simulating missing effects within the sequences from available modalities.
randomly shuffle the order of remaining modalities, which
[85] 25). Additionally,
Data Reconstruction. We input these remaining perturbed samples into the 3D U-Net-based reconstruction network, and obtain the output through a Rel.U function. The normalized slices from the original complete modalities are used as reconstruction objectives based on the combination loss of L1 and SSIM [59]. thus overall achieving self- supervuper pre-training in the first stage.

---

Stage I: Multi-gramular Modality Reconstruction
Stage 3: Incomplete M Modality dapliie Fine- mue-tuning
1 2
Complete
Excoder
Decoder
Incomoiicte Modality
Complate M Modality
Random Modallty Dropout
Hyyrii Carsisteucy Comtrraitt
Emcoder
Decoder
Randum Masking
II Random Modally Shutile
Encoder
Decoder
tteution dapter
Complete Modality
Stage 2: Modality-invariant Contrastive earning
Load [hsights
Encoder
Decoder
Operaalns
Hoseliy
Modiilty
0000
Bandon
M sdality Dropest
NT-Yeal
Aoseliy
Meslishy
Can BN BxLU
Ehimati-tte buttractinn
Sepervidin:
Hlment Haltpiattio
Figure 3: Multi-stage learning framework built on our encoder-decoder network. Stage I: complete modality reconstruction based on multi-gran gulti-graanlar random perturbations. Stage 2: modality-invane invariann contrastive learning for enhancing adaptive fine-tune tuning via hybrid consistency constraints.
modality representation. Stage 3: incomplete modality
3.2
Modality-invarian Contrastive Learning
The core idea behind this stage is to implicitly compensate for the feature biases introduced by missing modalities through contrastive learning.
Representation Construction. Without loss of generality, we consider a batch containing two input samples with complete modalities, iee,, 1 and 22. By applying random modality dropout to each of them while ensuring at least one and at most three modalities remain active, can obtain two extended samples (i₃ and I2) with missing modalities. And then, initialize the segmentation model by load the weights of the first stage except for the last output layer. (71, f1 T2, f2) then fed into the segmentation model to extract multi-level encoder features and generate the final predictions {Pi, P2, P2, Pa). respectively. After the global average pooling, those features are converted into four vector sets {fi) ( (f2)= and {fa) respectively. They are utilized for the following representation contrastive learning.
Contrastive Learning, The aforementioned representation vectors are utilized to construct positive- negative sample relationships. And introduce the NT-Xent loss 4 in each feature level to minimize distances between positive pairs and maximize separation of negative pairs. Specifically, for the 1t level, positive pairs are from the same input sample and its augmented variant, i.e., I and I (k (1)) Negative pairs from distinct input sources, e.g., (41, I2) and (2). Considering the vector set (Si, fi. fi. f we can calculate the NT-Xent loss follows:
exp(sim (C. f.)TT)
F(u, v) - log
Σx
218 -
[[ exp((inn sim sim(fi (1)
(2k 1, 21) + (2k, 2h:
LNT-Xent
2B 5
i=1
Rm
where and sim are the batch size and the cosine similarity. ][a is an indicator function that results in when l: of otherwise O. Such contrastive learning encourages the model to learn modality-invarian representations, focusing on the underlying target semantic features rather than being biased by modality-speciffc characteristics.
5

---

Segmentation Constraint. The common Dice loss lise is used to supervise the segmentation predictions as follows:
1 B Dice (P, Σ
Diice B PE(PLAA)
It is worth noting that despite applying the contrastive learning. the segmentation constraint is also necessary to guide the encoder's representation learning. This ensures the learned features are optimized and aligned for our ultimate objective, i.e,, the segmentation task.
3.3 Incomplete Modality Adaptive Fine-tuning
Due to potential errors in the reconstruction from the first stage and the positive-negative sample distance control in the second stage, we introduce the following adaptive fine-tuning guided by the complete modalities dynamically compensate for these representation errors. As shown in Fig. 3] our framework implements parallel pipelines to handle complete modality samples and their all potential incomplete counterparts (14 valid variants for 4 MRI modalities). The model directly generates the ideal intermediate features { [=") and segmentation prediction /' as the reference from complete modality samples via the forward propagation. And its prediction P is supervised by the GT mask using the Dice loss. The incomplete samples additionally require targeted optimization through lightweight adapters { A Ji=1. which progressively aligns their features {F+ and prediction / with those references. maintain the representation capabilities pre-trained from the first two stages, we freeze the entire encoder during this stage and only fine-tune the decoder and adapters.
Ji=1
Taking the 7th encoder stage as an example, we attach the adapter component to the frozen encoder stage E'. In the feature propagation path for incomplete multi-mode modal samples, the feature fi-1 from the previous encoder layer is first processed by E' to obtain the base features Fr Meanwhile, fi-1 also undergoes 3D convolutions and output the initial adaptive feature F Fd which then integrated with Fr via dlement-wise addition and sequential 3D convolutions. The generated feature Fý is fed into a 3D Swin Transformer 22 block to establish global contextual correlations between the F and Fi Fada-in- And then the global average pooling applied across channel and sequence dimensions, followed by the sigmoid to generate the mutual attention. We hope to capture the difficult semantic parts that cannot be perceived by the first two stages and then compensate Fip Therefore, we apply the reverse operation to the mutual attention map and generate the reverse attention map. And it is multiplied F and generate the adapted feature Fi Fada. which highlights differential information between modality-compii and -incomplete representations. Finally, the sum f of f and Ey replaces original Er as the real input to the subsequent process. The roles played by the above features need be activated with the help of feature-level consistency constraints.
Reverse Attention Adapter.
op
Hybrid Consistency Constraints, The consistency constraint process depicted in Fig. involves the levels of encoder features and final predictions. For the feature-level consistency, we compare the intermediate features { {Fi) } from modality-complete samples and (fr) [f*) compensated by the adapter from modality-inoomplte samples:
M
1
T.fc
F
(4)
B
1=1
where . denotes the number of potential valid -nnoopit-incommpete samples and it is 14 in our MRI experiments. Besides, the prediction-level consistency between the predicted segmentation maps from modality-complete and incomplete samples can be formulated by:
BB
1
Lpc
Dice (Ps, Pt Pk..)
(5)
By accumulating these consistency constraints, the parameters of the decoder and adapters are jointly optimized to minimize the difference between the representations from modality-complete and -incomplete samples.

---

Table I: Quantitative comparison of brain tumor segmentation on BraTS2020 32 1 and indicate that the larger scores and the smaller ones are better, respectively. Note: Official implementations used when available. All methods are evaluated under a unified missing modality setting when possible. Some methods use different training data and lack released code, making exact re-training infeasible. Extended results under various settings are in the Appendix.
Modility
TURDe (Willc)
sconee))
(Crec)
Ethancirg Stasion
(Ethancing! Shucppe PASSION
Mair Vice
Sitason
[50
SheSpac PASSION
531
[6]
Stasinn
[30
Shudppe
[5]]
BASSSON
[5
Nosted
[3
UnViikSag
23:22
58.54
63.10
54.83
75.14
5.36
37.30
42.33
50.36
56.31
2.31
16.34
22.15
50.23
35.19
24.61
60.52
57.31
50.92
68.57
27.40
45.19
$4,59
$4.75
79.20
33.85
44.34
64.95
60.10
79.20
6:15
$5.00
55.40
$6.10
69.41
6.77
39.64
34:36
19.91
45.70
1.55
14:13
18.25
22.
25.70
61.18
74.25
79.54
77.35
85.35
54,01
43.23
52.74
50.27
61.51
29.84
30.92
37.40
38.60
45.1
47.22
72.M1
70.00
75382
61.44
00/02
71.82
75.33
50.35
52.01
49.13
06.13
74,23
79.02
79.56
40.87
60.50
67.58
67.90
72.95
52.34
75.62
76,92
79.85
$4.60
56.87
66.00
77.13
78.32
90.23
70:54
$4.40
50.35
$2.90
66.22
47.36
60.20
50.55
07.90
31.61
16.5
48.18
46.05
51.34
22.37
67.55
21.40
77.21
77.96
13.15
42.90
$2.80
$1.60
59.71
1.26
12.20
36.02
$3.42
41.49
35.02
63.53
25.40
32.09
43.52
43.34
53.12
M2.80
87.96
76.52
78.94
54.05
09. 68
77.28
79.05
81.59
79.34
74.36
76.02
78.12
81.28
76.93
7%
M0.05
73.3M
77.15
07.03
78.64
16.26
71.78
75.50
Ansrapo:
14.81
M0.44
95.99
65.30
16.05
15.35
00.78
48. 14
35.01
53.91
63:30
Dev
23,09
10.47
10008
10.07
5.43
19:53
17.00
15.45
15.34
13:04
24.20
21.80
23.62
21.84
19.56
Table 2: Quantitative comparison of segmentation performance across RGB, Depth and Thermal modalities.
Modality
MOR--D Salieme Clhpas Sepnemiatinn (STERF 14 ROB-T Salemi Segmnntatian (V (VTIO 15
WHH-D Sementti Negrnemntton (ST ISUTN-BCIRD [4)
RCM Depth/hhomm
SSLSOD CAVER Pog/er ConnNet
100
[2]
823
UniMiSSeg
SSLS5D
[80]
L.SNer CANER CONTRINET Unik3KSeg
[5]]
(3]
Tosen/uiion
CMX CMMNNeX MaxiMMenor UniMiRSeg
[33]
772
453
861
.554
41.2
917
917
919
.30
M13
.929
938
51.0
52.4
51.9
53.0
53.2
Average
K21
788
M26
3933
350
M10
X59
X39
399
95.6
08.3
47.2
4K0
Std Dev
.000
J38
157
.137
J447
.076
.144
.10
.098
.47
5.4
49
5.6
6.1
3.1
4 Experiments
4.1 Datasets and Metrics
In this work, we conduct experimental comparisons on four popular visual multi image seg- mentation tasks to show the generalizability of the proposed method. 1) Brain Tumor Segmentation. We follow most brain tumor segmentation methods 122 63
TO, 30 use the BraTS2020 dataset (32), which contains 369 images with four modality scans: Tlee, T1, T2, and Flair, along with three annotated regions: enhancing tumor, tumor core, and whole tumor, which are mutually inclusive. The dataset is split into training (315), validation (17), and test (37) sets. II) RGB-D Salient Object Segmentation. We adopt the same training set as most methods [8 82 39 i.e., 1,485 samples from the NJUD (0) and 700 samples from the NLPR [4). The test dataset STERE 34], which contains 1,000 RGB and depth image pairs with complex scenes. III) RGB-T Salient Object Segmentation. We follow the setting of recent works 33 86 [1 the training set only contains the 2,500 samples from VT5000 [53] and adopt the V01000 [54] as the test set which contains 1,000 pairs of RGB-T images including more than 400 kinds of common objects collected in 10 types of scenes under different illumination conditions. IV) RGB-D Semantic Segmentation. SUN-RGBD (50] is the popular indoor scene benchmark with 37 classes. It contains 10,335 pairs of RGB-D images, with 5,285 pairs allocated for training and 5,050 for testing. We introduce some widely used metrics in each field for fair evaluation, including Dice for brain tumor segmentation, S S-measure 12 (S,,) for salient object segmentation and IoU for semantic segmentation.
4.2 Implementation Details
All experiments are conducted on one NVIDIA A800 GPU. We adopt basic image augmentation techniques to avoid overfitting, including random flipping. rotating and border clipping. We train the model for 300 epochs based on the Adam W optimizer [3I] with a warmup schedule, an initial learning rate of 0.0001, and a weight decay 0.00001. For RGB-D and RGB-T tasks, we concatenate the inputs into a 4-channel tensor to maintain the single-stream architecture. For fair comparison, we separately adopt ResNet-50 [8] and ConvNext-B [27] as the backbone for RGB-D/T salient object segmentation and RGB-D semantic segmentation, which are widely used in their respective fields.
7

---

PM
FB
(a) MRI-based Brain Tumor Segmentation
(b) RGB-D Semantic Segmenta- tion
(c) RGB-D Salient Object Seg- mentation
(d) RGB-T Salient Object Seg- mentation
Figure 4: Qualitative comparisons of predictions from different methods across different modality combinations. Best viewed on screen.
4.3 Evaluation
Quantitative Results. We conduct thorough comparisons on all the four tasks, as shown Tab. and Tab. 2 All these models are either directly tested for performance or retrained based on their publicly released code. To simulate missing modalities in practice and keep consistent with training, fill missing modalities in MRI with zero pixel values and directly copy other existing modalities to complete the dual input of any missing modality in RGB-D and RGB-T tasks. In addition, we follow cascade the intrinsic depth predictor as the input of the models for RGB-D salient object segmentation and RGB-D semantic segmentation tasks. It can be that our proposed UniMRSeg achieves excellent performance the best average performance and lowest standard deviation. These high accuracy, strong robustness and generalization capabilities across multiple modalities show its potential as unified, reliable, and efficient segmentation framework.
some methods with depth estimation functions 80 62 g5
all tasks with
Qualitative Results. In Fig. 4a PASSION 48 exhibits significant prediction fluctuations for enhancing tumor (blue) and tumor core (red) under missing modalities, while UniMRSeg maintains highly consistent predictions aligned with ground truth. TI and Flair modalities notably degrade PASSION's boundary discrimination for whole tumor (green), revealing mutually exclusive fusion deficiencies. In Fig. 4b UniMRSeg achieves precise door segmentation (red) by fusing color and geometric features across RGB/RGB-D modes, whereas CMX 771 fails completely with depth-only inputs. Leveraging spatial topology of chairs, UniMRSeg robustly segments blurred wall-adjacent boxes using pure depth data, demonstrating superior shape reasoning. Fig. 4 shows PopNet [62] produces identical spherical predictions in RGB/RGB-DD modes via embedded depth prediction, but degenerates to depth map binarization without RGB. In contrast, UniMRSeg reconstructs RGB semantics from depth and fuses cross-modal features to accurately segment humans and spheres. Fig. 4d verifies thermal modality's advantage in resolving RGB's 's color-similarity-nnuue sticking issues. Remarkably, UniMRSeg achieves complete target segmentation using thermal data alone.
4.4 Ablation Study
In this section, show the effectiveness of each component the brain tumor segmentation task with the most complex modality combination. The baseline model 3D-UNet without any pre-training.
Each Components. As shown in Tab. the three data perturbations in stage demonstrate the effectiveness of learning intra- and inter-modal features for guuti-granuulr reconstruction. Their accumulation during pre-training leads to over 14.7% performance gain over the baseline subse- quent segmentation. Stage 2 highlights the importance of jointly training the segmentation decoder and the encoder guided by spatial distance clustering, which promote each other. Stage further optimizes cross- modal alignment through reverse attention adapters and prediction-level consistency

---

Table 4: Ablation study of the reverse atten- tion adapter.
Table 3: Ablation study of each component.
Mixill
Core
Uahancing
22101
1110
Baseline
$1.00
38,40
A/l Rerver AbankeeMaaaal Absriie
18:13
06.8
66.33
Stage J. Mall granaler Medimit aeconsoracioo
ISc [.2]
+ Random Modality Dropout
66.58
55.47
42:25
1336
0015
90.0
Random Modality Shuffle
67.78
56.85
44.17
Table 5: Evaluation of different compensa- tion levels.
Random Spitial Masking
69.35
59.89
47.12
Modulity
Comruntiv
Learing (Unooder)
51.45
Segutestation Const (Decodet)
74,33
FERT 13
OREST LoG
Ancrig
Sugge J: Jncompier Mediality Adagtine Fine using (Sec
Whale
4 FFanire- 1 zvri Comistoncy (Adigeer)
78.2
00.38
90.25
62.00
20.22
+ Prediction: Level C Consisionty (Segmentaaion
80.64
73.35
63.10
Threr-Sigg Deeign Unijedd Suggle Stage
1500
19700
1971
Turre
8064
73.35
03.10
13
63.65
54.55
Unifled Single
10.08
44
04
84
0.3
02-
0.
(a) Baseline
(b) Stage
(c) Stage
(d) Stage 3
Figure 5: Average response value of feature maps of each encoding layer under different modal combinations at different stages.
constraints. The latter alone surpasses the stage decoder by 12.0%, showing the importance of enforcing prediction consistency across modalities. Gains in each stage are consistently and gradually improved, with the final model outperforming the baseline by over 44.6% average.
Three-Stage Design vs. Unified Single Stage. UniMRSeg is built upon a supe-suee pretraining framework, with training initialized from scratch starting at Stage 1. For each task (e.g.. BraTS 2020), self-supervised training is conducted solely on the task's own training set, rather than relying on externally labeled datasets such as ImageNet-pretraing design includes: 1) Pretrain both the encoder and decoder through arbitrary modality reconstruction. 2) Pretrain the encoder with contrastive learning. The segmentation task here is only used to guide the contrastive objective, not as a final goal. 3) Perform the downstream segmentation task. In this way, our three-stage design explicitly aims to reduce the representation gap between complete and incomplete modalities in the encoder-decodes space during the final segmentation stage. If all these tasks are trained jointly in a single-stage model, it would fall into the scope of multi-task learning. rather than self-supervised pretraining.
weights for initialization. Our three-stage
In multi-task learning, the goal is to leverage multiple complementary clues for collaborative learning. These clues can come from the data level [1 56 77). or from structural supervision types [47] 24 65
Most approaches often incorporate deliberate designs for task-sharing and task task-specific components to enable effective joint prediction across tasks. the three stages are forcibly merged into a single training stage, two major issues will arise: 1) The model is trained with inputs that involve Random Modality Dropout, Random Modality Shuffle, and Random Spatial Masking. The encoder is supervised using the NT-Xent contrastive loss, adapter is incorporated, and the model is simultaneously tasked with both segmentation and reconstruction. Such a fully entangled training setup lacks a clear task hierarchy, and there is no explicit coordination among the input, encoder, decoder, and output. 2) When all tasks are treated equally without ordering, joint optimization becomes highly difficult. the unified single-stage training attempt, the total loss involves six parts. As shown in Tab. 3 we compare the three-stage model with the unified single-stage model. We observed two phenomena during the experiments: 1) The single stage model failed to converge, with loss plateauing early. 2) The optimization process was highly unstable, with different losses fluctuating turn and failing to decrease consistently together. These phenomena and performace clearly indicate that unified single stage training is unable to achieve effective coordination among the various designs and supervision signals. The lack of a clear training order, combined with competing optimization objectives, leads to mutual interference.

---

Reverse Attention Adapter. Tab. 4 shows ablations on the reverse attention adapter. There are three key findings: 1) Removing reverse attention alone leads a 4.2% average drop, confirming its role in semantic compensation for missing modalities. 2) Subsequent elimination of mutual attention yields negligible performance variation, indicating that only emphasizing high-response regions between adapter layers and frozen encoding layers cannot achieve meaningful feature compensation. 3) Replacing the 3D Swin Transformer with a 3D convolution of similar parameters significantly degrades performance, validating the Transformer's advantage in cross-modal correlation modeling. Additionally, we compare freezing vs. fine-tuning the encoder during stage 3. Fine-tuning causes over 6.4% performance drops, indicating that it destroys the adapter-encoder synergy and the compensation mechanism relies on stable encoder representations rather than task-specific tuning.
To further explain this observation, we provide two perspectives: the inheritance between Stage 2 and Stage 3, and the rationale behind the lightweight reverse attention adapter (RAA). 1) Unlike general SSL methods 4 17 [6] that focus on learning generic representations without targeting specific downstream tasks, our contrastive learning in Stage task-aware. We co-train the segmentation head to guide the encoder toward modality-invariant features that are directly beneficial for segmentation, rather than general-purposs representations. This task-guided design ensures that the learned contrastive space aligns with the downstream segmentation objective. Therefore, fine-tuning the encoder in Stage 3 would undermine the task guided contrastive representations that were carefully established. 2) The RAA itself is designed as residual correction bridging the encoder representation gap between incomplete and complete modality inputs. Formally:
fine + A(fin)) Re feom-
(6)
where fine denotes encoder features from incomplete modalities, from represents encoder features from complete modalities, and AA()) is the learnable adapter. During Stage 3, both fine and feom are frozen, while only A is trained. This constrained setup ensures that the adapter focuses purely on compensating missing information, enabling stable and efficient optimization. Once the encoder is unfrozen, however, all three components become variables, making their roles unclear and the optimization unstable. In conclusion, freezing the encoder in Stage 3 is a deliberate and necessary design choice to preserve the task-guided contrastive representations learned carlier. The RAA thus works in synergy with a stable encoder to effectively compensate for missing modalities and maintain robust segmentation performance.
Hierarchical Compensation Mechanism. As shown in Tab. 5] single-level compensations each surpass the baseline. Cross-level combinations reveal nonlinear synergy. Notably, merely integrating input- and feature-level compensations outperforms existing methods in Tab. [ The fully compen- sated model achieves 36.2% average gain over baseline across categories, demonstrating consistent collaborative synergy among all three levels without mutual exclusion. To quantify modality rep- resentation gaps, Fig. 5 visualizes the average feature activation values across different encoding layers under various modality combinations. Fig. 5a displays the baseline visual distributions, where single-modality, dual-modality, and triple- combinations exhibit pronounced representation gaps compared to the full-modality setup. Fig. 5b shows results after stage pre-training, where the compensation capability for Linglemmddlin inputs is significantly enhanced. Fig. 5c illustratees the results under the joint constraints of contrastive learning and segmentation tasks, where the representations of all the modalities are further aligned. In Fig. [d the reverse attention adapter in stage 3 effectively compensates for cumulative errors from modality reconstruction in stage 1 and spatial discrepancy differentiation in stage 2, using only a small number of parameters.
5 Conclusion
In this work, we propose a novel modality-relax segmentation framework based on a hierarchical self- supervised compensation strategy. Through multi- granularity data perturbation, segmentation task- guided feature distance constraints, and the design of a reverse attention adapter, we simultaneously integrate three different self self-supervised techniques into the proposed UniMRSeg model and complete the representation compensation of the missing modality from the input level, feature level, and output level. UniMRReg is simple yet effective, achieving dominant performance in four different multi- modal image segmentation tasks. We hope that this research paradigm focusing on representation- level compensation can inspire more visual tasks that require modality-relat conditions in the future.
10

---

This work was supported by the National Natural Science Foundation of China under Grant 62431004, and by Dalian Science and Technology Innovation Foundation under Grant 2023JJ12GX015.
Acknowledgements::
References
[1] R. K. Ando and T. Zhang. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov).1817-1853, 2005.
[2] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. Multimae: Multi-modal multi-task masked autoencoders, In Proceedings of European Conference on Computer Vision, pages 348-367, 2022.
[3] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:834 848, 2017.
[4] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning. pages 1597-1607, 2020.
[5] T. Chen, L. Zhu, C. Deng, R. Cao, Y. Wang, S. Zhang, Z. Li, L. Sun, Y. Zang, and P. Mao. Sam-adapter: Adapting segment anything underperformed scenes. In Proceedings of IEEE International Conference on Computer Vision, pages 3367-3375, 2023.
[6] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXhv:2003.04297, 2020.
[7] Y. X. Chng, H. Zheng, Y. Han, X. Qiu, and G. Huang. Mask grounding for referring image segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages
2024.
[8] Y. Choi, M. A. Al-Masni, K.-. Jung, R.-E. Yoo, S.-Y. Lee, and D.-H. Kim. A single stage knowledge distillation network for brain tumor segmentation on limited mr image modalities. Computer Methods and Programs in Biomedicine, 240: 240:107644, 2023.
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pages 3213-3223, 2016.
[10] Y. Ding, X. Yu, and Y. Yang. Rinet: Region-awarr fusion network for incomplete multi-mode brain tumor segmentation. In Proceedings of IEEE International Conference on Computer Vision, pages 3975-3984, 2021.
[11] R. Dorent, S. Joutard, M. Modat, S. Ourselin, and T. Vercauteren. Hetero-modal variational encoder-decoder for joint modality completion and segmentation. Medical Image Computing and Computer Assisted CnrveentionMMICCAA 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part II 22, pages 74-82. Springer, 2019.
[12] D.-P. Fan, M.-M. Cheng, Y. Liu, T. Li, and A. Borji. Stracture-measure: A new way to evaluate foreground maps. In Proceedings of IEEE International Conference on Computer Vision, pages 4548 4488-455,.200 2017.
[13] S. Gupta, J. Hoffman, and J. Malik. Cross modal distillation for supervision transfer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 2827- 2836, 2016.
[14] A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, H. R. Roth, and D. Xu. Unetr: Transformers for 3d medical image segmentation. Proceedings of IEEE/CV Winter Conference Applications of Computer Vision, pages 574-584, 2022.
[15] M. Havaei, N. Guizard, N. Chapados, and Y. Bengio. Hemis: Hetero-modal image segmentation. In Proceedings of International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 469-477, 2016.

---

[16] K. He, X. Chen, S. Xie, Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vi- sion learners. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[17] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 9729-97299 9738, 2020.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[19] G. Hinton, O. Vinyals, and Dean. Distilling the knowledge neural network, arXiv preprint arXiv:1503.02531, 2015.
[20] R. Ju, L. Ge, W. Geng, T. Ren, and G. Wu. Depth saliency based on anisotropic center-surround difference. In Proceedings of International Conference on Image Processing, pages 1 1115-1119, 2014.
[21] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, al. Segment anything. In Proceedings of IEEE International Conference on Computer Vision, pages 4015-4026, 2023.
[22] A. Larrazabal, C. Martínez, J. Dolz, and E. Ferrante. Orthogonal ensemble networks for biomedical image segmentation. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 594-603, 2021.
[23] D. Li, B. Yang, W. Zhan, and X. He. Multi-category graph reasoning for multi-modal brain tumor segmentation. In Proceedings of International Conference on Medical Image Computing and Cosputer-Assisted Intervention, pages 445-455 2024.
[24] B. Lin, W. Jiang, P. Chen, Y. Zhang, S. Liu, and Y.-C. Chen. Mtmamba: Enhancing multi-task dense scene understanding by mamba-based decoders. In Proceedings of European Conference on Computer Vision, pages 3 34--330, 2024.
[25] H. Liu. D. Wei, Lu, Sun, L. Wang. and Y. Zheng. M3ae: Multimodal representation learning for brain tumor segmentation with missing modalities. In Proceedings of AAAI Conference on Artificial Intelligence, pages 1657-1665, 2023.
[26] W. Liu, X. Shen, C.-M. Pun, and X. Cun. Explicit visual prompting for low-level structure segmentations. In Proceedings OEEE Conference on Computer Vision and Pattern Recognition, pages 19434-19445, 2023.
[27] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 1 11976- 1 11986, 2022.
(28) 7. Liu, J. Ning. Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu. Video swin transformer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3202- 3211, 2022.
[29] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu. Video swin transformer. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3192- 3201, 2022.
[30] Z. Liu, J. Wei, R. Li, and J. Zhou. Sfusion: Self-attention based n-to-one multimodal fusion block. In Proceedings of International Conference on Medical Image Computing and Computer- Assisted intervention, pages 159-169, 2023.
[31] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
[32] B. H. Menze, A. Jakab, S. Bauer, J. Salpathy-Cramer K. Farahani, J. Kirby, Y. Burren, N. Porz, Slotboom, R. Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). 34(10):1933-2044 2014.
12

---

[33] M. Mistretta, A. Baldrati, L. Agnolucci, M. Bertini, and A. D. Bagdanov. Cross the gap. Exposing the intra-modal misalignment in clip via modality inversion. In Proceedings of International Conference on Learning Representations, 2025.
[34] Y. Niu, Y. Geng, X. Li, and F. Liu. Leveraging stereopsis for saliency analysis. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition, pages 454-461, 2012.
[35] Y. Pang, L. Zhang, X. Zhao, and H. Lu. Hierarchical dynamic filtering network for rgb-d salient object detection. In Proceedings of European Conference on Computer Vision, pages 235-252, 2020.
[36] Y. Pang, X. Zhao, T.-Z. Xiang, L. Zhang, and H. Lu. Zoom in and out: A mixed-scale triplet network for camouflaged object detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 2160-2170, 2022.
[37] Y. Pang, X. Zhao, T.-Z. Xiang, L. Zhang, and H. Lu. Zoommext: unified collaborative pyramid network for camouflaged object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[38] Y. Pang, X. Zhao, L. Zhang, and H. Lu. Multi-scale interactive network for salient object detection. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition, pages 9413-9422, 2020.
[39] Y. Pang, X. Zhao, L. Zhang, and H. Lu. Caver: Cross-modal view-mixed transformer for bi-modal salient object detection. IEEE Transactions on Image Processing, 32:892-908 20,4,, 2023.
[40] Y. Pang. X. Zhao, L. Zhang, and H. Lu. Comptr: Towards diverse bi-source dense prediction tasks via a simple yet general complementary transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[41] Y. Pang, X. Zhao, Zuo, L. Zhang, and H. Lu. Open-vocabulary camouflaged object segmenta- tion. In Proceedings of European Conference on Computer Vision, 2024.
[42] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016.
[43] H. Peng, B. Li, W. Xiong, W. Hu, and R. Ji. Rgbd salient object detection: benchmark and algorithms. In Proceedings of European Conference on Computer Vision, pages 92-109, 2014.
[44] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. 2021.
In Proceedings of International Conference on Machine Learning, pages
[45] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of International Conference Medical Image Computing and Cosputer-Asiitted Intervention, pages 234-241, 2015.
[46] D. She, Y. Zhang, 7. Zhang, H. Li, 7. Yan, and X. Sun. Eoformer: Edge-oriented transformer for brain tumor segmentation. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 333-343 2023.
[47] H. Shi, S. Ren, T. Zhang, and S. Pan. Deep multitask learning with progressive parameter sharing. In Proceedings of IEEE International Conference on Computer Vision, pages 19924- 19935, 2023.
[48] J. Shi, C. Shang, Z. Sun, L. Yu, X. Yang, and Z. Yan. Passion: Towards effective incomplete multi-modal medical image segmentation with imbalanced missing rates. In Proceedings of ACM International Conference on Multimedia, pages 456 2024.
[49] C. Shu, Y. Liu, J. Gao, Z. Yan, and C. Shen. Channel-wise knowledge distillation for dense prediction, In Proceedings of IEFE International Conference on Computer Vision, pages 5311-5320, 2.2..
13

---

[50] S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 567-576, 2015.
[51] H. Tang, Z. Li, D. Zhang, S. He, and J. Tang. Divide-and-conquer: Confluent triple-flow network for rgb-t salient object detection. IEEE Transactions on Pultern Analysis and Machine Intelligence, 2024.
[52] Y. Tang, D. Yang, W. Li, H. R. Roth, B. Landman, D. Xu, V. Nath, and A. Hatamizadeh. Self Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 20730-20740, 2022.
[53] Z. Tu, Y. Ma. Z. Li, C. Li, Xu, and Y. Liu. Rgbt salient object detection: A large-scale dataset and benchmark. IEEE Transactions on Multimedia, 25:4163 4176, 2022.
[54] Z. Tu, T. Xia, C. Li, X. Wang. Y. Ma, and J. Tang. Rgb-t image saliency detection via collaborative graph learning. IEEE Transactions Multimedia, 22(1):160-11) 173, 2019.
[55] H. Wang, Y. Chen, C. Ma, J. Avery, L. Hull, and G. Carneiro. Multi-modal learning with missing modality via shared-specific feature modelling. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pages 15878-15887, 2023.
[56] X. Wang, X. Zhang, Y. Cao, W. Wang, C. Shen, and T. Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.0m 032844,4 2023.
[57] Y. Wang. P. Cao, Q. Hou, L. Lan, J. Yang. X. Liu, and O. R. Zaiane. Progressively correcting soft labels via teacher team for knowledge distillation in medical image segmentation. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 521-530, 2024.
[58] Y. Wang, X. Chen, L. Cao, W. Huang, Sun, and Y. Wang. Multimodal token fusion for vision transformers. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 12186-12195, 2022.
[59] Z. Wang. E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In Asilomar Conference on Signals, Systems & Computers, volume 2, pages 1 1398-1402, 2003.
[60] C. Wei, H. Fan, Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 16668-14778, 2022.
[61] Wu, Ji, Y. Liu, Fu, M. Xu, Y. Xu, and Y. Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304. /2620, 2023.
[62] 7. Wu, D. P. Paudel, D.-P. Fan, J. Wang, S. Wang, C. Demonceaux, R. Timofte, and L. Van Gool. Source-free depth for object pop-out. In Proceedings of IEEE International Conference on Computer Vision, pages 1032-1042, 2023.
[63] Z. Xing, L. Yu, L. Wan, T. Han, and L. Zhu. Nestedformer: Nested modality-aware transformer for brain tumor segmentation. In Proceedings of International Conference Medical Image Computing and Cosputer-Assisted Intervention, pages 140-150, 2022.
[64] Y. Xu, Y. Bin, J. Wei, Y. Yang, G. Wang, and H. T. Shen. Multi-modal transformer with global-local alignment for composed query image retrieval. 25:8346-83577 2023.
[65] Y. Yang, P.-T. Jiang, Q. Hou, H. Zhang, J. Chen, and B. Li. Multi-task dense prediction via mixture of low-rank experts. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pages 27927-27937, 2024.
[66] B. Yin, X. Zhang, Z. Li, L. Liu, M.-M. Cheng, and Q. Hou. Dformer: Rethinking rgbd representation learning for semantic segmentation. In Proceedings of International Conference on Learning Representations, 2024.
14

---

[67] C. Yin and Q. Zhang, A multi modal framework for robots to learn manipulation tasks from human demonstrations. Journal of Intelligent & Robotic Systems, 107(4):56, 2023.
[68] Q. Yu, X. Zhao, Y. Pang, L. Zhang, and H. Lu. Multi-view aggregation network for dichotomous image segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3921-3930, 2024.
[69] Z. Zeng, Z. Peng. Yang, and W. Shen. Missing as masking: Arbitrary cross-modal feature reconstruction for incomplete multimodal brain tumor segmentation. In Proceedings of Interna- tional Conference on Medical Image Computing and Computer-Assisted Intervention, pages 424 433, 2024.
[70] X. Zhan, X. Pan, B. Dai, Z. Liu, D. Lin, and C. C. Loy. Self-supervised scene de-occlusion. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3784- 3792, 2020.
[71] J. Zhang H. Liu, K. Yang. X. Hu, R. Liu, and R. Stiefelhagen. Cmx: Cross-modal fusion for rgb- x semantic segmentation with transformers. IEEE Transactions on Intelligent Transportation Systems, 24(12): 14679-1469999 2023.
[72] J. Zhang, R. Liu, H. Shi, K. Yang. S. ReiB, K. Peng, H. Fu, K. Wang, and R. Stiefelhagen. Delivering arbitrary- modal semantic segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 1136-1144 136-1147,, 2023.
[73] J. Zhang, R. Liu, H. Shi, K. Yang, S. ReiB, K. Peng, H. Fu, K. Wang, and R. Stiefelhagen. Delivering cesitrary-modar semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1136-12 147, 2023.
[74] Y. Zhang, N. He, J. Yang, Y. Li, D. Wei, Y. Huang, Y. Zhang, Z. He, and Y. Zheng. mm- former: Multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation. In Proceedings of International Conference Medical Image Computing and Compuler-Assitted Intervention, pages 107-117, 2022.
[75] X. Zhao, S. Chang, Y. Pang, J. Yang. L. Zhang, and H. Lu. Adaptive multi-source predictor for zero-shot video object segmentation. International Journal of Computer Vision, pages 1-19, 2024.
[76] X. Zhao, Y. Pang, 7. Chen, Q- Yu, I.. Zhang, H. Liu, J. Zuo, and H. Lu. Towards automatic power battery detection: New challenge benchmark dataset and baseline. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 22020-22029, 2024.
[77] X. Zhao, Y. Pang, W. Ji, B. Sheng. J. Zuo, L. Zhang, and H. Lu. Spider: A unified framework for context-dependent concept understanding. In Proceedings of International Conference on Machine Learning, pages 60906-660 60926, 2024.
[78] X. Zhao, Y. Pang, J. Yang, L. Zhang, and H. Lu. Multi-source fusion and automatic predictor selection for zero-shot video object segmentation. In Proceedings of ACM International Conference on Multimedia, pages
2021.
[79] X. Zhao, Y. Pang, L. Zhang, and H. Lu. Joint learning of salient object detection, depth estimation and contour extraction. IEEE Transactions on Image Processing. 31:7350-7362. 2022.
[80] X. Zhao, Y. Pang, L. Zhang, H. Lu, and X. Ruan. Self-supervised pretraining for rgb-d salient object detection. In Proceedings of AAAI Conference on Artificial Intelligence, pages 3463-3477 3471, 2022.
[81] X. Zhao, Y. Pang, I. Zhang, H. Lu, and I.. Zhang. Suppress and balance: A simple gated network for salient object detection. In Proceedings of European Conference on Computer Vision, pages 31-51, 2020.
[82] X. Zhao, Y. Pang, L. Zhang, H. Lu, and I.. Zhang. Towards diverse binary segmentation via simple yet general gated network. International Journal of Computer Vision, 13)(10))4157- 4234, 2024.
15

---

[83] X. Zhao, L. Zhang, and H. Lu. Automatic polyp segmentation via multi-scale subtraction network. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 120-130, 2021.
[84] X. Zhao, L. Zhang, Y. Pang, H. Lu, and L. Zhang. A single stream network for robust and real-time rgb-d salient object detection. Proceedings of European Conference on Computer Vision, pages 646 646-662, 2020.
[85] Z. Zhao, J. Li, L. Wang, Y. Wang, and H. Lu. Maskmentor: Unlocking the potential of masked self-teaching for missing modality rgb-d semantic segmentation. In Proceedings of ACM International Conference on Mullimedia, pages 1915-1923, 2024.
[86] W. Zhou, Y. Zhu, Lei, Yang, and L. Yu. Lsnet: Lightweight spatial boosting network for detecting salient objects in rgb-thermal images. IEEE Transactions on Image Processing, 32:1329 1340, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions and scope detailed in the abstract and introduction. Section I.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations of our work in Appendix E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This is not theoretical paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in Appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included the implementation details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [No]
Justification:
The code will be released to facilitate further research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com / ssiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have included the implementation details and the hyperparameters in
Sec. 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report average and standard deviation values our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computing cost Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/pw..eerr....co//////////a
Answer: [Yes]
Justification:
We have read and understood the code of ethics and have done our best to
conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work does not present any potential negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: This work does not incorporate any large language models part of the core methods, any LLMs used were only for manuscript editing and formatting.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:///eeerrppppcc//ooeeeeeeeee for what should or should be described.
23

---

Table 6: Quantitative comparison of brain tumor segmentation BraTS2020. and 4 indicate that higher and lower scores are better, respectively. Note: The reported results of RFNet [T0] and mmFormer [74] are taken from the PASSION paper 48 where they are evaluated using the same training and test splits of BraTS2020.
Modality
Lioe soore(
WWile))
(Fahancing)
Flair TI
Tlee
Net
10
mmm Semmer
744
PASSION
18
RTNet UmMiR RiNec
10
mmi Vemer
[14
PASSION REN UmMMRSeg RiNNe
[15
10
mmm Cerner
724
PASSION RENeC UmMMRSeg
15
84.54
$3.60
40108
18.30
10-27
69.57
75.02
77.24
76.40
77:25
78.33
65.81
67.25
69.15
68.79
68.5 58.53
60:60
72.30
79.72
59.53
$6.00
57
63.32
11.09
78.66
30.42
40.12
62.28
83.24
83.95
84.45
64.20
61.74
58.
66.69
36.67
33.76
26.40
78.
74.07
76.74
76.45
81.46
81.45
80.50
79.83
82.06
68.56
70.70
70.34
71.38
85.51
$5.50
M6.78
$6.27
70.28
67.07
65.57
72,00
41.19
33.51
36,02
48.45
43.61
37.26
N7.73
70.70
42.17
31.47
18
$7.39
85.
$6.54
80.16
79.67
76.38
81,91
68:38
68.11
68.60
71.83
86.61
$7.99
87.46
88.81
81.67
80.71
79.54
82.41
69.47
70.85
72:24
74.93
67.36
38.54
87.45
89.46
81.97
81.25
78.58
83.68
68.76
69.91
70.18
74.37
K5.410
M5.40
83.04
89.04
K3.28
82.23
80.00
84.64
71.04
70.82
69.90
74.56
88.20
Average
67.51
83.67
83.05
85.97
75.02
73.08
70.51
77.12
55.35
54.91
$2.87
61.65
Si3 Dev
0.27
0.45
4.36
3.36
7.45
8.45
9.00
0.95
14.36
16.24
18.09
12.81
Table 7: Quantitative comparison of brain tumor segmentation BraTS2018. t and 4 indicate that higher and lower scores are better, respectively. Note: The reported results of mmFormer (74), M3AE (25) and M3FeCon [69] are taken from the M3FeCon [69]. where they evaluated using the same training and test splits of BraTS2018.
Modality
Dice score(Se)
Whole Tumor (Whole)
Tumor Core (Core)
Enhancing Tumor (Enhanning)
mmFormer
M3FeCon UniMRSeg mmFFrmmer
UniMRSeg mmnFormer
UniMRSeg
Flair TI Tice T2
74
25
69
74
25
69]
74
25
69
G
81.43
84.22
85.13
87.10
64.61
69.14
72.48
76.45
41.92
46.93
49.32
54.20
75.93
71.37
67.92
73.83
75.24
82.29
36.96
65.77
65.72
69.70
31.38
36.54
44.35
49,20
0
0
86.37
88.04
89.05
89.85
61.61
66.02
69.42
73.52
37.98
34.96
46.59
$1.35
83.22
85.58
86.61
88.57
79.07
84,75
84.85
73.13
74.39
76.55
76.75
74.75
76.50
79.16
83.25
79.01
83.11
82.88
83.97
72.77
74.58
76.78
78.41
87.46
88. 88.49
90.17
91.45
66.47
70.53
72.82
76.72
41.75
48.16
48.47
53,20
82.46
86.34
86.03
$8.15
69.89
71,46
72.01
77.43
43.85
44.73
50.17
55.12
87.99
89.31
90.46
91.78
70.18
70.59
72.81
76.95
46.47
40.57
$1.12
56.09
87.61
78.34
73.93
87.77
88.07
89.62
89,99
80.22
83.72
84.81
85.41
74.31
73:42
76.74
78.72
88.00
89.24
90.24
91.89
71.97
72.41
75.98
80.26
46.5
44.15
$2.63
60.12
79.94
83.08
85.06
86.79
90.48
80.89
84.13
84.39
85.49
73.49
73.37
77.33
79.36
89.93
89.56
90.69
92.45
86.23
84.24
85.03
86.44
76.36
74.85
77.81
79.87
83.28
85.18
86.32
88.90
73.42
77.05
78.28
80.67
58.65
59.23
63.84
67.02
Std Dev
6.37
5.29
5.25
3,50
8.02
7.34
6.00
5.05
16.31
16.17
14,05
12.25
Appendix
A Performance Comparison on Bra TS2020 and Bra TS2018
To enable fair comparisons with a broader range of methods, adopt the same data split settings as used in the PASSION paper (48) for BraTS2020, where the dataset is divided into 219 cases for training, 50 for validation, and 100 for testing. For BraTS2018,
follow the data split protocol from M3FeCon [69]. using 200 cases for training and 85 for testing. As shown in Tab. and Tab. method consistently achieves the best average performance and lowest standard deviation.
B Necessity of the Three-Stage Training Strategy
The proposed three-stage training pipeline, comprising Multi-granular Modality Reconstruction, Modality-Invariint Contrastive Learning, and Incomplete Modality Adaptive Fine-tuning, is designed to address the unique challenges of missing modality segmentation, which cannot be effectively tackled using standard end-to-end training.
24

---

Unlike single-modalin RGB image tasks that benefit from large-scale pre- pre-trained models, many other modalities such as depth, infrared, and medical imaging still lack general- purposs pre-trained encoders. As a result, most models initialized randomly suffer from suboptimal feature representa- tions and unstable performance. Moreover, simple end-to-end optimization struggles to bridge the representation gap between complete and incomplete modality inputs, leading to poor generalization to unseen modality combinations.
To this end, each stage in our training strategy plays complementary role in strengthening the model's robustness and adaptability:
Stage 1 (Modality Reconstruction): Enables the model to learn cross- modality structural priors and recover semantically meaningful representations from incomplete inputs.
Stage 2 (Contrastive Learning): Encourages the network to align features between complete and missing modality inputs, enhancing modality-invariat representations.
Stage 3 (Unified Fine-tuning): Refines prediction consistency across different modality combina-
tions and ensures deployment under a unitied model.
Each stage serves distinct and complementary role (input-level structure recovery, feature-level modality alignment, and output-level consistency). Extensive ablation studies (Sec. 4.4) validate
that each stage contributes distinct improvements in average accuracy. Without any of the stages, performance consistently degrade.
C Discussion Training Complexity and Efficiency
Although UniMRSeg adopts a three-stage training strategy, is deliberately designed for efficiency and simplicity. We analyze the training complexity from two perspectives: concise design and inference simplicity.
All three stages in UniMRSeg share unified 3D U-Net-style encoder-decoder backbone, integrated with lightweight 3D ASPP module. No additional complex or stacked modules introduced. This ensures both architectural clarity and training efficiency. Stage utilizes masked autoencoding-bue bbased reconstruction head. This head operates on top of the shared encoder and does not require extra mecoder-decodes branches, keeping the design compact. Stage applies contrastive loss to dtready-compute latent features, without introducing new network components. Stage 3 focuses refining segmentation via lightweight reverse attention adapter and decoder. The encoder is frozen, making this phase computationally efficient and requiring minimal fine-tuning.
Concise Design:
Single-Stage Inference with Unified Weights:
Despite the multi-stage training, inference remains a single-stage process. The final model trained at stage merges all learned representations into a unified network with shared parameters across all modality combinations. Unlike other methods that re- quire ensemble inference [25] 8 or modality-specinf branches [69 (74, UniMRSeg performs fast, unified inference without requiring modality recognition dynamic model selection. As shown in Tab. g UniMRSeg achieves comparable computational efficiency to state-of-the-art methods.
Table 8: Efficiency comparison.
Monnes
RENer LD]
mmFormer
[4] MEAE [2]
UniMIXsog
(MB))
87
FLOPs (G))
148
748
248
202
D Modality-Agnostic Robustness via Random Modality Shuffle
Previous multi-modal segmentation methods 69 74 10 25 often rely on fixed modality-to to-cnannel correspondence, where each input modality (e.g., T1, T2, Tlce, Flari) assigned to a specific encoder or channel during both training and inference. This design imposes a strong prior assumption: the modality type of each input channel must be known in advance and must strictly align with the model's is expected input structure during inference. However, such assumptions significantly limit the scalability and automation of AI-driven medical image analysis pipelines, especially in real-world applications where modality labels may be missing, inconsistent, or ambiguous.
To alleviate this constraint, we introduce random modality shuffle strategy in stage where the input modalities are randomly permuted at each training iteration. This forces the model to learn modality invariann representations and reduces its reliance on fixed input orders.
25

---

As shown in Tab. 9 even after ran- domly shuffling the modality-channe order five times during inference, the segmentation performance remains stable with minimal variation. These results demonstrate that the random shufile not only improves perfor- mance (see Tab. 3) but also enhances practical robustness and scalability. This aligns with the broader goal of building fully automated and general- izable medical AI systems.
Table Ablation study on the impact of different modality- channel orders during inference. Flair-T1-TIce-12 de- notes the best-performing fixed input order, while Aveve represents the average result of five inference runs using ran- dom modality shuffle.
Average Dice score(%)
Models
Whole Core Enhancing
Flair-Ti-THa
80.64 73.33
63.10
5-Ave
80.54 73.27
63.05
E
Limitations and Future Work
While the three-stage training strategy significantly improves model generalizability and segmentation accuracy under missing modalities, it inevitably increases the training pipeline complexity. This additional training overhead may pose challenges for practitioners time-constrained resource- limited environments. We acknowledge this trade-off and consider streamlining or accelerating the training process as a critical future direction. Several promising avenues achieve this include: I) Curriculum-basee Optimization. Introducing a curriculum learning schedule that gradually transitions from reconstruction to segmentation may allow for progressive learning in single stage. II) Modality-aware Parameter Sharing. Parameter-efficient fine-tuning (e.g., adapters or LoRA) across stages may allow most model weights be reused, reducing memory and time overhead. IIII Cross-task Shared Training Objectives. Reformulating the three tasks under shared objective (e.g.. information bottleneck or consistency maximization) may unify their gradients and reduce stage- specific training routines. These directions hold the potential to preserve the benefits of multi-stage training while simplifying the optimization process, making the proposed method more accessible to broader applications in multi-ma mudti-modal image analysis.
26


Reviews:
[{"review_id": "IAAieTRWKK", "reviewer": "Reviewer_sDPL", "summary": "The paper proposes the UniMRSeg unified modal relaxation segmentation framework, which addresses the issue of incomplete modalities in multi-modal segmentation through a hierarchical self-supervised compensation strategy (HSSC). The framework bridges the gap between complete and incomplete modal representations across input, feature, and output levels. \nSpecifically, it is implemented through: 1) multi-granularity modal reconstruction, 2) modal-invariant contrastive learning, 3) lightweight reverse attention adapters, and 4) hybrid consistency constraint fine-tuning. This approach eliminates the need for training dedicated models for different modal combinations, reducing deployment costs.", "questions": "1. The article may lack certain novelty, such as in the design of the Perturbation and Mask components, as these techniques have likely been extensively explored in domains like computer vision, natural language processing, or time series analysis. The article does not appear to clearly highlight a more distinctive motivation or design, mainly extending existing technologies. Additionally, the incorporation of lightweight reverse attention adapters in the model may also lack sufficient innovation, so the core innovative aspects of the paper need to be better emphasized.\n2. The article requires improvement in the visual presentation aspects, such as in the rendering of Figure 1 and Figure 2. The figures currently contain limited effective information or may not fully reflect the model's own structure or characteristics, necessitating further refinement.\n3. The article should compare its proposed model against a more competitive set of baseline models, and expand the experimental scale to further elucidate the performance advantages of the model presented in the paper.", "limitations": "The author mentions the limitations of the study in the article, mainly that the three-stage training strategy increases the complexity of the training process and introduces additional training overhead, which may pose challenges for practitioners with limited time or resources. I consider this explanation quite adequate, and such trade-offs are within an acceptable range for the sake of better performance. For other limitations, please refer to the Questions section.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The proposed model demonstrates superior performance.\nThe article is composed in a lucid and meticulous manner.\nThe insights provided for augmenting segmentation task performance in the multi-modal context are distinctive, and the paper's foundational premise is quite novel. The article presents a clear and rigorous approach to addressing the problem, which is highly commendable.\nThough the article possesses certain merits, there remain some shortcomings:\nThe motivation of the article is clear, but the novelty of the proposed method may be lacking.\nThe article is still deficient in experimentation, with an insufficient number of comparative methods, which may not fully showcase the superior performance of this model.\nThe article may have issues with unclear representations in some of the figures.", "was_revised": true, "final_justification": "The author further elaborated on the innovation of the proposed method in a specific context, enabling me to gain a more thorough understanding of the article's efforts and applications in promoting community development. In addition, the author provided a detailed explanation of why the baseline of the comparison method is limited and actively offered improvement plans, based on which I will reasonably raise my rating.", "rebuttal": "Dear Reviewer sDPL,   \n\nThank you for your time and careful review of our work. Below we address your questions and weaknesses mentioned:\n\n---\n>  [Q1 W1,  the article may lack certain novelty, such as in the design of the Perturbation and Mask components, as these techniques have likely been extensively explored in domains like computer vision, natural language processing, or time series analysis. The article does not appear to clearly highlight a more distinctive motivation or design, mainly extending existing technologies. Additionally, the incorporation of lightweight reverse attention adapters in the model may also lack sufficient innovation, so the core innovative aspects of the paper need to be better emphasized.]\n\nWe understand the concern regarding the novelty of individual components. We would like to emphasize that the novelty of UniMRSeg does not lie in standalone algorithmic innovations, but rather in the hierarchical, task-driven integration of multiple self-supervised mechanisms, jointly optimized to address the unique challenges of modality-relax segmentation. Specifically:\n\n**(1) Hierarchical Self-Supervised Compensation (HSSC) Framework**\n\n*(as already acknowledged and appreciated in your previous review)*\n\nUniMRSeg unifies pixel-level modality reconstruction, feature-level contrastive learning, and prediction-level label distillation into a single annotation-free compensation framework. Unlike previous works that treat self-supervised tasks in isolation, HSSC employs a three-stage curriculum, jointly optimized with a single shared model per task, enabling robust generalization across arbitrary missing-modality combinations.\n\n**(2) Hybrid Data Perturbation**\n\nWhile data perturbation techniques have been explored in various tasks, we design a hybrid perturbation mechanism specifically tailored to the modality-relax setting:\n\n* Random modality dropout simulates missing inputs;\n* Modality shuffle removes the reliance on fixed modality-channel mapping, encouraging modality-agnostic encoding;\n* Spatial masking mimics local degradation.\n\nAs discussed in Lines 585–606 and validated in Table 9, conventional models depend on strict input order and modality metadata, which limits their robustness in real-world scenarios. Our strategy removes these assumptions, improving deployment flexibility and generality.\n\n**(3) Lightweight Reverse Attention Adapter (RAA)**\n\nThe RAA  is designed as a residual correction to bridge the encoder representation gap between incomplete and complete modality inputs during Stage 3. Formally:\n\n\n$$\nf_{\\text{inc}} + \\mathcal{A}(f_{\\text{inc}}) \\approx f_{\\text{com}},\n$$\n\nwhere:\n\n* $f_{\\text{inc}}$: encoder features from the incomplete modality input,\n* $f_{\\text{com}}$: encoder features from the complete modality input,  \n* $\\mathcal{A}(\\cdot)$: the learnable adapter.\n\nDuring Stage 3, both $f_{\\text{inc}}$ and $f_{\\text{com}}$ are frozen, and only $\\mathcal{A}$ is trained. This well-constrained setup ensures that the adapter focuses purely on compensating the missing information, leading to stable and efficient optimization.\n\nTo further enhance the adapter’s ability to capture under-represented semantic cues, we apply reverse attention to amplify the residual signal. In this way, RRA can focus on the non-salient or poorly attended regions that conventional self-attention may overlook.\n\n\nOverall, our original intention was not to pursue isolated innovations on specific components or modules by introducing increasingly \"fancy\" designs in the hope of achieving strong results for unified modality-relax segmentation.  While such improvements may be valuable directions for future work, they are not the focus of this paper. Instead, this work aims to demonstrate that by strictly adhering to the motivation of Hierarchical Self-Supervised Compensation (HSSC) and designing around it in a principled manner, UniMRSeg is able to achieve strong and consistent performance across tasks. We hope that the motivation and explanation above can be well understood and appreciated. \n\n---\n\n>  [Q2 W3,  the article requires improvement in the visual presentation aspects, such as in the rendering of Figure 1 and Figure 2. The figures currently contain limited effective information or may not fully reflect the model's own structure or characteristics, necessitating further refinement. ]\n\nThank you for your suggestion. In the revised version, we will improve Figure 1 and Figure 2 as follows:\n\n**(1) Figure 1**\n\n   We will split the current figure into two subfigures for clarity:\n\n   * Figure 1a: A clear, labeled block diagram of the full architecture, including the encoder, decoder, perturbation module, contrastive learning module, and reverse attention adapter.\n   * Figure 1b: The benchmark box plot will be retained, with expanded labels and annotations to improve readability.\n\n**(2) Figure 2**\n\n   We will restructure Figure 2 to better highlight the three-stage training pipeline and the interconnections between stages, including:\n\n   * Inputs and outputs at each stage;\n   * The flow of gradients and supervision signals (e.g., reconstruction, contrastive, and segmentation losses);\n   * The dependency and progression between stages, as discussed in Response to [Q1 W1], emphasizing how Stage 1 prepares robust features, Stage2 aligns them with segmentation objectives, and Stage 3 adapts to incomplete modalities using frozen representations.\n\n---\n>  [Q3 W2,  the article should compare its proposed model against a more competitive set of baseline models, and expand the experimental scale to further elucidate the performance advantages of the model presented in the paper.]\n\nThank you for your suggestion. We fully agree that strong baselines and broad evaluations are essential to demonstrate the effectiveness and generality of our proposed method.\n\n**(1) On Baseline Competitiveness**\n\nWe would like to emphasize that Tables 1, 2, 6, and 7 already include state-of-the-art and highly competitive methods published in top-tier conferences/journals, such as:\n\n* PASSION (MM 2024)\n* GateNet (IJCV 2024)\n* EonTRINet (PAMI 2024)\n* MaskMentor (MM 2024)\n* M3FeCon (MICCAI 2024)\n\nThese methods represent the most advanced techniques in multi-modal/ missing modality  segmentation. \nMost importantly, they are among the few methods with publicly released code and pre-trained models, enabling fair and reproducible comparisons under consistent settings.\n\nWhile other recent approaches do exist, many do not provide released implementations, making standardized evaluation infeasible. We are cautious about comparing only reported numbers without aligned settings.\n\nIn order to fully address your suggestion and further strengthen the persuasiveness of our results, we report a performance comparison with the recently published model LS3M [1] (CVPR 2025, code not yet released). Although its implementation is unavailable, the reported results on BraTS 2018 are relatively aligned with our evaluation setup, allowing for an approximate comparison.\n\n| Method   |  Whole  Dice (%) ↑ | Core Dice (%) ↑ | Enhancing Dice (%) ↑  |\n| -------- | --------- | --------- | --------- |\n| LS3M     | 88.01     | 79.28     | 63.31     |\n| UniMRSeg | 88.90     | 80.67     | 67.02     |\n\n[1] Incomplete Multi-modal Brain Tumor Segmentation via Learnable Sorting State Space Model, CVPR 2025.\n\n\n**(2) On Evaluation Scope**\n\nUniMRSeg has been evaluated across diverse domains, including:\n\n* Medical segmentation: BraTS 2018, BraTS 2020\n* RGB-D semantic segmentation: SUN-RGBD\n* RGB-D salient object detection: STERE\n* RGB-T salient object detection: VT1000\n\nEach dataset is tested under all possible missing modality combinations, with a single unified model per task, to assess both task-level and modality-level generalization.  The scope of evalution is broader than most previous works.\n\n**(3) Planned Expansions in Final Version**\n\nTo further strengthen the evaluation, we will:\n\n* Include comparisons with generalist segmentation models (e.g., SAM2, SegGPT) under in-context setups using prompt masks.\n* Extend the evaluation to additional RGB-D SOD and SS datasets, such as SIP, NJUD, NLPR, and NYUD.\n\n\n\nIn summary, our current evaluation already reflects a wide and competitive scope. We are committed to further expanding the benchmarks and baselines to reinforce the value of our work. Thank you again for the constructive suggestion.\n\n---\n>  [Limitations: The author mentions the limitations of the study in the article, mainly that the three-stage training strategy increases the complexity of the training process and introduces additional training overhead, which may pose challenges for practitioners with limited time or resources. I consider this explanation quite adequate, and such trade-offs are within an acceptable range for the sake of better performance.]\n\nThank you very much for acknowledging our explanation regarding the three-stage training strategy. We also kindly refer you to our response to Reviewer PWbw’s [W1], where we further elaborate on the necessity of the three-stage pipeline and demonstrate its advantages over a unified single-stage alternative. We appreciate your understanding of the trade-off between performance and complexity."}, {"review_id": "hsfQG6MLap", "reviewer": "Reviewer_PWbw", "summary": "This paper introduces UniMRSeg, a unified framework for multi-modal image segmentation that handles missing modalities through a three-stage hierarchical self-supervised compensation strategy. The approach combines multi-granular modality reconstruction, modality-invariant contrastive learning, and incomplete modality adaptive fine-tuning with a reverse attention adapter. The authors evaluate their method on brain tumor segmentation, RGB-D object segmentation, and claim improved performance across all (missing) modality combinations.", "questions": "See weakness", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n  1. The problem of missing modality segmentation and the limitations of current complex inference setups. The unified framework leverages a single model sharing across modality combinations, which is conceptually appealing and addresses a practical deployment concern.\n  2. The reverse attention adapter design provides an interesting mechanism for identifying and compensating weak perceptual regions. Ablation also validates this design, removing the reverse attention mechanism causes a measurable drop (Table 4).\n- Weakness\n  1. The training pipeline's complexity raises questions about its necessity. While the ablation studies (Tables 3 and 5) demonstrate that each stage contributes to overall performance, it remains unclear why these components cannot be integrated into a single, unified loss function for joint optimization.\n  2. It is still not clear why fine-tuning of the encoder degrades performance (line 310-311). Could you please elaborate on that?\n  3. NestedFormer does not handle missing modality; it is unclear how the benchmark is done (Table 1).", "was_revised": true, "final_justification": "- The rebuttal clarified key concerns, particularly around the necessity of multi-staged training and benchmark settings\n- I agree with other reviewers that the performance improvement outweighs the complexity of the method", "rebuttal": "Dear Reviewer PWbw,\n\nMany thanks to your professional, detailed, and valuable reviews. We're going to response to your concerns one by one.\n\n---\n\n>  [W1,  the training pipeline's complexity raises questions about its necessity. While the ablation studies (Tables 3 and 5) demonstrate that each stage contributes to overall performance, it remains unclear why these components cannot be integrated into a single, unified loss function for joint optimization.]\n\n\nThank you for your suggestion. Prior to submission, we actually conducted the single-model unified training experiment. However, we chose not to include it in the paper, as our intention was to help readers focus more clearly on the self-supervised benefits brought by each stage individually. This is because the distinction between multi-stage and single-stage training essentially reflects a difference between self-supervised representation learning and multi-task learning.\n\n**(1) Self-supervised Representation Learning (Three Stages)**\n\nOur work is built upon a self-supervised pretraining framework, with training initialized from scratch starting at Stage 1. For each task (e.g., BraTS 2020), self-supervised training is conducted solely on the task’s own training set, rather than relying on externally labeled datasets such as ImageNet-pretrained weights for initialization. \nOur three-stage design includes: \n* Stage 1: Pretrain both the encoder and decoder through arbitrary modality reconstruction.\n* Stage 2: Pretrain the encoder with contrastive learning. The segmentation task here is only used to guide the contrastive objective, not as a final goal.\n* Stage 3: Perform the downstream segmentation task.\n\nThis design explicitly aims to reduce the representation gap between complete and incomplete modalities in the encoder-decoder space during the final segmentation stage. If all these tasks are trained jointly in a single-stage model, it would fall into the scope of multi-task learning, rather than self-supervised pretraining. \n\n**(2)  Multi-task Learning (Single Stage)**\n\nIn multi-task learning, the goal is to leverage multiple complementary clues for collaborative learning. These clues can come from the data level [1,2,3], or from structural supervision types [4,5,6]. Most approaches often incorporate deliberate designs for task-sharing and task-specific components to enable effective joint prediction across tasks. \nIn this work, if the three stages are forcibly merged into a single training stage, the following issues will arise: \n* The model is trained with inputs that involve Random Modality Dropout, Random Modality Shuffle, and Random Spatial Masking. The encoder is supervised using the NT-Xent contrastive loss, an adapter is incorporated, and the model is simultaneously tasked with both segmentation and reconstruction. Such a fully entangled training setup lacks a clear task hierarchy, and there is no explicit coordination among the input, encoder, decoder, and output.\n* When all tasks are treated equally without ordering, joint optimization becomes highly difficult. In the unified single-stage training attempt, the total loss involves six parts. The results on BraTS 2020 are as follow:\n\n| Setting                       | Whole  Dice (%) ↑ | Core Dice (%) ↑ | Enhancing Dice (%) ↑ | Training Stability | Convergence |\n| - | - | - | -  | -  | - |\n| Three Stages (Ours)            | 80.64          | 73.33       | 63.10           | ✓ Stable           | ✓ Easy      |\n| Single Stage (Unified Loss) | 20.32              | 13.67           | 10.03                | ✗ Unstable         | ✗ Poor      |\n\nWe observed the following phenomena  during the experiments:\n\n* The single stage model failed to converge, with loss plateauing early. \n* The optimization process was highly unstable, with different losses fluctuating in turn and failing to decrease consistently together.\n\nThese phenomena and performace clearly indicate that unified single stage training is unable to achieve effective coordination among the various designs and supervision signals. The lack of a clear training order, combined with competing optimization objectives, leads to mutual interference.\n\nIn summary, we would like to clarify two potential misconceptions:\n\n1. The three-stage training is **not overly complex**. Both theoretically and practically, it is a **simpler and more stable design**, especially when compared to directly mixing all objectives and modules in a single unified training phase, which leads to optimization difficulties.\n2. The three stages are **not repeating the same task**. Instead, they form a self-supervised learning (SSL) pipeline that does not rely on extra data, and progressively reduce the gap between incomplete and complete modalities at the **input, feature, and output** levels to improve segmentation performance.\n\nReferences:  \n[1] Segment Anything, ICCV 2023.  \n[2] SegGPT: Towards Segmenting Everything in Context, ICCV 2023.  \n[3] Spider: A Unified Framework for Context-dependent Concept Segmentation, ICML 2024.  \n[4] Deep Multitask Learning with Progressive Parameter Sharing, ICCV 2023.  \n[5] MTMamba: Enhancing Multi-Task Dense Scene  Understanding by Mamba-Based Decoders, ECCV 2024.  \n[6] Multi-Task Dense Prediction via Mixture of Low-Rank Experts, CVPR 2024.  \n\n---\n\n\n>  [W2, It is still not clear why fine-tuning of the encoder degrades performance (line 310-311). Could you please elaborate on that ?]\n\nWe respond from two perspectives: the inheritance between Stage 2 and Stage 3, and the design rationale of the lightweight reverse attention adapter.\n\n**(1) Inheritance between Stage 2 and Stage 3**\n\nUnlike general SSL methods (e.g.,  SimCLR, MoCo, MAE) that focus on learning generic representations without targeting specific downstream tasks, our contrastive learning in Stage 2 is task-aware. As stated in Lines 197–199, we co-train the segmentation head to guide the encoder toward learning modality-invariant features that are beneficial for segmentation, rather than general-purpose representations. This task-guided design ensures that the learned contrastive space aligns with the downstream segmentation objective. As shown in Table 3, this approach significantly boosts segmentation performance. Therefore, fine-tuning the encoder in Stage 3 would undermine the task-guided contrastive representations we carefully established. \n\n**(2) Lightweight Reverse Attention Adapter (RAA)**\n\nThe RAA  is designed as a residual correction to bridge the encoder representation gap between incomplete and complete modality inputs during Stage 3. Formally:\n\n\n$$\nf_{\\text{inc}} + \\mathcal{A}(f_{\\text{inc}}) \\approx f_{\\text{com}},\n$$\n\nwhere:\n\n* $f_{\\text{inc}}$: encoder features from incomplete modality input,\n* $f_{\\text{com}}$: encoder features from the complete modality input,  \n* $\\mathcal{A}(\\cdot)$: the learnable adapter.\n\nDuring Stage 3, both $f_{\\text{inc}}$ and $f_{\\text{com}}$ are frozen, and only $\\mathcal{A}$ is trained. This well-constrained setup ensures that the adapter focuses purely on compensating the missing information, enabling stable and efficient optimization. \nOnce the encoder is unfrozen, all three components become variables, making their roles unclear and the optimization unstable. This is why the \"finetune\" setting in Table 4 leads to significant performance degradation.\n\n\nIn conclusion, freezing the encoder in Stage 3 is a deliberate and necessary design choice to preserve the task-guided contrastive representations learned earlier. We will make this motivation and its empirical evidence more explicit in the revised manuscript. Thank you again for the insightful question.\n\n---\n>  [W3, NestedFormer does not handle missing modality; it is unclear how the benchmark is done (Table 1).]\n\nThank you for pointing this out. NestedFormer is not designed for missing modality settings. \nWe include it solely as a comparative baseline to highlight that segmentation methods designed for complete modalities perform significantly worse under missing modality conditions, due to the lack of explicit modeling for missing inputs. In contrast, all other methods in Table 1 are specifically designed to handle missing modality. We will clarify this purpose in the revised paper. \n\n---"}, {"review_id": "OJsULpPDB6", "reviewer": "Reviewer_ife8", "summary": "This paper proposes UniMRSeg, a unified segmentation framework designed to handle arbitrary missing modality combinations in multi-modal image segmentation tasks. UniMRSeg introduces a Hierarchical Self-Supervised Compensation (HSSC) strategy at three levels:\n(1) Input-level modality reconstruction using hybrid shuffled-masking;\n(2) Feature-level contrastive learning to obtain modality-invariant representations;\n(3) Output-level consistency enforced by lightweight reverse attention adapters. The model is validated across four benchmark tasks and achieves state-of-the-art performance with minimal variance across 21 missing modality combinations using a single unified model per task.", "questions": "See the Weaknesses part.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths:\n\n1.The motivation is good. The modality-relax setting addresses a highly practical and important real-world challenge, particularly in clinical scenarios with incomplete or corrupted data. UniMRSeg provides an effective and scalable solution to this issue.\n\n2.The HSSC is novelty. The proposed Hierarchical Self-Supervised Compensation strategy is a rare and elegant integration of multiple self-supervised mechanisms into a unified framework. Its design is well-aligned with the goal of modality-relax segmentation and contributes broader insights to the field of representation learning.\n\n3.The performance is good. UniMRSeg achieves state-of-the-art performance across four segmentation tasks. Notably, the authors go beyond the conventional focus on average accuracy across modality combinations by additionally reporting standard deviation, which highlights the model’s robustness and stability.It’s a highly commendable and often overlooked aspect.\n\n4.The ablation study is thorough.  Tables 3–5 convincingly demonstrate the contribution of each component and the necessity of the three-stage training. Figure 5 is particularly effective in intuitively showing how each stage contributes to improved representation and segmentation quality.\n\n## Weaknesses:\n1.Authors mainly focus on the analysis of modality combinations. It might be insightful to show the semantic importance. Whether some modalities (e.g., Flair vs T1ce) are inherently more informative or complementary to others.\n\n2.Although UniMRSeg is evaluated across distinct domains (e.g., medical and natural scenes), the paper does not explore whether a model trained on one task (e.g., BraTS) could transfer to another task (e.g., RGB-D segmentation). A discussion of cross-task generalizability would further strengthen the work.\n\n3.The current model does not explicitly infer which modalities are missing at test time. It would be helpful to understand whether this lack of modality-awareness could affect reliability in certain deployment settings.", "was_revised": true, "final_justification": "Thanks for the authors' positive response. It has addressed all of my concerns.", "rebuttal": "Dear Reviewer ife8,\n\nThank you very much for your valuable comments and suggestions, which are greatly helpful for improving both our current and future work. We address your questions as follows.\n\n---\n\n>  [W1, authors mainly focus on the analysis of modality combinations. It might be insightful to show the semantic importance. Whether some modalities (e.g., Flair vs T1ce) are inherently more informative or complementary to others.]\n\nThank you for this insightful comment. We agree that understanding the semantic importance of each modality is an important aspect of multi-modal learning.\nOur current work primarily focuses on evaluating robustness and generalization across arbitrary missing modality combinations, which explains our emphasis on combination-level performance (mean and standard deviation). However, we acknowledge that some modalities may be inherently more informative or complementary, especially in specific clinical tasks.\nIn fact, as shown in Table 1 of the paper, we observe that combinations including FLAIR generally lead to higher Dice scores, indirectly suggesting its stronger contribution. This observation aligns with clinical knowledge where FLAIR is often more sensitive to edema and tumor boundaries.  We will discuss the semantic importance in the final version.\n\n---\n\n>  [W2,  Although UniMRSeg is evaluated across distinct domains (e.g., medical and natural scenes), the paper does not explore whether a model trained on one task (e.g., BraTS) could transfer to another task (e.g., RGB-D segmentation). A discussion of cross-task generalizability would further strengthen the work.]\n\nWe agree that cross-task generalization is an important topic, especially for unified segmentation models. \nIn our current work, we design UniMRSeg to support task-specific but modality-relax generalization, i.e., within each segmentation domain (e.g., BraTS, SUNRGBD, STERE), the model handles all possible missing modality combinations using a single shared model. This is already a significant step toward practical unification, compared to previous works that require one model per combination. However, transferring a model trained on one task (e.g., brain tumor segmentation) to a fundamentally different task (e.g., RGB-D salient object segmentation) is non-trivial, due to large domain gaps in image appearance, semantics, and supervision forms. Such cross-task generalization is out of the current scope, but we agree it is a promising future direction.\nThat said, our architecture is compatible with such a generalization setting. In future work, we plan to explore whether the shared encoder in UniMRSeg can be reused across domains via task-specific adapters or prompting mechanisms. \n\n---\n\n>  [W3, the current model does not explicitly infer which modalities are missing at test time. It would be helpful to understand whether this lack of modality-awareness could affect reliability in certain deployment settings.]\n\nWe would like to clarify that our UniMRSeg does not rely on explicit modality-type awareness during inference. \nTo this end, we explicitly avoid designing an input pipeline that assumes modality-to-channel correspondence or requires modality labels. Instead, in Stage 1, we introduce a Random Modality Shuffle mechanism, where the order of input modalities is randomly permuted at each training iteration. This forces the model to learn modality-invariant representations rather than relying on fixed or predefined modality identity. \nIn the main paper (Table 9), we also demonstrate that our model maintains stable segmentation performance across five different random shuffle orders during inference without any prior knowledge of modality types. This shows that the learned representation is robust to modality ambiguity, and the model behaves as if it had already inferred or become invariant to missing or shuffled modality types. Hence, while we do not adopt an explicit modality classification or selection module (which may introduce additional complexity or assumptions), our design intentionally achieves the goal of modality-agnostic reasoning in a simpler and more scalable way."}, {"review_id": "0o5PdqUD2c", "reviewer": "Reviewer_mFFe", "summary": "The current paper presents UniMRSeg a unified multi-modal approach to image segmentation that can be trained to be robust in the event of missing modality at inference. The main contribution there is Hierarchical Self-Supervised Compensation (HSSC), a three-step training scheme that aims at making representations generated using incomplete modalities comparable to those generated using complete ones. The process works in the following way:\n\nThe (1) pre-training stage is based on the masked modality reconstruction and vigorous data augmentation (shuffling, masking) in order to develop cross-modal correspondences.\n\nA further step drops modality-specific supervision and contrastively learns the full and incomplete modalities in the feature space at the same time enforcing the downstream supervision at the segmentation task.\n\nThe last stage refines the model, tames the encoder parameters and implements two new mechanisms, a so-called reverse attention adapter that inverts the spatial attention to leverage the existing spatial knowledge, and the consistency constraints at the prediction level to distill the knowledge of the full-modality reference.\n\nThe unique strength of the suggested framework is that only one parameter-shared model is easily capable of processing an arbitrary subset of available modalities, a trait which was supported by empirical findings on brain tumor, salient object, and semantic segmentation benchmarks.", "questions": "The introduced three-stage complexity is the main contribution of the paper but, at the same time, it is a challenging element since it is very complex. There should be a stronger case as to why this sequential arrangement is preferable over a more direct, one-shot, end-to-end training regime that consolidates all the suggested losses, reconstruction, contrastive, and consistency, as multi-tasks. A comparison of the multi-stage procedure with such a single-stage alternative in an ablation would go a long way to support the case of taking up the added complexity.\n\nThere is also a need to have clarity with regard to reverse attention adapter. The authors explain the mechanism as the reverse attention and its mathematical and intuitive definitions are still ambiguous. What exactly is the reversed? What does it mean by explicitly compensate the weak perceptual semantics? Comparing this design with a more standard attention mechanism that learns to emphasize prominent shared feature, and also an ablation analysis would help in understanding the need of making the modification.\n\nIt would be useful to have these ablations in order to isolate contribution of individual stages better. As a case in point, how does the baseline model trained on purely the Stage 3 machinery- adapters + consistency loss, fine-tune without Stage 1 and 2 pre-training? Such comparison would give an insight on the load of the pre-training against the final refinement stage. More so, it would also be informative to benchmark the effectiveness of a baseline model that has been fine-tuned only on Stage 1 reconstruction, and then you would perform the standard fine-tuning on segmentation without Stage 2 or 3.\n\nConcerning the base line comparisons of previous works, as shown in Tables 1 and 2, there is need to give more information. Had the baseline models been trained end-to-end as per their papers or had they been trained under a multi-stage or pre-training regime to be fairly comparable? In case of end-to-end overlap, identified improvements in performance provided by UniMRSeg could, in some part, be attributed to large pre-training and not only to architectural or methodological advances.", "limitations": "The training pipeline, which is quite complex, is mentioned by the authors as one of the acknowledged limitations in the appendix. Although this observation is relevant, the issue should be better highlighted in the body of the paper since it relates to the viability and usability of the given approach. The proposed future task to make the process easier is properly credited, but at the same time it highlights the complexity of the situation at the present.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The current paper provides a brief but serious research of the issue of missing data in multi-modal learning: a clinically and scientifically significant issue. It has made its most significant contribution in developing a single framework which has been claimed to do inference without the full modal information. Trained as a three-stage training pipe-line, UniMRSeg utilises masked reconstruction, contrastive learning, and distillation modules that work across representation layers featuring different SNR regimes. State-of-the-art performance on four representative tasks with low variance, including brain tumour detection, salient object extraction, RGB-D segmentation and RGB-T semantic segmentation, empirically confirms the generalisability of the proposed across cohorts.\n\nRegardless of this strength, a number of caveats should be mentioned. Methodologically, the suggested framework is predominantly based on the existing self-supervised frameworks; the main novelty is in the sequential orchestration of them and not that of the novelties themselves. The training pipeline is also quite complex: three consecutive optimisation processes should be running simultaneously with repetitive freezing and unfreezing of various components of the network. This complexity can be threatening to reproducibility, hyperparameter sensitivity and challenging to deploy in practice. This means that the ablation insights provided by the authors, as interesting as they might be, cannot explain the need of such ordered pipeline; a direct comparison with the end-to-end architecture would have made the marginal gains brought about by the intermediate intermediate contrastive losses clear.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mFFe,\n\nWe sincerely thank you for your valuable comments and questions, which help us a lot to improve our work. We address your questions as follows. \n\n---\n> [W1, the suggested framework is predominantly based on the existing self-supervised frameworks; the main novelty is in the sequential orchestration of them and not that of the novelties themselves.]\n\nDue to the character limit in this rebuttal, we kindly refer you to our response to Reviewer sDPL's [Q1 W1], where we address these concerns in detail.\n\n---\n>   [Q1, the introduced three-stage complexity ...  a more direct, one-shot, end-to-end training regime that consolidates all the suggested losses, reconstruction, contrastive, and consistency, as multi-tasks ...]  \n[W2, the training pipeline is also quite complex ...  ]  \n  [W4, a direct comparison with the end-to-end architecture ...]  \n\nThank you for your suggestion. Prior to submission, we actually conducted the single-model unified training experiment. However, we chose not to include it in the paper, as our intention was to help readers focus more clearly on the self-supervised benefits brought by each stage individually. This is because the distinction between multi-stage and single-stage training essentially reflects a difference between self-supervised representation learning and multi-task learning.\n\n**(1) Self-supervised Representation Learning (Three Stages)**\n\nOur work is built upon a self-supervised pretraining framework, with training initialized from scratch starting at Stage 1. For each task (e.g., BraTS 2020), self-supervised training is conducted solely on the task’s own training set, rather than relying on externally labeled datasets such as ImageNet-pretrained weights for initialization. \nOur three-stage design includes: \n* Stage 1: Pretrain both the encoder and decoder through arbitrary modality reconstruction.\n* Stage 2: Pretrain the encoder with contrastive learning. The segmentation task here is only used to guide the contrastive objective, not as a final goal.\n* Stage 3: Perform the downstream segmentation task.\n\nThis design explicitly aims to reduce the representation gap between complete and incomplete modalities in the encoder-decoder space during the final segmentation stage. If all these tasks are trained jointly in a single-stage model, it would fall into the scope of multi-task learning, rather than self-supervised pretraining. \n\n**(2)  Multi-task Learning (Single Stage)**\n\nIn multi-task learning, the goal is to leverage multiple complementary clues for collaborative learning. These clues can come from the data level [1,2,3], or from structural supervision types [4,5,6]. Most approaches often incorporate deliberate designs for task-sharing and task-specific components to enable effective joint prediction across tasks. \nIn this work, if the three stages are forcibly merged into a single training stage, the following issues will arise: \n* The model is trained with inputs that involve Random Modality Dropout, Random Modality Shuffle, and Random Spatial Masking. The encoder is supervised using the NT-Xent contrastive loss, an adapter is incorporated, and the model is simultaneously tasked with both segmentation and reconstruction. Such a fully entangled training setup lacks a clear task hierarchy, and there is no explicit coordination among the input, encoder, decoder, and output.\n* When all tasks are treated equally without ordering, joint optimization becomes highly difficult. In the unified single-stage training attempt, the total loss involves six parts. The results on BraTS 2020 are as follow:\n\n| Setting                       | Whole  Dice (%) ↑ | Core Dice (%) ↑ | Enhancing Dice (%) ↑ | Training Stability | Convergence |\n| - | - | - | -  | -  | - |\n| Three Stages (Ours)            | 80.64          | 73.33       | 63.10           | ✓ Stable           | ✓ Easy      |\n| Single Stage (Unified Loss) | 20.32              | 13.67           | 10.03                | ✗ Unstable         | ✗ Poor      |\n\nWe observed the following phenomena  during the experiments:\n\n* The single stage model failed to converge, with loss plateauing early. \n* The optimization process was highly unstable, with different losses fluctuating in turn and failing to decrease consistently together.\n\nThese phenomena and performace clearly indicate that unified single stage training is unable to achieve effective coordination among the various designs and supervision signals. The lack of a clear training order, combined with competing optimization objectives, leads to mutual interference.\n\nIn summary, we would like to clarify two potential misconceptions:\n\n1. The three-stage training is **not overly complex**. Both theoretically and practically, it is a **simpler and more stable design**, especially when compared to directly mixing all objectives and modules in a single unified training phase, which leads to optimization difficulties.\n2. The three stages are **not repeating the same task**. Instead, they form a self-supervised learning (SSL) pipeline that does not rely on extra data, and progressively reduce the gap between incomplete and complete modalities at the **input, feature, and output** levels to improve segmentation performance.\n\nReferences:  \n[1] Segment Anything, ICCV 2023.  \n[2] SegGPT: Towards Segmenting Everything in Context, ICCV 2023.  \n[3] Spider: A Unified Framework for Context-dependent Concept Segmentation, ICML 2024.  \n[4] Deep Multitask Learning with Progressive Parameter Sharing, ICCV 2023.  \n[5] MTMamba: Enhancing Multi-Task Dense Scene  Understanding by Mamba-Based Decoders, ECCV 2024.  \n[6] Multi-Task Dense Prediction via Mixture of Low-Rank Experts, CVPR 2024.  \n\n---\n> [Q2, ... What exactly is the reversed? What does it mean by explicitly compensate the weak perceptual semantics? Comparing this design with a more standard attention mechanism ... an ablation analysis would help in understanding the need of making the modification.]\n\n**(1)  Definition of “Reverse” in Reverse Attention**\n\n\"Reverse\" refers to inverting the standard self-attention response. Traditional self-attention emphasizes high-activation (salient) regions. However, under missing modality conditions, some important semantic cues may receive low activation and be underrepresented. We compute reverse attention as `1 - self-attention` to explicitly highlight these under-attended regions, enabling targeted semantic compensation.\n\n**(2)  Purpose of Compensating Weak Perceptual Semantics**\n\nIdeally, if the features from incomplete and complete modalities are well aligned, their self-attention maps should exhibit similar activation distributions. In practice, due to missing inputs, incomplete modality features often lack activation in key semantic regions. These low-response areas typically contain complementary or suppressed information that standard attention overlooks. To address this, we introduce an adapter in Stage 3, trained to recover semantic cues from these weakly activated regions. Guided by reverse attention, the adapter focuses on residual perceptual gaps and compensates them explicitly. This enhances semantic completeness and improves prediction consistency under missing-modality scenarios.\n\n**(3)  Ablation Study** \n\nWe have conducted a detailed ablation analysis of the Reverse Attention Adapter (RAA) as shown in Table 4.  The \"mutual attention\" setting corresponds to a standard attention mechanism implemented with a 3D Swin Transformer. The results show that removing reverse attention (\"w/o Reverse Attention\") leads to a significant performance drop for UniMRSeg, while the performance gap between \"w/o Reverse Attention\" and \"w/o Reverse Attention + Mutual Attention\" is minimal. This strongly supports the effectiveness of the proposed reverse attention design. \n\nIn addition, we refer you to our response to Reviewer sDPL’s [Q1 W1] \"(3) Lightweight Reverse Attention Adapter (RAA)\", which further clarifies the motivation and advantages of the RAA.\n\n\n---\n\n> [Q3, It would be useful to have these ablations in order to isolate contribution of individual stages better ...]  \n[W3, This means that the ablation insights provided by the authors ... cannot explain the need of such ordered pipeline.]  \n\nWe fully agree on the importance of isolating the contribution of individual stages. Accordingly, we have already conducted detailed ablation experiments covering each stage independently, as well as all their combinations, which are presented in Table 5 of the submitted version. We kindly invite you to check it.\n\n---\n\n> [Q4, ... Had the baseline models been trained end-to-end as per their papers or had they been trained under a multi-stage or pre-training regime to be fairly comparable? ... performance provided by UniMRSeg could, in some part, be attributed to large pre-training and not only to architectural or methodological advances.]\n\nFirst, we would like to clarify that **UniMRSeg does not use any external data in any stage** of training. The different UniMRSeg models used for various segmentation tasks (Brain Tumor Segmentation / RGB-D SOD / RGB-T SOD / RGB-D Semantic Segmentation) are trained on the corresponding task’s training set in all three stages.  \nTherefore, the performance gains of UniMRSeg are **not attributable to large-scale pretraining**, but rather to the proposed architectural and learning design.\n\nSecond, to ensure fair comparison across methods in Tables 1, 2, 6, and 7, all baseline models are evaluated under the same training dataset splits. Specifically:\n\n* If a method reports results using the same dataset splits as ours, we directly cite the numbers from the original paper.\n* If the results are missing (e.g., for missing modality settings), or if different splits are used, we retrain the models using their released code, following the original training protocols as closely as possible.\n\nThis ensures that all results are evaluated under consistent and fair condition."}]

Author rebuttal:

Meta-review: The rebuttal addressed most concerns raised by reviewers, e.g., innovation and complexity. Three reviewers are very certain that they land on the positive side. The remaining one reviewer gave negative reviews originally, but did not participate in any discussion or even did not read the rebuttal. Thus, the weight of this reviewer is largely lowered down. The AC recommends acceptance of the paper. The authors are required to include the discussions during the rebuttal into the final version.
Decision: Accept (poster)