PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Rates

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 14:17:35

Abstract: Incomplete multi-modal image segmentation is a fundamental task in medical imaging to refine deployment efficiency when only partial modalities are available. However, the common practice that complete-modality data is visible during model training is far from realistic, as modalities can have imbalanced missing rates in clinical scenarios. In this paper, we, for the first time, formulate such a challenging setting and propose Preference-Aware Self-diStillatION (PASSION) for incomplete multi-modal medical image segmentation under imbalanced missing rates. Specifically, we first construct pixel-wise and semantic-wise self-distillation to balance the optimization objective of each modality. Then, we define relative preference to evaluate the dominance of each modality during training, based on which to design task-wise and gradient-wise regularization to balance the convergence rates of different modalities. Experimental results on two publicly available multi-modal datasets demonstrate the superiority of PASSION against existing approaches for modality balancing. More importantly, PASSION is validated to work as a plug-and-play module for consistent performance improvement across different backbones. Code will be available upon acceptance.

Main paper: 
 introduction: 
 Multiple imaging modalities are widely used in clinical practice like multi-parametric magnetic resonance imaging (MRI) [2] [3] [4] . Such homogeneous multi-modal data representing the same media type provides various tissue contrast views and spatial resolutions, making it possible to quantitatively categorize histological subregions [9, 43] . Nevertheless, due to factors like image degradation, patient motion-related artifacts, incorrect acquisition settings, and cost constraints, MRI sequences may be incomplete [15, 17, 22] . Consequently, despite the great success of multi-modal learning on medical segmentation [7, 16, 20, 41, 43] , it can hardly be directly deployed in practice. To deal with missing modalities, various approaches have been proposed for incomplete multi-modal segmentation. One straightforward strategy is to synthesize missing modalities with available full-modality data through generative adversarial networks [14] , which achieves full-modal segmentation with synthetic images [25, 32, 42] . Another typical solution is to transfer knowledge from full-modality networks to missing-modality ones [1, 39] , which requires superior teacher models trained with full-modality data and dedicates student models for each combination of missing modalities. Alternatively, another efficient approach is shared representation learning, which aims to train a unified segmentation model by sharing a spatial mapping across modalities with multiple modality-specific encoders and a shared decoder [5, 9, 10, 33, 44] . Compared to the former two methodologies, it provides greater flexibility with lower computational complexity and thus becomes the state-of-the-art (SOTA) paradigm.Unfortunately, previous studies tend to focus on deployment efficiency while idealizing the training process. In other words, they all assume perfect data training (𝑃𝐷𝑇 ) where data is available with full-modality. As illustrated in Figure 1 , in 𝑃𝐷𝑇 , incomplete multi-modal data is generated by randomly masking modalities with equal probabilities. One typical setting is to randomly mask modalities during each training round, making invisible modalities in one round visible in another round. The other setting is to randomly mask modalities before training and keep masked modalities invisible. Both settings assume different modalities share the same missing rates. Nevertheless, imperfect data exists not only in deployment but also during data collection and model training [21] , which is under-explored. Furthermore, the missing rates of different modalities may vary in practical scenarios [34] . For instance, in MRI modalities, T1 is the most commonly acquired while T2 usually is unavailable due to its sensitivity to artifacts [15] , resulting in situations where modalities are not equally available for model training and clinical evaluation [6, 19] . Inspired by this, we, for the first time, formulate such a scenario as imperfect data training (𝐼𝐷𝑇 ) where modalities own imbalanced missing rates and missing modalities are kept invisible throughout training, as illustrated in Figure 1 . Such imbalanced missing rates bring modality imbalance, where weak modalities (i.e., with higher missing rates) may be dominated by strong modalities (i.e., with lower missing rates), suffering from being under-trained. Despite great efforts on modality re-balancing [12, 28] , how to balance unequal training in 𝐼𝐷𝑇 for multi-modal segmentation has not been exploited.Under modality absence in model training, existing approaches encounter severe issues. Relying on full-modality training data as ground truth makes it challenging to supervise the reconstruction task in image synthesis [32, 42] and to train teacher models for knowledge transfer [1, 39] . In shared representation learning, the unified multi-modal model typically is trained by masking modalityspecific features in the fusion stage. However, during training, such "masked" modality data still plays a role in regularizing the unimodal performance [9, 33, 44] . Consequently, it would suffer from imbalanced regularization for weak modalities in 𝐼𝐷𝑇 .Based on the observation that, in multi-modal segmentation, multi-modal always performs better than uni-modal, and the knowledge gap between multi-modal and each uni-modal somewhat reflects modality dominance, we propose Preference-Aware Self diS-tillatION (PASSION) transferring multi-modal knowledge to all available uni-modals under the guidance of modality preference. Specifically, pixel-wise and semantic-wise self-distillation is first constructed to regularize the optimization objective of each unimodal in a unified framework. Compared to training via ground truth, it would re-balance each modality according to its distance to multi-modal knowledge. Then, considering the varying learning/optimization paces of different modalities, we define relative preference to estimate how strong each uni-modal is compared to others. Then, task-wise and gradient-wise preference-aware regularization is constructed to ensure a balanced convergence of different modalities. The main contributions are as follows:• We identify and formulate a more realistic and challenging task -incomplete multi-modality medical image segmentation with imbalanced missing rates. To our knowledge, it is the first exploration in multi-modal image segmentation. • We propose a new self-distillation solution PASSION to balance different modalities in both optimization objectives and paces. In addition, PASSION is proven, through extensive evaluation, to work as a plug-and-play module for performance improvement across various backbones. • We prove the superiority of PASSION against SOTA modalitybalancing approaches through comprehensive experiments on BraTS2020 and MyoPS2020 for incomplete multi-modal segmentation with imbalanced missing rates. 
related work 2.1 incomplete multi-modal segmentation: 
 Incomplete data is a common and long-standing issue in practical scenarios, particularly in clinical practice. The most commonly studied task is incomplete multi-modal medical image segmentation usually involving MRI images with various missing components [17] . As standard multi-modal segmentation methods built on full-modality data [20, 43] would encounter severe performance degradation given only incomplete modalities, it is of great value but also challenging.To pursue effective multi-modal medical image segmentation, extensive research has been broadly explored including image synthesis, knowledge transfer, and shared representation learning. Specifically, Sharma and Hamarneh [32] used a U-Net [8, 31] generator to impute missing modalities, with a generative adversarial network [14] learning to discriminate between real and synthesized inputs. Ma et al. [25] proposed a meta-learning algorithm [13] to reconstruct the features of missing modalities. Wang et al. [39] trained a teacher-student framework for each subset of modalities. Azad et al. [1] proposed a style-matching mechanism to reconstruct missing information from a full-modality network. Chen et al. [6] , Wang et al. [36] trained uni-modal models by transferring privileged knowledge from a full-modal teacher to each uni-modal student. Liu et al. [23] focused on self-supervised pre-training. However, in 𝐼𝐷𝑇 , there may not exist sufficient full-modality data as a target for image synthesis and full-modal teacher training, making them hard to deploy in real-world training.To overcome such limitations, Havaei et al. [17] and Dorent et al. [10] computed variational statistics to construct a uniform representation for segmentation. Chen et al. [5] and Ding et al. [9] performed shared representation fusion by modality re-weighting and attention-gating modules. Zhao et al. [45] introduced graph convolutional networks for incomplete brain tumor segmentation. Wang et al. [35] learned shared and specific features by distribution alignment and domain classification. Zhang et al. [44] and Shi et al. [33] introduced transformers to exploit both intra-and inter-modal dependence for feature fusion. Qiu et al. [30] proposed a group self-support algorithm to make use of the sensitive modality's specific features. However, such shared representation learning-based methods were proposed only for incomplete-modality inference while ignoring data missing in training. In other words, the frequency of each modality during training remains the same. Konwer et al. [21] discussed the limited full-modality data scenario and proposed a meta-learning method to enhance modality representations. Unfortunately, how to pursue effective multi-modal medical image segmentation with imbalanced missing rates is under-explored, which is more valuable in clinical practice.Due to modality discrepancy, multi-modal learning naturally encounters the concern of fairness and imbalance. Wang et al. [37] found that different modalities could overfit and generalize at different rates and thus obtain sub-optimal solutions when jointly training them using a unified optimization strategy. Peng et al. [28] proposed that the better-performing modality would dominate gradient updating while suppressing the learning process of other modalities. To address such issues, several approaches have been proposed to deal with modal imbalance. Du et al. [11] proposed to improve uni-modal performance through knowledge distillation from well-trained models. Xiao et al. [40] proposed to randomly drop the audio pathway during training as a regularization technique to adjust the learning paces between visual and audio pathways. Wang and Singh [38] proposed a re-weighting scheme by assigning lower weights to data samples with missing values to promote fairness. Peng et al. [28] chose to slow down the learning rate of the mighty modality by online modulation to lessen the inhibitory effect on other modalities. Fan et al. [12] performed stimulation on the particularly slow-learning modality by enhancing its clustering towards prototypes. Sun et al. [34] proposed the concept of relative advantage for modalities and adaptively regulated the supervision of all modalities during training. Though existing works all focus on classification instead of segmentation, they motivate us to pursue effective incomplete multi-modal medical image segmentation via re-balancing the convergence rates of different modalities in 𝐼 𝐷𝑇 . 
methodology 3.1 problem definition: 
 Given a multi-modal training dataset containing 𝑁 samples, each sample comprises 𝑀 modalities. Let [𝑃] for any positive integer 𝑃 denote the set {1, 2, . . . , 𝑃 }. Each training sample is denoted as({𝑥 𝑚 𝑛 } 𝑚∈ [𝑀 ] , {𝑦 𝑚 𝑛 } 𝑚∈ [𝑀 ]), where 𝑛 ∈ [𝑁 ] is sample index. For a segmentation task, it is to assign each pixel with an individual category label from 𝐾 categories. To formulate this, we use (𝑥 𝑚 𝑛,𝑖 , 𝑦 𝑚 𝑛,𝑖 ) to denote the raw input and label of pixel 𝑖 corresponding to modality 𝑚 and sample 𝑛. For multi-modal segmentation studied in this paper where all modalities share a common label,𝑦 𝑛,𝑖 = 𝑦 1 𝑛,𝑖 = 𝑦 2 𝑛,𝑖 = • • • = 𝑦 𝑀 𝑛,𝑖 holds.Let 𝐶 ∈ R 𝑁 ×𝑀 be the modality presence indication matrix, and let 𝐶 𝑛𝑚 denote the (𝑛, 𝑚)-th entry of 𝐶. Then 𝐶 𝑛𝑚 is 1, if modality 𝑚 of sample 𝑛 is available; otherwise 𝐶 𝑛𝑚 equals to 0. The missing rate 𝑀𝑅 of modality 𝑚, ∀𝑚 ∈ [𝑀], is calculated as 𝑀𝑅 𝑚 = (𝑁 -𝑛∈ [𝑁 ] 𝐶 𝑛𝑚 )/𝑁 . We assume 𝑀𝑅 𝑚 ∈ [0, 1) to ensure that at least one sample of each modality is available during training.Incomplete Multi-modal Segmentation Baseline. In existing research, the SOTA paradigm [9, 33, 44] adopts 𝑚 modality-specific encoders 𝐸 𝑚 and a shared fusion decoder 𝐷 𝑓 . Layer-wise features extracted from available 𝐸 𝑚 are skipped like U-Net to be fused in 𝐷 𝑓 . For the consistency of expression, we use 𝑧 𝑙 𝑛 = 𝐷 𝑙 𝑓 (𝑥 𝑛 ) as the fused features of all available modalities of sample 𝑛 in layer 𝑙, and𝑧 𝑚,𝑙 𝑛 = 𝐷 𝑙 𝑓 (𝑥 𝑚 𝑛 )as the features of only modality 𝑚. Denoting 𝑙 = 0 as the output layer, the overall objective of the paradigm isEQUATIONwhere ℓ 𝑑𝑖𝑐𝑒+𝑐𝑒 denotes the Dice and weighted cross-entropy loss and Up 2 𝑙 represents 2 𝑙 × upsampling. For simplification, we omit the low-dim transform and softmax operation 𝜎. In L 𝑡𝑎𝑠𝑘 , the former item imposes the final prediction objective and deep supervision, and the latter forces each modality to attain its optimal task-relevant representations. Denoting them L 𝑠𝑒𝑔 and L 𝑟𝑒𝑔 respectively, the overall objective in 𝐼𝐷𝑇 is re-formulated as:EQUATIONwhere L 𝑚 𝑟𝑒𝑔 denotes the regularization term of modality 𝑚 in L 𝑟𝑒𝑔 . Such an objective begs a problem: We cannot optimize unavailable modalities in 𝐼𝐷𝑇 . In other words, it will result in a severe imbalance across modalities given imbalanced missing rates, as L 𝑟𝑒𝑔 would emphasize more on modalities with lower missing rates for loss minimization. What's worse, as each single modality is not well optimized, it may cause severe fusion issues in the SOTA paradigm. Such observations motivate us to balance each modality's performance during unified end-to-end training given imbalanced modality missing rates.Motivated by knowledge distillation (KD) [18] , which aims to transfer the teacher's "dark knowledge" to students via soft labels, we treat multi-modal knowledge as a common objective for each available uni-modal to balance inter-modal learning. As multi-modal knowledge is learned through all modalities, it may be dominated by certain modalities of lower missing rates. When penalized through KD, such modalities would be less emphasized as they are closer to multi-modal knowledge (i.e., soft labels). In this way, it is promising to re-balance modalities in 𝐼 𝐷𝑇 . Unlike previous KD-based works relying on a separate complete multimodal teacher based on 𝑃𝐷𝑇 [1, 6, 36, 39] , we prefer a unified network to transfer knowledge from multi-modal to uni-modal, denoted as multi-uni self-distillation. On the one hand, it reduces the difficulty of training a sufficiently robust teacher model under 𝐼𝐷𝑇 . On the other hand, multi-modal knowledge is innate but under-exploited for uni-modal in a unified framework. Specifically, multi-uni self-distillation is composed of pixel-wise and semanticwise self-distillation described in the following.Pixel-wise Self-Distillation. As segmentation can be formulated as a pixel-level classification task, we propose to align the predictions of each pixel between multi-modal and uni-modal. It is based on the observation, validated through experiments, that feature-level alignment usually results in multi-modal performance degradation while logit alignment in deep supervision is a more robust option for imbalanced learning [26] . Therefore, pixel-wise multi-uni self-distillation for each uni-modal 𝑚 is formulated as:EQUATIONwhere 𝐾𝐿 denotes the Kullback-Leibler divergence, 𝜏 is the temperature hyper-parameter, 𝜎 is the softmax function, and 𝑧 𝑙 𝑛 and 𝑧 𝑚,𝑙 𝑛 represent features of multi-model and modality 𝑚 respectively from layer 𝑙 of sample 𝑛. Semantic-wise Self-Distillation. By learning more information from diverse modalities, the multi-modal teacher captures more robust intra-and inter-class representations [36] . Therefore, not only local pixel-wise knowledge but also global class-wise knowledge should be transferred to uni-modal. By using prototypes to represent the general features of a class, we hope to build multi-modal and uni-modal prototypes to achieve global knowledge transfer. where 𝑧 0 𝑛,𝑖 and 𝑧 𝑚,0 𝑛,𝑖 represent the multi-and uni-modal (i.e., modality 𝑚) output (i.e., layer 𝑙 = 0) features of pixel 𝑖 for sample 𝑛. 𝟙 represents an indicator being 1 if the argument is true or 0 otherwise. To measure the semantic richness within 𝑐 𝑚,𝑠 𝑛,𝑘 , we define 𝑆 𝑚,𝑠 𝑛,𝑘 to calculate the feature-prototype similarity for class 𝑘, defined asEQUATIONwhere 𝐶𝑜𝑠 (•, •) denotes cosine similarity. It should be noted that 𝑆 𝑚,𝑠 𝑛,𝑘 measures both intra-or inter-class semantic correlations depending on whether pixel 𝑖 belongs to class 𝑘 or not. Following this, the semantic richness of multi-modal prototypes 𝑐 𝑡 𝑛,𝑘 for class 𝑘 is defined asEQUATIONThen, the semantic-wise knowledge gap across all classes between uni-modal 𝑚 and multi-modal is defined as𝐷 𝑚 𝑛 = ∑︁ 𝑖 ∑︁ 𝑘 ∈ [𝐾 ]𝑆 𝑚,𝑠 𝑛,𝑘 (𝑖) -𝑆 𝑡 𝑛,𝑘 (𝑖)EQUATIONwhere ∥•∥ 2 denotes the ℓ 2 -norm. Accordingly, semantic-/class-wise knowledge transfer for modality 𝑚 is accomplished by minimizingL 𝑚 𝑝𝑟𝑜𝑡𝑜 = ∑︁ 𝑖 ∑︁ 𝑘 ∈ [𝐾 ]𝑆 𝑚,𝑠 𝑛,𝑘 (𝑖) -𝑆 𝑡 𝑛,𝑘 (𝑖)EQUATIONHere, using an L2-like loss is to further up-weight weak modalities for balanced optimization.As discussed above, multi-uni self-distillation equally transfers knowledge to available uni-modal to balance inter-modal learning. However, it may still make those modalities with higher missing rates struggle to keep up with others. This is because the recurring uni-modal wins at optimization more often than others. Therefore, dynamically evaluating how strong (or weak) each uni-modal is compared to others and balancing the learning paces across them in 𝐼𝐷𝑇 is crucial.Relative preference. Since different uni-modal students are born with unequal talents and expertise, measuring the learning progress of different students directly through their performance seems to be inappropriate. Noting that the multi-modal teacher usually prefers strong modalities that are easy to learn or have lower missing rates, we use the distance 𝐷 𝑚 𝑛 to represent the relative preference of the multi-modal teacher to uni-modal 𝑚. It is based on the observation that the higher 𝐷 𝑚 𝑛 is, the more modality 𝑚 is neglected. Given any sample 𝑛, the average distance is calculated as: D𝑛 = ∑︁EQUATIONwhere 𝐶 𝑛𝑚 indicates the presence of modality 𝑚 in sample 𝑛 according to the modality presence indication matrix 𝐶 defined in Sec. 3.1. With D𝑛 , the relative preference to modality m is formally defined as:EQUATION)When 𝑅𝑃 𝑚 𝑛 > 0 (resp., < 0), modality m is more preferred (resp., neglected) compared to other modalities for sample 𝑛. Especially, 𝑅𝑃 𝑚 𝑛 = 0, if modality m is at the balancing point or unavailable.Re-balancing regularization. Given the relative preference 𝑅𝑃 𝑚 of modality 𝑚 for each sample, it is expected to slow down the learning paces of the preferred modalities and speed up the neglected ones. Inspired by previous works on imbalanced multimodal learning [12, 28, 34] , we develop two regularization items for each modality: task-wise and gradient-wise.Noting that strong modalities occupy the center of gravity in early training due to their larger data amounts and easy learning properties. In task-wise, we propose to identify and push the slowlearning modalities based on sample-level relative preference. If 𝑅𝑃 𝑚 𝑛 is negative, we should further accelerate the slow-learning modality m. Therefore, we set a task mask 𝛿 𝑚 𝑛 . If modality 𝑚 is neglected, it should be accelerated for training by enhancing its semantic learning from multi-modal. Such a rule is formulated as:EQUATIONwhere 𝟙 represents an indicator being 1 if the argument is true or 0 otherwise. It should be noted that 𝛿 𝑚 𝑛 is to regularize semantic-wise self-distillation through L 𝑚 𝑝𝑟𝑜𝑡𝑜 . In terms of gradient-wise regularization, we propose to continually balance pixel-wise self-distillation through epoch-level relative preference. Let 𝛽 𝑚 denote the gradient-weighted coefficient of modality 𝑚 which is initialized as 1 1-𝑀𝑅 𝑚 to balance the inequality in modality amounts. Then we calculate epoch-average relative preferenceEQUATIONand apply gradient descent to update 𝛽 𝑚 every epoch byEQUATION)where 𝛾 is the updating rate and 𝑟 indexes the epoch. For modality 𝑚 with R𝑃 𝑚 > 0 or R𝑃 𝑚 < 0, its learning speed will be decreased or increased accordingly.Combining multi-uni self-distillation and preference-aware regularization, the overall optimization objective is written asEQUATIONwhere 𝜆 1 and 𝜆 2 are balancing hyper-parameters. 
experiments 4.1 datasets and evaluation metrics: 
 Two segmentation tasks with publicly-available multi-sequence MRI datasets are adopted for evaluation, including:(1) BraTS2020 [27] : The multimodal brain tumor segmentation challenge (BraTS2020) dataset consists of 369 cases with ground truth labels and four MRI modalities (i.e., T1, T1c, Flair, and T2). Ground truth is provided with the normal tissue and three tumor sub-regions including the necrotic and non-enhancing tumor core (NCR/NET), the peritumoral edema (ED), and GD-enhancing tumor (ET For image pre-processing, each brain MRI volume is re-sampled to the 1𝑚𝑚 3 resolution and each cardiac CMR scan is resampled into the same spatial resolution. Following [6, 9] , we cut out the black background areas outside the brain, center-crop the heart regions, and further normalize the intensity of each volume to zero mean and unit variance. Considering the specificity of two datasets, we perform the former task (i.e., BraTS2020) in 3D and the latter (i.e., MyoPS2020) in 2D to verify the flexibility of PASSION.For evaluation, two most commonly-used metrics are selected, including the Dice similarity coefficient (i.e., Dice) and Hausdorff distance (i.e., HD). Dice measures the voxel-wise accuracy and HD evaluates the surface distance. Higher Dice and lower HD indicate better segmentation performance. Quantitative evaluation is performed on subject-level volume segmentation to be consistent with the BraTS and MyoPS challenges.For comparison against SOTA imbalanced multi-modal learning approaches, mmFormer [44] is set as the backbone while for plugand-play evaluation another two SOTA multi-modal segmentation methods RFNet [9] and M2FTrans [33] are included as backbones. All models are implemented and modified with the same basicdimension size in Pytorch and trained using the optimizer AdamW [24] with an initial learning rate of 2e-4, a weight decay of 1e-4, and a batch size of 1 on NVIDIA Geforce RTX 3090 GPUs for 300 epochs. Specifically, we adopt a poly decay strategy with 𝑝 = 0.9 during training and set the temperature 𝜏 for pixel-wise self-distillation as 4 and the hyper-parameters 𝜆 1 and 𝜆 2 as 0.5 and 0.1 respectively for the self-distillation loss. The parameter setting for re-balancing Table 1 : Quantitative comparison on BraTS2020 and MyoPS2020 under various settings. 𝑀𝑅 is short for the missing rates of modalities (T1/T1c, Flair, T2) for BraTS2020 and (bSSFP, LGE, T2) for MyoPS2020 respectively. 𝑠, 𝑚, and 𝑙 correspond to small, medium, and large, set as 𝑠 = 0.2, 𝑚 = 0.5, and 𝑙 = 0.8 for BraTS2020 and 𝑠 = 0.3, 𝑚 = 0.5, and 𝑙 = 0.7 for MyoPS2020 respectively. 𝑃𝐷𝑇 denotes balanced modality distributions 𝑀𝑅 = (0, 0, 0) for perfect data training. Here, T1 and T1c share the same missing rate but are treated as separate modalities. supervision is 𝛾 = 0.01. During training, each brain volume is randomly cropped to 80 × 80 × 80 pixels, each cardiac scan is centercropped to 256×256 pixels, and both are then augmented by random rotation and intensity change. To emulate modality-imperfect data training, we randomly mask each sample's modalities according to the corresponding missing rates before training, directly drop those masked-modality inputs (i.e., invisible), and set corresponding encoded features as all-zero vectors in training.Quantitative evaluation. Quantitative comparison results by introducing SOTA modality-balancing approaches to the baseline (i.e., mmFormer [44] ) on BraTS2020 are summarized in Table 1 . Compared to PDT, segmentation under imbalanced modality missing rates is more challenging, resulting in apparent performance degradation of the baseline. Through modality re-balancing, PMR [12] achieves marginal performance improvements under most settings while ModDrop [40] even brings negative effects. This is because modalities are imbalanced not only in data amounts but also in semantics. For instance, Flair and T2 are much more informative than T1/T1c. Given relatively limited Flair and T2, it is more challenging to explore sufficient semantic information, making both PMR and ModDrop fail. Simply re-balancing modalities without considering semantic misalignment can be counter-productive. Quantitative evaluation. Quantitative comparison results on My-oPS2020 are summarized in Table 1 . Different from BraTS2020, modality-wise semantic information across modalities is closer in MyoPS2020. In other words, modalities contribute more equally to different sub-types instead of certain modalities being dominant for specific sub-types. As a result, both ModDrop and PMR achieve noticeable performance improvements by modality re-balancing. Comparatively, PASSION consistently achieves greater performance improvements under all modality settings, demonstrating its superiority in exploring richer semantic information. Visualized quantitative comparison in Figure 3 further validates this. Qualitative evaluation. Exemplar segmentation results on My-oPS2020 are illustrated in Figure 5 . Compared to BraTS2020, segmentation performance across different settings is more consistent even given only T2 (i.e., the least modality), indicating more equal semantic contributions of modalities. With more modalities, more false positives are recalled while more false positives are introduced. Comparatively, PASSION achieves the most consistent segmentation performance with the fewest false positives and false negatives.We first validate the concept of relative preference defined in Section 3.3 by plotting 𝑅𝑃 curves with and without PASSION and PASSION is expected to work as a plug-and-play module onto various backbones for modality re-balancing. To validate this, PASSION is integrated into SOTA incomplete multi-modal medical image segmentation methods and evaluated under various modality missing rates, as summarized in Table 3 . Compared to 𝑃𝐷𝑇 , there exists noticeable performance degradation under 𝐼𝐷𝑇 for all backbones. By introducing PASSION, consistent performance improvements across different sub-types under various modality combinations are achieved. One interesting observation is that adopting PASSION to address 𝐼𝐷𝑇 even outperforms 𝑃𝐷𝑇 for WT segmentation, proving the robustness and flexibility of PASSION. 
conclusion: 
 In this paper, we present PASSION to solve a new challenging task, namely incomplete multi-modal medical image segmentation with imbalanced missing rates. PASSION adopts pixel-wise and semantic-wise self-distillation to regularize modality-specific optimization objectives and preference-aware regularization in taskwise and gradient-wise to balance convergence rates of different modalities. Comprehensive evaluation demonstrates that PASSION outperforms SOTA modality-balancing approaches and works as a plug-and-play module for consistent performance improvement across various backbones. 
 
 Appendix:  


Decision: There is a consistent recommendation of acceptance from all reviews.

Reviews: review: paper_summary: paper_summary: A new self-distillation solution called PASSION has been proposed to balance different patterns in optimization objectives and pace. It has been proven on two datasets to serve as a plug-and-play module that enhances performance on various backbones.
strength_weakness: strength_weakness: 1. For the first time, a new scenario of imperfect data training has been proposed, exploring incomplete multimodal medical image segmentation with imbalanced missing rates.

2. The paper provides a clear description of the problem and detailed methodological details.
limitations: limitations: 1. What does the superscript 's' on all symbols after line 381 mean?

2. Do all modalities need to be pre-aligned? For multimodal segmentation tasks, registration is an important prerequisite step; what impact does this have on the applicability of this method?
paper_summary: paper_summary: This paper presents a novel approach, PASSION (Preference-Aware Self-diStillatION), for handling incomplete multi-modal medical image segmentation with imbalanced missing rates. The authors identify and address the realistic scenario where different imaging modalities have varying degrees of availability during model training. PASSION leverages self-distillation to balance optimization objectives across modalities and introduces task-wise and gradient-wise regularization to ensure balanced convergence rates. Experimental results on two datasets demonstrate the effectiveness of PASSION, which also functions as a plug-and-play module compatible with various backbones.
strength_weakness: strength_weakness: (1) Innovative Methodology: The paper introduces PASSION, a unique self-distillation method that balances optimization objectives and convergence rates for different modalities, addressing a significant gap in current research.

(2) Empirical Validation: The experimental results on BraTS2020 and MyoPS2020 datasets show that PASSION outperforms state-of-the-art methods in handling imbalanced modality availability, proving its practical utility in author's own settings.

(3) Plug-and-Play Module: PASSION is designed to be easily integrated into existing models, providing consistent performance improvements across different network architectures.
limitations: limitations: (1) Task settings: 

According to Table 3, there seems to be no significant performance difference (approximately 10%) between the IDT and PDT settings. Can the authors further explain the necessity of this task setup?

(2) Limited Evaluation: 

Although the experimental results are promising, the evaluation is limited to two datasets. Additional datasets, such as other BraTS datasets, could further validate the generalizability of PASSION. Moreover, it is recommended to include more method comparisons to verify the validity of the proposed method. In Table 3, the results show that the proposed method is effective in this setting, but it is unclear to what extent it is effective. It would be beneficial to add performance comparisons with other algorithms, such as the one mentioned in reference [30].

(3) Parameter Sensitivity: 

The paper does not discuss the sensitivity of PASSION to different hyperparameters, which is crucial for understanding its robustness and adaptability. For instance, the choice of missing rates in BraTS2020 (MR = 0.2, 0.4, 0.6, 0.8) needs further explanation. How were these parameters determined, and is there experimental data to justify these specific missing rates? Additionally, is the proposed method effective only with these parameters? It would be beneficial if the authors could add experiments to verify the sensitivity of the method to various hyperparameters.

(4) Suggested Citations:

  [1] Wang, Z., Hong, Y. (2023). A2FSeg: Adaptive Multi-modal Fusion Network for Medical Image Segmentation. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. 

  [2] Yansheng Qiu, Ziyuan Zhao, Hongdou Yao, Delin Chen, and Zheng Wang. 2023. Modal-aware Visual Prompting for Incomplete Multi-modal Brain Tumor Segmentation. In Proceedings of the 31st ACM International Conference on Multimedia (MM '23)
paper_summary: paper_summary: This paper introduces a novel approach to address the challenge of incomplete multi-modal medical image segmentation in the presence of imbalanced missing rates. Through Preference-Aware Self-diStillatION (PASSION), the authors propose a method that constructs both pixel-wise and semantic-wise self-distillation mechanisms. This approach aims to balance the optimization objectives across different modalities, enhancing segmentation performance. The paper presents a comprehensive analysis of the proposed method's effectiveness, highlighting its potential to significantly improve segmentation accuracy in challenging scenarios.
strength_weakness: strength_weakness: The paper addresses a more realistic and complex issue in the field of medical image segmentation, specifically incomplete multi-modality segmentation under imbalanced missing rates. This represents a significant advancement as it mirrors the practical conditions encountered in real-world clinical settings where data from different imaging modalities can be incomplete or unequally available.

The proposed method introduces a sophisticated self-distillation framework that operates at both pixel-wise and semantic-wise levels. The solution adjusts the optimization paces of different modalities, ensuring that each modality contributes appropriately to the overall segmentation task. This balancing act enhances the model's ability to effectively utilize available data, leading to improved segmentation performance even when faced with significant modality imbalances during training.
limitations: limitations: The proposed multi-unimodality distillation method bears a resemblance to the techniques presented in [1]. The paper does not adequately differentiate itself from this prior work, leading to an over-claim of its novelty. To strengthen its contribution, the authors should provide a clear comparison with this paper, highlighting the unique aspects and improvements of their approach.

The method described in the paper uses a simple concatenation of different modalities before passing them to the fusion decoder. This approach is relatively basic compared to more advanced fusion techniques utilized in RFNet and mmFormer, or even the simple mean fusion methods employed in [1] and HeMIS. The paper does not provide a rationale for why more sophisticated fusion methods were not considered or implemented. Given that advanced fusion techniques could potentially yield better segmentation performance, the authors need to justify their choice and discuss the limitations of their current fusion strategy.

References:
[1] Q. He, N. Summerfield, M. Dong, and C. Glide-Hurst, "Modality-Agnostic Learning for Medical Image Segmentation Using Multi-modality Self-distillation," *arXiv preprint arXiv:2306.03730*, 2023.
paper_summary: paper_summary: The author designed Preference-Aware Self-diStillatION (PASSION) for incomplete modality tasks set under imperfect data training. PASSION uses the prediction branch of the fusion as a teacher to transfer knowledge to the single modality feature extraction branches at both the pixel level and the semantic-wise level. To balance learning between modalities, task-level and gradient-level regularization were introduced.
strength_weakness: strength_weakness: The approach proposed in the paper is solving a problem of significant importance. They have proposed a novel approach and demonstrated state-of-the-art results on multiple defined datasets. They have described their approach, datasets, implementation details, evaluation metrics and results very well.
limitations: limitations: 1. This is an initial draft and is not yet complete. Figure 2 is difficult to comprehend due to the presence of numerous question marks.
2. Although, I am a bit unsure how this paper is directly related to multimedia but the approach discussed in the paper can certainly be extended to multimedia use cases as well and hence can act as a source of inspiration which can drive further research in the field of multimedia.