MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks
Authors: Alejandro Guerra-Manzanares
Farah Shamout
Venue: Accepted by TMLR ()

Abstract: Multimodal fusion leverages information across modalities to learn better feature representations with the goal of improving performance in fusion-based tasks. However, multimodal datasets, especially in medical settings, are typically smaller than their unimodal counterparts, which can impede the performance of multimodal models. Additionally, the increase in the number of modalities is often associated with an overall increase in the size of the multimodal network, which may be undesirable in medical use cases. Utilizing smaller unimodal encoders may lead to sub-optimal performance, particularly when dealing with high-dimensional clinical data. In this paper, we propose the Modality-INformed knowledge Distillation (MIND) framework, a multimodal model compression approach based on knowledge distillation that transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student. The teacher models consist of unimodal networks, allowing the student to learn from diverse representations. MIND employs multi-head joint fusion models, as opposed to single-head models, enabling the utilization of unimodal encoders in the case of unimodal samples without requiring imputation or masking of absent modalities. As a result, MIND generates an optimized multimodal model, enhancing both multimodal and unimodal representations. It can also be leveraged to balance multimodal learning during training. We evaluate MIND on binary classification and multilabel clinical prediction tasks using clinical time series data and chest X-ray images extracted from publicly available datasets. Additionally, we assess the generalizability of the MIND framework on three non-medical multimodal multiclass benchmark datasets. The experimental results demonstrate that MIND enhances the performance of the smaller multimodal network across all five tasks, as well as various fusion methods and multimodal network architectures, compared to several state-of-the-art baselines.

Published in Transactions on Machine Learning Research (01/2025)
MIND: Modality-Ifformed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks
Alejandro Guerra-Manzanaree
Engineering Division
New York University Abu Dhabi
Farah E. Shamout
farah. Strab.shamoudttyy..comm
Engineering Division
New York University Abu Dhabi
openreview. Reviewed on OpenReview: https: opterev//pe net/ net/forum? id= BhO Ymur
Abstract
Multimodal fusion leverages information modalities to learn better feature representa- tions with the goal of improving performance in fusion-based tasks. However, multimodal datasets, especially in medical settings, typically smaller than their unimodal counterparts, which can impede the performance of multimodal models. Additionally, the increase in the number of modalities is often associated with an overall increase the size of the multimodal network, which may undesirable in medical use cases. Utilizing smaller unimodal encoders may lead sub-optimal performance, particularly when dealing with digh-dimensional clini- cal data. In this paper, we propose the Modality--Nformed knowledge Distillation (MIN) framework, multimodal model compression approach based knowledge distillation that transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student. The teacher models consist of unimodal networks, allowing the student to learn from diverse representations. MIND employs multi-head joint fusion models, as opposed to single-head models, enabling the utilization of unimodal encoders in the case of unimodal samples without requiring imputation or masking of absent modalities. As result, MIND generates an optimized multimodal model, enhancing both multinodal and unimodal representations. It can also be leveraged to balance multimodal lcarning during training. We evaluate MIND on binary classification and multilabel clinical prediction tasks using clinical time series data and chest X-ray images extracted from publicly available datasets. Additionally, we assess the generalizability of the MIND framework on three non-medical multimodal multiclass benchmark datasets. The experimental results demonstrate that MIND enhances the performance of the smaller multimodal network across all five tasks, as well as various fusion methods and multimodal network architectures, compared several state-of-the-arl baselines.
1 Introduction
Healthcare decision-making like most human-based active reasoning (Smith & Gasser 2005 is inherently multimodal. This means that clinical decisions typically informed by multiple sources of information modalities, such as diagnostic imaging, clinical time-scrics data, or medical history, combined with clinical experience and espertise (Bale et al. 2012; Pauker & Kassirer 1980 Pines el al. 2023). As a result, recent research has demonstrated the potential of multimodal learning in modeling various clinical prediction tasks [Huang et al.] (2220), particularly in information fusion enhance downstream classification performance compared relying solely on unimodal network (Hayat et al.) 2022 Yang al. 22022 Wang et al. 2020a).
Note that the term unimodal model refers to modality-spechh neural network, i.e., model that processes

---

Published in Transactions on Machine Learning Research (01/2025)
samples consisting of single data modality (e.g., image), while the term multimodal model refers to network that processes samples consisting of multiple data modalities.
Despite recent advancements, the nature of the learning process in multimodal clinical prediction tasks remains unclear. Most research primarily focuses on achieving optimal fusion performance (Huang et al.) 2020), often overlooking additional challenges of multimodal training, such as optimizing modality utilization and modeling cross-modal interactions for effective training (Huang et al. 2022b] Wang et al. 2020b Wu ct al. 2022b). For instance, recent findings indicate that multimodal deep neural networks overfitting due to their larger capacity (Wang et al. 2020b). They tend to rely primarily on one of the available imput modalities, specifically the one that is fastest to learn from (Wu et al. 2022b). Since different modalities may generalize and overfit at different rates, naive joint training often leads to sub-optimal results WWang et al. 2020b).
prone to
The methodological challenges of multimodal learning are coupled with other challenges when applied healtheare. This includes the limited availability of large-scale multimodal clinical datasets, as well the sparse and heterogeneous nature of the input modalities, such as when fusing medical images with clinical time-series data. The increase in the nunber and heterogeneity of modalities entails the use of modality-specifie encoders, such as convolutional neural networks for images and recurrent neural networks for time-series data (Hayat et al. 2022). This requirement increases the overall size of the multimodal network, which may hinder deployment in clinically relevant use cases such as crivacy-preservin training (Baruch) et al. 2022). secure inference (Lou et al. 2021), , and applications on resource-constrained devices (Zhao et al, 2018
Our objective in this work is improve the compression of multimodal networks, both in terms of size and predictive performance when dealing with small multinodal datasets. We study this within the context of joint fusion, where two input data modalities have their encoded representations aggregated before applying a fusson layer. To achieve this objective, we propose a training framework based on knowledge distillation that incurs minimal modifications to the joint fusion architecture and learning objective, while siguificantly enhancing predictive performance and reducing overall size. We refer to this training framework as Modality- knowledge Distillation (MIND). Specifically, we make the following contributions:
We propose integrating mopality-speeific supervision signals into the joint fusion architecture. This approach enhances the representations of unimodal encoders by reducing the influence of the global model, allowing for a stronger focus on each unimodal encoder. As a result, this leads to enhanced performance in multimodal fusion.
We propose pre-training unimodal teachers to distill knowledge into modality-speciffc signals, com- pressing the knowledge from an ensemble larger unimodal models and enhancing the representations learned by the modality encoders of the student model. Our results demonstrate that this approach significantly improves fusion performance while utilizing more compact. student network.
We introduce additional weighting paper-parmmetrrs to emphasize modality-spccifie learning. Our empirical results demonstrate that this approach improves both unimodal and fusion (multimodal) performance, while also helping to balance modality learning during multimodal training.
2 Background
Knowledge distillation. Knowledge distillation (KD) (Hinton et al. 2015), also referred as model compression (Buciluă et al. 2006) teacher-student networks (Gou et al., 2021) , allows transferring the generalization capability of large model into a typically smaller model. The most common technique is response-based distillation, which leverages the output class probabilities of the larger model as soft targets for training the smaller model (Hinton et al. 2015). In the oflline knowledge distillation setting, the aim of the student network is to mimic the performance of the teacher network by approaching the softened responses of the pre-trained teacher network. The main rationale is that the function learned by a large model can be approximated by shallower model, which is computationally less expensive to train (Goul et al. 2021 Although increased network depth can improve learning, it is not always necessary desirable
2

---

Published in Transactions on Machine Learning Research (01/2025)
Ba & Caruana 2014). Other approaches involve approximating the features and layers of models, or the
relationships between instances, in online or self-distillation settings (Gou et al. 2021
KD aims to transfer the knowledge and generalization capabilities of a large model to a smaller model
Hinton et al, 2015), typically using student-teacher networks (Gou et al. 2021 In a multiclass classification
problem, the student network mimics the predictions of the teacher model during training (Gou et al., 2021
by minimizing:
C Cs(y.p(Ze: T = 1)) + CKD(P(Z, T) T))
(1)
supervised learning
knowledge distillation
where Ls is the supervised learning loss applied to the student network (Cross-Entropy (CE) loss), 2, are the logits of the student, T is a temperature parameter that regulates the importance of each target class, and p is the output probability obtained after applying softmax activation. LKD is the KD loss, which quantifics the divergence between the teacher (zt) and student. outputs (zæ); either using the Leilback-Leibler divergence loss (KL) or CE. In a multiclass sctting, when r 1, the supervised loss directly yields ÿ. In contrast, when T 1, KD generates the soft largets for the distillation loss.
Multimodal learning & knowledge distillation. Recent work leveraged KD for variety of multimodal learning tasks, such as for gencrating a multimodal student from unimodal teacher (Xue et al. 2021 ) addressing the limitations of CLIP (Dai et al., 2022), multimodal generation via cross-modal vision-language KD [Radford et al. 2021J, and optimized action recognition using multimodal sensors via wavelet KD (Quan et al. 2023). Other studies focused on handling missing modalities and improving computational efficiency by shifting from multimodal to unimodal attention (Agarwal ct al. 2021), or improving unlabeled image selection from pre-trained vision-language model (Zhang et al. 2022). The findings of previous work highlight the versatility and effectiveness of KD in improving the performance of various multimodal learning tasks. In our work, we focus on KD in the context of multimodal fusion networks, leveraging it to compress knowledge from an ensemble of teachers.
Knowledge distillation in healthcare. KD has been utilized to compress deep neural networks for a wide range of clinical prediction tasks. For unimodal tasks, several studies explored KD for federated learning
(Chen et al. 2022) and aulti-institution collaboration (Huang et al., 2023), medical text classification (Huang
ct al. 2022a electronic health record mining (Du & Hu, 2020). and multiclass classification for medical
images Yang et al. 2022 Ho & Gwak, 2020; Soni et al. 2019). Wang et al. (2020a) and Dou et al. (2020)
explored KD in the context of multimodal imaging data for multiclass classification, as well as for multiclass
cardiac structure and multi-organ segmentation, respectively. In another study (Dou et al., 2020). the authors proposed to learn shared representations among imaging modalitics to distill knowledge for modality segmentation, while Wang et al.] (2020a) distilled the knowledge of a unimodal teacher into a multimodal student. Another work (Hu et al.) 2020) investigated distilling knowledge from a multimodal teacher to unimodal student network for clinical image segmentation. In general, the use of KD for multimodal relevant problem in healthcare applications where multiple labels may be simultaneously present in a sample- - has been under-explored. In contrast, our work investigates the application of KD for model compression of multimodal networks for heterogencous medical modalities, focusing on binary and multilabel classification tasks.
clinical prediction tasks, and for multilabel classification in particular
Training of multimodal networks. Joint training of multimodal fusion networks is a challenging task due to the tendency of these networks to overfit one of the input modalities (Wang et al., 2020b).. As shown in recent work (Huang et al. 2022b), the different input modalities compete against each other during joint training, leading to sub-optimal joint training where only subset of the input modalities is learned efficiently, while the rest remain unexplored and contribute minimally (Wu et al. 2022b). Several techniques have been proposed to generate a more balanced training approach for multimodal learning (Huang el. al. 2022a Wu et al. 2022b Wang et al., 2020b Peng et al.) 2022), utilizing metrics to assess the degree of modality overfitting [Wang et al. 2020b Wu et al. 2022b). In this work, we leverage the conditional utilization rate (Wu et al.) (022b) to demonstrate that, as a by-product of our proposed framework, the weights of the distillation loss can be used to emphasize learning from specific modalities. This approach improves
3

---

Published in Transactions on Machine Learning Research (01/2025)
Muiti-modal joint fusion
MIND: Modality-Informed
Knowledge Distillation
fa
Uni modil knowledge pre franne
9AB
3 AB
fa
3,
fn
XA
9A
AAB
AAB
fA
9A
A
fB
9B
9AB
8
Utiimooaal knowledge trairnedd
X8 fB
9B
Figure Architecture of multimodal fusion network. A.1 Architecture for multimodal joint fusion, in recent work (Hayat et al. 2022). A.2 Modified base architecture that incorporates modality-specific classification heads with the loss shown in Equation 2 B. Architecture of the MIND framework, featuring the loss shown in Equation 5] which integrates unimodal pre- pre-trained teachers for KD into the unimodal encoders. (WA, (B) to emphasize encoder representation learning
It also includes modality weighting and improve modality utilization during multimodal training.
modality utilization and leads to a more balanced multimodal training process, enhancing both multimodal and unimodal predictions of the model trained via the MIND framework.
3 Methods
Preliminaries. In a simple multimodal fusion task, we assume the presence of two input modalities represented by XA and XB. The goal of the multimodal fusion task to jointly learn from both modalities predict set of ground-truth labels denoted by y. Due to heterogeneity, each modality is first processed by modelity-specifie encoders, such that 2 (x(x)) and Zg fD(XB), where ZA and Zø are latent feature represontations. These latent representations are concatenated and processed by a fusion classification layer, such that YBB = 9AB (Z,(ZA,22 2g). We then apply the binary cross-entropy (BCE) loss, denoted by LSAD SAB) and optimize fA, I',, and 9AB jointly, as shown in Figure 1 A.1.
Multimodal loss. We incorporate two additional loss terms by introducing classification modules for cach modality, such that Y (z(z)) and ÿB = (B(z)). Hence, the overall supervised learning loss becomes:
Ls LSAD (y, & Jusion) LSA 8A) + Lsp (8,
(2)
This modification allows for the independent use unimodal classification heads in the event unimodal samples at inference time, as illustrated in Figure [ A.2. Note that the term unimodal sample refers a data sample that contains only one modality (e.g., A or B), whereas a multimodal sample involves the presence of more than one modality (e.g., A and B).
4

---

Published in Transactions on Machine Learning Research (01/2025)
Unimodal teachers. We use KD compress the knowledge of an ensemble of unimodal teacher models into single, typically smaller student model. KD was originally conceived for multiclass classification problems, where different values of T are used to generate the soft targets. In our work, we propose to use KD for a wider range of predictive tasks inchuding binary and multilabel classification tasks. For multilabel classification, we modily the LKD lerm in Equation lo:
L Σ BCE ,''",
L
(3)
N
where L is the set possible labels in a multilabel classification problem, N the sample size, and s and denote the student and teacher networks, respectively.
Given an ensemble T of K teachers, we define new average ensemble prediction for i-th label (omitting without loss of generality):
(4)
K
such that the Ensemble Knowledge Distillation (EKD) loss is now denoted as LEKD(Y", (""))
In a comprehensive multimodal setting, the goal of the classification model is accurately classify both multimodal and unimodal samples, which is particularly relevant in healthcare applications where some modalities may be absent in sample. To improve the representations of multimodal and unimodal encoders, we introduce LEKD", which consists of an ensemble of unimodal teachers trained with sub-components of 2 for each modality, namely Ls, and L5.- Hence, we introduce a new learning objective term for each input modality to distill knowledge from unimodal teachers to the modality encoders of a smaller multimodal student, In the case two modalities (A and B), two terms are introduced:
Equation
LEKDY
LEKD
(5)
unimodal casembbl know ledwledge distillation for modality
unimodal casemble knowledge disiilationn tor modality B
We note that goal of the multimodal student network is to learn the distribution P(Y]XA,XBB) To enhance encoder representations and improve the learning of the multimodal distribution, we introduce the unimodal EKD components, consisting of LEKDY and LEKDE, which represent the distributions XAY and P(Y) respectively, as learned by the unimodal teachers.
Balancing multimodal training. We introduce weighting hyper- paper-parameters WA and wy to adjust the focus on the distillation components. This enhances encoder representation and modality learning, resulting in the overall loss defined as follows:
CRIND LSAH + LsA + Lsv + WA LEKDY + wB LEKDE
(6)
supervision signals
unimodal eusemble
knowledge distillation
A visual representation of the proposod MIND framework provided Figure D B. In particular, sctting WA,WB > minimizes both loss components (supervision signals and EKD), emphasizing distillation for unimodal encoder learning. Specifically, WA,WB > 1 prioritize knowledge distillation, while WA.WB = treat all loss components equally. When WA >> WB, the learning focuses on modality A, whereas WA < wB prioritizes modality B. This independent weighting can be leveraged to balance multimodal learning by emphasizing the less utilized modality with a larger weight. Setting WA, wp = disables distillation, reverting the learning objective to Equation 2
Overall, the MIND framework produces a smaller, optimized version of the original multimodal model. This compressed model make predictions using both multinodal (when both modalities present) and
5

---

Published in Transactions on Machine Learning Research (01/2025)
unimodal (single modality) imputs. By introducing weighted unimodal ensemble knowledge distillation, the MIND framework enhances and balances modality representation learning, significantly improving predictive performance for both multimodal and unimodal imputs. We provide a pseudo-code implementation of the MIND framework for training an enhancod, compact multimodal student network based on the MIND Iramework for two modalities Algorithmi
Algorithm MIND Framework
Require: Multimodal training dataset DABrra Multimodal validation dataset DABaal Modality A training dataset Decrann: Modality A validation dataset DA Modality B training dataset D Beracnt Modality training validation Direin Multimodal Model MAB, Loss function LXIND, Number of epochs N, Optimizer O, Weighting coefficients WA: WB
Ensure: Trained multimodal model MAB
1* Train T unimodal teacher models per modality */
/* Create ensemble of unimodal teachers per modality */
1: Σa- [TA.... .A,
2: Eg = [T"".... ....II]T
/* Train multimodal student model MAB */
3: Initialize model paramcters and add classification heads to modality encoders (MA, MB)
4: for epoch to N do Shuffle the training data DABirain for minibatch b in ABBrras do Forward pass: Compute predictions with all classification heads ÍAB. ÚA, Us, and teacher ensembles DEA DEg for minibatch b using models M, MA. MB, FA and EB Compute loss components LSAD (MAB, v), Ls (MA, v), Lso (DB. v), LEKDY (DA, DE,) and LEKDE (0B, 9Es) for minibatch following Equation 2 and Equation Compute LNuND(S LS.. , LEKDY. LEKDg WA wp) for minibatch following Equation Backward pass: Compute gradients of the loss with respect to model MAB parameters
8:
9:
10:
11:
Update model parameters using optimizer 0
12:
end for
13:
end for-0
4 Experiments
Datasets. To evaluate our approach, we focus on two clinical prediction tasks: predicting clinical conditions
(L = 25, Equation 3). and predicting in-hospital mortality after a 48-hour ICU stay (L 1, Equation 3 using Chest X-Ray (CXR) images and clinical lime-series data extracted from the patient's 's Electronic Health Records (EHR). We use chest X-ray images from MIMIC-CXR (Johnson et al.) (001b), where each image (xx) is replicated across three channels, yielding XA 22244x24x3, Associated clinical timo-scrics data (Xp) is obtained from MIMIC-IV (Johnson et al. 2023 with dimensions XB R* where t represents the number of time-steps based on the patient's 's TCU stay duration, and 76 denotes the number of pre- pro-psocessed features per time-step. We follow the dataset splits and multimodal architecture from the work of Hayat et al. 2022). where fA is parameterized by ResNet-34, f8 is parameterized as a two-layer LSTM and the fusion model 9AB parameterized as an LSST layer, referred to as MedFuse. We use the multimodal dataset. for training the MIND framework and basclines, containing samples with both modalitics present. For the clinical conditions (ask, the dataset comprises 7,728 training, 877 validation, and 2,161 test samples. For in-hospital mortality prediction, it consists of 4,885 training, 540 validation, and 1,373 test samples. When training unimodal models, we utilize all available data for each modality. Specifically, the CXR dataset comprises 124,671 training, 15,282 validation, and 36,625 test samples, while the EHR dataset contains 42,628 training, 4,802 validation, and 11,90 1,914 Lest samples.
Ensembles. In the MIND framework, we first pre-train the unimodal teachers using the available unimodal datasets. To ensure diversity, we train multiple models for each modality, following the architectures outlined in previous work (Hayat et al. 2022). For chest X-ray images, train ResNet-34, ResNet-18, and ResNet-11

---

Published in Transactions on Machine Learning Research (01/2025)
models. Similarly, for clinical time-series data, we train 2-layer, 3-layer, and 4-layer LSTM networks. The top three performing models from each modality are then combined to create ensembles. These ensembles of unimodal teachers are utilized to distill knowledge, as described in Equation G into smaller randomly initialized multimodal student model. This student model consists of a ResNet-10 and a 2-layer LSTM, serving as encoders for chest X-ray and time-series data, respectively.
Baselines. In this work, we focus on joint fusion, such that both encoders are trained from scratch (randomly initialized), although previous work primarily focuses fine-tuning pre-trained unimodal encoders (Huang et al. 2020 We believe that learning with randomly initialized encoders is more appropriate in our proposed setup for several reasons. First, randomly initialized encoders establish consistent bascline for fair comparisons across models, ensuring that any performance improvements reflect the architecture and training methods rather than pre-trained weights. Additionally, starting with random initialization helps isolate the specific learning dynamics of our setup, avoiding the potential of skewing the performance as a result of using fino-tuned pre-trained encoders. Our approach also introduces unique elements that may not align with assumptions of previous work, such as avoiding the need for pre-training the student. model, so we can assess how well the model learns from scratch to understand its full capabilities. Finally, using randomly initialized encoders eliminates biases from original training data associated with fine-tuning, allowing for a more objective evaluation of our method.
We compare the performance of MIND with the original multimodal model, MedFuse. The MIND model is three times smaller (in terms of learnable parameters) than MedFuse, as shown in Table A1 Further, we modify the original MudFuse architecture to adopt the loss introduced in Equation 2 denoted as MedFuse-3EE We also compare MIND to other related KD baselines, including TS (Wang et al., (0200a), MKE (Xue et al.) 2021), and UME (Du et al. 2023). For MKE, we evaluate two models: MKE-CXR and MKE-EHR. Further details about the baseline models can be found in Appendix AA.4 All MIND and baseline models trained for each specific task are all trained using the same dataset sizes, as described in Appendix and Appendix
Hyperparameter Tuning. All models, including baselines, are trained for a maximum of 300 epochs with early stopping after 40 epochs. We use batch size of 16 and the Adam optimizer across all experiments. Hyperparameter tuning involves optimizing the lcarning rate and weighting paramcters for MIND and all baselines. Further details on hyperparameler luning are provided in Appendix A We select the best models based on the checkpoint yielding the highest Area Under the Receiving Operating Characteristic (AUROC) on the validation set and present the results on the test set in terms of AUROC and Area Under the Precision-Recal Curve (AUPRC). We also report 95% confidence intervals with 1,000 iterations using the bootstrapping method (Efron & Tibshirani. 1994). For reproducibility, our code is included in Appendix A.55 as we aim to make it publicly available, following the pseudo-code implementation of the MIND framework in Algorithm I
We conduct our experiments on a shared high-performance computing cluster equipped with Nvidia A100 GPUs. For the clinical conditions task, training the models takes less than 20 hours (120-150 epochs), while for in-hospital mortality prediction, the models are trained in under two hours.
5
Results
5.1 Multimodal fusion performance results
Table I compares the performance of multimodal baselines with our proposed model on the test set. The MIND framework achieves the highest AUROC and AUPRC across both tasks, outperforming all baselines by over 1.4% AUROC and 2.3% AUPRC for clinical conditions prediction, and 1.6% AUROC and 1.4% AUPRC for in-hospital mortality prediction. Notably, MIND surpasses MedFuss and MedFuse-3H by over 3.4% AUROC and 5.4% AUPRC for clinical conditions, and by over 2.4% AUROC and 3.7% AUPRC for nospialt mortality. Validation sot results are in Appendix[BI. The results indicate that the MIND framework excels at compressing knowledge from unimodal teacher models while enabling unimodal predictions for unimodal samples an advantage not present in the compared frameworks.
7

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Multimodal fusion performance results. Performance results on the multimodal test set for the MIND model and all bascline models (MedFuse, MedFuso-3H, TS, MKE-CXR, MKE-EHR, and UME). The best results are highlighted in bold.
Clinical Conditions
In-hospital Mortality
Model
AUROC
AUPRC
AUROC
AUPRC
MedFuse Hayyt et al.
2022
0.748 (0.721, 0.774)
0.447 (0.402, 0.495)
0.816 (0.785, 0.847)
0.468 (0.398, 0.546)
MedFuse-3H
0.750 (0.724, 0.777)
0.452 (0.410, 0.500)
0.820 (0.787, 0.851)
0.461 (0.392, 0.541)
MKE-CXR Xue et al.
2021
0.711 (0.681, 0.739)
0.412 (0.372, 0.458)
0.707 (0.667, 0.747)
0.308 (0.255, 0.377)
MKE-EIIR [Xue al.
2021
0.744 (0.717, 0.770)
0.447 (0.405, 0.495)
0.827 (0.795, 0.856)
0.491 (0.422, 0.567)
UME Du al. 2023
0.767 (0.741, 0.793)
0.489 (0.450, 0.544)
0.818 (0.787, 0.849)
0.491 (0.419, 0.569)
TS
WWang et al. 2020s
0.768 (0.742, 0.794)
0.483 (0.439, 0.532)
0.828 (0.797, 0.857)
0,483 (0.414, 0.355)
MIND (Ours)
0.782 (0.757, 0.807)
0.506 (0.460, 0.556)
0.844 (0.815, 0.872)
0.505 (0.433, 0.587)
Acute (MIND) (TS)
Mixed (MIND)
Chrenic (MIND)
Mixed
Chronic
1.0
0.9
P-Pall ane ioctiettt dheee
Cther upper rusi chaces
eurry
mono Issmagr caused by butbbrrr sosual stteogyi
AUROC
Stack
Corductio dyctychmie
Dianet muliiua thth comple Catord
0.6
Conmary anthrrocterro de ant ofher hoor sirisos
Dicesser liv meteaiiiion
0.5
A
6
M N
W
Y
Clinical conditions
Figure Label-wise AUROC Performance. Comparison of AUROC performance between the MIND model and the best baseline (TS) for each clinical condition label and group (acute, mixed, and chronic).
Figure 2 shows the label-wise AUROC performance of the MIND model with TS, the best baseline. MIND outperforms TS in overall multimodal AUROC and AUPRC metrics and shows superior AUROC across all individual and group labels: 0.777 vs. 0.763 for acute conditions, 0.813 vs. 0.797 for mixed conditions, and 0.770 vs. 0.756 for chronic conditions. Similarly, for AUPRC (Figure B1 Appendix B.2), MIND scores 0.448 0.422 for acute conditions, 0.592 0.567 for mixed, and 0.551 0.534 for chronic conditions.
5.2 Quality of unimodal representations
We also evaluate the quality of the unimodal representations. The test set results are summarized Table 2 and Table for the clinical conditions and in-hospital mortality tasks, respectively. Additional results on the validation set are provided in Appendix B.3] For clinical conditions prediction using X-ray images, the MIND model achieves superior AUROC (0.709 vs. 0.663) and AUPRC (0.409 0.349) compared to MedFuse-3tt Similarly, for timo-scrics data, the MIND model outperforms MedFuso-3H with an AUROC of 0.746 vs. 0.715 and an AUPRC of 0.451 vs. 0.404. In the in-hospital mortality task, the MIND model improves chest. X-ray model performance by over 10% compared to MedFuse-3H while maintaining similar performance for clinical time series. These results demonstrate that the MIND model not only enhances multimodal performance but also ensures unimodal performance on par with unimodal models trained on much larger datasets (124,671 for CXR and 42,628 for EHR vs. 7,728 paired CXR-EHR samples).
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 2: Unimodal performance results for the clinical conditions task. Evaluation of unimodal and multimodal models per modality on the multimodal test sct, including the number of training samples per model. Note that unimodal models are trained on significantly larger datasets compared to the fusion models, which are trained on the multimodal set. The best results are highlighted in bold.
Chost X-Ray images
Clinical time series
Model
Training set
AUROC
AUPRC
AUROC
AUPRC
Multimodal
MedFuse-3H
7,728
0.663 (0.633, 0.693)
0.349 (0.313, 0.391)
0.715 (0.687, 0.743)
0.404 (0.365, 0.449)
MIND (Ours)
7,728
0.709 (0.680, 0.738)
0.409 (0.370, 0.455)
0.746 (0.719, 0.772)
0.451 (0.408, 0.499)
Unimodal
ResNet-34
124,671
0.704 (0.674, 0.733)
0.400 (0.361, 0.445)
ResNet-10
124,671
0.710 (0.681, 0.740)
0.413 (0.373, 0.459)
2-layer LSTM
42,628
0.744 (0.717, 0.771)
0.448 (0.406, 0 0.496)
4layer LSTM
42,628
0.742 (0.715, 0.768)
0.447 (0.404, 0.495)
Table Unimodal performance results for the in-hospital mortality prediction task. Evaluation of unimodal and multimodal models per modality on the multimodal test set, including the number of training samples per model. Note that unimodal models are trained on significantly larger datasets compared to the fusion models, which are trained on the multimodal set. The best results are highlighted in bold.
Chest X-ray images
Clinical time series
Model
Training set
AUROC
AUPRC
AUROC
AUPRO
Multimodal
MedFuse-3H
4,885
0.572 (0.529, 0.617)
0.201 (0.164, 0.251)
0.827 (0. 794, 0.857)
0.480 (0.406, 0.555)
MIND (Ours)
4,885
0.690 (0.648, 0.732)
0.290 (0.236, 0.358)
0.830 (0.795, 0.859)
0.502 (0.427, 0.577)
Unimodal
ResNet-34
124,671
0.681 (0.638, 0.724)
0.276 (0.226, 0.346)
ResNet-10
121,671
0.682 (0.643,, 0.721)
0.259 (0.214, 0.318)
2-layer LSTM
42,628
0.830 (0.801, 0.858)
0.489 (0.425, 0.569)
4-layer LSTM
42,628
0.823 (0.790, 0.853) 0.495 (0.418, 0.569)
These results highlight the MIND framework's ability to significantly enhance both multimodal and uni- modal performance. The unimodal encoders can be utilized with unimodal samples, achieving performance comparable to that of unimodal encoders trained on larger datascts. Unlike related work, our multimodal compression framework uniquely incorporates the use of unimodal encoders.
5.3 Ablation study sensitivity analysis
We conduct ablation studies to understand the impact of each component in Equation D To evaluate the benefits of ensembling, we experiment with both single-teacher KD and EKD, using three models per modality ensemble. Specifically, we consider six settings:
Supervised learning as in previous work (Hayat et al. 2022 without knowledge distillation.
2. Supervised learning with modified loss (Equation without knowledge distillation.
3. Supervised learning with unweighted unimodal single-teacher KD. Specifically, we set WA,wB in Equation 0 and use single unimodal model the modality teacher.
4. Supervised learning with weighted unimodal single-teacher KD. We apply Equation using a single unimodal model as the modality teacher.
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 4: Ablation study on MIND for multimodal and unimodal performance in the clinical conditions task. Fusion and unimodal performance on the multimodal test set for MIND with different loss components, Each setting indicates the components used in the modified loss for training and the type of knowledge distillation performed. The best results are highlighted in bold.
foss
AUROC
Fusson
AUPRC
AUROC
Chast X-Ray
AUPRC
AUROC
Thme Series
AUPRC
0.748 (0.716. 0.709)
(0.298, 0.489)
0.974)
(0.407, 0.198)
(0.64, 0.705)
(0,328
(0.685,
0:105 395, 0.450)
0,761 (8.785, 0.787)
(0.428, 0.520)
0.686 (0667, 0.716)
0.381 (0.344, 0.428)
0.730 (0.702, 0.766)
0.427 0.474)
4
4
IL706 (0.740, 0.791)
0.476 (0.432, 0.525)
0,630 (0.600, 0.718)
0.385 (0.345, (.427)
0.754 (1.706, 0.763)
0.428 (0.38 (0.386. 0.474)
5
V
0.778 (0.753, 0.800)
0. 1.116 00-153, 0.546)
0.095 (0.465, 0.725)
0.391 (0:35), 0.436)
0.758 (0.71 0.704)
0.137 (0:30), 0 0.461)
6
0.782 (0.757, (.507)
0.506 (0.000) 0.056)
0.709 (0.050, 0.738
0.409 (0.370, 0.455)
0.746 (0.718, 0.772)
0.451 (0.408, (.499)
5. Supervised learning with unweighted unimodal EKD. We set WA. = in Equation G ensemble of three leacher models.
and use an
6. MIND: supervised learning with weighted unimodal EKD (Equation
Table 4 shows the MIND setting results for the clinical conditions task, while Table B4 presents results for the in-hospital mortality task, both in terms of AUROC and AUPRC on the test set. Validation set results for both tasks can be found in Appendix B.4 We observe that adding supervised loss termss for both modalities slightly improves AUROC (from 0.743 to 0.748) and AUPRC (0.440 to 0.449). Incorporating the unimodal unweighted distillation component further improves AUROC and AUPRC to 0.761 and 0.472, respectively, with ensembling boosting these metrics to 0.766 and 0.476. Adding weighting coefficients to the distillation components significantly enhances performance on both multimodal and unimodal predictions. This demonstrates the benefits of distilling knowledge from pre-trained unimodal networks and highlights the importance of the weighting parameters in Equation G Collectively, all these factors collectively enhance the multimodal model, enabling unimodal predictions and achieving superior overall performance.
5.4 Ablation study II: balancing multimodal learning with WA and wB
We conduct a hyperparameter sensitivity analysis for the weighting parameters WA and WB introduced in Equation 0 These parameters help the model focus on unimodal representation learning, thereby improving the quality of unimodal predictions, as shown in Section [5.2] We evaluate their impact on enhancing the balance of multimodal learning using the conditional utilization rate per modality (Wu et al., (2022b), which adopt to characterize modality usage in the multimodal deep neural network. Specifically, we modify u for joint fusion multimodal neural nelworks as follows:
A() - A(SB)
A(BAB) uB A(YAB) - A(SA) A(
Uu
where A and B are two modalities without loss of generality, A(-) denotes a classification accuracy metric, ua computes the conditional utilization rate for modality A, and uB for modality B. The conditional utilization rate measures the marginal contribution of each modality to the fusion model. Following Wu et al. (2022b), duss is defined as the difference between conditional utilization rates: duti = uA up. This allows for the assessment of imbalanced modality usage within the multimodal fusion model. Specifically, duta E IR s.t. -1 1 dutil I 1, with extreme values indicating imbalanced modality usage.
We use the validation AUROC as the accuracy metric and test six settings: (i) Werr wehr 0, (ii) Werr < Wehr, (iii) Wear Wehr, (iv) Wear = Wehr, (v) Wear > Wehr, and (vi) Weer >> Wehr. Figure 3 shows the conditional utilization rates and their differences for the clinical conditions task. Table [3 details the specific values of the typer-parameters. In the setting where warr = Wehr = O, the fusion model tends to overfit the EHR modality, with utilization rate differences reaching up to 10% Assigning significantly different weights to the modalitics, as seen in the Wear >> Wehr and Wear < Wehr scttings, incrcases the conditional utilization rate of the modality with the larger weight while decreases it for the other, thereby affecting the balance of utilization rates during training. This effect can be leveraged to achieve more balanced multinodal learning,
10

---

Published in Transactions on Machine Learning Research (01/2025)
Uhh
Vxr
duch
0.14
0.00
0.00
0.12
0.85
0.10
-0.04
0.03
Value
V/luu
0.00
.0.00
0.82
0.0
26
100
0.50
26
100
Epoch
Epoch
Epoch
Wexr Welt 0
Waxr & Watr
Wexr < Welrr
Wexr Wehr
Wear Wetrr
Wezr > Wehr
Figure 3: Characterization of modality utilization in multimodal learning for the clinical con- ditions task. The line graphs depict the conditional utilization rates (Mezr, Kehr) and their difference (duas = Near - Wehrr every 5 training epochs for the six settings. Vertical lines indicate the epoch with the best AUROC performance for each model on the validation set.
as seen in the Werr > Wetr setting, which emphasizes the under-utilized CXR modality. Here, the EHR modality utilization never exceeds 10,, and the difference between the conditional utilization rales remains around 5%, the smallest observed. A smaller absolute value of duti indicates more balanced multimodal learning. Additionally, in this setting, the utilization is consistent with smoother uerr line. This indicates that assigning larger weight to the under-utilized modality results in more balanced multimodal learning, reducing overfitting of the dominant modality and increasing the use of the weaker modality. However, it's important to note that more balanced multimodal training does not always equate to better fusion performance; rather, it is related to improved unimodal encoder performance.
5.5 Additional Results on Multimodal Benchmark Datasets
The results in Section 5.1 demonstrate the utility of the MIND framework in clinical settings, which is the primary motivation for our study, particularly for multilabel and binary clinical tasks, using stato-of-tho-art architecture the base model. To evaluate the generalization and applicability of the MIND framework across diverse multimodal datasets, tasks, and architectures, we conducted additional validation on three multimodal, multiclass benchmark datasets: CREMA-D (Cao et al., 2014), S-MNIST (Khacef et al. 2019), and LUMA (Bezirganyan et al. 2024). We report the best accuracy results for each model in Table 5 For these experiments, the feature encoders consist. of ResNet-3/6 models, with the output from each encoder concatenated and used as input for the fusion model, which lincar layer. Additional implementation details, such dataset descriptions, the multimodal architectures employed, and their sizes, can be found in Appendix
As shown in Table 5 the MIND model consistently outperforms all multimodal basclines across all datasets. Additionally, the performance of the unimodal encoders either surpasses or is comparable to that of the baselines. Note that UME does not involve any multimodal training; simply averages the predictions of the unimodal models compute the multimodal prodiction. We observe that the MIND model improves multimodal prediction by an average of 1.5% and boosts the performance of the weaker modality by approximately 5.2% across tasks.
11

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Multimodal fusion performance results on multiclass classification tasks. Performance results the multimodal test set for the MIND model and all basclines (Vanilla, Vanilla-3H, TS, MKE-CXR, MKE-EHR, and UME) across the multiclass classification tasks defined by the CREMA-D, S-MNIST, and LUMA datasets. The best results are highlighted in bold.
CREMA-D
S-MNIST
LUMA
Model
Audio- Video
Audio
Video
Audio-Image
Image
Audio
Audio-Image
Audio
Image
Vanilla
0.599
0.182
0.273
0,970
0.967
0.417
0.847
0.385
0.250
Vanilla-3H
0.616
0.545
0.315
0.9666
0.981
0.604
0.870
0.539
0.580
MIE-AUDIO
Xue sl.
2021
0.612
0.555
0.235
0.661
0.515
0.521
0.715
0.456
0.346
MKE-VISUAL Xne et
2121
0.601
0.112
0.382
0.966
0.965
0.379
0.815
0.129
0.128
UME Du et al
2123
0.620
0.561
0.450
0.898
0.998
0.699
0.903
0.759
0.606
TS [Wan es al
2020a
0.625
0.328
0.274
0.970
0.930
0.473
0.894
0.610
0.509
MIND (Ours)
0.641
0.552
0.461
0.982
0.981
0.772
0.921
0.721
0.677
Overall, the results in Table I and Table 5 demonstrate that the MIND model significantly outperforms all baseline models across datasets, tasks, and architectures. These findings highlight the advantages of MIND over other multimodal training methods in terms of both multimodal and unimodal encoder performance for binary, multiclass, and multilabel Lasks. Specifically, we have validated our proposed framework on four multimodal datasets (MIMIC-IV- CREMA-D, S-MNIST and LUMA), five tasks, four architectures, and two fusson methods. MIND consistently outperforms all multimodal training basclines.
6 Discussion
Recent multimodal learning research highlights the challenges of learning from multiple modalities (Wu et al. 2022b Wang et al., 2020b). While increasing model size is sometimes feasible, it is not always desirable. Additionally, multimodal models often overfit to one modality, under-utilizing others. In this work, we propose MIND, a simple yet effective KD framework that distills knowledge from pre-trained unimodal teachers to a smaller multimodal student, enhancing both multimodal and unimodal predictive power. Our experimental results demonstrate significant benefits from this approach, likely due to unimodal encoders leveraging larger training datasets and learning better representations than those trained on the comparably smaller multimodal clinical datasets. To further improve ensemble impact, we hypothesize that stronger teachers and potentially a larger number of teachers are needed (Hinton et al. 2015). Notably, our framework is aechtectuuro-gggggcll. and can be casily adaptod to similar settings in other application domains.
While Buciluã et al. (2006), Fukuda et al. (2017), Asif et al. (2020), Li et al. (2021), and Wu el. al. (2022a) combine ensemble learning and offline response-base based knowledge distillation train compact networks, their approaches primarily focus on the unimodal networks. In contrast, we are concerned with the particularities of multimodal joint fusion networks, both in performance and size, thus propose incorporating and weighting modality-specific ensembles during student knowledge distillation as a distinctive feature. Despite methodological differences, MIND similarly avoids the need for large ensemble of classifiers, which demands significant resources during inference. In addition, the MIND framework can be used to address imbalanced multimodal learning during training and leverages modality encoders to handle unimodal samples, achieving predictive performance for unimodal samples that is on par with powerful unimodal models.
The introduction of the new learning objective (Equation 6) in our proposed approach does not compromise its applicability to real-world scenarios, particularly regarding its scalability as the number of modalities, M, increases. For two modalities, as defined in Equation 6 the loss consists of five components: three supervised learning losses and two weighted unimodal ensemble knowledge distillation losses. For M 3, the proposed loss would contain seven components: four supervised losses and three weighted unimodal ensemble knowledge distillation losses, incorporating one additional supervisod loss and one knowledge distillation loss for the new modality. Generalizing M modalities, the proposed loss would have 2M + loss terms, i.c., O(M), making our approach linearly scalable for practical settings.
12

---

Published in Transactions on Machine Learning Research (01/2025)
We customize the framework for multilabel classification, addressing less explored area in knowledge distillation, which typically focuses on multiclass classification. Multilabel classification is common in clinical prediction tasks, where multiple labels may be present in a single sample. We also evaluate our framework on a binary classification task. We demonstrate its effectiveness in two relovant clinical tasks: multilabel clinical conditions prediction and binary in-hospital mortality prediction, using the publicly available MIMIC-CXR and MIMIC-EHR datasets. In addition, we assess its performance on multimodal multiclass benchmark datasets, providing a comprehensive validation of our proposed framework across multilabel, multiclass, and binary classification tasks. Our code is publicly available to ensure reproducibility and facilitate future comparisons (see Appendix A.5
Limitations. Since our evaluation is limited to two clinical tasks, further experiments with additional clinical tasks and datasets are necessary. However, the scarcity of real-world multimodal medical datasets that include both images and clinical time-series data (heterogeneous modalities) poses significant challenge. Additionally, while our solution can help balance multimodal learning, it cannot ensure complete balance if one modality entirely dominates. In such cases, any multimodal approach may be inelficient. In future work, we aim to explore larger ensembles and employ stronger teachers. Despite the advantages of a compressed multimodal network during deployment, significant computational resources are still required during training. Specifically, our framework is embedded within the offline KD paradigm. The current state of the art in this KD paradigm establishes a two-step method: first, the training of various teacher models, followed by a second step for training the student model. This two-step process is essential to offline KD, which requires significant computational resources. However, to address this increase in resource demand, a distinctive feature of MIND is our proposal to ensemble the teacher models instead of discarding sub-optimal ones, thereby reducing computational waste. Finally, considering the socictal impact, while multimodal networks may enhance overall performance in the test cohort, further research is needed to assess their fairness and generalizability across patient subcohorts and at the instance level.
7 Broader Impact Statement
Potential Risks of Malicious Usage of Medical Models. The deployment of machine learning models in healthcare carries significant risks if misused. Malicious actors could potentially exploit these models to manipulate medical diagnoses, treatment plans, or patient data. For instance, altering a model's predictions could lead to incorrect diagnoses or inappropriate treatments, posing severe health risks to patients. In addition, the unauthorized access and manipulation of sensitive clinical data could lead to privacy breaches and identity theft.
Implicit Biases Learned from Knowledge Distillation. Machine learning models trained on clinical datasets are susceptible to inheriting and amplifying cxisting biases present in the training data. Knowledge distillation, a model training process where a smaller model learns from larger, pre-trained models, can propagate these biases. This can result in unequal treatment outcomes for different demographic groups, exacerbating health disparities. For instance, if the training data contains biases against certain ethnicities or genders, the distilled model may continue to exhibit these biases, leading to unfair treatment recommendations.
Privacy Concerns with MIMIC-IV and MIMIC-CXR data. The clinical datascts used in this research are MIMIC-IV (Johnson al. 2023) and MIMIC-CRR (Johnson al. 2019b). The authors would like to note that there are privacy concerns related to these datasets. In particular, the MIMIC-IV and MIMIC-CXR datasets are de-identified and adhere to strict privacy regulations, ensuring that patient confidentiality is maintained (Johnson et al. 2024 2019a). This allows researchers to utilize real-world clinical data for developing and testing machine learning models without compromising patient privacy.
While machine learning holds great promise for advancing healtheare delivery, it is critical to consider and address these broader impact concerns. Ensuring robust security measures, mitigating biases, and maintaining patient privacy are essential steps towards the responsible and ethical use of machine learning in healthcare.
13

---

Published in Transactions on Machine Learning Research (01/2025)
8 Conclusion
Multimodal learning offers potential enhancements for clinical prediction tasks by leveraging cross-modality interactions. However, integrating multiple modalities can lead to increased network complexity and size, posing challenges for such as modality heterogencity and missing data. Overall, our work addressos these challenges through weighted ensemble knowledge distillation, providing a promising approach to enhance fusion networks for real-world multimodal clinical data. Finally, we demonstrate the generalizability of our proposed approach by validating it across three multimodal benchmark datasets, as well two clinical tasks, two fusion methods, and various multimodal network architectures.
applications. Additionally, multimodal clinical data present hurdles
References
Dhruv Agarwal, Tanay Agrawal, Laura M Ferrari, and François Bremond. From multimodal to unimodal atlention in transformers using knowledge distillation. In 2021 17th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-8. IEEE, 2021.
Umar Asif, Jianbin Tang, and Stefan Harrer. Ensemble knowledge distillation for learning improved and efficient networks. In ECAT 2020, pp. 953-960. TOS Press, 2020.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.
Moran Baruch, Nir Drucker, Lev Greenberg, and Guy Moshkowich. A methodology for training homomorphic encryption friendly neural networks. In Applied Cryptography and Network Security Workshops: ACNS 2022 Satellite Workshops, AIBlock, AIHWS, AloTS, CIMSS, Cloud S&P, SCI, SecMT, SiMLA, Rome, Italy, June 20 23, 2022, Proceedings, pp. 336-553. Springer, 2022.
Louise Bate, Andrew Hutchinson, Jonathan Underhill, and Neal Maskrey, How clinical decisions are made. British journal of clinical pharmacology, 7A(4):614-62)) 2012.
Grigor Bezirganyan, Sana Sellami, Laure Berti- Equtil,quill, and Sébastien Fournier. Luma: A benchmark dataset for learning from uncertain and multimodal data. arXiv preprint arXiv:2406.0986, 2024.
Cristian Buciluă, Rich Caruana, and Alexandru Nieulescu-Mizil. Model compression. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 541, 2006.
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-330, 2014.
Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang, and Xing Xic. Mctafed: Federated learning among federations with eyelic knowledge distillation for personalized health arXiv preprint. arXiv:2206.08516 2022.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation clip via vision-language knowledge distillation. arXiv preprint arXiv:2203.0638) 2022.
Qi Dou, Quande Liu, Pheng Ann Heng, and Ben Glocker. Unpaired multi-modal segmentation via knowledge distillation. IEEE transactions on medical imaging, (9)::441 2425, 2020.
Chenzhuang Du, Jiaye Teng, Tingle Lí, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, and Hang Zhao. On uni-modal feature learning in supervisod multi-modal learning. In International Conference on Machine Learning, pp. 8632 8656. PMLR, 2023.
Hongbin George Du and Yanke Hu. Squeezebiobert: Biobert distillation for healthcare natural language processing. In Computational Data and Social Networks: 9th International Conference, CSoNet 2020, Dallas, TX, USA, December 11-13. 2020, Proceedings 9, pp. 193 201. Springer, 2020.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient knowledge distillation from an ensemble teachers. In Interspeech, pp. 3697-3701, 2017.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey.
International Journal of Computer Vision, 129:1789-1811 2021.
Nasir Hayat, Krzysztof J Geras, and Farah E Shamout, Medfuse: Multi- modal fusion with clinical time-series data and chest x-ray images. arXiv preprint arXiv:2207.07021, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0255 02531, 2015.
Thi Kieu Khanh Ho and Jeoughwan Gwak. Utilizing knowledge distillation in deep learning for classification of chest x-ray abnormalities. TEEE Access, 8:160749-16017 160761, 2020.
Minhao Hu, Matthis Maillard, Ya. Zhang, Tommaso Ciceri, Giammarco La Barbera, Isabelle Bloch, and Pictro Gori. Knowledge distillation from multi-modal to mono-modal segmentation networks. In Medical Image Computing and Computer Assisted Intervention MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part I 23, pp. 772-781. Springer, 2020.
Chung-ju Huang, Leye Wang, and Xiao Han. Vertical federated knowledge transfer via representation distillation for healthcare collaboration networks. In Proceedings of the ACM Web Conference 2023, pp. 4188-4199, 2023.
Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew P Lungren. Fusion of medical imaging and electronic health records using deep learning: systematic review and implementation guidelines. NPJ digital medicine, 3(1):136, 2020.
Szu-Chi Huang, Cheng-Fu Fu Cao, Po-Hsun Liao, Lung- Hao Lee, Po-Lei Lee, and Kuo-Kai Shyu. Enhancing chinese multi-label text classification performance with response-basely knowledge distillation. In Proceedings of the 34th Conference on Computational Linguistics and Speech Processing (ROCLING 2022), pp. 25-31, 2022a.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competition: What makes joint training of multi-modal network fail in deep (arning((provabai)) In International Conference on Machine Learning, pp. 9226 9259. PMLR, 2022b.
A. Johnson, T. Pollard, R. Mark, S. Berkowitz, and S. Horng. Mimic-cxr database (version 2.0.0), 2019a. URL ftctps://doi.org/// /ttp:////000.0//1000000////11110
13026/C2JTQQ
A. Johnson, L. Bulgarelli, T. Pollard, B. Gow, B. Moody, S. Horng, L. A. Celi, and R. Mark. Mimic-iy (version 3.1), 2024. URL http:ttsss///o.orrggge h
Alistair EW Johnson, Tom J Pollard, Seth Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, do-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019b.
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Benjamin Moody, Brian Gow, Li-wei H Lehman, el al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Lyes Khacef, Laurent Rodriguez, and Benoit Miramond. Written and spoken digits database for multimodal lcarning. 2019.
Xingjian Li, Haoyi Xiong, Zeyu Chen, Jun Huan, Cheng- Zhong Xu, and Dejing Dou. "in-network ensemble": Deep ensemble learning with diversified knowledge distillation. ACM Transactions on Intelligent Systems and Technology (TIST), 12(5):1-19, 2021.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Qian Lou, Yilin Shen, Hongxia Jin, and Lei Jiang. Safenet: A secure, accurate and fast neural network inference. Tn International Conference on Learning Representations, 2021.
Stephen G Pauker and Jerome P Kassirer. The threshold approach to clinical decision making. New England
Journal of Medicine, 302(20):1109-1113 1980.
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238-8247 2022.
Jesse M Pines, Ali S Raja, Fernanda Bellolio, and Christopher R Carpenter. Evidence-based emergency care: diagnostic testing and clinical decision rules. John Wiley & Sons, 2023.
Zhenzhen Quan, Qingshan Chen, Moyan Zhang, Weifeng Hu, Qiang Zhao, Jiangang Hou, Yujun Li, and Zhi Liu. Mawkdn: A multimodal fusion wavelet. knowledge distillation approach based cross-view attention for action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Linda Smith and Michael Gasser. The development of embodied coguition: Six lessons from babies. Artificial life, 11(1-2):13-25 29, 2005.
Ravi Soni, Jiahui Guan, Gopal Avinash, and V Ratna Saripalli. Hme: a hybrid reinforcement learning based model compression for healtheare applications. In 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE), pp. 146 151. IEEE, 2019.
Qi Wang, Liang Zhan, Paul Thompson, and Jiayu Zhou. Multimodal learning with incomplete modalities by knowledge distillation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1828- 1838, 2020a.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12695-12705, 2020b.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. Unified and effective ensemble knowledge distillation. arXiv preprint 20X0v:2204.00548, 2022a.
Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof Geras, Characterizing and overcoming the greody nature of learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp. 24043 24055. PMTR, 2022b.
Zihui Xue, Sucheng Ren, Zhengqi Gao, and Hang Zhao. Multimodal knowledge expansion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 854 863, 2021.
Litao Yang, Deval Mehta, Dwarikanath Mahapatra, and Zongyuan Ge. Leukocyte classification using multimodal architecture enhanced by knowledge distillation. In Medical Optical Imaging and Virtual Microscopy Image Analysis: First International Workshop, MOVI 2022, Held in Conjunction with MICCAT 2022, Singapore, September 18, 2022, Proceedings, pp. 63 72. Springer, 2022.
Zhengkun Zhang, Xiaojun Meng, Yasheng Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. Unins: A unified framework for multimodal summarization with knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11757 11764, 2022.
Zhuoran Zhao, Kamyar Mirzazad Barijough, and Andreas Gerstlauer. Deepthings: Distributed adaptive deep learning inference on resouree-consttaiieed iot edge clusters. IEEE Transactions Computer-Aided Design of Integrated Circuits and Systems, 37(11)22348-2888 -2329, 2018.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A Implementation details
A.1 Model size
Table A1 shows the sizes of the different model architectures. MedFuse (Hayat al. 2022 serves as the base
architecture for our work and is provided as a reference. While was originally proposed using ResNet-34 and 2-layer LSTM encoders, we adopt the same configuration for MIND (RosNct-10 and 2-layer LSTM) and all baselines for the sake of comparison. Notably, the MIND model is three limes smaller than the original architecture while providing unimodal classification capabilities.
Table A1: Size of multimodal models. We provide a summary of the number of trainable paramcters for each multimodal architecture. We use ResNet-10 and 2-layer LSTM encoders for the student network within the proposed MIND framework and for all baselines.
Model
No. of trainable parameters
ResNet-34 and 2-layer LSTM Hayal al. 2022 ResNet-10 and 2-layer LSTM (MIN)
23.9 M
7.5 M
A.2 Clinical dataset
In our experimental sctup, we use two publicly available datascts for two clinical prediction tasks: (i) multilabel clinical conditions prediction and (ii) in-hospital mortality prediction. For modality A, we use chest X-ray images (CXR) extracted from the MIMIC-CXR dataset (Johnson et al., 2019b). For modality B, we use electronic health record (EHR) time-series data extracted from the MIMIC-IV dataset (Johnson et al.) 2023). We link the modalities following Hayat et al. (2022). The dataset comprises all clinical time-series that have an associated chest X-ray image, ensuring that both modalities are present for each sample. We summarize the dataset in Table A2
Table A2: Summary of dataset. Size, shape and composition of the datasets used to train, validate, and evaluate the unimodal and multimodal models. The chest X-ray images, where a single image XAA E 2224x22423 are represented by dataset AA, and the clinical lime-series data, such that a sample x E Rtx76 where / is the number of time-steps based on patient's 's stay in the intensive care unit and 76 is the number of pre- pre-processed features per time-step, are represented by DB. For the multimodal datascts, we paired samples from DA and Dn, n, using the resulting datasets for our experiments.
Dataset
Training Validation
Test
Shape, composition
Unimodal
CXR (Da)
124671 42628
8813 4802
20747
XA
22
EHR (Dg)
11914
Xy E Rtx70
Multimodal
Clinical conditions
7756
882
2106
DAnDR
In-hospital mortality
4885
540
1373
DanDa
A.3 MIND architecture & implementation
We use MedFuse (Haya et al. 2022 as the multimodal bascline for our experimental sctup. The original implementation of MueeFuse employs an LSTM-based fusion module that processes sequence of modality representations provided by a ResNet-34 encoder for the chest X-ray images and a 2-layer LSTM encoder for the clinical time-serics data. In the MIND framework, we reduce the size of the image encoder to a ResNet-10 model to achieve model compression. In our experiments, we adhere the hyperparameters used in the original Fuse implementation for the randomly initialized multimodal fusion network.
17

---

Published in Transactions on Machine Learning Research (01/2025)
For the MIND model and all basclines, we perform random hyper-parametts tuning of the learning rates and weighting coefficients where applicable. We conduct minimum of 50 runs per model each task. For the unimodal baselines, we randomly select ten different learning rates and use the best-performing model based on the epoch with the highest AUROC on the validation set. All typer-parameters are summarized in Table
Table A3: Model prper-parameters. Description of the hyper-parameters used in our experimental sctup. We follow those suggested by Hayat et al. (2022) for the multinodal model with randomly initialized encoders and baselines. For the unimodal encoders, we evaluate different learning rates and select the best one based on the epoch with the highest AUROC on the validation set.
Hyper-paramett
Value/s
Batch size
16
Drop out
0.3
Epochs Early stopping
300
Patience = 40
Learning rate multimodal/basseeee /baselines
[1 10 ,..., 1 10 5]
Learning rate unimodal CXR
[1 x 10-4 ..... 1 x 10
Learning rate- unimodal EHR
[ x 10-4 ...., x 10-6
A.4 Multimodal Baselines
In alignment with our proposed response-based based knowledge distillation framework, MIND, we adapt the baselines for multilabel classification according to their corresponding papers, including MedFuse (Hayat ct al., 2022), MedFuse-3H, MKE (uue et al., 2021). UME (Du et al. 2023) and TS (Wang et al. 2020a). In our MIND framework and for all baselines, we save the best model checkpoint based on AUROC on the validation set during training and load the saved model to evaluate performance on the test set. Table A4 provides an overview of the loss functions proposed by cach bascline for multimodal training.
MedFuse (Hayat et al. 2022 serves the base architecture for all models and experiments in work. It is typical multimodal architecture proposed for medical tasks, featuring two modality encoders: ResNet-34 for CXR images and a 2-layer LSTM for lime-series data. The outputs from the encoders are concatenated and passed through an LSTM layer before the final multimodal classification head. To nnyudd
incorporate their methodology, we utilize the open-source code available at cas/MedFuse] In our experiments, we perform learning rate luning while adopting the default seltings for the other hyper-parameters.
McdFuso-3H. Like most multimodal networks, the original MedFuse architecture does not include classification head on the modality encoders, resulting in exclusively multimodal output. In our MIND framework, the first step is to modify the multimodal network architecture to leverage the encoders for samples with absent modalities, thus avoiding the need of masking or imputation. We extend the original MedFuse architecture to include three classification heads (3H), naming it MedFuse-3H, and use it as a baseline for our approach, In our experiments, We perform learning rate tuning while adopting the default. ModFuse scttings for the other prper-paramcters
TS WWagg et al. 2020al) proposes response-based knowledge distillation framework where teacher models, trained with large datasets that include samples with missing modalities, transfer knowledge via soft labels to multimodal student model. The multimodal student is trained using the soft labels from the unimodal teachers along with the supervision loss, as shown in Table A4 Originally proposed for multiclass classification, we adapt it for our multilabel classification framework. We perform random hyper- parameter tuning of the learning rate and the a, B distillation coefficients. Following Wang et al. (2020a), a, E {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7,0.8,00 0.9). We keep the other models and baselines, using the default MedFuse settings.
consistent across
18

---

Published in Transactions on Machine Learning Research (01/2025)
Multimodal Knowledge Expansion (MKE) (uee al. 2021 proposes response-based knowledge distillation framework composed of a unimodal teacher and a multimodal student. Originally designed to work with labeled and unlabeled data and evaluated on binary classification, multiclass classification, and segmentation tasks, we adapt it for our multilabel classification tasks framework. While it proposes the loss described Table A4 we train the multimodal student network using a method equivalent to the equation in the table, following the authors' instructions. Since proposes knowledge distillation (KD) from a single unimodal teacher to multimodal student, we split this baseline into two: MKE-EHR, where we use the best EHR. unimodal model the teacher, and MKE-CXR, where the best CXR unimodal model the teacher. We perform learning rate tuning in our experiments while adopting the default MedFuse settings for other syper-parameters
Uni-modal Ensemble (UME) Du et al. 2023) proposes the aggregation of predictions from independently trained unimodal models for multimodal samples on given task. The prediction for a multimodal instance is obtained by weighting the predictions of an ensemble unimodal models (one model per modality). Following Du et. al. (2023); we average the predictions from the unimodal models to provide the final prediction. Originally proposed for multiclass classification, we adapt it for multilabel classification framework. No further training is required using this framework, apart from the independent training of the unimodal models.
Using the notation introduced earlier, we provide a summary of the loss functions proposed by the MIND framework and all baseline models.
Table A4: Comparison of loss functions. Using the notation introduced earlier, we provide summary of the loss functions proposed by the MIND framework and all baseline models.
Model
Loss function
MedFuse (Hayal al.
2022
Csas
MedFuso-3F
LSAS + LSa + LsB
MKE-CXR (ue et al.
2021
LKDA + Lreg
MKE-EHH [ue et
2021
LKDE + 7x freg
UME (Du et al. 2023
Csa and Ls, (independently trained unimodal models)
TS WWang et 2020a
LSAN + x CKV% + CKDY
MIND (ours)
Lsan Ls. + Lsn + WA LEKDY we x LRKDg
A.5
Code for reproducibility
To ensure the reproducibility of our results, make code available at: https://gitnbb.com/mnudd
cai MIND
B Additional results on clinical tasks
B.1
Multimodal performance results the validation set
Table B1 presents the performance results of the MIND model and all baseline models on the multimodal validation set for both clinical tasks.
B.2 Label-wise AUPRC performance
compares the label-wise AUPRC performance of the MIND model with that of TS, the best bascline.
Figure B1
B.3
Unimodal performance results on the validation set
Table B2
presents the unimodal performance results for the clinical conditions prediction task on the validation provides the unimodal performance results for the in-hospital mortality prediction task on
set.
Table B3
19

---

Published in Transactions on Machine Learning Research (01/2025)
Table B1: Multimodal fusion performance results. Performance results on the multimodal validation set for the MIND model and all bascline models (ModFuse, McdFuse-3H, TS, MKE-CXR, MKE-EHR, and CME). The best results are highlighted in bold.
Clinical Conditions
In-hospital Mortality
Model
AUROC
AUPRC
AUROC
AUPRC
MedFuse
Hayat et al.
2022
0.757 (0.716, 0.795)
0.464 (0.397, 0.540)
0.833 (0.778, 0.883)
0.503 (0.395, 0.619)
MedFuse-311
0.760 (0.719, 0.798)
0.460 (0.393, 0.536)
0.835 (0.782, 0.884)
0.438 (0.348, 0.566)
WKKE-CXR
Xue al.
2021
0.718 (0.673, 0.761)
0.419 (0.360, 0.492)
0.727 (0.664, 0.791)
0.314 (0.239, 0.437)
MKE-EHI [uue et al.
2021
0.753 (0.713, 0.791)
0.456 (0.391, 0.531)
0.850 (0.807, 0.899)
0.544 (0.433, 0.672)
UME Du et al.
2023
0.775 (0.735, 0.813)
0.492 (0.425, 0.570)
0.850 (0.797, 0.900)
0.543 (0.423, 0.679)
TS [Wang al.
2020a
0.778 (0.738, 0.815)
0.490 (0.423, 0.568)
0.846 (0.795, 0.890)
0.563 (0.449, 0.667)
MIND (Ours)
0.790 (0.752, 0.826)
0.516 (0.446, 0.593)
0.866 (0.817, 0.908)
0.572 (0. 155, 0.692)
Acute (MIND)
Mixed (MIND)
Chronic (MIND Chronic
(TS)
Mixed
1.0
Aoura candenta suckler 360300
0.8
Comil surpral prousdarostt souddss
hnd electriye dirctenn
Other lower raco ralory
Noummne nocust caleed la butubbo yocal stidlogg
AUPRE
Shodt
COunlor evatiuteiisss
0.4
D- Didetto med Auo with comp fottoons
kidney
0.2
Denssor an cerroceeees and hot dieees
Eoeent al pepertnn on
0.0
5
Clinical conditions
Figure Bl: Label-wise AUPRC Performance. Comparison of AUPRC performance between the MIND model and the best baseline (TS) for each clinical condition label and group (acute, mixed, and chronic).
the validation set. Note that unimodal models are trained on much larger datasets compared to the fusion models, which are trained on the multimodal set.
Table B2: Unimodal performance results for the clinical conditions task on the validation set. Evaluation of unimodal and multinodal models per modality on the multimodal validation set, including the number of training samples per model.
CHEST X-RAY IMAGES AUROC
CLINICAL TIME SERIES AUROC
MODEL
TRAINING SET
AUPRC
AUPRC
Multimodal
MedFuse-3H
7,728
0.679 (0.632, 0.724)
0.371 (0.316, 0.439)
0.713 (0.669, 0.754)
0.401 (0.343, 0.472)
MIND (Ours)
7,728
0.710 (0.665, 0.754)
0.412 (0.353, 0.484)
0.754 (0.714, 0.792)
0.459 (0.394, 0.534)
Unimodal
ResNet-34
124,671
0.714 (0.700, 0.727)
0.442 (0.422, 0.464)
ResNet-10
124,671
0.719 (0.705, 0.732)
0.450 (0.430, 0.472)
2-layer LSTM
42,628
0.762 (0.743, 0.780)
0.420 (0.388, 0.456)
4-layer LSTM
42,628
0.759 (0.740, 0.777) 0.418 (0.386, 0.454)
20

---

Published in Transactions on Machine Learning Research (01/2025)
Table B3: Unimodal performance results for the in-hospital mortality task on the validation set. Evaluation of unimodal and multimodal models per modality the multimodal validation set, including the number of training samples per model.
CHEST X-RAY IMAGES AUPRO
CLINICAL TIME SERIES AUROC
MODEL
TRAINING SET
AUROC
AUPRC
Multimodal
MedFuse-3H
4,885
0.691 (0.629, 0.749)
0.257 (0.189, 0.356)
0.828 (0.775, 0.876)
0.445 (0.338, 0.580)
MIND (Ours)
4,885
0.712 (0.650, 0.775)
0.277 (0.208, 0.375)
0.858 (0.811, 0.898)
0.548 (0.432, 0.678)
Unimodal
ResNet-33
124,671
0.732 (0.715, 0.749)
0.183 (0.167, 0.205)
ResNet-10
124,671
0.725 (0.709, 0.742)
0.177 (0.160, 0.196)
2-layer LSTM
42,628
0.870 (0.846, 0.891)
0.528 (0.467, 0.595)
4-layer LSTM
42, 42,628
0.870 (0.846, 0.892) 0.536 (0.473, 0.601)
B.4 Ablation studies
Table B4 presents scnsitivity analysis (Ablation study I, Section 5.3 for the in-hospital mortality task the lest set, focusing on different loss components. Each setting indicates the components used in the modified loss for training and the type of knowledge distillation performed. Similarly, Table [55 provides the results on the validation set for Ablation Study 1 related to the prediction of clinical conditions, while Table B6 presents results for the in-hospital mortality prediction task. Table B7 specifies the ayper-paramcter values used in Ablation Study IT (Section 5.4), which is depicted in Figure
Table B4: Ablation study on MIND for multimodal and unimodal performance in the in-hospital mortality prediction task. Fusion and unimodal performance of MIND on the multimodal test set, focusing on different loss components.
Cras faai
MAre
AUROC
Fusion
AUPRC
AUROC
Cloot X-Ray
AUPRC
AUROC
Time
AUPRC
0.816 (0.785, 0.811)
0.468 (0:308, 0.546)
2
0.820 (0.787. 0.551)
0.461 (0:392. 0.541)
0.069 (0.025. 0.713)
0.278 (0.252, 0.359)
0.813 (0.783, 0.843)
0.642 (0.572, 0.523)
(0.792,
(0.389,
(0.219.
(0.803, 0,861)
0.382 (0:412, 0.563)
0.661
(0.613. 0.705)
0.277
(0.2332. 0.845)
11827 (0.754. 0.807)
0.478 (0.002) 0.656)
5
0.835 (0.804, 0.864)
0.491 (0.421. (.574)
0,882 (0.641. 0.721)
(1.25)
(0.237. 0.359)
(1827 (0.797. 0.856)
0.484 (0.410, 0.500)
6
0.844 (0.115, 0.872)
0.505 (0:453, 0.587)
0,889 (0.632 0.726)
0.290 (0.253, 0.354)
0.830 (0.801, 0.859)
0.302 (0.426, 0,577)
Table B5: Ablation study on MIND for multimodal and unimodal performance in the clinical conditions task. Fusion and unimodal performance of MIND on the multimodal validation set, focusing on different loss components.
Dess Csxi
Eao%
Canoy
Mare
AUROC
AUPRC
AUROC
X-Ray
AUPRC
AUROC
Time
AUPRC
1
0.755 (0.716, 0.704)
0.454 (0.250, 0.529)
2
0.767 (0.715, 0.711)
0.488 (0.399, (.546)
0.077 (0,629. 0.723)
11:770 [0,366 0
1.718 (0.1744 0.700)
0.417 (IL356, 0.485)
(0.794,
(0.416, 0.564)
0.655 (0.648.
013911 (0,033, 0.461]
0.730 (0,898, 0.775)
0.435 (0.371,
(0.738,
(0:421. 0.367)
(0.032, 0.744)
0.397
(0.2377 0.885)
(0. (0.700) 0.781)
0.637 (0.375,
0.784 (0.745, 0.820)
0.502 (0. 132, 0.580)
0.200 (0 05. 0.745)
0.000
(0.341, 0.4701
0.747 (0. 707 0.786)
0.645 (0.550, 0.520)
6
0.780 (0.752, 0.526)
0.516 (0-196, 0.305)
0.710
(0.065. 0.754)
0.412
(0.353. 0.481)
0.754
(0.7144 0.793)
0.459 (0.394, 0.551)
C Additional results on multimodal multiclass benchmark datasets
C.1 Datasets
CREMA-D (Cao al. 2014 an audio-visual dataset designed for speech emotion recognition. It includes
7,442 video clips from 91 actors, each speaking selection of 12 sentences. The utterances express, with
21

---

Published in Transactions on Machine Learning Research (01/2025)
Table B6: Ablation study on MIND for multimodal and unimodal performance in the in- in-hospital mortality task. Fusion and unimodal performance of MIND on the multimodal validation set, focusing different loss components.
Cea
faxce
Expe
LENDE
MAYe
AUROC
Faston
AUPRC
AUROC
Chest X-Ray
AUPRC
AUROC
Time Series
AUPRC
0.833 (6. 0.883)
0.503 (0.355, (.619)
2
0.835 (0.781. 0.885)
0.478 (0.388, 0.006)
0,821 (I.829, 0.749)
0.257 (0.189. 0.356)
0.828 0.775, 0.876)
0.445 (0,338, 0.690)
3
0,839 (0.754, 0.887)
0.467 (0,353, 0.606)
0.670 (0.209, 0.735)
0.248 10.1577 0.343)
0.802 (0.779, 0.581)
0.457 (0.355, 0.602)
0.842 (0,700) 0.803)
0.488 (0.381, 0.615)
0.678 (0.005 0.745)
0.293 10.201, 0.305)
0.531 (0/786, (.0))
0.475 (0.372, 0.000)
0,808 v20,800, 0,901)
0.545 (0.125. (.674)
0.028 (0.636, 0.79)
0.276 (0.2115 0.185)
01850
0.892)
(0.42). 0.663)
0,896 (0.817, 0.908)
0.572 (0.453, 0.502)
0.712 (0.830, 0.77%)
0.280 (0:211. 0.375)
(0.811, 0.808)
0,348 (0/432) 0.078)
Table B7: Sensitivity analysis of the weighting parameters WA and WB. Values of the seper-parametess used for each sctting in Ablation Study II (Section 5.4 where WA wear and wp Wehr- The corresponding learning curves are depicted in Figure 3
Setting
Werr
Welr
1. Werr Wehr 0
0
Werr & Wetrr
500
3. Werr < Wehr
10
100
Warr Weher
10
10
5. Wear > Wetor
100
10
6. Werr " Wehr
500
1
varying degrees of intensity, one of six common emotions: anger, happiness, disgust, fear, neutral, and sadness. The training set comprises 5,210 samples, while both the validation and test sets contain 1,116 samples each.
S-MNIST (Khacef et al. 2019 is an audio-visual dataset designed for multimodal fusion classification. It was created by pairing the original MNIST handwritten digits database with a spoken digits database extracted from Google Speech Commands. To construct the training, validation, and test sets, we randomly sampled 8,000 instances for the training set and 2,000 instances for the validation set from the original training dataset (10 classes, grayscale digits 0-9). Additionally, we randomly sampled 2,000 instances from the original test set to create our test set.
LUMA [Bezirganyan et al. 2024) is multimodal dataset designed for benchmarking multimodal learning. The image modality includes images from a 50-class subset of the CIFAR-10 10 and CIFAR-100 datasets, while the audio modality contains utterances of the class labels. The dataset is imbalanced, with the most prevalent 22 classes having 1,500 paired samples or more. To construct our dataset, we generate an evenly distributed sample, containing 1,500 paired instances per class. The resulting dataset, which includes 33,000 paired instances, is randomly split into training, validation, and test sets, with 25,000, 4,000, and 4,000 andio-video samples, respectively.
C.2 Architectures and Implementation
We utilize multimodal architecture composed of ResNet variants as modality encoders for both modalities. The outputs of the modality encoders are concatenated and passed through a linear layer that computes the final prediction. Specifically, the details of the vanilla multimodal architectures and their sizes are provided as follows:
For the S-MNIST datasel, we use ResNet-10 models, trained on the entire training set,, as unimodal teachers and ResNet-3 models as modality encoders for the compressed model, which is trained with fraction of the original dataset (8,000 randomly selected samples). The teacher models have approximately 4.9 million parameters, while the whole MIND model has only approximately 151,000 parameters (32.5x size reduction).
22

---

Published in Transactions on Machine Learning Research (01/2025)
For the CREMA-D dataset, use ResNet-18 models unimodal teachers and ResNet-6 models modality encoders for the compressed model. All models are trained on the full training dataset. Thee teacher models have approximately 11 million parameters, while the whole MIND model has only approximately 612,000 parameters (18x size roduction).
For the LUMA datasct, use ResNet-10 models unimodal teachers and ResNet-3 models modality encoders for the compressed model. All models are trained on the entire training dalaset. The teacher models have approsimately 4.9 million parameters, while the whole MIND model has only approximately 151,000 parameters (32.5x size reduction).
For the MIND model and all basclines, we perform random weighting coelficients where applicable. We conduct minimum of 50 per model for each task. For the unimodal baselines, randomly select ten different learning rates and use the best-performing model based the epoch with the highest classification accuracy on the validation set. In general, we use batch size of 64 and train the models for 50 to 100 epochs. The learning rates used are within the ranges described in Table A3
tuning of the learning rates and
23


Reviews:
[{"review_id": "BkslFTH4hX", "reviewer": "Reviewer_djCo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths** \n- This paper is well-organized and easy to read. \n- The proposed method is technically sound. \n- Extensive experimental results are provided to show the effectiveness of the proposed method. \n- Detailed visualizations and ablation results are also included. \n\n&nbsp;\n\n### **Weaknesses** \n- This paper mainly provides the experimental results on clinical prediction tasks. However, the proposed method can also be applied to conventional image classification or segmentation tasks. It would be interesting if the authors could also include the results on the conventional image tasks, e.g., on ImageNet or MS COCO.\n- There are some typos and style inconsistencies in the paper. For example, \"modality-specific\" appears inconsistently, sometimes with a hyphen and sometimes without (e.g., \"modality specific\"). Phrases such as \"modality utilization during multimodal training training\" appear, with the word \"training\" repeated unnecessarily (in the caption of Figure 1).\n- Some figures are very difficult to read. For example, there are too many lines in Figure 3. It is very difficult to compare different settings. It would be better to smooth the lines and use other strategies to make them easier to understand. \n- The technical contributions seem incremental. Seeing from Figure 1, it looks like the proposed method is very similar to Hayat et al. (2022). The only difference between A2 and the proposed method is using weighting hyper-parameters. Even though the novelty is not a mandatory requirement according to the TMLR reviewing guidelines, the authors still need to adjust their claims if the proposed method is very similar to Hayat et al. (2022).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a multimodal model compression framework based on knowledge distillation that transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into smaller multimodal students. They evaluate the proposed method on binary classification and multilabel clinical prediction tasks using clinical time series data and chest X-ray images extracted from publicly available datasets.  The experimental results demonstrate that their method improves the performance of the smaller multimodal network across all five tasks, as well as fusion methods and multimodal network architectures, with respect to several state-of-the-art baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors are requested to make changes to address the weaknesses I mentioned.", "broader_impact_concerns": "The authors claim their training data comes from public datasets. I think they might still need to explain that there are no privacy concerns when using those data. For example, they could include evidence of no privacy issues from the related dataset papers.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HowX9QHldc", "reviewer": "Reviewer_sy5L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nSoundness: This paper has a clear motivation for learning multimodal clinical representations from small datasets with small models. Existing large models may overfit and rely on one modality, leading to suboptimal results. The method is presented clearly, with the standard knowledge distillation loss, plus unimodal classification losses for individual modalities, and finally, it distills knowledge from an ensemble of unimodal teachers. Balancing weights are introduced for the KD terms. Standard metrics such as AUROC and AUPRC on MIMIC-CXR, plus ablation studies on each component and results on other multimodal datasets are provided. The code is provided and seems cleanly written.\n\nAudience: This paper provides a small network beneficial to clinical datasets and is useful for the medical ML community. The analysis of different KD loss components and the code can help the community.\n\n**Weaknesses**:\nSee requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Modality-INformed knowledge distillation (MIND), a (small) multimodal knowledge distillation model for small medical datasets. MIND distills modality-specific knowledge from ensembles of unimodal teacher models to a smaller multimodal student model. It uses an early fusion framework but leverages two additional modality-specific losses to enable independent unimodal modules for missing modalities. In addition, it has weighting parameters to balance the unimodal distillation components. On the clinical dataset MIMIC-CXR and non-clinical datasets CREMA-D, S-MNIST, and LUMA, the model achieves SOTA results on most metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Related work**: Could the authors discuss [1], which is very similar to this work, where the knowledge is transferred by learning the teacher’s behavior within each modality, and weighting schemes are also introduced. The following similar related work seemingly lessens the novelty of this work.\n\n[1] Jin, Woojeong, et al. \"Msd: Saliency-aware knowledge distillation for multimodal understanding.\" arXiv preprint arXiv:2101.01881 (2021).\n\n**Claim on missing modality**: Could the authors clarify the first paragraph in 5.1: “while enabling unimodal predictions when modalities are missing”. Experiments in Table 1 show that unimodal distillation helps but did Table 1 explicitly test the missing modality scenario? In general for the paper, the reviewer apologizes in advance if she or he missed it, but is there an explicit missing modality experiment (e.g., explicitly corrupt, mask out, or simply do not feed modality input into the model)? This is one of the claimed benefits in the paper, but it is unclear which tables show this (handling missing modality input better at inference time).", "broader_impact_concerns": "A broader impact statement on societal issues, such as the potential risks of malicious usage of medical models, as well as implicit biases learned from knowledge distillation, should be added.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ymnAhncuez", "reviewer": "Reviewer_zzZQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pros**:\n+ The proposed weighted unimodal ensemble knowledge distillation loss may interest some TMLR audiences. This loss can be used to compress models for multimodal tasks.\n+ The ablation study can generally verify the effectiveness of the key components in the proposed method.\n+ The overall performance is comparable to the other SOTA methods on various tasks/datasets.\n\n**Cons**:\n\nOverall, this paper is hard to follow, with some statements not well clarified or experimentally supported.\n\n1. Why do unimodal teacher networks allow the student to learn diverse representations? How about if the teacher model is not an unimodal network? And what is the unimodal model? Does it mean modal-specific models?\n\n2. Though the proposed method is claimed to improve the compression of multimodal networks, it is unclear whether the performance of the compressed networks owes to the extra large-scale datasets, compared with the other SOTA methods (see Q8). Regarding efficiency, the proposed method needs to train multiple teacher networks for each modality during the training stage, so it can be limited to only a few modalities and can be computationally expensive when there are many different modalities, e.g., MRI, PET, and Ultrasound in addition to X-ray.\n\n3. The proposed method looks like a two-step method, with the first step training various teacher models and the second step training a student model using the ensemble of teachers. A two-step method can need more training time and multiple teachers pre-trained on large-scale datasets can cost more computational resources. Due to the cumbersome pre-training of the teachers in the first step, one may consider using some other methods, e.g., MedFuse, for the first step training to save time/resources. That is, what will happen if other SOTA methods are used as the teacher model?\n\n4. On page 4, why are the randomly initialized encoders more comparable in the proposed setup (i.e., joint fusion, I suppose)? Particularly, why is the initialization scheme related to the setup?\n\n5. Though Sec. 5.4 analyzes balancing multimodal learning with \\omiga_A and \\omiga_A, it does not clearly show how to set \\omiga_A and \\omiga_A in Eq. (7)? What are the exact values of these two hyper-parameters during the training phase?\n\n6. The number of teachers for each modality is 3, but why 3 teachers are chosen? Is 3 the best?\n\n7. How is Eq. (6) used during training (except for plotting Fig. 3)? That is, when Fig. 3 is plotted, how to adjust the training hyperparameters/settings according to the Figure?\n\n8. Whether the CXR dataset and the EHR dataset used for training unimodal models are the extra data in addition to the datasets used for clinical conditions and in-hospital tasks? Whether the other competitors like MedFuse use the same amount of data as the CXR and EHR datasets or not?\n\n9. In Sec. 6, “Compared to existing work, only Bucilua et al. (2006) combine ensemble learning and knowledge distillation to train compact networks.” This statement seems to be incorrect. For instance, [A] also combines ensemble learning and knowledge distillation to train compact networks. Please conduct a comprehensive literature review to better highlight the difference between the proposed method and the others.\n\n[A] Fukuda, Takashi, et al. \"Efficient Knowledge Distillation from an Ensemble of Teachers.\" Interspeech. 2017.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a Modality-INformed knowledge Distillation (MIND) framework to leverage knowledge distillation to transfer knowledge from ensembles of multiple pre-trained deep neural networks of varying sizes into a smaller multimodal student. Experimental results show that the proposed method can achieve performance comparable to that of other SOTA methods on various datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the questions in the Cons and revise the manuscript accordingly.", "broader_impact_concerns": "There are no concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a knowledge distillation framework for multimodal clinical prediction tasks. Specifically, a teacher-student framework was proposed, where knowledge was transferred from ensembles of pre-trained deep neural networks of varying sizes into smaller student models. Experimental results on a few datasets show the effectiveness of the proposed method.

Overall, this paper presented some interesting findings, which were appreciated by the reviewers, receiving all positive recommendations (2 Leaning Accept, and 1 Accept). However, there are concerns about the novelty and significance of the proposed method. The technical contributions were found to be incremental (details please see the review comments), when compared to a few related works. But considering the motivation, technical soundness, the validated claims, and the findings this paper has presented, it is considered to be of interest to the audience of TMLR. Therefore, the AE recommends Accept.
Decision: Accept (poster)