ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities

Authors: julie  mordacq
leo  milecki
maria  vakalopoulou
steve  oudot
vicky  kalogeiton
l  mordacq
m  milecki
s  vakalopoulou
v  oudot
  kalogeiton
Venue: MIDL_io_2024_Conference
Published: 2024-01-31 16:56:55

Abstract: Multimodality has recently gained attention in the medical domain, where imaging or video modalities may be integrated with biomedical signals or health records. Yet, two challenges remain: balancing the contributions of modalities, especially in cases with a limited amount of data available, and tackling missing modalities. To address both issues, in this paper, we introduce the AnchoreD multimodAl Physiological Transformer (ADAPT), a multimodal, scalable framework with two key components: (i) aligning all modalities in the space of the strongest, richest modality (called anchor) to learn a joint embedding space, 
and (ii) a Masked Multimodal Transformer, leveraging both inter- and intra-modality correlations while handling missing modalities. We focus on detecting physiological changes in two real-life scenarios: stress in individuals induced by specific triggers and fighter pilots' loss of consciousness induced by g-forces. We validate the generalizability of ADAPT through extensive experiments on two datasets for these tasks, where we set the new state of the art while demonstrating its robustness across various modality scenarios and its high potential for real-life applications. Our code is available at https://github.com/jumdc/ADAPT.git.

Main paper: 
 introduction: 
 Monitoring physiological changes to external stimuli is crucial for assessing individuals' well-being, particularly in contexts with medical and safety implications. Examples include stress, a response to emotional, mental, and physical challenges (Schneiderman et al., 2005) , and a triggering or aggravating factor for various pathological conditions (Dimsdale, 2008) . High-performance environments, such as exposure to g-forces in aircraft, can lead to alterations in consciousness (Morrissette and McGowan, 2000) . At the same time, drowsiness during driving poses a critical physiological response with safety implications, contributing to road accidents and fatalities (Stewart, 2023) . Various sensors report physiological changes that may be detected visually (videos), acoustically (audio), or from biomedical signals (e.g., electrocardiograms). However, specific modalities may be missing during training and testing. Therefore, developing methods capable of handling missing modalities during both stages while balancing modalities' contributions is crucial to ensure robustness, notably when modalities with strong unimodal performances are severely missing.Various methods address the challenge of missing modalities, each with notable limitations, including (1) bias towards the most available modalities leading to sub-optimal performance (Konwer et al., 2023) , (2) dependence on complete modalities during training (Mallya and Hamarneh, 2022; Chen et al., 2021a) , (3) limited generalizability to more than two modalities (Ma et al., 2021 (Ma et al., , 2022)) , and (4) utilization of a shared encoder tailored for modalities with inputs of the same dimensions which complicates extension to heterogeneous modalities like imaging and biomedical signals (Konwer et al., 2023) .To address the above issues, we introduce the AnchoreD multimodAl Physiological Transformer (ADAPT) that is designed to operate effectively under missing modalities both during training and inference enabling robust real-life applicability. ADAPT consists of two key components. First, our goal is to embed all modalities in the same feature space. Instead of optimizing one loss per modality pair, which would result in quadratic growth of training time, we align each modality to one frozen modality, called anchor. It allows learning a joint embedding space with linear scalability and balancing each modality's contribution. We call this step the 'anchoring'. Second, it comprises a Masked Multimodal Transformer that leverages inter-and intra-modality correlations to concatenate features from different modalities into a unified representation. Additionally, we leverage masked attention from the transformers (Vaswani et al., 2017) to ensure flexibility in handling missing modalities similarly to Ma et al. (2022) ; Milecki et al. (2022) . When a modality is unavailable, its corresponding feature representation is masked. The transformer is trained using two objectives: self-supervised learning and the objective of the downstream task.ADAPT is applied to the challenging task of detecting physiological changes using multimodal medical data with missing modalities during training and inference. Specifically, we focus on detecting alterations in pilots' consciousness induced by g-forces in fighter jets and stress triggered in individuals by specific stimuli (Chaptoukaev et al., 2023) . We show that ADAPT outperforms the previous state of the art on both tasks and datasets while handling missing modalities. Extensive experiments demonstrate its robustness against missing modalities across various scenarios, highlighting its effectiveness for real-life applications.Our contributions are: (i) ADAPT, a modular framework that aligns multimodal representations to a common rich feature space; (ii) a modality-fusion strategy to handle missing modalities both at training and inference time; (iii) we set the new state of the art on two tasks and datasets and provide extensive evaluations highlighting ADAPT's superiority. 
related work: 
 Handling missing modalities. Missing modalities pose a persistent challenge in Multimodal Learning, particularly in medical imaging, due to privacy concerns or impractical data acquisition (Liang et al., 2021; Azad et al., 2022) . Various strategies have been explored to address this challenge. Knowledge Distillation (Hu et al., 2020; Mallya and Hamarneh, 2022; Wang et al., 2023b) involves learning from a teacher network trained on complete modality data. Generative modeling aims to impute missing inputs by generating synthetic data (Yoon et al., 2018; Sharma and Hamarneh, 2019) . Both approaches rely on complete modality at training, which can be insufficient for robust training. Another line of work is common space modeling, which learns a shared latent space from partially available modalities (Ma et al., 2021; Konwer et al., 2023; Wang et al., 2023a) . SMIL (Ma et al., 2021) perturbs the latent feature space to approximate the embedding of missing modalities but is limited to bi-modal datasets, limiting its generalizability. ShaSpec (Wang et al., 2023a) addresses more than two modalities by learning shared and specific features, but its use of a shared encoder complicates generalization to heterogeneous modalities of different dimensions. Additionally, shared latent space modeling may introduce biases toward the most available modalities (Konwer et al., 2023) . Simultaneously, cross-modal contrastive learning has shown impressive results (Zhang et al., 2022; Milecki et al., 2023; Li et al., 2023) by aligning multimodal data in a joint embedding space. Recently, ImageBind (Girdhar et al., 2023) aligned six modalities by relying on image-paired data and emphasized that aligning all pair combinations is unnecessary to bind more than two modalities together. Our proposed ADAPT advances this by training unimodal encoders solely with supervision from one modality, aligning them in a joint embedding space. It ensures that every modality contributes to the final representation, even if it is severely missing during training.Multimodal transformer. Transformers (Vaswani et al., 2017) have become the de facto approach for multimodal tasks (Recasens et al., 2023; Srivastava and Sharma, 2024) . They rely on the attention mechanism to model long-range dependencies with the flexibility to account for incomplete samples. Milecki et al. (2022) ; Ma et al. (2022) efficiently handle missing data in sequences and bimodal datasets through masked attention. ADAPT extends this to more than two modalities by leveraging attention to fuse them and exploring their inter-and intra-modal correlations while masking missing ones. We also perform a systematic study of missing modalities during training and testing, showing the versatility and potential of ADAPT for real-life scenarios. 
method: 
 This study addresses the detection of physiological changes using multimodal data, including video, audio, and biomedical signals. Real-world scenarios often involve missing modalities, motivating our goal to develop a modality-agnostic representation with broad applicability and to propose ADAPT -AnchoreD multimodAl Physiological Transformer. An overview of ADAPT is presented in Figure 1 .Notations. Let D = {(x i m ) M m=1 , y i } N i=1denote our training dataset, with M modalities and N labeled observations and x i = (x i m ) M m=1 the i-th observation (i.e., a family of m modality values) with y i ∈ Y = {0, .., J} its corresponding label (i.e., a physiological state). Given this input, we seek to train a neural network F, that associates to any observation, with any missing modality, a target label y ∈ Y.We train modality-specific encoders with a contrastive learning objective to align their representations to the one of the anchor. In this work, anchor is the video, as it can capture visually distinguishable physiological changes; however, any modality can be the anchor. In each minibatch, ADAPT takes up to M modalities, including video, audio, and biosignals, as input to produce a modality-agnostic representation for downstream tasks. It is trained in two steps. (i) Anchoring. We align the representations of all modalities via contrastive learning to the one of an anchor modality, i.e., the strongest and richest modality; here the video. (ii) Fusion. The encoders' features are concatenated and fed into the Masked Multimodal Transformer. When a modality is unavailable, the transformer masks its corresponding feature representations. The final representation (i.e., [CLS] token output) is used for downstream tasks.Modality-specific encoders. Each modality is encoded using a dedicated encoder. For video, we use the pre-trained Hiera (Ryali et al., 2023) encoder. For audio, each sample is encoded into a mel-spectrogram (a 2D acoustic time-frequency representation of sound), fed to BYOL-A (Niizumi et al., 2021) to obtain a 1-d feature. Biomedical signals are processed using 1D CNNs (Wang et al., 2023c; Ismail Fawaz et al., 2019) . We add a modality-specific linear projection head to each encoder to obtain a fixed size d dimensional embedding. ADAPT can be extended to other modalities by adding their respective encoders.Anchoring. We consider a pair of modalities with aligned observations (A, M m ), where A represents the anchor (video) and M m another modality. The anchor video x i a and its corresponding observation x i m are encoded using z i a =E a (x i a ) and z i m =E m (x i m ), respectively, where E a is a pre-trained and frozen video encoder and E m a DNN. Projection heads map the embeddings to f i a , f i m ∈R d . The loss is computed on f i a and f i m (Girdhar et al., 2023) :EQUATION)where τ is a temperature parameter τ ∈ R + , cos(., .) the cosine similarity, and B the batch size. In practice, we use a symmetric loss: L A,Mm + L Mm,A . To alleviate the modality gap (Liang et al., 2022) , we add Gaussian noise to the modality m representation (Gu et al., 2023) . We use a cosine schedule for the temperature parameter (Kukleva et al., 2023) . Given M modalities, we define the anchoring loss as L anchoring = M m=1,Mm̸ =A (L A,Mm + L Mm,A ).To effectively build modality-agnostic representations, we use the transformer (Vaswani et al., 2017) with N L attention blocks. For each sample, we stack the modality-specific representations,f i m ∈ R d , ∀m ∈ [1, M ], into a single matrix and prepend a special token [CLS], yielding a matrix F ∈ R (M +1)×d . Similarly to Liu et al. (2022) ; Nagrani et al. (2021) , the query, key and value are derived from F via:Q = W Q F , K = W K F and V = W V F where Q, K ∈ R (M +1)×d k and V ∈ R (M +1)×dv . Our modelization of inter-modal interactions differs from the usual cross-attention (Chen et al., 2021b; Jaegle et al., 2021) , which asymmetrically combines two separate embedding sequences of the same dimension. Using stacked features F allows generalization to any number of modalities, with linear scalability in the number of modalities instead of quadratic.Handling missing modalities. Inspired by Milecki et al. (2022) for missing follow-up patient examinations, we apply our strategy to deal with missing modalities to the scaled dot-product, core of each multi-head self-attention sub-layer. We consider one sub-layer with one head (h = 1) for simplicity. We use a masking binary matrix Z ∈ R (M +1)×(M +1) that specifies which modalities are missing: z ij = 1 if i and j are available, else z ij = 0. The output O ∈ R (M +1)×dv of the attention mechanism is, for O i each line of O:EQUATIONWhen h > 1, queries, keys, and values are linearly projected h times with different, learned linear projections, concatenated, and once again projected after the scaled-dot product.Modality dropout. We train the Masked Multimodal Transformer with a multi-view contrastive objective (Chen et al., 2020) . Drawing inspiration from Shi et al. (2022) , we mitigate the model's over-reliance on a single modality while enhancing its robustness in the absence of modalities through an augmentation technique called modality dropout. We leverage the masking scheme at the attention level to randomly mask input modalities. Given a batch Z, we create two simultaneous view Z ′ and Z ′′ . For each observation within Z ′ , we hide up to M -1 modalities following a uniform probability, M the number of modalities. Additionally, motivated by Han et al. (2020) , who recently showed the effect of additive noise on electrocardiograms, we add ϵ ∼ N (0, σ) on the biomedical signals to each view independently. We chose σ based on the amplitude of the signal. We use the infoNCE loss to enforce the similarity between the two views (Chen et al., 2020) on the final representation output given by the [CLS] token. Since the two representations are already mapped to the same dimension, following Jing et al. (2022) , we directly optimize the representations to enforce scalability and mitigate dimension collapse:L fusion = -B i=1 log exp(cos(CLS i ,CLS i ′ )/τ ) B k=1 exp(cos(CLS i ,CLS k ′ )/τ ). 
experiment setting: 
 Datasets. StressID (Chaptoukaev et al., 2023) Metrics. We use the balanced accuracy (ACC) and weighted F1 score (F1). For LOC, to ensure robustness to class imbalance (Huang et al., 2016; Luque et al., 2019) , we also report the true positive rate (TPR) as it ensures not missing out on pilots fainting; and the true negative rate (TNR) for completeness. We report metrics in the format mean(std) in %.Implementation details. The Anchoring and Masked Multimodal Transformer are trained on X * train and X train , respectively. A linear classifier is trained using the [CLS] token for the final task. We train for 70 epochs using AdamW optimizer, a starting learning rate of 1e -4 , followed by a cosine schedule and a linear warm-up of 4 epochs. Given their size difference, we set the batch size to 256 for LOC and 128 for StressID. To tackle LOC class imbalance, we use the Balanced Cross Entropy loss (Huang et al., 2016) (more in Appendix B.1).Comparison to the state of the art (Table 1 ) in the presence of all modalities. We compare ADAPT against unimodal baselines for video, audio, and biomedical signals concatenated (rows 1, 2 & 3), 'feature fusion' and 'decision-level fusion' (rows 4 & 5) (Chaptoukaev et al., 2023) , ShaSpec+ (Wang et al., 2023a ) (row 6) (more in Appendices B.2 and C).For StressID, we observe that ADAPT outperforms all methods from Chaptoukaev et al. ( 2023) by a notable margin; for instance, it outperforms 'decision-level fusion' by 4% in ACC and 6% in F1. Additionally, it remains highly competitive with ShaSpec+.For LOC, using only video (row 1) results in the best performance; however, video is unavailable in real-life scenarios. Instead, ADAPT handles missing modalities by leveraging representations from all modalities during training. Surprisingly, 'feature fusion', 'decisionlevel fusion', and ShaSpec+ lead to unbalanced metrics, i.e., they result in a high TNR while significantly sacrificing TPR (respectively 29.5%, 20.4% and 7.3%), showing they predict most samples as negative. This reveals their unsuitability for real-life cases with highly imbalanced classes where both TPR and TNR matter. Note that in our target scenarios, TPR is more important, as it is critical to detect pilots losing consciousness. By contrast, ADAPT results in a TPR of 69.5% (+40% vs. fusion methods) while maintaining a balanced TNR of 65.3%. This is further shown in Figure 2 , where ADAPT (blue crosses) strikes the Feature Fusion (Chaptoukaev et al., 2023) 79.3(2.5) 63.9(9.6) 97.0(0.9) 29.5( 20) 61(3) * 66(4) * Decision-level fusion (Chaptoukaev et al., 2023) 60.2(2.2) 99.0(0.0) 100.0(0.0) 20.5(4.5) 65(5) * 72(5) * ShaSpec+ (Wang et al., 2023a) 53.4(5.1) 97. best balance between TPR and TNR vs. other methods (red crosses). This testifies to ADAPT not being misled by the high class imbalance.Robustness to missing modalities (Table 2 ). We first report baseline results (row 1) on the default test set X test , i.e., no modality removed in StressID and 90% of videos missing for LOC. Then, we completely remove one or two modalities from X test and compare the results (∆) to the ones obtained on X test . † Methods from (Chaptoukaev et al., 2023) For LOC, ADAPT shows robustness in all scenarios, with a |∆|<8% and average |∆|=2.6 compared to the baseline. This is further shown in Figure 2 , where the balance between TNR and TPR (blue circles) remains consistent across all scenarios. Interestingly, for novideo, even though video-only provides strong unimodal performance (Table 6 ), ADAPT maintains high performances, indicating its capability of aligning representations in the video (anchor) space. Furthermore, for the real-life scenario where we remove both video and visual field (row 2), the results remain competitive with an average |∆|=2.72%, even though these modalities individually perform the best (Table 6 ). Additionally, no-audio demonstrates consistent results, keeping the TNR and TPR balanced (see Appendix C).For StressID, we remove audio and/or video, the most cumbersome modalities to acquire and examine the no-audio, no-video and real-life (i.e., no audio, no video) scenarios. The variation remains consistent for both no-audio and no-video: |∆|<8.3%. However, it is more consequent for real-life, with a significant drop in TPR for an equivalent TNR, as expected as we remove the richest modalities.Overall, even by removing modalities, ADAPT successfully detects stress or loss of consciousness with more than 60% ACC and more than 50% TPR, highlighting its ability to handle missing modalities, in contrast to all other methods unable to address this.Ablations. 1. Impact of the anchoring before fusion and choice of anchor (Table 3). Anchoring with video shows significant benefits, particularly in LOC with an 11.6% increase in ACC alongside consistent F1 scores. Similarly, for StressID, anchoring improves 69.5(2.9) 61.9(6.9) 77.1(6.3) real-life 62.0(3.2) 4.2 60.8(1.0) 3.6 68.5(4.3) -0.5 60.0(4.8) 9.5 66.9(9.5) -5.0 53.1(7.9) 24 no video 67.1(2.2) -1.1 66.2(3.9) -1.8 68.0(2.0) 0.0 61.2(4.6) 8.3 53.7(10.1) 8.5 68.8(7.1) 8.3 no audio 69.5(5.4) -3.3 63.0(12.0) 1.4 76.1(4.9) -8.1 68.3(2.9) 1.2 65.8(6.8) -4.0 70.7(7.4) 6.4 both ACC and F1 by 3.7%. Any anchor may be considered; we explore using the audio (row 3), but it leads to suboptimal performances. Overall, the anchor selection is driven by its robust unimodal performance, which remains effective despite high missing modalities. 2. Impact of feature configurations and fusion methods (Table 4 ). Compared to the 'feature fusion' and 'decision-level fusion' (rows 1,2, Chaptoukaev et al. (2023) ), our features and fusion method (last row) significantly increase ACC and F1 by 5.7% and 6.8%, respectively, further highlighting the advantages of anchoring. We also investigate applying anchoring to features from Chaptoukaev et al. (2023) (row 3) by solely training the projection head, as opposed to both the encoder and projection head. Although this yields decent results, the inability to learn features optimally is a drawback. Finally, the ADAPT entire pipeline (row 6) delivers competitive results while accommodating missing modalities. 
conclusions: 
 In this paper, we propose ADAPT, a modality-agnostic representation framework designed to operate effectively under missing modalities during both training and testing. Our framework has been challenged on two different tasks targeting the detection of physiological changes, outperforming the current state of the art while showcasing its superiority for handling missing modalities. Extensive ablations indicate the robustness of our method on different scenarios and strategies. Future work includes applications to other medical tasks.Appendix A. LOC dataset g-forces and fighter pilots. Earth's gravity, commonly referred to as 1g, induces a constant acceleration. Fighter pilots, however, encounter much higher accelerations, reaching up to 9g along the z-axis. This force induces a blood shift towards the lower body, reducing blood perfusion in the brain. Without mitigation, pilots can experience altered consciousness, encompassing diminished peripheral vision, central vision loss, Almost Loss Of Consciousness (ALOC), and G-Induced Loss of Consciousness (GLOC) (Morrissette and McGowan, 2000) , sometimes with fatal outcomes. Due to the eye's heightened sensitivity to hypoxia, initial symptoms are often visual. As retinal blood pressure drops below Intraocular pressure (usually 10-21 mm Hg), blood flow diminishes, impacting the retina initially in areas farthest from the optic disc and central retinal artery before progressing toward central vision. Fighter pilots train in centrifuges1 to apprehend the effects of +g z accelerations and perform the Anti-g Straining Maneuvers (AGSM). AGSM consist of muscle contractions in the lower limbs, synchronized with breathing exercices. These maneuvers aim to elevate arterial blood pressure to sustain blood volume in the brain (Pollock et al., 2021) . Each training is monitored by doctors through physiological data acquired in real time: electrocardiograms (ECG), electromyograms of the quadriceps (EMG), the acoustic breathing (AUDIO), the pressure on the pedals (PAL), the visual field (VF)2 , and through video monitoring. Several factors (Pollock et al., 2021; Park et al., 2015; Nunneley and Stribley, 1979) , may influence pilots' tolerance to g-forces, thus posing challenges for doctors in detecting alterations of consciousness, both in centrifuge simulations and in-flight. Moreover, expanding detection to real scenarios is not trivial. The modalities differ: the video and visual field data are impractical due to pilots' equipment (helmets and full-face O 2 masks) and technical constraints, respectively.Case study with medical doctors. Given the rare availability of videos, we conducted a case study with a sample of doctors in charge of the aeromedical training. The study aimed to assess the efficacy of relying solely on biomedical signals for identifying loss of consciousness. Protocol. Each doctor was presented with 20 complete launches accompanied by biomedical signals and tasked with labeling them as "Loss of consciousness" (Example in Figure 3 ) or "No loss of consciousness" (Example in Figure 4 ). Results. Surprisingly, doctors accurately classified 55% of launches with a variance of 6.53% using only biomedical signals. However, when provided with videos, doctors achieved 100% accuracy in launch classification. This underscores the crucial role of video data and emphasizes the necessity, from a medical standpoint, to incorporate its robust representation.Positive and negative samples. To detect alteration of consciousness, we train our models for binary classification (distinguishing between altered and unaltered ) on n-seconds windows. Such a window w t =[t-n, t] is considered positive if there is some consciousness alteration period (t s , t e ) such that t-n≥t s and t≤t e . Otherwise, w t is considered negative. We fix n = 3.1sec.  
 
 Appendix: Equipment capable of reproducing the intensity and jolt of the g-forces accelerations experienced in high-performance fighter jet.Self-filled in by pilots 


Decision: This paper received the final ratings with 1 borderline reject, 1 strong accept and 1 weak accept. The authors addressed most of the comments during the rebuttal while some justification and details remain to be further illustrated. After reconciling the comments, AC tend to accept the paper and strongly suggest the authors take the reviewers’ comments into the final revision.

Reviews: review: paper_summary: paper_summary: The proposed framework is to develop a modality-agnostic representation for multimodal physiological data by training modality-specific encoders with a contrastive learning objective to align their representations to an anchor modality. This is designed
to operate effectively under missing modalities during both training and testing.

Experiments conducted in the study demonstrate the effectiveness of ADAPT in maintaining high-quality results even when certain modalities are missing, showcasing its robustness and capability to align representations in the absence of specific modalities. The strength of ADAPT lies in its ability to address the challenges of missing modalities in multimodal learning, providing a solution for real-life applications where data collection may be incomplete.
main_review: main_review: Thanks for addressing my question. Authors provide clear comments, additional experiments. Though I think the idea of anchor is interesting, why choose video as an anchor is still unclear. I am willing to raise my score to 3r.
strength_weakness: strength_weakness: 1. I think the challenge of missing modality is not only a task to handle incomplete modality settings in both train and test stages, but also to try to reduce the effect of missing modality as much as we can. This paper focuses on the former one and just masks the missing modality's representation feature, which is not reasonable.
2. Experiments lack a detailed comparison with other state-of-the-art missing modality frameworks. Only two comparisons in one paper are insufficient.
3. The choice of anchor is unclear, can we use other modalities and how do they perform? Like, for the LOC dataset, most of the video data is missing; why not choose another modality to replace it? Please explain it from a theoretical and experimental view.
4. The experiment lacks the effect of different ratios of modality incompleteness on the experimental results. The relevant experiment setting can refer to *SMIL: Multimodal Learning with Severely Missing Modality (AAAI 2021)*.
questions: questions: 1. In common sense, the multimodal data has more information than unimodal data, which is helpful in real-life decisions. However, unimodal dataset outperformed the multimodal settings in Tab. 1.
2. Many modality-specific encoders are applied in the proposed frameworks. How can they influence the performance of this network?
3. Since the proposed method uses a masked Transformer, how does the maskx c operation influence the performance?

For other questions, please see weakness.
paper_summary: paper_summary: Focusing on detecting alterations of physiological responses in two real-life scenarios, namely: stress in individuals induced by specific triggers and fighter pilots’ loss of consciousness induced by g-forces.
main_review: main_review: The final rating reflects a comprehensive evaluation of the author's response to feedback throughout the revision process. Each comment was meticulously considered, leading to significant improvements in the clarity, depth, and accuracy of the work. The author's dedication to refining their manuscript based on the constructive critiques provided is commendable. This thorough engagement not only enhanced the overall quality of the submission but also demonstrated the author's commitment to excellence and receptiveness to feedback. The enhancements made as a result of addressing the comments have substantially contributed to the work's coherence, making it a valuable contribution to its field. Therefore, the final rating is justified by the author's successful efforts to integrate all provided feedback, ultimately elevating the work's academic and practical implications.
strength_weakness: strength_weakness: While the paper is short and concise, there are sections with grammatical errors that need addressing. Additionally, it lacks citations from recent papers, which is crucial for grounding the research in the current state of the field. These improvements are necessary for a more robust submission.
questions: questions: Nothing
paper_summary: paper_summary: This paper addresses a key question in multimodal fusion for medical data, balancing modality contributions when limited data with potential missingness in the modalities is present in the dataset. They propose a new multimodal deep learning framework named AnchoreD multimodAl Physiological Transformer (ADAPT), which in its contrastive training aligns all modalities in the space of the "strongest"anchor  to learn a joint embedding space via contrastive training and a masked multimodal trasnformer to align modalities and address missing modalities in the masking scheme. 

They evaluate on two different datasets, one for detection of stress in individuals induced by specific triggers and the next for fighter pilots’ loss of consciousness induced by g-forces. They evaluate the performance of the ADAPT model against several baselines to demonstrate improvements in classification performance.
main_review: main_review: I thank the authors for the detailed clarifications, additional experimental results and explanation of dataset characteristics for this application. Given that they have addressed my concerns adequately and improved the quality of the submission, I am willing to raise my score to 4
strength_weakness: strength_weakness: 1. The baselines being compared against seemed rather simplistic as described (In fact, very little description is available by way of architectural details). These are broadly classified as decision level (late fusion) and feature fusion (early/intermediate fusion?). On the other hand, here have been many proposed methods used in literature to achieve multimodal fusion and modality alignment, eg. using graph neural networks, CNNs, etc as deep learning methods or the classical models in (Chaptoukaev
et al., 2023), unsupervised learning (eg canonical correlation analysis).  It is unclear whether the proposed method would outperform these approaches for these tasks and datasets.

2. For the LOC dataset, it is really difficult to gauge whether the ADAPT method or any of the baselines is a clear winner based on all the metric comparison provided against individual modality predictors, especially the video modality. While I understand that the modality is rarely available in real-life, is it not used as the anchor (most important modality) for training ADAPT and is assumed to be present during training?

3. For the missing modalities experiment, if my understanding is correct, the modalities are considered missing only at test time and not during training (Tables 2, 7, 8). For a real evaluation of the impact of missing modalities, I would expect a comparison where the percentage of the modality data is considered missing both at train and test time and varied systematically, which is the more realistic setting to evaluate the robustness and success of ADAPT at addressing this issue.
questions: questions: Please refer to the points in weaknesses, especially the evaluation of robustness to missing modalities.

Comments: title: Official Comment by Reviewer UHJs
comment: Dear Reviewer,

Could you update your final rating for this paper?

Best
Thanks for addressing those concerns, I would change the rating.
Thanks for addressing those concerns. I would accept it now.
Thanks for addressing those concerns. I would accept it now.
Thanks for addressing those concerns. I would accept it now.
Thanks for addressing those concerns. I would accept it now.
We thank the reviewer for the insightful comments and questions. The major concerns regarding the challenges posed by the LOC dataset and how ADAPT addresses them were addressed in the comments to the submission. 

**W1. "The baselines compared against seemed rather simplistic[...]."**  This point is addressed in R2's comment.

**W2. "For the LOC dataset, it is really difficult to gauge whether the ADAPT [...]."** This point is addressed in the main comment.

**W3. “[...] For the missing modalities experiment, if my understanding is correct, the modalities are considered missing only at test time and not during training (Tables 2, 7, 8). [...]."** Relates also to **R1.W4 "The experiment lacks the effect of different ratios of modality incompleteness[...]."**
ADAPT and robustness to missing modalities. StressID and LOC datasets exhibit missing training and test sets modalities. StressID presents 18% of observations with missing video and 46% with audio missing, while LOC suffers from a significant 90% absence of videos across its dataset. We introduce a dedicated paragraph in Section 4, Subsection 4.1, to clarify the missing modalities across datasets. In this work, we focus on examining ADAPT's robustness to various subsets of modalities during inference (Table 2) while training with the inherent missing modalities present in both datasets. This approach aims to underscore ADAPT's real-life applicability. Our results demonstrate that ADAPT maintains competitive performance across the three studied modality scenarios: no video, no audio, and no audio with no video. 

Next, we answer point by point the questions raised by the reviewer: 

**Q1. "What are the class distributions for both the datasets? [...]"**
We apologize if this aspect wasn't clear in our initial submission. The LOC dataset exhibits a class imbalance of {1:50}, while the StressID dataset does not suffer from any imbalance. The imbalance ratio for the LOC dataset was specified in the metrics paragraph of subsection 4.1. In this study, we mitigate class imbalance using cost-sensitive learning, specifically employing the Cost-Sensitive Cross Entropy loss weighted by inverse class frequency (Huang et al., 2016; Wang et al., 2017). This loss function is utilized to train the classifiers of ADAPT, unimodal baseline, ShaSpec+, and Feature Fusion. In the revised version of the manuscript (Section 4, Subsection 4.1), we provide enhanced clarity by highlighting the imbalance ratio in the dataset presentation and adding further details on our approach to address it. 
ADAPT effectively handles class imbalance, achieving the highest True Positive Rate (TPR) on LOC among multimodal methods. Moreover, it tends to strike a superior balance between True Negative Rate (TNR) and TPR, as illustrated in Figure 2 and emphasized in the opening paragraph of section 4, subsection 4.2.

**Q2. "The authors should consider reporting the area under the precision recall curve (AU-PRC) to allow for easier comparison across methods."**
In this paper, we assess ADAPT using Class Balanced Accuracy, True Positive Rate (TPR), and True Negative Rate (TNR). These metrics are highly robust for evaluating performance on imbalanced datasets (Huang et al., 2016; Luque et al., 2019). While we initially considered precision, it has been shown to exhibit significant bias in the context of highly imbalanced datasets (Luque et al., 2019). Consequently, the AU-PRC would similarly suffer from this bias. Specifically, due to the prevalence of true negatives, the fraction of false positives may significantly outweigh the number of true positives, leading to a large denominator in the precision calculation and consequently deflating the precision due to false negatives.

**Q3. "The presentation and font of Table 4 makes it really hard to parse."**
We appreciate the reviewer's feedback. The revised manuscript now includes an updated table (Section 4, Subsection 4.2).
We thank the reviewer for their positive evaluation of our work and address point by point the weaknesses raised by the reviewer: 

**W1. “It lacks citations from recent papers.”**
Relates also to **R1.W2. “Experiments lack a detailed comparison with other state-of-the-art[...]”** and to **R3.W1. “The baselines compared against seemed rather simplistic[...]”**

We compare our approach directly with the methods utilized in (Chaptoukaev et al., 2023), which established the state-of-the-art for the StressID dataset. While acknowledging the importance of further comparisons, current methodologies exhibit limitations due to inherent dataset challenges. As detailed in the revised Section 2 of the manuscript, these methods suffer from (1) limited generalizability to more than two modalities (Ma et al., 2021), (2) bias towards the most abundant modalities leading to suboptimal performance, as observed in the severely underrepresented video modality in the LOC dataset (Konwer et al., 2023), and (3) an assumption of homogeneity among modalities, complicating extension to heterogeneous modalities like imaging and biomedical signals by utilizing shared encoders with uniform input dimensions (Wang et al., 2023).
Following the reviewer's suggestions, we compare against an additional state-of-the-art method, ShaSpec (Wang et al., 2023), a SOTA method that outperforms SMIL (Ma et al., 2021) on Audiovision-MNIST, altered to be compatible with our datasets. ShaSpec maximizes the utilization of all available input modalities during training and evaluation by learning shared and specific features for better data representation. However, due to the varying dimensions of our input modalities (e.g., 3D video, 1D biomedical signals), employing a shared encoder is nontrivial. Hence, we adopt an adapted version, ShaSpec+, where encoded inputs are fed to the shared encoder instead of raw inputs. To ensure a fair comparison with ADAPT, we use identical settings, including the same encoders (Hiera (Ryal et al. 2023) for video, Byol-a for audio, and 1D CNN for biomedical signals). Additionally, to address class imbalance for the LOC dataset, we substitute the cross-entropy loss with cost-sensitive cross-entropy (Huang et al. 2016). 
The Table (Table 1, Section 4 in the manuscript) below demonstrates the results. In the case of complete modality during testing, ADAPT demonstrates superior performance compared to both methods (Chaptoukaev et al., 2023) and the additional comparison with ShaSpec+ (Wang et al., 2023). Although ShaSpec+ yields competitive results on the Stressed dataset, it achieves the lowest true positive rate (TPR) (7.3%) on the LOC dataset, highlighting the relevance of ADAPT in extreme scenarios. Our experiments reveal ADAPT's potential to handle missing modalities robustly, even in extreme imbalance (e.g., 1:50 class ratio for LOC) or severe modalities missing during training.

**W2. “Sections with grammatical errors”**
Thank you for noticing this. Authors proofread several times the manuscript and corrected grammatical errors in the updated pdf.
The primary concern relating to the role of the video and, more generally, the anchor in our overall architecture has been addressed in our main comments to the reviewers. Substantial revisions were applied to the manuscript in the introduction, the related work (Section 2), and the section relating to the datasets (Section 4, Subsection 4.1). 
We answer the different questions and weaknesses raised by the reviewer point by point.

**W1. “I think the challenge of missing modality is not only a task to handle incomplete train and test stages, but also to try to reduce the effect of missing modality as much as we can. [...].”**
We thank the reviewer for raising this important point regarding the challenges inherent to missing modalities in multimodal learning. Our method effectively manages and mitigates the impact of missing modalities, as demonstrated in Table 1 (last row) and Table 2 (first row). ADAPT consistently performs well across datasets, regardless of full modality presence or missing modalities. 

|| LOC || StressID ||  
|:-| :-|:-|:-|:-|  
| | **ACC** | **F1** | **ACC** | **F1** |  
|full modality | 67.4(1.2) |76.9(2.5)| 69.5(3.7) | 75.9(4.3)|
| with missing modalities | 66.2(3.3)| 77.7(4.3) | 69.5(2.9)|70.0(3.1)|

Moreover, ADAPT maintains stability even when a modality is entirely absent from the test set, as shown in Table 2, underscoring its resilience.
Architecturally, ADAPT achieves robustness through two key mechanisms detailed in Section 2: anchoring and modality dropout. Anchoring involves embedding all modalities into the same latent space, mitigating biases towards the most available modalities, and ensuring scalability. We select video as the anchor due to its strong unimodal performance, even with significant missing data (e.g., video exhibits a missing ratio of 90% for LOC and 18% for StressID), leveraging its strong priors for downstream tasks where video may be unavailable. Additionally, we employ multi-view contrastive learning and a tailored data augmentation technique for multimodal learning: Modality dropout (Shi et al., 2022). Modality dropout mitigates the model’s reliance on any single modality, enhancing its robustness by randomly considering missing features of some modalities.
All in all, ADAPT effectively addresses the handling of missing modalities while ensuring robustness. 

**W2. “Experiments lack a detailed comparison with other state-of-the-art[...].”** This point is addressed in the comment to R2.

**W3. “The choice of anchor is unclear [...]”** This point is addressed in the general comment.

**W4. “The experiment lacks the effect of different ratios of modality incompleteness[...].”** This point is addressed in the comment to R3.

**Q1. “In common sense, the multimodal data has more information than unimodal data [...].”** This point is addressed in the general comment.

**Q2. “Many modality-specific encoders are applied in the proposed frameworks.”**
Our framework relies on a separate stream architecture rather than a single stream, motivated by several factors: (1) it enables using foundational models for aligning all modalities to the anchor (e.g., video); (2) using a shared encoder is challenging due to the heterogeneous modalities considered for both datasets and more precisely varying dimensions in modality inputs, such as 3D video and 1D biomedical signal. We have clarified this choice in the related work section of the revised manuscript to provide better insight into our approach. Regarding the selection of the encoders, we utilized state-of-the-art methods for each modality: Hiera (Ryal et al., 2023) for video, BYOL-A for audio (Niizumi et al., 2021), and 1D CNN for biomedical signals (Wang et al., 2023a). Additionally, for comparison, we considered using the handcrafted features proposed in (Chaptoukaev et al., 2023) in Table 4; however, as highlighted in row 3, it leads to suboptimal performances compared to ADAPT (row 6). Furthermore, we also employed our anchored unimodal encoders (rows 4 & 6) with (Chaptoukaev et al., 2023) fusion methods. It outperforms the fusion methods with handcrafted features by a large margin (+ 5% ACC and +6% in F1). 
In summary, our experiments highlight the powerful representations of our anchored unimodal encoders and, more generally, our separate stream architecture. 

**Q3. “Since the proposed method uses a masked Transformer [...]”**
In this work, we choose to leverage the mask in the attention mechanism from the transformer architecture. This choice is inspired by prior successful implementations (Milecki et al., 2022; Ma et al., 2022) in tasks such as follow-up patient examination completion and handling missing modalities in bimodal datasets.
We would like to thank the reviewers for evaluating our work and for their insightful comments and queries. We provide here additional clarifications and address the main comments. A detailed response to each reviewer is also given (R1 - UHJs, R2 - wJd1, R3 - 6ssF). 
We have submitted a revised version of the manuscript, which addresses the points highlighted by the reviewers and completes the bibliography. Modifications made to the manuscript are written in blue.


**R1 and R3 expressed concerns about the challenges posed by the LOC dataset and how ADAPT addresses them**, both in terms of multimodal architecture and the anchor choice. We addressed this issue more explicitly in our answer and the revised manuscript.

(i) *The extreme case of the LOC dataset.* The LOC dataset presents several extreme characteristics: (1) a severe class imbalance (1:50), (2) a high proportion of missing videos (present in only 10% of observations), while demonstrating strong unimodal performances compared to biomedical signals, (3) and impracticality in real-life data collection scenarios.
Given the rare availability of videos, we conducted a case study with a sample of doctors in charge of the aeromedical training. The study aimed to assess the efficacy of relying solely on biomedical signals for identifying loss of consciousness. Each doctor was presented with 20 complete launches accompanied by biomedical signals and tasked with labeling them as "Loss of consciousness" or "No loss of consciousness." Surprisingly, doctors accurately classified **only** 55% of launches with a variance of 6.53% using only biomedical signals. However, when provided with videos, doctors achieved 100% accuracy in launch classification (details on the case study were added in Appendix A). This underscores the crucial role of videos and emphasizes the necessity, from a medical standpoint, to incorporate their robust representation. Therefore, leveraging the strong priors of video data during training is imperative for achieving robust results on the LOC dataset, even in scenarios where video data is utterly absent during inference.
As demonstrated in Table 1, ADAPT effectively leverages video priors, surpassing other multimodal approaches and maintaining strong performance even when video data is absent from the test set (Table 2). Additionally, on more balanced datasets like the StressID dataset (with ½ positive and negative ratio), ADAPT consistently exhibits strong multimodal performance, outperforming unimodal methods and the previous state-of-the-art established by Chaptoukaev et al. (2023).

(ii) *The choice of anchor.* Anchoring facilitates the integration of all modalities into a unified feature space, enhancing training efficacy and ensuring linear scalability for joint representation. By aligning all modalities to a single representation (our anchor), we mitigate the quadratic complexity inherent in pairwise consideration of all modalities. As Konwer et al. (2023) highlighted, the absence of specific modalities in some observations can bias the shared latent space toward more available ones. Despite video being absent in 90% of samples in the LOC dataset, it exhibits robust unimodal performance when employing a pre-trained encoder. This observation motivates choosing video as the anchor to leverage its strong representation in the shared latent space. Table 3 illustrates the pivotal role of video. This choice yields a noteworthy increase of +12% in ACC on the LOC dataset while maintaining F1 scores and a +4% improvement in both ACC and F1 on the StressID dataset compared to ADAPT without anchored unimodal encoders (row 1). Moreover, as discussed in Section 3, any modality can serve as the anchor. To further support this assertion, we conduct additional experiments using audio as the anchor, for which we can also leverage a pre-trained encoder, yet consistently observe more robust results with video as the anchor (we modify Table 3 accordingly in the manuscript). 
In summary, the choice of the anchor is driven by its robust unimodal performance, which remains effective even in the presence of high missingness.