HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data

Authors: konstantin  hemker
nikola  simidjievski
mateja  jamnik
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 22:29:51

Abstract: Technological advances in medical data collection, such as high-throughput genomic sequencing and digital high-resolution histopathology, have contributed to the rising requirement for multimodal biomedical modelling, specifically for image, tabular and graph data. Most multimodal deep learning approaches use modality-specific architectures that are often trained separately and cannot capture the crucial cross-modal information that motivates the integration of different data sources. This paper presents the **H**ybrid **E**arly-fusion **A**ttention **L**earning **Net**work (HEALNet) – a flexible multimodal fusion architecture, which: a) preserves modality-specific structural information, b) captures the cross-modal interactions and structural information in a shared latent space, c) can effectively handle missing modalities during training and inference, and d) enables intuitive model inspection by learning on the raw data input instead of opaque embeddings. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models, substantially improving over unimodal and multimodal baselines whilst being robust in scenarios with missing modalities. The code is available at https://github.com/konst-int-i/healnet.

Main paper: 
 introduction: 
 A key challenge in Multimodal Machine Learning is multimodal fusion, which is the integration of structurally heterogeneous data into a common representation that reduces the dimensionality of the data whilst preserving salient biological signals [Steyaert et al., 2023] . Fusion approaches are well-studied in areas where there is a clearly defined shared semantic space, such as audio, visual, and text tasks like visual question answering [Goyal et al., 2016] , image captioning [Yu et al., 2020] , or multimodal dialogue [Liang et al., 2023] . However, healthcare data commonly consists of 2D or 3D images (histopathology and radiology), graphs (molecular data), and tabular data (multi-omics, electronic health records), where cross-modal relationships are typically more opaque and complex, the modalities often do not share semantics, and common representations are less explored. The Figure 1 : Overview of HEALNet (Hybrid Early-fusion Attention Learning Network) using a shared and modality-specific parameter space to learn from structurally different data sources in the same model (Fig. 1A ). The shared space is a learned latent embedding S that is iteratively updated through d attention-based fusion layers and captures the shared information between modalities. The hybrid early-fusion layer (Fig. 1B , and Eq. 3) learns the cross-attention weights W m = {W m X m ) which are shared between layers. These layers capture the structural information of each modality and encode it in the shared embedding after a pass through a self-normalising network (SNN) layer.fusion stage describes how far the multimodal representation is removed from the raw (unimodal) data, and is commonly categorised into early, intermediate, and late fusion [Baltrusaitis et al., 2019] .Early fusion approaches combine the raw data early in the pipeline, which allows the training of a single model from all data modalities simultaneously. However, most of these approaches use simple operations such as concatenation, which removes structural information, or take the Kronecker product [Chen et al., 2022a] , which can lead to exploding dimensions when applied to multiple modalities and large matrices. Late fusion, on the other hand, trains separate models for each modality at hand, which allows capturing the salient structural information but prevents the model from learning interactions between modalities [Liang et al., 2023] . Intermediate fusion approaches attempt to overcome this trade-off by learning a low-level representation (embedding) for each modality before combining them. This can result in discovering the cross-modal interactions whilst taking advantage of each modality's internal data structure. The problem with many intermediate fusion approaches is that the learnt latent representation is not interpretable to human experts, and handling missing modalities is often noisy [Cui et al., 2023] . To overcome these issues, we posit that there is a need for more sophisticated early fusion methods, which we refer to as hybrid early-fusion, that: a) preserve structural information and b) learn cross-modal interactions, and c) work on the raw data, thus allowing for in-model explainability.In this paper, we propose Hybrid Early-fusion Attention Learning Network (HEALNet, Figure 1 ), a novel hybrid early-fusion approach that leverages the benefits of early and intermediate fusion approaches, and scales to any number of modalities. The main idea behind HEALNet is to use both a shared and modality-specific parameter space in parallel within an iterative attention architecture. Specifically, a shared latent bottleneck array is passed through the network and iteratively updated, thus capturing shared information and learning tacit interactions between the data modalities. Meanwhile, attention weights are learned for each modality and are shared between layers to learn modality-specific structural information. We demonstrate the multimodal utility of HEALNet on survival analysis tasks on four cancer sites from The Cancer Genome Atlas (TCGA) data, combining multi-omic (tabular) and histopathology slides (imaging) data. Our results show that HEALNet achieves state-of-the-art concordance Index (c-Index) performance compared to other fusion models on all four cancer datasets for multimodal patient survival prediction. More specifically, HEALNet leads to an average improvement of up to 7% compared to the best unimodal benchmarks and up to 4.5% compared to the best early, intermediate, and late fusion benchmarks, which we see as a promising validation of our hybrid early-fusion paradigm. In summary, the contributions of our proposed HEALNet include:• Preserving the modality-specific structure: HEALNet outperforms unimodal tabular (omic) and imaging (histopathology) baselines without a dedicated modality-specific network topology.• Learning cross-modal interactions: HEALNet effectively captures cross-modal information, achieving a significantly higher multimodal uplift compared to existing early, intermediate, and late fusion baselines.• Handling missing modalities: We show that HEALNet effectively handles missing modalities at inference time without introducing further noise to the model, a common problem in the clinical use of multimodal models.• Model inspection: HEALNet is explainable 'by design' since the modality-specific attention weights can provide insights about what the model has learned without the need for a separate explanation method. We believe that they are useful for model debugging and validation alongside domain experts. 
related work: 
 In this paper, we focus on multimodal learning problems from biomedical data, where the data modalities are structurally heterogeneous, specifically combining image (e.g., Whole Slide Imagery (WSI)) and tabular (e.g., omic and clinical) data. This aspect is different and more general than approaches that focus on combining homogeneous modalities, such as multi-omic data, where the combined modalities have the same structural formalism (tabular) [Sammut et al., 2021 , Dai et al., 2021] . As such, our work closely relates to several approaches for multimodal data fusion that consider learning from WSI images and genomic data. Namely, Cheerla and Gevaert [2019] introduce a two-step procedure, combining a self-supervised pre-training step with a downstream fine-tuning for survival analysis. In the first self-supervised step, a modality-specific embedding is trained and optimised using a similarity loss (similar to contrastive learning) between the embeddings. The latter step includes (supervised) fine-tuning of a survival model, trained and optimised using the Cox loss. HEALNet, on the other hand, implements a sequential architecture for end-to-end training to learn across modalities without requiring a separate pre-training step. The model architecture is flexible to allow for both training on the raw input data as well as leveraging pre-trained model encoders, which may further improve its performance on a particular task of interest.HEALNet's multimodal fusion capabilities build on attention architectures [Vaswani et al., 2017] .A popular such architecture is the Perceiver [Jaegle et al., 2021] , which uses iterative self-and cross-attention layers and achieves impressive performance across various unimodal tasks. However, this architecture is restricted to single modalities unless inputs are concatenated before training, which may remove salient structural signals. In a multimodal context, cross-attention has been used as a core component for several intermediate and late fusion models. 'Multimodal co-attention' (MCAT) [Chen et al., 2021] is an attention-based fusion approach that uses a tabular modality as the query and the imaging modality as the key and value array to train a cross-attention unit. Xu and Chen [2023] further extend this concept by introducing a refined co-attention mechanism based on optimal transport, where one modality is used to better contextualise the other. Such co-attention approaches only scale to two modalities since the co-attention units can only take in one set of query-key-value inputs. It also requires having a 'primary' modality that should be contextualised by a 'secondary' modality, which may not always be the case. Chen et al. [2022a] present an intermediate fusion approach that first constructs modality-specific embeddings before passing them through a gating-based attention mechanism, combining the output via a Kronecker product. In turn, the resulting high-dimensional '3D multimodal tensor' is used for a downstream survival prediction task. Similarly, Chen et al. [2022b] propose a late fusion approach which implements modality-specific model encoders before combining each model's output via a gating mechanism. Here, the attention mechanism is only applied to the imaging modality (WSI), which serves as a method for learning more general representations by combing the patch-level latents, but also allows for explainable post-hoc image analysis of the identified regions. An inherent limitation of such an approach is that the encoders are trained separately (without the other modality's context), and consequently, the explanations only account for unimodal information.Another common limitation of all the above approaches is that they expect fixed tensor dimensions during training and inference. An alternative to this is presented with sequential fusion methods such as by Swamy et al. [2023] , where modalities can be skipped if not present. Currently, the problem with this approach is that it only works on 1D tensors and latent states, relying on encoders specific to this architecture.In contrast, HEALNet overcomes many of the limitations of the related fusion approaches. First, its design readily scales to more than two modalities without additional computational overhead beyond the one introduced by the additional modality-specific encoders. Namely, the iterative attention mechanism alleviates the use of fusion operators (such as a Kronecker product) that render highdimensional embeddings, and allows for combining many modalities while preserving the structural information of each. Moreover, HEALNet can learn cross-modal interactions since the modalities are used as mutual context for each other's updates. This also allows for more holistic explanations compared to late fusion approaches that use modality-specific models trained in isolation. 
healnet: 
 Preliminaries. Let X m represent data from modality m = 1, ..., j ∈ N. Let X m ∈ R p×n be either a tabular dataset with p features and n samples or an image dataset X m ∈ R h×w×c×n with n images with height h, width w and channels c. The goal of a multimodal fusion approach is to learn a fusion function f () such that y = f (X 1 , ..., X j ). A conventional design of such a system (such as the related work discussed in Section 2) is to first learn a modality-specific function g m (), which learns an intermediate representation h m = g m (X m ), and then apply a fusion function f () for predicting the target variable ŷ = f (h 1 , ..., h j ).Architecture. We depict HEALNet in Figure 1 . Instead of computing h m and applying a single fusion function f (), HEALNet uses an iterative learning setup. Let t denote a step, where the total number of steps T = d × j for the number of fusion layers d ∈ θ. Let S t represent a latent array shared across modalities, initialised at S 0 where S ∈ R a×b for embedding dimensions a, b ∈ N and is updated at each step. First, instead of learning an intermediate representation h m as encoded inputs for X m , we compute the attention weights asEQUATIONfor each modality m at each step t. Second, we learn an update function ψ() to be applied at each step. The update of S with modality m is given byEQUATIONfor total time steps T and attention function α (Equation 1). For parameter efficiency, the final implementation uses weight sharing between layers. Across modalities, each early-fusion layer becomes an update function of the formEQUATIONThe final function for generating a prediction only takes the final state of the shared array and returns the predictions of the target variable ŷ = f (S T ) as a fully-connected layer.Figure 1 depicts a high-level visual representation of this approach, showing: (a) Hybrid Early-fusion Attention Learning Network, and its key component (b) the early fusion layer (as given in Equation 3). We use attention layers since they: a) make fewer assumptions about the input data (e.g., compared to a convolutional network), and b) their ability to provide context to the original modality through the cross-attention mechanism. We start by initialising a latent embedding variable, which is iteratively used as a query into each of the fusion layers, and is updated with information from the different modalities at each layer pass. We chose the iterative attention paradigm due to its highly competitive performance on a range of unimodal tasks [Jaegle et al., 2021] . Passing the modalities through the shared latent array helps to significantly reduce the dimensionality whilst learning important structural information through the cross-attention layers. The HEALNet pseudocode is detailed further in Appendix A.Preserving structural information. To handle heterogeneous modalities, we use modality-specific cross-attention layers α() and their associated attention weights a(t)m , whilst having the latent array S shared between all modalities. Sharing the latent array between modalities allows the model to learn from information across modalities, which is repeatedly passed through the model (Figure 1A ). Meanwhile, the modality-specific weights between the cross-attention layers (Figure 1B ) focus on learning from inputs of different dimensions, as well as learning the implicit structural assumptions of each modality. Specifically, in this work, the employed attention mechanism refers to the original scaled dot product attention from [Vaswani et al., 2017] , with adjustments for tabular and image data.Formally, given a tabular dataset as a matrix X m = {x(11) m , ..., x (np)m }, with n ∈ N samples and p ∈ P features (e.g., a gene expression), we aim to learn the weight matrices W  and W (v) m that act as a linear transformation for S and X m that form the queries (q(q) m , W (k) m ,(n) m ), keys (k (n) m ) and values (v (n)m ) for each sample passed into the layer. The general scale dot-product attention generates attention scores for each feature and can be expressed in Cartesian Notation asEQUATIONIn other words, for each channel p and sample n, an attention layer calculates the normalised and scaled attention weight being given the context of all other features for that sample. This has the benefit that the attention scores are always specific to each input given to the attention layer.From this, we can extract both the normalised attention matrix A as well as the context matrixC p (q, K, V ) = P i=1 A p,i × v i, which is the attention-weighted version of the original input x. In our case, we need to combine multiple inputs to apply the iterative attention mechanism (i.e., cross-attention) -these inputs are the latent S and the input matrix X m for each modality. To do this, we use the latent array as the query and the input tensor as the keys and values, respectively. Given a latent array S, we define the query for each sample q(n) m = W (q)m S and the keys and values ask m = W (k) m x and v (n) m = W (v) m for all samples n ∈ [1, N ].Intuitively, the iterative cross-attention can be seen as aligning the query to each modality individually, but not aligning the modalities themselves to ensure that its unique signal is captured. At each time step, the query for the next update provides context from the other modalities of previous updates.High-dimensional biomedical data. Attention-based architectures are typically trained on vast datasets (which are commonly available for vision and language tasks). The challenges of working with biomedical data, however, are their high dimensionality whilst often having relatively few samples (i.e., patients). For example, a dataset (such as TCGA-BLCA) contains whole slide images of approximately 6.4 gigapixels (80k × 80k pixels) in its highest resolution and includes thousands of multi-omic features, but only from a few hundred patients in total. This leads to two common problems in digital pathology -overfitting [Holste et al., 2023] and high computational complexity.First, to counteract overfitting, HEALNet implements both L1 and L2 regularisation. Considering the relatively large number of parameters required for the attention layers, we found L1 regularisation to be important. Beyond that, we opted for a self-normalising neural network (SNN) block, due to its proven robustness and regularisation properties [Klambauer et al., 2017] .Second, handling the extremely high resolution of the whole slide images (WSIs) within computational constraints is also a challenge. We address this by extracting non-overlapping 256x256 pixel patches on the 2x and 4x downsampled whole-slide image (~0.5 and 1.0µm per pixel respectively). For comparability with other work, we extract a 2048-dimensional feature vector for each patch using a standard ResNet50 pre-trained on the Kather100K dataset, which consists of 100k histopathology images of both healthy tissue and colorectal cancer tissue [Pocock et al., 2022] . While HEALNet also achieves competitive results on the raw patch data, this requires more significant downsampling to be computationally feasible at scale.A common challenge in clinical practice is missing data modalities during inference. Namely, in practical scenarios, while models have been trained on multiple modalities, there is a great chance that not all data modalities are available for predicting the patient's outcome. Therefore, multimodal approaches must be robust to such scenarios. Typical intermediate fusion approaches would need to randomly initialise or impute a tensor of the same shape, or sample the latent space for a semantically similar replacement to pass into the fusion function f (h 1 , ..., h j ; θ) at inference, which is likely to introduce noise. In contrast, HEALNet overcomes this issue by design: the iterative paradigm can simply skip a modality update step (Equation 3) at inference time in a noise-free manner. Note that these practical benefits also extend to training scenarios, where a (typically small) number of samples are missing some modalities. Rather than imputing this data or completely omitting the samples, HEALNet can train and utilise all the available data using the same update principle. 
experiments: 
 Datasets. We empirically evaluate the utility of HEALNet on survival analysis tasks on four cancer datasets from The Cancer Genome Atlas (TCGA). Concretely, we use modalities which are structurally heterogeneous such as the ones formalised in a tabular or image dataset. Our tabular data structures consist of three sources: bulk gene expressions (RNAseq), mutations (whole-genome sequencing), and copy number variations. HEALNet treats these as three separate modalities, while for the baselines that only support two modalities we had to concatenate them -in continuation of this paper we refer to these as omic modality. The WSI modality includes H&E-stained whole slide tissue images of the same patient cohorts as in the omic modality. Namely, the four cancer datasets that we include are Muscle-Invasive Bladder Cancer (BLCA, n=436), Breast Invasive Carcinoma (BRCA, n=1021), Cervical Kidney Renal Papillary Cell Carcinoma (KIRP, n=284), and Uterine Corpus Endometrial Carcinoma (UCEC, n=538) (further dataset details are provided in Appendix C). These specific sites were chosen based on their sample size (BRCA, BLCA, and UCEC are some of the largest TCGA datasets), performance indicators reported in previous unimodal studies (e.g., KIRP highest on omic, UCEC highest on WSI only [Chen et al., 2022b] ), and other omic properties (e.g., BLCA and UCEC are known for their very high gene mutation rate [Ma et al., 2022] ).Task setup. We focus on modelling overall survival. For each patient, we are provided with right-censored censorship status c (a binary variable on whether the outcome was observed at the end of the study or not) and survival months since data recording. In line with our baselines [Chen et al., 2022b] , we take non-overlapping quartiles k for the uncensored patients and apply them to the overall population, which assigns a categorical survival risk to each patient. For the survival task, our prediction model and the baselines are set to output logits of these survival buckets y logits = f (X m , S; θ, ρ). Using these, we calculate the hazard as the sigmoid of the logits f hazard = 1 1e -y logits , and the corresponding survival as f survival = k 1 1 -f hazard . The survival, hazards, censorship status and discretised survival label are all used to calculate the negative log-likelihood (NLL) loss from the proportional hazards model defined in [Zadeh and Schmid, 2021] . The concordance index is then calculated by comparing all study subject pairs to determine the fraction of pairs in which the predictions and outcomes are concordant [Brentnall and Cuzick, 2018] .During development, we found that the proportion of uncensored patients sometimes can be as little as 15% of the cohort (UCEC). Applying the survival bins from such a small sub-sample led to very imbalanced discretised survival on the full cohort. To counteract this, we added the option to apply survival class weighting in the loss function, implemented as the inverse weight of the survival bins. Additionally, note that the NLL loss and the concordance index are sometimes only loosely related. Therefore, our loss weighting helped to stabilise the correlation between the NLL loss and the c-Index. To ensure fair comparison and comparability with the baselines, we employed the same NLL loss and weighting for training HEALNet. Nevertheless, HEALNet is readily extensible and can be implemented with other survival loss functions such as the Cox loss, which can potentially lead to more stable c-Index results [Cheerla and Gevaert, 2019] .Baselines. In all experiments, we compare HEALNet to state-of-the-art uni-and multimodal approaches that utilise different fusion strategies. This includes Porpoise [Chen et al., 2022b] , which uses a late fusion gating method to combine the latent representations learned from a self-normalising network (SNN) on the omic modality and an attention-based multiple-instance learning network (AMIL) [Chen et al., 2022b] . In terms of intermediate fusion, we include MCAT [Chen et al., 2021] and MOTCAT [Xu and Chen, 2023 ], which leverage co-attention between latent representations resulting from modal-specific encoders. Furthermore, we compare to MultiModN [Swamy et al., 2023] , a sequential fusion approach for multi-task learning with a pre-defined set of encoders. We also include the Perceiver model [Jaegle et al., 2021] , which has shown strong performance on various unimodal tasks through its iterative attention mechanism. Following the original implementation, we benchmark its multimodal integration capabilities using early fusion via concatenation. Finally, for each of our multimodal baselines, we trained unimodal variants as reported in the task-specific papers [Chen et al., 2022b [Chen et al., , 2021]] , as well as unimodal models trained with HEALNet using a single modality. We report the best unimodal model out of the set. The full results of all unimodal baselines can be found in Appendix B.Implementation details. For each experiment we employ 5-folds of repeated random sub-sampling (Monte Carlo cross-validation) with a 70-15-15 split for the training, validation and test sets. All reported results show the models' performance on test data that was not used during training or Table 1 : Mean and standard deviation of the concordance Index on four survival risk categories. We trained HEALNet and all baselines on four TCGA tasks and report the performance on the test set across five folds. HEALNet outperforms all of its multimodal baselines and three out of four unimodal baselines in absolute c-Index performance. validation. We re-train all of the baseline models using the code reported in the respective papers. All models have been run under the same circumstances and using the same evaluation framework (including data splits and loss weighting). For hyperparameter tuning, we ran a Bayesian Hyperparameter search [Bergstra et al., 2013] for all training parameters across models. Model-specific parameters of the baselines were tuned if the optimal parameters on the TCGA datasets were not available. The final set of hyperparameters can be found in Appendix D. All experiments were run on a single Nvidia A100 80GB GPU running on a Ubuntu 22.04 virtual machine. HEALNet is implemented in the PyTorch framework and available open-source at https://github.com/konst-int-i/healnet. 
results: 
 The results of the survival analysis are summarised in Table 1 , showing the mean and standard deviation of the c-Index across five cross-validation folds. Across all tested cancer sites, HEALNet outperforms all multimodal baselines. This corresponds to an improvement over the multimodal baselines of approximately 7%, 1%, 3% and 6% on the BLCA, BRCA, KIRP, and UCEC tasks respectively. HEALNet also exhibits more stable behaviour compared to the multimodal baselines, as evident from the standard deviation (across folds), which is lower in three out of four datasets.The unimodal baselines shown in Table 1 correspond to the best-performing models from a selection of unimodal baselines we trained (refer to the complete list in Appendix B). Compared to the better of the two unimodal baselines, HEALNet achieves an approximately 10% higher c-Index on BLCA and BRCA, a 4% higher c-Index on KIRP, and a nearly equivalent performance on UCEC. We refer to this as a multimodal uplift, which is illustrated in Figure 2 , where we compare the improvement of the different multimodal models and fusion strategies to the best unimodal model. Note that the UCEC dataset is an example of modality dominance, where all informative signal stems from one modality (in this case WSI), while the signal from the other modality can be either non-informative or noisy. Intermediate and late fusion approaches, which directly combine modalities, are less robust in such cases. For instance, in the case of Porpoise and MCAT, this can even lead to performance degradation. Since HEALNet is more robust to such noise, it leads to performance comparable to the unimodal variant. To further assess the robustness of HEALNet, we evaluate its performance in scenarios with missing modalities. Specifically, using HEALNet trained on four modalities (WSI + 3×Omics), we investigate its performance when modalities are missing during inference. Note that half of the test samples include only a WSI modality, while the other half is an Omics modality, chosen randomly. The unimodal baseline corresponds to the predictions of the available modality in the same way that a late fusion model would use two unimodal models followed by an XOR gating mechanism to make its prediction. For completeness, we also report results where the whole test set consists of samples with either Omics or WSI modality, rather than a combination of both. Note that the unimodal baselines are HealNet models, trained on a single modality. The results from this analysis, given in Table 2 , show that our proposed HEALNet, pre-trained on both modalities, archives stable and generally better performance than a late fusion baseline (as commonly performed in practice). 
discussion: 
 Structure-preserving fusion. The results support our hypothesis that HEALNet is able to learn the structural properties of each modality and convert the structural signal into better performance.The quantitative evidence of this is given by HEALNet's absolute c-Index performance across tasks (Table 1 ), performing substantially better than the early fusion baseline that employs concatenation. Additionally, HEALNet allows for further qualitative analysis of this behaviour, as we can visualise the sample-level attention of different regions of the whole slide image. The attention maps in Figure 3 show a sample where the model identified multiple patches in the same region -distilling the wider image down to local information for which we would typically use convolutional networks.HEALNet learns about high-level structural relationships by using a hybrid of modality-specific and shared parameters. For each modality, we learn attention weights (Equation 1) simultaneously in an end-to-end process. We believe that this distinguishes our hybrid early-fusion approach from conventional early and intermediate fusion methods. Instead of removing structural information (i.e., concatenation) or creating excessively large input tensors (i.e., Kronecker product), our hybrid early-fusion is able to preserve such structures by design. Similar to intermediate fusion methods, we use a shared latent space to capture cross-modal dependencies. However, instead of creating a latent space through multiple encodings before combining them via a downstream function (f (h 1 , ..., h j ; ϕ)), HEALNet learns an update function (Equation 3) that iteratively updates the shared latent with modality-specific information. Nevertheless, a limitation of such an approach is akin to training a higher number of parameters (i.e., large attention matrices), but on relatively few samples, making it prone to overfitting. Hence, we found that HEALNet can be sensitive to the choice of regularisation mechanisms, even though the current regularisation techniques (such as L1 + SNN) have shown to be effective (see Figure 4 in Appendix E).Cross-modal learning. The motivation of using a hybrid early-fusion over a late fusion approach is to enable the model to learn cross-modal interactions that are unavailable to modality-specific models trained in isolation. We can see the effect of this in Figure 2 , showing HEALNet's substantially higher uplift compared to the late fusion benchmark. Additionally, Table 2 shows that even when a modality is missing, HEALNet outperforms the best unimodal models that are present, indicating that the multimodal embedding is implicitly inferring some information about the missing modality.We note, however, that using a multimodal model is not always a requirement, especially in the presence of modality dominance, which we see on the UCEC dataset. This dominance can be explained by the relatively high morphological variety that has been found in endometrial carcinomas (i.e., UCEC), which is expressed by spindled, stromal, or extremely large cells [Rabban, 2020] . This visual signal can only be picked up by the WSI, which is a trend we saw consistently across baselines. Nonetheless, HEALNet is robust to such cases, achieving comparable performance to the best unimodal model. Upon further inspection of the HEALNet's Omics attention weights on the UCEC task, we found that they barely changed since their initialisation. HEALNet was able to (correctly) inhibit this signal, which is not the case for the other multimodal baselines where it leads to a loss in performance.A key benefit of using iterative attention is that we can skip updates if modalities are missing at inference time without adding additional noise. For many intermediate fusion methods, missing modalities introduce noise since the fusion function f () expects an intermediate representation h m for all modalities. This requires initialising a random array or doing a latent search for a similar array to impute the missing portion. A practical approach to this challenge is a late fusion approach, which requires training and keeping several unimodal alternatives, that can act as a substitution. This, however, can be computationally intensive. HEALNet, on the other hand, overcomes this challenge by design. We believe that this underlines a key benefit of hybrid early-fusion -handling mixed missing modalities, at inference time, which takes advantage of multimodal training, without introducing additional noise.Computational complexity. Another advantage of HEALNet's sequential architecture design is that it scales linearly with respect to both sample size n and the number of modalities m. More specifically, since the cross-attention and self-normalising layers that comprise the fusion layers scale with O(n), each fusion layer has a complexity of O(mn). As a result, the runtime only depends on the number of fusion layers d, which in our setup is a hyperparameter (refer to Appendix D for the hyperparameter values used in this work). Compared to other competitive multimodal baselines, HEALNet scales much more efficiently. For instance, MCAT [Chen et al., 2021] is natively designed for two modalities.Scaling MCAT to m > 2 would require calculating the modality-guided cross-attention for all unique pairwise combinations, resulting in O(m 2 n). This quadratic scaling trend is also similar for other baselines that implement Kronecker product for combining modalities, such as Porpoise [Chen et al., 2022b] and MOTCAT [Xu and Chen, 2023 ]. Note that, in all cases, including HEALNet, the actual runtime will further depend on several additional factors and designs, such as the choice of the modality-specific encoders and the size of their embeddings within the underlying multimodal model.Inspections and explanations. Finally, another design benefit of using attention on the raw input data is that it allows for instance-level insights into the model's behaviour, without the need for additional post-hoc explanation methods. Figure 3 shows what parts of the sample the model attends to on average across layers. For images, one can create a high-level heatmap of the cell tissue to highlight relevant regions for more detailed insights into the tumour microenvironment and disease progression.In turn, these regions can be further analysed post-hoc, such as via nucleus segmentation. To showcase this capability, in Figure 3 , we take the highest attention patches and perform nucleus segmentation Figure 3 : Illustration of model's inspection capabilities using HEALNet on a high-risk patient of the UCEC study. We use the mean modality-specific attention weights across layers to highlight high-risk regions and inspect high-attention omic features. Individual patches can be used for further clinical or computational post-hoc analysis such as nucleus segmentation. We observe that the high-risk regions exhibit a very high concentration and different arrangement of epithelial cells (red) which is commonly associated with the origin of various cancer types [Coradini et al., 2011] .into epithelial cells, lymphocytes, macrophages, and neutrophils using a HoverNet [Graham et al., 2019] pre-trained on the MoNuSAC dataset [Verma et al., 2021] . For the high-risk regions, we observe a higher concentration of epithelial cells that are commonly associated with various cancer origins [Coradini et al., 2011] , which we see as initial validation that the high-risk patches identified by the model capture salient biological signal. 
conclusion: 
 We introduce HEALNet, a flexible hybrid early-fusion approach for multimodal learning. HEALNet has several distinctive and beneficial properties, suitable for applications in biomedical domains: 1) it preserves structural signal of each modality through modality-specific attention, 2) it learns cross-modal interactions due to its iterative architecture, 3) it effectively handles missing modalities, and 4) it enables easy model inspection. The experimental evaluation highlights the importance of fusing data early in the model pipeline to capture the cross-modal signal leading to better overall model performance. While in this work we focus only on survival analysis using modalities from digital pathology and genomic data, we believe that our framework can also be extended to other domains (and modalities) such as radiology or precision oncology, as well as other tasks such as diagnosis or predicting treatment response. 
 
 Appendix: HEALNet is a novel and flexible multimodal approach able to leverage complex heterogeneous biomedical data. Technological advances in medical data collection, such as high-resolution histopathology and high-throughput genomic sequencing, have inspired the development of novel multimodal approaches, able to address a variety of challenging modelling tasks in many biomedical applications. Our work extends the state-of-the-art in this area, introducing a performant architecture with several distinctive and beneficial properties suitable for many clinically-relevant applications. Specifically, we demonstrate the utility of our architecture on four real-world applications related to cancer prognosis using histopathology and multi-omic data collected by The Cancer Genome Atlas (TCGA) consortium.As such, the primary aim and impact of our work is advancing data analysis capabilities in critical domains such as medicine and biology, focusing on complex tasks that require simultaneous modelling of multimodal data. The ability of our architecture to effectively handle missing modalities during inference can support many clinically relevant scenarios. For instance, this includes scenarios where HEALNet has been trained using costly data, but it can still be used to predict outcomes for patients in clinics that do not have access to sophisticated data collection technologies. Moreover, as HEALNet is explainable 'by design', it can enable easy model inspection as well as insights into the cross-modal interactions, which can facilitate trustworthiness and better adoption in such critical domains.To this end, our work has only been evaluated in a strictly research setting. Further applications of our work in scenarios with sensitive data, such as clinical practice, introduce some challenges. As our primary focus is biomedical applications, data privacy must be carefully managed. Furthermore, as HEALNet is intended to serve as a decision-aiding tool, it bears risks of decision bias. Mitigating this involves further extensive evaluations, clinical trials, and medical regulation to ensure its reliability and safety before wider adoption in clinical settings.HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data Output: Logits of survival predictions Ŷlogits ∈ {ŷ (1) , . . . , ŷ(n) }, where ŷ(i) ∈ R, ∀i = 1, . . . , n.1: S 0 ← R c l ×d l // S 0 is the latent array with S 0 [i, j] ∼ U(0, 1) ∀i ∈ {1, . . . , c l }, j ∈ {1, . . . , d l } for latent channels c l and latent dimensions d l . 2: for fusion layer l = 0, . . . , d -1 do 3:S l+j ← F usionU pdate(S l , X m ) // Equation 34: end for 5: Ŷlogits ← LinearHead(S T ) // where total timesteps T = md ∈ N 6: Ŷhazard = sigmoid( Ŷlogits ) m = α(X m , S t ; ϕ am ) // Attention, Equation 17:S t+m = ψ(S t , am ) // Latent Update, Equation 28:S t+m ← SN N (S t+m ) 9: end for 10: return S t+j We found that HEALNet can be sensitive to the choice of regularisation mechanisms, even though the current regularisation techniques, such as L1 + SNN, have shown to be effective.  


Decision: The paper presents a method for fusing multi-modal health data for survival prediction in cancer using an iterative cross-attention structure on raw data modalities.
The method can deal with missing modalities. The reviewers point out that this is important in the real-world health data context, where missing variables is a major problem. The method is evaluated on four data sets. The paper has a deep discussion of related work was considered as very detailed.

Similarity to other approaches in the literature and the limitation to survival analysis are mentioned as the main weaknesses.

Yet, after deep discussion with the authors who provided additional results, three out of four reviewers voted towards acceptance of the paper. The one reviewer voting for a weak reject was also the only one to not respond to the rebuttal discussions on the paper.

Reviews: review: paper_summary: paper_summary: The authors present a multi-modal fusion architecture, named HEALNet, which aims to preserve modality-specific structural information and capture cross-modal interactions in a shared latent space. HEALNet enables intuitive model inspection by learning directly from raw data inputs instead of opaque embeddings. The iterative paradigm can skip modality update steps to handle missing modalities. The authors validate the framework through multi-modal survival analysis on four cancer datasets.
strength_weakness: strength_weakness: 1. HEALNet shares a very similar structure with Perceiver. The authors should highlight the innovative parts more clearly to help readers understand which aspects are novel and which are based on related work.
2. The evaluation is limited to survival analysis. Given that this is a multi-modal fusion framework, it would be beneficial to perform additional tasks, such as classification or regression, to better evaluate its performance.
3. The sample sizes for the four datasets used are relatively small. Instead of 5-fold repeated random subsampling, more repetitions should be conducted, and confidence intervals should be reported to ensure the reliability of the results.
4. In addition to the concordance index, other metrics or plots such as cumulative proportion surviving plots should be reported to better visualize the results.
questions: questions: 1. Higher performance was reported in the compared paper MOTCat [1]. Why did this happen?
2. Since the modalities are different, what is the rationale for sharing weights across all fusion layers in HEALNet? How is modality-specific structural information preserved during the fusion process?
limitations: limitations: 1. The evaluation of HEALNet is limited to survival analysis, which may not fully capture its potential across different biomedical tasks.
2. The small sample sizes and limited number of datasets may affect the generalizability of the results.
3. The absence of a comprehensive set of performance metrics limits the understanding of the model's strengths and weaknesses.
paper_summary: paper_summary: This paper presents HEALNet, a method for end-to-end multimodal fusion of mixed-type biomedical data. In contrast to methods like feature concatenation or Kronecker product fusion, HEALNet employs an iterative cross-attention structure that operates on the raw input modalities, representing a hybrid between early and intermediate-fusion approaches. This allows for the necessary cross-modal interaction, while natively handling missing modalities at test time and aiding interpretability (by operating on raw data rather than embeddings). HEALNet outperforms existing state-of-the-art multimodal fusion methods on survival analysis from histopathology and multi-omic data.
strength_weakness: strength_weakness: - Treatment of hyperparameters is somewhat confusing. It is unclear if Bayesian hyperparameter optimization (HPO) was applied to all methods or just to HEALNet. If HPO was applied to all methods, then why do their hyperparameters not appear in Table 5? These are important details to ensure a fair comparison between methods.
- The latent update in Equation 2 is unclear, and this seems to be a critical component of the proposed method. What *exactly* is this function $\psi(\cdot)$? Also, there is an additional parameter $\rho$ to $\psi(\cdot)$ in the pseudocode that is not present in Equation 2. This needs to be clarified.
questions: questions: - Was Bayesian HPO applied to all methods or only HEALNet? Why does only HEALNet appear in Table 5? Please clarify this and which exact hyperparameters underwent optimization; for instance, it is confusing that all shared hyperparameters appear to be identical except the L1 regularization term.
- What exactly is the update function $\psi(\cdot)$? What is the parameter $\rho$ in the pseudocode, and why is it not present in Equation 2?

Minor comments:
- L18: Unnecessary to use MMML acronym if never used again
- L83: “additive approach to combining modalities (rather than handling them in parallel).” It is unclear what is meant by “additive” – sequential?
- L118: “fusing operators” -> “fusion operators”
- L125: Remove semicolon
- L145: I would write “…Learning Network and its key component, (b) the early fusion layer (as given in Equation 3).” 
- Eq 4: Use \left[ and \right] to make square brackets larger
- L186: Can omit “contains”
- L187: Consider citing https://ieeexplore.ieee.org/abstract/document/10230356/, which addresses multimodal fusion methods to prevent overfitting in multimodal fusion. This is particularly relevant since this paper also considered fusion of histopathology imaging with tabular data, and forms results in contrast to the parameter-intensive Kronecker product as well.
limitations: limitations: Several limitations are adequately addressed in Section 6.
paper_summary: paper_summary: The authors present HEALNet as an early fusion approach for integrating different data modalities. HEALNet utilizes an end-to-end training process with an additive method for combining modalities, rather than handling them in parallel. This strategy enables HEALNet to scale and adapt effectively to datasets of varying sizes and characteristics without requiring explicit pre-training.
strength_weakness: strength_weakness: - The methodology is not clearly written and is difficult to understand.
- The presentation also falls short. For instance, Figure 3 features a bar plot that is unreadable.
- Additionally, the color scheme in the attention map seems unusual; typically, a blue-red gradient is used as it is more intuitive.
- The visualization analysis is lacking. There is only one figure displaying the model's inspection capabilities, which features a WSI attention map.
- The iterative attention mechanism, with multiple steps and updates, can be computationally expensive. This could be particularly challenging when dealing with very high-dimensional data or when scaling up to larger datasets.
- There is no ablation study regarding the number of the fusion layers. The number of fusion layers is a critical parameter that can significantly affect the computational efficiency of the model.
questions: questions: - Could you include metrics such as as Floating Point Operations (FLOPs) and the number of trainable parameters to provide a more comprehensive understanding of the computational cost and efficiency of the HEALNet approach?
- We have seen in recent papers [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://arxiv.org/html/2405.11618v1) that in cases where representations come from different sources or modalities the different  repressentations are aligned in a meaningful way to facilitate effective learning and integration. How does your model handle discrepancies in feature scales or semantic differences between modalities?
limitations: limitations: The methodology needs to be clearer. The dimensions of the cross-attention layers are not clearly defined, which complicates understanding of how these layers align and integrate features from modalities of differing dimensions. Additionally, the visualizations and illustrations require significant improvement, as current figures, such as bar plots and attention maps, are difficult to interpret and hard to read. Enhancing these visual elements and providing clearer explanations of the methodology will improve the quality of the  paper.
paper_summary: paper_summary: The paper proposes an attention based early-fusion method for multi-modal learning. Specifically the paper uses a perceiver style iterative updates to a latent vector using  a fusion layer. The fusion layer sequentially updates the latent vector using cross-attention with each of the modalities. The approach supports cross-modal interactions and provides a way to handle missing modalities at inference time.

The authors compare the approach against existing baselines for fusing biomedical modalities and  show promising results on the survival prediction task.
strength_weakness: strength_weakness: Its unclear how much the iterative fusion is helpful as the parameters are shared. Can the authors provide more some information about when the iterative fusion is useful and how many iterations are typically needed.

The proposed iterative style updates to the latent vector seems similar to an LSTM style aggregation. In many VLM and multimodal papers (https://arxiv.org/pdf/2405.09818, https://arxiv.org/pdf/2304.08485), the modality specific tokens are projected to a shared latent space and are aggregated through self-attention across the shared space. This provides more opportunity for cross modal interactions than iterative cross-attention. Have the authors compared with such approaches?

The results are shown for a specific task (survival prediction) and specific modalities where the datasets are small and models are prone to overfitting. Its unclear if this approach works well for problems which need more complex interactions and where dataset sizes are larger.
questions: questions: Its unclear how the features from different patches in the WSI aggregated? What is the embedding dimension?

Do the early-fusion models have more parameters? If so are they more prone to overfitting on these datasets?

Since the latent array is updated sequentially with information from different modes, how much does the order of fusing modalities impact the result?
limitations: limitations: Given the weaknesses and open questions, I'm basing my decision but would be open to changing the decision post rebuttal once the questions and weaknesses are addressed.

Comments: title: A gentle reminder
comment: Dear AC,

We really appreciate your time and effort in managing our submission.

Even though we addressed all of reviewer `a9ao` questions and concerns, we've noted that they haven't  responded nor acknowledged our rebuttal thus far, even though they showed positive outlook on our work. We believe further interaction can be beneficial, and we are keen to discuss any remaining clarifications in the final hours of the discussion period.

We would appreciate your assistance in reminding the reviewer to respond/acknowledge our rebuttal.

Kind regards, Authors
Thanks for acknowledging our response and your positive outlook - we are glad that we were able to further clarify our work. We would appreciate if you could update your increased positive score, as stated in your last comment, in the system before the deadline of the discussion period.
Dear reviewer `a9ao`,

Thank you again for your thoughtful review and positive attitude toward our work! We have addressed all of your questions and concerns. As the discussion period is drawing to a close, we are keen on getting your feedback.

We are looking forward to your response. Thank you for your time.

Kind regards,
The Authors
I have carefully considered the authors' rebuttal and appreciate the additional clarity provided. As a result, I am updating my score to 5.
Thank you for responding to our rebuttal. In our initial response we briefly discuss the computational complexity of HEALNet, and specifically the role of the fusion layers (as per the last weakness point). We also added new experiments to show that HEALNet is able to scale seamlessly to larger datasets, where the computational cost of the iterative attention mechanism is alleviated by sharing weights across the fusion layers. This makes the number of parameters independent of the number of fusion layers (i.e., iterations). 

More formally, for any multimodal model, it is important to scale with both number of samples $n$ and the number of modalities $m$. For instance, MCAT (our most competitive baseline) is natively designed for two modalities. To scale this approach to $m>2$, one would need to calculate the modality-guided cross-attention for all unique pairwise combinations ${m \choose 2} = \frac{m(m-1)}{2}$ which is $\mathcal{O}(m^2)$. 

A **key advantage of HEALNet’s** sequential setup is that it scales linearly. Since the cross-attention and SNN layers used within the fusion layers scale with $\mathcal{O}(n)$, each fusion layer has a complexity of $\mathcal{O}(mn)$ such that the runtime then mainly depends on the number of fusion layers $d$. Note that, HEALNet we consider $d$ to be a hyper-parameter with the values provided in Appendix E. We plan to post the number of parameters of HEALNet and the baselines on the KIRP dataset later today. 

We acknowledge that the actual runtime will depend on several other factors such as the size of the modality-specific embeddings within each model. As we didn't measure FLOPs at this instance, we provide the BigO analyses of our method with respect to the other baselines. Assuming we use the same encoders for each model, the time complexity for each fusion operation is summarized below.

| Model | Time Complexity with number of modalities $m$ and samples $n$ |
| --- | --- |
| Perceiver (early, concat) | $\mathcal{O}(nm)$  |
| MultiModN (sequential) | $\mathcal{O}(nm)$ |
| MCAT(intermediate) | $\mathcal{O}(m^2n)$ |
| Porpoise (late, Kronecker) | $\mathcal{O}(m^2n)$ |
| HEALNet | $\mathcal{O}(nm)$ |

We hope that this clarified the remaining concerns and will be reflected in your updated score.
Thanks to the authors for addressing my questions.

The method provides a simple way to aggregate information from different modalities without losing their structural information. However its unclear how the results show here translate to other problems.

Based on this I am updating my score from 4 to 5.
I have reviewed the authors' rebuttal and appreciate the clarifying comments. However, my question regarding the inclusion of metrics such as Floating Point Operations (FLOPs) and the number of trainable parameters remains unanswered. So, does my question about the number of iterations that are required. Therefore, I intend to maintain my score.
Dear Reviewers, 

Thank you very much for your time and effort in reviewing our paper and responding to our rebuttal.

We would like to encourage those reviewers who are yet to respond to the rebuttal to engage in discussions for the final stretch of the discussion period. 

Kind regards, 

Authors
Thank you very much for responding to our rebuttal! Per your suggestion, we will further highlight the role of having multiple fusion layers in the manuscript. We briefly wanted to clarify the open points below:

> “If the modalities are complementary its unclear why a concatenation approach doesn't work as well, while it may not handle missing modalities.”

We agree that, in principle, the concatenation approach would work well if all modalities had the same tensor dimensions. However, if we are processing structurally different modalities (e.g., 1D vs 2D vs 3D tensors), one would need to flatten or aggregate the excessive dimensions to ensure shape consistency for the projection. This leads to two problems: 1) we may end up with excessively large 1D tensors which 2) remove spatial signal if we e.g., flatten an image representation. In practice, we cannot always assume shape consistency between our modalities and the presence of all modalities for every sample (especially in clinical practice) - both of these aspects informed our design choices. We touch on these aspects in L89 & L111-114. 

> “Are the tokens from different modalities concatenated and passed through iterative fusion? Wouldn't it have similar paramters to HEALNet then (the Q,K,V projection dims remain similar)”

Contrary to the Perceiver, in HEALNet each modality has its *own* Q,K,V projection (as indicated in Figure 1), which allows us to keep the input data intact irrespective of the tensor dimensions. That is, HEALNet does not require to aggregate excessive dimensions which helps to maintain the structural signal of each modality.
>having multiple fusion layers (i.e., iterations) helps to prevent overfitting

Thanks I think it makes sense that this helps reduce influence of the order of modalities. The choice of weight sharing makes sense with this design, I think its worth highlighting this in the paper.

>Don’t align what shouldn’t be aligned

While I do agree the VL models are aligned, the idea of projecting to a shared representation space in which interactions happen is also being used in the cross-attention interaction, although the iterative fusion does reduce dependency slightly.
If the modalities are complementary its unclear why a concatenation approach doesn't work as well, while it may not handle missing modalities.

Not sure I understand the perceiver early fusion setup well. Are the tokens from different modalities concatenated and passed through iterative fusion? Wouldn't it have similar paramters to HEALNet then (the Q,K,V projection dims remain similar)

> The final embedding dimension is a 2D Vector of size num_patches x encoding_dim
This is indeed a large sequence length.
I acknowledge that I have read the authors' rebuttal and thank them for the clarifying comments. I would encourage the authors to be as specific as possible when justifying hyperparameter choices, even if this material is relegated to the appendix. I will maintain my original score.