HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
Authors: Konstantin Hemker
Nikola Simidjievski
Mateja Jamnik
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal
fusion
computational pathology
multiomic analyses
Primary area: machine_learning_for_healthcare
TL;DR: Flexible method for multimodal fusion on different biomedical data structures

Abstract: Technological advances in medical data collection, such as high-throughput genomic sequencing and digital high-resolution histopathology, have contributed to the rising requirement for multimodal biomedical modelling, specifically for image, tabular and graph data. Most multimodal deep learning approaches use modality-specific architectures that are often trained separately and cannot capture the crucial cross-modal information that motivates the integration of different data sources. This paper presents the **H**ybrid **E**arly-fusion **A**ttention **L**earning **Net**work (HEALNet) – a flexible multimodal fusion architecture, which: a) preserves modality-specific structural information, b) captures the cross-modal interactions and structural information in a shared latent space, c) can effectively handle missing modalities during training and inference, and d) enables intuitive model inspection by learning on the raw data input instead of opaque embeddings. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models, substantially improving over unimodal and multimodal baselines whilst being robust in scenarios with missing modalities. The code is available at https://github.com/konst-int-i/healnet.

HEALNet:
Multimodal Fusion for Heterogeneous Biomedical Data
Konstantin Hemker
Nikola Simidjievski
Department of Computer Science & Technology
PBCI, Department of Oncology University of Cambridge Cambridge, United Kingdom
University of Cambridge
Cambridge, United Kingdom
konstantin, www.aatttiinmmmeeeee...mm www.ehecwl....mmmm..m........ww. cam ac
77779@cam
ac ac.
Mateja Jamnik
Department of Computer Science & Technology
University of Cambridge
Cambridge, United Kingdom
mateja.
jangjjajjmmkkkkl jamnik@cl. cam ac uk
Abstract
Technological advances in medical data collection, such high-throughput ge- nomic sequencing and digital high-resolution histopathology, have contributed to the rising requirement for multimodal biomedical modelling, specifically for image, tabular and graph data. Most multimodal deep learning approaches use modelity-specifie architectures that are often trained separately and cannot cap- ture the crucial cross moosl information that motivates the integration of difffrennt data sources. This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet) flexible multimodal fusion architecture, which: a) pre- serves modality-specifei structural information, b) captures the cross-modal in- teractions and structural information in shared latent space, c) can effectively handle missing modalities during training and inference, and d) enables intuitive model inspection by learning the raw data input instead of opaque embeddings. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data four cancer datasets from The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-thed -frt----tth--ttt performance compared to other end-to-end trained fu- sion models, substantially improving over unimodal and multimodal baselines whilst being robust in scenarios with missing modalities. The code is available at httts:///iitub: http://ggittuub.ccoooonesttt www/konat.....ooooooooooooo.a.oe int-iniiiiii i/healnet
Introduction
A key challenge in Multimodal Machine Learning is multimodal fusion, which is the integration of structurally heterogeneous data into common representation that reduces the dimensionality of the data whilst preserving salient biological signals [Steyaert al. 2023 Fusion approaches well-studied in areas where there a clearly defined shared semantic space, such as audio, visual, and text tasks like visual question answering [Goyal et al., 2016), image captioning (Yu et al. 2020). or multimodal dialogue [Liang el al., 2023 However, healthcare data commonly consists of 2D or 3D images (histopathology and radiology), graphs (molecular data), and tabular data (multi-omics, electronic health records), where cross-modal relationships are typically more opaque and complex, the modalities often do not share semantics, and common representations are less explored. The
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hybrid Early-fusion Attention Learning Network
Fusion Layer
Hitoopahhoocoo cayy
Multi Oml data
K.
V
V
K
V
5,
Cioes Accentteee Sompentt
Croas Arteeeeeeean ten
Eyla tha a
Syta (ToP))
Cooss www.Aceeetitn
Lattet Embeerding
Fuaion Layer
Fallon Layer
Fusioo Layer
Haraedk
Modality-spciffc weights
Shared Weights acrosa fusion layar
W
Figure 1: Overview of HEALNet (Hybrid Early-fusion Attention Learning Network) using a shared and modality-specifee parameter space to leam from structurally different data sources in the same model (Fig. 1A). The shared space is a learned latent embedding S that is iteratively updated through d attention-based fusion layers and captures the shared information between modalities. The hybrid early-fusion layer (Fig. IB, and Eq. 33 learns the cross-attention weights Wm (W(4)) (k) Wh(e) for each modality corresponding to the queries (Qm. w S), keys (Km, = w X,,,, and values (Vn = wi(t) xm.) which are shared between layers. These layers capture the structural information of each modality and encode it in the shared embedding after pass through a self- nolfanormalisng network (SNN) layer.
fusion stage describes how far the multimodal representation is removed from the raw (unimodal) data, and is commonly categorised into early, intermediate, and late fusion (Baltrusaitis et al.! 2019)
Early fusion approaches combine the raw data early the pipeline, which allows the training of single model from all data modalities simultaneously. However, most of these approaches use simple operations such as concatenation, which removes structural information, or take the Kronecker product [Chen et al. 2022a), which can lead to exploding dimensions when applied to multiple modalities and large matrices. Late fusion, on the other hand, trains separate models for each modality at hand, which allows capturing the salient structural information but prevents the model from learning interactions between modalities [Liang et al.) 2023). Intermediate fusion approaches attempt to overcome this trade-off by learning low-level representation (embedding) for each modality before combining them. This can result in discovering the cross-modal interactions whilst taking advantage of each modality's 's internal data structure. The problem with many intermediate fusion approaches is that the learnt latent representation is not interpretable to human experts, and handling missing modalities is often noisy [Cui al., 2023]. To overcome these issues, we posit that there is need for more sophisticated early fusion methods, which we refer to as hybrid early-fusion, that: a) preserve structural information and b) learn cross-modal interactions, and c) work on the raw data, thus allowing for in-model explainability.
In this paper, we propose Hybrid Early-fusion Attention Learning Network (HEALNet, Figure D.. a novel hybrid early-fusion approach that leverages the benefits of early and intermediate fusion approaches, and scales to any number of modalities. The main idea behind HEALNet is use both a shared and modality-specific parameter space in parallel within an iterative attention archi- tecture. Specifically, shared latent bottleneck array is passed through the network and iteratively updated, thus capturing shared information and learning tacit interactions between the data modalities. Mcanwhile, attention weights are learned for each modality and are shared between layers to learn modality-speciffe structural information. We demonstrate the multimodal utility of HEALNet on survival analysis tasks on four cancer sites from The Cancer Genome Atlas (TCGA) data, combining multi-omic (tabular) and histopathology slides (imaging) data. Our results show that HEALNet achieves of-te-of-the-arr concordance Index (c-Index) performance compared to other fusion models on all four cancer datasets for multimodal patient survival prediction. More specifically, HEALNet leads to an average improvement of up to 7% compared to the best unimodal benchmarks and up to 4.5% compared to the best early, intermediate, and late fusion benchmarks, which we see as promising validation of our hybrid early-fusion paradigm. In summary, the contributions of our proposed HEALNet include:

---

Preserving the modality-specieee structure: HEALNet outperforms unimodal tabular (omic) and imaging (histopathology) baselines without dedicated moddalitysppecifi network topology.
Learning cross-modal interactions: HEALNet effectively captures cross-modal information, achiev- ing a significantly higher multimodal uplift compared to existing early, intermediate, and late fusion baselines.
Handling missing modalities: We show that HEALNet effectively handles missing modalities at inference time without introducing further noise to the model, a common problem in the clinical use of multimodal models.
Model inspection: HEALNet is explainable 'by design' since the mopelity-specific attention weights can provide insights about what the model has learned without the need for a separate explanation method. We believe that they are useful for model debugging and validation alongside domain experts.
2
Related Work
In this paper, we focus on multimodal learning problems from biomedical data, where the data modalities are structurally heterogeneous, specifically combining image (e.g.. Whole Slide Imagery (WSI) and tabular (e.g., omic and clinical) data. This aspect is different and more general than approaches that focus on combining homogeneous modalities, such as multi-omic data, where the combined modalities have the same structural formalism (tabular) (Sammut et al. [021] Dai al. 2021]. As such, our work closely relates to several approaches for multimodal data fusion that consider learning from WSI images and genomic data. Namely, Cheerla and Gevaert 2019 supervise pre-training step with a downstream fine-tuning for survival analysis. In the first self- step, a modality-specific embedding is trained and optimised using a similarity loss (similar contrastive learning) between the embeddings. The latter step includes (supervised) fine-tuning of survival model, trained and optimised using the Cox loss. HEALNet, on the other hand, implements a sequential architecture for end-to-end training to learn across modalities without requiring a separate pre-training step. The model architecture is flexible to allow for both training on the raw input data as well as leveraging pre-trained model encoders, which may further improve its performance on particular task of interest.
introduce two-step procedure, combining
HEALNet's multimodal fusion capabilities build on attention architectures (Vaswani al. 2017]. A popular such architecture is the Perceiver (Jaegle et al.) 2021], which uses iterative self- and cross-attention layers and achieves impressive performance across various unimodal tasks. However, this architecture is restricted to single modalities unless inputs are concatenated before training, which may remove salient structural signals. In a multimodal context, cross-attention has been used as a core component for several intermediate and late fusion models. 'Multimodal co-attention' (MCAT) [Chen et al.] 202T] is an attention- based fusion approach that uses tabular modality as the query and the imaging modality as the key and value array train cross-attention unit. Xu and Chen 2023) further extend this concept by introducing a refined co-attention mechanism based on optimal transport, where one modality is used to better contextualise the other. Such co-attention approaches only scale to two modalities since the co-attention units can only take in one set of query-key-valur inputs. It also requires having a 'primary' modality that should be contextualised by a 'secondary'' modality, which may not always be the case.
Chen et al. [202aa] present an intermediate fusion approach that first constructs modality-specinf embeddings before passing them through gating-based attention mechanism, combining the output via a Kronecker product. In turn, the resulting high-dimensional '3D multimodal tensor' is used for a downstream survival prediction task. Similarly, Chen et al. [2022b] propose a late fusion approach which implements modality-speciffi model encoders before combining each model's output via gating mechanism. Here, the attention mechanism is only applied to the imaging modality (WSI), which serves as method for learning more general representations by combing the patch-level latents, but also allows for explainable post-hoc image analysis of the identified regions. An inherent limitation of such an approach is that the encoders are trained separately (without the other modality's context), and consequently, the explanations only account for unimodal information.
Another common limitation of all the above approaches is that they expect fixed tensor dimensions during training and inference. An alternative to this presented with sequential fusion methods such as by Swamy et al. [2023), where modalities can be skipped if not present. Currently, the problem
3

---

with this approach is that it only works on ID tensors and latent states, relying on encoders specific to this architecture.
In contrast, HEAL overcomes many of the limitations of the related fusion approaches. First, its design readily scales to more than two modalities without additional computational overhead beyond the one introduced by the additional modality-specific encoders. Namely, the iterative attention mechanism alleviates the use of fusion operators (such as Kronecker product) that render high- dimensional embeddings, and allows for combining many modalities while preserving the structural information of each. Moreover, HEALNet can learn cross-modal interactions since the modalities are used as mutual context for each other's 's updates. This also allows for more holistic explanations compared to late fusion approaches that use modality-speciff models trained in isolation.
3 HEALNet
Preliminaries. Let Xm represent data from modality = 1, E N. Let Xm E < be either tabular dataset with p features and n samples or an image dataset Xm Rxxwxexn with n images with height h, width and channels The goal of multimodal fusion approach is to learn a fusion function f( such that y = f(X L.... Xj). ). A conventional design of such a system (such as the related work discussed in Section 2 is to first learn modality-speciff function gm (). which learns an intermediate representation hm = 9m (Xm)), and then apply a fusion function f() for predicting the target variable ý f(h,, ..... h2).
Architecture. We depict HEALNet in Figure I Instead of computing h. and applying a single fusion function S(,, HEALNee uses an iterative learning setup. Let L denote step, where the total number of steps = d x j for the number of fusion layers E 0. Let S; represent a latent array shared across modalities, initialised at So where S E Raxb for embedding dimensions b E N and is updated at each step. First, instead of learning an intermediate representation hmm as encoded inputs for X compute the attention weights as
a (S Xm):
for each modality m at each step t. Second, we learn an update function (+) to be applied at each step. The update of 5 with modality m is given by
Se+1m = (S,, a
(2)
for total time steps T and attention function a (Equation [ For parameter efficiency, the final implementation uses weight sharing between layers. Across modalities, each early-fusion layer becomes an update function of the form
Set (1,,,,),..,,;)))
The final function for generating a prediction only takes the final state of the shared array and returns the predictions of the target variable f(ST) as a fully-connected layer.
diepietss a high-level visual representation of this approach, showing: (a) Hybrid Early-fusion Attention Learning Network, and its key component (b) the early fusion layer (as given in Equation B}. We use attention layers since they: a) make fewer assumptions about the input data (c.g., compared to convolutional network), and b) their ability to provide context to the original modality through the cross-attention mechanism. We start by initialising a latent embedding variable, which is iteratively used query into each of the fusion layers, and is updated with information from the different modalities at each layer pass. We chose the iterative attention paradigm due to its highly competitive performance range of unimodal tasks (Jaegle et al. 2021]. Passing the modalities through the shared latent array helps to significantly reduce the dimensionality whilst learning important structural information through the cross-attention layers. The HEALNet pseudocode is detailed further in Appendix A
Preserving structural information. To handle heterogeneous modalities, use modality-specinee
cross-attention layers ax( and their associated attention weights (()) whilst having the latent array S shared between all modalities. Sharing the latent array between modalities allows the model to learn from information across modalities, which is repeatedly passed through the model (Figure [T]. Mcanwhile, the -ppeiit weights between the cross-attention layers (Figure [ focus

---

learning from imputs of different dimensions, as well as learning the implicit structural assumptions of each modality. Specifically, in this work, the employed attention mechanism refers to the original scaled dot product attention from [Vaswani et al. 2017]. with adjustments for tabular and image data.
Formally, given a tabular dataset as matrix Xmm {am
(211 ..... (np)) }. with E N samples and p E P features (e.g., a gene expression), we aim to learn the weight matrices W w and Whi") that act as a linear transformation for S and Xm that form the queries (gm)) (nln), ) keys (kmb) and values (v((²) for each sample passed into the layer. The general scale dot- not-product attention generates attention scores for each feature and can be expressed in Cartesian Notation as
exply, k;,") E, explep . kf))
agp: K) =
Aj [L.N.
In other words, for each channel p and sample an attention layer calculates the normalised and scaled attention weight being given the context of all other features for that sample. This has the benefit that the attention scores are always specific to each input given to the attention layer. From this, we can extract both the normalised attention matrix A as well the context matrix Cp(o, E Ap.i X Ut, which is the attention-weighted version of the original input x. In our case, we need to combine multiple inputs to apply the iterative attention mechanism (i.c., cross-attention) these inputs are the latent S and the input matrix Xm for each modality. To do this, we use the latent array as the query and the input tensor as the keys and values, respectively. Given a latent array S, we define the query for each sample glox) wrel S and the keys and values as km = whth> x and tmm = Wilh' for all samples n E NJ. Intuitively, the iterative cross-attention can be seen aligning the query to each modality individually, but not aligning the modalities themselves to ensure that its unique signal is captured. At each time step, the query for the next update provides context from the other modalities of previous updates.
High-dimensional biomedical data. Attention- based architectures are typically trained on vast datasets (which are commonly available for vision and language tasks). The challenges of working with biomedical data, however, are their high dimensionality whilst often having relatively few samples (i.c., patients). For example, dataset (such as TCGA-BLCA) contains whole slide images of approximately 6.4 gigapixels (80k: x 80k pixels) in its highest resolution and includes thousands of multi-omic features, but only from few hundred patients in total. This leads to two common problems in digital pathology - overfitting [Holste et al. 2023 and high computational complexity.
First, to counteract overfitting, HEALNet implements both L1 and L2 regularisation. Considering the relatively large number of parameters required for the attention layers, we found L1 regularisation to be important. Beyond that, we opted for a self-normalising neural network (SNN) block, due to its proven robustness and regularisation properties [Klambauer et al. 2017]-
Second, handling the extremely high resolution of the whole slide images (WSIs) within computa- tional constraints also a challenge. We address this by extracting non-overlapping 256x256 pixel patches on the 2x and 4x downsampled whole-slide image (-0.5 and 1.0pm per pixel respectively). For comparability with other work, we extract a 2048-dimensions. feature vector for each patch using a standard ResNet50 pre-trained on the Katherl00K dataset, which consists of 100k histopathology images of both healthy tissue and colorectal cancer tissue [Pocock et al. 2022). While HEALNet also achieves competitive results on the raw patch data, this requires more significant downsampling to be computationally feasible at scale.
Handling missing modalities. A common challenge in clinical practice is missing data modalities dur- ing inference. Namely, in practical scenarios, while models have been trained on multiple modalities, there is great chance that not all data modalities are available for predicting the patient's outcome. Therefore, multimodal approaches must be robust to such scenarios. Typical intermediate fusion ap- proaches would need to randomly initialise or impute tensor of the same shape, or sample the latent space for a semantically similar replacement to pass into the fusion function f(h! ...., h²: #) at in- ference, which is likely to introduce noise. In contrast, HEALNet overcomes this issue by design; the iterative paradigm can simply skip a modality update step (Equation at inference time in noise-free manner. Note that these practical benefits also extend to training scenarios, where a (typically small) number of samples are missing some modalities. Rather than imputing this data or completely omit- ting the samples, HEALNet can train and utilise all the available data using the same update principle.
5

---

4 Experiments
Datasets. We empirically evaluate the utility of HEALNet on survival analysis tasks on four cancer datasets from The Cancer Genome Atlas (TCGA). Concretely, we use modalities which are structurally heterogeneous such the ones formalised in a tabular or image dataset. Our tabular data structures consist of three sources: bulk gene expressions (RNAseq), mutations (whole-genome sequencing), and copy number variations. HEALNet treats these three separate modalities, while for the baselines that only support two modalities we had to concatenate them - in continuation of this paper we refer to these as omic modality. The WS/ modality includes H&E-stained whole slide tissue images of the same patient cohorts as in the omic modality. Namely, the four cancer datasets that we include are Muscle-Invasive Bladder Cancer (BLCA, n=436), Breast Invasive Carcinoma (BRCA, n=1021), Cervical Kidney Renal Papillary Cell Carcinoma (KIRP, n=284), and Uterine Corpus Endometrial Carcinoma (UCEC, n=538) (further dataset details are provided in Appendix [Cp These specific sites were chosen based on their sample size (BRCA, BLCA, and UCEC are some of the largest TCGA datasets), performance indicators reported in previous unimodal studies (e.g.. KIRP highest on omic, UCEC highest on WSI only [Chen et al. (022b)), ) and other omic properties BLCA and UCEC are known for their very high gene mutation rate [Ma et al., 2022)) ).
Task setup. We focus on modelling overall survival. For each patient, we are provided with right-censored censorship status (a binary variable on whether the outcome was observed at the end of the study or not) and survival months since data recording. In line with our baselines [Chen al. 2022b], we take non-overlapping quartiles l for the uncensored patients and apply them to the overall population, which assigns categorical survival risk to each patient. For the survival task, our prediction model and the baselines are set to output logits of these survival buckets Hogits = f(Xm: S; p). Using these, we calculate the hazard as the sigmoid of the logits
fhazard le Mlagsts and the corresponding survival as feurvival = II hazards, censorship status and discretised survival label are all used to calculate the negative log-lik iigllikelihood (NLL) loss from the proportional hazards model defined in (Zadeh and Schmid 2021). The concordance index then calculated by comparing all study subject pairs to determine the fraction of pairs in which the predictions and outcomes are concordant [Brentnall and Cuzick 2018
fhazard. The survival,
During development, we found that the proportion of uncensored patients sometimes can be as little as 15% of the cohort (UCEC), Applying the survival bins from such a small sub-sample led to very imbalanced discretised survival on the full cohort. To counteract this, we added the option to apply survival class weighting in the loss function, implemented as the inverse weight of the survival bins. Additionally, note that the NLL loss and the concordance index are sometimes only loosely related. Therefore, our loss weighting helped to stabilise the correlation between the NLL loss and the c-dxx. To ensure fair comparison and comparability with the baselines, we employed the same NLL loss and weighting for training HEALNet. Nevertheless, HEALNet is readily extensible and can be implemented with other survival loss functions such as the Cox loss, which can potentially lead to more stable c-Index results [Cherlla and Gevaert 2019].
Baselines. In all experiments, we compare HEALNet to state-of-the-art uni- and multimodal ap- proaches that utilise different fusion strategies. This includes Porpoise [Chen et al. (0226). which uses late fusion gating method to combine the latent representations learned from normaaioma network (SNN) on the omic modality and an attention-base
multiple-instanee learning network (AMIL) [Chen et al., 2022b). In terms of intermediate fusion, we include MCAT [Chen et al., 2021) and MOTCAT [Xu and Chen 2023], which leverage co-attention between latent representations encoders. Furthermore, we compare to MultiModN [Swamy et al. 2023), sequential fusion approuch for multi-task learning with pre-defined set of encoders. We also include the Perceiver model [Jaegle ct al. 2021], which has shown strong performance on various unimodal tasks through its iterative attention mechanism. Following the original implementation, we benchmark its multimodal integration capabilities using early fusion via concatenation. Finally, for each of our multimodal baselines, we trained unimodal variants as reported in the task-specific papers (Chen et al. 20226 2021]. as well as unimodal models trained with HEALNet using a single modality. We report the best unimodal model out of the set. The full results of all unimodal baselines can be found in Appendix B
resulting from
Implementation details. For each experiment we employ 5-folds of repeated random sub-sampling (Monte Carlo cross-vale validation) with 70-15-15 split for the training, validation and test sets. All reported results show the models' performance on test data that was not used during training or

---

Table 1: Mean and standard deviation of the concordance Index on four survival risk categories. We trained HEALNet and all baselines on four TCGA tasks and report the performance on the test set across five folds. HEALNet outperforms all of multimodal baselines and three out of four unimodal baselines in absolute c-Index performance.
Model
BLCA
BRCA
KIRP
UCEC
Uni-modal (Omics)
0.606 0.019
0.580 + 0.027
0.780 + 0.035
0.550 + 0.026
Uni-modal (WSI)
0.556 = 0.039
0.550 + 0.037
0.533 + 0.099
0.630 + 0.028
Porpoise (Late)
0.620 0.048
0.630
0.040
0.790
0.041
0.590
0.034
MCAT (Intermediate)
0.620 + 0.040
0.589 +: 0.073
0.789 1 0.087
0.589 1 0.062
MOTСAТ (Intermediate)
0.631 = 0.051
0.607 0.069
0.810 + 0.062
0.587 - 0.083
MultiModN (Sequential Fusion)
0.551 = 0.060
0.582 = 0.084
0.753 + 0.152
0.610 0.121
Perceiver (Early Fusion)
0.565 = 0.042
0.566 + 0.068
0.783 + 0.135
0.623 + 0.107
HEALNet (ours)
0.668 0.036
0.638
0.073
0.812 + 0.055
0.626 + 0.037
MOTCH frammadiin)
Uplitt 8e
5
MutMochhSSqaani Tuila))
Perce (Earl Fusion)
0
-10
Figure 2: Mean percentage uplift of all multimodal models compared to the best unimodal baseline. Across all tested TCGA cancer sites, HEALNet's ' hybrid early-fusion paradigm outperforms early, intermediate, and late fusion methods.
validation. We re- re-train all of the baseline models using the code reported in the respective papers. All models have been run under the same circumstances and using the same evaluation framework (includ- ing data splits and loss weighting). For hyperparameter tuning, we Bayesian Hyperparameter search [Bergstra et al. (2013) for all training parameters across models. Model-specift parameters of the baselines were tuned if the optimal parameters the TCGA datasets were not available. The final set of hyperparameters can be found in Appendix D All experiments were run on a single Nvidia A 100 80GB GPU running on Ubuntu 22.04 virtual machine. HEALNet is implemented in the PyTorch framework and available open-source at (tttttss////. /tte::///tttttt www /oost- int- 3/healnet
5 Results
The results of the survival analysis are summarised in Table showing the mean and standard deviation of the c-Index across five cols-validation folds. Across all tested cancer sites, HEALNet outperforms all multimodal baselines. This corresponds to an improvement over the multimodal baselines of approximately 7%, 1%, 3% and 6% on the BLCA, BRCA, KIRP, and UCEC tasks respectively. HEALNet also exhibits more stable behaviour compared to the multimodal baselines, as evident from the standard deviation (across folds), which lower in three out of four datasets.
The unimodal baselines shown in Table T correspond to the hest-performing models from a selection of unimodal baselines we trained (refer the complete list in Appendix B). Compared to the better of the two unimodal baselines, HEALNet achieves an approximately 10% higher c-Index on BLCA and BRCA, 4% higher c-Ixx on KIRP, and a nearly equivalent performance on UCEC. We refer to this as a multimodal uplift, which is illustrated in Figure where we compare the improvement of the different multimodal models and fusion strategies the best unimodal model. Note that the UCEC dataset is an example of modality dominance, where all informative signal stems from one modality (in this case WSI), while the signal from the other modality can be either non-informative or noisy. In- termediate and late fusion approaches, which directly combine modalities, are less robust in such cases. For instance, in the case of Porpoise and MCAT, this can even lead to performance degradation. Since HEALNet is more robust to such noise, it leads to performance comparable to the unimodal variant.
7

---

Table 2: Analysis of the performance of HEALNet, trained on all modalities, in scenarios with missing modalities at inference, compared to unimodal baselines. Each test sample contains only one of either the Omic or WSI modality. The scenarios include test sets consisting of samples with only Omic modality, only WSI modality or combination of both (at random). HEALNet achieves a higher c-Index across datasets, implying effective encoding of cross-modal information and handling different amounts of data with missing modalities.
Test
100% Omics
100%
WSI HEALNet
50% WSI + 50% Omics HEALNet
WSI+Omic HEALNet
Uni-modal
HEALNet
Uni-modal
Uni-modal
BLCA 0.606
0.618
0,487
0.501
0.547
0.612
0.668
BRCA
0,556
0.571
0.529
0.539
0.543
0.541
0.638
KIRP
0.771
0.773
0.518
0.526
0.644
0.714
0.812
UCEC
0,509
0.529
0.558
0.584
0,533
0.580
0.626
To further assess the robustness of HEALNet, evaluate its performance in scenarios with missing modalities. Specifically, using HEALNet trained on four modalities (WSI+ + 3 Omics), we investigate its performance when modalities are missing during inference. Note that half of the test samples include only a WSI modality, while the other half is an Omics modality, chosen randomly. The unimodal baseline corresponds the predictions of the available modality in the same way that late fusion model would use two unimodal models followed by an XOR gating mechanism to make its prediction. For completeness, either Omics or WSI modality, rather than a combination of both. Note that the unimodal baselines HealNet models, trained on a single modality. The results from this analysis, given in Table 2 show that our proposed HEALNet, pre-trained on both modalities, archives stable and generally better performance than a late fusion baseline (as commonly performed in practice).
also report results where the whole test set consists of samples with
6 Discussion
Structure-ppesseviine fusion. The results support our hypothesis that HEALNet is able learn the structural properties of each modality and convert the structural signal into better performance. The quantitative evidence of this is given by HEALNet's absolute c-Index performance across tasks (Table D performing substantially better than the early fusion baseline that employs concatenation. Additionally, HEALNet allows for further qualitative analysis of this behaviour, as we can visualise the sample-level attention of different regions of the whole slide image. The attention maps in Figure 3 show sample where the model identified multiple patches in the same region - distilling the wider image down to local information for which we would typically use convolutional networks.
HEALNet learns about high-level structural relationships by using hybrid of mpdality-ppeeffic and shared parameters. For each modality, we learn attention weights (Equation []] simultancously in an end-to-end process. We believe that this distinguishes our hybrid early-fusion approach from conventional early and intermediate fusion methods. Instead of removing structural information (i.e., concatenation) or creating excessively large input tensors (i.e., Kronecker product), our hybrid early- furioo is able to preserve such structures by design. Similar to intermediate fusion methods, we use a shared latent space to capture cross-modes modal dependencies. However, instead of creating latent space through multiple encodings before combining them via a downstream function (f(,, ((,,,.., hy; Φ)), HEALNet learns an update function (Equation 33 that iteratively updates the information. Nevertheless, a limitation of such an approach
shared latent with modality-specin is akin to training a higher number of parameters (i.c., large attention matrices), but on relatively few samples, making it prone to overfitting. Hence, we found that HEALNet can be sensitive to the choice of regularisation mechanisms, even though the current regularisation techniques (such as LI + SNN) have shown to be effective (see Figure in Appendix B
Cross-modal learning. The motivation of using a hybrid early-fusion over a late fusion approach is to enable the model to learn cross modal interactions that are unavailable to moodality-speciff models trained in isolation. We can see the effect of this in Figure showing HEALNet's substantially higher uplift compared to the late fusion benchmark. Additionally, Table 2 shows that even when a modality is missing, HEAL Net outperforms the best unimodal models that are present, indicating that the multimodal embedding is implicitly inferring some information about the missing modality.
8

---

We note, however, that using multimodal model is not always requirement, especially in the pres- ence of modality dominance, which we see on the UCEC dataset. This dominance can be explained by the relatively high morphological variety that has been found in endometrial carcinomas (i.e., UCEC), which is expressed by spindled, stromal, or extremely large cells [Rabban 2020]. This visual signal can only be picked up by the WSI, which is trend we saw consistently across baselines. Nonetheless, HEALNet is robust to such cases, achieving comparable performance to the best unimodal model. Upon further inspection of the HEALNet's Omics attention weights on the UCEC task, we found that they barely changed since their initialisation. HEALNet was able to (correctly) inhibit this signal, which is not the case for the other multimodal baselines where it leads to loss in performance.
Missing modality handling, A key benefit of using iterative attention is that we can skip updates if modalities are missing at inference time without adding additional noise. For many intermediate fusion methods, missing modalities introduce noise since the fusion function f') expects an intermediate representation hmm for all modalities. This requires initialising random array or doing latent search for similar array to impute the missing portion. practical approach to this challenge is late fusion approach, which requires training and keeping several unimodal alternatives, that can act as a substitution. This, however, can be computationally intensive. HEALNet, on the other hand, overcomes this challenge by design. We believe that this underlines key benefit of hybrid early-fusion - handling mixed missing modalities, at inference time, which takes advantage of multimodal training, without introducing additional noise.
Computational complexity. Another advantage of HEALNet's sequential architecture design is that it scales linearly with respect to both sample size n and the number of modalities m. More specifically, since the cross-attention and self-normalising layers that comprise the fusion layers scale with On)). each fusion layer has a complexity of ((m)). As result, the runtime only depends on the number of fusion layers d, which in our setup is a hyperparameter (refer to Appendix [D for the hyperparameter values used in this work). Compared to other competitive multimodal baselines, HEALNet scales much more efficiently. For instance, MCAT [Chen et al.. 202T is natively designed for two modalities. Scaling MCAT to vn > 2 would require calculating the modality-guide cross-attention for all unique pairwise combinations, resulting in O(m²n). This quadratic scaling trend is also similar for other baselines that implement Kronecker product for combining modalities, such as Porpoise (Chen et al.) 20226] and MOTCAT [uu and Chen 2023). Note that, in all cases, including HEALNet, the actual runtime will further depend on several additional factors and designs, such the choice of the modality-specific encoders and the size of their embeddings within the underlying multimodal model.
Inspections and explanations. Finally, another design benefit of using attention on the raw input data is that it allows for instance-level insights into the model's behaviour, without the need for additional post-hoc explanation methods. Figure 3 shows what parts of the sample the model attends to on average across layers. For images, one can create a high-level heatmap of the cell tissue to highlight relevant regions for more detailed insights into the tumour microenvironment and discase progression. In turn, these regions can be further analysed post-hoc, such as via nucleus segmentation. To showcase this capability, in Figure 3 we take the highest attention patches and perform nucleus segmentation
Whole Slide Image
Attention Map
Post-hoc patch analysis
Omic Feeture Attention
Detasen: UCEC
Highest
tp
Symphooyte
Macrophage
Neurooph
Figure 3: Illustration of model's inspection capabilities using HEALNet on a high-risk patient of the UCEC study. We use the mean modality-specine attention weights across layers to highlight high-risk regions and inspect high-attention omic features. Individual patches can be used for further clinical or computational post-hoc analysis such as nucleus segmentation. We observe that the high-risk regions exhibit a very high concentration and different arrangement of epithelial cells (red) which is commonly associated with the origin of various cancer types [Coradini et al. 2011].

---

into epithelial cells, lymphoeytes, macrophages, and neutrophils using a HoverNet (Graham et al. 2019] pre-trained on the MoNuSAC dataset (Verma et al. 202T). For the high-risk regions, we observe a higher concentration of epithelial cells that are commonly associated with various cancer origins [Coradini et af.) 2011], which we see as initial validation that the high-risk patches identified by the model capture salient biological signal.
7 Conclusion
We introduce HEALNet, flexible hybrid earty-fusion approach for multimodal learning. HEALNet has several distinctive and beneficial properties, suitable for upplications in biomedical domains: 1) it preserves structural signal of each modality through modality-specciff attention, 2) it learns cross-modal interactions due to its iterative architecture, 3) it effectively handles missing modalities, and 4) enables easy model inspection. The experimental evaluation highlights the importance of fusing data early in the model pipeline to capture the cross-modal signal leading to better overall model performance. While in this work we focus only on survival analysis using modalities from digital pathology and genomic data, we believe that our framework can also be extended to other domains (and modalities) such as radiology or precision oncology, as well as other tasks such as diagnosis predicting treatment response.
Broader Impact Statement
HEALNet is a novel and flexible multimodal approach able to leverage complex heterogeneous biomedical data. Technological advances in medical data collection, such as high-resolution histopathology and high-throughput genomic sequencing, have inspired the development of novel multimodal approaches, able to address variety of challenging modelling tasks in many biomedical applications. Our work extends the state-of-the-art this area, introducing a performant architecture with several distinctive and beneficial properties suitable for many dinically-relevant upplications. Specifically, we demonstrate the utility of our architecture on four real-world applications related to cancer prognosis using histopathology and multi-omic data collected by The Cancer Genome Atlas (TCGA) consortium.
As such, the primary aim and impact of our work is advancing data analysis capabilities in critical do- mains such as medicine and biology, focusing on complex tasks that require simultaneous modelling of multimodal data. The ability of our architecture to effectively handle missing modalities during inference can support many clinically relevant scenarios. For instance, this includes scenarios where HEALNet has been trained using costly data, but it can still be used to predict outcomes for patients in clinics that do not have access to sophisticated data collection technologies. Moreover, as HEALNet is explainable 'by design', it can enable easy model inspection well as insights into the cross-modal interactions, which can facilitate trustworthiness and better adoption in such critical domains.
To this end, our work has only been evaluated in a strictly research setting. Further applications of our work in scenarios with sensitive data, such as clinical practice, introduce some challenges. As our primary focus is biomedical applications, data privacy must be carefully managed. Furthermore, as HEALNet is intended to serve as decision-aiding tool, bears risks of decision bias. Mitigating this ves further extensive evaluations, clinical trials, and medical regulation to ensure its reliability and safety before wider adoption in clinical settings.
Acknowledgments and Disclosure of Funding
The authors would like to thank Philip Schouten for his insightful clinical feedback. KH acknowl- edges support from the Gates Cambridge Trust via the Gates Cambridge Scholarship. NS and MJ acknowledge the support of the U.S. Army Medical Research and Development Command of the Department of Defense; through the FY22 Breast Cancer Research Program of the Congression- ally Directed Medical Research Programs, Clinical Research Extension Award GR ANT11769177.. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.
10

---

References
Sandra Steyaert, Marija Pizurica, Divya Nagaraj. Priya Khandelwal, Tina Hernandez- Seaaandde-Buusard, and Olivier Gevaert. Multimodal data fusion for cancer biomarker discovery with deep Iearn- ing. Nature Machine Intelligence. 5(4):351-362, April 2023. ISSN 2522-5839. doi: 10. s42256-023-006333
URL httt:t/ps:/wwe www.w/..tur.... com
Yash Goyal, Akrit Mohapatra, Devi Parikh, and Dhruv Batra. Towards Transparent AI Systems: Interpreting Visual Question Answering Models, September 2016. URL http::/arxxx org/
abs/ 08974 1rXiv:1608.08974 [cs].
Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. Multimodal Transformer With Multi-View Visual Representation for Image Captioning. IEEE Transactions on Circuits and Systems for Video Technology, 30(12):4467-4400 December 2020. ISSN 1558-2205. doi: 10.1109/CSVT.2019 2947482. Conference Name: IEEE Transactions on Circuits and Systems for Video Technology.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Moreney. Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions, February 2023. URL http
//arxiv a229 03430 arXiv:2209.03433 003430 [es].
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis Philippe Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, arXiv:
41(2):423-4433 February 2019. ISSN 19393539. doi: 00.1109/TAAMM2218..2727 1705.09406 Publisher: IEEE Computer Society.
Richard J. Chen, Ming Y. Lu, Jingwen Wang, Drew F. K. Williamson, Scott J. Rodig. Neal Linde- man, and Faisal Mahmood. Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis. IEEE Transactions on Medical Imag- ing, (14):7557-770 April 2022a. ISSN 1558-254X. doi: 20199TM1.2200.30211 Conference Name: IEEE Transactions on Medical Imaging.
Can Cui, Haichun Yang. Yaohong Wang, Shilin Zhao, Zuhayr Asad, Lori A. Coburn, Keith T. Wilson, Bennett A. Landman, and Yuankai Huo. Deep multimodal fusion of image and non- image data in disease diagnosis and prognosis: review. Progress in Biomedical Engineering. 5(2):022001, April 2023. ISSN 1091. doi: 10.18/8/2516- 10911/cc2iccc URL https org/ 10. 1088/25161--2 1091/accfe Publisher: IOP Publishing.
Stephen John Sammut, Mireia Crispin-Ortuce Suet Feung Chin, Elena Provenzano, Helen A. Bardwell, Wenxin Ma, Wei Cope, Ali Dariush, Sarah Jane Dawson Jean E. Abraham, Janet Dunn, Louise Hiller, Jeremy Thomas, David A. Cameron, John M.S. Bartlett, Larry Hayward, Paul Pharoah, Florian Markowetz, Oscar M. Rueda, Helena M. Earl, and Carlos Caldas. Multi- omic machine learning predictor of breast cancer therapy response. Nature 2021 601:7894, 601 (7894):623-629) December 2021. ISSN 1476-4687, doi: 10. 103//41556--21-042278--- URL Publisher: Nature Publishing
htttps://ww..nn www.w.nwwte...mmm com http://rrcceeee/1118--000000 Group.
Yin Dai, Yifan Gao, and Fayu Liu. TransMed: Transformers Advance Multi-Modal Medical Image Classification. Diagnostics, 11(8):1384, August 2021. ISSN 2075-4418. doi: 10.3390/ diagnostics 1081384. URL hhttps://wwt http://wwwwmmpp.... 2owm22055-4411/11/8/15555 Publisher: Multidisciplinary Digital Publishing Institute.
Number:
Anika Cheerla and Olivier Gevaert. Deep learning with multimodal representation for pancancer
prognosis prediction. Bioinformatics, 45(14))444-i4454 July 2019. ISSN 1367-4803, 1460-2059, httpp///mmoooiioooooooiic...ee
doi: 10. 10.1103/hhoomfommacc/bbbb2..2. URL
arriicle/36/14//4////2222
Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. Advances in Neural Information Processing Systems, 30, 2017.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General Perception with Iterative Attention. ICMI,, March 2021, doi: 10.485.0//rxiv. 2103.03206. URL https://arxi /awivv..vv.///////i//aaa///aaava org/ 03206v22 arXiv: 2103.03206.

---

Richard J. Chen, Ming Y. Lu, Wei-Hung Weng, Tiffany Y. Chen, Drew Fk. Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images. In 2021 JEEE/CVE International Conference on Computer Vision (ICCV), pages 4005-4005, Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-66552-812-5. doi: 10. 109/CCC489922.0211 000998. htt.ltttss://eeeepl..0 ieee atg/document/9970077
Yingxue Xu and Hao Chen. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pages 21241-21251, October 2023.
Richard J. Chen, Ming Y. Lu, Drew F.K. Williamson, Tiffany Y. Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, and Faisal Mahmood. Pan-cancer integrative histology-genona analysis via multimodal deep learning. Cancer Cell, 40(8):865- 878.e6, August 2022b. ISSN 1535-6108. doi: Press.
Publisher: Cell
Vinitra Swamy, Malika Satayeva, Jibril Frej. Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, and Many-Anne Hartley. Multimodn-multimodaa multi-task, interpretable modular networks. arXiv preprint arXiv:2309. 14/ 2023.
Gregory Holste, Douwe van der Wal. Hans Pinckaers, Rikiya Yamashita, Akinori Mitani, and Andre Esteva. Improved multimodal fusion for small datasets with auxiliary supervision. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1-5, 2023, doi: 10.1 10.100///113777..0 202....0.........566
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-Normalizing Neu- ral Networks, September 2017. URL URL htttp://axxiv
http:///axxi..oorraa/////// wwtg/a////////ee// 02515 arXiv:1706.02515
[es, stat].
Johnathan Pocock, Simon Graham, Quoe Dang Vu, Mostafa Jahanifar, Srijay Deshpande, Giorgos Hadjigeorghiou, Adam Shephard, Raja Muhammad Saad Bashir, Mohsin Bilai, Wenqi Lu, David Epstein, Fayyaz Minhas, Nasir M. Rajpoot, and Shan E. Ahmed Raza. TIAToolbox as an end-to- end library for advanced tissue image analytics. Communications Medicine, 2(1):1-14, September 2022. ISSN 2730-664X. doi: 10. 01884443866-022-00886-3333 URL https: / /ww. nature. com/ 0ww.ccle///43886--22-0000000
Number: 1 Publisher: Nature Publishing Group.
Fayang Ma, Kyle Laster, and Zigang Dong. The comparison of cancer gene mutation frequencies in Chinese and U.S. patient populations. Nature Communications, 13(1):5651, September 2022. ISSN
2041-1723, doi: 10. 01118/444467--23333333333 111467-022-3355122 Number: Publisher: Nature Publishing Group.
URL/tttps: //www. nature. com/a homm/rtiiles//
Shekoufeh Gorgi Zadeh and Matthias Schmid, Bias Ctoss-Enttopy-Bart -Bsed Training of Deep Survival Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(9): Conference
3126-3137, September 2021. ISSN 1939-3539. doi: 10. 10.1109/TPAMI.CC
Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
Adam R Brentnall and Jack Cuzick. Use of the concordance index for predictors of censored survival data. Statistical Methods in Medical Research, 27(8): 2373, August 2018. ISSN 0962-2802 doi: 10.1177/09228822166888 16680224.. URL http::////6l...0g/00001///////// Publisher: SAGE Publications Ltd STM.
James Bergstra, Daniel Yamins, and David Cox. Making Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures. In Proceedings of the 30th International Conference Machine Leurning, pages 115-123, PMLR, February 2013. URL
httpp:://proceeddngas mlr. http://ss.....r.e.eeeeeeeee Betesss/v2//ergstraall html ISSN: 1938-7228.
Joseph T. Rabban. Distinction of low-grade endometrioid carcinoma in endometrial samplings from high-grade mimics: A practical bastern-based approach. Diagnostic Histopathology, 26(6): 237-247, June 2020. ISSN 1756-2317, 1876-7621. doi: 10. h.016//.mpdpp.2220.oo
Danila Coradini, Claudia Casarsa, and Saro Oriana. Epithelial cell polarity and tumorigenesis: New perspectives for cancer detection and treatment. Acta Pharmacologica Sinica, 32(5):552-564, May 2011. ISSN 1745-7254, doi: 10. 10.1038/aps.2011.22
12

---

Simon Graham, Quoe Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis. 58: 101563, December 2019. ISSN 13618415. doi: 10.1016/j.media.220 101563. URL (tKlitttps///iikiinghhu elsevier com/
Ruchika Verma, Neeraj Kumar, Abhijeet Patil, Nikhil Cherian Kurian, Swapnil Rane, Simon Gra- ham, Quoe Dang Vu, Micke Zwager, Shan E. Ahmed Raza, Nasir Rajpoot, Xiyi Wu, Huai Chen, Yijie Huang, Lisheng Wang, Hyun Jung, G. Thomas Brown, Yanling Liu, Shuolin Liu, Scyed Alireza Fatemi Jahromi, Ali Asghar Khani, Ehsan Montahaei, Mahdieh Soleymani Baghshah, Hamid Behroozi, Pavel Semkin, Alexandr Rassadin, Prasad Dutande, Romil Lodaya, Ujjwal Baid, Bhakti Baheti, Sanjay Talbar, Amirreza Mahbod, Rupert Ecker, Isabella Elllinger, Zhipeng Luo, Bin Dong, Zhengyu Xu, Yuehan Yao, Shuai Lv, Ming Feng, Kele Xu, Hasib Zunair, Abdessamad Ben Hamza, Steven Smiley, Tang-Kai Yin, Qi-Rui Fang, Shikhar Srivastava, Dwarikanath Mahapa- tra, Lubomira Trnavska, Hanyun Zhang, Priya Lakshmi Narayanan, Justin Law, Yinyin Yuan, Abhiroop Tejomay. Aditya Mitkari, Dinesh Koka, Vikas Ramachandra, Lata Kini, and Amit Sethi. MoNuSAC2020: A Multi-Organ Nuclei Segmentation and Classification Challenge. IEEE transactions on medical imaging, 40(12):3413-33 December 2021. ISSN 1558-254X. doi: 10.1109/MM120221300038
Alistair EW Johnson, Tom Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
13

---

Appendix
HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data
A HEALNet Pseudocode
Algorithm HEALNet
Input: Training data Xm {at .......)))) for modalities m {1...., and samples
N (1,...,n]] nn.,
where m, N N.
Input: Survival training labels = ....., for S E {l....,s} survival bins, where S, (,y()) IN
Input: Number of fusion layers d E N.
Output: Logits of survival predictions Yiogite ( n. I....nn.
g(m),, where g() R, 1: So Raxa // So is the latent array with Soli.j] ~ 1) (I,... ah....aajje E (l....,d for latent channels e and latent dimensions dj.
2: for fusion layer = 0...., d do
3: Si+j & d'ussonUpppat (d) Xm) Equation
4: end for
5: Ylogits <- Linear Head( Sr) where total timesteps md
6: Yhazard = sigmoid 9 togits)
7: Loss =
Y Yhozara)
8: return Y Yhazard
Algorithm 2 Hybrid Early-fusion Layer (FusionUpdate)
Input: Latent array S, IR xd at time step E (1....,Th:
Input: Training data Xm = {a] ,...,,? m (i)) for modalities m E {1......} and samples N (1....,, n), where each s² e TRd and m, N,d, E N. corresponds each modality's channel dimensions.
Output: St+j E Raxd
1: for modality
1,....j do
2: Qm +- W(()) Si+m-1 where W:(a) E RR I/ query
3: Km + w Xm where (k) E IR // key
Vm t wil) Xm where w(()) E Rd // value
4:
dam + {Wile) (i(k), w(l")
5:
6:
dm = a(Xms 0mm))) // Attention, Equation
(
7: Stmm = (S,, (1)) Latent Update, Equation
8: St+m < SNN
9:
end for
10: return St+3
14

---

B Experimental Results
While the main paper includes selected experimental results, where we only show the best uni-modal performance, the full list of experimental results can be found below.
Table 3: Full results of mean and standard deviation of the concordance Index on four survival risk categories. We trained HEALNet and all baselines on four The Cancer Genome Atlas (TCGA) tasks and report the performance on the hold-out test set across five cross-validation folds. A selection of these results are presented in Table [
Modalities
Model (+Fusion Type)
BLCA
BRCA
KIRP
UCEC
Purpoise (Late)
0.590 0.043
0.580
0.027
0,780 + 0.035
0.550 + 0.026
MCAT (Intermediate)
0.552 0.036
0.484
0.045
0.721 + 0.084
0.542 + 0.065
Omic
MultiModN (Sequential)
0.575 0.037
0.422
0.070
0.764 + 0.149
0.440 + 0.086
HEALNet (ours)
0.606 0.019
0.556 + 0.035
0.771 + 0.135
0.509 + 0.016
Porpoise (Early)
0.540 + 0.030
0.550 0.037
0.520 + 0.037
0.630 + 0.028
MCAT (Intermediate)
0.556 + 0.039
0.489 0.039
0.533 - 0.099
0.602 = 0.068
WSI
MultiModNN (Sequential)
0.500 + 0.040
0.510 1 0.055
0.484 - 0.102
0.489 - 0.022
HEALNet (ours)
0.487 0.046
0.529 + 0.042
0.518 = 0.123
0.558 = 0.072
Porpoise (Late)
0.620 + 0.048
0.630 + 0.040
0.790 + 0.041
0.590 - 0.034
MCAT (Intermediate)
0.620 + 0.040
0.589 + 0.073
0.789 + 0.087
0,589 + 0.062
Omic WSI
MultiModN (Sequential)
0,551 + 0.060
0.582 + 0.084
0.753 + 0.152
0.610 - 0.121
Perceiver (Farly)
0.565 + 0.042
0.566 + 0.068
0.783 + 0.135
0.623 + 0.107
HEAL.ee (ours)
0.668 + 0.036
0.638 + 0.073
0.812 + 0.055
0,626 + 0.107
Additional experiments
We provide an extended analysis of the performance of HEALNet on combining tabular and time series modalities, evaluated on intensive care data from the Medical Information Mart for Intensive Care (MIMIC-III -III Johnson et al. [2016). We train models for two separate tasks: predicting patient mortality ('MORT'), ). formulated as a multi-class classification task; and disease classification ('ICD-9' codes), which we formulate as a binary classification task. We use both clinical variables and small time series data various vital signs measured at 24 time steps. Both tasks have sample size of n 32616 and the same feature set for different task labels. We report the average AUC in the case of ICD9 and Macro-AUC ("one-vs-rest") in the case of MORT, averaged across five folds.
Table 4: Mean and standard deviation of classification performance. We trained HEALNet and all baselines on two MIMIC-II tasks and report the performance on the test set across five folds. HEALNet outperforms all multi-modal and uni-modal baselines in classification performance.
Model (+Fusion Type)
ICD9
MORT
UniModal (Tabular)
0.73 +0.023
0.658
UniModal (Time Series)
0.700
0.715+0016
Perceiver (Early)
0.733+a008
0.723
MultiModN (Sequential)
0.500
0.500
MCAT (Intermediate)
0.500....000
0.500 +0.000
HEALNet (ours)
0.767 100022
0.748 0.009
15

---

C
Dataset Details
The results shown in this paper here are based upon data generated by the TCGA Research Network: https://www. cancer.. www.gooo/te /tga/ The Cancer Genome Atlas (TCGA) is an open-source genomics program run by the United State National Cancer Institute (NCT) and National Human Genome Research Institute, containing a total of 2.5 petabyts of genomic, epigenomic, transcriptomic, and proteomic data. Over the years, this has been complemented by multiple other data sources such as the whole slide tissue images, which we use in this project. It contains data on 33 different cancer types for over 20,000 patients. Across the four cancer sites the scope of this paper, we process total of 2.5 Terabytes of imaging and omics data, the vast majority of which is taken up by the high-resolution whole slide images. Specifically, the four cancer sites are:
Urothelial Bladder Carcinoma (BLCA): Most common type of bladder cancer, where the carcinoma starts in the urothelial cells lining the inside of the bladder.
Breast Invasive Carcinoma (BRCA): Commonly referred to as invasive breast cancer refers cancer cells that have spread beyond the ducts of lobules into surrounding breast tissue.
Kidney Renal Papillary Cell Carcinoma (KIRP): Type of kidney cancer characterised by the growth of papillae within the tumour which is multi-focal, meaning that they frequently occur in more than location in the kidney.
Uterine Corpus Endometrial Carcinoma (UCEC): Most common type of uterine cancer arising in the endometrium, i.e., the lining of the uterus.
Table 5: Overview data availability and dimensionality of the four TCGA datasets used for experiments.
Property
BLCA
BRCA
KIRP
UCEC
Slide samples
436
1,019
297
566
Omic samples
437
1,022
284
538
Overlap (n used)
436
1,019
284
538
Omic features used
2,191
2,922
1,587
1,421
Sample WSI resolution (px)
79,968
79,653
35,855 x 34,985
72,945 53,994
105,672 x 71,818
Censorship share
53.9%
86.8%
84.5%
85.5%
Survival bin sizes
[72, 109, 172]
[403, 289, 172, 155]
[43, 56, 113, 72]
[68, 143, 83, 244]
Disk space (GB)
883
275
756
16

---

D Hyperparameters
Table Overview of Hyperparameters used in the final code implementation. Parameters that are consistent across datasets can be set in the conf saaafigmmmnnuuu yml and dataset-specific parameters can be set in best.bbyeerparmeterrr yml
Scope
Paramter
BLCA
BRCA
KIRP
Output dims
Patch Size
256
256
256
256
Loss
NLL
NLL
NLL
NLL
Loss weighting
inverse
inverse
inverse
inverse
Subset
uncensored
uncensored
uncensored
Batch Size Epochs
Shared
Early stopping patience
Scheduler
OneCycle LR 0.008
OneCyele LR
OneCycle LR
OneCycle LR
Max
0.008
0.008
0.008
Momentum
0.92
0.92
0,92
0,92
Optimizer
Adam
Adam
Adam
Adam
Ll reg
0.00001
0.000006
0.00004
Layers
Shared Latent dims Attention heads
25 119
25 119
25 119
25 119
8
8
HEALNet
Dims per head
103
Attention dropout
0.08
0.46
0.32
0.25
Feedforward dropout
0.47
0.36
0,05
0.06
Model Size Omic
256x256
256x256
256x256
256x256
Model Size WSI
1024 x 256 256
1024 x 256 256
1024 x 256 256
1024 256 256
MCAT/MOTCAT
Dropout
0.25
0.25
0.25
Fusion Method
bilineur
bilineur
concat
concat
Layers
2
2 126
2
Latent dims
119
62
Perceiver
Attention heads
Attention dropout
0.08
0.46
0.32
0.25
Feedforward dropout
0.47
0.36
0.05
0.06
17

---

E Sensitivity to the regularisation mechanism
We found that HEALNet can be sensitive the choice of regularisation mechanisms, even though the current regularisation techniques, such as L1 SNN, have shown to be effective.
Train Loss
0.25
0.15
0.05
Epoch
10
20 Validation Loss
30
40
50
0.34
0.32
0.28
0.24
10
20
30
40
50
Figure 4: Effect of the regularisation mechanism. We show the train (top) and validation (bottom) losses on the KIRP dataset, of HEALNet variants with no regularisation (blue), only LI regularisation (indigo), and LI regularisation self-normalising network layer (green).
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:[Yes]
Justification:
We provide extensive experimental evidence and discussion that supports our as well as the Supplementary
claims and contributions. Please refer to Sections and material.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed in Section 0
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
provide a pseudo-code of our approach in Appendix A discuss all experimental details in Section including details of the datasets (Appendix C) and hyperparametrs (Appendix) D
Justification: We discuss our approach detail in Section 3
We also provide the code for
reproducing the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The datasets (detailed in Appendix C are a generated by the TCGA Research Network (https: : /wwp//wwww... cancer gov/ /ov/)) and are publicly available. We provide the code for reproducing the results.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yes
Justification: We discuss all experimental settings in Section 4 including details of the datasets (Appendix C and hyperparametrs (Appendix D
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table I (in Section 5) and Table 3 (in Appendix B report mean and standard deviation of the Concordance Index across different folds.
8. Experiments Compute Resources
19

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:[ Yes]
Justification: Reported in 'Implementation details' in Section
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://gww.papppss..ccc/oolllals
Answer: [Yes]
Justification: The research conducted conforms with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer to Section 7
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:[NA]
Justification: [NA]
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We provide the details of our approach as well as the source code. The code available at htttt::// thub. com www/komsss//.eeecc www.:oootttttttteeeooe int 1/healnet:
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No].
Justification: [NA]
20


Reviews:
[{"review_id": "TFpMnkTrXm", "reviewer": "Reviewer_a9ao", "summary": "The authors present a multi-modal fusion architecture, named HEALNet, which aims to preserve modality-specific structural information and capture cross-modal interactions in a shared latent space. HEALNet enables intuitive model inspection by learning directly from raw data inputs instead of opaque embeddings. The iterative paradigm can skip modality update steps to handle missing modalities. The authors validate the framework through multi-modal survival analysis on four cancer datasets.", "questions": "1. Higher performance was reported in the compared paper MOTCat [1]. Why did this happen?\n2. Since the modalities are different, what is the rationale for sharing weights across all fusion layers in HEALNet? How is modality-specific structural information preserved during the fusion process?", "limitations": "1. The evaluation of HEALNet is limited to survival analysis, which may not fully capture its potential across different biomedical tasks.\n2. The small sample sizes and limited number of datasets may affect the generalizability of the results.\n3. The absence of a comprehensive set of performance metrics limits the understanding of the model's strengths and weaknesses.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and clearly written, with a logical flow of information that is easy to follow.\n2. The iterative paradigm's ability to skip modality update steps to handle missing modalities is a practical approach.", "weaknesses": "1. HEALNet shares a very similar structure with Perceiver. The authors should highlight the innovative parts more clearly to help readers understand which aspects are novel and which are based on related work.\n2. The evaluation is limited to survival analysis. Given that this is a multi-modal fusion framework, it would be beneficial to perform additional tasks, such as classification or regression, to better evaluate its performance.\n3. The sample sizes for the four datasets used are relatively small. Instead of 5-fold repeated random subsampling, more repetitions should be conducted, and confidence intervals should be reported to ensure the reliability of the results.\n4. In addition to the concordance index, other metrics or plots such as cumulative proportion surviving plots should be reported to better visualize the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Comments on Weaknesses**\n\n**Clarification on Novelty:** As you rightly point out, the general idea of iterative cross-attention with latent state passing has been previously suggested, which we acknowledge as a starting point for HEALNet’s architecture (L89, L113, L151). While other iterative modelling architectures (such as the Perceiver) handle a range of uni-modal tasks, HEALNet introduces a novel way of how iterative attention can be leveraged for learning effective multi-modal representations. Concretely, the Perceiver architecture requires concatenation of the input modalities. This becomes particularly problematic for modalities with misaligned dimensions and channels, as you either need to flatten all tensors or aggregate across channels, both of which trades off information for shape compatibility. The Perceiver paper demonstrates multimodal capabilities by concatenating audio and video embeddings, reducing the dimensions of the video and treats them as a single modality thereafter — this is identical to the Early Fusion baseline shown in Table~1, which HEALNet outperforms. HEALNet learns modality-specific latent updates to ensure that the modality shapes can be kept intact. Additionally, learning modality-specific updates becomes valuable in the handling of missing modalities. In such scenarios, an early fusion approach like the Perceiver would need to impute the missing values, which is noisy. In contrast, HEALNet’s setup allows to skip the missing updates at train and inference time. \n\n**Classification and Regression tasks:** We agree that the method is general which was a key design objective of the architecture. The focus on survival analysis in this paper was chosen for two main reasons. \n\nFirst, this paper focuses on multimodal data in the context of biomedical tasks. In these domains, most tasks that are a) clinically relevant and b) have sufficient data available to deal with time-to-event labels, where the outcomes of some study subjects is unobserved (right-censored data). Neither classification nor regression on these task labels is suited to handle time-to-event or the concept of censorship accurately. \n\nSecond, it is worth noting that the survival analysis task was chosen to ensure comparability with relevant baselines. Under the hood, the survival analysis implementation is using a classifier to predict the hazard bins of a proportional hazards survival model. \n\nArchitecturally, the only thing that would change to adapt to a classification task is the very final layer (”Head”) in Figure 1A. \n\n**Sample size**: In the field of cancer pathology, The Cancer Genome Atlas (TCGA) contains by far the largest patient cohorts. It is worth noting that given the high dimensions of the whole slide images (see L185), even a single cancer cohort (e.g., TCGA-BLCA) contains over 500GB of data. This is also the case for the benchmarked previously published studies. \n\n**Confidence Intervals:**  We reported standard deviation instead of confidence intervals for two reasons. First, the target variable does not follow a t-distribution ([see this plot](https://filetransfer.io/data-package/djTYYGjo#link)), which is a crucial assumption for the valid interpretation of confidence intervals. As such, reporting standard deviations across the repeated random sub-sampling folds is a more valid estimate of the model’s confidence. Second, using standard deviations was in line with the benchmarked methods, which improves the comparability when directly contrasting the approaches.\n\n**Metrics beyond c-Index:** We agree that Kaplan-Meier plots provide an in-depth representation of the survival probabilities on the test set, but the actual visual differences are often subtle and barely detectible by a naked eye. Therefore, the c-Index coupled with the standard deviation across folds provides a better way of comparing results, as it is also commonly found in the published baselines. \n\n## **Responses to questions**\n\n**Higher performance reported in MOTCat [1] paper:**\n\nTo make the results across experimental setups comparable, we re-ran all experiments under the same experimental conditions and the tunable sets of hyperparameters from the original paper, which we show in Table~1. The different results can be explained by the fact that this paper’s experimental setup is quite different from the one reported in [1]. While [1] predicts disease-specific survival (DSS), HEALNet is trained to predict general survival (GS). General survival does not filter for case deaths unrelated to the disease and is therefore a noisier prediction task. This filtering is evident by the smaller sample sizes on the same cancer cohorts in [1]. Getting higher c-Index results is a general trend we observed in studies predicting DSS instead of GS[2]. \n\nHEALNet follows the same experimental setup, evaluation, and cohort selection as in several multimodal benchmarks (Porpoise, MCAT). \n\n**Rationale for weight sharing**: \n\nWe would would like to clarify two points: First, the weights are only shared between the sets of modality-specific weights across the fusion layer. For example, the cross-attention update for modality 1 and 2 would be shared across all the fusion layers, but no weights are shared between the modality 1 and 2 within or across fusion layers. Second, the weight sharing helps to 1) improve the parameter efficiency of the model and prevent exploding parameter size with a high number of layers and modalities and 2) we found it to prevent overfitting. \n\n[1] Xu, Y., Chen, H., 2023. Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction, in: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV51070.2023.01942\n\n[2] Jaume, G., et al., 2023. Modelling dense multimodal interactions between biological pathways and histology for survival prediction. *arXiv preprint arXiv:2304.06819*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5IgzlCv1sT", "reviewer": "Reviewer_ZYCi", "summary": "This paper presents HEALNet, a method for end-to-end multimodal fusion of mixed-type biomedical data. In contrast to methods like feature concatenation or Kronecker product fusion, HEALNet employs an iterative cross-attention structure that operates on the raw input modalities, representing a hybrid between early and intermediate-fusion approaches. This allows for the necessary cross-modal interaction, while natively handling missing modalities at test time and aiding interpretability (by operating on raw data rather than embeddings). HEALNet outperforms existing state-of-the-art multimodal fusion methods on survival analysis from histopathology and multi-omic data.", "questions": "- Was Bayesian HPO applied to all methods or only HEALNet? Why does only HEALNet appear in Table 5? Please clarify this and which exact hyperparameters underwent optimization; for instance, it is confusing that all shared hyperparameters appear to be identical except the L1 regularization term.\n- What exactly is the update function $\\psi(\\cdot)$? What is the parameter $\\rho$ in the pseudocode, and why is it not present in Equation 2?\n\nMinor comments:\n- L18: Unnecessary to use MMML acronym if never used again\n- L83: “additive approach to combining modalities (rather than handling them in parallel).” It is unclear what is meant by “additive” – sequential?\n- L118: “fusing operators” -> “fusion operators”\n- L125: Remove semicolon\n- L145: I would write “…Learning Network and its key component, (b) the early fusion layer (as given in Equation 3).” \n- Eq 4: Use \\left[ and \\right] to make square brackets larger\n- L186: Can omit “contains”\n- L187: Consider citing https://ieeexplore.ieee.org/abstract/document/10230356/, which addresses multimodal fusion methods to prevent overfitting in multimodal fusion. This is particularly relevant since this paper also considered fusion of histopathology imaging with tabular data, and forms results in contrast to the parameter-intensive Kronecker product as well.", "limitations": "Several limitations are adequately addressed in Section 6.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The problem motivation is very clear, and the explanation of related prior work is thorough. Care is taken to properly inform the reader of relevant background so that the authors can demonstrate why the proposed method is unique.\n- The structure and logical flow of the paper is excellent. Tables and illustrations are of high quality throughout.\n- The proposed method is interesting and is designed to solve domain-specific problems in biomedical multimodal fusion such as handling missing modalities and interpretability, while still enabling powerful cross-modal learning.\n- The experiments appear to be soundly conducted, and results are convincing against competitive relevant baselines.", "weaknesses": "- Treatment of hyperparameters is somewhat confusing. It is unclear if Bayesian hyperparameter optimization (HPO) was applied to all methods or just to HEALNet. If HPO was applied to all methods, then why do their hyperparameters not appear in Table 5? These are important details to ensure a fair comparison between methods.\n- The latent update in Equation 2 is unclear, and this seems to be a critical component of the proposed method. What *exactly* is this function $\\psi(\\cdot)$? Also, there is an additional parameter $\\rho$ to $\\psi(\\cdot)$ in the pseudocode that is not present in Equation 2. This needs to be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Responses to Weaknesses and Questions** \n\n**Hyperparameter optimization:** We would like to clarify that we ran the Bayesian Hyperparameter optimisation for all baselines. The shared parameters in Table 5 were ran equally for all baselines from Table 1. We acknowledge that Table 5 currently does not show the final hyperparameters of the baselines, but we will add these for the final manuscript. Please see the relevant hyperparameters that we tuned for the baselines below. Note that this choice was mostly informed by the original implementations of the respective papers. \n\n- MCAT/MOTCAT/Porpoise:\n    - `model_size_omic` : Embedding size of early layers for the omic data (pre-set choices between ‘large’ and ‘small’ from original papers)\n    - `model_size_wsi` : Same as model_size_omic, but for the WSI embedding\n    - `dropout` : Dropout applied after each layer of original implementation\n    - `fusion_method` : choice between concatenation and bilinear fusion\n- Perceiver (early):\n    - `layers` : total number of iterations\n    - `latent_dims` : dimensions of latent bottleneck\n    - `attention_heads` : number of attention heads\n    - `attn_dropout` : dropout after each self-attention layer\n    - `ff_dropout` : dropout after each feedforward pass\n- MultiModN:\n    - `embedding_dims` : size of latent embedding\n    \n\n**Latent Update in Eq. 2:**  \n\n- **Clarification on $\\psi(\\cdot)$**: In the main body, the update function $\\psi(\\cdot)$ represents the learnable function (i.e., the cross-attention update) instead of a fixed closed-form expression. In the pseudocode in Algorithm 2, given the attention matrix $attn \\in \\mathbb{R}^{b \\times i \\times j}$ for batch size $b$, number of queries $i$, and keys/values $j$, for modality $m$ at time step $t$ and the value matrix $v \\in \\mathbb{R}^{b \\times j \\times d}$ for the dimensions of the modality value vector $d$, the update mechanism is $\\psi(S_t, a^{(t)}_m) = \\sum_j attn_{b,i,j} \\cdot v_{b,j,d}$.  We will make this part clearer in the updated version.\n- **Clarification on $\\rho$:** Thanks for catching this. It is a leftover notation (typo) that we missed in the last version of the manuscript. We will correct this accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A14zDgb1M9", "reviewer": "Reviewer_5LQ9", "summary": "The authors present HEALNet as an early fusion approach for integrating different data modalities. HEALNet utilizes an end-to-end training process with an additive method for combining modalities, rather than handling them in parallel. This strategy enables HEALNet to scale and adapt effectively to datasets of varying sizes and characteristics without requiring explicit pre-training.", "questions": "- Could you include metrics such as as Floating Point Operations (FLOPs) and the number of trainable parameters to provide a more comprehensive understanding of the computational cost and efficiency of the HEALNet approach?\n- We have seen in recent papers [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://arxiv.org/html/2405.11618v1) that in cases where representations come from different sources or modalities the different  repressentations are aligned in a meaningful way to facilitate effective learning and integration. How does your model handle discrepancies in feature scales or semantic differences between modalities?", "limitations": "The methodology needs to be clearer. The dimensions of the cross-attention layers are not clearly defined, which complicates understanding of how these layers align and integrate features from modalities of differing dimensions. Additionally, the visualizations and illustrations require significant improvement, as current figures, such as bar plots and attention maps, are difficult to interpret and hard to read. Enhancing these visual elements and providing clearer explanations of the methodology will improve the quality of the  paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of the authors to implement an additive approach to combine modalities instread of processing them in parallel is promising.", "weaknesses": "- The methodology is not clearly written and is difficult to understand.\n- The presentation also falls short. For instance, Figure 3 features a bar plot that is unreadable.\n- Additionally, the color scheme in the attention map seems unusual; typically, a blue-red gradient is used as it is more intuitive.\n- The visualization analysis is lacking. There is only one figure displaying the model's inspection capabilities, which features a WSI attention map.\n- The iterative attention mechanism, with multiple steps and updates, can be computationally expensive. This could be particularly challenging when dealing with very high-dimensional data or when scaling up to larger datasets.\n- There is no ablation study regarding the number of the fusion layers. The number of fusion layers is a critical parameter that can significantly affect the computational efficiency of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Comments on weaknesses**\n\n- **“methodology is not clearly written and difficult to understand”**: We would like to point to the step-by-step methodology, which is discussed in detail in Section 3, illustrated in Figure 1, and presented as pseudocode in Appendix A. We believe this to be a very detailed description of HEALNet. Nevertheless,  we would be grateful if the reviewer could specify which parts of the methodology are unclear, so we can further improve the clarity of our manuscript.\n- **“[…] Figure 3 features a bar plot that is unreadable”**: We would kindly like to point out that the aim of Figure 3 is to provide a high-level *illustration* of the general inspection capabilities of HEALNet that help users validate the model, as well as understand the results. We believe we mention this in the caption of Figure 3. In this work, Figure 3 is meant to showcase the capabilities of HEALNet, rather than to deep dive into the biology of UCEC. The size/formatting of the Figure 3 is due to page limitations. Still, Figure 3 is sufficiently high resolution that the individual feature names can be read on the PDF version of the paper. Nevertheless, we appreciate the feedback and will provide a larger version of the image in the Appendix.\n- **“iterative attention mechanims […] can be computationally expensive”**: The computational cost of the iterative attention mechanism is alleviated by sharing weights across the fusion layers, which makes the number of parameters independent of the fusion layers. It is worth noting that the existing tasks of the paper are already dealing with very high-dimensional datasets (see L181 onwards). To show that HEALNet also works for datasets with a larger sample size, we ran some additional experiments comparing HEALNet to relevant baselines on a disease classification (ICD9) and patient mortality (MORT) prediction tasks of MIMIC-III (n=32,616). The table below shows the AUC and Macro AUC for the respective tasks. The modalities in MIMIC are tabular electronic health record features and time series on vital signs and test results.\n\n| Model/Dataset | ICD9 (n=32,616) [AUC] | MORT (n=32,616)  [MacroAUC] |\n| --- | --- | --- |\n| Uni-modal (EHR) | 0.731 ± 0.023 | 0.658 ± 0.001 |\n| Uni-modal (Time Series) | 0.700 ± 0.013 | 0.715 ± 0.016 |\n| Perceiver (Early) | 0.733 ± 0.028 | 0.723 ± 0.015 |\n| Porpoise (Late) | 0.628 ± 0.020 | 0.617 ± 0.015 |\n| HEALNet | 0.767 ± 0.022 | 0.748 ± 0.009 |\n- **“[...] number of fusion layers is a critical parameter that can significantly affect the computational efficiency of the model”:** As previously mentioned, the number of fusion layers does not affect the computational efficiency of the model due to the implementation of weight sharing.\n\n### **Responses to questions**  \n\n- **Handling discrepancies in feature scales and semantic differences between modalities:** We would like to point out a crucial distinction between multimodal modelling tasks in the “standard” Vision & Language (V&L) setting and ones over Biomedical modalities (beyond V&L):\n    - Current **Vision & Language** approaches rely on the assumption that the same semantic concepts are expressed within the modalities (e.g., an image and its text description). Moreover, these semantic concepts are typically explicit (e.g., an image with a bird and a description that includes ‘bird’). Therefore, both modalities follow the same training distribution. In turn, V&L approaches align the latent representations to reflect that distribution across both modalities, and leverage these explicitly expressed semantic concepts through a contrastive objectives, such as the one in the linked Tangled paper [1].\n    - For **Biomedical modalities beyond V&L**, we generally cannot make such assumptions. Often times, the signal from different modalities (e.g., morphological features, genomics, and transcriptomics) follows different distributions and we want to maintain that separation to build a more predictive model. If we used projections into the same latent space combined with contrastive objectives, this would likely mute the signal from the separate modalities. This defeats the purpose of cross-modal learning for these modalities as it becomes more difficult to learn from both distributions. It is worth noting that while we separately align the latent representation to each modality, we never align the modality representation with each other to prevent such a muting effect. Moreover, on a broader note, if we were to make a similar assumption as in [1], such an approach would be applicable and scale only to two modalities - which is the case in [1], where the authors learn from WSI and gene expression profiles only. HEALNet, on the other hand, is robust and can readily scale to many different modalities of any size, as shown in the MIMIC experiments above.\n- While we appreciate and thank the reviewer for the reference [1],  we would like to point out that paper [1] was published just three days before the NeurIPS deadline. Nevertheless, we have benchmarked some approaches that perform a projection and alignment in the same latent space. Specifically, the MCAT [2] baseline also aligns both modalities directly through the “genomics-guided cross-attention” where the dot product of the WSIs and the multiomic data is minimised. Another example is the Perceiver (Early Fusion) [3] baseline which is first combining the input modalities to a common tensor representation and then passes them through multiple cross-and self-attention layers. Note that HEALNet consistently outperforms both of these methods, as shown in Table 1 and Section 5.\n    \n    \n\n[1] Jaume et al. (2024) Transcriptomics-guided Slide Representation Learning in Computational Pathology \n\n[2] Chen, R.J. et al. (2021). Multimodal co-attention transformer for survival prediction in gigapixel whole slide images.\n\n[3] Jaegle, A., et al. (2021) Perceiver: General perception with iterative attention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ad7mhkSKRu", "reviewer": "Reviewer_YCoh", "summary": "The paper proposes an attention based early-fusion method for multi-modal learning. Specifically the paper uses a perceiver style iterative updates to a latent vector using  a fusion layer. The fusion layer sequentially updates the latent vector using cross-attention with each of the modalities. The approach supports cross-modal interactions and provides a way to handle missing modalities at inference time.\n\nThe authors compare the approach against existing baselines for fusing biomedical modalities and  show promising results on the survival prediction task.", "questions": "Its unclear how the features from different patches in the WSI aggregated? What is the embedding dimension?\n\nDo the early-fusion models have more parameters? If so are they more prone to overfitting on these datasets?\n\nSince the latent array is updated sequentially with information from different modes, how much does the order of fusing modalities impact the result?", "limitations": "Given the weaknesses and open questions, I'm basing my decision but would be open to changing the decision post rebuttal once the questions and weaknesses are addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and provides good context on existing methods.\n\nThe proposed approach is simple and explained clearly.\n\nThe iterative style fusion is useful as it can support cross-modal interactions without exploding the embedding dimension and can also handle missing modalities at inference time.", "weaknesses": "Its unclear how much the iterative fusion is helpful as the parameters are shared. Can the authors provide more some information about when the iterative fusion is useful and how many iterations are typically needed.\n\nThe proposed iterative style updates to the latent vector seems similar to an LSTM style aggregation. In many VLM and multimodal papers (https://arxiv.org/pdf/2405.09818, https://arxiv.org/pdf/2304.08485), the modality specific tokens are projected to a shared latent space and are aggregated through self-attention across the shared space. This provides more opportunity for cross modal interactions than iterative cross-attention. Have the authors compared with such approaches?\n\nThe results are shown for a specific task (survival prediction) and specific modalities where the datasets are small and models are prone to overfitting. Its unclear if this approach works well for problems which need more complex interactions and where dataset sizes are larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Comments on weaknesses:**\n\n**Usefulness of iterative fusion:** \n\nIn short, having multiple fusion layers (i.e., iterations) helps to prevent overfitting. During development, we found that the modality-specific updates only learn how to operate on the latent embedding $S$ directly after the previous modality update if you only have one fusion layer. Increasing the number of fusion layers means that $S$ is updated with all modalities multiple times such that information from each modality becomes part of the context vector for every other modality. We share the weights between the fusion layers to prevent layer specificity such that the modality-specific updates become context-agnostic. This is why the model also delivers robust performance when skipping a modality at inference time (Table 2). \n\nRegarding the number of fusion layers needed, we ran some initial ablation studies during development, where we did not find a clear pattern on the exact number of fusion layers needed, but found that $d>1$ was an effective regulariser. \n\n**Cross-modal interactions & other architectures:** \n\nUsing the shared latent space projection in a “monolithic” architecture is something that we originally considered and decided against for three main reasons: \n\n- **Don’t align what shouldn’t be aligned:** First, we would like to point out a crucial distinction of multimodal modelling in the Vision & Language (V&L) domain and on biomedical modalities beyond V&L.\n    - **Vision & Language** models rely on the assumption that the same semantic concept is explicitly expressed across modalities (e.g., an image and its description). In other words, both modalities follow the same training distribution and V&L models consequently align the latent representations to reflect this distribution across both modalities, such as through the contrastive objectives in the linked Chameleon paper.\n    - For **Biomedical modalities beyond V&L**, we don’t want to make this assumption. Often, the signal from different modalities (e.g., morphological features, genomics, and transcriptomics) follows different distributions and we want to maintain this separation to build a more predictive model. If we used projections into the same latent space combined with contrastive objectives, this would align the signal from the separate distributions. This defeats the purpose of cross-modal learning for these modalities as it becomes more difficult to learn from both distributions.\n    - **We have benchmarked** several approaches which leverage projection and alignment in the same latent space. The MCAT baseline also aligns both modalities directly through the “genomics-guided cross-attention” where the dot product of the WSIs and the multiomic data is minimised. Another example is the Perceiver (Early Fusion) baseline which is first combining the input modalities to a common tensor representation and then passes them through multiple cross-and self-attention layers. HEALNet **consistently outperforms** these methods (Table 1).\n- **Scaling up to many modalities:** The iterative attention mechanism alleviates the use of fusion operators that can result in high-dimensional latents (e.g., Kronecker product [1]). This allows for combining many  modalities while preserving the structural information of each as well as learning cross-modal interactions.\n- **Improved handling of missing modalities:** Another benefit of an iterative architecture instead of a monolithic one is that each layer pass does not require all modalities at the same time.\n\n**Larger datasets:** \n\nWe would like to point out that for cancer pathology tasks The Cancer Genome Atlas (TCGA) contains by far the largest patient cohorts available in the public domain with each cancer cohort being over 500GB in size (L192-198). \n\nTo show HEALNet’s performance on larger sample sizes, we ran some additional experiments comparing HEALNet to a subset of baselines (due to resource constraints in the rebuttal period) on a disease classification (ICD9) and patient mortality (MORT) prediction tasks of MIMIC-III (n=32,616). The modalities in MIMIC are tabular EHR features and time series on vital signs and test results. The results are shown in the general comment of the rebuttal. \n\n### **Responses to questions** \n\n- **[…]unclear how the features from different patches in the WSI aggregated? […]**\n    - The final embedding dimension is a 2D Vector of size `num_patches x encoding_dim` . The `encoding_dim` refers to the dimensions of the Kather100k pre-trained encoder, which results in a 2048-dimensional feature vector per patch (see L195).\n- **Do the early-fusion models have more parameters? […]**\n    - This highly depends on the early fusion method used. A commonly used early fusion methods is the Kronecker product [1] which is constructing a high-dimensional input tensor, meaning that models using Tensor fusion would require a higher number of parameters in the earlier layers than HEALNet. However, other early fusion models use aggregation methods (e.g., concatenation and averaging across excess channels, as done in the Perceiver). In these cases, the parameters required in the early layers would be lower than HEALNet’s, but notably this approach also collapses an entire dimension, which may remove salient signal.\n- **[…] how much does the order of fusing modalities impact the result?**\n    - During development of HEALNet, we performed preliminary analyses on the influence of the order of modalities on the downstream performance: We did not find any significant difference. Since all modalities become the context of each other’s update as long as $d>1$, the order of the modalities does not matter.\n\nReferences: \n\n[1] Zadeh, A. et al (2017). Tensor fusion network for multimodal sentiment analysis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### **General comment**

We would like to thank all reviewers for their time and insightful comments. We are encouraged that you acknowledge HEALNet as having an excellent structure and logical flow, sound experiments, and an interesting methodological contribution, as well as being clear, thorough, and unique. We also thank the reviewers for picking up on minor comments and typos which we will incorporate in the final manuscript. In the reviewer comments, we provide detailed answers to your questions with additional explanations of the raised issues. 

### **Additional experiments**

We would like to particularly highlight additional experiments that we ran based on the comments around scaling the method to larger datasets (i.e., larger sample size). To address this, we ran HEALNet some relevant baselines on a disease classification (ICD9) and patient mortality (MORT) prediction tasks of MIMIC-III (n=32,616). The table below shows the AUC and Macro AUC for the respective tasks. The modalities in MIMIC are tabular electronic health record features and time series on vital signs and test results. 

| Model/Dataset | ICD9 (n=32,616) [AUC] | MORT (n=32,616)  [MacroAUC] |
| --- | --- | --- |
| Uni-modal (EHR) | 0.731 ± 0.023 | 0.658 ± 0.001 |
| Uni-modal (Time Series) | 0.700 ± 0.013 | 0.715 ± 0.016 |
| Perceiver (Early) | 0.733 ± 0.028 | 0.723 ± 0.015 |
| Porpoise (Late) | 0.628 ± 0.020 | 0.617 ± 0.015 |
| **HEALNet** | **0.767 ± 0.022** | **0.748 ± 0.009** |

We did not have time to run all benchmarks given the time and resource constraints during the rebuttal, but we plan to add the full experimental extension to the final manuscript.

Meta-review: The paper presents a method for fusing multi-modal health data for survival prediction in cancer using an iterative cross-attention structure on raw data modalities.
The method can deal with missing modalities. The reviewers point out that this is important in the real-world health data context, where missing variables is a major problem. The method is evaluated on four data sets. The paper has a deep discussion of related work was considered as very detailed.

Similarity to other approaches in the literature and the limitation to survival analysis are mentioned as the main weaknesses.

Yet, after deep discussion with the authors who provided additional results, three out of four reviewers voted towards acceptance of the paper. The one reviewer voting for a weak reject was also the only one to not respond to the rebuttal discussions on the paper.
Decision: Accept (poster)