MultiMoDN—Multimodal, Multi-Task, Interpretable Modular Networks
Authors: Vinitra Swamy
Malika Satayeva
Jibril Frej
Thierry Bossy
Thijs Vogels
Martin Jaggi
Tanja Käser
Mary-Anne Hartley
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Learning
Multimodal Learning
Multi-task learning
Missingness
Interpretability
TL;DR: An inherently interpretable modular network which sequentially fuses inputs from multiple data modalities while being robust to biased patterns of missingness, evaluated on healthcare, education and meteorology tasks

Abstract: Predicting multiple real-world tasks in a single model often requires a particularly diverse feature space. Multimodal (MM) models aim to extract the synergistic predictive potential of multiple data types to create a shared feature space with aligned semantic meaning across inputs of drastically varying sizes (i.e. images, text, sound). Most current MM architectures fuse these representations in parallel, which not only limits their interpretability but also creates a dependency on modality availability. We present MultiModN, a multimodal, modular network that fuses latent representations in a sequence of any number, combination, or type of modality while providing granular real-time predictive feedback on any number or combination of predictive tasks. MultiModN's composable pipeline is interpretable-by-design, as well as innately multi-task and robust to the fundamental issue of biased missingness. We perform four experiments on several benchmark MM datasets across 10 real-world tasks (predicting medical diagnoses, academic performance, and weather), and show that MultiModN's sequential MM fusion does not compromise performance compared with a baseline of parallel fusion. By simulating the challenging bias of missing not-at-random (MNAR), this work shows that, contrary to MultiModN, parallel fusion baselines erroneously learn MNAR and suffer catastrophic failure when faced with different patterns of MNAR at inference. To the best of our knowledge, this is the first inherently MNAR-resistant approach to MM modeling. In conclusion, MultiModN provides granular insights, robustness, and flexibility without compromising performance.

MultiModN--Mllinndaaa MulkiTTakk,, Interpretable Modular Networks
Vinitra Swamy* EPFL
Malika Satayeva* EPFL
Jibril Frej EPFL
Thierry Bossy EPFL
vinitra.swamy @ epfl.ch
malika.satayevaeeepl..d
jibiil.fre@eepp..co
bieery.bossy@epfll.c
Thijs Vogels EPFL
Martin Jaggi EPFL
Tanja Käser
Mary-Anne Hartley*
EPFL
Yale, EPFL
cojj..oogelsecpf..c
martin.jagg@eepf..cl
tanja.kaser@opfl.ch
mary-anne. waw.aannnhaallcyyylll..o
Abstract
Predicting multiple real-world tasks in single model often requires particularly diverse feature space. Multimodal (MM) models aim to extract the synergistic predictive potential of multiple data types create shared feature space with aligned semantic meaning across inputs of drastically varying sizes (i.e. images, text, sound). Most current MM architectures fuse these representations paral- lel, which not only limits their interpretability but also creates dependency on modality availability. We present MultiModN, multimodal, modular network that fuses latent representations in sequence of any number, combination, or type of modality while providing granular real-time predictive feedback on any number combination of predictive tasks. MultiModN's composable pipeline is interpretable-by -edeeaegg------iinnn.. as well as innately multi-task and robust to the fun- damental issue of biased missingness. We perform four experiments on several benchmark MM datasets across 10 real-world tasks (predicting medical diagnoses, academic performance, and weather), and show that Mul MMldiMMdNN's ' sequential MM fusion does not compromise performance compared with a bascline of parallel fusion. By simulating the challenging bias of missing not-at-randon (MNAR), this work shows that, contrary to MultiModN, parallel fusion baselines erroneously learn MNAR and suffer catastrophic failure when faced with different patterns MNAR at inference. To the best our knowledge, this is the first inherently MNAR-rrsistan approach to MM modeling. In conelusion, MultiModN provides granular insights, robustness, and flexibility without compromising performance.
Introduction
The world is richly multimodal and intelligent decision-making requires integrated understanding of diverse environmental signals, known as embodied intelligence ①1. Until recently. advances deep learning have been mostly compartmentalized as computer vision for images, natural language processing for text, and so on. Multimodal (MM) learning has emerged from the need to address real-world tasks that cannot be robustly represented by a single signal type as well as the growing availability and diversity of digitized signals 2 4
data modality, creating disembodied domains such
Some examples sentiment from facial expression, text, and sound 8 9 10 a combination of sensors 12
diagnosis from combination of medical tests and imagery (5] 6 77. estimating
and identifying human activities from
denotes equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

TASKS: Any number or combination tosks IT) can be decoded
cachtaayy step providing fine-g grained, updatable inturpretabillly
tarks
module;
module,
Single-tack Decoder
11
drr
d
dl
d
dTTl
Halt--takk
d
[D] #
Soj
Sj
M
exchnumber d comb nutional inpuis
emby
emb
smbhyj
umbeddinn
Huttinodal Eiseeders rumbet)
1
e e|M]
Muttineodal Receders
11
IMJ
MODALITIES: Any number comb nation of different modalitees [M] Image, sound, toc... can be
Messing skpors
i Musicg
mod:
mod
mods
mod
added in composable
sequential pipelie compr sing available imputs
TXT
(a) MultiModN
(b) P-Pusion
Figure 1: Comparison of modular MultiModN (a) vs. monolithic P-Fusion (b). MultiModN inputs any number/combination of modalities (mod) into a sequence of mod-specific encoders (e). It can skip over missing modalities. A state (s) is passed to the subsequent encoder and updated. Each state can be fed into any number/combination of decoders (d) to predict multiple tasks. Modules identified grey blocks comprising an encoder, state, and set of decoders. P-Fusior on is a monolithic model. It inputs a fixed number/combination of modalities (mod) into mod-specific encoders (e). Missing modalities are padded and encoded. Embeddings (emb) concatenated and provided to a single decoder in parallel (d) to predict a single task.
The richer representations from synergistic data types also have the potential increase the task space, where single set of representations can generalize to several tasks. Multi-task (MT) learning has not only been shown benefit the performance of individual tasks but also has the potential to greatly reduce computational cost through shared feature extraction [33.
In short, multimodal and multi-task learning hold significant potential for centanccentric machine learning and can be summarized respectively as creating a shared feature space from various data types and deriving their relative semantic meaning across several tasks.
Limitations of current multimodal models. Current MM models propose a parallel integration of modalities, where representations are fused and processed simultaneously 2 B 4 Parallel fusion (hereafter P-Fusi on) creates several fundamental limitations that we address in this work.
The most important issue we seek to resolve in current MM architectures is their dependence on modality availability where all modalities for all data points are required inputs during both training and inference. Modallttysppeciff missingness is common real-world problem and can fundamentally bias the model when the missingness of a modality predictive of the label (known as missing not-at- random, MNAR). The common solution of restricting learning to data points with a complete set of modalities creates models that perform inequitably in populations with fewer available resources (i.e. when the pattern of MNAR is different between train and test sets). In complex real-world datasets, there is often no intersection of complete availability, thus necessitating the exclusion of modalities or significantly limiting the train set. On the other hand, imputation explicitly featurizes missingness, thus risking to create a trivial model that uses the presence of features rather than their value for prediction [4 The MNAR issue is particularly common in medicine, where modality acquisition is dependent on the decision of the hcalthcare worker (i.e. the decision that the model is usually attempting to emulate). For example, patient with a less severe form of a discase may have less intensive monitoring and advanced imagery unavailable. If the goal is to predict prognosis, the model could use the missingness of test rather than its value. This is fundamental flaw and can lead to catastrophic failure in situations where the modality is not available for independent reasons (for instance resource limitations). Here, the featurized missingness would inappropriately categorize the patient in a lower severity class. For equitable real-world predictions, is critical to adapt predictions available resources, and thus allow composability of inputs inference.
Another key issue current techniques that this work addresses is model complexity. Parallel fusion of various input types into single vector make many post-hoc interpretability techniques difficult

---

or impossible [16]. Depending where the fusion occurs, it may be impossible to decompose modality-specife predictive importance.
In this work, we leverage network modularization, compartmentaliizng each modality and task into independent encoder and decoder modules that inherently robust to the bias of MNAR and can he assembled in any combination or number at inference while providing continuous modality-specine predictive feedback.
Contributions. We propose MultiModN, a multimodal extension of the work of Trottet et al. [7], which uses flexible sequence of model and task-agnostic encoders produce evolving latent representation that can be queried by any number or combination of multi-task, model-agnostic decoder modules after each input (showcased in Figure J. Specifically, demonstrate that our modular approach of sequential MM fusion:
[1] matches parallel MM fusion (P-Fusion) for a range of real-world tasks across several bench- mark datasets, while contributing distinct advantages, such as being:
[2] composable at inference, allowing selection of any number or combination of available inputs,
[3] is robust to the bias of missing not-at-random (MNAR) modalities,
[4] is inherently interpretable, providing granular modality-specific predictive feedback, and
[5] is easily extended to any number or combination of tasks.
We provide an application-agnosti open-source framework for the implementation of Mul tiModN: https://gis thub. com/ coom/ppfl-10 wiggobaaleellth/W.M. limits our model performance to fairly compare the multimodal fusion step. At equivalent perfor- mance, our model architecture is by far superior to the baseline by virtue of being inherently modular, interpretable, composable, robust to systematic missingness, and multi-task.
Our experimental setup purposely
2 Background
Approaches to MM learning can be categorized by the depth of the model at which the shared feature space is created (2). Late fusion (decision fusion) processes inputs in separate sub-networks, only combining the outputs at the decision-level, using separate model aggregation technique to make a final prediction. While simple, late fusion fails to capture relationships between modalities and is thus not traly multimodal. Early fusion (feature fusion), combines modalities at the input level, allowing the model to learn a joint representation. Concatenating feature vectors is popular and simple approach [18 [1], but the scale of deployment is particularly limited by the curse of dimensionality. Finally, intermediate fusion (model fusion) seeks to fine-tune several feature extraction networks from the parameters of downstream classifier.
Parallel Multimodal Fusion (P-Fusion). Recently, Soenksen et al. 20 proposed a fusion architecture which demonstrated the utility of multiple modalities in the popular MM medical benchmark dataset, MIMIC 21 22). Their framework (HAIM or Holistic Artificial Intelligence in Medicine) generates single modlltty embeddings, which are concatenated into single one- dimensional multimodal fusion embedding. The fused embedding is then fed to a single-task classifier. This work robustly demonstrated the value of multimodality across several tasks and rich combination of heterogeneous sources. HAIM consistently achieved an average improvement of 6-33% AUROC (area under the receiver operating characteristic curve) across all tasks in comparison to single-modality models. We use this approach as a P-Fusion baseline against our sequential fusion approach of MModN and extend it to several new benchmark datasets and multiple tasks.
Soenksen et al. [20 perform over 14.324 experiments on 12 binary classification tasks using every number and combination of modalities. This extreme number of experiments, was necessary because the model is not composable nor capable multi-task (MT) predictions. Rather, a different model is needed for each task and every combination of inputs for each task. In contrast, MultiModi is an extendable network, to which any number of encoders and decoders can be added. Thus, most of the 14,324 experiments could technically be achieved within one MultiMode model.
Several other recent architectures utilize parallel fusion with transformers. UNiT (Unified Trans- former) [23 is promising multimodal, multi-task transformer architecture; however, it remains monolithic, trained on the union of all inputs (padded when missing) fed in parallel. This not only exposes the model to patterns of systematic missingness during training but also reduces model
3

---

interpretability and portability 24 s recent work has found similar results on the erratic behavior of transformers to missing modalities, although only tested on visual/text inputs. LLMs have also recently been used to encode visual and text modalities [25] but it is not clear how tabular and time-series would be handled or how this would affect the context window at inference. Combining predictive tasks with LLMs will also greatly impact interpretabilityy. introducing hallucinations and complex predictive contamination where learned textual bias can influence outcomes.
Modular Sequential Multimodal Fusion. A module of a modular model is defined as a self- contained computational unit that can be isolated, removed, added, substituted, or ported. is also desirable for modules to be order invariant and idempotent, where multiple additions of the same module have no additive effect. We design MultiModN to encode individual inputs, whereby module- exclusion can function as input skippablity, allowing missing inputs to be skipped without influencing predictions. Thus, modular models can have various input granularities, training strategies, and aggregation functions. Some popular configurations range from hierarchies with shared layers to ensemble predictions and teacher-trainer transfer learning approaches [26 27
We expand on the sequential modular network architecture proposed by Trottet et al. [17 called MoDN (Modular Decision Networks) as means of sequential MM fusion. MoDN trains series of feature-specific encoder modules that produce a latent representation of a certain size (the state). Modules can be strung together in a max--nd-maad an input into the next. Therefore, the state has an additive evolution with each selected encoder. A series of decoders can query the state at any point for multiple tasks from various combinations of inputs, giving MoDN the property of combinatorial generalization.
sequence by feeding the state of one encoder as
Thus, we extend MoDN learn multiple tasks from multimodal inputs. By aligning feature extraction pipelines between MultiModh and the P-Fusion baseline (inspired by HAIM) we can achieve a better understanding of the impact of monolithic- parrallll fusion vs. sequential-moduu MM fusion. Figure I provides a comparison between P- Fussion and MultiModN, also formalized below.
Problem formulation
Context. We propose multi-task supervised learning framework able to handle any number or combination of inputs of varying dimension, irrespective of underlying bias in the availability of these inputs during training. We modularize the framework such that each input and task is handled by distinct encoder and decoder modules. The inputs represent various data modalities (i.e. image, sound, text, time-series, tabular, etc.). We assume that these inputs have synergistic predictive potential for a given target and that creating a multimodal shared feature space will thus improve model performance. The tasks represent semantically related observations. We hypothesize that jointly training on semantically related tasks will inform the predictions of each individual task.
Notation. Formally, given a set of modalities (features) M = {modd,....... mod, Mi and set of tasks (targets) T = beusk,,...,,sshhl, ). let X' = {(11, (12, (2). (I... (NN) represent a multimodal, multi-task dataset with N data points - WNN)
Each point x has |M| modalities (inputs): x = (mmodd ....; Imod mi } and is associated with a set of |T] targets (tasks): from x-rays, CT), for simplicity, we consider sources and modalities as equal mod elements in M.
{yeask...... Vtask / }. Modalities comprise various sources (e.g. images
Multimodal, multi-task, modular formulation. We decompose each data point x into [M| sequen- tial encoder modules specific to its constituent modalities and each target into [T] decoder modules specific to its constituent tasks such that any combination number of modalities can be used to predict any combination or number of tasks. Our objective is to learn a set of function modules, F. Each function module within this set, represented as fj E F maps combinations of modalities M, to combinations of tasks Tr, i.e. f M, - T. It is important to note that Mj an element of the powerset of all modalities and 7; is an element of the powerset of all tasks.
Extension to time-series. In the above formulation, the M encoder modules are handled in sequence, thus naturally aligning inputs with time-series. While the formulation does not change for time-series data, it may be optimized such that ʃj represents single time step. This is relevant the roal-world
'The equivalent transformer architecture has 427,421 trainable parameters for the EDU dataset (Sec. 5 while MultiMode achieves better performance with 12,159 parameters.

---

setting of a data stream, where inference takes place at the same time as data is being received (i.e. predicting student performance at each week of course as the course is being conducted). The continuous prediction tasks (shown for EDU and Weather in Sec. (6) demonstrate how MultiModi can be used for incremental time-series prediction.
4 MultiModN: Multimodal, Multi-task, Modular Networks (Our model)
Building on 17 (summarized and color-coded in Figure Tap. the MultiModN architecture consists of three modular elements: a set of State vectors S == (S....... ) set of modality
Encoders & fei,.... eIMI and set of task-specific Decoders D { {di, (di,.... d State so is randomly initialized and then updated sequentially by e; to 8t. Each 8; can be decoded by one, any combination, all elements of DD to make set of predictions. All encoder and decoder parameters are subject to training.
States (S). Akin to hidden state representations in Recurrent Neural Networks (RNNs), the state of MultiModN is a vector that encodes information about the previous inputs. As opposed to RNNs, state updates are made by any number combination of modular, modality-specific encoders and each state can be decoded by modular, task-specific decoders. Thus the maximum number of states by any permutation of encoders is n!. For simplicity, we limit the combinatorial number of states to single order (whereby e; should be deployed before 6i4 1) in which any number or combination of encoders may be deployed (i.e. one or several encoders can be skipped any point) long as the order is respected. Thus, the number of possible states for given sample is equal to 2 MI, Order invariance could be achieved by training every permutation of encoders |M]. i.e. allowing encoders to be used in any order at inference, as opposed to this simplified implementation of MultiModN in which order is fixed. At each step i, the encoder e; processes the previous state, 8;-1. as an input and outputs an updated state 8; of the same size. When dealing with series, we denote si(0) as the state representing time before any modalities have been encoded, and as $,(0.1,4,5) as the state at time/ after being updated by encoders et, 64 and es, in that order.
Encoders (E). Encoders are modularized to represent a single modality, i.c. |E| = |.M]. An encoder e, takes input the combination of single modality (of any dimension) and the previous state 5;-1- Encoder e, then outputs a s;, updated with the new modality. For simplicity, we fix the state size between encoders. Due modularization, MultiModN is model-agnostic, whereby encoders can be of any type of architecture (i.e. Dense layers, LSTM, CNN). For experimental simplicity, we use a single encoder design with simple dense layer architecture. The input vectors in our experiments are ID. When a modality is missing, the encoder is skipped and not trained (depicted in Figure []].
Decoders (D). Decoders take any state as input and output prediction. Each decoder is assigned to a single task, that [D] = [T], i.e. MultiModN is not multiclass, but multi-task (although single task may be multiclass). Decoders are also model-agnostic. Our implementation has regression, binary, and multiclass decoders across static targets or changing time-series targets. Decoder parameters shared across the different modalities. The decoder predictions are combined across modalitiessmodules by averaging the loss. Interestingly, a weighted loss scheme could force the model to emphasize certain tasks over others.
As shown in 17. MultiModN can be completely arder-invariant and idempotent if randomized during training. For interpretability, sequential inference (in any order) is superior to parallel input due to its decomposability, allowing the user to visualize the effect of each input and aligning with Bayesian reasoning.
Quantification of Modularity. The modularity of a network can be quantified, whereby neurons are represented by nodes (vertices) and connections between neurons as edges. There are thus comparatively dense connections (edges) within a module and sparse connections between them. Partitioning modules is an No-coepeete problem (28]. We present modules that are defined a priori, whereby a module comprises one encoder e; connected to one state 8;, which is in turn connected to set of [T] tasks (a module is depicted as grey box in Figure Tap. Following a formalization of modularity quantitation proposed by Newman et al. [29], we compute the modularity score for Mult t MiMMNN and show that it tends to perfect modularity score of 1 with each added modality and each added task. When viewed at the network granularity of these core elements, P-Fusion on is seen
as monolithic model with a score of 0. The formula is elaborated in Appendix Sec. B
5

---

4.1 P-Fusion: Parallel Multimodal Fusion (Baseline)
We compare our results to a recent multimodal architecture inspired by HAIM (Holistic Al in Medicine) [20]. As depicted in Figure Tb. HAIM also comprises three main elements, namely, fixed set of modality-specifff encoders & = f
which create a fixed set of embeddings B (embl,...,mbbb,x that is concatenated and fed into a single-task decoder (d1). HAIM achieved state-of-the-a results on the popular and challenging benchmark MIMIC dataset, showing consistently that multimodal predictions were between 6% and 33% better than single modalities.
Encoders (E). Contrary to the flexible and composable nature of MultiModN, the sequence of encoders in P- P-Fusion is fixed and represents unique combination of modalities. It is thus unable to skip over modalities that are missing, instead padding with neutral values and explicitly embedding the non-missing modalities. The encoders are modality-specifer pre-trained neural networks.
Embeddings (B), Multimodal embeddings are fused in parallel by concatenation.
Decoders (D). Concatenated embeddings are passed single-task decoder.
Architecture alignment. We align feature extraction between MultiModN and P-Fusion to best isolate the effect of sequential (MultiModN)
parallel (P- (P-Fusion) fusion. As depicted in Appendix Figure 8 we let MultiModN take as input the embeddings created by the P-Fusion pre-trained encoders. Thus both models have identical feature extraction pipelines. No element of the MultiModN pipeline proposed in Figure ia changed. The remaining encoders and decoders in both models simple dense layer networks (two fully connected ReLU layers and one layer for prediction). Importantly, MultiModN encoders and decoders are model- -ageos-a and can be of any architecture.
5 Datasets
We compare MultiModN and P-Fusion three popular multimodal benchmark datasets across 10 real-world tasks spanning three distinct domains (healthcare, education, metcorology). The healtheare dataset (MIMIC) is particularly challenging in terms of multimodal complexity, incorporating imputs of vastly varying dimensionality. Education (EDU) and Wcather2k have a particular focus on time-series across modalities. Appendix Sec. details features, preprocessing, and tasks (taskı-1o).
MIMIC. MIMIC 30 is set of deidentified electronic medical records comprising over 40, 000 crit- ical care patients at large tertiary care hospital in Boston. The feature extraction pipeline replicated according to our baseline of P-Fusion (20), making use of petient-level feature embeddings extracted from pre-trained models as depicted in Appendix Figure 8 We select the subcohort of 921 patients who have valid labels for both diagnoses and all four modalities present. We use all four modalities inputs: chest x-rays (image), chart events (time-scries), demographic information (tabular), and cchocardiogram notes (text). For simplicity, we focus on two diagnostic binary classification tasks: cardiomegaly (taski) and enlarged cardiomediastinum (fask2). These tasks were selected for their semantic relationship and also because they were reported to benefit from multimodality (20]. Thus, we have four modality-spenn -peciific encoders and two binary classification diagnostic decoders.
Education (EDU). This educational time-series dataset comprises 5, 611 students with over 1 million interactions in 10-week Massively Open Online Course (MOOC), provided to globally diverse population. is benchmarked several recent works 31
32 33 Our modeling setting is
replicated from related literature, with 45 handcrafted time-series features regarding problem and
video modalities extracted for all students at each weekly time-step [34]. We use two modality- specific encoders (problem and video) and three popular decoder targets: binary classifiers (task:-1) of pass/fail and drop-out, and a continuous target of next week's 's performance (task;) [35).
Weather2k. Weather2k is 2023 benchmark dataset that combines tabular and time-series modalities for weather forecasting [6). The data is extracted from 1, 1686 ground weather stations covering 6 million km2, with 20 features representing hourly interactions with meteorological measurements and three static features representing the geographical location of the station. We create five encoders from different source modalities: geographic (static), air, wind, land, and rain and align with the benchmark prediction targets (6) on five continuous regression targets: short (24 hr), medium (72 hr), long term (720 hr) temperature forecasting, relative humidity and visibility prediction (faskso-10).

---

MARRRAA HASIN TARE Toa Karro Proohaly
Tal 1. Tullio
Phuses Taal lesd
THE
All
Baliel bangleten)) Hear Fac 315334)
HuldP
002
Pabee
Fake
Pake pasdicc
(a) MIMIC
(b) EDU
(c) Weather
Figure 2: MultiModN does not compromise performance in single-tasks. AUROC for six binary prediction tasks in (a) MIMIC, (b) EDU, and (c) Weather2k. Tasks predicted by P- P-Fusion are compared with MultiModN, 95% CIs are shaded.
Experiments
Overview. We align feature extraction pipelines between MultiModN and the P-Fusion baseline in order to isolate the impact of parallel-monolitic vs. sequential depicted in Appendix Sec. B). We thus do not expect significant difference in performance, but rather aim to showcase the distinct benefits that can be achieved with modular sequential multimodal fusion without compromising baseline performance. In the following subsections, we perform four experiments to show these advantages. [1] Mult MMoNN performance is not compromised compared to P-Fusion in single-task predictions. [2] Multi iModN is able to extend multiple tasks, also without compromising performance. [3] MultiModN is inherently composable and interpretable, providing modality-specific predictive feedback. [4]
fusion (described in an and
is resistant to MNAR bias and avoids catastrophic failure when missingness patterns are different between train and test settings.
Model evaluation and metries. All results represent a distribution of performance estimates model trained 5 times with different random weight initializations for the state vector and weights. Each estimate uses completely independent test set from an 10-10-10 K-Fold crai-test-validdtiio split, stratified on one or more of the prediction targets. We report metrics (macro AUC, BAC, MSE) with 95% confidence intervals, aligned with domain specifcc literature of each dataset 34 20 36).
Hyperparameter selection. Model architectures were selected among the following hyperparame- ters: state representation sizes [1, 5, 10, 20, 50, 100], batch sizes [8, 16, 32, 64, 128], hidden features [16, 32, 64, 128], dropout [0, 0.1, 0.2, 0.3], and attention [0, 1]. These values were grouped into while keeping the others fixed (within groups). Appendix Figure 9 shows that MultiModN robust to changing batch size, while dropout rate and hidden layers negatively impact larger models (possibly overlitting). The most specific parameter to MultiModN state size. As expected, we see negative impacts at size extremes, where small states likely struggle to transfer features between steps, while larger ones would be prone overfitting.
3 categories (small, medium, large). We vary
6.1 Exp. 1: Sequential modularization in Mult iModN does not compromise performance
Setup. A single-task model was created for each laski across all three datasets. Fach model takes all modalities as input. We compare MultiModN and P-Fusion in terms of performance. AUROCs can be visualized in Figure 2 while BAC and MSE are detailed in Table As feature extraction pipelines between MultiModN and P-Fusion are aligned, this experiment seeks to investigate sequential modular fusion compromises model performance. To compress the multiple predictions of time-series into a single binary class, we select a representative time step (EDU tasks3-4 at 60% course completion) average over all time steps (Weather tasksy- evaluated on a 24h window).
Results. Both MultiModN and P-Fusion achieve state-of-the-arr results on single tasks using multimodal inputs across all 10 targets. In Figure 2c] we binarize the continuous weather task (humidity prediction) as an average across all time steps. The task particularly challenging for the P- Fusioon baseline, which has random performance (AUROC: 0.5). Compared with P- Fusion,

---

MIMIC
Education (EDU)
Weather
Cardiomegaly
ECM
Success
Dropout
Next Week Tomp. (24b) Temp (72h)
Tomp (720h)
Humidity
Visibility
Metric
BAC
BAC
BAC
BAC
MSE
ASE
MSE
MSE
MSE
MSE
MultiMoNN
0.75 20.04
0.71 +000 0,93 20.04 0.83 +0,02 0.11 #0.01
0.03 =0.01
0.03 #0.01 0.03 =0.01
0.02 =0.11
0.10 =0.01
P-Fusion
0.75 +0.02
0.69 +0.03 0.92 +0.83 0.87
+0.05 0.01 +0.01 0.02 =0.01
0.03 +0.01
0.02 =0.01
0.03
=0.01 0.08 +0.02
Table 1: MultiModN does not compromise performance in single-tasks, Performance for binary and continuous prediction tasks in MIMIC, EDU, and Weather, comparing P-Fusion and MultiModN. 95% CIs are shown. ECM: Enlarged Cardiomediastinun Temp: Temperature.
EDU
Waather
AUC Store (bog. over soodh
P-Pazion (Raselins)
Puliioon Ranekons
06
MM (Singe Task
Med Mool (äagle Took)
M4rr (160) Taos)
Wen Mouth plunt Thoy
jast Overs
0.4-
0.04
0.2
0.02
(Blegle
(Bi Tack]
(Au TasA)
MustMoNN Tock)
0.00
Cardormepay
BCW
Sucoees (PTT)
Dispout
Wendy Part.
Tump (241)
Tump (10)))
Temp (C2Rg)
Via biy
Figure 3: Multi-task MultiModN maintains baseline performance in individual tasks. Single- and multi-task MultiModN on the prediction of individual tasks, compared with the monolithic P-Fusion (can only be single-task). AUC for binary (left) and MSE for continuous (right). Error bars: 95% Cls.
shows a 20% improvement, which significant at the p < 0.05 level. As the temporality of this task is particularly important, it could be hypothesized that the sequential nature of Mul tiModN better represents sereeseries inputs. Nevertheless, all weather targets are designed as regression tasks and show state-of of-the-art MSE scores in Table [ where MultiModN achieves baseline performance.
We provide an additional parallel fusion transformer baseline with experimental results showcased in Appendix Sec. [2.4 The results indicate that MultiModN matches or outperforms the multimodal transformer in vast majority of single- and multi-task settings, and comes with several inter- pretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for classification and MSE for regression tasks), Mul tiMoNN beats the transformer baseline significantly in 7 tasks, overlaps 95% CIs in 11 tasks, and loses slightly (0.01) in 2 regression tasks.
Mult MiMoN matches P-Fusion performance across all 10 tasks in all metrics reported across all three multimodal datasets. Thus, modularity does not compromise predictive performance.
6.2
Exp. 2:
Multi-task MultiModN maintains baseline performance in individual tasks
The modular design of MultiModN allows it to train multiple task- decoders and deploy them in any order or combination. While multi-task models have the potential to enrich feature extraction (and improve the model), it is critical to note that all feature extraction from the raw input is performed before Mult tiMoNN is trained. MultiModN is trained on embeddings extracted from pre- trained models (independently of its own encoders). This is done purposely to best isolate the effect of parallel- monoolihic vs. eoquential-modulle for each dataset, predicting the set of tasks in that dataset, i.e. tasks) in MIMIC, tasks3-5 in EDU, and tasksg_ 10 in Weather) and compare this 10 single-task MultiModN models (one for each tasks1-1o). Monolithic models, like P- P-Fusion on are not naturally extensible to multi-task predictions. Thus P-Fusion on (grey bars in Figure 3) can only be displayed for single-tast models. This experiment aims to compare MultiM iModN performance between single- and multi-task architectures to ensure that this implementation does not come a cost to the predictive performance of individual tasks.
Setup.
fusion. We train three multi-task MultiModN models (one
Results. In Figure 3 we compare the single-task P-Fusion (grey bars), to single- and multi- task implementations of MultiModN (in color). The results demonstrate that MultiModi is able to maintain its performance across all single-prediction tasks even when trained on multiple tasks. We additionally include the results of our model on various numbers and combinations of inputs, described further in Appendix Sec. E.5 The baseline would have to impute missing features in these combinations, exposing it catastrophic failure the event of systematic missingness (Sec. 6.4).
8

---

MultiModN has the significant advantage of being naturally extensible to the prediction of multiple tasks without negatively impacting the performance of individual tasks.
6.3 Exp. 3: MultiModN has inherent modality-speciffic local and global model explainabilty
Setup. Parallel MM fusion obfuscates the con- tribution of individual inputs and requires add-on or post hoc methods to reveal unimodal contri-
Individual Modaltty Contribution:: Global interpretab Tachnique
Pitation MaRtts
your
butions and cross-modal interactions [37 38 39].
23 Soenksen et al. [20] used Shapley values [40] to
Loies
P-huain Hirrpsii ity poubile for rolikk redaltas.
derive marginal modality contributions. While these post hoc methods provide valuable insight, they are computationally expensive and challeng- ing or impossible to deploy at inference. In contrast, MultiModi confers inherent modality- specific interpretability, where the contribution of each input can be decomposed by module. We use taski in MIMIC to compute two measures: [1] Importance score, where each encoder is de- ployed alone, providing predictive importance of single modality by subtracting predictions made from the prior state. This can be computed across all data points or individual data points. [2] Cu- mulative probability, where the prediction from each multi-task decoder is reported in sequence (i.e. given the previously encoded modalities). We demonstrate this on random patient from the test set, who has true label of O for both tasks. Further plots are in Appendix Sec. E.2
Curnlaatve Prodictions: intepritabiity Technique
09
61
Figure 4: Inherent modality-specifie model explainability in MultiModN. Heatmaps show individual modality contributions (IMC) (top) and cumulative contributions (CP) (bottom): re- spectively importance score (global explain- ability) or cumulative probability (local ex- plainability). The multi-task MultiMoaN for taski-2 in MIMIC is compared to two single- task P-Fusion models. IMC are only possi- ble for MultiModN (only 1 modality encoded, rest are skipped). CP are made sequentially from states encoding all previous modalities. P-Fusion unable to naturally decompose modality-speciff contributions (can only make predictions once all modalities are encoded). IMC is computed across all patients in the test set. CP is computed for a single patient, (true label 0 for both task1-2). The CP heatmap shows probability ranging from confident neg- ative diagnosis (0) perfect uncertainty and confident positive diagnosis (1)
Results. Monolithic P-Fusion models cannot be decomposed into any mpeelity- predic- tions, and its (single-task) prediction is only made after inputting all modalities. In contrast, Figure 4 shows Mul t tMModN provides granular insights for both importance score and cumulative prediction. We observe that the Text modality is the most important. The cumulative prediction shows the prior strongly predicts positivity in both classes and thus that So has learned the label prevalence.
The predictions naturally produced by MultiModN provide diverse and granular interpretations.
6.4 Exp. 4: MultiModN is robust to catastrophic f'illure from biased missingness
Setup. MultiModN is designed to predict any number or combination of tasks from any number or combination of modalities. A missing modality is skipped (encoder e; is not used) and not padded/encoded. Thus, MultiModN avoids featurizing missingness, which is particularly advanta- geous when missingness is MNAR. Featurizing MNAR can result in catastrophic failure when MNAR patterns differ between train and test settings. We demonstrate MultiModN' inherent robustness to catastrophic MNAR failure by training Mul MultiModN and P-Fusion on four versions of MIMIC with various amounts (0, 10, 50, or 80%) of MNAR by artificially removing one modality in one class only. Figure 5 compares Mult t MModN and P-Fusion on task; when tested in a setting that has either no missingess or where the MNAR pattern is different (i.e. label-flipped).
Results. Figure shows a dramatic catastrophic failure of P-Fusion in a label-flipped MNAR test set (black solid line) compared with MultiModN. P-Fusion: is worse than random at 80% MNAR (AUROC: 0.385). In contrast, MultiModN only loses 10% in MNAR flip, remarkably, matching performance in test with no missingness. Further plots in Appendix E.3

---

MultiModN is robust to catastrophic missingness (MNAR failure) where P-Fusion is not.
7
Conclusion
We present MultiModN, a novel sequential modular multimodal (MM) architecture, and demonstrate its dis- tinct advantages over traditional monolithic MM models which process inputs in parallel.
0.7
By aligning the feature extraction pipelines between MultiModi and its baseline P-Fusion, better iso- late the comparison between modular-sequential MM fusion vs. monolithic-parallel MM fusion. We perform four experiments across 10 complex real-world MM tasks in three distinct domains. We show that neither the sequential modularization of MultiModN nor its ex- tension multi-task predictions compromise the predic- tive performance on individual tasks compared with the monolithic baseline implementation. Training multi- task model can be challenging to parameterize across We perform no specific calibration and show that MultiModN is ro- bust to cross-task bias. Thus, at no performance cost, modularization allows the inherent benefits of multi-task modeling, as well as providing interepretable insights into the predictive potential of each modality. The most significant benefit of MultiModN is its natural robustness to catastrophic failure due to differences in missingness between train and test settings. This a frequent and fundamental flaw of many domains and especially im- pacts low-rew reworueee settings where modalities may be missing for reasons independent of the missingness in the train More generally, modularization creates a set of self-contained modules, composable in any number or combination according to available inputs and desired outputs. This composability not only provides enormous flexibility at inference but also reduces the computational cost of deployment. Taken together, these features allow MultiModN make resource-adapted predictions, which have a particular ad advantage for real-world problems in resource-limited settings.
066
Calemrooil Aerpning
05
Malh
(P
Test set with to mileingness
Madh
Percentage of MMMA train set,
Figure 5: MultiModN is robust to catas- trophic MNAR failure. Impact of MNAR missingess on MultiModN vs. P-Fusion. Both models are trained on four versions of the MIMIC dataset with 80% MNAR. They are then tested on either a test set with no MNAR missing- ness (-- or a test set where the biased missingness is label-flipped, i.e. MNAR occurs in the other binary class as com- pared with the train (-). Results for task depicted. C195% shaded.
inter- and cross-task performance [13 41
Limitations and future work. The main limitation for studying MM modeling is the scarcity of large-scale, open-source, MM datasets that cover multiple real-world tasks, especially for time-series. Additionally, while MultiModN is theoretically able to handle any number or combination of modali- ties and tasks, this has not been empirically tested. Having high combinatorial generalization comes at computational and performance cost, where the 'memory' of a fixed-size state representation will likely saturate at scale. The performance of MultiModN is purposely limited in this work by fixing the feature extraction pipeline, to best isolate the effect of sequential fusion. Future work leveraging MultiModi model-agnostic properties would be able to explore the potential performance benefit. This is particularly interesting for time-series, for which the state 'memory' may need to be parameterized to capture predictive trends of varying shapes and lengths.
8 Acknowledgements
This project was substantially co-financed by the Swiss State Secretariat for Education, Research and Innovation (SERI).
References
[1] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 2021.
10

---

[2] Tadas Baltrusaitis, Chaitanya Ahuja, and Louis- Phiilipp Morency. Multimodal machine learn- ing: survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelliencee 2019.
[3] Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey. IEEE Access, 2019.
[4] Arnab Barua. A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions. IEEE Access, 2023.
[5] Ming Fan, Wei Yuan, Wenrui Zhao, Maosheng Xu, Shiwei Wang, Xin Gao, and Lihua Li. Joint prediction of breast cancer histological grade and ki-67 expression level based on dce-mri and dwi radiomics. IEEE Journal of Biomedical and Health Informatics, 2020.
[6] Kevin M Boehm, Pegah Khosravi, Rami Vanguri, Jianjiong Gao, and Sohrab P Shah. Harnessing multimodal data integration to advance precision oncology. Nature Reviews Cancer, 2022.
[7] Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical AL Nature Medicine, 2022.
[8] Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, and Jie Zhou. Bridging text and video: A uni- versal multimodal transformer for audio-visual scene-aware dialog. IEEE ACM Transactions on Audio Speech and Language Processing, 2021.
[9] Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, and Amir Hussain. Multimodal sentiment analysis: Addressing key issues and setting up the baselines. IEEE Intelligent Systems, 2018
[10] Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Po- ria. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowledge- Based Systems, 2018.
[11] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. Empirical Methods in Natural Language Processing (EMNLP), 2017.
[12] Zeeshan Ahmad and Naimul Mefraz Khan. Human action recognition using deep multilevel multimodal (M²) fusion of depth and inertial sensors. IEEE Sensors Journal, 2020.
[13] Michael Crawshaw. Multi-task learning with deep neural networks: A survey. ArXiv, 2020.
[14] John W. Graham. Missing data analysis: Making it work in the real world. Annual Review of Psychology, 2009.
[15] Tlamelo Emmanuel, Thabiso Maupong, Dimane Mpoeleng, Thabo Semong, Banyatsang Mphago, and Oteng Tabona. A survey on missing data in machine learning. Journal of Big Data, 2021.
[16] Gargi Joshi, Rahee Walambe, and Ketan Kotecha. A Review on Explainability in Multimodal Deep Neural Nets. IEEE Access, 2021.
[17] Cécile Trottet, Thijs Vogels, Kristina Keitel, Alexandra Kulunkina, Rainer Tan, Ludovico Cobuccio, Martin Jaggi, and Mary-Anne Hartley. Modular clinical decision support networks (MoDN)-updatable interpretable, and portable predictions for evolving clinical environments. medRxiv. 2022.
[18] Shih-Cheng Huang, Anuj Pareck, Saced Seyyedi. Imon Banerjee, and Matthew P Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digital Medicine, 2020.
[19] Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. Multimodal machine learning in precision health: scoping review. NPJ Digital Medicine, 2022.

---

[20] Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux. Kimberly Villalobos Carballo, Liangyuan Na. Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ Digital Medicine, 2022.
[21] Alistair EW Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV (version 1.0). PhysioNet, 2021.
[22] Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih- ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. MIMIC-CXR-JPC large publicly available database of labeled chest radiographs. ArXiv, 2019.
[23] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1439-1449, 2021.
[24] Mengmeng Ma. Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18177-18186, 2022.
[25] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736, 2022.
[26] Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. Modular deep learning. ArXiv, 2023.
[27] Mohammed Amer and Tomás Maul. A review of modularization techniques in artificial neural networks. Artificial Intelligence Review, 2019.
[28] Ulrik Brandes, Daniel Delling, Marco Gaertler, Robert Gorke, Martin Hoefer, Zoran Nikoloski, and Dorothea Wagner. On modularity clustering. IEEE Transactions on Knowledge and Data Engineering, 2008.
[29] M. E. J. Newman and M. Girvan. Finding and evaluating community structure in networks. Physical Review E, 2004.
[30] Ary Goldherger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Thung-Kang Peng, and H Eugene Stan- ley. PhysioBank, PhysioToolke and PhysioNet: components of new research resource for complex physiologic signals. Circulation, 2000.
[31] Vinitra Swamy, Bahar Radmehr, Natasa Kreo, Mirko Marras, and Tanja Käser. Evaluating the Explainers: Black-box explainable machine learning for student success predic- tion in MOOCs. Educational Data Mining (EDM), 2022.
[32] Mina Shirvani Boroujeni and Pierre Dillenbourg. Discovery and temporal analysis of MOOC study patterns. Journal of Learning Analytics, 2019.
[33] Mohammad Asadi, Vinitra Swamy, Jibril Frej, Julien Vignoud, Mirko Marras, and Tanja Käser. Ripple: Concept- based interpretation for raw time series models in education. AAAI Conference on Artificial Intelligence (EAAI), 2023.
[34] Vinitra Swamy, Mirko Marras, and Tanja Käser. Meta transfer learning for early success prediction in MOOCs. In ACM Conference on Learning at Scale (L@S), 2022.
[35] Cheng Ye and Gautam Biswas. Early prediction of student dropout and performance in MOOCs using higher granularity temporal information. Journal of Learning Analytics, 2014.
[36] Xun Zhu, Yutong Xiong, Ming Wu, Gaozhen Nie, Bin Zhang, and Ziheng Yang. Weather2k: multivariate spatio-temporal benchmark dataset for meteorological forecasting based on real-time observation data from ground weather stations. AISTATS, 2023.
12

---

[37] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang. Louis- Philippe Morency, and Ruslan Salakhutdinov. Multiviz: Towards visualizing and understanding multimodal models. In International Conference on Learning Representations (ICLR), 2023.
[38] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[39] Torsten Wörtwein, Lisa Sheeber, Nicholas Allen, Jeffrey Cohn, and Louis-Philippe Morency. Beyond additive fusion: Learning non-additive multimodal interactions. Empirical Methods in Natural Language Processing (EMNLP), 2022.
[40] Benedek Rozemberezki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivér Kiss, Sebastian Nilsson, and Rik Sarkar. The shapley value in machine learning. In International Joint Conference on Artificial Intelligence, 2022.
[41] Partoo Vafaeikia, Khashayar Namdar, and Farzad Khalvati. brief review of deep multi-task learning and auxiliary task learning. ArXiv, 2020.
[42] Vinitra Swamy, Sijia Du, Mirko Marras, and Tanja Kaser. Trusting the Explainers: Teacher validation of explainable artificial intelligence for course design. In LAK23: 13th International Learning Analytics and Knowledge Conference, pages 345 3366, 2023.
[43] Mirko Marras, Julien Tuang Tu Vignoud, Tanja Kaser, al. Can feature predictive power generalize? benchmarking early predictors of student success across flipped and online courses. In Proceedings of the 14th International Conference on Educational Data Mining, pages 150-160, 2021.
[44] Daniel FO Onah, Jane Sinclair, and Russell Boyatt. Dropout rates of massive open online courses: behavioural patterns. EDULEARNI4 proceedings, pages 5825-5834, 2014.
[45] Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong
Lu. Negbio: nigh-perfonmance tool for negation and uncertainty detection in radiology reports.
AMIA Summits on Translational Science Proceedings, 2018:188, 2018.
[46] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
13

---

A MultiModN Framework Implementation
A task- and modality-agnostic open-source framework MultiModN solution has been implemented in Python using PyTorch as the primary machine learning framework. The /multimodn package contains the MultiModN model and its components. The /datasets package is responsible of preparing the data inputs for MultiModN. Some examples using the public Titanic dataset have been provided.
The code is available at: httpe///tthaaooo///rggggoalaaaat
Grocken
Zaeck
Sulk
Prunket
$4 beirdeen
$41
J11
DeCOk'
JIJ Terges
JJJ
Targes
Targdt
Figure 6: Architecture of MultiModN code. The upper panel shows more detailed depiction of sequential encoding using series of model-agnostic encoders which receive inputs of variable dimension to create the evolving state vector, which represents the shared feature space. The lower panel shows how each state can be probed by any number of target decoders.
A.1 MultiModi metrics
During training and evaluation, the metrics of the model are stored a log at each epoch in matrix of dimensions (E 1) D, where E is the number of encoders and D the number of decoders. Each row represents the metrics for a target at each state of the model.
A.2
Code structure
A.2.1
MultiModN
/mul timodn package contains the Mult iModN model and its modules:
[1] Encoders: honndeeww/mullimmdcceeecodrrrr
[2] Decoders: heodderww/muliinoodddddddeerrr
[3] State: /mmltimodn/statee.p
A.2.2 Datasets
/dataset package contains the MultiModDataset abstract class, compatible with Mul tiModM.
Specific datasets are added in the /dataset directory and must fulfill the following requirements:
Contain dataset class that inherit MultiModDataset or has method to convert into MultiModDataset
Contain a sh script responsible of getting the data and store it in /data folder
_gettttm_....................__. function of MultiModDataset subclasses must yield elements of the following shape:
tuple
data: [torch. Tensor),
targets: numpy .ndaray,
14

---

(optional) encoding_sequence
numpy. ndarray
namely tuple containing an array of tensors representing the features for each subsequent encoder, numpy array representing the different targets and optionally numpy array giving the order in which to apply the encoders to the subsequent data tensors. Note: data and encoding. _equeecee must have the same length.
Missing values. The user is able choose keep missing values (nan values). Missing values can be present in the tensors yielded by the dataset and are managed by MultiModN.
A.2.3 Pipelines
/pipeline package contains the training pipelines using MultiModd for Multimodal Learning. follows the following steps:
Create Mul MoltiiddDDttset and the dataloader associated
Create the list of encoders according to the features shape the MultiModDataset
Create the list of decoders according to the targets of the MultiModDataset
Init, train and test the Mul tiMMMM model
Store the trained model, training history and save learning curves
A.3
Quick start
Quick start rumning MultiModN on Titanic example pipeline with Multilayer Perceptron encoder:
/
python3 pipelimee/tiiaaiicctttttcc...
py
Open pippl/hee//tttamlc//iaiiiiiiiii png to look at the training curves.
B
Additional details about MultiModN Architecture
4n
4x1
91
92
g3
Figure 7: Schematic representation of the modules (g, groups) of MultiModN. encoders, vector, d: decoders. Each module connected by single edge between 8, and 6n+11 There are [M] groups (i.e. input-specific modules) and [T] decoders per module.
Modularity. In the following, we detail the computation of Multi MooMi's modularity measure, The total number of edges in MultiModN module is [T] + The total number of modules is |M] and there are |M - edges connecting consecutive modules, which makes for a total number of edges in the entire MultiModN model of [ + -
To compute the modularity following using the formalization proposed by Newman et [2] we need to define groups In the case of Mul MiMdN,,,N, each group corresponds to one module. Let G be the matrix whose components giy the fraction of edges in the original network that connect vertices in group to those in group
15

---

Within MultiModN, each group contains (I7] + 1) edges and is connected to adjacent groups by two edges, with the exception of 91 and glml- which are connected to only one other group (cf. Figure y
Thus, G is tridiagonal matrix whose diagonal elements Gi are equal to (17) + 1)/m and whose upper and lower diagonal elements are equal to 1/m:
11
1
1
[T|+1
G
m
[T
I
The modularity measure is defined by = tr(G) - ||
where tr(G), is the trace of G and ||G (T and we have G2 equal to:
the of elements of G?. The trace of G is equal to
M
(1T)+))++1
2([T|+1)
1
2(17++1)
(1T)+))2+2
2(IT++1)
1
2([7++1)
(17++))+2 2(17++1)
1
m
2(TT++1)
+1)++)++
2([T++1)
1
2([7++1)
(IT++1)++1
Hence, we have:
|G [IM[[T++ [[[[]|+1)] + 2) - M - + +1) - 22]
m²
M 6 6| + M - 47 - 10
2 4 +4 - M
It is important to note that when M increases, the trace will tend to (IT) + 1)/((T) + for large number of tasks. Morcover, when |i|icrccasss ||G will decrease towards to 0. Thus, for large number of tasks, we have a modularity measure that increases and tends to with the number of modalities.
Architecture alignment between P-Fusion and MultiModN. We purposely align the feature extraction pipelines P-Fusion and MultiModN in order to best isolate the effect monolithic- parallel fusion vs. modular-seques sequential fusion. In Figure 8 we see how the alignment is limited to the input, where both MultiModN and P-Fusion share the feature-extraction of each modality, where the MultiModN encoders receive an embedding (emb). No element of MultiModi is changed as described in Figure] [ Embeddings from missing data can be skipped (i.e. not encoded) by MultiModN.
16

---

tasks
tapkpp
tasks
caskys
module;
modulen
dp
dml
d1 ....
diT)
Nulti-task Decoders
MultiModN Architecture unchanged
S1j
MiMj
each combinat on inputs
11
e M
Multimodal Encoders
Missing embedaingg skipped
emb
jemb
emb
Separate embeddlings
P-fusion
embeddingg that MultiModh takes sequence (instead
11
e
en
Multimodal Encoders
concatenating)
mod
mod
mod
missing
TXT
P- P-Fusion and MultiModN architectures. We purposely ensure that the feature extraction pipelines are aligned between P-Fusion and MultiModN. To this end, we use the embeddings (emb) produced by P-Fusion as inputs into the MultiModi encoders (e). No element of MMMMMMMMMo MMMdN is changed. MultiModN encoders (e) are in orange, the MultiModN state (s) is in blue and multi-task Mult MultiModN decoders (d) are green.
Figure 8: Alignment
For MIMIC, feature extraction for each modularity is replicated from previous work (20 and use embeddings generated from set of pre-trained models. For Weather and Education, there were no pre-existing embedding models, design autoencoders trained to reconstruct the original input features from a latent space. We keep the autoencoder's encoder and decoder structure exactly aligned with the encoders of MultiModN (two ReLU activated, fully-connected Dense layers and a third layer either generating the state representation with a ReLU activation or the final prediction with sigmoid activation). We also align the number of trainable parameters with Mul MultiModN's modality-based encoders for fair baseline comparison by selecting appropriate state representation size per modality to equal the state representation in Mul MoMNN. The remaining hyperparameters are left exactly the same (batch size, hidden layer size, dropout rate, optimization metrics, loss function).
C Datasets and Tasks
description of all tasks is provided Table 2 In the following paragraphs, we detail the prepro- cessing decisions on the datasets for context and reproducibility.
MIMIC. The data includes four input modalities (tabular, textual, images, and time-series) derived from several sources for each patient. We align our preprocessing pipeline exactly with the study from which our baseline of P-Fusiin is derived (20] (described in 4.1) To this end, we use petient-level
feature embeddings extracted by the pre-trained models described in 20 and depicted in Figure g
The dataset is combination of MIMIC databases; MIMIC-IV v.1.0 21 and MIMIC-CXR-JPG v.2.0.0 (22 After gaining authorized access to PhysioNet online repository [30], the embedding dataset can downloaded via this link: httpp:://ss /tctps:/ppysiontt
httgg/coneet////ee haaimtt/haimmmii maaimmmuttin multimodal/
1.0. 1/ The dataset comprises 45, 050 samples, each corresponding to time point during patient's hospital stay when chest X-ray was obtained. covers total of 8, 655 unique patient stays. To ensure data quality and limit our experiments to two thematic tasks (diagnosis of Lask; cardiomegaly and laskg: enlarged cardiomediastinum) we remove duplicates (based on image id and image acquisition time), and retain only relevant samples that have valid labels for both targets of interest,
17

---

Le. both task; and task2 are either present (1) or absent (0). Subsequent experiments for these tasks are thus performed on the 921/45, 050 selected relevant patients.
EDU. This dataset involves hand-crafted features extracted for 5, 611 students across 10 weeks of data. The preprocessing of data is an exact replication of several related works using the same dataset (34 31 42 based on 4 feature sets determined as predictive for MOOC courses in [43]. 45 features regarding problem and video data are extracted per student per week, covering features like Delay Lecture, which calculates the average delay in viewing video lectures after they are released to students or TotalClicksProblem, the number of clicks that a student has made on problems this week. The features are normalized with min- max normalization, and missing values are imputed with zeros that have meaning i.e. no problem events in a week is correctly inferred as zero problems attempted. In this setting, missingness is a valued, predictive feature of the outcome, and thus we do not perform missingness experiments on this dataset. Mul tiMM has the ability to select whether missingenss is encoded or not, and thus it would not suffer disadvantage in setting where missingness should be featurized.
In MOOCs, a common issue is that students join a course and never participate in any assignments, homeworks, or quizzes. This could be due to registering aspirationally, to read some material, or to watch videos 44.. An instructor can easily classify students who have never completed an assignment as failing students. As introduced by Swamy et al. in 34 and used with this dataset in related work EDU has removed students that were predicted to fail in the first two weeks simply by having turned in no assignments (99% confidence of failing with an out-of-the-box logistic regression model, where the confidence threshold was tuned over balanced accuracy calculations). It has been shown that including these students will artificially increase the performance of the model, providing even better results than those showcased by students to test a more challenging modeling problem.
[3 42 331.
in this work (3) We thus exclude these
Weather. The Weather2k dataset, presented in (36), covers features from 1, 866 weather stations with 23 features covering seven different units of measurements (degrees, meters, HPA, celsius, percentage, mss millimeters). To align these features on vastly different scales, we normalize the data. We use the large extract (R) provided by the authors instead of the smaller representative sample also highlighted in the benchmark paper (S) [3).. We use the first 24 hourly measurements as input to train the MultiModN model and calculate the five regression tasks as determined in Table 2 below.
Tasks. As showcased in Table 2 our evaluation covers 10 binary and regression tasks in two settings: static (one value per datapoint) continuous (changing values per datapoint, per timestep).
Task
Type
Namte
Description
MIMIC
Binzyy
Stack
Canionegssl
to process radiology octes, resulting disgnostic outcomes: poszive, negative, uncertais, missing.
Labels determined per using NogBio CheXpert
2
Stokk Binzy
Caddonedlattiiou
Enlarged
The set label values idenical to ose for cardonogaly.
Labels devernined per (D) using NegBoo and CheXperi
[b
EDU
Binnay
Stack
Prediction
Success
Fnd of coarse fase-fuil prodiction (per stalant) as per on the counse
Comtinauus Dinayy
Prediction
Dropout
Not for
scudent any non zero on or problem
decoder steps for timesteps, multiclass task by sepanttin video or groblem involvement until
nest week
easily
course, not.
course into sepurate classes.
Continacus Regression
Performance
Porecssting
Week
Moving average (per student, per week)
Competencttiionn (number problems the student has passed weekl. Competrnciiirengeh (cxtent to which stndest passes a quiz getting the maximum grade with atlempis).
performance buseline paper [L] Staden Shape (cecieving the mutimuum quiz grade on unempt).
from removed
Continacus
Regression
Temperature Poressating
Term
Changing air temperature measteemntts (collectol per station, per houz), shifted by 24 hourly meseurements (1 day).
Weather
Continacus
Regression
Tamperature
Porecssting
Classinn air tompee ature meaausement (collected per station, per houz), shilted by 72 hourly meseurements (3 days).
Continacus
Regression
Temperature Fowecnsting
Loog Torm
Changing air temperature measurements (collected per station, per hour), shifted by 720 hourly messureenns (30 days).
Regresison
Stucke
Relative Hunidity
Instantancous lunnddity relative saturation as percentage 48h from 2.5 meters above the ground.
used as benchmark Eocccastng tuck 56]
Stutke Regresison
Vishslity
used benchmark Socceasing tuck [36
stean horizontal visibility
48 from 2.8 maters ahove the ground.
Table 2: Description of all 10 tasks used to evaluate iModi
18

---

Note that for tasky, the three features used to calculate next week's performance were not included in the original imput features because of possible data leakage, as student performance on quizzes directly contributes to their overall grade (Pass/Fail).
For the AUROC curves on tasks9-10 10 in Figure 2 conduct binarization of the two last regression tasks. To align with the regression task, we conduct a static forecasting prediction per station for the relative humidity or visibility with a time window of 24h for the 48th timestep (one day in advance). While Multi MModN is capable of making a prediction at each continuous timestep, P-Fusi on is not able to do this without separate decoder at each timestep, and therefore to compare the tasks we must choose a static analysis. We choose a threshold based on the normalized targets: 0.75 for humidity and 0.25 for visibility (selected based on the distribution of the feature values for the first 1000 timesteps), and evaluate the predictions as a binary task over this threshold.
Analogously, in the Section E. analysis below the binarization of the remaining regression tasks, we express the continuous regression tasks for temperature forecasting tasks8-8 as a static binary task. To do this, we evaluate the prediction from the full window (24th timestep) the respective forecasting timestep (48, 96, 744). The threshold we select is 0.3, closely corresponding to the normalized mean of the temperature (0.301) over the first 1000 timesteps.
D Model Optimization
Table indicates the chosen hyperparameters for the experiments conducted in Section the paper, selected based on the optimal hyperparameters for the multi-task settings (all the tasks for a dataset predicted jointly). The single tasks use the same hyperparameters as the multi-task settings. For saving the best model across training epochs in the time series settings (EDU, Weather), our optimization metric saved the ones with best validation set results the most short term task. Therefore, for EDU, we use MSE of tasks, predicting next week performance, and taskş for Weather, forecasting temperature within a day). The intuition is that choosing the best model the short term task would allow the model to emphasize stronger short-term connections, which in turn would improve long term performance.
Hidden State
of Timesteps
Batch Dropout Rate
Save Best Model
Task
Layer Size
Rep. (chosen metrie)
Size
Size
16
0.2
32
50
taski Val BAC + Macro AUROC
2 MIMIC
16
0.2
32
50
task2 Val BAC + Macro AUROC
16
32
50
tasksi-2 -2 BAC + Macro AUROC
10
64
20
tasky Accuracy
10
64
0.1
32
20
tasks Val Accuracy
5 EDU
10
64
0.1
32
20
tasks, Val MSE
10
64
0.1
32
20
task, Val MSE
24
128
0.1
32
20
taska Val MSE
24
128
0.1
32
20
tasky MSE
24
128
32
20
tasks Val MSE
9
24
128
0.1
32
20
taska Val MSE
10
24
128
0.1
32
20
task1o Val MSE
Weather 24
128
32
20
tasks MSE
Table 3: Hyperparameters selected for each experiment. We tuned the hyperparameters for the multi-task models (MIMIC, EDU, Weather) and used the same hyperparameters for each single-task model for fair comparison.
In Figure B we examine case study of MultiModN's changing performance on tasks, student success prediction in the EDU dataset, by varying hyperparameters across three model architectures (chosen for small, medium, and large hyperparameter initializations). We note that batch size is fairly robust across all three initial model settings, with large batch size on the largest model having slightly variable performance. Examining changing dropout rate, we note that with medium and large models, change dropout impacts performance considerably. This allows us to hypothesize that high dropout on larger state representations does not allow the model to learn everything it can from the data.
19

---

Looking at varied hidden layer size, we see comparable performance for the small and medium initializations, but note that in the large case, having smaller hidden layer size is important to maintain performance. Even as MultiModN performance trends upwards with larger hidden layer size (i.e. 128) for the large initialization, the confidence interval is large, so performance is not stable. Lastly, observing state representation size, we see that when state representation is too small for the task (i.e. 1, 5), the small and large models are adversly impacted. Additionally, when state representation is too large (i.e. 100), performance seems to drop increase variability again. It is therefore important to tune MultiModN and find the right state representation size for the dataset and predictive task(s).
BAC Predziin
Hyperparamater
Hyperpaaaettr
Hyperparmettr
0.2
Smad (10 hetee-
1- 12
Sruil EX2 buish 10 (axx--9244 924)
23 0.
Medium XM sanch sede1-
Hödden
build teakee 30 staaaad
(128 butch ledken
0.1
92
03
9
106
Batch Size
Dropout Rate
Hiddon Layer Size
State Representatio Sizo
Figure 9: MultiModN hyperparameter selection across four parameters on task3 the EDU dataset (Pass/Fail). Each individual parameter is varied on the x-axis (dr: dropout rute) with all other initializations fixed (grouped in small, medium, and large values). These are compared in terms of balanced accuracy (BAC). 95% CIs are shaded.
Experimental Setup. For the results reported in Sections 6.1 6.2 and 6.4 we perform 5-fold stratified cross-validation with 80-10-10 train-validation-tess split. Due to the time- series nature of the EDU and Weather datasets, we orient the stratification the real labels associated with the longest-term task. For EDU, tasks (end of course pass-fail prediction) and for Weather tasky (30-day temperature forecasting) chosen for the stratification split.
an alternative approach, regarding the MIMIC dataset, two-step procedure was implemented to address the imbalanced class ratios, given the absence of a priori- tized task. Initially, a new dummy label was assigned to each sample, indicating positivity both pathologies are present and negativity otherwise. Subsequently, label was assigned to each unique hospital stay based on the aggregated labels from the first step. A hospital stay was considered positive if the number of times sample from that stay has been found positive is greater than or equal to the half of samples with the same hospital stay ID. The latter, outlined in [20], ensured that information was leaked on the hospital stay level during stratification.
10-
0.8-
Temperature forecesting
22.35 Taak
Texk 720
PF Tosk ALE--0S--00
0.2-
MutiMMedt Esingle
WulMecdUSS--la TAU 2.062
WLllMcdd-ST--tat 24UD-28082222
0.0-
Experiments were conducted using the same architecture in PyTorch (MIMIC) and TensorFlow (EDU, Weather), to provide multiple implementations across training frameworks for ease of use. We use Adam optimizer with gradient clipping across all experiments.
Falee positive rate
Figure 10: AUROC for three additional binary prediction tasks in Weather2k. Tar- gets predicted by P-Fusion are compared to MultiModN, 9I%CIs are shaded.
E
Additional Experiments
E.1
Single task
present the binarized results (AUROC curves) for several additional regression tasks (lasksg-8 for Weather) in Figure 10
The specific details of binarization discussed above in Appendix
20

---

We note that the confidence intervals overlap for P-Fusion and MultiModi over all Weather tasks. However, it clear that the performance of MultiModN varies lot (large Cls). This could be due to the design of the binarization originally in the benchmark paper (36]. this was introduced as a regression task. Another contributing factor to large Cls could be that the model was trained across all timesteps but only evaluated on one timestep for comparable binarization. Despite these caveats, we can statistically conclude that P-Fusion and MultiModi performance are comparable on these additional tasks.
Section C
E.2 Interpretability
We perform an interpretability analysis for the EDU dataset, analogous to the local and global interpretability analysis on MIMIC from Figure 4 The global analysis (IMC) is conducted over all students for the first week of the course. We note interesting findings: specifically that problem interactions are more important for tasks3-4 while video interactions more important for tasks-
Individual Modality Contributions: Global Interpretability Technique
(importance scores)
1.0
P-Fusion MulbModN
Tesk
Task
0.5
Problem
Video
P-Fusion interpretability not possible individual modalities.
Cumulative Predictions:
Local Interpretability Technique (probabilitie)
MultiMMoNN
Tack
1.0
Task
Task
Prior [Itandaro)
Problem
Video
Label 0.5
P-Fusion
P-Fusion Interpretability is not possible at internediate steps.
Task
Task
Prior
Problem
Video all previoza)
provicss)
Inherent modality-specific model explainability in MultiModN for tasks3-5- Heatmaps show individual modality contributions (IMC) (top) and cumulative contributions (CP) (bottom): respectively importance score (global explainability) or cumulative probability (lo- cal explainability). The multi-task MultiModN for taska-5 in EDU is compared two single-task P-Fusion models. IMC are only possible for MultiModM (only modality encoded, rest are skipped). CP are made sequentially from states encoding all previous modalities. P-Fusion on is unable natu- rally decompose modality-speciff contributions (can only make predictions once all modalities are encoded). IMC is computed across all students in the test set. CP is computed for single student, (true label = for tasks3-4 and 0 for tasks). The CP heatmap shows probability ranging from confident negative diagnosis (0) perfect uncertainty and confident positive diagnosis (1).
Figure 11:
The student selected for the CP local analysis passes the course (1 for task3) and does not dropout (1 for task,), but does not have strong performance the next week (~> 0 tasks). This analysis is also conducted across the first week of course interactions. We see that P P-Fusion cannot produce modality interpretations and predicts the incorrect label. However, Mul MulliMMdNN able to identify a changing confidence level across modalities, eventually ending on the right prediction for all tasks. The confidence for laska increases with the student's problem interactions and reduces for their video interactions. This could have potential for designing an intervention to improve student learning outcomes. We note that for lasks, both the students' problem and video interactions contribute similarly to the prediction.
21

---

E.3 Missingness
We expand on the missingness experiments presented in 6.4 Here, we present further control
experiments (training data bussing--t-randon MAR) in both MIMIC tasks (taskı: diagnosis of Cardiomegaly 12 and task2: diagnosis of Enlarged Cardiomediastinum 13
In the first two subplots of each figure, both P-Fusion (black) and MultiModN (red) are trained on MNAR data and then evaluated either on a test set risk of catastrophic failure (where the pattern of MNAR is label-flipedd, first figure) or on test set with no missingness. As can be seen the first figures, P-Fusion suffers catastrophic failure the MNAR flip, becoming worse than random when single modality is missing 80%, opposed to MultiModN, which only decreases AUROC about 10%. When test set has no missing values, P-Fusion and Mul tiModN are not significantly different, proving that the catastrophic failure of P-Fusion on is due to MNAR. This is further confirmed in the last two plots each task, where the models are trained on MAR data and evaluated on test sets either without missing values MAR missingness.
Figure 12: Detailed missingness experiments for laski (Cardiomegalyy, P-Fusion (black) and MultiModN (red) are trained on MIMIC data where various percentages of a single modality are missing (0 80%) either for single class (MNAR. first two plots) or without correlation to either class (MAR, last two plots). The AUROCs are shown for each when evaluated on test sets which either have risk of catastrophic failure (first plot, MNAR with label flip) test sets without missingness or MAR missingness. CI95% shaded.
Figure 13: Detailed missingness experiments for tasky (Enlarged Cardiomediastinum) P-Fusion (black) and MultiModN (red) are trained MIMIC data where various percentages of a single modality are missing (0 or 80%) either for a single class (MNAR, first two plots) or without correlation either class (MAR, last evaluated on test sets which either have a risk of catastrophic failure (first plot, MNAR with label flip) or test sets without missingness MAR missingness. C195% shaded.
plots). The AUROCs shown for each when
E.4 Comparison P-Fusion Transformer
Additional experiments with Transformer have been conducted 10 tasks across three datasets. Results are showcased below in two tables (left for tasks1-4 and right for tasks; with 95% Cls.
The hyperparameter-tunee tuned architecture (based on head size, number of transformer blocks, MLP units) for EDU and Weather is transformer model with transformer blocks, heads size 256, dropout of 0.25, MIP units 128 with dropout 0.4, batch size 64, trained for 50 epochs with cross-entropy loss. For MIMIC, the most performant (tuned) transformer architecture includes transformer blocks with 3 heads of size 128, MLP units 32 with batch size 32. We train this architecture each decoder
22

---

task individually and all tasks together for total of 13 new models with the exact preprocessing steps as the P-Fusion and MultiModN experiments. The results indicate that MultiModN often outperforms or at least matches the P- F-Fusion Transformer benchmark in the vast majority of single task and multi-task settings, and comes with several interpretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for the classification tasks and MSE for the regression tasks), MultiModN beats the Transformer baseline significantly in tasks, overlaps 95% Cls in 1l tasks, and loses very slightly (by 0.01) in 2 regression tasks.
Regresssion
Task
Encoder
MSE
R2
ClassiEcation
EDU
Single
Weedly Perf.
Task
Encoder
BAC
Mus
3.91 =3.40
649.05 508.01
Single
Single
0.00
0.80
MIMIC
Cardiomegaly
Muti
0.50 6:7
0.43 nicc
x0:0
Single
0.50 x
0.44 0.08
Single
0.00 10.0
0.71 6.1.
ECM
Temp.
WEATHER
Single
0.50 + b.1
0.40 x
Single
a.00
0.82
Sucoess (P/F)
Tomp. (720h)
EDU
Single
0.53 8.22
0.83
Single
0.01 2001
0.69 8:22
Dropout
Humidity
a 6.22
0.02
Muls
2.00
0.00
Single
Visibiitty
Mus
0.60
Figure 14: Performance of the P-Fusion Transformer on 10 classification and regression tasks across 3 datasets. Results are showcased with 95% confidence intervals. BAC and MSE are the primary evaluation metrics for classification and regression respectively.
E.5
Additional Inference Settings
Modalittis (Inference)
Task
MultiModN
P-Fusion
[demo]
0.50
0.50
CM
[text]
0.61
[ts]
0.52
0.53
[visual]
0.75
0.75
Modelities
[demo]
0.50
0.51
Task
(Inference)
MultIModN
P-Fuston
EDM
[text]
0.59
0.52
[demojtextits]
0.64
0.60
[ts]
0.53
0.54
[demo/visual]text]
0.75
0.75
[visual]
0.71
0.72
|demolvisual(ts|
0.77
0.76
[demo]text]
0.55
0.57
|iisualtextitss
0.74
0.75
[demojts]
/
0.54
0.52
[demojtextits]
0.56
CM
[demolvisual]
EDM
[demo/visualjtext]
0.72
0.68
[text]ts]
0.61
0.59
[demolvisualits]
|visual[text]
|visua[text(ts/
0.71
0.67
|visual[ts]
0.76
0.76
|demo/visualltexttts
[demo]text]
0.56
[demo/visual]txxttss
0.71
0.68
[demojts|
M
0.47
0.56
ECM
[demoovvsual]
0.69
[text]ts]
x
0.60
0.51
|vsual[text]
0.71
0.68
\visualits]
0.72
0.71
Detailed modality inference experiments for MultiModN in comparison P- P-Fusion on. In these experiments, different combinations of modalities and orderings the time of inference are used for the two tasks in the MIMIC dataset. All 95% CIs overlap between the two models.
Figure 15:
23

---

To provide insight into performance gain, we performed additional experiments to showcase the benefits of modularity with vastly different training and inference settings. The results of 30 new experiments of inference encoders, each performed with 5-fold cross-validation are included in Figure We compare P-Fusion and MulNiModN on both tasks of the MIMIC dataset using all possible combinations of four input modalities at test time. MultiModN ignores missing modalities whereas P-Fusion imputes and therefore encodes missing modalities.
13
We note that the performance at inference for P-Fusion and Mul tiModN has no significant differences for all experiments (using 95% Cls). Figure I3 shows that, on average, P-Fusion on tends to overfit more to the most dominant (visual) modality. When this modality is missing (at random or completely random), MultiModN performs better on combination of the remaining modalities (demo, text, time series). In the case of missing modalities, the observed effect in Figure 15 is weak confidence intervals overlap. Considering the MNAR (missing not-at-random) scenario described in Sec. 6.4 the difference becomes significant.
24


Reviews:
[{"review_id": "wbAxDRSzMq", "reviewer": "Reviewer_pFnK", "summary": "This paper proposes a modular multimodal model that fuses latent representations in a sequence of modality to conduct a combination of predictive tasks. It utilizes a flexible sequence of model and task-agnostic encoders to produce an evolving latent representation for a combination of multi-task, model-agnostic decoder modules to conduct downstream tasks. The authors conduct experiments on benchmarks of different domains to show the effectiveness of the proposed method. The experiments show that the proposed method outperforms traditional monolithic multi-modal models.", "questions": "1. The authors should make the figures in the experiments part more clear to see the comparison of the proposed methods and the baseline.", "limitations": "The authors have addressed the technical limitations in the paper. I do not find any negative societal impact of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and well-organized. The design of methods and experiments are clear and easy to understand.\n2. The proposed method is novel and effective. The modules are task-agnostic and can be adapted on any number of tasks on different modalities.\n3. The experiments are substantial and convincing. The proposed method shows significantly better performance compared with traditional methods.", "weaknesses": "1. It seems that for single tasks the proposed method cannot provide improvement compared with the baseline.\n2. It would be helpful to support the claim that the proposed model can handle  any number/combination of tasks by adding more experiments of combinatorial tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**SUMMARY:**  \nWe kindly thank the reviewer for this careful evaluation of our work and greatly appreciate the positive review of our “excellent” presentation, methodological soundness, and worthwhile contribution.\n\nWe strongly agree with all points raised. We have addressed each in a point-by-point response below and feel that the resulting edits have further improved the manuscript.\n\n**STRENGTHS:**  \nWe are happy that the reviewer feels the paper is well-written, well-organized, clear, and easy to understand. We are also thrilled to see that they appreciate the “novelty and effectiveness” of our proposed model, and that our experiments are substantial and convincing.\n\n**WEAKNESSES:**  \nThe reviewer notes that for single tasks, **our method does not improve compared with the baseline**. \nIndeed, this is intended. It is critical to understand that we (purposely) do not aim to improve on the baseline in terms of performance. Rather, we purposely limit the model’s ability to do so, by aligning the pre-extracted features. This is to isolate the comparison of the fusion step. In a phenomenon called “negative transfer”, multimodal multi-task models often have degraded performance compared to their single-modality counterparts [1]. \n\nWe argue that our model is by far superior to the baseline by virtue of being modular, interpretable, composable, robust to systematic missingness, and multi-task *without* impacting performance compared to the parallel baseline which does not have these other features and will make this point more explicit in the final manuscript.\n\nThe reviewer would like to see **more support for our method’s performance using various numbers and combinations of inputs** at inference. This is an excellent suggestion. We now include additional experiments detailing the results of our model on various numbers and combinations of inputs (see Figure 1 of supplemental PDF information). The baseline would have to impute missing features in all these combinations, exposing it to catastrophic failure in the event of systematic missingness as explained in Section 6.4 and supplemental Section E.3.\n\n**QUESTIONS:**  \nThank you for the suggestion to improve the experimental figures for clearer understandability. As recommended, we have made several formatting edits to better distinguish MultiMoDN and the baseline directly in the figures. The revised figures are showcased in the attached PDF, with differences highlighted in yellow.\n\n**LIMITATIONS:**  \nWe are happy that the reviewer feels we have appropriately addressed all limitations in the paper.\n\n**REFERENCES:**  \n[1] Liu, Shengchao, Yingyu Liang, and Anthony Gitter. \"Loss-balanced task weighting to reduce negative transfer in multi-task learning.\" AAAI 2019.\n"}, {"review_id": "luREqweYgw", "reviewer": "Reviewer_iERT", "summary": "This paper describes an architecture for multimodal multitask learning which is robust to missing modalities/tasks both at training and test time. This is composed of:\n- encoder specific modules\n- (assuming an ordering among modalities) a hidden states which depends on the output of a modality specific encoder, and which feds into the next encoder,\n- a decoder that takes the state and produced an output for each task.\nThe method is tested on three multimodal datasets, namely MIMIC, EDU and Weather2K. ", "questions": "See comments above in the weakness section.\nThe major question is about 1) relation to prior work, particularly in the vision/language domain, 2) use of more standard benchmarks (in addition to the chosen ones), 3) use of RNN as opposed to transformer, 4) various clarifications of the approach.", "limitations": "No concern.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- multimodal multitask learning is a very interesting topic which is very relevant to this venue\n- the idea to make a multimodal multitask system robust to missing modalities is clearly good\n- the overall motivation and intuition behind the proposed architecture is sensible and intuitive", "weaknesses": "- The major limitation is the lack of reference and comparison to other multimodal multitask approaches, particularly those based on transformers in the vision/language domain. For instance,\n[1] UniT: Multimodal Multitask Learning with a Unified Transformer \nRonghang Hu Amanpreet Singh CVPR 2021\n[2] Are Multimodal Transformers Robust to Missing Modality?\nMengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, Xi Peng CVPR 2022 \nThe submission would be much stronger if the authors would discuss their contribution relative to these works, and possibly compare empirically their architecture to these ones using the same benchmarks\n- Related to the point above, there should be at least discussion about the recent trend to turn every task to text generation using a large language model (LLM), and multimodal multitask system leveraging LLMs. For instance:\n[3] Flamingo: a Visual Language Model for Few-Shot Learning\nJean-Baptiste Alayrac, Jeff Donahue et al. NeurIPS 2022\n- Related to transformers, the choice of using a RNN seems rather unnatural because it enforces an artificial ordering over modalities. Instead transformers operate on sets (of any size). I wonder whether the authors have considered replacing the RNN with a transformer.\n- The paper lacks clarity. The method should be described in a more formal way to better understand the implementation details. For instance, I am unclear:\n  - whether decoder parameters are shared across the different modalities,\n  - how the decoders predictions are combined across modalities/modules,\n  - what parameters are actually subject to training,\n  - what happens in the encoder when a modality is missing\n- The empirical results are weak. In fact, the proposed method often times works worse than the simple fusion baseline.\n- It would be nice if the authors demonstrated the benefits of modularity, for instance by:\n  - adding a new task or modality over time\n  - improving a task module with a subset of modalities at training time, the task improves also when at test time other modalities are used, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**SUMMARY:**  \nWe kindly thank the reviewer for the careful evaluation of our work and greatly appreciate the review. We address each concern in the point-by-point response below and feel that the resulting edits have further improved the manuscript.\n\n**STRENGTHS:**  \nWe are happy that the reviewer feels the paper covers an interesting topic that is very relevant to NeurIPS. We are also glad that the reviewer agrees on the importance of making multimodal models resilient to the common issue of systematic missingness and that they find our approach sensible and intuitive.\n\n**WEAKNESSES:**  \nWe agree with the reviewer’s opinion that our submission would be much stronger if compared to **further multimodal multitask baselines, particularly transformers in the vision/language domain**. We had actually tested several baselines (e.g., BiLSTMs, various compositions of P-Fusion), none of which improved upon MultiMoDN. For readability, they were excluded. We now include a transformer baseline selected with hyperparameter tuning on all 3 datasets. The results are found in Table 1 of the PDF and will be included in the final submission. \n\nThe results indicate that MultiModN outperforms or at least matches the transformer benchmark in the vast majority of single and multitask settings, and comes with several interpretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for classification and MSE for regression tasks), MultiModN beats the transformer baseline significantly in 7 tasks, overlaps 95% CIs in 11 tasks, and loses very slightly (by 0.01) in 2 regression tasks.\n\nWe agree with the reviewer that a **larger discussion on the recent multimodal trends using transformers and LLMs** would improve the paper. UNiT is a promising multimodal, multitask transformer architecture, however, it remains monolithic, trained on the union of all inputs (padded when missing) inputted into the model in parallel. This risks exposing the model to systematic missingness during training and reduces model interpretability (requiring all modalities to be represented even if not present) and portability (the transformer has 427,421 trainable params for EDU while MultiModN achieves better performance with 12,159). [1]’s recent work has found similar results on the erratic behavior of transformers to missing modalities, but is only tested on visual/text inputs.\n\nThe recommendation of the LLM approach Flamingo is interesting but is also limited to only 2 modalities (visual/text). It is not clear how tabular and time series would be handled or how this would affect the context window at inference. Combining predictive tasks with LLMs will also greatly impact interpretability, introducing hallucinations and creating a model complexity that may use learned text bias to influence predictions. \n\n**The reviewer wonders whether transformers are superior as they don’t impose an order which may be “unnatural”.** MultiMoDN can be completely order-invariant and idempotent as shown in [2] and only requires randomization during training to achieve this. For interpretability, we argue that sequential inference (in any order) is far superior to parallel input due to its decomposability: allowing the user to visualize the effect of each input and aligning with Bayesian reasoning. \n\nWe will present this improved discussion of our positioning in relation to recent multimodal transformers and LLMs in the final paper.\n\nThe reviewer makes excellent recommendations to improve the **clarity of our paper and the formalism of our architecture**. We have integrated all of them, specifically detailing that:\n  - Decoder parameters are indeed shared across the different modalities.  \n  - The decoder predictions are combined across modalities/modules by averaging the loss. It is interesting to note that a weighted loss scheme could force the model to emphasize certain tasks over others.  \n  - All encoder and decoder parameters are subject to training, except for the pretrained encoders used to generate embeddings in the MIMIC dataset (used to replicate the exact baseline parallel fusion setting).  \n  - When a modality is missing, the encoder is skipped and not trained. (This is also depicted in Figure 1 of the original submission).  \n\nThe reviewer claims that the **empirical results are weak because our model is not superior to the baseline in terms of performance**.\nIt is critical to understand that we (purposely) do not aim to improve on the baseline in terms of performance. Rather, we limit the model’s ability to do so, by aligning the pre-extracted features to isolate the comparison of the fusion step. Multimodal multi-task models often have degraded performance compared to their single modality counterparts [3]. We argue that our model is by far superior to the baseline by virtue of being modular, interpretable, composable, robust to systematic missingness and multi-task without impacting performance.\n\nThe reviewer would like to see more support for our method’s performance using **various numbers and combinations of inputs at inference**. This is an excellent suggestion. We now include the results of our model on various numbers and combinations of inputs (see Figure 1 of the attached PDF). The baseline would have to impute missing features in all of these combinations, exposing it to catastrophic failure in the event of systematic missingness (Section 6.3).\n\n**LIMITATIONS:**  \nWe are happy that the reviewer feels we have appropriately addressed all limitations in the paper.\n\n**REFERENCES:**   \n[1] Ma, Mengmeng, et al. \"Are multimodal transformers robust to missing modality?\" CVPR 2022.   \n[2] Trottet, Cecile, et al. \"Modular Clinical Decision Support Networks (MoDN)—Updatable, interpretable, and portable predictions for evolving clinical environments.\" PLOS 2023.  \n[3] Liu, Shengchao, et al. \"Loss-balanced task weighting to reduce negative transfer in multi-task learning.\" AAAI 2019."}, {"review_id": "S1dqTxM3IT", "reviewer": "Reviewer_mEsN", "summary": "This paper presents MultiModN, a modular network that can deal with multimodal multitask problems and is inherently interpretable. MultiModN architecture consists of one encoder module for each modality, and one decoder module for each task. Each encoder module takes in a previous state and one modality input, and outputs the next state; the final state is obtained by sequentially passing the state through the encoder for each available modality (therefore this model can deal with data points with missing modalities as well).  The authors conducted experiments on 10 tasks across 3 real-world domains with vastly different modalities. MultiModN was compared to P-fusion, and both models used the exact same preprocessing steps and feature extraction to ensure fairness. The experiments showed that MultiModN is able to achieve similar performance compared to P-fusion while being inherently interpretable. Moreover, the authors conducted additional experiments that intentionally correlates missing modalities and certain labels in the training set, and shows that MultiModN is less affected by the correlation than P-Fusion (i.e. more robust against MNAR).", "questions": "Q1.  Since all encoders are dense-layers (I assume that means MLPs), is it correct that the inputs to the encoders are the features extracted from each modality, which is always a single vector?\n\nQ2. The only part of the paper that I am completely lost on is line 149-151 \"Extension to time series\". I understand that the M encoders are applied sequentially, but what does that have to do with time series? For time-series modality, doesn't the feature extractor turn it into a single vector such that it can go through the dense layers of the encoders (as in Q1)? I also don't see how this \"extension\" is relevant to any other part of the paper.", "limitations": "The authors have adequately addressed the limitations. \n\nI believe there is no foreseeable potential negative social impact in this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper studies the important problem of building robust and interpretable models for solving multimodal real-world tasks. Although the new model did not improve performance, improving model interpretability and robustness to spurious correlations is also extremely important for real-world settings where mistakes can have real impacts.\n\n2. The experiments are done in a very well controlled setting. The authors purposefully used the exact same module architectures and feature extractors as the baseline to ensure fair comparison, and confidence intervals are always included. The experiments convincingly showed that the model is indeed inherently interpretable, and that the model is robust to MNAR compared to P-fusion and does not perform worse than P-fusion overall.\n\n3. The paper presentation is generally quite good. The methodology description is clear and detailed, and the experiment details are all included. The figures and tables are nice looking and easy to understand. ", "weaknesses": "The experiments are restricted to the exact same settings and pre-processing steps as P-fusion. While this ensures the fairness of comparison, it also means that we don't have evidence on whether this modular approach will also be able to achieve similar performance or alleviate MNAR problem if we have a different baseline approach / pre-processing. Perhaps a small proof-of-concept experiment showing that this approach can also work under different settings will greatly improve the significance and impact of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**SUMMARY:**  \nWe kindly thank the reviewer for this careful evaluation of our work and greatly appreciate the positive review. We strongly agree with all points raised.\nWe have addressed each in a point-by-point response below and feel that the resulting edits have further improved the manuscript.\n\n**STRENGTHS:**\nWe are happy that the reviewer feels the paper addresses the important problem of multimodality in real-world settings. We are additionally encouraged that the reviewer believes the experiments are conducted in a “well-controlled setting” and that the paper presentation is strong with “clear and detailed descriptions”. We are immensely grateful that reviewer 3 recognizes the contributions of the paper as we intended!\n\n**WEAKNESSES:**  \nThe reviewer rightly explains that our **intentional alignment of architectures (to ensure fair comparison) purposely limits our ability to make a performance comparison to the baseline**, and views this as a weakness.\n\nTo provide insight into performance gains, we performed additional experiments as suggested to showcase the benefits of modularity with vastly different training and inference settings. The results of 30 new experiments of inference encoders, each performed with 5-fold cross-validation are included in Figure 1 of the rebuttal. We compare P-Fusion and our approach on both tasks of the MIMIC dataset using all possible combinations of four inputs modalities at test time. MultiModN ignores missing modalities whereas P-fusion imputes and therefore encodes missing modalities. \n\nWe note that the performance at inference for P-Fusion and MultiModN has no significant differences for all experiments (using 95% CI). Figure 1 shows that, on average, P-Fusion tends to overfit more to the most dominant (visual) modality. When this modality is missing (at random or completely at random), Multi-ModN performs better on a combination of the remaining modalities (demo, text, time series). In the case of missing modalities, the observed effect in Figure 1 is weak – confidence intervals overlap. Considering the MNAR (missing not at random) scenario described in Experiment 3 of the paper (Section 6.3), the difference becomes significant. \n\n**QUESTIONS:**\nThe reviewer had 2 questions:\n\n**Regarding the inputs of encoders**: yes in our case, the input is always a single vector and the encoders are always MLP. For the MIMIC dataset that contains images as an input modality, we use the pre-trained CNN to preprocess the data, which results in a vector. But nothing in the design of MultiModN restricts us to the use of MLP encoders that expects a 1D vector as an input. The pre-trained CNN could be included in the encoder as a frozen part, so that the encoder would receive images as input without any changes in the operations applied on the input. Any suitable architecture can be used as an encoder. We chose the presented simplified architecture to demonstrate the approach. We agree that this point needs to be clarified and made explicit. We will state explicitly that in our experiments, all encoders are MLP that expect 1D vectors as input.  \n\nThe second question of the reviewer **discusses the role and relevance of the \"extension to time series” section** in the architecture. We agree that the description of the time series setting of the MultiModN architecture (relevant in EDU and Weather) can be better described.  As the reviewer mentioned correctly, in the MIMIC setting, we show adding the time-series directly as a modality, which embeds the data as a single vector.   \n\nHowever, in real-world settings with data streams, real-time data is collected continuously over weeks and months at the same time the model is expected to inference. For example, the education version of MultiModN can make a prediction of student pass-fail success at week 1, week 2, week 3, etc. with new data from each week. MultiModN allows us to simply add the new week’s data (i.e. week 3) and update the predictions on all tasks (without requiring the model to inference again on week 1 or week 2 predictions). As described in the appendix, the “continuous” classification or regression tasks (Tasks 4 through 8) predict at every timestep with the new data from that timestep (each hour for the temperature forecasting and each week for the student success or dropout prediction).\n\n**LIMITATIONS:**  \nWe are happy that the reviewer feels we have appropriately addressed all limitations in the paper.\n"}]

Author rebuttal:
We are grateful to the reviewers for their careful evaluation and very happy to receive positive and high-quality reviews. We agree with all points raised and have been able to reflect and respond to each in detail.

We highlight a major concern of the only reviewer (R2) recommending rejection: our model does not show significant performance improvement compared with the baseline. This is an important misunderstanding, as we actually purposely limit our model performance to fairly compare the multimodal fusion step. At equivalent performance, our model architecture is by far superior to the baseline by virtue of being inherently modular, interpretable, composable, robust to systematic missingness, and multi-task. The aim was to achieve these advantages without impacting performance compared to the baseline, which does exhibit these advantages.

Indeed, R3 cites this approach as an important strength: *“The experiments are done in a very well controlled setting. The authors purposefully used the exact same module architectures and feature extractors as the baseline to ensure fair comparison, and confidence intervals are always included. The experiments convincingly showed that the model is indeed inherently interpretable, and that the model is robust to MNAR compared to P-fusion and does not perform worse than P-fusion overall.”*

**R1** provides a very positive review and has recommended acceptance, citing our *“convincing and sound experiments”* described in a *“well-written, organized, and clear”* manner. They have requested the following:
1. **More support for our method’s ability to use various numbers and combinations of inputs at inference.** We provide 30 new inference experiments in Figure 1 in the attached PDF. The experiments test all combinations of different inference settings from the training setting and show the expected/desired outcome, where our method is not significantly different from the baseline (all 95% CIs overlap) despite having the various advantages of modularity (robustness to systematic missingness, interpretability, portability).
2. **Further clarity on the experimental figures.** We have formatted Figures 3, 4, and 5 and Appendix Figure 11 to better highlight the difference between our model and the baseline. These improved figures are included as Figure 2 in the attached PDF (changes highlighted in yellow).

**R2** provides a generally positive review (*“interesting topic”*, *“relevant to NeurIPS”*, *“sensible and intuitive approach”*),  but recommends rejection. They cite several issues motivating this recommendation which have now been fully addressed:
1. **The reviewer recommends a transformer baseline and a commentary on multi-modal LLM approaches.** Based on the related work mentioned by R2, we discuss in detail how our work compares to other approaches (in vision and text). Notably, we have conducted additional experiments to now include a multitask, multimodal transformer baseline (found in Table 1 of the attached results), which is overall less performant than our architecture and does not have the advantages of interpretability or modularity. We further explain how LLMs are not well suited to the task at hand, as they greatly degrade interpretability and introduce the risk of hallucination from previously biased text inputs. 
2. **The reviewer asks for various clarifications regarding the model architecture** regarding decoder parameter sharing, decoder loss, trained parameters, and treatment of missing modalities. We have elaborated on each of these areas formally in our architecture section and would like to thank the reviewer for their careful perusal.

**R3** provided a positive review stating *“improving model interpretability and robustness to spurious correlations is extremely important for real-world settings where mistakes can have real impacts”* and recommends acceptance:
1. **The reviewer requests that we experiment with diverse train and test settings** beyond an exact comparison with P-Fusion, aligning with R1. Our new experiments in Figure 1 address this concern. 
2. **The reviewer requests architecture clarifications on the inputs of the encoders and the “extension to time series” section**. The input vectors in our experiments are 1D. Depending on the nature of the modalities in the dataset, we support other encoders (CNN, transformers, and other pretrained models) to determine the ideal encoding for each input/modality. For time-series, we present the real-world setting of a data stream, where inference takes place at the same time as data is being received (i.e. predicting student performance at each week of a course as the course is being conducted). These settings are relevant for continuous prediction tasks in EDU and Weather, showing how MultiModN can be used for incremental time series prediction. We will state these descriptions explicitly in the paper.

To summarize the additional experiments, we provide the following results to strengthen our paper in the PDF attachment:
- **New results on all 10 tasks across 3 datasets from a multimodal multitask transformer architecture** based on the related work highlighted by R2 (Table 1). These results show that MultiModN is comparable or better than the Transformer in both single task and multi-task settings, in addition to having advantages that are not present with the Transformer (size, modularity, interpretability).
- **30 new inference modality experiments comparing P-Fusion and MultiModN in different settings and tasks** (Figure 1). These indicate comparable or often better performance in contrast with P-Fusion across diverse inference settings.
- **Improved diagrams from the results sections** to clearly understand the difference between the baseline and MultiModN (Figure 2). 

Overall, we thank the reviewers for their thoughtful and expert advice. We hope that we have adequately addressed reviewer concerns and further improved confidence in our work.

Meta-review: The paper proposes a new multimodality model that leverages a sequential design to process different modalities. Reviewers generally appreciate the thoroughly controlled experiments, the importance of the problem, and the intuitive architecture design. The concern of reviewer iERT also seems to have been addressed in the rebuttal.
Decision: Accept (poster)