Title: MultiMoDN—Multimodal, Multi-Task, Interpretable Modular Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Learning

Multimodal Learning

Multi-task learning

Missingness

Interpretability
Abstract: Predicting multiple real-world tasks in a single model often requires a particularly diverse feature space. Multimodal (MM) models aim to extract the synergistic predictive potential of multiple data types to create a shared feature space with aligned semantic meaning across inputs of drastically varying sizes (i.e. images, text, sound). Most current MM architectures fuse these representations in parallel, which not only limits their interpretability but also creates a dependency on modality availability. We present MultiModN, a multimodal, modular network that fuses latent representations in a sequence of any number, combination, or type of modality while providing granular real-time predictive feedback on any number or combination of predictive tasks. MultiModN's composable pipeline is interpretable-by-design, as well as innately multi-task and robust to the fundamental issue of biased missingness. We perform four experiments on several benchmark MM datasets across 10 real-world tasks (predicting medical diagnoses, academic performance, and weather), and show that MultiModN's sequential MM fusion does not compromise performance compared with a baseline of parallel fusion. By simulating the challenging bias of missing not-at-random (MNAR), this work shows that, contrary to MultiModN, parallel fusion baselines erroneously learn MNAR and suffer catastrophic failure when faced with different patterns of MNAR at inference. To the best of our knowledge, this is the first inherently MNAR-resistant approach to MM modeling. In conclusion, MultiModN provides granular insights, robustness, and flexibility without compromising performance.

MultiModN-Multimodal, Multi-Task, Interpretable Modular Networks

Abstract

Predicting multiple real-world tasks in a single model often requires a particularly diverse feature space. Multimodal (MM) models aim to extract the synergistic predictive potential of multiple data types to create a shared feature space with aligned semantic meaning across inputs of drastically varying sizes (i.e. images, text, sound). Most current MM architectures fuse these representations in parallel, which not only limits their interpretability but also creates a dependency on modality availability. We present MultiModN , a multimodal, modular network that fuses latent representations in a sequence of any number, combination, or type of modality while providing granular real-time predictive feedback on any number or combination of predictive tasks. MultiModN 's composable pipeline is interpretable-by-design, as well as innately multi-task and robust to the fundamental issue of biased missingness. We perform four experiments on several benchmark MM datasets across 10 real-world tasks (predicting medical diagnoses, academic performance, and weather), and show that MultiModN 's sequential MM fusion does not compromise performance compared with a baseline of parallel fusion. By simulating the challenging bias of missing not-at-random (MNAR), this work shows that, contrary to MultiModN , parallel fusion baselines erroneously learn MNAR and suffer catastrophic failure when faced with different patterns of MNAR at inference. To the best of our knowledge, this is the first inherently MNAR-resistant approach to MM modeling. In conclusion, MultiModN provides granular insights, robustness, and flexibility without compromising performance.

1 Introduction

The world is richly multimodal and intelligent decision-making requires an integrated understanding of diverse environmental signals, known as embodied intelligence [1]. Until recently, advances in deep learning have been mostly compartmentalized by data modality, creating disembodied domains such as computer vision for images, natural language processing for text, and so on. Multimodal (MM) learning has emerged from the need to address real-world tasks that cannot be robustly represented by a single signal type as well as the growing availability and diversity of digitized signals [2, 3, 4]. Some examples are diagnosis from a combination of medical tests and imagery [5, 6, 7], estimating sentiment from facial expression, text, and sound [8, 9, 10, 11], and identifying human activities from a combination of sensors [12].

𝑚𝑜𝑑

1

𝑒

1

𝑠

1j

𝑚𝑜𝑑

|M|

𝑒

|M|

𝑠

|M|j

, … ,

, … ,

𝑑

1

𝑑

|T|

𝑡𝑎𝑠𝑘

|T|

, … ,

module

𝑛

Multimodal

Encoders

State:

One for

each number &

combination of

inputs

Multi-task

Decoders

𝑡𝑎𝑠𝑘

1

𝑑

1

𝑑

|T|

, … ,

, … ,

𝑑

1

𝑑

|T|

module

1

𝑠

0j

TASKS:

Any number or combination of tasks

|T|

can be decoded

each/any step providing fine-grained, updatable  interpretability

MODALITIES:

Any number

or combination of different

modalities

|M|

(image,

sound, text…) can be

added in a composable

sequential pipeline

comprising available inputs

𝑚𝑜𝑑

missing

Missing inputs skipped

(a)

MultiModN

𝑒𝑚𝑏

1

𝑒𝑚𝑏

|M|

Multimodal

Encoders

Concatenated

embedding

Single-task

Decoder

(

fixed number)

𝑒𝑚𝑏

2

|D| = 1

Missing inputs

padded & encoded

𝑑

1

𝑒

1

𝑒

|M|

𝑒

2

𝑚𝑜𝑑

1

𝑚𝑜𝑑

|M|

𝑚𝑜𝑑

missing

𝑡𝑎𝑠𝑘

1

(b)

P-Fusion

The richer representations from synergistic data types also have the potential to increase the task space, where a single set of representations can generalize to several tasks. Multi-task (MT) learning has not only been shown to benefit the performance of individual tasks but also has the potential to greatly reduce computational cost through shared feature extraction [13].

In short, multimodal and multi-task learning hold significant potential for human-centric machine learning and can be summarized respectively as creating a shared feature space from various data types and deriving their relative semantic meaning across several tasks.

Limitations of current multimodal models. Current MM models propose a parallel integration of modalities, where representations are fused and processed simultaneously [2, 3, 4]. Parallel fusion (hereafter P-Fusion ) creates several fundamental limitations that we address in this work.

The most important issue we seek to resolve in current MM architectures is their dependence on modality availability where all modalities for all data points are required inputs during both training and inference. Modality-specific missingness is a common real-world problem and can fundamentally bias the model when the missingness of a modality is predictive of the label (known as missing not-atrandom, MNAR). The common solution of restricting learning to data points with a complete set of modalities creates models that perform inequitably in populations with fewer available resources (i.e. when the pattern of MNAR is different between train and test sets). In complex real-world datasets, there is often no intersection of complete availability, thus necessitating the exclusion of modalities or significantly limiting the train set. On the other hand, imputation explicitly featurizes missingness, thus risking to create a trivial model that uses the presence of features rather than their value for the prediction [14, 15]. The MNAR issue is particularly common in medicine, where modality acquisition is dependent on the decision of the healthcare worker (i.e. the decision that the model is usually attempting to emulate). For example, a patient with a less severe form of a disease may have less intensive monitoring and advanced imagery unavailable. If the goal is to predict prognosis, the model could use the missingness of a test rather than its value. This is a fundamental flaw and can lead to catastrophic failure in situations where the modality is not available for independent reasons (for instance resource limitations). Here, the featurized missingness would inappropriately categorize the patient in a lower severity class. For equitable real-world predictions, it is critical to adapt predictions to available resources, and thus allow composability of inputs at inference.

Another key issue of current techniques that this work addresses is model complexity . Parallel fusion of various input types into a single vector make many post-hoc interpretability techniques difficult

or impossible [16]. Depending on where the fusion occurs, it may be impossible to decompose modality-specific predictive importance.

In this work, we leverage network modularization, compartmentalizing each modality and task into independent encoder and decoder modules that are inherently robust to the bias of MNAR and can be assembled in any combination or number at inference while providing continuous modality-specific predictive feedback.

Contributions. We propose MultiModN , a multimodal extension of the work of Trottet et al. [17], which uses a flexible sequence of model and task-agnostic encoders to produce an evolving latent representation that can be queried by any number or combination of multi-task, model-agnostic decoder modules after each input (showcased in Figure 1). Specifically, we demonstrate that our modular approach of sequential MM fusion:

matches parallel MM fusion ( P-Fusion ) for a range of real-world tasks across several benchmark datasets, while contributing distinct advantages, such as being:

composable at inference , allowing selection of any number or combination of available inputs,

is robust to the bias of missing not-at-random (MNAR) modalities ,

is inherently interpretable , providing granular modality-specific predictive feedback, and

is easily extended to any number or combination of tasks .

We provide an application-agnostic open-source framework for the implementation of MultiModN : https://github.com/epfl-iglobalhealth/MultiModN . Our experimental setup purposely limits our model performance to fairly compare the multimodal fusion step. At equivalent performance, our model architecture is by far superior to the baseline by virtue of being inherently modular, interpretable, composable, robust to systematic missingness, and multi-task.

2 Background

Approaches to MM learning can be categorized by the depth of the model at which the shared feature space is created [2]. Late fusion (decision fusion) processes inputs in separate modality-specific sub-networks, only combining the outputs at the decision-level, using a separate model or aggregation technique to make a final prediction. While simple, late fusion fails to capture relationships between modalities and is thus not truly multimodal. Early fusion (feature fusion), combines modalities at the input level, allowing the model to learn a joint representation. Concatenating feature vectors is a popular and simple approach [18, 19], but the scale of deployment is particularly limited by the curse of dimensionality. Finally, intermediate fusion (model fusion) seeks to fine-tune several feature extraction networks from the parameters of a downstream classifier.

Parallel Multimodal Fusion ( P-Fusion ). Recently, Soenksen et al. [20] proposed a fusion architecture which demonstrated the utility of multiple modalities in the popular MM medical benchmark dataset, MIMIC [21, 22]. Their framework (HAIM or Holistic Artificial Intelligence in Medicine) generates single-modality embeddings, which are concatenated into a single onedimensional multimodal fusion embedding. The fused embedding is then fed to a single-task classifier. This work robustly demonstrated the value of multimodality across several tasks and a rich combination of heterogeneous sources. HAIM consistently achieved an average improvement of 6-33% AUROC (area under the receiver operating characteristic curve) across all tasks in comparison to single-modality models. We use this approach as a P-Fusion baseline against our sequential fusion approach of MultiModN and extend it to several new benchmark datasets and multiple tasks.

Soenksen et al. [20] perform over 14,324 experiments on 12 binary classification tasks using every number and combination of modalities. This extreme number of experiments, was necessary because the model is not composable nor capable of multi-task (MT) predictions. Rather, a different model is needed for each task and every combination of inputs for each task. In contrast, MultiModN is an extendable network, to which any number of encoders and decoders can be added. Thus, most of the 14,324 experiments could technically be achieved within one MultiModN model.

Several other recent architectures utilize parallel fusion with transformers. UNiT (Unified Transformer) [23] is a promising multimodal, multi-task transformer architecture; however, it remains monolithic, trained on the union of all inputs (padded when missing) fed in parallel. This not only exposes the model to patterns of systematic missingness during training but also reduces model

interpretability and portability 1 . [24]'s recent work has found similar results on the erratic behavior of transformers to missing modalities, although it is only tested on visual/text inputs. LLMs have also recently been used to encode visual and text modalities [25], but it is not clear how tabular and time-series would be handled or how this would affect the context window at inference. Combining predictive tasks with LLMs will also greatly impact interpretability, introducing hallucinations and complex predictive contamination where learned textual bias can influence outcomes.

Modular Sequential Multimodal Fusion. A module of a modular model is defined as a selfcontained computational unit that can be isolated, removed, added, substituted, or ported. It is also desirable for modules to be order invariant and idempotent, where multiple additions of the same module have no additive effect. We design MultiModN to encode individual inputs, whereby moduleexclusion can function as input skippablity , allowing missing inputs to be skipped without influencing predictions. Thus, modular models can have various input granularities, training strategies, and aggregation functions. Some popular configurations range from hierarchies with shared layers to ensemble predictions and teacher-trainer transfer learning approaches [26, 27].

We expand on the sequential modular network architecture proposed by Trottet et al.[17] called MoDN (Modular Decision Networks) as a means of sequential MM fusion. MoDN trains a series of feature-specific encoder modules that produce a latent representation of a certain size (the state ). Modules can be strung together in a mix-and-match sequence by feeding the state of one encoder as an input into the next. Therefore, the state has an additive evolution with each selected encoder. A series of decoders can query the state at any point for multiple tasks from various combinations of inputs, giving MoDN the property of combinatorial generalization.

Thus, we extend MoDN to learn multiple tasks from multimodal inputs. By aligning feature extraction pipelines between MultiModN and the P-Fusion baseline (inspired by HAIM) we can achieve a better understanding of the impact of monolithic-parallel fusion vs. sequential-modular MM fusion. Figure 1 provides a comparison between P-Fusion and MultiModN , also formalized below.

3 Problem formulation

Context. We propose a multi-task supervised learning framework able to handle any number or combination of inputs of varying dimension, irrespective of underlying bias in the availability of these inputs during training. We modularize the framework such that each input and task is handled by distinct encoder and decoder modules . The inputs represent various data modalities (i.e. image, sound, text, time-series, tabular, etc.). We assume that these inputs have synergistic predictive potential for a given target and that creating a multimodal shared feature space will thus improve model performance. The tasks represent semantically related observations. We hypothesize that jointly training on semantically related tasks will inform the predictions of each individual task.

Notation. Formally, given a set of modalities (features) M = { mod 1 , . . . , mod |M| } and a set of tasks (targets) T = { task 1 , . . . , task |T | } , let X = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x N , y N ) } represent a multimodal, multi-task dataset with N data points ( x 1 , . . . , x N ).

Each point x has |M| modalities (inputs): x = { x mod 1 , . . . , x mod |M| } and is associated with a set of |T | targets (tasks): y = { y task 1 , . . . , y task |T | } . Modalities comprise various sources (e.g. images from x-rays, CT), for simplicity, we consider sources and modalities as equal mod elements in M .

Multimodal, multi-task, modular formulation. We decompose each data point x into |M| sequential encoder modules specific to its constituent modalities and each target y into |T | decoder modules specific to its constituent tasks such that any combination or number of modalities can be used to predict any combination or number of tasks. Our objective is to learn a set of function modules , F . Each function module within this set, represented as f i j ∈ F maps combinations of modalities M j to combinations of tasks T i , i.e. f i j : M j →T i . It is important to note that M j is an element of the powerset of all modalities and T i is an element of the powerset of all tasks.

Extension to time-series. In the above formulation, the M encoder modules are handled in sequence, thus naturally aligning inputs with time-series. While the formulation does not change for time-series data, it may be optimized such that f i j represents a single time step. This is relevant in the real-world

setting of a data stream, where inference takes place at the same time as data is being received (i.e. predicting student performance at each week of a course as the course is being conducted). The continuous prediction tasks (shown for EDU and Weather in Sec. 6) demonstrate how MultiModN can be used for incremental time-series prediction.

4 MultiModN : Multimodal, Multi-task, Modular Networks (Our model)

Building on [17] (summarized and color-coded in Figure 1a), the MultiModN architecture consists of three modular elements: a set of State vectors S = { s 0 , . . . , s |M| } , a set of modality-specific Encoders E = { e 1 , . . . , e |M| } , and a set of task-specific Decoders D = { d 1 , . . . , d |T | } . State s 0 is randomly initialized and then updated sequentially by e i to s i . Each s i can be decoded by one, any combination, or all elements of D to make a set of predictions. All encoder and decoder parameters are subject to training.

States ( S ). Akin to hidden state representations in Recurrent Neural Networks (RNNs), the state of MultiModN is a vector that encodes information about the previous inputs. As opposed to RNNs, state updates are made by any number or combination of modular, modality-specific encoders and each state can be decoded by modular, task-specific decoders. Thus the maximum number of states by any permutation of n encoders is n ! . For simplicity, we limit the combinatorial number of states to a single order (whereby e i should be deployed before e i +1 ) in which any number or combination of encoders may be deployed (i.e. one or several encoders can be skipped at any point) as long as the order is respected. Thus, the number of possible states for a given sample is equal to 2 |M| . Order invariance could be achieved by training every permutation of encoders |M| ! , i.e. allowing encoders to be used in any order at inference, as opposed to this simplified implementation of MultiModN in which order is fixed. At each step i , the encoder e i processes the previous state, s i -1 , as an input and outputs an updated state s i of the same size. When dealing with time-series, we denote s t (0) as the state representing time t before any modalities have been encoded, and as s t (0 , 1 , 4 , 5) as the state at time t after being updated by encoders e 1 , e 4 and e 5 , in that order.

Encoders ( E ). Encoders are modularized to represent a single modality, i.e. |E| = |M| . An encoder e i takes as input the combination of a single modality (of any dimension) and the previous state s i -1 . Encoder e i then outputs a s i , updated with the new modality. For simplicity, we fix the state size between encoders. Due to modularization, MultiModN is model-agnostic, whereby encoders can be of any type of architecture (i.e. Dense layers, LSTM, CNN). For experimental simplicity, we use a single encoder design with a simple dense layer architecture. The input vectors in our experiments are 1D. When a modality is missing, the encoder is skipped and not trained (depicted in Figure 1).

Decoders ( D ). Decoders take any state s i as input and output a prediction. Each decoder is assigned to a single task, that is |D| = |T | , i.e. MultiModN is not multiclass, but multi-task (although a single task may be multiclass). Decoders are also model-agnostic. Our implementation has regression, binary, and multiclass decoders across static targets or changing time-series targets. Decoder parameters are shared across the different modalities. The decoder predictions are combined across modalities/modules by averaging the loss. Interestingly, a weighted loss scheme could force the model to emphasize certain tasks over others.

As shown in [17], MultiModN can be completely order-invariant and idempotent if randomized during training. For interpretability, sequential inference (in any order) is superior to parallel input due to its decomposability, allowing the user to visualize the effect of each input and aligning with Bayesian reasoning.

Quantification of Modularity. The modularity of a network can be quantified, whereby neurons are represented by nodes (vertices) and connections between neurons as edges. There are thus comparatively dense connections (edges) within a module and sparse connections between them. Partitioning modules is an NP-complete problem [28]. We present modules that are defined a priori , whereby a module comprises one encoder e i connected to one state s i , which is in turn connected to a set of |T | tasks (a module is depicted as a grey box in Figure 1a). Following a formalization of modularity quantitation proposed by Newman et al. [29], we compute the modularity score for MultiModN and show that it tends to a perfect modularity score of 1 with each added modality and each added task. When viewed at the network granularity of these core elements, P-Fusion is seen as a monolithic model with a score of 0. The formula is elaborated in Appendix Sec. B.

4.1 P-Fusion : Parallel Multimodal Fusion (Baseline)

We compare our results to a recent multimodal architecture inspired by HAIM (Holistic AI in Medicine) [20]. As depicted in Figure 1b, HAIM also comprises three main elements, namely, a fixed set of modality-specific encoders E = { e 1 , . . . , e |M| } which create a fixed set of embeddings B = { emb 1 , . . . , emb |M| } , that is concatenated and fed into a single-task decoder ( d 1 ). HAIM achieved state-of-the-art results on the popular and challenging benchmark MIMIC dataset, showing consistently that multimodal predictions were between 6% and 33% better than single modalities.

Encoders ( E ). Contrary to the flexible and composable nature of MultiModN , the sequence of encoders in P-Fusion is fixed and represents a unique combination of modalities. It is thus unable to skip over modalities that are missing, instead padding with neutral values and explicitly embedding the non-missing modalities. The encoders are modality-specific pre-trained neural networks.

Embeddings ( B ). Multimodal embeddings are fused in parallel by concatenation.

Decoders ( D ). Concatenated embeddings are passed to a single-task decoder.

Architecture alignment. We align feature extraction between MultiModN and P-Fusion to best isolate the effect of sequential ( MultiModN ) vs. parallel ( P-Fusion ) fusion. As depicted in Appendix Figure 8, we let MultiModN take as input the embeddings created by the P-Fusion pre-trained encoders. Thus both models have identical feature extraction pipelines. No element of the MultiModN pipeline proposed in Figure 1a is changed. The remaining encoders and decoders in both models are simple dense layer networks (two fully connected ReLU layers and one layer for prediction). Importantly, MultiModN encoders and decoders are model-agnostic and can be of any architecture.

5 Datasets

We compare MultiModN and P-Fusion on three popular multimodal benchmark datasets across 10 real-world tasks spanning three distinct domains (healthcare, education, meteorology). The healthcare dataset (MIMIC) is particularly challenging in terms of multimodal complexity, incorporating inputs of vastly varying dimensionality. Education (EDU) and Weather2k have a particular focus on time-series across modalities. Appendix Sec. C details features, preprocessing, and tasks ( task 1 -10 ).

MIMIC. MIMIC [30] is a set of deidentified electronic medical records comprising over 40 , 000 critical care patients at a large tertiary care hospital in Boston. The feature extraction pipeline replicated according to our baseline of P-Fusion [20], making use of patient-level feature embeddings extracted from pre-trained models as depicted in Appendix Figure 8. We select the subcohort of 921 patients who have valid labels for both diagnoses and all four modalities present. We use all four modalities as inputs: chest x-rays (image), chart events (time-series), demographic information (tabular), and echocardiogram notes (text). For simplicity, we focus on two diagnostic binary classification tasks: cardiomegaly ( task 1 ) and enlarged cardiomediastinum ( task 2 ). These tasks were selected for their semantic relationship and also because they were reported to benefit from multimodality [20]. Thus, we have four modality-specific encoders and two binary classification diagnostic decoders.

Education (EDU). This educational time-series dataset comprises 5 , 611 students with over 1 million interactions in a 10-week Massively Open Online Course (MOOC), provided to a globally diverse population. It is benchmarked in several recent works [31, 32, 33]. Our modeling setting is replicated from related literature, with 45 handcrafted time-series features regarding problem and video modalities extracted for all students at each weekly time-step [34]. We use two modalityspecific encoders (problem and video) and three popular decoder targets: binary classifiers ( task 3 -4 ) of pass/fail and drop-out, and a continuous target of next week's performance ( task 5 ) [35].

Weather2k. Weather2k is a 2023 benchmark dataset that combines tabular and time-series modalities for weather forecasting [36]. The data is extracted from 1 , 866 ground weather stations covering 6 million km 2 , with 20 features representing hourly interactions with meteorological measurements and three static features representing the geographical location of the station. We create five encoders from different source modalities: geographic (static), air, wind, land, and rain and align with the benchmark prediction targets [36] on five continuous regression targets: short (24 hr), medium (72 hr), long term (720 hr) temperature forecasting, relative humidity and visibility prediction ( tasks 6 -10 ).

(a)

MIMIC

(b)

EDU

(c)

Weather

1.0

0.8

rate

0.6

positive

Semantically related tasks

Semantically related tasks

Semantically related tasks

Task 1 - Cardiomegaly

Task 3 - Success Prediction (Pass/Fail)

Task 9 - Humidity

Task 2 - Enlarged Cardiomediastinum

Task 4 - Dropout next week (T/F)

Task 10 - Visibility

Baseline

Baseline

Baseline

0.2

=P-Fusion-Task 2 AUC=0.82±0.05

P-Fusion-Task 1 AUC=0.82±0.02

=PI-Fusion-Task 4 AUC=0.87±0.04

PI-Fusion-Task 3 AUC=0.92±0.02

PI-Fusion-Task 10 AUC=0.50±0.00

PI-Fusion-Task 9 AUC=0.50±0.00

MultiModN (single task)

MultiModN (single task)

MultiModN (single task)

MultiModN-Task 1 AUC=0.82±0.05

MultiModN-Task 3 AUC=0.93±0.03

MultiModN-Task 9 AUC=0.70±0.14

0.0

MultiModN-Task 2 AUC=0.76±0.06

MultiModN-Task 4 AUC=0.83±0.02

MultiModN-Task 10 AUC=0.61±0.09

0:0

0.2

0.4

0.6

0.8

1.0

0:0

0.2

0.4

0.6

80

1.0

0:0

0.2

0.4

0.6

0.8

1.0

False positive rate

False positive rate

False positive rate

6 Experiments

Overview. We align feature extraction pipelines between MultiModN and the P-Fusion baseline in order to isolate the impact of parallel-monolithic vs. sequential-modular fusion (described in 4.1 and depicted in Appendix Sec. B). We thus do not expect a significant difference in performance, but rather aim to showcase the distinct benefits that can be achieved with modular sequential multimodal fusion without compromising baseline performance . In the following subsections, we perform four experiments to show these advantages. [1] MultiModN performance is not compromised compared to P-Fusion in single-task predictions. [2] MultiModN is able to extend to multiple tasks, also without compromising performance. [3] MultiModN is inherently composable and interpretable, providing modality-specific predictive feedback. [4] MultiModN is resistant to MNAR bias and avoids catastrophic failure when missingness patterns are different between train and test settings.

Model evaluation and metrics. All results represent a distribution of performance estimates on a model trained 5 times with different random weight initializations for the state vector and weights. Each estimate uses a completely independent test set from an 80-10-10 K-Fold train-test-validation split, stratified on one or more of the prediction targets. We report metrics (macro AUC, BAC, MSE) with 95% confidence intervals, as aligned with domain-specific literature of each dataset [34, 20, 36].

Hyperparameter selection. Model architectures were selected among the following hyperparameters: state representation sizes [1, 5, 10, 20, 50, 100], batch sizes [8, 16, 32, 64, 128], hidden features [16, 32, 64, 128], dropout [0, 0.1, 0.2, 0.3], and attention [0, 1]. These values were grouped into 3 categories (small, medium, large). We vary one while keeping the others fixed (within groups). Appendix Figure 9 shows that MultiModN is robust to changing batch size, while dropout rate and hidden layers negatively impact larger models (possibly overfitting). The most specific parameter to MultiModN is state size. As expected, we see negative impacts at size extremes, where small states likely struggle to transfer features between steps, while larger ones would be prone to overfitting.

6.1 Exp. 1: Sequential modularization in MultiModN does not compromise performance

Setup. Asingle-task model was created for each task 1 -10 across all three datasets. Each model takes all modalities as input. We compare MultiModN and P-Fusion in terms of performance. AUROCs can be visualized in Figure 2 while BAC and MSE are detailed in Table 1. As feature extraction pipelines between MultiModN and P-Fusion are aligned, this experiment seeks to investigate if sequential modular fusion compromises model performance. To compress the multiple predictions of time-series into a single binary class, we select a representative time step (EDU tasks 3 -4 at 60% course completion) or average over all time steps (Weather tasks 9 -10 evaluated on a 24h window).

Results. Both MultiModN and P-Fusion achieve state-of-the-art results on single tasks using multimodal inputs across all 10 targets. In Figure 2c, we binarize the continuous weather task (humidity prediction) as an average across all time steps. The task is particularly challenging for the P-Fusion baseline, which has random performance (AUROC: 0.5). Compared with P-Fusion ,

[Baseline]

[Baseline]

[Baseline]

[Baseline]

MultiModN shows a 20% improvement, which is significant at the p < 0 . 05 level. As the temporality of this task is particularly important, it could be hypothesized that the sequential nature of MultiModN better represents time-series inputs. Nevertheless, all weather targets are designed as regression tasks and show state-of-the-art MSE scores in Table 1 where MultiModN achieves baseline performance.

We provide an additional parallel fusion transformer baseline with experimental results showcased in Appendix Sec. E.4. The results indicate that MultiModN matches or outperforms the multimodal transformer in the vast majority of single- and multi-task settings, and comes with several interpretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for classification and MSE for regression tasks), MultiModN beats the transformer baseline significantly in 7 tasks, overlaps 95% CIs in 11 tasks, and loses slightly (0.01) in 2 regression tasks.

MultiModN matches P-Fusion performance across all 10 tasks in all metrics reported across all three multimodal datasets. Thus, modularity does not compromise predictive performance.

6.2 Exp. 2: Multi-task MultiModN maintains baseline performance in individual tasks

Setup. The modular design of MultiModN allows it to train multiple task-specific decoders and deploy them in any order or combination. While multi-task models have the potential to enrich feature extraction (and improve the model), it is critical to note that all feature extraction from the raw input is performed before MultiModN is trained. MultiModN is trained on embeddings extracted from pretrained models (independently of its own encoders). This is done purposely to best isolate the effect of parallel-monolithic vs. sequential-modular fusion. We train three multi-task MultiModN models (one for each dataset, predicting the set of tasks in that dataset, i.e. tasks 1 -2 in MIMIC, tasks 3 -5 in EDU, and tasks 6 -10 in Weather) and compare this to 10 single-task MultiModN models (one for each tasks 1 -10 ). Monolithic models, like P-Fusion are not naturally extensible to multi-task predictions. Thus P-Fusion (grey bars in Figure 3) can only be displayed for single-task models. This experiment aims to compare MultiModN performance between single- and multi-task architectures to ensure that this implementation does not come at a cost to the predictive performance of individual tasks.

Results. In Figure 3 we compare the single-task P-Fusion (grey bars), to single- and multitask implementations of MultiModN (in color). The results demonstrate that MultiModN is able to maintain its performance across all single-prediction tasks even when trained on multiple tasks. We additionally include the results of our model on various numbers and combinations of inputs, described further in Appendix Sec. E.5. The baseline would have to impute missing features in these combinations, exposing it to catastrophic failure in the event of systematic missingness (Sec. 6.4).

MultiModN has the significant advantage of being naturally extensible to the prediction of multiple tasks without negatively impacting the performance of individual tasks.

6.3 Exp. 3: MultiModN has inherent modality-specific local and global model explainabilty

MultiModN

P-Fusion

Individual Modality Contributions: Global Interpretability Technique

(importance scores)

(+ all previous)

(+ all previous)

(+ all previous)

(+ all previous)

Cumulative Predictions: Local Interpretability Technique

(probabilities)

P-Fusion interpretability is not possible for individual modalities.

Prior

Tabular

Image

Text

Time Series

(Random)

(+ all previous)

(+ all previous)

(+ all previous)

(+ all previous)

Prior

Tabular

Image

Text

Time Series

(Random)

True

Label

…

0.9

0.1

0.0

1.0

Tabular

Image

Text

Time Series

1.0

0

Task 1

Task 2

Task 1

Task 2

Task 1

Task 2

0.5

MultiModN

P-Fusion

P-Fusion interpretability is not possible at intermediate steps.

■

Setup. Parallel MM fusion obfuscates the contribution of individual inputs and requires add-on or post hoc methods to reveal unimodal contributions and cross-modal interactions [37, 38, 39]. Soenksen et al. [20] used Shapley values [40] to derive marginal modality contributions. While these post hoc methods provide valuable insight, they are computationally expensive and challenging or impossible to deploy at inference. In contrast, MultiModN confers inherent modalityspecific interpretability, where the contribution of each input can be decomposed by module. We use task 1 -2 in MIMIC to compute two measures: [1] Importance score , where each encoder is deployed alone, providing predictive importance of a single modality by subtracting predictions made from the prior state. This can be computed across all data points or individual data points. [2] Cumulative probability , where the prediction from each multi-task decoder is reported in sequence (i.e. given the previously encoded modalities). We demonstrate this on a random patient from the test set, who has a true label of 0 for both tasks. Further plots are in Appendix Sec. E.2.

Results. Monolithic P-Fusion models cannot be decomposed into any modality-specific predictions, and its (single-task) prediction is only made after inputting all modalities. In contrast, Figure 4 shows MultiModN provides granular insights for both importance score and cumulative prediction. We observe that the Text modality is the most important. The cumulative prediction shows the prior strongly predicts positivity in both classes and thus that S 0 has learned the label prevalence.

The predictions naturally produced by MultiModN provide diverse and granular interpretations.

6.4 Exp. 4: MultiModN is robust to catastrophic failure from biased missingness

Setup. MultiModN is designed to predict any number or combination of tasks from any number or combination of modalities. A missing modality is skipped (encoder e i is not used) and not padded/encoded. Thus, MultiModN avoids featurizing missingness, which is particularly advantageous when missingness is MNAR. Featurizing MNAR can result in catastrophic failure when MNAR patterns differ between train and test settings. We demonstrate MultiModN 's inherent robustness to catastrophic MNAR failure by training MultiModN and P-Fusion on four versions of MIMIC with various amounts (0, 10, 50, or 80%) of MNAR by artificially removing one modality in one class only. Figure 5 compares MultiModN and P-Fusion on task 1 when tested in a setting that has either no missingess or where the MNAR pattern is different (i.e. label-flipped).

Results. Figure 5 shows a dramatic catastrophic failure of P-Fusion in a label-flipped MNAR test set ( black solid line ) compared with MultiModN . P-Fusion is worse than random at 80% MNAR (AUROC: 0.385). In contrast, MultiModN only loses 10% in MNAR flip, remarkably, matching performance in a test with no missingness. Further plots in Appendix E.3.

MultiModN is robust to catastrophic missingness (MNAR failure) where P-Fusion is not.

7 Conclusion

We present MultiModN , a novel sequential modular multimodal (MM) architecture, and demonstrate its distinct advantages over traditional monolithic MM models which process inputs in parallel.

By aligning the feature extraction pipelines between MultiModN and its baseline P-Fusion , we better isolate the comparison between modular-sequential MM fusion vs. monolithic-parallel MM fusion. We perform four experiments across 10 complex real-world MM tasks in three distinct domains. We show that neither the sequential modularization of MultiModN nor its extension to multi-task predictions compromise the predictive performance on individual tasks compared with the monolithic baseline implementation. Training a multitask model can be challenging to parameterize across inter- and cross-task performance [13, 41]. We perform no specific calibration and show that MultiModN is robust to cross-task bias. Thus, at no performance cost, modularization allows the inherent benefits of multi-task modeling, as well as providing interepretable insights into the predictive potential of each modality. The most significant benefit of MultiModN is its natural robustness to catastrophic failure due to differences in missingness between train and test settings. This is a frequent and fundamental flaw of many domains and especially impacts low-resource settings where modalities may be

Catastrophic Forgetting

(

P-Fusion

)

MNAR resistant

(

MultiModN

)

MultiModN

P-Fusion

Reaction to MNAR in train set

(Baseline)

(Baseline)

0.9

0.8

0.7

0.6

0.5

Test set with label flipped MNAR

MultiModN

Test set with no missingness

0.4

！

MultiModN

0

10

50

80

Percentage ofMNAR in train set,%

AUROC

missing for reasons independent of the missingness in the train set. More generally, modularization creates a set of self-contained modules, composable in any number or combination according to available inputs and desired outputs. This composability not only provides enormous flexibility at inference but also reduces the computational cost of deployment. Taken together, these features allow MultiModN to make resource-adapted predictions, which have a particular advantage for real-world problems in resource-limited settings.

Limitations and future work. The main limitation for studying MM modeling is the scarcity of large-scale, open-source, MM datasets that cover multiple real-world tasks, especially for time-series. Additionally, while MultiModN is theoretically able to handle any number or combination of modalities and tasks, this has not been empirically tested. Having a high combinatorial generalization comes at a computational and performance cost, where the 'memory' of a fixed-size state representation will likely saturate at scale. The performance of MultiModN is purposely limited in this work by fixing the feature extraction pipeline, to best isolate the effect of sequential fusion. Future work leveraging MultiModN model-agnostic properties would be able to explore the potential performance benefit. This is particularly interesting for time-series, for which the state 'memory' may need to be parameterized to capture predictive trends of varying shapes and lengths.

References

Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications , 2021.

Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019.

Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey. IEEE Access , 2019.

Arnab Barua. A systematic literature review on multimodal machine learning: Applications, challenges, gaps and future directions. IEEE Access , 2023.

Ming Fan, Wei Yuan, Wenrui Zhao, Maosheng Xu, Shiwei Wang, Xin Gao, and Lihua Li. Joint prediction of breast cancer histological grade and ki-67 expression level based on dce-mri and dwi radiomics. IEEE Journal of Biomedical and Health Informatics , 2020.

Kevin M Boehm, Pegah Khosravi, Rami Vanguri, Jianjiong Gao, and Sohrab P Shah. Harnessing multimodal data integration to advance precision oncology. Nature Reviews Cancer , 2022.

Julián N Acosta, Guido J Falcone, Pranav Rajpurkar, and Eric J Topol. Multimodal biomedical AI. Nature Medicine , 2022.

Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, and Jie Zhou. Bridging text and video: A universal multimodal transformer for audio-visual scene-aware dialog. IEEE / ACM Transactions on Audio Speech and Language Processing , 2021.

Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, and Amir Hussain. Multimodal sentiment analysis: Addressing key issues and setting up the baselines. IEEE Intelligent Systems , 2018.

Navonil Majumder, Devamanyu Hazarika, Alexander Gelbukh, Erik Cambria, and Soujanya Poria. Multimodal sentiment analysis using hierarchical fusion with context modeling. KnowledgeBased Systems , 2018.

Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. Empirical Methods in Natural Language Processing (EMNLP) , 2017.

Zeeshan Ahmad and Naimul Mefraz Khan. Human action recognition using deep multilevel multimodal ( M 2 ) fusion of depth and inertial sensors. IEEE Sensors Journal , 2020.

Michael Crawshaw. Multi-task learning with deep neural networks: A survey. ArXiv , 2020.

John W. Graham. Missing data analysis: Making it work in the real world. Annual Review of Psychology , 2009.

Tlamelo Emmanuel, Thabiso Maupong, Dimane Mpoeleng, Thabo Semong, Banyatsang Mphago, and Oteng Tabona. A survey on missing data in machine learning. Journal of Big Data , 2021.

Gargi Joshi, Rahee Walambe, and Ketan Kotecha. A Review on Explainability in Multimodal Deep Neural Nets. IEEE Access , 2021.

Cécile Trottet, Thijs Vogels, Kristina Keitel, Alexandra Kulunkina, Rainer Tan, Ludovico Cobuccio, Martin Jaggi, and Mary-Anne Hartley. Modular clinical decision support networks (MoDN)-updatable, interpretable, and portable predictions for evolving clinical environments. medRxiv , 2022.

Shih-Cheng Huang, Anuj Pareek, Saeed Seyyedi, Imon Banerjee, and Matthew P Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ Digital Medicine , 2020.

Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo. Multimodal machine learning in precision health: A scoping review. NPJ Digital Medicine , 2022.

Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas. Integrated multimodal artificial intelligence framework for healthcare applications. NPJ Digital Medicine , 2022.

Alistair EW Johnson, Lucas Bulgarelli, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. MIMIC-IV (version 1.0). PhysioNet , 2021.

Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chihying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. ArXiv , 2019.

Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1439-1449, 2021.

Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18177-18186, 2022.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716-23736, 2022.

Jonas Pfeiffer, Sebastian Ruder, Ivan Vuli´ c, and Edoardo Maria Ponti. Modular deep learning. ArXiv , 2023.

Mohammed Amer and Tomás Maul. A review of modularization techniques in artificial neural networks. Artificial Intelligence Review , 2019.

Ulrik Brandes, Daniel Delling, Marco Gaertler, Robert Gorke, Martin Hoefer, Zoran Nikoloski, and Dorothea Wagner. On modularity clustering. IEEE Transactions on Knowledge and Data Engineering , 2008.

M. E. J. Newman and M. Girvan. Finding and evaluating community structure in networks. Physical Review E , 2004.

Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation , 2000.

Vinitra Swamy, Bahar Radmehr, Natasa Krco, Mirko Marras, and Tanja Käser. Evaluating the Explainers: Black-box explainable machine learning for student success prediction in MOOCs. Educational Data Mining (EDM) , 2022.

Mina Shirvani Boroujeni and Pierre Dillenbourg. Discovery and temporal analysis of MOOC study patterns. Journal of Learning Analytics , 2019.

Mohammad Asadi, Vinitra Swamy, Jibril Frej, Julien Vignoud, Mirko Marras, and Tanja Käser. Ripple: Concept-based interpretation for raw time series models in education. AAAI Conference on Artificial Intelligence (EAAI) , 2023.

Vinitra Swamy, Mirko Marras, and Tanja Käser. Meta transfer learning for early success prediction in MOOCs. In ACM Conference on Learning at Scale (L@S) , 2022.

Cheng Ye and Gautam Biswas. Early prediction of student dropout and performance in MOOCs using higher granularity temporal information. Journal of Learning Analytics , 2014.

Xun Zhu, Yutong Xiong, Ming Wu, Gaozhen Nie, Bin Zhang, and Ziheng Yang. Weather2k: A multivariate spatio-temporal benchmark dataset for meteorological forecasting based on real-time observation data from ground weather stations. AISTATS , 2023.

Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, LouisPhilippe Morency, and Ruslan Salakhutdinov. Multiviz: Towards visualizing and understanding multimodal models. In International Conference on Learning Representations (ICLR) , 2023.

Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Torsten Wörtwein, Lisa Sheeber, Nicholas Allen, Jeffrey Cohn, and Louis-Philippe Morency. Beyond additive fusion: Learning non-additive multimodal interactions. Empirical Methods in Natural Language Processing (EMNLP) , 2022.

Benedek Rozemberczki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivér Kiss, Sebastian Nilsson, and Rik Sarkar. The shapley value in machine learning. In International Joint Conference on Artificial Intelligence , 2022.

Partoo Vafaeikia, Khashayar Namdar, and Farzad Khalvati. A brief review of deep multi-task learning and auxiliary task learning. ArXiv , 2020.

Vinitra Swamy, Sijia Du, Mirko Marras, and Tanja Kaser. Trusting the Explainers: Teacher validation of explainable artificial intelligence for course design. In LAK23: 13th International Learning Analytics and Knowledge Conference , pages 345-356, 2023.

Mirko Marras, Julien Tuang Tu Vignoud, Tanja Kaser, et al. Can feature predictive power generalize? benchmarking early predictors of student success across flipped and online courses. In Proceedings of the 14th International Conference on Educational Data Mining , pages 150-160, 2021.

Daniel FO Onah, Jane Sinclair, and Russell Boyatt. Dropout rates of massive open online courses: behavioural patterns. EDULEARN14 proceedings , pages 5825-5834, 2014.

Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. Negbio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Summits on Translational Science Proceedings , 2018:188, 2018.

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 590-597, 2019.

A MultiModN Framework Implementation

A task- and modality-agnostic open-source framework MultiModN solution has been implemented in Python using PyTorch as the primary machine learning framework. The /multimodn package contains the MultiModN model and its components. The /datasets package is responsible of preparing the data inputs for MultiModN . Some examples using the public Titanic dataset have been provided.

The code is available at: https://github.com/epfl-iglobalhealth/MultiModN .

Encoders

Encoder 1

2, 3, ..., T-1

EncoderT

Encoder

Encoder

ndus

2 inputs

(random)

Stateo

Encoder 1

State,

Encoder T

StateT

111

111

111

Target

Target

Target

Decoders

Decoders

Decoders

↑11

↑11

Targets

Targets

Targets

A.1 MultiModN metrics

During training and evaluation, the metrics of the model are stored in a log at each epoch in a matrix of dimensions ( E +1) ∗ D , where E is the number of encoders and D the number of decoders. Each row represents the metrics for a target at each state of the model.

A.2 Code structure

A.2.1 MultiModN

/multimodn package contains the MultiModN model and its modules:

A.2.2 Datasets

/dataset package contains the MultiModDataset abstract class, compatible with MultiModN .

Specific datasets are added in the /dataset directory and must fulfill the following requirements:

Contain a dataset class that inherit MultiModDataset or has a method to convert into a MultiModDataset

Contain a .sh script responsible of getting the data and store it in /data folder

namely a tuple containing an array of tensors representing the features for each subsequent encoder, a numpy array representing the different targets and optionally a numpy array giving the order in which to apply the encoders to the subsequent data tensors. Note: data and encoding_sequence must have the same length.

Missing values. The user is able to choose to keep missing values ( nan values). Missing values can be present in the tensors yielded by the dataset and are managed by MultiModN .

A.2.3 Pipelines

/pipeline package contains the training pipelines using MultiModN for Multimodal Learning. It follows the following steps:

Create MultiModDataset and the dataloader associated

Create the list of encoders according to the features shape of the MultiModDataset

Create the list of decoders according to the targets of the MultiModDataset

Init, train and test the MultiModN model

Store the trained model, training history and save learning curves

A.3 Quick start

Quick start running MultiModN on Titanic example pipeline with a Multilayer Perceptron encoder:

Open pipelines/titanic/titanic_mlp.png to look at the training curves.

B Additional details about MultiModN Architecture

d1

d2

ILlp

d1

d2

d1

d2

ILlp

d1

d2

S1

S2

S3

S|M|

e1

e2

e3

91

92

93

g|M)

Modularity. In the following, we detail the computation of MultiModN 's modularity measure. The total number of edges in a MultiModN module is |T | +1 . The total number of modules is |M| and there are |M| 1 edges connecting consecutive modules, which makes for a total number of edges in the entire MultiModN model of m = |M| ( |T | +2) -1 .

To compute the modularity following using the formalization proposed by Newman et al [29], we need to define groups. In the case of MultiModN , each group corresponds to one module . Let G be the matrix whose components g ij is the fraction of edges in the original network that connect vertices in group i to those in group j .

Within MultiModN , each group contains ( |T | +1) edges and is connected to adjacent groups by two edges, with the exception of g 1 and g |M| , which are connected to only one other group (cf. Figure 7).

Thus, G is a tridiagonal matrix whose diagonal elements G ii are equal to ( |T | +1) /m and whose upper and lower diagonal elements are equal to 1 /m :

The modularity measure is defined by Q = tr ( G ) -∥ G 2 ∥ , where tr ( G ) , is the trace of G and ∥ G 2 ∥ is the sum of elements of G 2 . The trace of G is equal to |M| m ( T +1) and we have G 2 equal to:

Hence, we have:

It is important to note that when M increases, the trace will tend to ( |T | +1) / ( |T | +2) ≈ 1 for a large number of tasks. Moreover, when |M| increases || G 2 || will decrease towards to 0. Thus, for a large number of tasks, we have a modularity measure that increases and tends to 1 with the number of modalities.

Architecture alignment between P-Fusion and MultiModN . We purposely align the feature extraction pipelines of P-Fusion and MultiModN in order to best isolate the effect of monolithicparallel fusion vs. modular-sequential fusion. In Figure 8, we see how the alignment is limited to the input, where both MultiModN and P-Fusion share the feature-extraction of each modality, where the MultiModN encoders receive an embedding ( emb ). No element of MultiModN is changed as described in Figure 1. Embeddings from missing data can be skipped (i.e. not encoded) by MultiModN .

𝑚𝑜𝑑

1

𝑒

1

𝑒𝑚𝑏

1

𝑒

2

𝑚𝑜𝑑

|

𝑚

|

𝑒

n

𝑒𝑚𝑏

n

Multimodal

Encoders

Separate

embeddings

𝑒𝑚𝑏

2

𝑚𝑜𝑑

missing

Missing embeddings skipped

P-fusion

creates

embeddings that

MultiModN takes

in sequence

(instead of

concatenating)

MultiModN

Architecture

unchanged

𝑡𝑎𝑠𝑘

|

𝒯

|

, … ,

𝑡𝑎𝑠𝑘

1

𝑒

1

𝑠

1j

𝑒

|M|

𝑠

|M|j

, … ,

, … ,

𝑑

1

𝑑

|T|

module

𝑛

Multimodal

Encoders

State:

One for

each number &

combination of

inputs

Multi-task

Decoders

, … ,

𝑑

1

𝑑

|T|

module

1

𝑡𝑎𝑠𝑘

|

𝒯

|

, … ,

𝑡𝑎𝑠𝑘

1

For MIMIC, feature extraction for each modularity is replicated from previous work [20] and we use embeddings generated from a set of pre-trained models. For Weather and Education, as there were no pre-existing embedding models, we design autoencoders trained to reconstruct the original input features from a latent space. We keep the autoencoder's encoder and decoder structure exactly aligned with the encoders of MultiModN (two ReLU activated, fully-connected Dense layers and a third layer either generating the state representation with a ReLU activation or the final prediction with a sigmoid activation). We also align the number of trainable parameters with MultiModN 's modality-based encoders for a fair baseline comparison by selecting an appropriate state representation size per modality to equal the state representation in MultiModN . The remaining hyperparameters are left exactly the same (batch size, hidden layer size, dropout rate, optimization metrics, loss function).

C Datasets and Tasks

A description of all tasks is provided in Table 2. In the following paragraphs, we detail the preprocessing decisions on the datasets for context and reproducibility.

MIMIC. The data includes four input modalities (tabular, textual, images, and time-series) derived from several sources for each patient. We align our preprocessing pipeline exactly with the study from which our baseline of P-Fusion is derived [20] (described in 4.1). To this end, we use patient-level feature embeddings extracted by the pre-trained models described in [20] and depicted in Figure 8.

The dataset is a combination of two MIMIC databases: MIMIC-IV v.1.0 [21] and MIMIC-CXR-JPG v.2.0.0 [22]. After gaining authorized access to PhysioNet online repository [30], the embedding dataset can be downloaded via this link: https://physionet.org/content/haim-multimodal/ 1.0.1/ . The dataset comprises 45 , 050 samples, each corresponding to a time point during a patient's hospital stay when a chest X-ray was obtained. It covers a total of 8 , 655 unique patient stays. To ensure data quality and limit our experiments to two thematic tasks (diagnosis of task 1 : cardiomegaly and task 2 : enlarged cardiomediastinum), we remove duplicates (based on image id and image acquisition time), and retain only relevant samples that have valid labels for both targets of interest,

i.e. both task 1 and task 2 are either present (1) or absent (0). Subsequent experiments for these tasks are thus performed on the 921 / 45 , 050 selected relevant patients.

EDU. This dataset involves hand-crafted features extracted for 5 , 611 students across 10 weeks of data. The preprocessing of data is an exact replication of several related works using the same dataset [34, 31, 42] based on 4 feature sets determined as predictive for MOOC courses in [43]. 45 features regarding problem and video data are extracted per student per week, covering features like Delay Lecture , which calculates the average delay in viewing video lectures after they are released to students or TotalClicksProblem , the number of clicks that a student has made on problems this week. The features are normalized with min-max normalization, and missing values are imputed with zeros that have meaning i.e. no problem events in a week is correctly inferred as zero problems attempted. In this setting, missingness is a valued, predictive feature of the outcome, and thus we do not perform missingness experiments on this dataset. MultiModN has the ability to select whether missingenss is encoded or not, and thus it would not suffer a disadvantage in a setting where missingness should be featurized.

In MOOCs, a common issue is that students join a course and never participate in any assignments, homeworks, or quizzes. This could be due to registering aspirationally, to read some material, or to watch videos [44]. An instructor can easily classify students who have never completed an assignment as failing students. As introduced by Swamy et al. in [34] and used with this dataset in related work [31, 42, 33], EDU has removed students that were predicted to fail in the first two weeks simply by having turned in no assignments (99% confidence of failing with an out-of-the-box logistic regression model, where the confidence threshold was tuned over balanced accuracy calculations). It has been shown that including these students will artificially increase the performance of the model, providing even better results than those showcased by MultiModN in this work [34]. We thus exclude these students to test a more challenging modeling problem.

Weather. The Weather2k dataset, presented in [36], covers features from 1 , 866 weather stations with 23 features covering seven different units of measurements (degrees, meters, HPA, celsius, percentage, ms -1 , millimeters). To align these features on vastly different scales, we normalize the data. We use the large extract (R) provided by the authors instead of the smaller representative sample also highlighted in the benchmark paper (S) [36]. We use the first 24 hourly measurements as input to train the MultiModN model and calculate the five regression tasks as determined in Table 2 below.

Tasks. As showcased in Table 2, our evaluation covers 10 binary and regression tasks in two settings: static (one value per datapoint) or continuous (changing values per datapoint, per timestep).

Note that for task 4 , the three features used to calculate next week's performance were not included in the original input features because of possible data leakage, as student performance on quizzes directly contributes to their overall grade (Pass/Fail).

For the AUROC curves on tasks 9 -10 in Figure 2, we conduct a binarization of the two last regression tasks. To align with the regression task, we conduct a static forecasting prediction per station for the relative humidity or visibility with a time window of 24h for the 48th timestep (one day in advance). While MultiModN is capable of making a prediction at each continuous timestep, P-Fusion is not able to do this without a separate decoder at each timestep, and therefore to compare the tasks we must choose a static analysis. We choose a threshold based on the normalized targets: 0.75 for humidity and 0.25 for visibility (selected based on the distribution of the feature values for the first 1000 timesteps), and evaluate the predictions as a binary task over this threshold.

Analogously, in the Section E analysis below on the binarization of the remaining regression tasks, we express the continuous regression tasks for temperature forecasting tasks 6 -8 as a static binary task. To do this, we evaluate the prediction from the full window (24th timestep) at the respective forecasting timestep (48, 96, 744). The threshold we select is 0.3, closely corresponding to the normalized mean of the temperature (0.301) over the first 1000 timesteps.

D Model Optimization

Table 3 indicates the chosen hyperparameters for the experiments conducted in Section 6 of the paper, selected based on the optimal hyperparameters for the multi-task settings (all the tasks for a dataset predicted jointly). The single tasks use the same hyperparameters as the multi-task settings. For saving the best model across training epochs in the time series settings (EDU, Weather), our optimization metric saved the ones with best validation set results on the most short term task. Therefore, for EDU, we use MSE of task 5 , predicting next week performance, and task 6 for Weather, forecasting temperature within a day). The intuition is that choosing the best model on the short term task would allow the model to emphasize stronger short-term connections, which in turn would improve long term performance.

In Figure B, we examine a case study of MultiModN 's changing performance on task 3 , student success prediction in the EDU dataset, by varying hyperparameters across three model architectures (chosen for small, medium, and large hyperparameter initializations). We note that batch size is fairly robust across all three initial model settings, with large batch size on the largest model having slightly variable performance. Examining changing dropout rate, we note that with medium and large models, change in dropout impacts performance considerably. This allows us to hypothesize that high dropout on larger state representations does not allow the model to learn everything it can from the data.

Looking at varied hidden layer size, we see comparable performance for the small and medium initializations, but note that in the large case, having a smaller hidden layer size is important to maintain performance. Even as MultiModN performance trends upwards with larger hidden layer size (i.e. 128) for the large initialization, the confidence interval is large, so performance is not stable. Lastly, observing state representation size, we see that when state representation is too small for the task (i.e. 1, 5), the small and large models are adversly impacted. Additionally, when state representation is too large (i.e. 100), performance seems to drop or increase variability again. It is therefore important to tune MultiModN and find the right state representation size for the dataset and predictive task(s).

Small [32 batch - 16 hidden - 10 state]

Medium [64 batch - 32 hidden - 20 state]

Large [128 batch - 64 hidden - 30 state]

Small [16 hidden - 10 state - 0.2 dr]

Medium [32 hidden - 20 state - 0.2 dr]

Large [64 hidden - 30 state - 0.2 dr]

Small [32 batch - 10 state - 0.2 dr]

Medium [64 batch - 20 state - 0.2 dr]

Large [128 batch - 30 state - 0.2 dr]

Small [32 batch - 16 hidden - 0.2 dr]

Medium [64 batch - 32 hidden - 0.2 dr]

Large [128 batch - 64 hidden - 0.2 dr]

State Representation Size

BAC Score (avg over 5 seeds)

Experimental Setup. For the results reported in Sections 6.1, 6.2 and 6.4, we perform 5-fold stratified cross-validation with 80-10-10 train-validation-test split. Due to the timeseries nature of the EDU and Weather datasets, we orient the stratification the real labels associated with the longest-term task. For EDU, task 3 (end of course pass-fail prediction) and for Weather task 8 (30-day temperature forecasting) is chosen for the stratification split.

In an alternative approach, regarding the MIMIC dataset, a two-step procedure was implemented to address the imbalanced class ratios, given the absence of a prioritized task. Initially, a new dummy label was assigned to each sample, indicating positivity if both pathologies are present and negativity otherwise. Subsequently, a label was assigned to each unique hospital stay based on the aggregated labels from the first step. A hospital stay was considered positive if the number of times sample from that stay has been found positive is greater than or equal to the half of samples with the same hospital stay ID. The latter, as outlined in [20], ensured that no information was leaked on the hospital stay level during stratification.

Experiments were conducted using the same architecture in PyTorch (MIMIC) and TensorFlow (EDU, Weather), to provide multiple implementations across training frameworks for ease of use. We use an Adam optimizer with gradient clipping across all experiments.

E Additional Experiments

E.1 Single task

We present the binarized results (AUROC curves) for several additional regression tasks ( tasks 6 -8 for Weather) in Figure 10. The specific details of binarization are discussed above in Appendix

1.0

0.8

rate

0.6

Truepositive

Temperatureforecasting

Task 6 - 24 hrs

Task 7 - 72 hrs

0.4

Task 8 - 720 hrs

Baseline

PIF-Task 6 AUC=0.86±0.02

PIF-Task 7 AUC=0.81±0.02

0.2

PIF-Task 8 AUC=0.89±0.01

MultiModN (single task)

MultiModN-ST-Task 6AUC=0.71±0.19

MultiModN-ST-Task 7AUC=0.86±0.09

0.0

MultiModN-ST-Task 8 AUC=0.81±0.15

0:0

0.2

0.4

0.6

0.8

1.0

Falsepositiverate

Section C. We note that the confidence intervals overlap for P-Fusion and MultiModN over all Weather tasks. However, it is clear that the performance of MultiModN varies a lot (large CIs). This could be due to the design of the binarization as originally in the benchmark paper [36], this was introduced as a regression task. Another contributing factor to large CIs could be that the model was trained across all timesteps but only evaluated on one timestep for a comparable binarization. Despite these caveats, we can statistically conclude that P-Fusion and MultiModN performance are comparable on these additional tasks.

E.2 Interpretability

We perform an interpretability analysis for the EDU dataset, analogous to the local and global interpretability analysis on MIMIC from Figure 4. The global analysis (IMC) is conducted over all students for the first week of the course. We note interesting findings: specifically that problem interactions are more important for tasks 3 -4 while video interactions are more important for task 5 .

MultiModN

P-Fusion

Individual Modality Contributions: Global Interpretability Technique

(importance scores)

(+ all previous)

(+ all previous)

(+ all previous)

(+ all previous)

Cumulative Predictions: Local Interpretability Technique

(probabilities)

P-Fusion interpretability is not possible for individual modalities.

Prior

Tabular

Image

Text

Time Series

(Random)

(+ all previous)

(+ all previous)

Prior

Problem

Video

(Random)

True

Label

Problem

Video

1.0

0

Task 3

Task 4

0.5

MultiModN

P-Fusion

Task 5

Task 3

Task 4

Task 5

Task 3

Task 4

Task 5

(+ all previous)

(+ all previous)

Prior

Problem

Video

(Random)

1.0

0

0.5

P-Fusion interpretability is not

possible at intermediate steps.

The student selected for the CP local analysis passes the course (1 for task 3 ) and does not dropout (1 for task 4 ), but does not have strong performance in the next week ( ∼ 0 in task 5 ). This analysis is also conducted across the first week of course interactions. We see that P-Fusion cannot produce modality-specific interpretations and predicts the incorrect label. However, MultiModN is able to identify a changing confidence level across modalities, eventually ending on the right prediction for all tasks. The confidence for task 3 increases with the student's problem interactions and reduces for their video interactions. This could have potential for designing an intervention to improve student learning outcomes. We note that for task 5 , both the students' problem and video interactions contribute similarly to the prediction.

E.3 Missingness

We expand on the missingness experiments presented in 6.4. Here, we present further control experiments (training on data missing-at-random, MAR) in both MIMIC tasks ( task 1 : diagnosis of Cardiomegaly 12 and task 2 : diagnosis of Enlarged Cardiomediastinum 13).

In the first two subplots of each figure, both P-Fusion (black) and MultiModN (red) are trained on MNAR data and then evaluated either on a test set at risk of catastrophic failure (where the pattern of MNAR is label-flipped, first figure) or on a test set with no missingness. As can be seen in the first figures, P-Fusion suffers catastrophic failure in the MNAR flip, becoming worse than random when a single modality is missing at 80%, as opposed to MultiModN , which only decreases AUROC about 10%. When the test set has no missing values, P-Fusion and MultiModN are not significantly different, proving that the catastrophic failure of P-Fusion is due to MNAR. This is further confirmed in the last two plots of each task, where the models are trained on MAR data and evaluated on test sets either without missing values or MAR missingness.

1.0

-8'0

0.6

AUROC

0.4

0.2

P-Fusion

MultiModN

Test set with no missingness [trained on MNAR]

P-Fusion

MultiModN

MAR test [trained on MAR]

MultiModN

P-Fusion

Test set with no missingness [trained on MAR]

MultiModN

P-Fusion

Percentage of MNAR in train set, %

80

Percentage of MNAR in train set, %

80

Percentage of MAR in train set, %

80

0

Percentage of MAR in train set, %

80

1.0

0.8

0.6

AUROC

0.4

0.2

Test setwith label flipped MNAR[trained on MNAR]

Test set with no missingness [trained on MNAR]

MAR test [trained on MAR]

Test set with no missingness [trained on MAR]

P-Fusion

MultiModN

P-Fusion

MultiModN

MultiModN

P-Fusion

P-Fusion

MultiModN

0.0

0

Percentage of MNAR in train set,%

08

0

Percentage of MNAR in train set,%

08

0

Percentage of MAR in train set,%

80

0

Percentage ofMAR in train set,%

80

E.4 Comparison to a P-Fusion Transformer

Additional experiments with a Transformer have been conducted on 10 tasks across three datasets. Results are showcased below in two tables (left for tasks 1 -4 and right for tasks 5 -10 ) with 95% CIs.

The hyperparameter-tuned architecture (based on head size, number of transformer blocks, MLP units) for EDU and Weather is a transformer model with 4 transformer blocks, 4 heads of size 256, dropout of 0.25, MLP units 128 with dropout 0.4, batch size 64, trained for 50 epochs with cross-entropy loss. For MIMIC, the most performant (tuned) transformer architecture includes 2 transformer blocks with 3 heads of size 128, MLP units 32 with batch size 32. We train this architecture on each decoder

task individually and all tasks together for a total of 13 new models with the exact preprocessing steps as in the P-Fusion and MultiModN experiments. The results indicate that MultiModN often outperforms or at least matches the P-Fusion Transformer benchmark in the vast majority of single task and multi-task settings, and comes with several interpretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for the classification tasks and MSE for the regression tasks), MultiModN beats the Transformer baseline significantly in 7 tasks, overlaps 95% CIs in 11 tasks, and loses very slightly (by 0.01) in 2 regression tasks.

Classification

MIMIC

EDU

Regression

EDU

WEATHER

MultiModN

P-Fusion

Modalities

(Inference)

CM

Task

Task

Encoder

MSE

R2

Single

0.01 ± 0.01

0.21± 0.06

Weekly Perf.

Task

Encoder

BAC

F1

Multi

3.91 ±3.49

-649.05 ±506.01

Single

0.50±0.01

0.43± 0.02

Single

0.00± 0.01

0.80± 0.10

Cardiomegaly

Temp. (24h)

Multi

0.50±0.02

0.43 ± 0.03

Multi

0.00 ±0.01

0.75 ± 0.08

Single

0.50±0.02

0.44 ±0.03

Single

0.00 ±0.01

0.71 ±0.18

ECM

Temp. (72h)

Multi

0.50 ±0.01

0.42 ± 0.01

Multi

0.00±0.01

0.77 ±0.08

Single

0.50±0.01

0.40 ± 0.01

Single

0.00±0.01

0.82± 0.13

Success (P/F)

Multi

0.96 ± 0.01

0.95 ± 0.02

0.00 ±0.01

0.78 ±0.03

Single

0.83±0.22

0.83 ± 0.24

nodo

Humidity

Multi

0.28 ± 0.02

0.27±0.02

Multi

0.03±0.01

0.00 ±0.41

Single

0.02 ± 0.01

0.82 ±0.11

Visibility

Multi

0.05 ± 0.02

0.50 ± 0.12

E.5 Additional Inference Settings

CM

MultiModN

P-Fusion

Modalities

(Inference)

CM

ECM

CM

ECM

Task

0.47

MultiModN

P-Fusion

Modalities

(Inference)

CM

ECM

CM

Task

ECM

CM

ECM

CM

ECM

0.47

|demol

0.50

0.50

[text]

0.61

0.58

|s|

0.52

0.53

Ivisual]

0.75

0.75

|demol

®

0.50

0.51

|text]

0.59

0.52

|demoltextlts|

0.64

0.60

[ts]

0.53

0.54

|demolvisualltext]

0.75

0.75

|visual|

0.71

0.72

|demolvisuallts]

0.77

0.76

[demoltext]

0.55

0.57

.LM

0.74

0.75

|demolts|

OM

0.54

0.52

|demoltextlts|

0.56

0.51

|demolvisuall

0.74

0.76

|demolvisual|text]

0.72

0.68

[text]ts]

DM

0.61

0.59

|demolvisuallts|

0.73

0.70

[visualltext]

0.74

0.74

[visual/text/ts]

LM

0.71

0.67

[visual(ts]

.M

0.76

0.76

|demolvisualltext|ts]

0.74

0.75

[demoltext]

0.56

0.52

|demolvisualltext/ts]

0.71

0.68

|demolts]

OM

0.56

|demolvisuall

0.69

0.71

[text]ts]

DM

0.60

0.51

[visualltext]

0.71

0.68

[visuallts]

0.72

0.71

ECM

CM

ECM

CM

To provide insight into performance gains, we performed additional experiments to showcase the benefits of modularity with vastly different training and inference settings. The results of 30 new experiments of inference encoders, each performed with 5-fold cross-validation are included in Figure 15. We compare P-Fusion and MultiModN on both tasks of the MIMIC dataset using all possible combinations of four input modalities at test time. MultiModN ignores missing modalities whereas P-Fusion imputes and therefore encodes missing modalities.

Wenote that the performance at inference for P-Fusion and MultiModN has no significant differences for all experiments (using 95% CIs). Figure 15 shows that, on average, P-Fusion tends to overfit more to the most dominant (visual) modality. When this modality is missing (at random or completely at random), MultiModN performs better on a combination of the remaining modalities (demo, text, time series). In the case of missing modalities, the observed effect in Figure 15 is weak - confidence intervals overlap. Considering the MNAR (missing not-at-random) scenario described in Sec. 6.4, the difference becomes significant.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MultiModN, a modular network that can deal with multimodal multitask problems and is inherently interpretable. MultiModN architecture consists of one encoder module for each modality, and one decoder module for each task. Each encoder module takes in a previous state and one modality input, and outputs the next state; the final state is obtained by sequentially passing the state through the encoder for each available modality (therefore this model can deal with data points with missing modalities as well).  The authors conducted experiments on 10 tasks across 3 real-world domains with vastly different modalities. MultiModN was compared to P-fusion, and both models used the exact same preprocessing steps and feature extraction to ensure fairness. The experiments showed that MultiModN is able to achieve similar performance compared to P-fusion while being inherently interpretable. Moreover, the authors conducted additional experiments that intentionally correlates missing modalities and certain labels in the training set, and shows that MultiModN is less affected by the correlation than P-Fusion (i.e. more robust against MNAR).\n\nSTRENGTHS:\n1. This paper studies the important problem of building robust and interpretable models for solving multimodal real-world tasks. Although the new model did not improve performance, improving model interpretability and robustness to spurious correlations is also extremely important for real-world settings where mistakes can have real impacts.\n\n2. The experiments are done in a very well controlled setting. The authors purposefully used the exact same module architectures and feature extractors as the baseline to ensure fair comparison, and confidence intervals are always included. The experiments convincingly showed that the model is indeed inherently interpretable, and that the model is robust to MNAR compared to P-fusion and does not perform worse than P-fusion overall.\n\n3. The paper presentation is generally quite good. The methodology description is clear and detailed, and the experiment details are all included. The figures and tables are nice looking and easy to understand.\n\nWEAKNESSES:\nThe experiments are restricted to the exact same settings and pre-processing steps as P-fusion. While this ensures the fairness of comparison, it also means that we don't have evidence on whether this modular approach will also be able to achieve similar performance or alleviate MNAR problem if we have a different baseline approach / pre-processing. Perhaps a small proof-of-concept experiment showing that this approach can also work under different settings will greatly improve the significance and impact of this work.\n\nQUESTIONS:\nQ1.  Since all encoders are dense-layers (I assume that means MLPs), is it correct that the inputs to the encoders are the features extracted from each modality, which is always a single vector?\n\nQ2. The only part of the paper that I am completely lost on is line 149-151 \"Extension to time series\". I understand that the M encoders are applied sequentially, but what does that have to do with time series? For time-series modality, doesn't the feature extractor turn it into a single vector such that it can go through the dense layers of the encoders (as in Q1)? I also don't see how this \"extension\" is relevant to any other part of the paper.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations. \n\nI believe there is no foreseeable potential negative social impact in this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an architecture for multimodal multitask learning which is robust to missing modalities/tasks both at training and test time. This is composed of:\n- encoder specific modules\n- (assuming an ordering among modalities) a hidden states which depends on the output of a modality specific encoder, and which feds into the next encoder,\n- a decoder that takes the state and produced an output for each task.\nThe method is tested on three multimodal datasets, namely MIMIC, EDU and Weather2K.\n\nSTRENGTHS:\n- multimodal multitask learning is a very interesting topic which is very relevant to this venue\n- the idea to make a multimodal multitask system robust to missing modalities is clearly good\n- the overall motivation and intuition behind the proposed architecture is sensible and intuitive\n\nWEAKNESSES:\n- The major limitation is the lack of reference and comparison to other multimodal multitask approaches, particularly those based on transformers in the vision/language domain. For instance,\n[1] UniT: Multimodal Multitask Learning with a Unified Transformer \nRonghang Hu Amanpreet Singh CVPR 2021\n[2] Are Multimodal Transformers Robust to Missing Modality?\nMengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, Xi Peng CVPR 2022 \nThe submission would be much stronger if the authors would discuss their contribution relative to these works, and possibly compare empirically their architecture to these ones using the same benchmarks\n- Related to the point above, there should be at least discussion about the recent trend to turn every task to text generation using a large language model (LLM), and multimodal multitask system leveraging LLMs. For instance:\n[3] Flamingo: a Visual Language Model for Few-Shot Learning\nJean-Baptiste Alayrac, Jeff Donahue et al. NeurIPS 2022\n- Related to transformers, the choice of using a RNN seems rather unnatural because it enforces an artificial ordering over modalities. Instead transformers operate on sets (of any size). I wonder whether the authors have considered replacing the RNN with a transformer.\n- The paper lacks clarity. The method should be described in a more formal way to better understand the implementation details. For instance, I am unclear:\n  - whether decoder parameters are shared across the different modalities,\n  - how the decoders predictions are combined across modalities/modules,\n  - what parameters are actually subject to training,\n  - what happens in the encoder when a modality is missing\n- The empirical results are weak. In fact, the proposed method often times works worse than the simple fusion baseline.\n- It would be nice if the authors demonstrated the benefits of modularity, for instance by:\n  - adding a new task or modality over time\n  - improving a task module with a subset of modalities at training time, the task improves also when at test time other modalities are used, etc.\n\nQUESTIONS:\nSee comments above in the weakness section.\nThe major question is about 1) relation to prior work, particularly in the vision/language domain, 2) use of more standard benchmarks (in addition to the chosen ones), 3) use of RNN as opposed to transformer, 4) various clarifications of the approach.\n\nLIMITATIONS:\nNo concern.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a modular multimodal model that fuses latent representations in a sequence of modality to conduct a combination of predictive tasks. It utilizes a flexible sequence of model and task-agnostic encoders to produce an evolving latent representation for a combination of multi-task, model-agnostic decoder modules to conduct downstream tasks. The authors conduct experiments on benchmarks of different domains to show the effectiveness of the proposed method. The experiments show that the proposed method outperforms traditional monolithic multi-modal models.

STRENGTHS:
1. This paper is well-written and well-organized. The design of methods and experiments are clear and easy to understand.
2. The proposed method is novel and effective. The modules are task-agnostic and can be adapted on any number of tasks on different modalities.
3. The experiments are substantial and convincing. The proposed method shows significantly better performance compared with traditional methods.

WEAKNESSES:
1. It seems that for single tasks the proposed method cannot provide improvement compared with the baseline.
2. It would be helpful to support the claim that the proposed model can handle  any number/combination of tasks by adding more experiments of combinatorial tasks.

QUESTIONS:
1. The authors should make the figures in the experiments part more clear to see the comparison of the proposed methods and the baseline.

LIMITATIONS:
The authors have addressed the technical limitations in the paper. I do not find any negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07