SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MultiModN—Multimodal, Multi-Task, Interpretable Modular Networks
Vinitra Swamy*
vinitra.swamy@epfl.ch
Malika Satayeva*
malika.satayeva@epfl.ch
Jibril Frej
jibril.frej@epfl.ch
Thierry Bossy
thierry.bossy@epfl.ch
Thijs Vogels
thijs.vogels@epfl.ch
Tanja Käser*
tanja.kaser@epfl.ch
Mary-Anne Hartley*
Yale, EPFL
mary-anne.hartley@yale.edu
Predicting multiple real-world tasks in a single model often requires a particularly diverse feature space. Multimodal (MM) models aim to extract the synergistic predictive potential of multiple data types to create a shared feature space with aligned semantic meaning across inputs of drastically varying sizes (i.e. images, text, sound). Most current MM architectures fuse these representations in parallel, which not only limits their interpretability but also creates a dependency on modality availability. We present MultiModN, a multimodal, modular network that fuses latent representations in a sequence of any number, combination, or type of modality while providing granular real-time predictive feedback on any number or combination of predictive tasks. MultiModN's composable pipeline is interpretable-by-design, as well as innately multi-task and robust to the fundamental issue of biased missingness. We perform four experiments on several benchmark MM datasets across 10 real-world tasks (predicting medical diagnoses, academic performance, and weather), and show that MultiModN's sequential MM fusion does not compromise performance compared with a baseline of parallel fusion. By simulating the challenging bias of missing not-at-random (MNAR), this work shows that, contrary to MultiModN, parallel fusion baselines erroneously learn MNAR and suffer catastrophic failure when faced with different patterns of MNAR at inference. To the best of our knowledge, this is the first inherently MNAR-resistant approach to MM modeling. In conclusion, MultiModN provides granular insights, robustness, and flexibility without compromising performance.
The world is richly multimodal and intelligent decision-making requires an integrated understanding of diverse environmental signals, known as embodied intelligence [1]. Until recently, advances in deep learning have been mostly compartmentalized by data modality, creating disembodied domains such as computer vision for images, natural language processing for text, and so on. Multimodal (MM) learning has emerged from the need to address real-world tasks that cannot be robustly represented by a single signal type as well as the growing availability and diversity of digitized signals [2, 3, 4]. Some examples are diagnosis from a combination of medical tests and imagery [5, 6, 7], estimating sentiment from facial expression, text, and sound [8, 9, 10, 11], and identifying human activities from a combination of sensors [12].
(a) MultiModN
(b) P-Fusion
Figure 1: Comparison of modular MultiModN (a) vs. monolithic P-Fusion (b). MultiModN inputs any number/combination of modalities (mod) into a sequence of mod-specific encoders (e). It can skip over missing modalities. A state (s) is passed to the subsequent encoder and updated. Each state can be fed into any number/combination of decoders (d) to predict multiple tasks. Modules are identified as grey blocks comprising an encoder, a state, and a set of decoders. P-Fusion is a monolithic model. It inputs a fixed number/combination of modalities (mod) into mod-specific encoders (e). Missing modalities are padded and encoded. Embeddings (emb) are concatenated and provided to a single decoder in parallel (d) to predict a single task.
The richer representations from synergistic data types also have the potential to increase the task space, where a single set of representations can generalize to several tasks. Multi-task (MT) learning has not only been shown to benefit the performance of individual tasks but also has the potential to greatly reduce computational cost through shared feature extraction [13].
In short, multimodal and multi-task learning hold significant potential for human-centric machine learning and can be summarized respectively as creating a shared feature space from various data types and deriving their relative semantic meaning across several tasks.
Limitations of current multimodal models. Current MM models propose a parallel integration of modalities, where representations are fused and processed simultaneously [2, 3, 4]. Parallel fusion (hereafter P-Fusion) creates several fundamental limitations that we address in this work.
The most important issue we seek to resolve in current MM architectures is their dependence on modality availability where all modalities for all data points are required inputs during both training and inference. Modality-specific missingness is a common real-world problem and can fundamentally bias the model when the missingness of a modality is predictive of the label (known as missing not-at-random, MNAR). The common solution of restricting learning to data points with a complete set of modalities creates models that perform inequitably in populations with fewer available resources (i.e. when the pattern of MNAR is different between train and test sets). In complex real-world datasets, there is often no intersection of complete availability, thus necessitating the exclusion of modalities or significantly limiting the train set. On the other hand, imputation explicitly featurizes missingness, thus risking to create a trivial model that uses the presence of features rather than their value for the prediction [14, 15]. The MNAR issue is particularly common in medicine, where modality acquisition is dependent on the decision of the healthcare worker (i.e. the decision that the model is usually attempting to emulate). For example, a patient with a less severe form of a disease may have less intensive monitoring and advanced imagery unavailable. If the goal is to predict prognosis, the model could use the missingness of a test rather than its value. This is a fundamental flaw and can lead to catastrophic failure in situations where the modality is not available for independent reasons (for instance resource limitations). Here, the featurized missingness would inappropriately categorize the patient in a lower severity class. For equitable real-world predictions, it is critical to adapt predictions to available resources, and thus allow composability of inputs at inference.
Another key issue of current techniques that this work addresses is model complexity. Parallel fusion of various input types into a single vector makes many post-hoc interpretability techniques difficult
or impossible [16]. Depending on where the fusion occurs, it may be impossible to decompose modality-specific predictive importance.
In this work, we leverage network modularization, compartmentalizing each modality and task into independent encoder and decoder modules that are inherently robust to the bias of MNAR and can be assembled in any combination or number at inference while providing continuous modality-specific predictive feedback.
Contributions. We propose MultiModN, a multimodal extension of the work of Trottet et al. [17], which uses a flexible sequence of model and task-agnostic encoders to produce an evolving latent representation that can be queried by any number or combination of multi-task, model-agnostic decoder modules after each input (showcased in Figure 1). Specifically, we demonstrate that our modular approach of sequential MM fusion:
[1] matches parallel MM fusion (P-Fusion) for a range of real-world tasks across several benchmark datasets, while contributing distinct advantages, such as being:
[2] composable at inference, allowing selection of any number or combination of available inputs,
[3] is robust to the bias of missing not-at-random (MNAR) modalities,
[4] is inherently interpretable, providing granular modality-specific predictive feedback, and
[5] is easily extended to any number or combination of tasks.
We provide an application-agnostic open-source framework for the implementation of MultiModN: https://github.com/epfl-iglobalhealth/MultiModN. Our experimental setup purposely limits our model performance to fairly compare the multimodal fusion step. At equivalent performance, our model architecture is by far superior to the baseline by virtue of being inherently modular, interpretable, composable, robust to systematic missingness, and multi-task.
Approaches to MM learning can be categorized by the depth of the model at which the shared feature space is created [2]. Late fusion (decision fusion) processes inputs in separate modality-specific sub-networks, only combining the outputs at the decision-level, using a separate model or aggregation technique to make a final prediction. While simple, late fusion fails to capture relationships between modalities and is thus not truly multimodal. Early fusion (feature fusion), combines modalities at the input level, allowing the model to learn a joint representation. Concatenating feature vectors is a popular and simple approach [18, 19], but the scale of deployment is particularly limited by the curse of dimensionality. Finally, intermediate fusion (model fusion) seeks to fine-tune several feature extraction networks from the parameters of a downstream classifier.
Parallel Multimodal Fusion (P-Fusion). Recently, Soenksen et al. [20] proposed a fusion architecture which demonstrated the utility of multiple modalities in the popular MM medical benchmark dataset, MIMIC [21, 22]. Their framework (HAIM or Holistic Artificial Intelligence in Medicine) generates single-modality embeddings, which are concatenated into a single one-dimensional multimodal fusion embedding. The fused embedding is then fed to a single-task classifier. This work robustly demonstrated the value of multimodality across several tasks and a rich combination of heterogeneous sources. HAIM consistently achieved an average improvement of  $6 - 33\%$  AUROC (area under the receiver operating characteristic curve) across all tasks in comparison to single-modality models. We use this approach as a P-Fusion baseline against our sequential fusion approach of MultiModN and extend it to several new benchmark datasets and multiple tasks.
Soenksen et al. [20] perform over 14,324 experiments on 12 binary classification tasks using every number and combination of modalities. This extreme number of experiments, was necessary because the model is not composable nor capable of multi-task (MT) predictions. Rather, a different model is needed for each task and every combination of inputs for each task. In contrast, MultiModN is an extendable network, to which any number of encoders and decoders can be added. Thus, most of the 14,324 experiments could technically be achieved within one MultiModN model.
Several other recent architectures utilize parallel fusion with transformers. UNiT (Unified Transformer) [23] is a promising multimodal, multi-task transformer architecture; however, it remains monolithic, trained on the union of all inputs (padded when missing) fed in parallel. This not only exposes the model to patterns of systematic missingness during training but also reduces model
interpretability and portability<sup>1</sup>. [24]'s recent work has found similar results on the erratic behavior of transformers to missing modalities, although it is only tested on visual/text inputs. LLMs have also recently been used to encode visual and text modalities [25], but it is not clear how tabular and time-series would be handled or how this would affect the context window at inference. Combining predictive tasks with LLMs will also greatly impact interpretability, introducing hallucinations and complex predictive contamination where learned textual bias can influence outcomes.
Modular Sequential Multimodal Fusion. A module of a modular model is defined as a self-contained computational unit that can be isolated, removed, added, substituted, or ported. It is also desirable for modules to be order invariant and idempotent, where multiple additions of the same module have no additive effect. We design MultiModN to encode individual inputs, whereby module-exclusion can function as input skippability, allowing missing inputs to be skipped without influencing predictions. Thus, modular models can have various input granularities, training strategies, and aggregation functions. Some popular configurations range from hierarchies with shared layers to ensemble predictions and teacher-trainer transfer learning approaches [26, 27].
We expand on the sequential modular network architecture proposed by Trotter et al.[17] called MoDN (Modular Decision Networks) as a means of sequential MM fusion. MoDN trains a series of feature-specific encoder modules that produce a latent representation of a certain size (the state). Modules can be strung together in a mix-and-match sequence by feeding the state of one encoder as an input into the next. Therefore, the state has an additive evolution with each selected encoder. A series of decoders can query the state at any point for multiple tasks from various combinations of inputs, giving MoDN the property of combinatorial generalization.
Thus, we extend MoDN to learn multiple tasks from multimodal inputs. By aligning feature extraction pipelines between MultiModN and the P-Fusion baseline (inspired by HAIM) we can achieve a better understanding of the impact of monolithic-parallel fusion vs. sequential-modular MM fusion. Figure 1 provides a comparison between P-Fusion and MultiModN, also formalized below.
Context. We propose a multi-task supervised learning framework able to handle any number or combination of inputs of varying dimension, irrespective of underlying bias in the availability of these inputs during training. We modularize the framework such that each input and task is handled by distinct encoder and decoder modules. The inputs represent various data modalities (i.e. image, sound, text, time-series, tabular, etc.). We assume that these inputs have synergistic predictive potential for a given target and that creating a multimodal shared feature space will thus improve model performance. The tasks represent semantically related observations. We hypothesize that jointly training on semantically related tasks will inform the predictions of each individual task.
Notation. Formally, given a set of modalities (features)  $\mathcal{M} = \{mod_1,\dots ,mod_{|\mathcal{M}|}\}$  and a set of tasks (targets)  $\mathcal{T} = \{task_1,\ldots ,task_{|\mathcal{T}|}\}$ , let  $\mathcal{X} = \{(x_1,y_1),(x_2,y_2),\ldots ,(x_N,y_N)\}$  represent a multimodal, multi-task dataset with  $N$  data points  $(x_{1},\ldots ,x_{N})$
Each point  $x$  has  $|\mathcal{M}|$  modalities (inputs):  $x = \{x_{mod_1},\ldots ,x_{mod_{|\mathcal{M}|}}\}$  and is associated with a set of  $|\mathcal{T}|$  targets (tasks):  $y = \{y_{task_1},\dots ,y_{task_{|\mathcal{T}|}}\}$ . Modalities comprise various sources (e.g. images from x-rays, CT), for simplicity, we consider sources and modalities as equal mod elements in  $\mathcal{M}$ .
Multimodal, multi-task, modular formulation. We decompose each data point  $x$  into  $|\mathcal{M}|$  sequential encoder modules specific to its constituent modalities and each target  $y$  into  $|\mathcal{T}|$  decoder modules specific to its constituent tasks such that any combination or number of modalities can be used to predict any combination or number of tasks. Our objective is to learn a set of function modules,  $\mathcal{F}$ . Each function module within this set, represented as  $f_{j}^{i} \in \mathcal{F}$  maps combinations of modalities  $\mathcal{M}_{j}$  to combinations of tasks  $\mathcal{T}_{i}$ , i.e.  $f_{j}^{i}: \mathcal{M}_{j} \to \mathcal{T}_{i}$ . It is important to note that  $\mathcal{M}_{j}$  is an element of the powerset of all modalities and  $\mathcal{T}_{i}$  is an element of the powerset of all tasks.
Extension to time-series. In the above formulation, the  $\mathcal{M}$  encoder modules are handled in sequence, thus naturally aligning inputs with time-series. While the formulation does not change for time-series data, it may be optimized such that  $f_{j}^{i}$  represents a single time step. This is relevant in the real-world
<sup>1</sup>The equivalent transformer architecture has 427,421 trainable parameters for the EDU dataset (Sec. 5) while MultiModN achieves better performance with 12,159 parameters.
setting of a data stream, where inference takes place at the same time as data is being received (i.e. predicting student performance at each week of a course as the course is being conducted). The continuous prediction tasks (shown for EDU and Weather in Sec. 6) demonstrate how MultiModN can be used for incremental time-series prediction.
4 MultiModN: Multimodal, Multi-task, Modular Networks (Our model)
Building on [17] (summarized and color-coded in Figure 1a), the MultiModN architecture consists of three modular elements: a set of State vectors  $\mathcal{S} = \{s_0, \dots, s_{|\mathcal{M}|}\}$ , a set of modality-specific Encoders  $\mathcal{E} = \{e_1, \dots, e_{|\mathcal{M}|}\}$ , and a set of task-specific Decoders  $\mathcal{D} = \{d_1, \dots, d_{|\mathcal{T}|}\}$ . State  $s_0$  is randomly initialized and then updated sequentially by  $e_i$  to  $s_i$ . Each  $s_i$  can be decoded by one, any combination, or all elements of  $\mathcal{D}$  to make a set of predictions. All encoder and decoder parameters are subject to training.
States  $(\mathcal{S})$ . Akin to hidden state representations in Recurrent Neural Networks (RNNs), the state of MultiModN is a vector that encodes information about the previous inputs. As opposed to RNNs, state updates are made by any number or combination of modular, modality-specific encoders and each state can be decoded by modular, task-specific decoders. Thus the maximum number of states by any permutation of  $n$  encoders is  $n!$ . For simplicity, we limit the combinatorial number of states to a single order (whereby  $e_i$  should be deployed before  $e_{i + 1}$ ) in which any number or combination of encoders may be deployed (i.e. one or several encoders can be skipped at any point) as long as the order is respected. Thus, the number of possible states for a given sample is equal to  $2^{|M|}$ . Order invariance could be achieved by training every permutation of encoders  $|\mathcal{M}|!$ , i.e. allowing encoders to be used in any order at inference, as opposed to this simplified implementation of MultiModN in which order is fixed. At each step  $i$ , the encoder  $e_i$  processes the previous state,  $s_{i - 1}$ , as an input and outputs an updated state  $s_i$  of the same size. When dealing with time-series, we denote  $s_{t(0)}$  as the state representing time  $t$  before any modalities have been encoded, and as  $s_{t(0,1,4,5)}$  as the state at time  $t$  after being updated by encoders  $e_1, e_4$  and  $e_5$ , in that order.
Encoders  $(\mathcal{E})$ . Encoders are modularized to represent a single modality, i.e.  $|\mathcal{E}| = |\mathcal{M}|$ . An encoder  $e_i$  takes as input the combination of a single modality (of any dimension) and the previous state  $s_{i-1}$ . Encoder  $e_i$  then outputs a  $s_i$ , updated with the new modality. For simplicity, we fix the state size between encoders. Due to modularization, MultiModN is model-agnostic, whereby encoders can be of any type of architecture (i.e. Dense layers, LSTM, CNN). For experimental simplicity, we use a single encoder design with a simple dense layer architecture. The input vectors in our experiments are 1D. When a modality is missing, the encoder is skipped and not trained (depicted in Figure 1).
Decoders  $(\mathcal{D})$ . Decoders take any state  $s_i$  as input and output a prediction. Each decoder is assigned to a single task, that is  $|\mathcal{D}| = |\mathcal{T}|$ , i.e. MultiModN is not multiclass, but multi-task (although a single task may be multiclass). Decoders are also model-agnostic. Our implementation has regression, binary, and multiclass decoders across static targets or changing time-series targets. Decoder parameters are shared across the different modalities. The decoder predictions are combined across modalities/modules by averaging the loss. Interestingly, a weighted loss scheme could force the model to emphasize certain tasks over others.
As shown in [17], MultiModN can be completely order-invariant and idempotent if randomized during training. For interpretability, sequential inference (in any order) is superior to parallel input due to its decomposability, allowing the user to visualize the effect of each input and aligning with Bayesian reasoning.
Quantification of Modularity. The modularity of a network can be quantified, whereby neurons are represented by nodes (vertices) and connections between neurons as edges. There are thus comparatively dense connections (edges) within a module and sparse connections between them. Partitioning modules is an NP-complete problem [28]. We present modules that are defined a priori, whereby a module comprises one encoder  $e_i$  connected to one state  $s_i$ , which is in turn connected to a set of  $|\mathcal{T}|$  tasks (a module is depicted as a grey box in Figure 1a). Following a formalization of modularity quantitation proposed by Newman et al. [29], we compute the modularity score for MultiModN and show that it tends to a perfect modularity score of 1 with each added modality and each added task. When viewed at the network granularity of these core elements, P-Fusion is seen as a monolithic model with a score of 0. The formula is elaborated in Appendix Sec. B.
4.1 P-Fusion: Parallel Multimodal Fusion (Baseline)
We compare our results to a recent multimodal architecture inspired by HAIM (Holistic AI in Medicine) [20]. As depicted in Figure 1b, HAIM also comprises three main elements, namely, a fixed set of modality-specific encoders  $\mathcal{E} = \{e_1,\dots ,e_{|\mathcal{M}|}\}$  which create a fixed set of embeddings  $\mathcal{B} = \{emb_1,\dots ,emb_{|\mathcal{M}|}\}$ , that is concatenated and fed into a single-task decoder  $(d_{1})$ . HAIM achieved state-of-the-art results on the popular and challenging benchmark MIMIC dataset, showing consistently that multimodal predictions were between  $6\%$  and  $33\%$  better than single modalities.
Encoders  $(\mathcal{E})$ . Contrary to the flexible and composable nature of MultiModN, the sequence of encoders in P-Fusion is fixed and represents a unique combination of modalities. It is thus unable to skip over modalities that are missing, instead padding with neutral values and explicitly embedding the non-missing modalities. The encoders are modality-specific pre-trained neural networks.
Embeddings (B). Multimodal embeddings are fused in parallel by concatenation.
Decoders  $(\mathcal{D})$ . Concatenated embeddings are passed to a single-task decoder.
Architecture alignment. We align feature extraction between MultiModN and P-Fusion to best isolate the effect of sequential (MultiModN) vs. parallel (P-Fusion) fusion. As depicted in Appendix Figure 8, we let MultiModN take as input the embeddings created by the P-Fusion pre-trained encoders. Thus both models have identical feature extraction pipelines. No element of the MultiModN pipeline proposed in Figure 1a is changed. The remaining encoders and decoders in both models are simple dense layer networks (two fully connected ReLU layers and one layer for prediction). Importantly, MultiModN encoders and decoders are model-agnostic and can be of any architecture.
5 Datasets
We compare MultiModN and P-Fusion on three popular multimodal benchmark datasets across 10 real-world tasks spanning three distinct domains (healthcare, education, meteorology). The healthcare dataset (MIMIC) is particularly challenging in terms of multimodal complexity, incorporating inputs of vastly varying dimensionality. Education (EDU) and Weather2k have a particular focus on time-series across modalities. Appendix Sec. C details features, preprocessing, and tasks  $(task_{1-10})$ .