SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MIMIC. MIMIC [30] is a set of deidentified electronic medical records comprising over 40,000 critical care patients at a large tertiary care hospital in Boston. The feature extraction pipeline replicated according to our baseline of P-Fusion [20], making use of patient-level feature embeddings extracted from pre-trained models as depicted in Appendix Figure 8. We select the subcohort of 921 patients who have valid labels for both diagnoses and all four modalities present. We use all four modalities as inputs: chest x-rays (image), chart events (time-series), demographic information (tabular), and echocardiogram notes (text). For simplicity, we focus on two diagnostic binary classification tasks: cardiomegaly  $(task_{1})$  and enlarged cardiemiastinum  $(task_{2})$ . These tasks were selected for their semantic relationship and also because they were reported to benefit from multimodality [20]. Thus, we have four modality-specific encoders and two binary classification diagnostic decoders.
Education (EDU). This educational time-series dataset comprises 5,611 students with over 1 million interactions in a 10-week Massively Open Online Course (MOOC), provided to a globally diverse population. It is benchmarked in several recent works [31, 32, 33]. Our modeling setting is replicated from related literature, with 45 handcrafted time-series features regarding problem and video modalities extracted for all students at each weekly time-step [34]. We use two modality-specific encoders (problem and video) and three popular decoder targets: binary classifiers  $(task_{3-4})$  of pass/fail and drop-out, and a continuous target of next week's performance  $(task_5)$  [35].
Weather2k. Weather2k is a 2023 benchmark dataset that combines tabular and time-series modalities for weather forecasting [36]. The data is extracted from 1,866 ground weather stations covering 6 million  $\mathrm{km}^2$ , with 20 features representing hourly interactions with meteorological measurements and three static features representing the geographical location of the station. We create five encoders from different source modalities: geographic (static), air, wind, land, and rain and align with the benchmark prediction targets [36] on five continuous regression targets: short (24 hr), medium (72 hr), long term (720 hr) temperature forecasting, relative humidity and visibility prediction ( $tasks_{6-10}$ ).
(a) MIMIC
(b) EDU
(c) Weather
Figure 2: MultiModN does not compromise performance in single-tasks. AUROC for six binary prediction tasks in (a) MIMIC, (b) EDU, and (c) Weather2k. Tasks predicted by P-Fusion are compared with MultiModN.  $95\%$  CIs are shaded.
Overview. We align feature extraction pipelines between MultiModN and the P-Fusion baseline in order to isolate the impact of parallel-monolithic vs. sequential-modular fusion (described in 4.1 and depicted in Appendix Sec. B). We thus do not expect a significant difference in performance, but rather aim to showcase the distinct benefits that can be achieved with modular sequential multimodal fusion without compromising baseline performance. In the following subsections, we perform four experiments to show these advantages. [1] MultiModN performance is not compromised compared to P-Fusion in single-task predictions. [2] MultiModN is able to extend to multiple tasks, also without compromising performance. [3] MultiModN is inherently composable and interpretable, providing modality-specific predictive feedback. [4] MultiModN is resistant to MNAR bias and avoids catastrophic failure when missingness patterns are different between train and test settings.
Model evaluation and metrics. All results represent a distribution of performance estimates on a model trained 5 times with different random weight initializations for the state vector and weights. Each estimate uses a completely independent test set from an 80-10-10 K-Fold train-test-validation split, stratified on one or more of the prediction targets. We report metrics (macro AUC, BAC, MSE) with  $95\%$  confidence intervals, as aligned with domain-specific literature of each dataset [34, 20, 36].
Hyperparameter selection. Model architectures were selected among the following hyperparameters: state representation sizes [1, 5, 10, 20, 50, 100], batch sizes [8, 16, 32, 64, 128], hidden features [16, 32, 64, 128], dropout [0, 0.1, 0.2, 0.3], and attention [0, 1]. These values were grouped into 3 categories (small, medium, large). We vary one while keeping the others fixed (within groups). Appendix Figure 9 shows that MultiModN is robust to changing batch size, while dropout rate and hidden layers negatively impact larger models (possibly overfitting). The most specific parameter to MultiModN is state size. As expected, we see negative impacts at size extremes, where small states likely struggle to transfer features between steps, while larger ones would be prone to overfitting.
6.1 Exp. 1: Sequential modularization in MultiModN does not compromise performance
Setup. A single-task model was created for each  $task_{1-10}$  across all three datasets. Each model takes all modalities as input. We compare MultiModN and P-Fusion in terms of performance. AUROCs can be visualized in Figure 2 while BAC and MSE are detailed in Table 1. As feature extraction pipelines between MultiModN and P-Fusion are aligned, this experiment seeks to investigate if sequential modular fusion compromises model performance. To compress the multiple predictions of time-series into a single binary class, we select a representative time step (EDU  $tasks_{3-4}$  at  $60\%$  course completion) or average over all time steps (Weather  $tasks_{9-10}$  evaluated on a 24h window).
Results. Both MultiModN and P-Fusion achieve state-of-the-art results on single tasks using multimodal inputs across all 10 targets. In Figure 2c, we binarize the continuous weather task (humidity prediction) as an average across all time steps. The task is particularly challenging for the P-Fusion baseline, which has random performance (AUROC: 0.5). Compared with P-Fusion,
MIMIC	Education (EDU)	Weather
Cardiomegaly	ECM	Success	Dropout	Next Week	Temp. (24h)	Temp (72h)	Temp (720h)	Humidity	Visibility
Metric	BAC	BAC	BAC	BAC	MSE	MSE	MSE	MSE	MSE	MSE
MultiModN	0.75 ±0.04	0.71 ±0.03	0.93 ±0.04	0.83 ±0.02	0.01 ±0.01	0.03 ±0.01	0.03 ±0.01	0.03 ±0.01	0.02 ±0.01	0.10 ±0.01
P-Fusion	0.75 ±0.02	0.69 ±0.03	0.92 ±0.03	0.87 ±0.05	0.01 ±0.01	0.02 ±0.01	0.03 ±0.01	0.02 ±0.01	0.03 ±0.01	0.08 ±0.02
Table 1: MultiModN does not compromise performance in single-tasks. Performance for binary and continuous prediction tasks in MIMIC, EDU, and Weather, comparing P-Fusion and MultiModN.  $95\%$  CIs are shown. ECM: Enlarged Cardiomediastinum, Temp: Temperature.
Figure 3: Multi-task MultiModN maintains baseline performance in individual tasks. Single- and multi-task MultiModN on the prediction of individual tasks, compared with the monolithic P-Fusion (can only be single-task). AUC for binary (left) and MSE for continuous (right). Error bars:  $95\%$  CIs.
MultiModN shows a  $20\%$  improvement, which is significant at the  $p < 0.05$  level. As the temporality of this task is particularly important, it could be hypothesized that the sequential nature of MultiModN better represents time-series inputs. Nevertheless, all weather targets are designed as regression tasks and show state-of-the-art MSE scores in Table 1 where MultiModN achieves baseline performance.
We provide an additional parallel fusion transformer baseline with experimental results showcased in Appendix Sec. E.4. The results indicate that MultiModN matches or outperforms the multimodal transformer in the vast majority of single- and multi-task settings, and comes with several interpretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for classification and MSE for regression tasks), MultiModN beats the transformer baseline significantly in 7 tasks, overlaps  $95\%$  CIs in 11 tasks, and loses slightly (0.01) in 2 regression tasks.
MultiModN matches P-Fusion performance across all 10 tasks in all metrics reported across all three multimodal datasets. Thus, modularity does not compromise predictive performance.
6.2 Exp. 2: Multi-task MultiModN maintains baseline performance in individual tasks
Setup. The modular design of MultiModN allows it to train multiple task-specific decoders and deploy them in any order or combination. While multi-task models have the potential to enrich feature extraction (and improve the model), it is critical to note that all feature extraction from the raw input is performed before MultiModN is trained. MultiModN is trained on embeddings extracted from pretrained models (independently of its own encoders). This is done purposely to best isolate the effect of parallel-monolithic vs. sequential-modular fusion. We train three multi-task MultiModN models (one for each dataset, predicting the set of tasks in that dataset, i.e.  $tasks_{1-2}$  in MIMIC,  $tasks_{3-5}$  in EDU, and  $tasks_{6-10}$  in Weather) and compare this to 10 single-task MultiModN models (one for each  $tasks_{1-10}$ ). Monolithic models, like P-Fusion are not naturally extensible to multi-task predictions. Thus P-Fusion (grey bars in Figure 3) can only be displayed for single-task models. This experiment aims to compare MultiModN performance between single- and multi-task architectures to ensure that this implementation does not come at a cost to the predictive performance of individual tasks.
Results. In Figure 3 we compare the single-task P-Fusion (grey bars), to single- and multi-task implementations of MultiModN (in color). The results demonstrate that MultiModN is able to maintain its performance across all single-prediction tasks even when trained on multiple tasks. We additionally include the results of our model on various numbers and combinations of inputs, described further in Appendix Sec. E.5. The baseline would have to impute missing features in these combinations, exposing it to catastrophic failure in the event of systematic missingness (Sec. 6.4).
MultiModN has the significant advantage of being naturally extensible to the prediction of multiple tasks without negatively impacting the performance of individual tasks.
6.3 Exp. 3: MultiModN has inherent modality-specific local and global model explainability
Figure 4: Inherent modality-specific model explainability in MultiModN. Heatmaps show individual modality contributions (IMC) (top) and cumulative contributions (CP) (bottom): respectively importance score (global explainability) or cumulative probability (local explainability). The multi-task MultiModN for  $task_{1-2}$  in MIMIC is compared to two single-task P-Fusion models. IMC are only possible for MultiModN (only 1 modality encoded, rest are skipped). CP are made sequentially from states encoding all previous modalities. P-Fusion is unable to naturally decompose modality-specific contributions (can only make predictions once all modalities are encoded). IMC is computed across all patients in the test set. CP is computed for a single patient, (true label = 0 for both  $task_{1-2}$ ). The CP heatmap shows probability ranging from confident negative diagnosis (0) to perfect uncertainty and confident positive diagnosis (1).
Setup. Parallel MM fusion obfuscates the contribution of individual inputs and requires add-on or post hoc methods to reveal unimodal contributions and cross-modal interactions [37, 38, 39]. Soenksen et al. [20] used Shapley values [40] to derive marginal modality contributions. While these post hoc methods provide valuable insight, they are computationally expensive and challenging or impossible to deploy at inference. In contrast, MultiModN confers inherent modality-specific interpretability, where the contribution of each input can be decomposed by module. We use  $task_{1-2}$  in MIMIC to compute two measures: [1] Importance score, where each encoder is deployed alone, providing predictive importance of a single modality by subtracting predictions made from the prior state. This can be computed across all data points or individual data points. [2] Cumulative probability, where the prediction from each multi-task decoder is reported in sequence (i.e. given the previously encoded modalities). We demonstrate this on a random patient from the test set, who has a true label of 0 for both tasks. Further plots are in Appendix Sec. E.2.
Results. Monolithic P-Fusion models cannot be decomposed into any modality-specific predictions, and its (single-task) prediction is only made after inputting all modalities. In contrast, Figure 4 shows MultiModN provides granular insights for both importance score and cumulative prediction. We observe that the Text modality is the most important. The cumulative prediction shows the prior strongly predicts positivity in both classes and thus that  $S_0$  has learned the label prevalence.
The predictions naturally produced by MultiModN provide diverse and granular interpretations.
6.4 Exp. 4: MultiModN is robust to catastrophic failure from biased missingness
Setup. MultiModN is designed to predict any number or combination of tasks from any number or combination of modalities. A missing modality is skipped (encoder  $e_i$  is not used) and not padded/encoded. Thus, MultiModN avoids featurizing missingness, which is particularly advantageous when missingness is MNAR. Featurizing MNAR can result in catastrophic failure when MNAR patterns differ between train and test settings. We demonstrate MultiModN's inherent robustness to catastrophic MNAR failure by training MultiModN and P-Fusion on four versions of MIMIC with various amounts (0, 10, 50, or  $80\%$ ) of MNAR by artificially removing one modality in one class only. Figure 5 compares MultiModN and P-Fusion on  $task_1$  when tested in a setting that has either no missingness or where the MNAR pattern is different (i.e. label-flipped).
Results. Figure 5 shows a dramatic catastrophic failure of P-Fusion in a label-flipped MNAR test set (black solid line) compared with MultiModN. P-Fusion is worse than random at  $80\%$  MNAR (AUROC: 0.385). In contrast, MultiModN only loses  $10\%$  in MNAR flip, remarkably, matching performance in a test with no missingness. Further plots in Appendix E.3.
We present MultiModN, a novel sequential modular multimodal (MM) architecture, and demonstrate its distinct advantages over traditional monolithic MM models which process inputs in parallel.
By aligning the feature extraction pipelines between MultiModN and its baseline P-Fusion, we better isolate the comparison between modular-sequential MM fusion vs. monolithic-parallel MM fusion. We perform four experiments across 10 complex real-world MM tasks in three distinct domains. We show that neither the sequential modularization of MultiModN nor its extension to multi-task predictions compromise the predictive performance on individual tasks compared with the monolithic baseline implementation. Training a multitask model can be challenging to parameterize across inter- and cross-task performance [13, 41]. We perform no specific calibration and show that MultiModN is robust to cross-task bias. Thus, at no performance cost, modularization allows the inherent benefits of multi-task modeling, as well as providing interpretable insights into the predictive potential of each modality. The most significant benefit of MultiModN is its natural robustness to catastrophic failure due to differences in missingness between train and test settings. This is a frequent and fundamental flaw of many domains and especially impacts low-resource settings where modalities may be
Figure 5: MultiModN is robust to catastrophic MNAR failure. Impact of MNAR missingness on MultiModN vs. P-Fusion. Both models are trained on four versions of the MIMIC dataset with  $0 - 80\%$  MNAR. They are then tested on either a test set with no MNAR missingness  $(\texttt{- - - })$  or a test set where the biased missingness is label-flipped, i.e. MNAR occurs in the other binary class as compared with the train  $(-)$ . Results for task1 depicted. CI95% shaded.
missing for reasons independent of the missingness in the train set. More generally, modularization creates a set of self-contained modules, composable in any number or combination according to available inputs and desired outputs. This composability not only provides enormous flexibility at inference but also reduces the computational cost of deployment. Taken together, these features allow MultiModN to make resource-adapted predictions, which have a particular advantage for real-world problems in resource-limited settings.
Limitations and future work. The main limitation for studying MM modeling is the scarcity of large-scale, open-source, MM datasets that cover multiple real-world tasks, especially for time-series. Additionally, while MultiModN is theoretically able to handle any number or combination of modalities and tasks, this has not been empirically tested. Having a high combinatorial generalization comes at a computational and performance cost, where the 'memory' of a fixed-size state representation will likely saturate at scale. The performance of MultiModN is purposely limited in this work by fixing the feature extraction pipeline, to best isolate the effect of sequential fusion. Future work leveraging MultiModN model-agnostic properties would be able to explore the potential performance benefit. This is particularly interesting for time-series, for which the state 'memory' may need to be parameterized to capture predictive trends of varying shapes and lengths.
This project was substantially co-financed by the Swiss State Secretariat for Education, Research and Innovation (SERI).
[1] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature Communications, 2021.
MultiModN is robust to catastrophic missingness (MNAR failure) where P-Fusion is not.
[37] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multiviz: Towards visualizing and understanding multimodal models. In International Conference on Learning Representations (ICLR), 2023.
[38] Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[39] Torsten Wörtwein, Lisa Sheeber, Nicholas Allen, Jeffrey Cohn, and Louis-Philippe Morency. Beyond additive fusion: Learning non-additive multimodal interactions. Empirical Methods in Natural Language Processing (EMNLP), 2022.
[40] Benedek Rozemberczki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivier Kiss, Sebastian Nilsson, and Rik Sarkar. The shapley value in machine learning. In International Joint Conference on Artificial Intelligence, 2022.
[41] Partoo Vafaeikia, Khashayar Namdar, and Farzad Khalvati. A brief review of deep multi-task learning and auxiliary task learning. ArXiv, 2020.
[42] Vinitra Swamy, Sijja Du, Mirko Marras, and Tanja Kaser. Trusting the Explainers: Teacher validation of explainable artificial intelligence for course design. In LAK23: 13th International Learning Analytics and Knowledge Conference, pages 345–356, 2023.
[43] Mirko Marras, Julien Tuang Tu Vignoud, Tanja Kaser, et al. Can feature predictive power generalize? benchmarking early predictors of student success across flipped and online courses. In Proceedings of the 14th International Conference on Educational Data Mining, pages 150-160, 2021.
[44] Daniel FO Onah, Jane Sinclair, and Russell Boyatt. Dropout rates of massive open online courses: behavioural patterns. EDULEARN14 proceedings, pages 5825-5834, 2014.
[45] Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. Negbio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Summits on Translational Science Proceedings, 2018:188, 2018.
[46] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghloo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
A MultiModN Framework Implementation
A task- and modality-agnostic open-source framework MultiModN solution has been implemented in Python using PyTorch as the primary machine learning framework. The /multimodn package contains the MultiModN model and its components. The /datasets package is responsible of preparing the data inputs for MultiModN. Some examples using the public Titanic dataset have been provided.
The code is available at: https://github.com/epfl-iglobalhealth/MultiModN.
Figure 6: Architecture of MultiModN code. The upper panel shows a more detailed depiction of sequential encoding using a series of model-agnostic encoders which receive inputs of variable dimension to create the evolving state vector, which represents the shared feature space. The lower panel shows how each state can be probed by any number of target decoders.
A.1 MultiModN metrics
During training and evaluation, the metrics of the model are stored in a log at each epoch in a matrix of dimensions  $(E + 1)*D$ , where  $E$  is the number of encoders and  $D$  the number of decoders. Each row represents the metrics for a target at each state of the model.
A.2 Code structure
A.2.1 MultiModN
/multimodn package contains the MultiModN model and its modules:
[1] Encoders:/multimodn/encoders
[2] Decoders:/multimodn/decoders
[3] State:/multimodn/state.py
A.2.2 Datasets
/dataset package contains the MultiModDataset abstract class, compatible with MultiModN.
Specific datasets are added in the /dataset directory and must fulfill the following requirements:
- Contain a dataset class that inherit MultiModDataset or has a method to convert into a MultiModDataset
- Contain a .sh script responsible of getting the data and store it in /data folder
__*_ function of MultiModDataset subclasses must yield elements of the following shape:
tuple  
( data: [torch,Tensor], targets: numpy.ndarray,
(optional) encoding_sequence: numpy.ndarray
namely a tuple containing an array of tensors representing the features for each subsequent encoder, a numpy array representing the different targets and optionally a numpy array giving the order in which to apply the encoders to the subsequent data tensors. Note: data and encoding_sequence must have the same length.
Missing values. The user is able to choose to keep missing values (nan values). Missing values can be present in the tensors yielded by the dataset and are managed by MultiModN.
A.2.3 Pipelines
/pipeline package contains the training pipelines using MultiModN for Multimodal Learning. It follows the following steps:
- Create MultiModDataset and the dataloader associated
- Create the list of encoders according to the features shape of the MultiModDataset
- Create the list of decoders according to the targets of the MultiModDataset
- Init, train and test the MultiModN model
- Store the trained model, training history and save learning curves
A.3 Quick start
Quick start running MultiModN on Titanic example pipeline with a Multilayer Perceptron encoder:
./datasets/titanic/get_data.sh
python3pipelines/titanic/titanic_mlppipeline.py
Open pipelines/titanic/titanic_mlp.png to look at the training curves.
B Additional details about MultiModN Architecture
Figure 7: Schematic representation of the modules  $(g, \text{groups})$  of MultiModN.  $e$ : encoders,  $s$ : state vector,  $d$ : decoders. Each module is connected by a single edge between  $s_n$  and  $e_{n+1}$ . There are  $|\mathcal{M}|$  groups (i.e., input-specific modules) and  $|\mathcal{T}|$  decoders per module.