SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The hyperparameter-tuned architecture (based on head size, number of transformer blocks, MLP units) for EDU and Weather is a transformer model with 4 transformer blocks, 4 heads of size 256, dropout of 0.25, MLP units 128 with dropout 0.4, batch size 64, trained for 50 epochs with cross-entropy loss. For MIMIC, the most performant (tuned) transformer architecture includes 2 transformer blocks with 3 heads of size 128, MLP units 32 with batch size 32. We train this architecture on each decoder
task individually and all tasks together for a total of 13 new models with the exact preprocessing steps as in the P-Fusion and MultiModN experiments. The results indicate that MultiModN often outperforms or at least matches the P-Fusion Transformer benchmark in the vast majority of single task and multi-task settings, and comes with several interpretability, missingness, and modularity advantages. Specifically, using the primary metric for each task (BAC for the classification tasks and MSE for the regression tasks), MultiModN beats the Transformer baseline significantly in 7 tasks, overlaps  $95\%$  CIs in 11 tasks, and loses very slightly (by 0.01) in 2 regression tasks.
Figure 14: Performance of the P-Fusion Transformer on 10 classification and regression tasks across 3 datasets. Results are showcased with  $95\%$  confidence intervals. BAC and MSE are the primary evaluation metrics for classification and regression respectively.
E.5 Additional Inference Settings
Figure 15: Detailed modality inference experiments for MultiModN in comparison to P-Fusion. In these experiments, different combinations of modalities and orderings at the time of inference are used for the two tasks in the MIMIC dataset. All  $95\%$  CIs overlap between the two models.
To provide insight into performance gains, we performed additional experiments to showcase the benefits of modularity with vastly different training and inference settings. The results of 30 new experiments of inference encoders, each performed with 5-fold cross-validation are included in Figure 15. We compare P-Fusion and MultiModN on both tasks of the MIMIC dataset using all possible combinations of four input modalities at test time. MultiModN ignores missing modalities whereas P-Fusion imputes and therefore encodes missing modalities.
We note that the performance at inference for P-Fusion and MultiModN has no significant differences for all experiments (using  $95\%$  CIs). Figure 15 shows that, on average, P-Fusion tends to overfit more to the most dominant (visual) modality. When this modality is missing (at random or completely at random), MultiModN performs better on a combination of the remaining modalities (demo, text, time series). In the case of missing modalities, the observed effect in Figure 15 is weak - confidence intervals overlap. Considering the MNAR (missing not-at-random) scenario described in Sec. 6.4, the difference becomes significant.