SlowGuess/NeurIPS_2023_OCR huggingface dataset:
multi-label classification since a single movie may be associated with multiple genres (see Appendix ??). SMIL [31], as a baseline approach, employs the pre-trained BERT to extract the textual features, while image features are extracted using the pre-trained VGG-19. The text features are subsequently normalized with respect to the visual features using the proposed RegBN before the fusion process occurs. The fusion operation involves a concatenation layer. The fused features are passed through two fully-connected layers to obtain the classification labels. The details are provided in Appendix ??. Table 3 presents the quantitative results obtained from the experiments. The table indicates that applying a normalization layer such as PMDN or RegBN can lead to an improvement in the performance over the baseline model. This highlights the importance of the normalization step in multimedia data. In particular, RegBN was found to enhance the F1 score of the baseline by an average of  $8.86\%$  across all three metrics.
Experiments with the AV-MNIST dataset [31]: The small AV-MNIST dataset comprises two modalities, namely, audio and image. The audio modality is obtained from Free Spoken DigitsDataset $^2$ , which includes 1,500 raw audio recordings from three different speakers. Despite the independence of the image and audio modalities, the primary objective of this experiment is to investigate whether the confounding effects induced by the speakers have an impact on the classification results. In line with the previous experiment, SMIL [31] serves as the baseline method. The obtained classification results for the combinations of baseline+MDN, baseline+PMDN, and baseline+RegBN are  $98.43\%$ ,  $98.68\%$ , and  $99.11\%$ , respectively. Figure 3 illustrates t-SNE plots for the different approaches. In this experiment, tSNE is applied to the features extracted from a fully-connected layer after the concatenation of normalized image features with audio features. The figure demonstrates that the classes are more separable for RegBN as compared to MDN or PMDN, in particular, class '2'. This result implies that our normalization method effectively removes the confounding effects caused by different speakers, thereby providing more flexibility to the baseline method for accurate classification. The results on AV-MNIST are reported in Appendix ??
4.2 Affective computing
In this section, we explore the potential application of RegBN in multimodal emotion-sentiment analysis using the IEMOCAP [8], CMU-MOSI, and CMU-MOSEI [63] datasets, which are divided into aligned and non-aligned multimodal time-series (see Appendices ??, ??, & ??). The baseline method (BL) for this task is the Multimodal Transformer (MulT) [54], which is a ViT developed for analyzing multimodal language sequences. MulT fuses video  $(v)$ , audio  $(a)$ , and language  $(l)$  at three levels (see Appendix ??): 1) language fusion  $(v \to l \& a \to l)$ , 2) audio fusion  $(v \to a \& l \to a)$ , and 3) video fusion  $(a \to v \& l \to v)$ . The outputs of one or more levels are then aggregated for
$^{2}$ Free Spoken Digit Dataset is available at https://github.com/Jakobovski/free-spoken-digit-dataset
(a) Only image
(e) RegBN
(b) Only spectrogram
(c) MDN
(d) PMDN
Figure 3: tSNE visualization of the features extracted from a-b) an unimodal image and an unimodal audio, and c-e) the multimodal model with different normalization methods. Each data point represents a sample.
Table 4: Results of multimodal emotion analysis on IEMOCAP [8] with word aligned multimodal sequences. The baseline (BL) is Multimodal Transformer (MulT) [54].
Method	Loss	Happy (%)	Sad (%)	Angry (%)	Natural (%)
Train↓	Test↓	Acc↑	F1↑	Acc↑	F1↑	Acc↑	F1↑	Acc↑	F1↑
BL	0.106	0.536	86.3	84.0	81.5	80.6	86.5	86.4	69.5	69.1
BL+RegBN	0.009	0.452	87.4	83.0	84.3	84.1	88.2	88.1	73.4	73.2
classification. Here, we report the results of all three fusion levels combined, while the detailed results for each level are provided in Appendix ??. The quantitative results of the emotion analysis experiment are reported in Table 4. The table shows that RegBN improves the classification performance of MulT in most cases, particularly for the "natural" class of the word-aligned experiment. Table 5 presents the results of the multimodal sentiment analysis on the CMU-MOSI and CMU-MOSEI datasets. In both the word-aligned and unaligned experiments, decoupling the multimodal features with RegBN improves training and inference performance.
4.3 Healthcare diagnosis
The ADNI dataset [19] includes 3D MRI scans of patients along with rich clinical information organized in a low-dimensional tabular format (see Appendix ??). The dataset is subject to confounding effects such as age, sex, and level of education, which must be considered to prevent biased evaluation results, as noted in studies [59, 39]. Similar to [60], we partitioned the data into five separate and non-overlapping folds, ensuring that each fold had a balanced distribution of diagnosis, age, and sex. In this experiment, we utilized a 3D ResNet [60] for extracting MRI features, while tabular features were extracted using an MLP network proposed in [14] (see Appendix ??). These features were then concatenated and fed into another MLP network for diagnosis classification. We applied different normalization techniques to the unimodal MRI and tabular features prior to concatenation. Table 6 reports the mean and standard deviation obtained by averaging the performance of the multimodal over these five folds. The complete results can be found in Appendix ??. RegBN emerged as the most effective technique, outperforming other normalization methods in terms of both ACC and BA metrics. Moreover, RegBN's ability to relax the interdependencies between the different modalities facilitated lower training loss in the multimodal network.
4.4 Robotics
The Vision&Touch dataset [23] encompasses recordings of simulated and real robotic arms that are equipped with visual (RGB and depth), force, and proprioception sensors (see Appendix ??). We employed 'Making Sense of Vision and Touch' (MSVT) developed by Lee et al. [23] as a baseline (refer to Appendix ??). For this experiment, we apply normalization techniques to decouple the visual RGB and force from depth information. Table 7 summarizes the results, while detailed results can be found in Appendix ??. The flow loss values in these tables indicate that MSVT converges more effectively when used in conjunction with RegBN. This suggests that RegBN is able to effectively remove dependencies between the RGB and depth pair, as well as the force and depth pair.
Table 5: Multimodal sentiment analysis results multimodal on CMU-MOSEI & MOSI [63] with word aligned multimodal sequences. The baseline (BL) method is MulT [54].
Method	Dataset	Loss	Sentiment (%)
Training↓	Test↓	Acc2↑	Acc5↑	Acc7↑	F1↑	Corr↑
BL	CMU-MOSEI	0.452	0.636	80.3	51.9	50.3	80.1	67.1
BL+RegBN	CMU-MOSEI	0.438	0.611	81.1	52.2	50.5	81.2	66.6
BL	CMU-MOSI	0.403	0.632	81.4	42.5	37.5	82.0	69.7
BL+RegBN	CMU-MOSI	0.267	0.546	81.8	42.3	38.6	82.3	69.1
Table 6: Training's cross-entropy (CE) loss along test's accuracy (ACC) and balanced accuracy (BA) results on the ADNI dataset [19]. Baseline is a combination of techniques developed in [14, 60].
Method	Norm. Params. (#)	Training CE loss↓	Test
ACC (%)↑	BA (%)↑
BL+BN	512	0.641±0.01	48.8±4.4	48.3±4.5
BL+MDN	0	0.619±0.03	50.4±4.8	50.1±5.6
BL+PMDN	4,096	0.632±0.03	49.7±6.2	49.6±7.5
BL+RegBN	0	0.596±0.01	53.0±3.1	52.3±3.7
Table 7: Results of MSVT [23] incorporating various normalization methods on Vision&Touch [23]
Method	Norm. Params. (#)	Training	Test Accuracy (%)↑
Flow loss↓	Total loss↓
BL	-	0.212	0.563	86.2
BL+PMDN	131,072	0.194	0.454	87.9
BL+RegBN	0	0.052	0.231	91.5
Table 8: Classification accuracy results on the synthetic dataset in the presence of a confounder. Due to the randomness of the experiment, we reported the mean and std of results over 100 runs.
Normalization Method	Norm. params. (#)	Experiment I	Experiment II
MLP	CNN	MLP	CNN	MLP	CN
reference	-	-	87.5	87.5	75.0	75.0
BN	256	112	96.4±0.4	94.9±0.2	83.1±0.8	85.2±0.3
MDN	0	0	91.9±0.7	89.7±0.7	79.1±0.6	82.2±0.7
PMDN	2048	201,728	93.4±0.6	92.7±0.5	80.8±0.7	84.3±0.8
RegBN	0	0	87.3±0.9	88.2±0.8	76.2±0.8	76.8±0.9
4.5 Synthetic dataset
Drawing from the synthetic dataset in study [30], we designed a synthetic experiment for a binary classification task. Images with a size of  $64 \times 64$  are split into four regions with identical Gaussians in their background. The regions along the main diagonal are linked to the class label, while the off-diagonal ones act as confounders that do not impact the true label. We created metadata of size 16 for this dataset that includes the true binary label, the confounder value, one fake label, 12 randomly generated fake features, and 1 column with a value of one. The binary classes have overlapping, so the accuracy drops based on the amount of overlapping (see Appendix ??). In Experiment I, the theoretical maximum accuracy of classification is  $87.5\%$ , while for Experiment II, it is  $75\%$ . The base networks employed here are a simple CNN and an MLP-based neural network. The dataset and base networks are detailed in Appendices ?? and ??, respectively. The classification results are presented in Table 8, with more detailed results available in Appendix ??. The results reveal that conventional normalization techniques, such as BN, GN, and IN, are unable to remove the confounding effects. The performance of the base networks that employed MDN was superior compared to those with conventional normalization methods. However, MDN still exhibits shortcomings in dealing with confounding effects. Despite the incorporation of a high number of learnable parameters, PMDN was unsuccessful in eliminating confounding effects, as the inclusion of fake values in the metadata led to the deception of the multimodal model. RegBN demonstrated its ability to remove confounder effects by yielding performance closest to the theoretical maximum in both experiments.
4.6 Computational cost and ablation study
We employed an NVIDIA GTX 1080 Ti with 12GB VRAM for image experiments and an NVIDIA A100 with 80GB VRAM for video experiments. Since RegBN learns the projection matrix through Eqs. 4-7, it eliminates the need for learnable parameters, resulting in enhanced computational efficiency for multimodal models. During the training process, RegBN achieves a frame rate of approximately 124 fps when normalizing a pair of layers with dimensions  $256 \times 1024$ . In comparison, conventional normalization techniques achieve a frame rate of around 13,000 fps for the normalization of one layer of dimensions  $256 \times 1024$ . Regarding inference time, RegBN exclusively relies on the updated projection matrix and therefore does not require additional computation. RegBN is frequently used before the fusion blocks in multimodal models, so the training time of a few RegbBN layers is negligible. Results for the ablation study are provided in Appendix ??
This study presented a novel normalization method designed for dependency and confounding removal in multimodal data and models. The experimental results demonstrated the effectiveness of RegBN in significantly enhancing the accuracy and convergence of multimodal models, rendering it a promising normalization method for multimodal heterogeneous data. The current version of RegBN employs L-BFGS on a single GPU, which will be improved in the future. Our aspiration is that this work effectively leverages new opportunities in exploring and harnessing multi-modality data within multimodal analysis.
This work was supported by the Munich Center for Machine Learning (MCML) and the German Research Foundation (DFG). The authors gratefully acknowledge the computational and data resources provided by the Leibniz Supercomputing Centre. The authors thank Emre Kavak for his valuable input on the synthetic dataset design.
[60] Tom Nuno Wolf, Sebastian Pölsterl, Christian Wachinger, Alzheimer's Disease Neuroimaging Initiative, et al. Daft: A universal module to interweave tabular data and 3d images in cnns. NeuroImage, 260:119505, 2022.
[61] Tom Nuno Wolf, Sebastian Pölsterl, and Christian Wachinger. Don't panic: Prototypical additive neural network for interpretable classification of alzheimers disease. In International Conference on Information Processing in Medical Imaging, pages 82-94. Springer, 2023.
[62] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[63] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259, 2016.
[64] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236–2246, 2018.
[65] Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. Training confounder-free deep learning models for medical applications. Nature communications, 11(1):6010, 2020.