SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Morteza Ghahremani $^{1,2}$  Christian Wachinger $^{1,2}$
$^{1}$ Lab for AI in Medical Imaging (AI-Med), Department of Radiology,
Technical University of Munich (TUM), Germany
$^{2}$ Munich Center for Machine Learning (MCML), Germany
{morteza.ghahremani, christian.wachinger}@tum.de
Recent years have witnessed a surge of interest in integrating high-dimensional data captured by multisource sensors, driven by the impressive success of neural networks in integrating multimodal data. However, the integration of heterogeneous multimodal data poses a significant challenge, as confounding effects and dependencies among such heterogeneous data sources introduce unwanted variability and bias, leading to suboptimal performance of multimodal models. Therefore, it becomes crucial to normalize the low- or high-level features extracted from data modalities before their fusion takes place. This paper introduces RegBN, a novel approach for multimodal Batch Normalization with REGularization. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different data sources. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. We validate the effectiveness of RegBN on eight databases from five research areas, encompassing diverse modalities such as language, audio, image, video, depth, tabular, and 3D MRI. The proposed method demonstrates broad applicability across different architectures such as multilayer perceptrons, convolutional neural networks, and vision transformers, enabling effective normalization of both low- and high-level features in multimodal neural networks. RegBN is available at https://mogvision.github.io/RegBN.
Multimodal models, which adeptly fuse information from a diverse range of sources, have yielded promising results and found many applications such as language and vision [27, 29, 44, 17, 51], multimedia [2, 35, 12], affective computing [63, 41, 64, 48, 56], robotics [23, 49, 24], human-computer interaction [43, 41], and healthcare diagnosis [40, 60, 36, 14, 61]. Multimodal machine learning presents distinctive computational and theoretical research challenges due to the diversity of data sources involved [25, 26]. The impressive versatility and efficacy of multimodal neural network models can be attributed to their ability to effectively integrate and leverage heterogeneous data.
Multimodal models process heterogeneous information obtained from multisource sensors by extracting their features. Subsequently, the extracted features are fused at different levels (including early, middle, or late fusion [58, 35, 36]) to address specific tasks, such as classification, recognition, description, and segmentation [25, 26, 12, 1, 10]. Heterogeneous information from multisource sensors, however, is susceptible to confounding effects caused by extraneous variables or multiple distributions [30, 7, 5, 11, 57, 32, 6]. Confounding variables pertain to external factors that introduce bias (either positive or negative) in the relationship between the variables being studied [42, 50]. The complexity of confounders emerges from their potential pervasiveness across diverse data modal-
Figure 1: The presence of heterogeneous data often entails external confounding effects (denoted by 'C' in the figure) and partial dependencies, which impede the efficient training of a multimodal network. This study recommends the normalization of low- and high-level features extracted by foundation models through the use of RegBN. This allows the normalised features to be rendered independent, enabling a multimodal network to discern underlying patterns and optimize its performance.
Mutually Dependent
ities. For instance, in image analysis, confounders might encompass lighting variations, while in audio classification, speaker attributes like race or gender can be confounding factors. In video parsing, backgrounds play a role, and in dementia diagnosis, the education level of patients can be a confounder. Furthermore, positive or negative correlations can exist among heterogeneous data that impact the distributions of the learned features [50, 30]. These factors pose challenges for a multimodal network to accurately uncover the actual relationship between variables for a given task. Ignoring the confounding effects and partial dependencies in features or data can result in substantial drawbacks, including the deviation of a multimedia model from its global minimum, reduction in the speed and stability of neural network training, and the generation of unreliable or misleading outcomes. Consequently, it is recommended to normalize features from different modalities before their fusion [30, 55]. In this study, we show that the normalization step can help mitigate the confounding effects and enhance the overall performance and reliability of the multimodal networks.
Normalization techniques have proven to be highly effective in enhancing the speed and stability of neural network training. By re-centering and re-scaling the input layers, the normalization methods provide a promising solution to overcome the challenges of training deep neural networks. Batch normalization (BN) [18], layer normalization (LN) [4], group normalization (GN) [62], and instance normalization (IN) [62] are popular normalization techniques that have been used in the foundation of many state-of-the-art neural networks such as ResNet [15], DenseNet [16], Inception [52], and others. The aforementioned normalization methods are designed to standardize feature distributions and do not take into account confounding effects and dependencies among features. Recent progress in multimodal learning leverages the potential of extensive multimodal data representations. CLIP [45], ALIGN [20], and MaMMUT [22] are multimodal foundations devised for images, text, audio, and video. Expanding this scope, ImageBind [13] and Gato [46] tackle diverse tasks across multiple modalities. Dealing with large-scale multimodality data, however, may introduce new obstacles such as modality heterogeneity, modality imbalance, confounding, and intermodal variabilities. These emphasize the necessity of developing a normalization technique that is dedicated to multimodal data.
To tackle these challenges, several methods [30, 55, 65] have recently been proposed that use metadata—the data that provides information about given data—for the normalization of the features in a neural network. However, these studies predominantly focus on metadata and still encounter issues related to confounding effects. Motivated by these findings, we introduce a novel normalization method for multimodal heterogeneous data, referred to as RegBN, aimed at removing confounding effects and dependencies from low- and high-level features before fusing those. Our approach entails leveraging regularization to promote independence among heterogeneous data from multisource
sensors (Figure 1). RegBN facilitates the training of deep multimodal models while ensuring the prediction of reliable results. Our key contributions are as follows:
- As a normalization module, RegBN can be integrated into multimodal models of any architecture such as multilayer perceptrons (MLPs), convolutional neural networks (CNNs), vision transformers (ViTs), and other architectures.
- RegBN possesses the capability to be applied to a vast array of heterogeneous data types, encompassing text, audio, image, video, depth, tabular, and 3D MRI.
- RegBN undergoes comprehensive evaluation on a collection of eight datasets, including multimedia, affective computing, healthcare diagnosis, and robotics.
The findings indicate that RegBN consistently leads to substantial improvements in inference accuracy and training convergence regardless of the data type or method used.
Normalization methods in deep learning: BN [18], LN [4], GN [62], and IN [62] are conventional normalization methods that normalize the input features by re-centering and re-scaling. Unlike such methods that process input layers separately, RegBN takes two input layers and produces corresponding mutually independent output layers. The input layers could be any  $n$ -dimensional low-/high-level features, metadata, or raw data.
Confounding effect removal: Several statistical techniques have been developed for regressing out confounders [42, 33, 38]. Such methods are basically developed for confounding effect removal in healthcare. In recent years, several studies used deep neural networks for coping with confounders [65, 30, 55]. Lu et al. [30] introduced a novel layer normalization module called 'metadata normalization' (MDN), which is designed specifically for neural networks that incorporate metadata. A closed-form solution to linear regression was developed by the authors to capture the relationship between feature layers and metadata. MDN, however, exhibits certain limitations such as its weak performance on small mini-batches and the requirement of the entire metadata for computing its matrix inverse. A penalty-based approach called PMDN [55] was proposed as an improvement over MDN. PMDN addresses the shortcomings of MDN by using learnable parameters instead of estimating MDN's matrix inverse. PMDN is designed for metadata with a small number of features due to the huge number of required learning parameters. PMDN also requires a time-consuming two-step training procedure to optimize its learnable parameters. In addition, a major limitation of PMDN is how to effectively train the learnable parameters, as these parameters are trained solely based on the model's loss function, and models are typically unaware of the presence of confounding effects.
We leverage the great potential of regularization as a solution to address the aforementioned issues. Our proposed normalization technique does not rely on learnable parameters and is capable of operating efficiently on small mini-batches. Most importantly, our approach produces more accurate results and is applicable to various types of multimodal neural networks, not limited to those incorporating metadata, and its ability to operate at both low- and high-feature levels. RegBN is the first normalization method to address the dependency and confounding issues in multimodal data, making it a unique and promising contribution to the field.
3 RegBN: A regularization method for normalization of multimodal data
Given a trainable multimodal neural network (e.g., MLPs, CNNs, ViTs) with multimodality backbones  $\mathcal{A}$  and  $\mathcal{B}$ . Let  $f^{(l)}\in \mathcal{R}^{b\times n}$  represent the  $l$ -th layer of the multimodal network for modality  $\mathcal{A}$  with batch size  $b$  and  $n_1\times \ldots \times n_N$  features that are flattened into a vector of size  $n$ . In a similar vein, we define  $g^{(k)}\in \mathcal{R}^{b\times m}$  as the  $k$ -th layer of the multimodal network for modality  $\mathcal{B}$  with  $m_{1}\times \ldots \times m_{M}$  features that are flattened into a vector of size  $m$ . Layers  $l$  and  $k$  can be positioned directly prior to the fusion step. Depending on the fusion approach employed,  $f^{(l)}$  and  $g^{(k)}$  can contain low-level features, high-level features, or latent representations of the multimodal model for early, middle, or late fusion, respectively (see Appendix ??). Features  $f^{(l)}$  and  $g^{(k)}$  pertain to two distinct modalities and are subject to immeasurable unknown confounders. Moreover, they also share partial common information. The goal is to find potential similarities (mainly caused by confounding factors and dependencies during data collection) between these layers and then remove those (Figure 1). Mutually-independent layers can be then fused in the multimodal network for a
Figure 2: (a) In a mini-batch, there is a risk of falling into local minima; we fix this problem by seeding  $\lambda_{+}$  at different locations, detailed in Section 3.2. (b) 2D visualization of the loss surface of the multimodal SMIL model [31] on AV-MNIST (see Section 4.1). The numbers inside the plot indicate the classification loss, while the color bar reports the L1 loss of the projection weights in RegBN, i.e.  $\Delta W_{t}$ . RegBN's weights converge into their global minimum as long as the multimodal model reaches its global minimum.
given task. To this end, one can represent  $f^{(l)}$  based on  $g^{(k)}$  as a linear regression model
$$
f ^ {(l)} = W ^ {(l, k)} g ^ {(k)} + f _ {r} ^ {(l)}. \tag {1}
$$
Here,  $W^{(l,k)}$  is a projection matrix of size  $n \times m$  and  $f_r^{(l)}$  represents the difference between  $f^{(l)}$  and its corresponding map in the domain of  $g^{(k)}$ , also known as the residual. Ideally, the residual map does not contain any features of  $g^{(k)}$ , so  $f_r^{(l)}$  and  $g^{(k)}$  are mutually independent. RegBN minimizes the linear relationship between these layers via
$$
\hat {W} ^ {(l, k)} = \underset {W ^ {(l, k)}} {\arg \min } \left\| f ^ {(l)} - W ^ {(l, k)} g ^ {(k)} \right\| _ {2} ^ {2} \quad w. r. t. \quad \left\| W ^ {(l, k)} \right\| _ {F} = 1. \tag {2}
$$
The constraint in terms of the Frobenius norm,  $\left\| W^{(l,k)}\right\|_F = \sum_{i=1}^n \sum_{j=1}^m \omega_{i,j}^{(l,k)2} = 1$ , guarantees that the network does not experience vanishing and exploding gradients during training and inference. Eq. 2 is a regularization method for ill-conditioned problems [37, 9, 34, 47]. A Lagrangian multiplier offers an equivalent formulation  $\mathcal{F}$  to the optimization defined at Eq. 2:
$$
\mathcal {F} \left(W ^ {(l, k)}, \lambda_ {+}\right) = \left\| f ^ {(l)} - W ^ {(l, k)} g ^ {(k)} \right\| _ {2} ^ {2} + \lambda_ {+} \left(\left\| W ^ {(l, k)} \right\| _ {F} - 1\right). \tag {3}
$$
In this equation,  $\lambda_{+}$  is a positive Lagrangian multiplier, playing the role of a dual variable. Minimizing  $\mathcal{F}(W^{(l,k)},\lambda_{+})$  over  $W^{(l,k)}$  yields the projection matrix
$$
\hat {W} ^ {(l, k)} = \left(g ^ {(k) \top} g ^ {(k)} + \hat {\lambda} _ {+} \mathbf {I}\right) ^ {- 1} g ^ {(k) \top} f ^ {(l)}, \tag {4}
$$
where,  $\hat{\lambda}_{+}$  is the estimated Lagrangian multiplier obtained through minimization of the term  $\left\| W^{(l,k)}\right\|_{F} - 1$ ;  $\mathbf{I}$  is an identity matrix and superscript  $\top$  represents the transpose operator. We employ singular value decomposition (SVD) and limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) [28] for solving Eq. 2. The solution is detailed in Appendix ??, and the proposed method is summarized in Algorithm 1. It is worth noting that the projection weights are not directly trained by the multimodal criterion since there is a danger of being fooled by confounders. Instead, the projection weights are learned through Eqs. 2-4 and updated recursively over chunks of training data, detailed below.
3.1 Update of the projection matrix
The projection matrix,  $W^{(l,k)}$ , is calculated for every mini-batch so we recursively update that via the exponential moving average's approach [21], which decays the mean gradient and variance for each
Algorithm 1 Pseudocode of RegBN
1: Inputs:  $f^{(l)} \in \mathcal{R}^{b \times n}$  and  $g^{(k)} \in \mathcal{R}^{b \times m}$ :  $l$ -th and  $k$ -th layer of an MML, respectively
2: Other inputs:  $t$  (timestep/mini-batch), projection weights  $W^{(l,k)}$ ,  $\Lambda_{t-1}$  (a collection of estimated  $\lambda_{+}$  values until previous timestep), training;
3: Compute SVD of  $g^{(k)}$ : SVD $(g^{(k)}) = U\Sigma V^* = \sum_{i=1}^{m} \sigma_i u_i v_i^*$
4: Set  $\Lambda_{t}$  through Eq. 8 and given  $\Lambda_{t - 1}$
5: for  $\lambda$  in  $\Lambda_t$  do
6: Initialize  $\lambda_{+}$  with  $\lambda$  and then apply the L-BFGS algorithm to approximate Eq. 7;
7: Store the estimated  $\lambda_{+}$  and the approximation error;
8: end for
9: Choose  $\lambda_{+}$  with the lowest approximation error as the sub-optimal  $\hat{\lambda}_{+}$ ;
10: Update  $\Lambda_t\colon \Lambda_t\gets \Lambda_{t - 1}\cup \bar{\lambda}_t$
11: Insert  $\hat{\lambda}_{+}$  to Eq. 4 for computation of the projection matrix:  $\hat{W}^{(l,k)} = \sum_{i=1}^{m} \left( \frac{\sigma_i u_i f^{(l)}}{\sigma_i^2 + \hat{\lambda}_+} \right) v_i$ ;
12: if training then
13:Normalize the input feature layer using  $\hat{W}^{(l,k)}$  ..  $f_{r}^{(l)}\gets f^{(l)} - \hat{W}^{(l,k)}g^{(k)}$
14: end if
15: Update the projection weights  $W_{t}^{(l,k)}$  via Eqs. 5 and 6;
16: if not training then
17:Normalize the input feature layer using  $W_{t}^{(l,k)}\colon f_{r}^{(l)}\gets f^{(l)} - W_{t}^{(l,k)}g^{(k)};$
18: end if
19: return  $f_{r}^{(l)}$ ,  $W_{t}^{(l,k)}$ , and  $\Lambda_{t}$
variable exponentially. Let  $W_{t-1}^{(l,k)}$  denote the updated projection matrix until the previous timestep. We define  $\Delta W_t$  as the mean absolute error between the currently predicted projection matrix and the previously updated projection matrix:  $\Delta W_t = \left\| \hat{W}^{(l,k)} - W_{t-1}^{(l,k)} \right\|_1$ . The projection matrix is then updated at timestep  $t$  via
$$
W _ {t} ^ {(l, k)} = \left(1 - \gamma_ {t} \frac {m _ {t}}{\sqrt {\nu_ {t} + \epsilon}}\right) \hat {W} ^ {(l, k)} + \gamma_ {t} \frac {m _ {t}}{\sqrt {\nu_ {t} + \epsilon}} W _ {t - 1} ^ {(l, k)}. \tag {5}
$$
Here,  $\gamma_{t}$  is the learning rate of the multimodal model at timestep  $t$ ;  $m_{t}$  and  $\nu_{t}$  are the first and second moments, respectively, derived from
$$
m _ {t} = \frac {\beta_ {1} m _ {t - 1} + (1 - \beta_ {1}) \Delta W _ {t}}{1 - \beta_ {1} ^ {t}}, \nu_ {t} = \frac {\beta_ {2} \nu_ {t - 1} + (1 - \beta_ {2}) \Delta W _ {t} ^ {2}}{1 - \beta_ {2} ^ {t}}, \tag {6}
$$
where  $\beta_{1},\beta_{2}\in (0,1)$  represent constant exponential decay rates. As mentioned in Algorithm 1, the projection matrix is updated during training only (Algorithm 1: Steps 11-13), and we use the latest updated projection weights for validation or inference (Algorithm 1: Steps 15-17).
3.2 Avoiding falling into local minima
Multimodal networks often contain several backbones (e.g., MLPs, CNNs, ViTs), depending on the number of multiple heterogeneous data sources. Identifying the global minima of such networks is a challenging problem, as the networks may fall into local minima. On the other hand, optimization of Eq. 3 relies on  $\lambda_{+}$ , which may introduce several local minima. Conventional regression methods treat  $\lambda_{+}$  as a constant/hyperparameter, while predetermined hyperparameters usually increase the risk of falling into local minima. To prevent this, we adopt a mini-batch-wise approach for the estimation of  $\lambda_{+}$ . The objective is to estimate suboptimal  $\lambda_{+}$  with the L-BFGS optimization algorithm in a way that meets
$$
\hat {\lambda} _ {+} = \underset {\lambda_ {+}} {\arg \min } \frac {\partial \mathcal {F} (W ^ {(l , k)} , \lambda_ {+})}{\partial \lambda_ {+}} = \underset {\lambda_ {+}} {\arg \min } \left(\left\| \left(g ^ {(k) \top} g ^ {(k)} + \lambda_ {+} \mathbf {I}\right) ^ {- 1} g ^ {(k) \top} f ^ {(l)} \right\| _ {F} - 1\right). \tag {7}
$$
Quasi-Newton methods like L-BFGS rely on the initial value of  $\lambda_{+}$  for minimization of the optimization problems like Eq. 7 (see Figure 2a). Due to the limited number of local minima that exist per mini-batch, we adopt a method whereby we initialize  $\lambda_{+}$  with multiple random values within a
Table 1: The experimental section covers a diverse range of research areas, dataset sizes, input modalities (in the form of  $i$  : image,  $l$  : language,  $v$  : video,  $a$  : audio,  $s$  : 3D snippet-level features,  $m$  : 3D MRI,  $d$  : depth,  $t$  : tabular,  $f$  : force sensor,  $p$  : proprioception sensor), and prediction tasks.
Area	Dataset	Modalities	Samples (#)	Prediction task
Training	Test
Affective computing	CMU-MOSEI	{v,a,l}	18,118	4,659	sentiment
Affective computing	CMU-MOSI	{v,a,l}	1,513	686	sentiment
Affective computing	IEMOCAP	{v,a,l}	3,515	938	emotion
Multimedia	MM-IMDb	{i,l}	18,160	7,799	movie genre
Multimedia	LLP	{a,v,s}	10,649	1,200	video parsing
Multimedia	AV-MNIST (small)	{i,a}	1,045	450	classification
Multimedia	AV-MNIST	{i,a}	60,000	10,000	classification
Healthcare diagnosis	ADNI	{m,t}	1,073	268	dementia diagnosis
Robotics	Vision&Touch	{i,d,f,p}	40,546	22,800	contact
-	Synthetic dataset	{i,t}	10,000	1,000	classification
predetermined range  $\Lambda_{p} = [\lambda_{p_{1}},\dots ,\lambda_{p_{C}}]$  as well as with the median of the suboptimal  $\lambda_{+}$  values predicted in the previous mini-batches,  $\mu_{\frac{1}{2}}(\Lambda_{t - 1})$
$$
\boldsymbol {\Lambda} _ {t} = \Lambda_ {p} \cup \mu_ {\frac {1}{2}} (\Lambda_ {t - 1}) = \left[ \lambda_ {p _ {1}}, \dots , \lambda_ {p _ {C}} \right] \cup \mu_ {\frac {1}{2}} \{\left[ \hat {\lambda} _ {1}, \dots , \hat {\lambda} _ {t - 1} \right] \}. \tag {8}
$$
By creating a loop over  $\Lambda_{t}$ , we first initialize  $\lambda_{+}$  and then estimate its true value using the L-BFGS method. We also store the approximation error and after the termination of the loop, the suboptimal value of  $\lambda_{+}$  for a given mini-batch is determined by selecting the estimated value that yields the lowest approximation error. After calculating the suboptimal lagrangian multiplier, we update  $\Lambda$ , i.e.,  $\Lambda_{t} \gets \Lambda_{t-1} \cup \hat{\lambda}_{t}$ . As illustrated in Figure 2b, when the multimodal model approaches its global minimum, the parameters of RegBN also converge.
Heterogeneous data captured by diverse multisource sensors are utilized to verify the usefulness and effectiveness of RegBN in multiple data contexts such as language, audio, 2D image, video, depth, 3D MRI, and tabular data. RegBN is applied to eight datasets that are summarized in Table 1. Details on the datasets and baseline methods are provided in Appendices ?? & ??. The default parameters and settings for RegBN are reported in Appendix ?? Our code is openly available at https://mogvision.github.io/RegBN. Experimental details are provided in Appendix ?? Here, we summarize the main results. The findings of this investigation provide insights into the performance of RegBN across various data modalities and highlight its potential as a robust normalization technique. PMDN relies on a considerable number of learnable parameters specifically tailored for metadata with limited dimensions. MDN necessitates substantial RAM resources, particularly when estimating its inverse matrix on large-scale datasets.
4.1 Multimedia
Experiments with the LLP dataset [53]: Audio-Visual Video Parsing (AVVP) [53] (see Appendix ??) is employed as a baseline for parsing individual audio, visual, and audio-visual events under both segment-level and event-level metrics over the LLP dataset. The normalization module is employed to decouple the audio features from the video features, ensuring their independence. Table 2 reports that RegBN improves the baseline performance in nine out of ten metrics. RegBN brings about improvements in all audio-visual video parsing subtasks, measured both at the segment-level and event-level metrics. These findings suggest that decoupling audio and video allows AVVP to produce more accurate predictions of event categories on a per-snippet basis. $^{1}$
Experiments with the MM-IMDb dataset [3]: The MM-IMDb dataset was curated for the purpose of predicting movie genres through the use of either image or text modality. This task involves
<sup>1</sup>Appendix ?? reports the validation results of AVVP with and without RegBN on the LLP dataset.
Table 2: Audio-visual video parsing accuracy (%) of AVVP [53], as baseline (BL), on the LLP dataset [53] for different normalization techniques. A and V stand for audio and visual, respectively.
Method	Segment-Level	Event-Level
A↑	V↑	A-V↑	Type↑	Event↑	A↑	V↑	A-V↑	Type↑	Event↑
BL	60.1	52.9	48.9	54.0	55.4	51.3	48.9	43.0	47.7	48.0
BL+PMDN	59.7	53.0	48.8	53.6	55.3	51.4	49.1	42.9	47.7	48.2
BL+RegBN	60.2	53.3	48.9	54.0	55.2	52.0	49.5	43.1	47.9	49.3
Table 3: Multi-label classification scores (F1 score) of baseline SMIL [31] (denoted by BL) with/without normalization on the MM-IMDb dataset [3]
Method	Norm. Params. (#)	F1 Score (%)
Samples↑	Micro↑	Weighted↑
BL	-	49.62	51.18	48.95
BL+PMDN	65,536	49.85	51.44	49.36
BL+RegBN	0	54.82	55.37	52.83