Title: RegBN: Batch Normalization of Multimodal Data with Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multimodal Data

Multimodality

Batch Normalization

Heterogeneous data

Regularization

Confounder

Confounding Effect Removal

Data Dependency
Abstract: Recent years have witnessed a surge of interest in integrating high-dimensional data captured by multisource sensors, driven by the impressive success of neural networks in integrating multimodal data. However, the integration of heterogeneous multimodal data poses a significant challenge, as confounding effects and dependencies among such heterogeneous data sources introduce unwanted variability and bias, leading to suboptimal performance of multimodal models. Therefore, it becomes crucial to normalize the low- or high-level features extracted from data modalities before their fusion takes place. This paper introduces RegBN, a novel approach for multimodal Batch Normalization with REGularization. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different data sources. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. We validate the effectiveness of RegBN on eight databases from five research areas, encompassing diverse modalities such as language, audio, image, video, depth, tabular, and 3D MRI. The proposed method demonstrates broad applicability across different architectures such as multilayer perceptrons, convolutional neural networks, and vision transformers, enabling effective normalization of both low- and high-level features in multimodal neural networks. RegBN is available at https://mogvision.github.io/RegBN.

RegBN: Batch Normalization of Multimodal Data with Regularization

Abstract

Recent years have witnessed a surge of interest in integrating high-dimensional data captured by multisource sensors, driven by the impressive success of neural networks in integrating multimodal data. However, the integration of heterogeneous multimodal data poses a significant challenge, as confounding effects and dependencies among such heterogeneous data sources introduce unwanted variability and bias, leading to suboptimal performance of multimodal models. Therefore, it becomes crucial to normalize the low- or high-level features extracted from data modalities before their fusion takes place. This paper introduces RegBN, a novel approach for multimodal Batch Normalization with REGularization. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different data sources. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. We validate the effectiveness of RegBN on eight databases from five research areas, encompassing diverse modalities such as language, audio, image, video, depth, tabular, and 3D MRI. The proposed method demonstrates broad applicability across different architectures such as multilayer perceptrons, convolutional neural networks, and vision transformers, enabling effective normalization of both lowand high-level features in multimodal neural networks. RegBN is available at https://mogvision.github.io/RegBN .

1 Introduction

Multimodal models, which adeptly fuse information from a diverse range of sources, have yielded promising results and found many applications such as language and vision [27, 29, 44, 17, 51], multimedia [2, 35, 12], affective computing [63, 41, 64, 48, 56], robotics [23, 49, 24], humancomputer interaction [43, 41], and healthcare diagnosis [40, 60, 36, 14, 61]. Multimodal machine learning presents distinctive computational and theoretical research challenges due to the diversity of data sources involved [25, 26]. The impressive versatility and efficacy of multimodal neural network models can be attributed to their ability to effectively integrate and leverage heterogeneous data.

Multimodal models process heterogeneous information obtained from multisource sensors by extracting their features. Subsequently, the extracted features are fused at different levels (including early, middle, or late fusion [58, 35, 36]) to address specific tasks, such as classification, recognition, description, and segmentation [25, 26, 12, 1, 10]. Heterogeneous information from multisource sensors, however, is susceptible to confounding effects caused by extraneous variables or multiple distributions [30, 7, 5, 11, 57, 32, 6]. Confounding variables pertain to external factors that introduce bias (either positive or negative) in the relationship between the variables being studied [42, 50]. The complexity of confounders emerges from their potential pervasiveness across diverse data modal-

Multimodallow/high-level

MutuallyIndependentMultimodal

NormalizationviaRegBN

Features

Features

Confounder

ities. For instance, in image analysis, confounders might encompass lighting variations, while in audio classification, speaker attributes like race or gender can be confounding factors. In video parsing, backgrounds play a role, and in dementia diagnosis, the education level of patients can be a confounder. Furthermore, positive or negative correlations can exist among heterogeneous data that impact the distributions of the learned features [50, 30]. These factors pose challenges for a multimodal network to accurately uncover the actual relationship between variables for a given task. Ignoring the confounding effects and partial dependencies in features or data can result in substantial drawbacks, including the deviation of a multimedia model from its global minimum, reduction in the speed and stability of neural network training, and the generation of unreliable or misleading outcomes. Consequently, it is recommended to normalize features from different modalities before their fusion [30, 55]. In this study, we show that the normalization step can help mitigate the confounding effects and enhance the overall performance and reliability of the multimodal networks.

Normalization techniques have proven to be highly effective in enhancing the speed and stability of neural network training. By re-centering and re-scaling the input layers, the normalization methods provide a promising solution to overcome the challenges of training deep neural networks. Batch normalization (BN) [18], layer normalization (LN) [4], group normalization (GN) [62], and instance normalization (IN) [62] are popular normalization techniques that have been used in the foundation of many state-of-the-art neural networks such as ResNet [15], DenseNet [16], Inception [52], and others. The aforementioned normalization methods are designed to standardize feature distributions and do not take into account confounding effects and dependencies among features. Recent progress in multimodal learning leverages the potential of extensive multimodal data representations. CLIP [45], ALIGN [20], and MaMMUT [22] are multimodal foundations devised for images, text, audio, and video. Expanding this scope, ImageBind [13] and Gato [46] tackle diverse tasks across multiple modalities. Dealing with large-scale multimodality data, however, may introduce new obstacles such as modality heterogeneity, modality imbalance, confounding, and intermodal variabilities. These emphasize the necessity of developing a normalization technique that is dedicated to multimodal data.

To tackle these challenges, several methods [30, 55, 65] have recently been proposed that use metadata-the data that provides information about given data-for the normalization of the features in a neural network. However, these studies predominantly focus on metadata and still encounter issues related to confounding effects. Motivated by these findings, we introduce a novel normalization method for multimodal heterogeneous data, referred to as RegBN, aimed at removing confounding effects and dependencies from low- and high-level features before fusing those. Our approach entails leveraging regularization to promote independence among heterogeneous data from multisource

sensors (Figure 1). RegBN facilitates the training of deep multimodal models while ensuring the prediction of reliable results. Our key contributions are as follows:

As a normalization module, RegBN can be integrated into multimodal models of any architecture such as multilayer perceptrons (MLPs), convolutional neural networks (CNNs), vision transformers (ViTs), and other architectures.

RegBN possesses the capability to be applied to a vast array of heterogeneous data types, encompassing text, audio, image, video, depth, tabular, and 3D MRI.

RegBN undergoes comprehensive evaluation on a collection of eight datasets, including multimedia, affective computing, healthcare diagnosis, and robotics.

The findings indicate that RegBN consistently leads to substantial improvements in inference accuracy and training convergence regardless of the data type or method used.

2 Related work

Normalization methods in deep learning : BN [18], LN [4], GN [62], and IN [62] are conventional normalization methods that normalize the input features by re-centering and re-scaling. Unlike such methods that process input layers separately, RegBN takes two input layers and produces corresponding mutually independent output layers. The input layers could be any n -dimensional low-/high-level features, metadata, or raw data.

Confounding effect removal : Several statistical techniques have been developed for regressing out confounders [42, 33, 38]. Such methods are basically developed for confounding effect removal in healthcare. In recent years, several studies used deep neural networks for coping with confounders [65, 30, 55]. Lu et al. [30] introduced a novel layer normalization module called 'metadata normalization' (MDN), which is designed specifically for neural networks that incorporate metadata. A closed-form solution to linear regression was developed by the authors to capture the relationship between feature layers and metadata. MDN, however, exhibits certain limitations such as its weak performance on small mini-batches and the requirement of the entire metadata for computing its matrix inverse. A penalty-based approach called PMDN [55] was proposed as an improvement over MDN. PMDN addresses the shortcomings of MDN by using learnable parameters instead of estimating MDN's matrix inverse. PMDN is designed for metadata with a small number of features due to the huge number of required learning parameters. PMDN also requires a time-consuming two-step training procedure to optimize its learnable parameters. In addition, a major limitation of PMDN is how to effectively train the learnable parameters, as these parameters are trained solely based on the model's loss function, and models are typically unaware of the presence of confounding effects.

We leverage the great potential of regularization as a solution to address the aforementioned issues. Our proposed normalization technique does not rely on learnable parameters and is capable of operating efficiently on small mini-batches. Most importantly, our approach produces more accurate results and is applicable to various types of multimodal neural networks, not limited to those incorporating metadata, and its ability to operate at both low- and high-feature levels. RegBN is the first normalization method to address the dependency and confounding issues in multimodal data, making it a unique and promising contribution to the field.

3 RegBN: A regularization method for normalization of multimodal data

Given a trainable multimodal neural network (e.g., MLPs, CNNs, ViTs) with multimodality backbones A and B . Let f ( l ) ∈ R b × n represent the l -th layer of the multimodal network for modality A with batch size b and n 1 × . . . × n N features that are flattened into a vector of size n . In a similar vein, we define g ( k ) ∈ R b × m as the k -th layer of the multimodal network for modality B with m 1 × . . . × m M features that are flattened into a vector of size m . Layers l and k can be positioned directly prior to the fusion step. Depending on the fusion approach employed, f ( l ) and g ( k ) can contain low-level features, high-level features, or latent representations of the multimodal model for early, middle, or late fusion, respectively (see Appendix ?? ). Features f ( l ) and g ( k ) pertain to two distinct modalities and are subject to immeasurable unknown confounders. Moreover, they also share partial common information. The goal is to find potential similarities (mainly caused by confounding factors and dependencies during data collection) between these layers and then remove those (Figure 1). Mutually-independent layers can be then fused in the multimodal network for a

(+V'M)d

W

0.8

9.4×10-3

16

68

0.6

0.4

7.4×10-3

0.2

8

5.7×10-3

F0°0

-0.2

3.8×10-3

36

24

16

20

0.4

0.6

84

60

48

64

40

56

1.9×10-3

80

72

0.8

-0.8-0.6-0.4-0.2

0.0

0.2

0.4

0.6

0.8

given task. To this end, one can represent f ( l ) based on g ( k ) as a linear regression model

Here, W ( l,k ) is a projection matrix of size n × m and f ( l ) r represents the difference between f ( l ) and its corresponding map in the domain of g ( k ) , also known as the residual. Ideally, the residual map does not contain any features of g ( k ) , so f ( l ) r and g ( k ) are mutually independent. RegBN minimizes the linear relationship between these layers via

The constraint in terms of the Frobenius norm, ∥ ∥ W ( l,k ) ∥ ∥ F = ∑ n i =1 ∑ m j =1 ω ( l,k ) 2 i,j = 1 , guarantees that the network does not experience vanishing and exploding gradients during training and inference. Eq. 2 is a regularization method for ill-conditioned problems [37, 9, 34, 47]. A Lagrangian multiplier offers an equivalent formulation F to the optimization defined at Eq. 2:

In this equation, λ + is a positive Lagrangian multiplier, playing the role of a dual variable. Minimizing F ( W ( l,k ) , λ + ) over W ( l,k ) yields the projection matrix

where, ˆ λ + is the estimated Lagrangian multiplier obtained through minimization of the term ∥ ∥ W ( l,k ) ∥ ∥ F -1 ; I is an identity matrix and superscript ⊤ represents the transpose operator. We employ singular value decomposition (SVD) and limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) [28] for solving Eq. 2. The solution is detailed in Appendix ?? , and the proposed method is summarized in Algorithm 1. It is worth noting that the projection weights are not directly trained by the multimodal criterion since there is a danger of being fooled by confounders. Instead, the projection weights are learned through Eqs. 2-4 and updated recursively over chunks of training data, detailed below.

3.1 Update of the projection matrix

The projection matrix, W ( l,k ) , is calculated for every mini-batch so we recursively update that via the exponential moving average's approach [21], which decays the mean gradient and variance for each

Algorithm 1 Pseudocode of RegBN

1: Inputs : f ( l ) ∈ R b × n and g ( k ) ∈ R b × m : l -th and k -th layer of an MML, respectively

2: Other inputs : t (timestep/mini-batch), projection weights W ( l,k ) , Λ t -1 (a collection of estimated λ + values until previous timestep), training ;

4: Set Λ t through Eq. 8 and given Λ t -1 ;

3: Compute SVD of g ( k ) : SVD ( g ( k ) ) = U Σ V ∗ = ∑ m i =1 σ i u i v ∗ i

5: for λ in Λ t do

6: Initialize λ + with λ and then apply the L-BFGS algorithm to approximate Eq. 7;

7: Store the estimated λ + and the approximation error;

8: end for

9: Choose λ + with the lowest approximation error as the sub-optimal ˆ λ + ;

10: Update Λ t : Λ t ← Λ t -1 ∪ ˆ λ t ;

11: Insert ˆ λ + to Eq. 4 for computation of the projection matrix: ˆ W ( l,k ) = ∑ m i =1 ( σ i u i f ( l ) σ 2 i +ˆ λ + ) v i ;

12: if training then

13: Normalize the input feature layer using ˆ W ( l,k ) : f ( l ) r ← f ( l ) -ˆ W ( l,k ) g ( k ) ;

14: end if

15: Update the projection weights W ( l,k ) t via Eqs. 5 and 6;

16: if not training then

17: Normalize the input feature layer using W ( l,k ) t : f ( l ) r ← f ( l ) -W ( l,k ) t g ( k ) ;

18: end if

19: return f ( l ) r , W ( l,k ) t , and Λ t

variable exponentially. Let W ( l,k ) t -1 denote the updated projection matrix until the previous timestep. We define ∆ W t as the mean absolute error between the currently predicted projection matrix and the previously updated projection matrix: ∆ W t = ∥ ∥ ∥ ˆ W ( l,k ) -W ( l,k ) t -1 ∥ ∥ ∥ 1 . The projection matrix is then updated at timestep t via

Here, γ t is the learning rate of the multimodal model at timestep t ; m t and ν t are the first and second moments, respectively, derived from

where β 1 , β 2 ∈ (0 , 1) represent constant exponential decay rates. As mentioned in Algorithm 1, the projection matrix is updated during training only (Algorithm 1: Steps 11-13), and we use the latest updated projection weights for validation or inference (Algorithm 1: Steps 15-17).

3.2 Avoiding falling into local minima

Multimodal networks often contain several backbones (e.g., MLPs, CNNs, ViTs), depending on the number of multiple heterogeneous data sources. Identifying the global minima of such networks is a challenging problem, as the networks may fall into local minima. On the other hand, optimization of Eq. 3 relies on λ + , which may introduce several local minima. Conventional regression methods treat λ + as a constant/hyperparameter, while predetermined hyperparameters usually increase the risk of falling into local minima. To prevent this, we adopt a mini-batch-wise approach for the estimation of λ + . The objective is to estimate suboptimal λ + with the L-BFGS optimization algorithm in a way that meets

Quasi-Newton methods like L-BFGS rely on the initial value of λ + for minimization of the optimization problems like Eq. 7 (see Figure 2a). Due to the limited number of local minima that exist per mini-batch, we adopt a method whereby we initialize λ + with multiple random values within a

predetermined range Λ p = [ λ p 1 , . . . , λ p C ] as well as with the median of the suboptimal λ + values predicted in the previous mini-batches, µ 1 2 (Λ t -1 ) :

By creating a loop over Λ t , we first initialize λ + and then estimate its true value using the L-BFGS method. We also store the approximation error and after the termination of the loop, the suboptimal value of λ + for a given mini-batch is determined by selecting the estimated value that yields the lowest approximation error. After calculating the suboptimal lagrangian multiplier, we update Λ , i.e., Λ t ← Λ t -1 ∪ ˆ λ t . As illustrated in Figure 2b, when the multimodal model approaches its global minimum, the parameters of RegBN also converge.

4 Experiments

Heterogeneous data captured by diverse multisource sensors are utilized to verify the usefulness and effectiveness of RegBN in multiple data contexts such as language, audio, 2D image, video, depth, 3D MRI, and tabular data. RegBN is applied to eight datasets that are summarized in Table 1. Details on the datasets and baseline methods are provided in Appendices ?? & ?? . The default parameters and settings for RegBN are reported in Appendix ?? . Our code is openly available at https://mogvision.github.io/RegBN . Experimental details are provided in Appendix ?? . Here, we summarize the main results. The findings of this investigation provide insights into the performance of RegBN across various data modalities and highlight its potential as a robust normalization technique. PMDN relies on a considerable number of learnable parameters specifically tailored for metadata with limited dimensions. MDN necessitates substantial RAM resources, particularly when estimating its inverse matrix on large-scale datasets.

4.1 Multimedia

Experiments with the LLP dataset [53] : Audio-Visual Video Parsing (AVVP) [53] (see Appendix ?? ) is employed as a baseline for parsing individual audio, visual, and audio-visual events under both segment-level and event-level metrics over the LLP dataset. The normalization module is employed to decouple the audio features from the video features, ensuring their independence. Table 2 reports that RegBN improves the baseline performance in nine out of ten metrics. RegBN brings about improvements in all audio-visual video parsing subtasks, measured both at the segment-level and event-level metrics. These findings suggest that decoupling audio and video allows AVVP to produce more accurate predictions of event categories on a per-snippet basis. 1

Experiments with the MM-IMDb dataset [3] : The MM-IMDb dataset was curated for the purpose of predicting movie genres through the use of either image or text modality. This task involves

multi-label classification since a single movie may be associated with multiple genres (see Appendix ?? ). SMIL [31], as a baseline approach, employs the pre-trained BERT to extract the textual features, while image features are extracted using the pre-trained VGG-19. The text features are subsequently normalized with respect to the visual features using the proposed RegBN before the fusion process occurs. The fusion operation involves a concatenation layer. The fused features are passed through two fully-connected layers to obtain the classification labels. The details are provided in Appendix ?? . Table 3 presents the quantitative results obtained from the experiments. The table indicates that applying a normalization layer such as PMDN or RegBN can lead to an improvement in the performance over the baseline model. This highlights the importance of the normalization step in multimedia data. In particular, RegBN was found to enhance the F1 score of the baseline by an average of 8.86% across all three metrics.

Experiments with the AV-MNIST dataset [31] : The small AV-MNIST dataset comprises two modalities, namely, audio and image. The audio modality is obtained from Free Spoken DigitsDataset 2 , which includes 1,500 raw audio recordings from three different speakers. Despite the independence of the image and audio modalities, the primary objective of this experiment is to investigate whether the confounding effects induced by the speakers have an impact on the classification results. In line with the previous experiment, SMIL [31] serves as the baseline method. The obtained classification results for the combinations of baseline+MDN, baseline+PMDN, and baseline+RegBN are 98.43%, 98.68%, and 99.11%, respectively. Figure 3 illustrates t-SNE plots for the different approaches. In this experiment, tSNE is applied to the features extracted from a fully-connected layer after the concatenation of normalized image features with audio features. The figure demonstrates that the classes are more separable for RegBN as compared to MDN or PMDN, in particular, class '2'. This result implies that our normalization method effectively removes the confounding effects caused by different speakers, thereby providing more flexibility to the baseline method for accurate classification. The results on AV-MNIST are reported in Appendix ?? .

4.2 Affective computing

In this section, we explore the potential application of RegBN in multimodal emotion-sentiment analysis using the IEMOCAP [8], CMU-MOSI, and CMU-MOSEI [63] datasets, which are divided into aligned and non-aligned multimodal time-series (see Appendices ?? , ?? , & ?? ). The baseline method (BL) for this task is the Multimodal Transformer (MulT) [54], which is a ViT developed for analyzing multimodal language sequences. MulT fuses video ( v ), audio ( a ), and language ( l ) at three levels (see Appendix ?? ): 1) language fusion ( v → l & a → l ), 2) audio fusion ( v → a & l → a ), and 3) video fusion ( a → v & l → v ). The outputs of one or more levels are then aggregated for

(a) Only image

(b) Only spectrogram

(c) MDN

(d) PMDN

(e) RegBN

0

2

4

6

8

1

3

5

7

9

classification. Here, we report the results of all three fusion levels combined, while the detailed results for each level are provided in Appendix ?? . The quantitative results of the emotion analysis experiment are reported in Table 4. The table shows that RegBN improves the classification performance of MulT in most cases, particularly for the 'natural' class of the word-aligned experiment. Table 5 presents the results of the multimodal sentiment analysis on the CMU-MOSI and CMU-MOSEI datasets. In both the word-aligned and unaligned experiments, decoupling the multimodal features with RegBN improves training and inference performance.

4.3 Healthcare diagnosis

The ADNI dataset [19] includes 3D MRI scans of patients along with rich clinical information organized in a low-dimensional tabular format (see Appendix ?? ). The dataset is subject to confounding effects such as age, sex, and level of education, which must be considered to prevent biased evaluation results, as noted in studies [59, 39]. Similar to [60], we partitioned the data into five separate and non-overlapping folds, ensuring that each fold had a balanced distribution of diagnosis, age, and sex. In this experiment, we utilized a 3D ResNet [60] for extracting MRI features, while tabular features were extracted using an MLP network proposed in [14] (see Appendix ?? ). These features were then concatenated and fed into another MLP network for diagnosis classification. We applied different normalization techniques to the unimodal MRI and tabular features prior to concatenation. Table 6 reports the mean and standard deviation obtained by averaging the performance of the multimodal over these five folds. The complete results can be found in Appendix ?? . RegBN emerged as the most effective technique, outperforming other normalization methods in terms of both ACC and BA metrics. Moreover, RegBN's ability to relax the interdependencies between the different modalities facilitated lower training loss in the multimodal network.

4.4 Robotics

The Vision&Touch dataset [23] encompasses recordings of simulated and real robotic arms that are equipped with visual (RGB and depth), force, and proprioception sensors (see Appendix ?? ). We employed 'Making Sense of Vision and Touch' (MSVT) developed by Lee et al. [23] as a baseline (refer to Appendix ?? ). For this experiment, we apply normalization techniques to decouple the visual RGB and force from depth information. Table 7 summarizes the results, while detailed results can be found in Appendix ?? . The flow loss values in these tables indicate that MSVT converges more effectively when used in conjunction with RegBN. This suggests that RegBN is able to effectively remove dependencies between the RGB and depth pair, as well as the force and depth pair.

4.5 Synthetic dataset

Drawing from the synthetic dataset in study [30], we designed a synthetic experiment for a binary classification task. Images with a size of 64 × 64 are split into four regions with identical Gaussians in their background. The regions along the main diagonal are linked to the class label, while the off-diagonal ones act as confounders that do not impact the true label. We created metadata of size 16 for this dataset that includes the true binary label, the confounder value, one fake label, 12 randomly generated fake features, and 1 column with a value of one. The binary classes have overlapping, so the accuracy drops based on the amount of overlapping (see Appendix ?? ). In Experiment I, the theoretical maximum accuracy of classification is 87.5%, while for Experiment II, it is 75%. The base networks employed here are a simple CNN and an MLP-based neural network. The dataset and base networks are detailed in Appendices ?? and ?? , respectively. The classification results are presented in Table 8, with more detailed results available in Appendix ?? . The results reveal that conventional normalization techniques, such as BN, GN, and IN, are unable to remove the confounding effects. The performance of the base networks that employed MDN was superior compared to those with conventional normalization methods. However, MDN still exhibits shortcomings in dealing with confounding effects. Despite the incorporation of a high number of learnable parameters, PMDN was unsuccessful in eliminating confounding effects, as the inclusion of fake values in the metadata led to the deception of the multimodal model. RegBN demonstrated its ability to remove confounder effects by yielding performance closest to the theoretical maximum in both experiments.

4.6 Computational cost and ablation study

We employed an NVIDIA GTX 1080 Ti with 12GB VRAM for image experiments and an NVIDIA A100 with 80GB VRAM for video experiments. Since RegBN learns the projection matrix through Eqs. 4-7, it eliminates the need for learnable parameters, resulting in enhanced computational efficiency for multimodal models. During the training process, RegBN achieves a frame rate of approximately 124 fps when normalizing a pair of layers with dimensions 256 × 1024 . In comparison, conventional normalization techniques achieve a frame rate of around 13,000 fps for the normalization of one layer of dimensions 256 × 1024 . Regarding inference time, RegBN exclusively relies on the updated projection matrix and therefore does not require additional computation. RegBN is frequently used before the fusion blocks in multimodal models, so the training time of a few RegbBN layers is negligible. Results for the ablation study are provided in Appendix ?? .

5 Conclusion and future work

This study presented a novel normalization method designed for dependency and confounding removal in multimodal data and models. The experimental results demonstrated the effectiveness of RegBN in significantly enhancing the accuracy and convergence of multimodal models, rendering it a promising normalization method for multimodal heterogeneous data. The current version of RegBN employs L-BFGS on a single GPU, which will be improved in the future. Our aspiration is that this work effectively leverages new opportunities in exploring and harnessing multi-modality data within multimodal analysis.

References

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems , 34:24206-24221, 2021.

Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovi´ c, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems , 33:25-37, 2020.

John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. In 5th International conference on learning representations workshop. , 2017.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Anas Belouali, Samir Gupta, Vaibhav Sourirajan, Jiawei Yu, Nathaniel Allen, Adil Alaoui, Mary Ann Dutton, and Matthew J Reinhard. Acoustic and language analysis of speech for suicidal ideation among us veterans. BioData mining , 14(1):1-17, 2021.

Ioana Bica and Mihaela van der Schaar. Transfer learning on heterogeneous feature spaces for treatment effects estimation. arXiv preprint arXiv:2210.06183 , 2022.

Brandon M Booth, Louis Hickman, Shree Krishna Subburaj, Louis Tay, Sang Eun Woo, and Sidney K D'Mello. Bias and fairness in multimodal machine learning: A case study of automated video interviews. In Proceedings of the 2021 International Conference on Multimodal Interaction , pages 268-277, 2021.

Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation , 42:335-359, 2008.

D Calvetti, S Morigi, L Reichel, and F Sgallari. Tikhonov regularization and the l-curve for large discrete ill-posed problems. Journal of computational and applied mathematics , 123(1-2): 423-446, 2000.

Joseph DelPreto, Chao Liu, Yiyue Luo, Michael Foshey, Yunzhu Li, Antonio Torralba, Wojciech Matusik, and Daniela Rus. Actionsense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment. Advances in Neural Information Processing Systems , 35:13800-13813, 2022.

Aymeric Dieuleveut, Gersende Fort, Eric Moulines, and Geneviève Robin. Federatedem with heterogeneity mitigation and variance reduction. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 29553-29566. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ f740c8d9c193f16d8a07d3a8a751d13f-Paper.pdf .

Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. Advances in Neural Information Processing Systems , 34:15748-15761, 2021.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15180-15190, 2023.

Paul Hager, Martin J Menten, and Daniel Rueckert. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. accepted for CVPR , 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Yan Huang, Yuming Wang, Yunan Zeng, and Liang Wang. Mack: multimodal aligned conceptual knowledge for unpaired image-text matching. Advances in Neural Information Processing Systems , 35:7892-7904, 2022.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al. The alzheimer's disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine , 27(4):685-691, 2008.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR , 2015.

Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple architecture for joint learning for multimodal tasks. arXiv preprint arXiv:2303.16839 , 2023.

Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8943-8950. IEEE, 2019.

Michelle A Lee, Brent Yi, Roberto Martín-Martín, Silvio Savarese, and Jeannette Bohg. Multimodal sensor fusion with differentiable filters. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 10444-10451. IEEE, 2020.

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. arXiv preprint arXiv:2107.07502 , 2021.

Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, LouisPhilippe Morency, and Ruslan Salakhutdinov. Multiviz: Towards visualizing and understanding multimodal models. In The Eleventh International Conference on Learning Representations , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming , 45(1-3):503-528, 1989.

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining taskagnostic visiolinguistic representations for vision-and-language tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ c74d97b01eae257e44aa9d5bade97baf-Paper.pdf .

Mandy Lu, Qingyu Zhao, Jiequan Zhang, Kilian M Pohl, Li Fei-Fei, Juan Carlos Niebles, and Ehsan Adeli. Metadata normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10917-10927, 2021.

Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 2302-2310, 2021.

Daniel Martin, Sandra Malpica, Diego Gutierrez, Belen Masia, and Ana Serrano. Multimodality in vr: A survey. ACM Computing Surveys (CSUR) , 54(10s):1-36, 2022.

Shammi More, Simon B Eickhoff, Julian Caspers, and Kaustubh R Patil. Confound removal and normalization in practice: A neuroimaging based sex prediction case study. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part V , pages 3-18. Springer, 2021.

Henrique Gomes Moura, Edson Costa Junior, Arcanjo Lenzi, and Vinicius Carvalho Rispoli. On a stochastic regularization technique for ill-conditioned linear systems. Open Engineering , 9(1):52-60, 2019.

Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems , 34:14200-14213, 2021.

Marla Narazani, Ignacio Sarasua, Sebastian Pölsterl, Aldana Lizarraga, Igor Yakushev, and Christian Wachinger. Is a pet all you need? a multi-modal study for alzheimer's disease using 3d cnns. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 66-76. Springer, 2022.

Arnold Neumaier. Solving ill-conditioned and singular linear systems: A tutorial on regularization. SIAM review , 40(3):636-666, 1998.

Sebastian Pölsterl and Christian Wachinger. Estimation of causal effects in the presence of unobserved confounding in the alzheimer's continuum. In Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28-June 30, 2021, Proceedings 27 , pages 45-57. Springer, 2021.

Sebastian Pölsterl and Christian Wachinger. Identification of causal effects of neuroanatomy on cognitive decline requires modeling unobserved confounders. Alzheimer's & Dementia , 2022.

Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger. Combining 3d image and tabular data via the dynamic affine feature map transform. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part V 24 , pages 688-698. Springer, 2021.

Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. A review of affective computing: From unimodal analysis to multimodal fusion. Information fusion , 37:98-125, 2017.

Mohamad Amin Pourhoseingholi, Ahmad Reza Baghestani, and Mohsen Vahedi. How to control confounding effects by statistical analysis. Gastroenterology and hepatology from bed to bench , 5(2):79, 2012.

Jennifer Preece, Helen Sharp, and Yvonne Rogers. Interaction design: beyond human-computer interaction . John Wiley & Sons, 2015.

Yanyuan Qiao, Yuankai Qi, Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: Historyenhanced and order-aware pre-training for vision-and-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-14, 2023. doi: 10.1109/TPAMI.2023. 3234243.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent, 2022. URL https://arxiv. org/abs/2205.06175 , 3, 2022.

Dominic Richards, Sahand Negahban, and Patrick Rebeschini. Distributed machine learning with sparse heterogeneous data. Advances in Neural Information Processing Systems , 34: 18008-18020, 2021.

Philipp V Rouast, Marc TP Adam, and Raymond Chiong. Deep learning for human affect recognition: Insights and new developments. IEEE Transactions on Affective Computing , 12(2): 524-543, 2019.

Septimiu E. Salcudean, Hamid Moradi, David G. Black, and Nassir Navab. Robot-assisted medical imaging: A review. Proceedings of the IEEE , 110(7):951-967, 2022. doi: 10.1109/ JPROC.2022.3162840.

Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. A survey of multimodal sentiment analysis. Image and Vision Computing , 65:3-14, 2017.

Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. arXiv preprint arXiv:2210.06031 , 2022.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weaklysupervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 436-454. Springer, 2020.

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting , page 6558. NIH Public Access, 2019.

Anthony Vento, Qingyu Zhao, Robert Paul, Kilian M Pohl, and Ehsan Adeli. A penalty approach for normalizing feature distributions to build confounder-free models. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part III , pages 387-397. Springer, 2022.

Gyanendra K Verma. Multimodal affective computing: Affective information representation, modelling, and analysis. 2023.

Christian Wachinger, Anna Rieckmann, and Sebastian Pölsterl. Detect and correct bias in multi-site neuroimaging datasets. Medical Image Analysis , 67:101879, 2021.

Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in neural information processing systems , 33:4835-4845, 2020.

Junhao Wen, Elina Thibeau-Sutre, Mauricio Diaz-Melo, Jorge Samper-González, Alexandre Routier, Simona Bottani, Didier Dormont, Stanley Durrleman, Ninon Burgos, Olivier Colliot, et al. Convolutional neural networks for classification of alzheimer's disease: Overview and reproducible evaluation. Medical image analysis , 63:101694, 2020.

Tom Nuno Wolf, Sebastian Pölsterl, Christian Wachinger, Alzheimer's Disease Neuroimaging Initiative, et al. Daft: A universal module to interweave tabular data and 3d images in cnns. NeuroImage , 260:119505, 2022.

Tom Nuno Wolf, Sebastian Pölsterl, and Christian Wachinger. Don't panic: Prototypical additive neural network for interpretable classification of alzheimer's disease. In International Conference on Information Processing in Medical Imaging , pages 82-94. Springer, 2023.

Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV) , pages 3-19, 2018.

Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259 , 2016.

AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2236-2246, 2018.

Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. Training confounder-free deep learning models for medical applications. Nature communications , 11(1):6010, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on  mitigate the influence of unwanted variability and bias in multimodal data. To this end, the authors proposed a novel non-parametric regularization technique, i.e., RegBN. The proposed method utilizing Frobenius norm to constrain cross modality consistency, enabling model to discern underlying patterns. Comprehensive experiments show the usefulness of proposed approach in various datasets from different research areas.\n\nSTRENGTHS:\n1. Paper is well-written and follows a good structure. \n2. Comprehensive performance evaluation are shown, involving various recent multimodal methods and seven datasets, that proposed approach is working in practice.\n3. Mitigating unwanted variability and bias of multimodal data is an important topic and few methods have touched this field. The proposed method is interesting. Improving the reliability of multimodal learning by normalizing multimodal feature is a new perspective.\n\nWEAKNESSES:\nMajor\n\n- Method: The proposed method mitigates the influence of unwanted variability and bias by minimizes the linear relationship between multimodal layers. In Figure 1, the authors claim that \"by leveraging RegBN, these data sources can be rendered independent, thereby enabling a multimodal network to discern underlying patterns and optimize its performance\". To the best of my knowledge, exploring consistency and complementarity[1] may be two mainly underlying reason why multimodal model outperforms the unimodal. Does the proposed method ignore to explore the cross-modal consistency?\n\n- Analysis: In line 108, the authors claim that \"The goal is to find potential similarities (mainly caused by confounding factors and dependencies during data collection)\". In my view, this is an essential assumption for this paper. I think the authors should provide more discussion about why potential confounding factors and dependencies cause these similarities. It could be necessary to give some explanation and citations.\n- Comparisons: In line 33 \"For instance, the race or gender of speakers in audio classification, backgrounds in video parsing, or the education level of patients in dementia diagnosis is commonly recognized as confounders.\". This statements seems very closely to the motivation of DRO[2]. It is appreciated to give some discussion about the relationship between RegBN and DRO.\n\nMinor\n\n- Ambiguous notations: In line 101 and 102, what is the meaning of N and M?\n\n[1] Deep Partial Multi-View Learning, Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu and Qinghua Hu, IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE T-PAMI)\n\n[2] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781–6792. PMLR, 2021.\n\nQUESTIONS:\nPlease address the issues raised in the weakness section.\n\nLIMITATIONS:\nPlease address the issues raised in the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper introduces an approach for the normalization of multimodal data called RegBN. The RegBN aligns multimodal features by a learnable projection matrix with regularization in the form of the Frobenius norm. The regularization strength is updated with the L-BFGS optimization algorithm. Extensive experiments on various modalities are conducted to verify the method.

STRENGTHS:
- The paper is well-written and easy to follow.
- Aligning multimodal data from the perspective of the normalization method can be promising.
- Extensive experimental results show the effectiveness of the proposed RegBN.

WEAKNESSES:
- The method is not very novel for the reviewer. Essentially, RegBN is an l2 regression with regularization in the form of the Frobenius norm on weight. Moreover, L-BFGS has been widely adopted to update the regularization strength.
- From the extensive experiments, RegBN is actually a multimodal feature fusion module. Other than baselines of normalization methods such as MDN and PMDN, more comparisons with other multimodal feature fusion modules such as [1] and [2] should be provided.
- Why the method is called RegBN. It is not clear how RegBN is connected with the normalization method.

[1] Yikai Wang et al. Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion.  2021
[2] Yikai Wang et al. Deep Multimodal Fusion by Channel Exchanging. 2020

QUESTIONS:
Please see the weaknesses. Overall, this paper proposes a simple but effective scheme to better fuse multimodal features. The technical contribution is limited. The experimental results are comprehensive and convincing.  It would be better to provide more comparative experiments as suggested.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach called RegBN for normalizing multimodal data before fusion in neural networks. The integration of heterogeneous multimodal data poses challenges due to confounding effects and dependencies among different data sources, which can introduce variability and bias. RegBN addresses these issues by using the Frobenius norm as a regularizer term. The method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. The effectiveness of RegBN is validated on eight databases from various research areas, covering diverse modalities. The proposed method shows broad applicability across different neural network architectures, enabling effective normalization of both low- and high-level features in multimodal models. Overall, RegBN offers a promising solution for improving the performance of multimodal models by addressing normalization challenges.\n\nSTRENGTHS:\nThe paper presents a novel idea regarding multi-modal fusion. The proposed method is model-agnostic, i.e., the authors provide details about how to apply RegBN within different frameworks, including late fusion, layer (intermediate) fusion, and early fusion. This method does not require significant modification of the original framework and can be plugged in regardless of the encoder's architecture. \n\nExtensive experiments are conducted, including eight datasets covering applications in multimedia, affective computing, healthcare diagnosis, and robotics. On all datasets, the proposed method outperformed the considered baseline: vanilla model, or model with normalization in other manner such as BatchNorm or PMDN. \n\nThis work provides a relatively unique exploration of the topic of multi-modal learning which I believe is worth presenting to the community.\n\nWEAKNESSES:\nAuthors conducted experiments between with/without RegBN and also compared with other normalization methods.\nIt will be interesting to see how does RegBN help regardling of the problem of imbalanced modality utilization, as pointed out in recently papers [1, 2, 3]\n\n[1] Wang, Weiyao et al. “What Makes Training Multi-Modal Classification Networks Hard?” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 12692-12702.\n[2] Huang, Yu et al. “What Makes Multimodal Learning Better than Single (Provably).” Neural Information Processing Systems (2021).\n[3] Wu, Nan et al. “Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks.” ICML 2022.\n\nQUESTIONS:\n- Typo in the caption of Figure 3. Subfigures are indexed incorrectly. \n- The authors should remember to correct the misstatement in the main paper about SMIL for MM-IMDb.\n\nLIMITATIONS:\nAuthors mentioned that the current algorithm works only on single GPU.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes RegBN for the normalization of multimodal data. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different sources of data. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. Experiments on eight databases from five research areas demonstrate the effectiveness of the proposed method.

STRENGTHS:
- This paper proposes a novel RegBN method for the normalization of multimodal data.
- Extensive experiments demonstrate the effectiveness of the proposed method.

WEAKNESSES:
- My main concern is that the proposed method lacks comparison with other competitive normalization methods such layer norm and group norm.
- Using eight databases from five research areas is not necessary. The authors should focus on two or three datasets.
- Synthetic multimodal dataset should be provided to thoroughly evaluate the effectiveness of the proposed method.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel regularization approach tailored for multimodal/heterogenous data fusion, which can be applied off-the-shelf to modern neural network architectures. The approach, namely RegBN, targets the reduction/removal of confounding factors and partial dependencies that are usually present among features extracted from different modalities. To do so, RegBN is implemented as a regularization technique that promotes feature independence across data modalities. The regularization is iteratively optimized over mini-batches using a separate solver not relying on back propagation and gradient descent. This separation is critical as otherwise the network could simply learn to use the confounding factors instead of removing them. Experiments are performed on several multimodal datasets based on different modalities and spanning across different applications, such as multimedia, affective computing, healthcare diagnosis, and robotics. Moreover, a synthetic dataset is designed to further highlight the impact of RegBN. The results showcase a positive impact in all the test cases.\n\nSTRENGTHS:\n1. The approach is novel and grounded with theoretical motivation\n2. The method can be plugged off-the-shelf to most multimodal architectures, possibly leading to vaste adoption\n3. Wide breadth of experiments that explore several multimodal scenarios\n4. Positive results in all scenarios\n\nWEAKNESSES:\n1. No integration with recent widely adopted multimodal foundation models, such as CLIP, ImageBind, and many others. Despite I understand that RegBN purpose contrasts with the multimodal alignment objective driving those models, I believe it would be valuable to bring them to the discussion. It would also be beneficial for the positioning of the paper.\n2. The boost is sometimes minor, e.g see Table 2, 5, I.3, and in some settings even slightly negative, e.g., see Table 5 and I.4 (video fusion rows). Also, it is unclear the high difference between AV-MNIST results reported in the main paper (~ 99% accuracy) and the ones reported in table I.3 (~71% accuracy). Furthermore, why for the experiments on LLP, training metrics are reported in Table 2 and validation metrics in the supplementary? \n3. As RegBN requires a separate optimization, its integration affects the time complexity of the model training. The Authors provides multiple examples on how RegBN could/should be integrated in existing architectures, see Figure I.1, however, in most cases a single RegBN layer is added to the network. It is unclear whether this choice is suboptimal for performance in favor efficiency, or whether it is useless to have multiple RegBN as in I.1c.\n4. Minor:\n    1. The writing is for sure enjoyable, however there are some parts that could have better explained / could be less ambiguous. For instance, in section 3: _\"Ideally, the residual layer ... RegBN minimizes the linear relationship between these layers via: equation (2)\"_, but then equation (2) does only contain $f$ and $g$, and thus it is completely tacit the key insight on why solving equation(2) would provide $f_r$ orthogonal to $g$, as well as it is not explicit that $f_r$ corresponds to the normalization of $f$. Moreover, Figure 1 provides a general idea of the impact of RegBN, but it could have been more aligned with the formulation, e.g., by adding $f$, $f_r$, and $g$; and in general the caption could be expanded to better describe it — what does the grey band represent?. Finally the positioning could have been more clear by stating why recent multimodal foundation models like CLIP are out of the scope.\n    2. “multimodAl” often mistyped as “multimodEl”; Table I.5, 3rd column, video fusion, (70.6) vs (81.4) seems a bit suspicious; Color gradient is not explained in Figure I.9; wrong x axis label in Figure I.10;\n\nQUESTIONS:\n1. Do the Authors envision any use/application of RegBN for large foundation models, e.g., is it meant for whenever they are employed for downstream tasks?\n2. What could be the reason why RegBN affects negatively performance? Is it due to the fact that when evaluating some dataset in-distribution learning spurious correlation might provide a boost?  Please comment also on the last two sentences of weakness # 2.\n3. Is it enough to apply RegBN once as shown in I.1a,b,d or might be worth to apply it to several layers?\n4. Consider slightly modifying sec. 3 and Figure 1, as well as adding somewhere (introduction or related work) a few positioning sentences wrt foundation models. Fix other typos/mistakes.\n\nLIMITATIONS:\nSocietal impact is not discussed. I believe that RegBN could provide a benefit in reducing stereotypical biases that currently affect multimodal models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-29"}