RegBN: Batch Normalization of Multimodal Data with Regularization
Authors: MORTEZA GHAHREMANI
Christian Wachinger
Venue: NeurIPS 2023 poster (2023)
Keywords: Multimodal Data
Multimodality
Batch Normalization
Heterogeneous data
Regularization
Confounder
Confounding Effect Removal
Data Dependency
TL;DR: RegBN is a novel normalization method for multimodal data based on regularization that mitigates the confounding effects and the underlying dependencies among multimodal data, thereby enhancing the learning of multimodal representations.

Abstract: Recent years have witnessed a surge of interest in integrating high-dimensional data captured by multisource sensors, driven by the impressive success of neural networks in integrating multimodal data. However, the integration of heterogeneous multimodal data poses a significant challenge, as confounding effects and dependencies among such heterogeneous data sources introduce unwanted variability and bias, leading to suboptimal performance of multimodal models. Therefore, it becomes crucial to normalize the low- or high-level features extracted from data modalities before their fusion takes place. This paper introduces RegBN, a novel approach for multimodal Batch Normalization with REGularization. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different data sources. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. We validate the effectiveness of RegBN on eight databases from five research areas, encompassing diverse modalities such as language, audio, image, video, depth, tabular, and 3D MRI. The proposed method demonstrates broad applicability across different architectures such as multilayer perceptrons, convolutional neural networks, and vision transformers, enabling effective normalization of both low- and high-level features in multimodal neural networks. RegBN is available at https://mogvision.github.io/RegBN.

RegBN: Batch Normalization of Multimodal Data with Regularization
Morteza Ghahremani 1.2
Christian Wachinger
Lab for AI in Medical Imaging (AI-Med), Department of Radiology, Technical University of Munich (TUM), Germany
2 Munich Center for Machine Learning (MCML), Germany
(morteza. ghahr ghanremani,, christian. dbristian.wachinger wachinger) Stum. de
Abstract
Recent years have witnessed surge interest integrating high-dimensional data captured by multisource sensors, driven by the impressive success of neural networks integrating multimodal data. However, the integration hetero- geneous multimodal data poses significant challenge, confounding effects and dependencies among such heterogeneous data sources introduce unwanted variability and bias, leading suboptimal performance of multimodal models. Therefore, it becomes crucial to normalize the low- or high-level features ex- tracted from data modalities before their fusion takes place. This paper introduces RegBN, novel approach for multimodal Batch Normalization with REGular- ization. RegBN uses the Frobenius norm regularizer term address the side effects of confounders and underlying dependencies among different data sources. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. We validate the effectiveness of RegBN on eight databases from five research areas, encompassing diverse modalities such as language, audio, image, video, depth, tabular, and 3D MRI. The proposed method demonstrates broad applicability across different architectures such as multilayer perceptrons, convolutional neural networks, and vision transformers, enabling effective normalization of both low- and high-level features in multimodal neural networks. RegBN is available at https: : / ht/////oggvioooooo/oo gi github....... io/ii///eee Lo/RegBN.
1 Introduction
Multimodal models, which adeptly fuse information from a diverse range sources, have yielded promising results and found many applications such as language and vision [27, 29, 44, 17, 51], multimedia [2, 35, 12), affective computing [63, 41, 64, 48, 56], robotics [23, 49, 24], human- computer interaction [43, 41], and healthcare diagnosis [40, 60, 36, 14, 61]. Multimodal machine learning presents distinctive computational and theoretical research challenges due to the diversity of data sources involved [25, 26]. The impressive versatility and efficacy of multimodal neural network models can be attributed to their ability to effectively integrate and leverage heterogeneous data.
Multimodal models process heterogeneous information obtained from multisource sensors by ex- tracting their features. Subsequently, the extracted features are fused different levels (including early, middle, or late fusion [58, 35, 36]) to address specific tasks, such as classification, recognition, description, and segmentation [25, 26, 12, 1, 10]. Heterogeneous information from multisource sensors, however, is susceptible to confounding effects caused by extraneous variables or multiple distributions [30, 7, 11, 57, 32, 6]. Confounding variables pertain to external factors that introduce bias (either positive negative) in the relationship between the variables being studied [42, 50]. The complexity confounders emerges from their potential pervasiveness across diverse data modal-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

M
Multimodal low/high level
Mutually Independent Multimodal Features
Normalization via RegBN
Features
Confounder
Mutuallly Dependent
The presence of heterogeneous data often entails external confounding effects (denoted by 'C" in the figure) and partial dependencies, which impede the efficient training of a multimodal network. This study recommends the normalization of low- and high-level features extracted by foundation models through the use of RegBN. This allows the normalised features to be rendered independent, enabling multimodal network discern underlying patterns and optimize its perfor- mance.
Figure 1:
ities. For instance, in image analysis, confounders might encompass lighting variations, while in audio classification, speaker attributes like race or gender can be confounding factors. In video parsing, backgrounds play role, and in dementia diagnosis, the education level of patients can be a confounder. Furthermore, positive or negative correlations can exist among heterogeneous data that impact the distributions of the learned features [50, 30]. These factors pose challenges for multimodal network to accurately uncover the actual relationship between variables for given task. Ignoring the confounding effects and partial dependencies in features or data can result in substantial drawbacks, including the deviation of multimedia model from its global minimum, reduction in the speed and stability of neural network training, and the generation of unreliable or misleading out- comes. Consequently, it is recommended to normalize features from different modalities before their fusion [30, 55]. this study, we show that the normalization step can help mitigate the confounding effects and enhance the overall performance and reliability of the multimodal networks.
Normalization techniques have proven to be highly effective in enhancing the speed and stability of neural network training. By re-centering and re-scaling the input layers, the normalization methods provide a promising solution overcome the challenges of training deep neural networks. Batch normalization (BN) [18], layer normalization (LN) [4], group normalization (GN) [62]. and instance normalization (IN) [62] are popular normalization techniques that have been used in the foundation of many sate-off-hhe-at neural networks such as ResNet [15], DenseNet [16], Inception [52], and others. The aforementioned normalization methods are designed to standardize feature distributions and do not take into account confounding effects and dependencies among features. Recent progress in multimodal learning leverages the potential of extensive multimodal data representations. CLIP [45], ALIGN [20], and MaMMUT [22] are multimodal foundations devised for images, text, audio, and video. Expanding this scope, ImageBind [13] and Gato [46] tackle diverse tasks across multiple modalities. Dealing with large-scale multimodality data, however, may introduce new obstacles such as modality heterogeneity, modality imbalance, confounding, and intermodal variabilities. These emphasize the necessity of developing normalization technique that is dedicated to multimodal data.
To tackle these challenges, several methods [30, 55, 65] have recently been proposed that use metadata- the data that provides information about given data- the normalization of the features in neural network. However, these studies predominantly focus on metadata and still encounter issues related to confounding effects. Motivated by these findings, we introduce a novel normalization method for multimodal heterogeneous data, referred to RegBN, aimed at removing confounding effects and dependencies from low- and high-level features before fusing those. Our approach entails leveraging regularization to promote independence among heterogencous data from multisource

---

sensors (Figure 1). RegBN facilitates the training of deep multimodal models while ensuring the prediction of reliable results. Our key contributions are follows:
As normalization module, RegBN can be integrated into multimodal models of any architecture such as multilayer perceptrons (MLPs), convolutional neural networks (CNNs), vision transformers (ViTs), and other architectures.
RegBN possesses the capability to be applied to vast array of heterogeneous data types, encompassing text, audio, image, video, depth, tabular, and 3D MRI.
RegBN undergoes comprehensive evaluation on a collection of eight datasets, including multimedia, affective computing, healthcare diagnosis, and robotics.
The findings indicate that RegBN consistently leads to substantial improvements in inference accuracy and training convergence regardless of the data type method used.
Related work
Normalization methods in deep learning: BN [18], LN [4], GN [62], and IN [62] are conventional normalization methods that normalize the input features by re-centering and re-scaling. Unlike such methods that process input layers separately, RegBN takes two input layers and produces corresponding mutually independent output layers. The input layers could be any n--iimensional low-/high-level features, metadata, or raw data.
Confounding effect removal: Several statistical techniques have been developed for regressing out confounders [42, 33,38.. 38]. Such methods are basically developed for confounding effect removal in healthcare. recent years, several studies used deep neural networks for coping with confounders [65, 30, 55]. Lu et al. [30] introduced novel layer normalization module called 'metadata normalization' (MDN), which is designed specifically for neural networks that incorporate metadata. A closed-form solution to linear regression was developed by the authors to capture the relationship between feature layers and metadata. MDN, however, exhibits certain limitations such as its weak performance on small mini-batches and the requirement of the entire metadata for computing its matrix inverse. A penalty-basee approach called PMDN [55] was proposed as an improvement over MDN. PMDN addresses the shortcomings of MDN by using learnable parameters instead of estimating MDN's matrix inverse. PMDN is designed for metadata with a small number of features due to the huge number of required learning parameters. PMDN also requires time-consuming two-step training procedure to optimize its learnable parameters. In addition, a major limitation of PMDN is how to effectively train the learnable parameters, as these parameters are trained solely based on the model's loss function, and models are typically unaware of the presence of confounding effects.
We leverage the great potential of regularization Our proposed normalization technique does not rely on learnable parameters and is capable of operating efficiently small results and is applicable to various types of multimodal neural networks, not limited to those incorporating metadata, and its ability operate at both low- and high-feature levels. RegBN is the first normalization method to address the dependency and confounding issues in multimodal data, making it unique and promising contribution the field.
solution to address the aforementioned issues.
Most importantly, our approach produces more accurate
3 RegBN: A regularization method for normalization of multimodal data
Given trainable multimodal neural network (e.g., MLPs, CNNs, ViTs) with multimodality backbones A and B. Let (1) E Raxn represent the 1-th layer of the multimodal network for modality A with batch size b and n2 X .... x Nn features that are flattened into a vector size n. In similar vein, we define (0)) RRxm as the k-th layer of the multimodal network for modality B with mi x... ... X mM features that are flattened into a vector of size m. Layers and k can be positioned directly prior to the fusion step. Depending on the fusion approach employed, f(() and (k)) can contain low-level features, high-level features, or latent representations of the multimodal model for early, middle, or late fusion, respectively (see Appendix ??). Features f() and (*) pertain to two distinct modalities and are subject to immeasurable unknown confounders. Moreover, they also share partial common information. The goal is to find potential similarities (mainly caused by confounding factors and dependencies during data collection) between these layers and then remove those (Figure l).
layers can be then fused in the multimodal network for
3

---

08
F(W.2.)
02
57x10
Maxto
-04
19xt0
-08
A
(a) Extrema mini- and seeded values {)1, A2, A3) for finding sub-optimal X+
(b) 2D visualization of multimodal LeNet model with RegBN on AV-MNIST [31]
there is a risk of falling into local minima; we fix this problem by seeding A_ different locations, detailed in Section 3.2. (b) 2D visualization of the loss surface of the multimodal SMIL model [31] on AV-MNIST (see Section 4.1). The numbers inside the plot indicate the classification loss, while the color bar reports the loss of the projection weights in RegBN, i.e. AW, RegBN's weights converge into their global minimum as long as the multimodal model reaches its global minimum.
Figure 2: (a) In a
given task. To this end, one can represent f(()) based on (kk) as linear regression model
(1) Here, W((.A) is a projection matrix of size n x m and f represents the difference between f() and its corresponding map in the domain of (*)), also known as the residual. Ideally, the residual map does not contain any features of (h,,,, so f( and (( are mutually independent. RegBN minimizes the linear relationship between these layers via
f() = wilkk) (k)) + f!")
(1,A,)) arg min W(l.)
| W((N) (()
W.F.
|
=
The constraint in terms of the Frobenius norm, || Σn プ Wi
(1,k) 1, guarantees
that the network does not experience vanishing and exploding gradients during training and inference. Eq. is regularization method for ill-conditioned problems [37, 9, 34, 47]. Lagrangian multiplier offers an equivalent formulation F to the optimization defined at Eq. 2:
F(W(L.) X+) 1 w((,), g
(/TWRR) -1)
+ +
In this equation, A is positive Lagrangian multiplier, playing the role of dual variable. Minimizing F(W(L,) X+) over W(lk) yields the projection matrix
(
(M) (()) + +X-1) g (M) (())
where, 24 is the estimated Lagrangian multiplier obtained through minimization of the term [W 1: I an identity matrix and superseript T represents the transpose operator. We em- ploy singular value decomposition (SVD) and limited-memory Sroyden--Feetche--Goolloo F Goldfarb-Shanno (L-BFGS) [28] for solving Eq. The solution is detailed in Appendix ??. and the proposed method summarized in Algorithm 1. It is worth noting that the projection weights are not directly trained by the multimodal criterion since there a danger of being fooled by confounders. Instead, projection weights are learned through Eqs. 2-4 and updated recursively over chunks of training data, detailed below.
3.1 Update of the projection matrix
The projection matrix, W(L)), calculated for every mini-batch so we recursively update that via the exponential moving average's approach [21]. which decays the mean gradient and variance for each

---

Algorithm 1 Pseudocode of RegBN
Inputs: f() R and (()) E R 1-th and k-th layer of an MMI., respectively
2: Other inputs: (timestep/mini-batch)) projection weights W(l,s), A (a collection of estimated
X. values until previous timestep), training;
3: Compute SVD of g ( (VD(g UEV* = -E O.4io
4: Set A, through Eq. 8 and given A,-
5: for À in A, do
6: Initialize X with À and then apply the L-BFGS algorithm to approximate Eq.
7: Store the estimated X and the approximation error;
8: end for
9: Choose A_ with the lowest approximation error as the sub-optimal X4;
10: Update Aq: Ae +- At-1 U A;;
11: Insert Å to Eq. 4 for computation of the projection matrix: W -E
Cives ((0)
Usi
o7+A,
12: if training then
Normalize the input feature layer using V7,(,,); f:0 t- f() W((1AA)
13:
end if
15: Update the projection weights W!llkk) via Eqs. 5 and 6;
16: if not training then
Normalize the input feature layer using W (((,)); f( t- f() w
17:
end if
19: return ,1 w.(lk)) and Ae
variable exponentially. Let W((.)) t-1 denote the updated projection matrix until the previous timestep. We define AW, the mean absolute error between the currently predicted projection matrix and the The projection matrix is then
previously updated projection matrix: AW, = [V updated at timestep t via
W!l
(1
Me Vv:
m.
W(lA)
+ 7t
+
Vu +
Here, is the learning rate of the multimodal model at timestep t; m. and is are the first and second moments, respectively, derived from
8 -3) 1t
B21--1 (1 32)
my
1
1 1 35
where B1, f2 E (0, 1) represent constant exponential decay rates. As mentioned in Algorithm the
projection matrix is updated during training only (Algorithm I: Steps 11-13), and we use the latest updated projection weights for validation inference (Algorithm 1: Steps 15-17).
3.2 Avoiding falling into local minima
Multimodal networks often contain several backbones (e.g., MLPs, CNNs, ViTs), depending on the number of multiple heterogeneous data sources. Identifying the global minima of such networks is challenging problem, as the networks may fall into local minima. On the other hand, optimization of Eq. 3 relies on A+, which may introduce several local minima. Conventional regression methods treat X as constanthyperparamneer while predetermined hyperparameters usually increase the risk of falling into local minima. To prevent this, we adopt a mini-batch-wist approach for the estimation of A4. The objective is to estimate suboptimal X4 with the L-BFGS optimization algorithm in way that meets
OF(W((A)))) = arg min X
2 = arg min A
(TT ( + +A,1) (K) 1 foo (7)
-1)
ax_
Quasi-Newtor methods like L-BFGS rely on the initial value of X4 for minimization of the opti- mization problems like Fq. 7 (see Figure 2a). Due to the limited number of local minima that exist per mini-batch we adopt method whereby initialize x7 with multiple random values within
5

---

Table 1: The experimental section covers a diverse range of research areas, dataset sizes, input modalities (in the form of i: image, I: language, v: video, a: audio, s: 3D snippet-level features, m: 3D MRI, d: depth, I: tabular, f: force sensor, p: proprioception sensor), and prediction tasks.
Area
Dataset
Modalities
Samples (#)
Prediction task
Training
Test
Affective computing
CMU-MOSEI CMU-MOSI IEMOCAP
{
18,118
4,659
sentiment
Affective computing
{v,
1,513
686
sentiment
Affective computing
{v,a.{}
3,515
938
emotion
Multimedia
MM-IMDb
{i,1}
18,160
7,799
movie genre
Multimedia
LLP
{a
10,649
1,200
video parsing
Multimedia
AV-MNIST (small)
{i,a}
1,045
450
classification
Multimedia
AV-MNIST
{i.a)
60,000
10,000
classification
Healthcare diagnosis
ADNI
{m,t)
1,073
268
dementia diagnosis
Robotics
Vision& Touch
(.d,
40,546
22,800
contact
Synthetic dataset
(i,4)
10,000
1,000
classification
predetermined range A, Dep...................... Apel as well as with the median of the suboptimal X values predicted in the previous mini-batches, pu (Ac-1):
A. Ap Wu (Ac-1) Pm....., Apel U Ung 1
(8)
By creating a loop over A,, we first initialize X and then estimate its true value using the L-BFGS method. We also store the approximation error and after the termination of the loop, the suboptimal value of X for given mini-batch is determined by selecting the estimated value that yields the lowest approximation error. After calculating the suboptimal lagrangian multiplier, we update A, ie.. A₁ + A U År As illustrated Figure 2b, when the multimodal model approaches its global minimum, the parameters of RegBN also converge.
Experiments
Heterogencous data captured by diverse multisource sensors utilized to verify the usefulness and effectiveness RegBN in multiple data contexts such as language, audio, 2D image, video, depth, 3D MRI, and tabular data. RegBN is applied eight datasets that are summarized in Table 1. Details on the datasets and baseline methods are provided Appendices ?? & ??. The default parameters and settings for RegBN reported in Appendix ??. Our code is openly available at
github. io/RegBN. Experimental details are provided in Appendix ??. Here, we summarize the main results. The findings of this investigation provide insights into the performance of RegBN across various data modalities and highlight its potential as robust normalization technique. PMDN relies considerable number of learnable parameters specifically tailored for metadata with limited dimensions. MDN necessitates substantial RAM resources, particularly when estimating its inverse matrix large-scale datasets.
4.1 Multimedia
Experiments with the LLP dataset [53]: Audio-Visii Video Parsing (AVVP) [53] (see Ap- pendix ??) employed as baseline for parsing individual audio, visual, and audio-visual events under both segment-level and event-level metrics over the LLP dataset. The normalization module is employed to decouple the audio features from the video features, ensuring their independence. Table reports that RegBN improves the baseline performance in nine out of ten metrics. RegBN brings about improvements in all audio-visual video parsing subtasks, measured both at the segment-level and event-level metrics. These findings suggest that decoupling audio and video allows AVVP to produce more accurate predictions of event categories on a per-snippet basis.
Experiments with the MM-IMDb dataset [3]: The MM-IMDb dataset was curated for the purpose of predicting movie genres through the use of either image or text modality. This task involves
Appendix ?? reports the validation results of AVVP with and without RegBN the LLP dataset.

---

Table 2: Audio-visual video parsing accuracy (%) of AVVP [53], as baseline (BL), on the LLP dataset [53] for different normalization techniques. A and V stand for audio and visual. respectively.
Method
Segment- Level
Level
Af
V1
A-V1
Typet
Event
Af
Vt
A-V1
Typet
Event
BL
60.1
52.9
48.9
54.0
55.4
51.3
48.9
43.0
47.7
48.0
BL+PMDN
59.7
53.0
48.8
53.6
55.3
51.4
49.1
42.9
47.7
48.2
BL+RegBN
60.2
53.3
48.9
54.0
55.2
52.0
49.5
43.1
47.9
49.3
Table 3: Multi-label classification scores (FI score) of baseline SMIL [3]] (denoted by BL)
with/without normalization on the MM-IMDb dataset [3]
Method
Norm. Params.
Fl Score (%)
Samplest
Microf
Weightedf
BL BL+PMDN
49.62
51.18
48.95
65,536
49.85
51.44
49.36
BL+RegBN 0
54.82
55.37
52.83
multi-label classification since a single movie may be associated with multiple genres (see Ap- pendix ??). SMIL [31], as baseline approach, employs the pre-trained BERT extract the textual features, while image features are extracted using the pre-trained VGG-19. The text features are subsequently normalized with respect to the visual features using the proposed RegBN before the fusion process occurs. The fusion operation involves a concatenation layer. The fused features are passed through two fully-connected layers to obtain the classification labels. The details are provided in Appendix ??. Table 3 presents the quantitative results obtained from the experiments. The table indicates that applying a normalization layer such PMDN or RegBN can lead an improvement in the performance over the baseline model. This highlights the importance of the normalization step in multimedia data. In particular, RegBN was found to enhance the F1 score of the baseline by an average of 8.86% across all three metrics.
Experiments with the AV-MNIST dataset [31]: The small AV-MNIST dataset comprises two modal- ities, namely, audio and image. The audio modality is obtained from Free Spoken DigitsDataset? which includes 1,500 raw audio recordings from three different speakers. Despite the independence of the image and audio modalities, the primary objective of this experiment is to investigate whether the confounding effects induced by the speakers have impact on the classification results. In line with the previous experiment, SMIL [31] serves as the baseline method. The obtained classification results for the combinations baseline+MDN, baseline+PMDN, and baseline+RegBN are 98.43% 98. 18.68% 38,, and 99.1 1%, respectively. Figure 3 illustrates t-SNE plots for the different approaches. In this experiment, iSNE is applied to the features extracted from a fully-connected layer after the concatenation of normalized image features with audio features. The figure demonstrates that the classes are more separable for RegBN as compared to MDN or PMDN, in particular, class '2'. This result implies that normalization method effectively removes the confounding effects caused by different speakers, thereby providing more flcxibility to the baseline method for accurate classification. The results on AV-MNIST are reported in Appendix ??.
4.2 Affective computing
In this section, we explore the potential application of RegBN multimodal emotion sentimen analysis using the IEMOCAP [8], CMU-MOSI, and CMU-MOSEI [63] datasets, which are divided into aligned and non-aligned multimodal time-series (see Appendices ??, ??, & ??). The baseline method (BL) for this task is the Multimodal Transformer (MulT) [54], which is ViT developed for analyzing multimodal language sequences. MulT fuses video (v), audio (a), and language (I) at three levels (see Appendix ??): 1) language fusion (v - [ & + (), 2) audio fusion (v
& + a), and 3) video fusion (a > v & + v). The outputs of one or more levels are then aggregated for
2 Free Spoken Digit
Dataset is available at
https://githe com/ haan//akobossk//
free- spoeen-ppon--iigge digi--dattttt-t-eee
7

---

(a) Only image (b) Only spectrogram
(c) MDN
(d) PMDN
(e) RegBN
0
8
Figure 3: iSNE visualization of the features extracted from a-b) unimodal image and an unimodal audio, and e-e) the multimodal model with different normalization methods. Each data point represents a sample.
Table 4: Results of multimodal emotion analysis on IEMOCAP [8] with word aligned multimodal sequences. The baseline (BL) Multimodal Transformer (MulT) [54].
Method
Loss
Happy (%)
Sad (%)
Angry (%)
Natural (%)
Train
Test
Acct
FIf
Acct
Flf
Acct
FIf
Acct
F11
BL
0.106
0,536
86.3
84.0
81.5
80.6
86.5
86.4
69.5
69.1
BL+RegBN
0.009
0.452
87.4
83.0
84.3
84.1
88.2
88.1
73.4
73.2
classification. Here, we report the results of all three fusion levels combined, while the detailed results for each level are provided in Appendix ??. The quantitative results of the emotion analysis experiment are reported in Table 4. The table shows that RegBN improves the classification performance of MulT in most cases, particularly for the "natural" class of the word-aligned experiment. Table 5 presents the results of the multimodal sentiment analysis on the CMU-MOSI and CMU-MOSEI datasets. In both the word-aligned and unaligned experiments, decoupling the multimodal features with RegBN improves training and inference performance.
4.3 Healthcare diagnosis
The ADNI dataset [19] includes 3D MRI scans of patients along with rich clinical information orga- nized in low-dimensional tabular format (see Appendix ??). The dataset is subject to confounding effects such as age, sex, and level of education, which must be considered to prevent biased evaluation results, as noted in studies [59, 39]. Similar to [60]. we partitioned the data into five separate and non-overlapping folds, ensuring that each fold had a balanced distribution of diagnosis, age, and sex. In this experiment, we utilized a 3D ResNet [60] for extracting MRI features, while tabular features were extracted using an MLP network proposed in [14] (see Appendix ??). These features were then concatenated and fed into another MLP network for diagnosis classification. We applied different normalization techniques to the unimodal MRI and tabular features prior to concatenation. Table 6 reports the mean and standard deviation obtained by averaging the performance the multimodal over these five folds. The complete results can be found in Appendix ??. RegBN emerged as the most effective technique, outperforming other normalization methods terms of both ACC and BA metrics. Moreover, RegBN's ability to relax the interdependencies between the different modalities facilitated lower training loss in the multimodal network.
4.4 Robotics
The Vision&Toucch dataset [23] encompasses recordings of simulated and real robotic arms that are equipped with visual (RGB and depth), force, and proprioception sensors (see Appendix ??). We employed 'Making Sense of Vision and Touch' (MSVT) developed by Lee et al. [23] as a baseline (refer to Appendix ??). For this experiment, we apply normalization techniques to decouple the visual RGB and force from depth information. Table summarizes the results, while detailed results can be found in Appendix ??. The flow loss values in these tables indicate that MSVT converges more effectively when used in conjunction with RegBN, This suggests that RegBN is able to effectively remove dependencies between the RGB and depth pair, as well as the force and depth pair.
8

---

Table 5: Multimodal sentiment analysis results multimodal on CMU-MOSHI & MOSI [63] with word aligned multimodal sequences. The baseline (BL) method is MulT [54].
Method
Dataset
Loss
Sentiment (%)
Training
Test
Acc2t
Accs
Accr
FI1
Corr
BL
CMU-MOSEI
0.452
0.636
80.3
51.9
50.3
80.1
67.1
Beg
CMU-MOSEI
0.438
0.611
81.1
52.2
50.5
81.2
66.6
BL
CMU-MOSI
0.403
0.632
81.4
42.5
37.5
82.0
69.7
BL+RegBN
CMU-MOSI
0.267
0.546
81.8
42.3
38.6
82.3
69.1
Table 6: Training's cross-entropy (CE) loss along test's accuracy (ACC) and balanced accuracy (BA)
results on the ADNI dataset [19]. Baseline is a combination techniques developed in [14, 60].
Method
Norm. Params.
Training CE
Test
(#)
loss
ACC (%) f
BA (%)}
BL+BN
512
0.64110.01
48.8+4.4
48.324.5
BL+MDN
0
0.619#0.03
50.4+4.8
50.1
BL+PMDN
4,096
0.632+0.03
49.7#6.2
49.6+7.5
BL+RegBN
0
0.596+0.01
53.0+3.1
52.343.7
Table Results of MSVT [23] incorporating various normalization methods on Vision&Touch [23]
Method
Norm. Params.
Training
Test Accuracy
Flow loss
Total loss.
(%)1
BL BL+PMDN
0.212
0.563
86.2
131,072
0.194
0.454
87.9
BL+RegBN
0
0.052
0.231
91.5
Table 8: Classification accuracy results on the synthetic dataset in the presence of confounder. Due the randomness of the experiment, we reported the mean and std of results over 100 runs.
Normalization
Norm. params. (#)
Experiment I
Experiment II
Method
MLP
CNN
MLP
CNN
MI.P
CN
reference
87.5
87.5
75.0
75.0
BN
256
112
96.4+0.4
94.9+0.2
83.1
85.240.3
MDN
0
91.9+0.7
89.7+0.7
79.1+0.6
82.2+0.7
PMDN
2048
201.728
93,4+0.6
92.7+0.5
80.8+0.7
84.3+0.8
RegBN
0
87.3+0.9
88.240.8
76.2+0.8
76.8+0.9

---

4.5 Synthetic dataset
Drawing from the synthetic dataset study [30], we designed synthetic experiment for binary classification task. Images with size of x4 are split into four regions with identical Gaussians in their background. The regions along the main diagonal are linked the class label, while the off-diagonal ones act as confounders that do not impact the true label. We created metadata of size 16 for this dataset that includes the true binary label, the confounder value, one fake label, 12 randomly generated fake features, and column with value of one. The binary classes have overlapping, so accuracy drops based on the amount of overlapping (see Appendix ??). In Experiment the theoretical maximum accuracy of classification is 87.5%, while for Experiment II, it is 75%. The base networks employed here are a simple CNN and an MLP-based neural network. The dataset and base networks are detailed in Appendices ?? and ??, respectively. The classification results are presented in Table 8, with more detailed results available in Appendix ??. The results reveal that conventional normalization techniques, such BN, GN, and IN, are unable remove the confounding effects. The performance of the base networks that employed MDN was superior compared to those with conventional normalization methods. However, MDD still exhibits shortcomings in dealing with confounding effects. Despite the incorporation a high number of learnable parameters, PMDN was unsuccessful in eliminating confounding effects, the inclusion of fake values in the metadata led to the deception of the multimodal model. RegBN demonstrated its ability to remove confounder effects by yielding performance closest to the theoretical maximum in both experiments.
4.6 Computational cost and ablation study
We employed an NVIDIA GTX 1080 Ti with 12GB VRAM for image experiments and an NVIDIA A100 with 80GB VRAM for video experiments. Since RegBN learns the projection matrix through Eqs. 4-7, it eliminates the need for learnable parameters, resulting in enhanced computational efficiency for multimodal models. During the training process, RegBN achieves frame rate of approximately 124 fps when normalizing pair of layers with dimensions 256 x 1024. In comparison, conventional normalization techniques achieve frame rate of around 13,000 fps for the normalization of one layer of dimensions 256 x 1024. Regarding inference time, RegBN exclusively relies on the updated projection matrix and therefore does require additional computation. RegBN is frequently used before the fusion blocks in multimodal models, so the training time of a few RegbBN layers is negligible. Results for the ablation study are provided in Appendix ??.
5 Conclusion and future work
This study presented novel normalization method designed for dependency and confounding removal in multimodal data and models. The experimental results demonstrated the effectiveness of RegBN in significantly enhancing the accuracy and convergence of multimodal models, rendering promising normalization method for multimodal heterogeneous data. The current version of RegBN employs L-BFGS on a single GPU, which will be improved in the future. Our aspiration is that this work effectively leverages new opportunities in exploring and harnessing multi-modality data within multimodal analysis.
Acknowledgements
This work was supported by the Munich Center for Machine Learning (MCML) and the German Re- search Foundation (DFG). The authors gratefully acknowledge the computational and data resources provided by the Leibniz Supercomputing Centre. The authors thank Emre Kavak for his valuable input the synthetic dataset design.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206-24221, 2021.
[2] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[3] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multi- modal units for information fusion. In 5th International conference on learning representations workshop. 2017.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv 167.0000770 2016.
[5] Anas Belouali, Samir Gupta, Vaibhav Sourirajan, Jiawei Yu, Nathaniel Allen, Adil Alaoui, Mary Ann Dutton, and Matthew J Reinhard. Acoustic and language analysis of speech for suicidal ideation among veterans. BioData mining, 14(1):1-17, 2021.
[6] Ioana Bica and Mihaela van der Schaar. Transfer learning on heterogeneous feature spaces for treatment effects estimation. arXiv preprint arXiv:2210.06188: 2022.
[7] Brandon M Booth, Louis Hickman, Shree Krishna Subburaj, Louis Tay, Sang Eun Woo, and Sidney K D'Mello. Bias and fairness in multimodal machine learning: A case study of automated video interviews. In Proceedings of the 202/ International Conference Multimodal Interaction, pages 268-277 2021.
[8] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
[9] D Calvetti, S Morigi, L Reichel, and F Sgallari. Tikhonov regularization and the l-curve for large discrete ill-posed problems. Journal of computational and applied mathematics, 123(1-2): 423 446, 2000.
[10] Joseph DelPreto, Chao Liu, Yiyue Luo, Michael Foshey, Yunzhu Li, Antonio Torralba, Wojeiech Matusik, and Daniela Rus. Actionsense: multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment. Advunces in Neural Information Processing Systems, 35: 13800-13813, 2022.
[11] Aymeric Dieuleveut, Gersende Fort, Eric Moulines, and Geneviève Robin. Federated- em with heterogeneity mitigation and variance reduction. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 29553-29566. Curran Associates, Inc., 2021. URL https: //wpc//einccceeee..eoooeaeo meurips.cc/paper. cctppccpppaffaeeeeeeeeee cnmeer_filespppeeeee (ttppprrfiessspppe//2221111. ///////////2///llee// f740c8d9c193f 115.1311177777711111153333 P pdf.
[12] Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. Multimodal and multilingual embeddings for large-scale speech mining. Advances in Neural Information Processing Systems, 34:15748-15761, 2021.
[13] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[14] Paul Hager, Martin Menten, and Daniel Rueckert. Best of both worlds: Multimodal contrastive learning with tabular and imaging data. accepted for CVPR, 2023.
11

---

[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[17] Yan Huang, Yuming Wang, Yunan Zeng, and Liang Wang. Mack: multimodal aligned conceptual knowledge for unpaired image-text matching. Advances in Neural Information Processing Systems, 35:7892-7904, 2022.
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. pmlr, 2015.
[19] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski. Paula J Britson, Jennifer L. Whitwell, Chadwick Ward, et al. The alzheimer's 's disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Reso- nance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, (7(4):685-6) 691, 2008.
[20] Chao Jia, Yinfei Yang. Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916.
PMLR, 2021.
[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[22] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple architecture for joint learning for multimodal tasks. arXiv preprint arXiv:2303. /6839, 2023.
[23] Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference Robotics and Automation (ICRA), pages 8943-8950. IEFF, 2019.
[24] Michelle A Lee, Brent Yi, Roberto Martin-Martin, Silvio Savarese, and Jeannette Bohg. Multi- modal sensor fusion with differentiable filters. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2020.
[25] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslic Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning, arXiv preprint MxXi::2177007000 2021.
[26] Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis- Philippe Morency, and Ruslan Salakhutdinov. Multiviz: Towards visualizing and understanding multimodal models. In The Eleventh International Conference on Learning Representations, 2023.
[27] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
(28) Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 55(155)5503-528 1989.
[29] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task- agnostic visiolinguistic representations for vision-and sidon-and-langeee lan-uuge tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bue, E. Fox, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpss:/prooeediigg/ wtwp:eeeedddnnnnnnnninneeee wtteripsccc//eeeoeeeeeeeooooeeee wwtpcc/ppppeeaeeeeeeeeee http:/pper..flleeppee/////// /ttee///ppperr00///iieee 37467700eee2237744444ddbdaallaa2 Paper .df.
12

---

[30] Mandy Lu, Qingyu Zhao, Jiequan Zhang, Kilian M Pohl, Li Fei-Fei, Juan Carlos Niebles, and Ehsan Adeli. Metadata normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10917-10927, 2021.
[31] Mengmeng Ma. Jian Ren, Long Zhao. Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2302-2310, 2021.
[32] Daniel Martin, Sandra Malpica, Diego Gutierrez, Belen Masia, and Ana Serrano. Multimodality in vr: A survey, ACM Computing Surveys (CSUR), 54(105):1-36, 2022.
[33] Shammi More, Simon B Eickhoff, Julian Caspers, and Kaustubh R Patil. Confound removal and normalization in practice: A neuroimaging based sex prediction case study, In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part V, pages 3-18. Springer, 2021.
[34] Henrique Gomes Moura, Edson Costa Junior, Arcanjo Lenzi, and Vinieius Carvalho Rispoli. On a stochastic regularization technique for ill-conditioned linear systems. Open Engineering, 9(1):52 60, 2019.
[35] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances Neural Information Processing Systems, 34:10000-141113 2021.
[36] Marla Narazani, Ignacio Sarasua, Sebastian Pölsterl, Aldana Lizarraga, Igor Yakushev, and Christian Wachinger. Is pet all you need? multi-modal study for alzheimer's s disease using 3d cnns. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 66-76. Springer, 2022.
[37] Arnold Neumaier. Solving ill-conditioned and singular linear systems: A tutorial on regulariza- tion. SIAM review, 40(3):636-666, 1998.
[38] Sebastian Pölsterl and Christian Wachinger. Estimation of causal effects in the presence of unobserved confounding in the alzheimer's 's continuum. In Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28-June 30, 2021, Proceedingss 27, pages 45-57. Springer, 2021.
[39] Sebastian Pölsterl and Christian Wachinger. Identification of causal effects of neuroanatomy on cognitive decline requires modeling unobserved confounders. Alzheimer's & Dementia, 2022.
[40] Sebastian Pölsterl, Tom Nuno Wolf, and Christian Wachinger. Combining 3d image and tabular data via the dynamic affine feature map transform. In Medical Image Computing and Computer Assisted Ceervention-MICC.. 2021: 24th International Conference, Strasbourg, France, September 27-October I, 2021, Proceedings, Part V 24, pages 688-698. Springer, 2021.
[41] Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. A review of affective computing: From unimodal analysis to multimodal fusion. Information fusion, 37:98 125, 2017.
[42] Mohamad Amin Pourhoseingholi, Ahmad Reza Baghestani, and Mohsen Vahedi. How to control confounding effects by statistical analysis. Gastroenterology and hepatology from bed to bench, 5(2):79, 2012.
[43] Jennifer Preece, Helen Sharp, and Yvonne Rogers. Interaction design: beyond human-computer interaction. John Wiley & Sons, 2015.
[44] Yanyuan Qiao, Yuankai Qi. Yicong Hong, Zheng Yu, Peng Wang, and Qi Wu. Hop+: History- enhanced and order-aware pre-training for vision-and-languaa navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-14, 2023. doi: 10.1 M1111109TTMMM00 2023. 3234243.
13

---

[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 9748-8763. PMLR, 2021.
[46] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Marth, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent, 2022. URL. https://arxiv. org/abs/2205.06 061000000611..5.0 3, 2022.
[47] Dominic Richards, Sahand Negahban, and Patrick Rebeschini. Distributed machine learning with sparse heterogeneous data. Advances in Neural Information Processing Systems, 34: 2021.
[48] Philipp V Rouast, Marc TP Adam, and Raymond Chiong. Deep learning for human affect recognition: Insights and new developments. IEEE Transactions on Affectivv Computing, 12(2):
2019.
[49] Septimiu Salcudean, Hamid Moradi, David G. Black, and Nassir Navab. Robot-assisted medical imaging: A review. Proceedings of the IEEE, 110(7):951-667, 2022. doi: 10. 109/ JPR0C.2002.3162840.
[50] Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3-14, 2017.
[51] Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre-training with multimodal temporal contrastive learning. arXiv preprint arXiv:2210.06031, 2022.
[52] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-20 1126282,, 2016.
[53] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio-visual video parsing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages Springer, 2020.
[54] Yao-Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, page 6558. NIH Public Access, 2019.
[55] Anthony Vento, Qingyu Zhao, Robert Paul, Kilian M Pohl, and Ehsan Adeli. A penalty approach for normalizing feature distributions to build confounder-fres models. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2022: 25th International Conference, Singapore, September 18 22, 2022, Proceedings, Part III, pages 387-397. Springer, 2022.
[56] Gyanendra K Verma. Multimodal affective computing: Affective information representation, modelling, and analysis. 2023.
[57] Christian Wachinger, Anna Rieckmann, and Sebastian Pölsterl. Detect and correct bias in multi-site ncuroimaging datasets. Medical Image Analysis, 67:101879, 2021.
[58] Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in neural information processing systems, 33:4835 4845, 2020.
[59] Junhao Wen, Elina Thibeau-Sutre. Mauricio Diaz-Melo, Jorge Samper-González, Alexandre Routier, Simona Bottani, Didier Dormont, Stanley Durrleman, Ninon Burgos, Olivier Colliot, al. Convolutional neural networks for classification alzheimer's disease: Overview and reproducible evaluation. Medical image analysis, 63: 63:101694, 2020.
14

---

[60] Tom Nuno Wolf, Sebastian Pölsterl, Christian Wachinger, Alzheimer's Disease Neuroimaging Initiative, et al. Daft: A universal module to interweave tabular data and 3d images in cnns. Neurolmage, 260:1 19505, 2022.
[61] Tom Nuno Wolf, Sebastian Pölsterl, and Christian Wachinger. Don't panic: Prototypical additive neural network for interpretable classification of alzheimer's 's disease. In International Conference Information Processing in Medical Imaging, pages 92-94. Springer, 2023.
[62] Yuxin Wu and Kaiming He. Group normalization. In Proceedings the European conference on computer vision (ECCV), pages 3-19, 2018.
[63] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis Louis-Philippe Morency. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 1rX:::1600..0625,, 2016.
[64] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Couis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, 2018.
[65] Qingyu Zhao, Ehsan Adeli, and Kilian M Pohl. Training confounder free deep learning models for medical applications. Nature communications, 11(1):6010, 2020.
15


Reviews:
[{"review_id": "uX224aDQ0q", "reviewer": "Reviewer_Fqi7", "summary": "The paper presents a novel regularization approach tailored for multimodal/heterogenous data fusion, which can be applied off-the-shelf to modern neural network architectures. The approach, namely RegBN, targets the reduction/removal of confounding factors and partial dependencies that are usually present among features extracted from different modalities. To do so, RegBN is implemented as a regularization technique that promotes feature independence across data modalities. The regularization is iteratively optimized over mini-batches using a separate solver not relying on back propagation and gradient descent. This separation is critical as otherwise the network could simply learn to use the confounding factors instead of removing them. Experiments are performed on several multimodal datasets based on different modalities and spanning across different applications, such as multimedia, affective computing, healthcare diagnosis, and robotics. Moreover, a synthetic dataset is designed to further highlight the impact of RegBN. The results showcase a positive impact in all the test cases.   ", "questions": "1. Do the Authors envision any use/application of RegBN for large foundation models, e.g., is it meant for whenever they are employed for downstream tasks?\n2. What could be the reason why RegBN affects negatively performance? Is it due to the fact that when evaluating some dataset in-distribution learning spurious correlation might provide a boost?  Please comment also on the last two sentences of weakness # 2.\n3. Is it enough to apply RegBN once as shown in I.1a,b,d or might be worth to apply it to several layers?\n4. Consider slightly modifying sec. 3 and Figure 1, as well as adding somewhere (introduction or related work) a few positioning sentences wrt foundation models. Fix other typos/mistakes.", "limitations": "Societal impact is not discussed. I believe that RegBN could provide a benefit in reducing stereotypical biases that currently affect multimodal models. ", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The approach is novel and grounded with theoretical motivation\n2. The method can be plugged off-the-shelf to most multimodal architectures, possibly leading to vaste adoption\n3. Wide breadth of experiments that explore several multimodal scenarios\n4. Positive results in all scenarios", "weaknesses": "1. No integration with recent widely adopted multimodal foundation models, such as CLIP, ImageBind, and many others. Despite I understand that RegBN purpose contrasts with the multimodal alignment objective driving those models, I believe it would be valuable to bring them to the discussion. It would also be beneficial for the positioning of the paper.\n2. The boost is sometimes minor, e.g see Table 2, 5, I.3, and in some settings even slightly negative, e.g., see Table 5 and I.4 (video fusion rows). Also, it is unclear the high difference between AV-MNIST results reported in the main paper (~ 99% accuracy) and the ones reported in table I.3 (~71% accuracy). Furthermore, why for the experiments on LLP, training metrics are reported in Table 2 and validation metrics in the supplementary? \n3. As RegBN requires a separate optimization, its integration affects the time complexity of the model training. The Authors provides multiple examples on how RegBN could/should be integrated in existing architectures, see Figure I.1, however, in most cases a single RegBN layer is added to the network. It is unclear whether this choice is suboptimal for performance in favor efficiency, or whether it is useless to have multiple RegBN as in I.1c.\n4. Minor:\n    1. The writing is for sure enjoyable, however there are some parts that could have better explained / could be less ambiguous. For instance, in section 3: _\"Ideally, the residual layer ... RegBN minimizes the linear relationship between these layers via: equation (2)\"_, but then equation (2) does only contain $f$ and $g$, and thus it is completely tacit the key insight on why solving equation(2) would provide $f_r$ orthogonal to $g$, as well as it is not explicit that $f_r$ corresponds to the normalization of $f$. Moreover, Figure 1 provides a general idea of the impact of RegBN, but it could have been more aligned with the formulation, e.g., by adding $f$, $f_r$, and $g$; and in general the caption could be expanded to better describe it — what does the grey band represent?. Finally the positioning could have been more clear by stating why recent multimodal foundation models like CLIP are out of the scope.\n    2. “multimodAl” often mistyped as “multimodEl”; Table I.5, 3rd column, video fusion, (70.6) vs (81.4) seems a bit suspicious; Color gradient is not explained in Figure I.9; wrong x axis label in Figure I.10;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation to the reviewer for providing valuable comments and suggestions. We have provided a point-by-point response, as outlined below.\n\n*C#1 & Q#1: \nC) No integration with recent widely adopted multimodal foundation models,...\nQ) Do the Authors envision any use/application of RegBN for large foundation models...?*\n\n**Authors' Response**: We highly value the reviewer's insightful suggestion. RegBN is capable of being employed within the architecture of any multimodal technique. In a preliminary experiment involving CLIP, we performed fine-tuning on a pre-trained CLIP model trained on MS-COCO captions [1*]. The results on CIFAR10 are as follows: \nTop-1 Accuracy: CLIP without RegBN: 15.07%, CLIP with RegBN: 16.02%. \nTop-5 Accuracy: CLIP without RegBN: 66.8%, CLIP with RegBN: 67.05%. \nFor larger-scale datasets like MS-COCO, layer fusion seems promising to unlock RegBN's full potential (compared to late fusion), and we will prioritize exploring this avenue as you kindly recommended. In accordance with both this comment and the one provided in Question #3, we are motivated to expand the content of the **fourth** paragraph in the Introduction section.\n\n*C#2 and Q#2: \nC) The boost is sometimes minor, ...?*\nQ) What could be the reason why RegBN affects negatively performance?...*\n\n**Authors' Response**: We extend our gratitude for the detailed points. To ensure providing detailed responses, we have categorized and re-ordered the comments into three parts and have addressed each segment as outlined below: \nPart i) AV-MNIST vs. Small AV-MNIST: For enhanced visual representation, we opted to employ small AV-MNIST, consisting of 1,500 raw audio recordings from three speakers and 1,500 MNIST images (manuscript: lines 195-199). Table I.3 presents the outcomes for AV-MNIST. We acknowledge that AV-MNIST is indeed more challenging (compared to small AV-MNIST) due to a preprocessing step that is elaborated in Supplementary-Section C.3. This involves a 75% reduction of visual modality energy via PCA, which results in a noticeable drop in classification performance on the solo visual modality from around 99% on MNIST to 64.3%.\nPart ii) Minor Boost and Fusion Strategy: We recognize the intrinsic challenge in defining and quantifying confounding factors within real-world data as well as in modality imbalance. Different data acquisition settings introduce varying ranges of confounders, which might intensify due to diverse conditions. RegBN's performance is dependent on data specifications. For instance, compression on AV-MNIST exerts a considerable impact on classification results, while a dataset without compression might yield more accurate outcomes (footnote 1). Moreover, the fusion strategy adopted holds a substantial influence on results. \nPart iii) Experiments on LLP: There must have been confusion about the results in Table 2, which are not from the training set but from the test set. We will clarify it in the manuscript. We reported validation results in Supplementary-Section E.2 to showcase RegBN's impact on training from an accuracy perspective. \n\nFootnote 1: In the case of the original AV-MNIST, we reached out to the authors for the dataset, but regrettably, only the compressed version was made available to us.\n\n\n*C#3 & Q#3:\nComment: As RegBN requires a separate optimization, ... whether it is useless to have multiple RegBN as in I.1c.\nQ:  Is it enough to apply RegBN once ... or might be worth applying it to several layers?*\n\n**Authors' Response**: We are grateful for the insightful comment. From a technical standpoint, RegBN establishes mutual independence between the data of two input modalities. Hence, when applying the RegBN layer successively to the same pair of inputs, the resulting outputs are anticipated to remain independent. Therefore, it is advisable not to utilize RegBN repeatedly over a particular pair of modalities unless there is a shift in the processing context. For instance, modalities X and Y can achieve mutual independence via RegBN, and subsequently, their output can be rendered mutually independent with other modalities as well. As a preliminary experiment, we incorporated RegBN into the MLP network outlined in Section D.6. This integration involved utilizing RegBN K times within the synthetic experiment (Section E.6). For Experiment I, the results for different values of K (K\\in{1,2,...,5}) fall within the range of [87.19, 87.43]. Given the inherent randomness of the synthetic dataset, the observed changes in the results (\\~0.25%) are relatively minor and opting for `K=1' would be the most suitable choice.\nIn the context of layer fusion (Fig. I1.c), where RegBN is employed multiple times, the input feature maps at each instance differ from one another. RegBN can be employed as long as its inputs are not mutually independent. A portion of this part is mentioned as Q4 of the Suppl. Q&A section.\n\n*Minor Cs and Q #3: \nC) The writing is for sure enjoyable, however ...\nQ: Consider slightly modifying sec. 3 and ... Fix other typos/mistakes. *\n\n**Authors' Response**: We acknowledge and apologize for the errors highlighted, all of which will be duly rectified. In line with the reviewer's observations, the accurate value for video fusion in the 3rd column of Table I.5 is indeed 80.6. we have also planned to revise Section 3 regarding the foundation models and modality imbalance. We would like to emphasize our deep appreciation for the reviewer's vigilance in pointing out the inaccuracies and oversights.\n\n*Limitation: Societal impact is not discussed...*\n\n**Authors' Response**: We have planned to enrich the manuscript with a thorough exploration of the societal implications associated with the use of RegBN.\n\nOnce again, we deeply appreciate all the comments and suggestions.  We hope we have addressed every comment satisfactorily. We kindly invite the reviewer to raise any additional questions or points during the forthcoming reviewer-author discussion period."}, {"review_id": "zzra5DjWDI", "reviewer": "Reviewer_eB1v", "summary": "This paper proposes RegBN for the normalization of multimodal data. RegBN uses the Frobenius norm as a regularizer term to address the side effects of confounders and underlying dependencies among different sources of data. The proposed method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. Experiments on eight databases from five research areas demonstrate the effectiveness of the proposed method.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a novel RegBN method for the normalization of multimodal data.\n- Extensive experiments demonstrate the effectiveness of the proposed method.\n", "weaknesses": "- My main concern is that the proposed method lacks comparison with other competitive normalization methods such layer norm and group norm.\n- Using eight databases from five research areas is not necessary. The authors should focus on two or three datasets.\n- Synthetic multimodal dataset should be provided to thoroughly evaluate the effectiveness of the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for the reviewer's comments and feedback on our study. We have taken into account all the comments and kindly request the reviewer to consider our responses, which we have addressed in a meticulous manner.\n\n*Comment #1: My main concern is that the proposed method lacks comparison with other competitive normalization methods such layer norm and group norm.*\n\n**Authors' Response**: Thank you for your insights. Since traditional normalization techniques such as Batch normalization (BN), layer normalization (LN), and group normalization (GN) are not explicitly designed for multimodal data and confounder removal, resulting in unsatisfactory results in the experiments that were omitted from the main manuscript. Our manuscript prioritizes the presentation of techniques demonstrating acceptable performance or contextual relevance. To ensure transparency and comprehensiveness, we have already included the results of such techniques in specific experiments, notably the synthetic experiment (Section E.6) and the Healthcare domain experiment (Section E.4). This allows for a holistic understanding of the methodology's performance across varying scenarios. We have updated Supplementary-Sections E.4 and E.6-Tables I.7 and I.9 to include LN results (please find the tables in the attachment). As anticipated, the LN outcomes in both experiments are comparable to those of traditional normalization techniques such as BN and GN.\n\n*Comment #2: Using eight databases from five research areas is not necessary. The authors should focus on two or three datasets.*\n\n**Authors' Response**: Our motivation to assess and validate the proposed method across diverse modalities was driven by the abundance of multimodal data such as video, 2D/3D image, text, tabular, audio, etc. The wide range of experiments presented in this study has provided us with a robust platform to showcase the versatility of RegBN—a facet that has been warmly embraced by our fellow reviewers. We also opted for publicly available and community-accepted benchmark datasets for reproducibility. To ensure readability, we presented a condensed result overview in the manuscript while reserving the detailed results in the Supplementary file for clarity. Our hope is that RegBN gains widespread adoption as a normalization method in multimodal learning, and we are keen to observe the performance of RegBN on multimodal data from other sensors.\n\n*Comment #3: Synthetic multimodal dataset should be provided to thoroughly evaluate the effectiveness of the proposed method.*\n\n**Authors' Response**: Sections 4.5 & E.6 provide the results for synthetic datasets with a single channel. To address the synthetic multimodal dataset, we conducted an experiment utilizing Prototypical Modality Rebalance (PMR)  [1*], one of the latest developed techniques tailored for mitigating modality imbalance within multimodal learning. Our experiment was conducted on the Colored-and-gray MNIST dataset [2*]. Similar to the MNIST dataset, Colored-and-gray MNIST contains 60,000 training samples and 10,000 test samples. Each sample comprises a gray-scale image (as the first modality) and a monochromatic image (as the second modality). The latter exhibits a strong color correlation with its corresponding digit label. As noted in [1*], the monochromatic images within the training set are robustly color-correlated with their respective digit labels. In contrast, the validation set consists of 10,000 samples, wherein the color correlation of monochromatic images with their labels is weaker. The RegBN in this experiment was applied to raw input images. The  results of the test set are as follows:\n\nPMR:\t [acc: 32.97%,   acc_gray: 19.57%  acc_color: 16.26%]\n\nPMR with RegBN: [acc: 37.04%,   acc_gray: 20.61%  acc_color: 21.44%]\n\nWe have also attached the training loss and validation accuracy. As shown in the attached plots, the performance of PMR in both training loss and validation score is boosted in the presence of RegBN. RegBN provides PMR with an enhanced capability to be trained deeper (lower loss) by rendering independency between the two modalities. The optimization mechanism of RegBN, detailed in Sections 3.1&3.2, does not allow the multimodal learning to fall into its local minima. This increased freedom contributes significantly to an improvement in PRM performance. We have condensed a portion of this discussion into Question 2 of the Supplementary's Q&A section (Section G). \n\nIt is worth noting that the Colored-and-gray MNIST dataset should not be considered a multimodal synthetic dataset. However, it does hold some relevance to Comment \\#3, as the input images encompass images with varying channel numbers.\n\n[1*] Fan et\\~al. PMR: Prototypical Modal Rebalance for Multimodal Learning. CVPR, 2023.\n\n[2*] Kim et\\~al.. Learning not to learn: Training deep neural networks with biased data. CVPR, 2019.\n\nOnce more, we extend our gratitude for your valuable comments and feedback on this study. Should any additional questions or clarifications arise, we enthusiastically encourage you to raise them during the upcoming reviewer-author discussion period scheduled from August 10 to August 16. "}, {"review_id": "8ob7DyEMIu", "reviewer": "Reviewer_saMu", "summary": "The paper introduces a novel approach called RegBN for normalizing multimodal data before fusion in neural networks. The integration of heterogeneous multimodal data poses challenges due to confounding effects and dependencies among different data sources, which can introduce variability and bias. RegBN addresses these issues by using the Frobenius norm as a regularizer term. The method generalizes well across multiple modalities and eliminates the need for learnable parameters, simplifying training and inference. The effectiveness of RegBN is validated on eight databases from various research areas, covering diverse modalities. The proposed method shows broad applicability across different neural network architectures, enabling effective normalization of both low- and high-level features in multimodal models. Overall, RegBN offers a promising solution for improving the performance of multimodal models by addressing normalization challenges.\n\n\n", "questions": "- Typo in the caption of Figure 3. Subfigures are indexed incorrectly. \n- The authors should remember to correct the misstatement in the main paper about SMIL for MM-IMDb. \n\n\n", "limitations": "Authors mentioned that the current algorithm works only on single GPU. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel idea regarding multi-modal fusion. The proposed method is model-agnostic, i.e., the authors provide details about how to apply RegBN within different frameworks, including late fusion, layer (intermediate) fusion, and early fusion. This method does not require significant modification of the original framework and can be plugged in regardless of the encoder's architecture. \n\nExtensive experiments are conducted, including eight datasets covering applications in multimedia, affective computing, healthcare diagnosis, and robotics. On all datasets, the proposed method outperformed the considered baseline: vanilla model, or model with normalization in other manner such as BatchNorm or PMDN. \n\nThis work provides a relatively unique exploration of the topic of multi-modal learning which I believe is worth presenting to the community. ", "weaknesses": "Authors conducted experiments between with/without RegBN and also compared with other normalization methods.\nIt will be interesting to see how does RegBN help regardling of the problem of imbalanced modality utilization, as pointed out in recently papers [1, 2, 3]\n\n[1] Wang, Weiyao et al. “What Makes Training Multi-Modal Classification Networks Hard?” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 12692-12702.\n[2] Huang, Yu et al. “What Makes Multimodal Learning Better than Single (Provably).” Neural Information Processing Systems (2021).\n[3] Wu, Nan et al. “Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks.” ICML 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for the reviewer's comments and insightful suggestions. Below, we have provided a point-by-point response to each comment, encompassing all concerns, for your careful consideration.\n\n*Comment: Authors conducted experiments between with/without RegBN and also compared with other normalization methods. It will be interesting to see how does RegBN help regardling of the problem of imbalanced modality utilization, as pointed out in recently papers [1, 2, 3]. ([1] Wang, Weiyao et al. “What Makes Training Multi-Modal Classification Networks Hard?” CVPR-2020. [2] Huang, Yu et al. “What Makes Multimodal Learning Better than Single (Provably).” NeurIPS-2021. [3] Wu, Nan et al. “Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks.” ICML-2022.)*\n\n**Authors' Response**: We thank the reviewer for the insightful comment. Modality imbalance is a common occurrence in multimodal learning. Fortunately, RegBN can present a valuable solution to address the issues arising from modality imbalance. We conducted an experiment utilizing Prototypical Modality Rebalance (PMR)  [1*], one of the latest developed techniques tailored for mitigating modality imbalance within multimodal learning. Our experiment was conducted on the Colored-and-gray MNIST dataset [2*]. Similar to the MNIST dataset, Colored-and-gray MNIST contains 60,000 training samples and 10,000 test samples. Each sample comprises a gray-scale image (as the first modality) and a monochromatic image (as the second modality). The latter exhibits a strong color correlation with its corresponding digit label. As noted in [1*], the monochromatic images within the training set are robustly color-correlated with their respective digit labels. In contrast, the validation set consists of 10,000 samples, wherein the color correlation of monochromatic images with their labels is weaker. The RegBN in this experiment was applied to raw input images. The  results of the test set are as follows:\n\nPMR:\t [acc: 32.97%,   acc_gray: 19.57%  acc_color: 16.26%]\n\nPMR with RegBN: [acc: 37.04%,   acc_gray: 20.61%  acc_color: 21.44%]\n\nWe have also attached the training loss and validation accuracy as Figure 1. As shown in the attached plots, the performance of PMR in both training loss and validation score is boosted in the presence of RegBN. RegBN provides PMR with an enhanced capability to be trained deeper (lower loss) by rendering independency between the two modalities. The optimization mechanism of RegBN, detailed in Sections 3.1&3.2, does not allow the multimodal learning to fall into its local minima. This increased freedom contributes significantly to an improvement in PRM performance. We have condensed a portion of this discussion into Question 2 of the Supplementary's Q&A section (Section G).\n\n[1*] Fan et\\~al. PMR: Prototypical Modal Rebalance for Multimodal Learning. CVPR, 2023.\n[2*] Kim et\\~al.. Learning not to learn: Training deep neural networks with biased data. CVPR, 2019.\n\n*Questions: 1. Typo in the caption of Figure 3. Subfigures are indexed incorrectly. 2. The authors should remember to correct the misstatement in the main paper about SMIL for MM-IMDb.*\n\n**Authors' Response**:  We highly thank the reviewer for reminding us of the typos. We apologize for the errors and we will correct them in our potential final version. \n\n*Limitation: Authors mentioned that the current algorithm works only on single GPU.*\n\n**Authors' Response**: The initial release of RegBN on GitHub, which will also be referenced in the manuscript's final version, features support for a single GPU. Our ongoing efforts involve the development of a multiple-GPU implementation of L-BFGS, aiming to optimize RegBN for training with large-scale datasets. As progress is made, we will update and release this enhanced version on the GitHub repository.\n\nOnce again, we thank the reviewer for every comment. Our responses aim to address all concerns. If more questions or suggestions arise, we invite the reviewer to discuss them during the Aug 10-16 reviewer-author period. \n"}, {"review_id": "6I5UafXgPZ", "reviewer": "Reviewer_JscA", "summary": "This paper introduces an approach for the normalization of multimodal data called RegBN. The RegBN aligns multimodal features by a learnable projection matrix with regularization in the form of the Frobenius norm. The regularization strength is updated with the L-BFGS optimization algorithm. Extensive experiments on various modalities are conducted to verify the method.", "questions": "Please see the weaknesses. Overall, this paper proposes a simple but effective scheme to better fuse multimodal features. The technical contribution is limited. The experimental results are comprehensive and convincing.  It would be better to provide more comparative experiments as suggested.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- Aligning multimodal data from the perspective of the normalization method can be promising.\n- Extensive experimental results show the effectiveness of the proposed RegBN.", "weaknesses": "- The method is not very novel for the reviewer. Essentially, RegBN is an l2 regression with regularization in the form of the Frobenius norm on weight. Moreover, L-BFGS has been widely adopted to update the regularization strength.\n- From the extensive experiments, RegBN is actually a multimodal feature fusion module. Other than baselines of normalization methods such as MDN and PMDN, more comparisons with other multimodal feature fusion modules such as [1] and [2] should be provided.\n- Why the method is called RegBN. It is not clear how RegBN is connected with the normalization method.\n\n[1] Yikai Wang et al. Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion.  2021\n[2] Yikai Wang et al. Deep Multimodal Fusion by Channel Exchanging. 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the comments and feedback provided by the reviewer. We would like to address the concerns raised regarding the presentation of our method in the manuscript. We reordered the comments to ensure an efficient and comprehensive response, taking into account overlapping concerns. \n\n*Comments #1&2: \nC1) From the extensive experiments, RegBN is actually a multimodal feature fusion module. Other than baselines of normalization methods such as MDN and PMDN, more comparisons with other multimodal feature fusion modules such as [1] and [2] should be provided.  \nC2) Why the method is called RegBN. It is not clear how RegBN is connected with the normalization method.*\n\n\n**Authors' Response**: We have already addressed this concern meticulously earlier in Supplementary-Section A: Multimodal Normalization and Fusion. In that section, we provided a clear discussion about fusion and normalization. In broad terms, fusion and normalization are distinct subjects with unique tasks. RegBN functions by normalizing input X in relation to input Y, resulting in a normalized X with the same dimensions as the original input X. This stands in contrast to fusion, where inputs X and Y are combined to generate one or more outputs with distinct content and dimensions. Hence, RegBN, as a normalization method, can be used in the structure of any fusion or deep learning method. We are well informed about the latest advancements in the fusion domain, and it is noteworthy that the paper [2] recommended by the reviewer had been previously cited as ref [51] in the manuscript. Our manuscript comprehensively covers the nuances of early, layer, and late fusion, as depicted and elaborated in Figure I.1 in Supplementary. We have incorporated a segment of this talk into Supplementary-Section G, addressing Question 3 for more clarification.\n\n*Comment #3: The method is not very novel for the reviewer. Essentially, RegBN is an l2 regression with regularization in the form of the Frobenius norm on weight. Moreover, L-BFGS has been widely adopted to update the regularization strength.*\n\n**Authors' Response**: Based on the previous comment, there may have been confusion by the reviewer about the actual role of RegBN (as a fusion or a normalization layer) and, therefore, also the confusion about its novelty. The other respected reviewers clearly stated the high novelty of RegBN. The novelty of RegBN lies in addressing challenges for multimodal data normalization. Kindly take into account that we do not assert novelty in our use of L-BFGS. We chose it as a practical tool for implementing our method. However, our approach and the objective of this study remain adaptable to any future minimization methodologies.\n\n\nWe trust that the provided responses address the reviewer’s concerns regarding RegBN. We have taken diligent measures to ensure comprehensive coverage of RegBN details within both the manuscript and Supplementary. As the reviewer-author discussion period is scheduled from August 10 to August 16, we eagerly await any forthcoming questions and will be delighted to offer further clarification during this time."}, {"review_id": "bsrYsrfS8D", "reviewer": "Reviewer_6Kg3", "summary": "This paper focuses on  mitigate the influence of unwanted variability and bias in multimodal data. To this end, the authors proposed a novel non-parametric regularization technique, i.e., RegBN. The proposed method utilizing Frobenius norm to constrain cross modality consistency, enabling model to discern underlying patterns. Comprehensive experiments show the usefulness of proposed approach in various datasets from different research areas.", "questions": "Please address the issues raised in the weakness section.", "limitations": "Please address the issues raised in the weakness section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Paper is well-written and follows a good structure. \n2. Comprehensive performance evaluation are shown, involving various recent multimodal methods and seven datasets, that proposed approach is working in practice.\n3. Mitigating unwanted variability and bias of multimodal data is an important topic and few methods have touched this field. The proposed method is interesting. Improving the reliability of multimodal learning by normalizing multimodal feature is a new perspective.", "weaknesses": "Major\n\n- Method: The proposed method mitigates the influence of unwanted variability and bias by minimizes the linear relationship between multimodal layers. In Figure 1, the authors claim that \"by leveraging RegBN, these data sources can be rendered independent, thereby enabling a multimodal network to discern underlying patterns and optimize its performance\". To the best of my knowledge, exploring consistency and complementarity[1] may be two mainly underlying reason why multimodal model outperforms the unimodal. Does the proposed method ignore to explore the cross-modal consistency?\n\n- Analysis: In line 108, the authors claim that \"The goal is to find potential similarities (mainly caused by confounding factors and dependencies during data collection)\". In my view, this is an essential assumption for this paper. I think the authors should provide more discussion about why potential confounding factors and dependencies cause these similarities. It could be necessary to give some explanation and citations.\n- Comparisons: In line 33 \"For instance, the race or gender of speakers in audio classification, backgrounds in video parsing, or the education level of patients in dementia diagnosis is commonly recognized as confounders.\". This statements seems very closely to the motivation of DRO[2]. It is appreciated to give some discussion about the relationship between RegBN and DRO.\n\nMinor\n\n- Ambiguous notations: In line 101 and 102, what is the meaning of N and M?\n\n[1] Deep Partial Multi-View Learning, Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu and Qinghua Hu, IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE T-PAMI)\n\n[2] Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pages 6781–6792. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for the comments and feedback. We have addressed each comment carefully and provided a point-by-point response, as outlined below.\n\n*Question #1: Method: The proposed method mitigates the influence of unwanted variability and bias by minimizes the linear relationship between multimodal layers. In Figure 1,... To the best of my knowledge, exploring consistency and complementarity [1] may be two mainly underlying reason why multimodal model outperforms the unimodal. Does the proposed method ignore to explore the cross-modal consistency?*\n\n**Authors' Response**: In [1], the authors aim to improve performance with missing data. When “views” or data sources are missing for a specific instance, the missing data is imputed. The consistency is used for the generation of the missing information as constraints. In our work, we are not considering missing data, which could be the subject of future work. In the case of complete observations, having the same information multiple times in the network does not provide benefits to the classification and rather makes the network more inefficient (or equivalently called imbalanced modality), as demonstrated in our experiments. RegBN aims at making the different modalities' data independent by removing confoundings. As demonstrated by the quantitative and qualitative experimental results, ensuring independence can improve the reliability of analyses and predictions by leveraging the synergies between different types of information while minimizing confounding impacts between modalities. Thanks to this comment, we have included a concise discussion on consistency, complementarity, and multimodal normalization in the Supplementary’s Q&A section-Q1.\n\n*Comment #2: Analysis: In line 108, the authors claim that \"The goal is ...\" It could be necessary to give some explanation and citations.*\n\n**Authors' Response**:  We apologize if the text is not clear about the confounders. By definition, a confounder is the common cause of treatment and outcome. Hence, there are similarities between confounder and treatment. We want to remove those similarities (dependencies) to avoid biased results. As an example, the imaging device may impact the characteristics of images, as it is the case for medical images. If the task is to predict diagnosis, and the diagnoses are not balanced across scanners, the model can learn to differentiate scanners instead of disease. Hence, we want to remove the dependence of the scanner on the images. \nWe will enhance the clarity of the Introduction Section by incorporating the following sentences:\n“Confounding variables pertain to external factors that introduce bias (either positive or negative)  in the relationship between the variables being studied [1*,2*]. The complexity of confounders emerges from their potential pervasiveness across diverse data modalities. For instance, in image analysis, confounders might encompass lighting variations, while in audio classification, speaker attributes like race or gender can be confounding factors. In video parsing, backgrounds play a role, and in dementia diagnosis, the education level of patients can be a confounder. Furthermore, positive or negative correlations can exist among heterogeneous data that impact the distributions of the learned features [2*, 3*].”\n\n[1*] Elwood JM, editor. Causal Relationships in Medicine. Oxford: Oxford University Press; 1988. p. 332.\n\n[2*] Soleymani et~al.. A survey of multimodal sentiment analysis. Image and Vision Computing, 65, pp.3-14, 2017.\n\n[3*] Manuscript: ref 28\n\n*Comment #3: Comparisons: In line 33 \"For instance,...\". This statement seems very close to the motivation of DRO[2]. It is appreciated to give some discussion about the relationship between RegBN and DRO.*\n\n**Authors' Response**:  The paper in [2] introduces JTT and compares it to group DRO in the experiments. JTT addresses the problem that classifiers may have an overall high accuracy, but certain groups may have low accuracy. JJT first learns a classifier on the input data and then assigns higher weights to training samples that were misclassified. Subsequently, training a second model with the weighted training set pays more attention to those misclassified samples. This can help with the worst group’s accuracy. \nIn RegBN, we address the normalization of multimodal data, which is a very different problem from JTT. We consider multiple input modalities. For instance, consider the example that we would have audio recordings and the gender of the speaker as input. JTT would first train a model and maybe, the samples with the worst accuracy would mainly contain men. By putting more emphasis on the worst samples in the second training, the performance of men's audio recordings may increase. For RegBN, we would have the audio data and gender as inputs. The RegBN layer would now create a latent representation of audio that is independent of the gender variable. In the next step, the classification layer takes normalized audio and gender as its inputs, wherein the audio features are gender-neutral. \n\n*Minor Comment: Ambiguous notations: In lines 101 and 102, what is the meaning of N and M?*\n\n**Authors' Response**:  We thank the reviewer for highlighting this point. In this study,  'f' and 'g’ are N- and M-dimensional feature maps, respectively. We will make the necessary revision to the text to accurately reflect this.\n\nWe sincerely appreciate the reviewer's insightful suggestions, and we are committed to integrating them into the final version of the manuscript. We are hopeful that our detailed responses have effectively attended to all of the reviewer's concerns. In the event that any additional questions or points of clarification arise, we kindly invite the reviewer to raise those during the forthcoming reviewer-author discussion period.  Your comment is greatly appreciated, and we look forward to engaging in further discussions during the mentioned timeframe."}]

Author rebuttal:
We sincerely thank the reviewers for their insightful and constructive feedback. We have incorporated their suggestions and responded comprehensively to their comments. Since the reviewer comments do not share common themes, we have addressed them individually. We have incorporated a Q&A section in the Supplementary file that concisely summarizes the discussion related to specific questions raised by the reviewers. This section, provided in one page with 4 questions, is intended to proactively address potential inquiries that readers might have about the same matter.  We hope that our study will be well-received as a valuable contribution to the NeuroIPS' focus on theory & application. We remain available for any further discussions or inquiries the reviewers may have.

It is important to mention that the enclosed file includes figures and tables that have been prepared to address some of the comments from two respected reviewers with the IDs: eB1v and saMu.

Best regards, 

Authors


Meta-review: There is a clear consensus among reviewers that the paper introduces a novel approach towards solving the problem of multimodal data normalization that is model-agnostic and reduces mutual information among multimodal features. The paper is well motivated and provides good experimentation and ablative study which adds to the merit of the work. Some questions were raised on the technical aspect of the proposed method and its limitations as well as discussions of the results section e.g. RegBN requiring as separate optimization process along the network training, lack of experimentation on multimodal foundational models.  Authors have addressed several points in their rebuttal which have alleviated most reviewers’ concerns. The AC finds a strong support from reviewers to merit the paper for publication. It is highly encouraged for the authors to take the advantage on the discussions raised by reviewers for their final revision using from both pre-/post-rebuttal phase comments.
Decision: Accept (poster)