On Uni-modal Feature Learning in Multi-modal Learning
Authors: Chenzhuang Du
Jiaye Teng
Tingle Li
Yichen Liu
Tianyuan Yuan
Yue Wang
Yang Yuan
Hang Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Supervised Multi-modal Late-fusion Learning

Abstract: We abstract the features of multi-modal data into 1) uni-modal features, which can be learned from uni-modal training, and 2) paired features, which can only be learned from cross-modal interaction. Multi-modal joint training is expected to benefit from cross-modal interaction on the basis of ensuring uni-modal feature learning. However, recent late-fusion training approaches still suffer from insufficient learning of uni-modal features on each modality and we prove that this phenomenon does hurt the model's generalization ability.
Given a multi-modal task, we propose to choose targeted late-fusion learning method from Uni-Modal Ensemble (UME) and the proposed Uni-Modal Teacher (UMT), according to the distribution of uni-modal and paired features. We demonstrate that, under a simple guiding strategy, we can achieve comparable results to other complex late-fusion or intermediate-fusion methods on multi-modal datasets, including VGG-Sound, Kinetics-400, UCF101, and ModelNet40.

Under review as a conference paper at ICLR 2023
ON UNI-MODAL FEATURE LEARNING IN SUPERVISED MULTI-MODAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We abstract the features of multi-modal data into 1) uni-modal features, which be learned from uni-modal training, and 2) paired features, which can only learned from cross rross-modal interaction. Multi-modal joint training is expected benefit from cross-modal interaction the basis of ensuring uni-modal feature learning. However, recent late-fusion training approaches still suffer from insuf- ficient learning of uni-modal features on each modality, and we prove that this phenomenon does hurt the model's generalization ability. Given a multi-modal task, we propose to choose targeted late-fusion learning method from Uni-Modal Ensemble (UME) and the proposed Uni-Modal Teacher (UMT), according to the distribution of uni-modal and paired features. We demonstrate that, under simple guiding strategy, we can achieve comparable results to other complex late-fusion or intermediate-fusion methods on multi-modal datasets, including VGG-Sound, Kinetics-400, UCF101 101,,,, and ModelNet40.
Uni-Modal Training
Multi-Modal Training
Modallty
Modality
Modality
Modality
uni-modal festures
paired features
uni-modal festuuees
paired fealures
learned features
unleamed features
Figure Overview Modality Laziness. Although multi-modal joint training provides the op- portunity for cross-modal interaction to learn paired features, the model easily saturates and ignores the uni-modal features that are hard to learn but also important to generalization.
INTRODUCTION
Multi-modal signals, e.g., vision, sound, text, are ubiquitous in our daily life, allowing us to per- ceive the world through multiple sensory systems. Inspired by the crucial role that multi-modal interactions play in human perception and decision (Smith Gasser 2005). substantial efforts have been made to build effective and reliable computational multi-modal systems in fields like multi- media computing (Wang et al. 2020 Xiao et al.) 22200), representation learning (Arandjelovic & Zisserman 2017 Radford et al.
2021 and robotics (Chen al. 2020a).
According to how the features of multi-modal data can be learned, we abstract them into two cate- gories: (1) uni-modal features, which can be learned from uni-modal training, and (2) paired fea- tures, which can only be learned from cross-modas interaction. In this paper, we focus on multi- modal tasks where uni-modal priors meaningful (ay et al. |217]] Chen al. 2020b). Ideally, CCeen
we hope that multi-modal joint training can learn paired features through cross-modal interactions on the basis of ensuring that enough uni-modal features leamne..
Uni- modal prior here means that we get predictions only according modality in multi-modal tasks.

---

Under review as a conference paper at ICLR 2023
However, recent late- fusion methods still suffer from learning insufficient uni-modal representations of each modality (Peng et al. 2022). We term this phenomenon as Modality Laziness and illustrate that in Figure [ We theoretically characterize Modality Laziness and prove that it does hurt the generalization ability of the model, especially when uni-modal features are dominant in the given task. Besides the laziness problem, another shortcoming of recent late-fusion approaches is that they are complex to implement. For example, G-Blending (Wang et al.) 2020) needs an extra split of data tanetttiig-o-gggnerliiziiie
to estimate the again and again. Peng et al. (2022) proposes OGM-GE, which dynamically adjusts the gradients of different modalities during training. However, it needs to tune too many hyper-parament parameters [ includ- ing the start and end epoch of the gradient modulation, an "alpha" used to calculate the coefficients for the modulation and whether adaptive Gaussian noise Enhancement (GE) is needed. The more complicated thing is that these hyper-parameter need to be re-tuned on new datasets.
ratio to re-weight the losses and then re-train the model
To this end, more simple and effective methods are urgently needed. We pay attention to the learning of uni-modal features and propose to choose targeted late-fusion training method from Uni-Modal Ensemble (UME) and proposed Uni-Modal Teacher (UMT) according to the distribution of uni- modal and paired features. If both uni-moda and paired features are essential, UMT is effective, which helps multi- mudti-modal models better learn uni- animmodal features via uni-modal distillation; if both modalities have strong uni-modal features and paired features are not important enough, UME is properer, which combines predictions of uni-modal models and completely avoids insufficient learn- ing of uni-modal features. We also provide an empirical trick to decide which one to use. Under this guidance, we achieve comparable results to other complex late-fusion or intermediate-fusion methods on multiple multi-modal datasets, including VGG-Sound (Chen et al., 20206), Kinetics- 400 (Kay et al. 2017]. UCF101 (Soomro et al.) 2012) and ModelNet40 (Wu et al. 2022)
2 RELATED WORK
Multi-modal training approaches aim to train multi-modal model by using all modalities simul-
taneously (Liang et al. 2021). including audio-visual classification (Peng et al. 2022| Xiao et al. 2020
Panda et al. 2021) action recognition (Wang al. 2020 Panda et al. 202T) , visual ques-
tion ans answering Agrawal et al. 2018) and RGB-D segmentation (Park et al. 2017 Hu et al. 2019
Seichter et al. 2020). There are several different fusion methods, including carly/middle fusion (Se-
ichter et al.
2020, Nagrani et al. 2021 Wu et al. 2022) and late fusion (Wang et al. 2020 Peng al., 2022, Fayek & Kumar 2020p. In this paper, we mainly consider the late-fusion methods following Wang et (2020). which is convenient and straightforward to evaluate the learning of uni-modal features. We demonstrate that simple late-fusion approaches can outperform approaches
with more complex model architecture (Wu et al. 2022. Xiao et al. 2020).
learning theory is still at an early age. A line of work focuses understanding multi-view tasks (Amini et al. 2009 Xu et al. [013 Arora ct 2d.66 Allen-Zhu & Li 2020), and our assumption on the data structure partially stems from Allen-Zhu & (2020). Huang al. (2021) explains multi-modal learning is potentially better than
Multi-modal learning theory. The research on
uni-modal learning and Huang et al. (2022) explains why failure exists in multi-modal learning. Our
paper investigates the different types of features in multi-modal data and provides solutions for the
weakness of multi-modal learning.
Knowledge distillation was introduced to compress the knowledge from an ensemble into smaller and faster model but still preserve competitive generalization power (Buciluă et al.) 2006 Hinton 2021 Allen-Zhu & Li, 2020). In this paper, we propose
al.] 2015 Tian et al., 2019 Gou et
Uni-Modal Teacher to leverage uni-modal distillation for joint training to help the learning of uni-
modal features, without involving cross-modal knowledge distillation (Pham et al. 2019 Gupta
et al. 2016 Tan & Bansal 2020; Garcia et al. 2018 Luo et al. 2018).
3 ANALYSIS, LEARNING GUIDANCE AND THEORY
In this section, we show the drawbacks and advantages of joint training. On one hand, joint training results in insufficient learning of uni-modal features (Modality Laziness). On the other hand, it
https: www.ghttbb.ooomm com Saooo//GWw--aaa Lab /aaplaaaaa/./.... BWWMMGGGCVPP222222
2

---

Under review as a conference paper at ICLR 2023
Table 1: Top test accuracy (in %) of linear evaluation encoders from various multi-modal late- fusion training methods and uni-modal training on VGG-Sound and UCFI0I.
VGG-Sound
UCF101
Method
RGB Encoder
Audio Encoder
RGB Encoder
Opt-Flow Encoder
Linear-Fusion
15,56
43.44
75.66
48.08
MLP-Fusion
14.52
40.01
75.65
51.89
Attention-Fuse
13.31
43.97
74.84
7.72
G-Blending
17.69
43.90
74.91
44.49
OGM-GE
15.60
41.95
73.54
65.03
Uni-Modal Training 23.17
45.15
77.08
74.99
--- Uni-Video Best ACC
0.8
Uni-Opt Flow Best ACC
Text Acc
0.4
Uni-Videa
0.1
LinearHead
Linear-Head
Atterion-tted
Attention-teed
MLPHeed
G Blending
G Blending
OGKIGE
OCAIGE
Epoch
Epoch
(a) RGB encoder evaluation on (GG-Sound.
(b) Optical flow encoder evaluation on UCF101.
Figure 2: By building a linear classifier on encoders and checking the top-1 accuracy, we evaluate the RGB encoder in VGG-Sound and the optical flow encoder in UCFI01 from different multi-modal late-fusion methods.
allows interactions between modalities learn representations beyond uni-modal features, namely paired features. Based on this, we offer guidance on multi-moda late-fusion learning. Finally, we provide a theoretical analysis of Modality Laziness and justification for our solution.
The importance of uni-modal prior varies across different multi-modal tasks. In tasks like video classification (Chen et al.) (2220b) and action recognition (Feichtenhofer al.) 2016 Wang et al. 2020), uni-modal models can achieve good performance alone, suggesting that uni- modal priors in these settings are essential. Visual question and answering (VQA) (Agrawal al.) 2018) is a counter example. Specifically, the same image with different text questions may have totally different labels, making it pointless to check its uni-modal accuracy. this paper, we focus the tasks where uni-modal priors essential, following Wu et al. (022): Peng et al. (0022)
Discussion.
3.1 INSUFFICIENT LEARNING UNI-MODAL FEATURES MULTIIMODAL TRAINING
This subsection illustrates that existing multi-modal late-fusion training methods suffer from Modal- ity Laziness. Even recent methods, G-Blending (Wang al. 2020 and OGM-GE (Peng et al. 2022). are no exception.
In multi-modal late-fusion learning, each modality is encoded by its corresponding encoder and then fusion module is applied on top of them to produce outputs. By building classifier on frozen encoder (Chen et al. 2020c), we assess the learned representations the encoder:
As Table shows, all encoders from multi-moda training worse than those from uni- modal traning, especially the RGB encoder VGG-Sound and optical flow encoder in UCF101. No matter which optimizer used (Appendix A.3.
3

---

Under review as a conference paper at ICLR 2023
Table 2: Top-1 test accuracy of multi-modal models and uni-modal models certain classes of VGG-Sound. Avg Pred: average the two uni-modal models' predictions directly. Linear Clf: train multi-modal classifier on top of uni-modal trained encoders. Naive Fusion: train multi-modal late-fusion model from scratch.
Class
164
303
33
255
91
4
152
127
155
mean acc
Uni-RGB
4
3
4
12
2
15
Uni-Audio
30
34
10
43
50
18
53
32
27.7
Avg Pred
37
37
7
28
63
21
51
30
28.4
Linear Clf
35
15
33
27
60
65
26
2
53
49
36.5
Naive Fusion
48
22
55
67
26
40
39.5
Algorithm Uni-Modal Teacher (UMT) for late-fusion learning
Input:
Uni-modal supervised pre-trained models E F
pretrain pretrain random initialized late-
fusion multi-modal model Fm"m, iteration number N, loss weight Atask, Adiatill- for 0 to N do
Sample multi-modal data {xmL X Y} }}~ D.
Compute uni-modal pre-trained features fpré fpra of the data by F F pretrain pretroin"
Compute the prediction and features Y, _ f from multi-modal model.
Compute the losses between Y, fm, fmn and Y, fpre fpre and multiply by the
Atakk Adistill, Adistsul, respectively.
Update the multi-modal model by SGD or its variant. end for Return: A multi-modal model trained UMT.
As Figure 2 shows, throughout the training process, the two encoders mentioned above not only cannot achieve comparable performance to their uni-modal counterparts but are far worse than them.
3.2 How DOES MODEL BENEFIT FROM MULTI-MODAL TRAINING?
In Sec 3.1 we empirically show that recent late-fusion methods suffer from insufficient learning of uni-modal features. Combining predictions from uni-modal trained models avoids laziness by nature, which raises another question: How does multi-modal model benefit from mudti-modal training? We answer this question by investigating different models on VGG-Sound and find that the model learns some representations beyond uni-modal features.
As Table 2 shows, certain classes VGG-Sound, the accuracy naive fusion (navie fusion or naive multi-modal late-fusion learning means carefully designed tricks are used) exceeds the sum of the accuracy of the two uni-modal models. Besides, we evaluate two other methods. One to directly average the uni uni-modal models' predictions, which has little cross- moos--oddal interaction. The other one is to train a multi-modal linear classifier on top of uni-modal pre-trained encoders, where modalities
interact with each other through the linear layer. We find that naive fusion training, which owns maximum freedom of cross-modal interaction among these models, gets the best mean accuracy across these classes, suggesting that joint training enables the model learn representations beyond uni-modal features, which we term as paired features. They are type of feature that uni-modal training cannot learn.
We also analyze more datasets and find that different datasets have different characteristics, we put more experimental analysis and interpretation in Appendix A.7.
We offer more explanations on paired features in Appendix 10
3.3 GUIDANCE ON MULTI-MODAL LEARNING
Given a multi-modal task, both uni-modal features and paired features are essential, Uni-Modal Teacher (UMT) is effective; if both modalities have strong uni-modal features and paired features
4

---

Under review as a conference paper at ICLR 2023
not important enough, simply combining the predictions of uni-modal models works well, which is named as Uni-Modas Ensemble (UME).
UMT. Uni-Modal Teacher (UMT) is proposed for late-fusion joint training. It distills the pre- uni-modal features to the corresponding parts multi-moda late-fusion models. Distilling knowl- edge from uni-modal models help multi-modal models learn uni-modal features better, which happens in feature-level. The framework of UMT is shown in Algorithm and Figure 4 Noting that we use the same backbone in uni-modal and multi-modal model for a specified modality. The backbones and loss function used can be found in Sec More details can be found in Appendix
pre-trained
UME. If both modalities have strong uni-modal features, joint training does more harm than good. Combining predictions of uni-modal models avoids insufficient learning of uni-modal features by nature. Firstly, we can train uni-modal models independently. Then, we can give final output by weighting the predictions of uni-modal models. The simple ensemble method named as Uni- Modal Ensemble (UME). We demonstrate that UME can show competitive performance on certain multi-modal datasets.
An empirical trick to decide which method to use. We can train a multi-modal linear classifier on uni-modal pre-trained encoders and compare that with averaging predictions of uni-modal models. benefit from cross-modal interaction in
If the performance of the classifier is better, it means this task and we can choose UMT, where cross modsl interactions are preserved while guaranteeing improved learning of uni-modal features; otherwise, the simple gross-modal interaction does more harm than good because of the strong uni-modal features of each modality, and we can choose UME, which avoids Modality Laziness completely.
3.4 THEORETICAL CHARACTERIZATION AND JUSTIFICATION
In this subsection, we characterize Modality Laziness of Sec 3.1 from feature learning perspective and prove it does hurt the generalization of the model. And then, we give justification for the learning guidance proposed in See 3.3
Before diving into the technical details, we first provide some intuition behind the proof. Our goal is to show that how Modality Laziness happens in multi-modal joint training, and we refer Figure 3 as an illustration. Here we omit the effect of paired features for easier to understand the intuition. During the naive multi-modal training process, learning those easy-to-learn features suffices to reach zero training error (point A in Figure 3. However, the model is under-trained at point A, and the pero-training-ee
region stops us from further training. As a comparison, uni-modal models can learn more features and achieve point B, outperforming point A.
We next give Modality Laziness a theoretical explanation under a simple but effective regime. We mainly consider cases with two modalities n: and I= and similar techniques can be directly generalized to the cases with more modalities.
Data distribution. We formalize the distribution of the multi-mode features. Specifically. abstract the features into uni-modal features (Definition 3.1) and paired features (Definition 3.2
to describe the core differences between uni-modal training and multi-modal joint training. We consider the binary classification regime where the label y has a uniform distribution over 1-
without loss of generality. Such simplification is self-contained to describe the differences between uni-modal features and paired features.
Definition 3.1 (Uni-modal features, which can be learned from uni-modal training). The i-th uni- modal feature (f(x)"))) in modality zi generated a
w.p. p(fi), uf(( > 0;
w.p. 1 p(f)) -(i), ysi(x")) = 0; w.p. (fi), vf((am)) <
The i-th uni-modal feature (g: (2) in modality =² similarly generated with parameters p(g:) and e(g:). We simplify "with probability" "w.p."
5

---

Under review as a conference paper at ICLR 2023
n2
Training Procedure
n
Testing Procedure
B
F2
F22
F22
Test error decreases
Rulloodd
x
F1
F1
F1
I: Region where training error
Region where training error
5, J: the feature set learned from modality x in uni/multt-modal learning
Nalve multi-modal training reaches Point
B: Combining modal trained models reaches
Figure 3: An illustration of the feature learning results of uni-modal training and muulii-moda train- ing without considering paired features. uni-modal training, modality " learns feature set Fü However, naive joint training learns less features of each modality than uni-modal training when get- ting zero training error (namely F(). Uncontroversially, combining predictions of individual trained uni-modal models (B) outperforms naive joint training (A).
Definition 3.2 (Paired features, which can only be learned from cross-modal interaction). The j-th paired feature h is generated as:
w.p. p(h)), uhy( (h""
(x))) > 0;
w.p. 1 - -(h,) - =(h)), yh (an (x O;
w.p. e(hj), phy (an"))yya"" (x"") O.
When the context is clear, we abuse the notation T; to represent either fi (uni-modal feature in modality x=1), ), 9s (uni-modal feature in modality z²²), or h2 (paired feature). We name p(r,) the predicting probability of feature ri- When T; is present (meaning that r; * 0), we
I[r; > 0) - I[r; < 0) to predict y. Otherwise (r; = 0), we random guess uniformly over (-1, To simplify the discussion, we always assume e(fi) = p(ft)/c, where e is fixed constant. For
the ease of notations, we define the empty feature in Definition 3.3
Definition 3.3 (Empty Feature). Empty feature e; is kind of uni-modal feature (or paired feature) with p(e;)= = <(ex)= = 0.
Evaluation procedure. When the context is clear, we abuse r; denote the learned features. For each data point, we random guess ú on 1-1, 1} uniformly when Σ, I(r; > 0) = Σ, I(r; < 0). Otherwise, we predict the label by ÿ 21 I(r. > 0) Σ, I(ra 0)) We define the error as Σ, I(yr; < 0) Σ, I(yr; > 0).
Training procedure. (a.) multi-modal joint training, which directly train the model using both modality = and modality n (b.) uni-modal ensemble, which firstly train the features via independent training mm and m2 separately), and then combine the x -earnedd features and -learred features.
During the training process, we first initialize all the features with empty features es to imitate ran- dom initialization. The models then learn the features in descending order of predicting probability, meaning that the powerful features (with large predicting probability) are learned firs Our goal is to minimize the training error to zero
We now state our main theorem in Theorem 3.4 demonstrating naive joint training learn fewer uni-modal features compared to uni-modal training, which hurts the model's generalization.
Theorem 3.4. In uni-modal ensemble, assume that the training procedure learns bmt features in modality =h and learns bmz features in modality == We order the probability of uni-modal fea-
WWe abuse the notation h simplify the notations where h(x""1) and h(="²)) can have different forms.
Recent works have demonstrated that neural networks indeed prefer casy-to-learn features (Shah et al.
2020
Pezeshki et al. 2020).
We always assume that the training error be minimized to zero

---

Under review as a conference paper at ICLR 2023
tures (both xm and x² ) in decreasing order of predicting probability p. namely, P(1 p ..... In multi-modal training approaches, assume that the training procedure learns kmi uni-modal features in modality x=m,, learns km2 uni-modal features in modality x²'², and learns Kpa paired features with predicting probability p(h1)....,, ".b..... We provide three types of laziness:
(a. Quantity Laziness: kml + km2 + Kpp S min/bbi bm
(b. Uni-modal Laziness: Each modality in multi-modal training approaches performs worse than uni-modal training.
(c. Performance Laziness: Consider new testing point, then for every ô >> 0, if the following inequality holds:
Σ p(h,) sElRpal
Σ
PM + A
VE vE(b tbual
where A(0) (8(kipa bms -ims bmz kmmz) (og(1/5), then with probability at least 1 - 6, uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.
In theorem 3.4 we describe three notations of laziness problem: Quantity Laziness indicates that the number features learned naive multi-modal training less than uni-modal training. Uni- modal Laziness shows encoders from multi-modal training perform worse than from uni-modal training because of Quantity Laziness, which fits the experimental results in sec3.1. Performance Laziness compares the performance of multi-modal joint training approaches with Uni-Modal En- semble, demonstrating that when mni-modal features dominate, combining uni-modal predictions is more effective. We defer the complete proof to Appendix [B.1 and generalize that to modali- ties (Appendix B. p.
We give a concrete example in Appendix B.3 to better illustrate Theorem (3.4]
We prove that UMT proposed in Sec 3.3 indeed helps uni-modal feature learning and can also learn some easy-to-learn paired features in Theorem 3.5 and Appendix B.3
Theorem 3.5. Denote the paired features by hi.... h2 with corresponding predicting probability p(h1).... ..... p(hL). Assume that distillation can boost the training priority by pd >> O. If there exists paired features whose predicting probability exceeds the boosting probability pd, namely, the set S is not empty:
S = (hç p(h.) > po) # d.
Then UMT helps uni-modal feature learning and can also learn easy-to-learn paired features.
4 EXPERIMENTS
In Sec 3.4. we justify our method theoretically. In this section, we firstly introduce the experimental setup and then demonstrate that choosing a suitable learning method from UMT and UME can outperforming other complex late-fusion or intermediate-ffssion methods in various multi-modal tasks.
4.1 EXPERIMENTAL SETUP
Dataset. We run experiments four datasets. Kinetics-400 (Kay et al. 2017) is video recognition dataset with 240k videos for training and 19k for validation. We treat the two modalities, RGB and audio, the inputs. VGG-Sound (Chen et al. 2020b) is audio-visual classification dataset which contains over 200k video clips for 309 different sound classes. UCF101 (Soomro et al., 2012) is action recognition dataset with 101 action categories, including 7k videos for training and 3k for testing. ModelNer40
3D object classification task with 9,483 training samples and 2,468 test samples. Following Wu et al. (2022). we treat the front and rear view two modalities.
Training Settings. In VGG-Sound, UCF101 and ModelNet40, we use ResNet as our backbone, all with 18 layers. As for Kinetics-400, use 50 101 layers' ResNet encode the inputs. Noting that 3D CNN is used for visual data of VGG-Sound and Kinetics-400, The data preprocessing, hyper-parameters, optimizer can be found in the Appendix A.l and A.2
The prohahility is taken the randomness of the testing point

---

Under review as a conference paper at ICLR 2023
Table Comparison between averaging uni- Table 4: modal predictions and multi- molli- classifier fiers from the multi-modal linear classifier trained on uni-modal pre- pre-trained encoders. trained on uni-modal pre-trained encoders in UCFI01 This evaluation method borrows from Peng et al. (2022).
Evaluation of uni-modal classi-
Dataset
MM Clf
Avg Preds
VGG-Sound
51.0
46.1
Model
RGB
Opt-flow
Kinetics-400
76.4
74.8
UCFI01
84.4
86.8
Uni-Clf from MM Cif
68.2
52.9
ModelNet40 91.7
91.9
Uni-Modal Model
77.1
75.0
Table 5: Results of different late-fusion methods. Table 6: Comparison between UMT and means the result comes from its original paper
Audio-Visual SlowFast (Xiao et al. (2220)
on Kinetics-400. AVSlowFast an repre- sentative intermediate fusion method.
Method
VGG-Sound
Kinetics-400
Linear-Head
49.5
74.3
MLP-Heed
44.8
74.8
Method
RGB Encoder
Acc
Atten-Head
49.8
74.1
AVSlowFast
SlowFast-50
77.0
49.9
73.2
UMT (ours)
SlowFast-50
78.1
G-Blending
50.4
75.8*
OGM-GE
50.6*
74.5
AVSlowFast UMT (ours)
SlowFast 101
78.8
SlowFast- 101
79.4
UMT (ours)
53.5
76.8
4.2 AN EMPIRICAL TRICK TO DECIDE WIICH LEARNING METHOO TO USE.
We train a multi-modal linear classifier on frozen uni-modal pre-trained encoders and compare that with averaging uni-modal predictions. As Table 3 shows, in VGG-Sound and Kinetics-400. the classifier is better, meaning cross-modal interaction can benefit the classifier in the two datasets. However, in UCF101 and ModelNet40, averaging uni-modal predictions performs well. To explore why the classifier fails in UCF101, we check the uni-modal classifiers of the newly trained multi- modal linear layer (details in Appendix (A..I]). As Table shows, they are far worse than the uni- modal models. The result shows the simple linear classifier suffers from serious Modality Laziness in UCF101, which negatively impact the performance. Both modalities ModelNet40 also have strong modal features and can achieve 89% accuracy individually. Averaging uni-modal predic- tions avoids laziness problem and achieves competitive performance. Based on the above analysis, we perform UMT on VGG-Sound and Kinetics-400, and UME on UCF101 and ModelNet40.
4.3 UMT IS AN EFFECTIVE REGULARIZER FOR UNI-MODAL FEATURE LEARNING
In this subsection, demonstrate that Uni-Modal Teacher outperforms other multi- training methods in VGG-Sound and Kinetics-400. We use MSELoss as the distillation loss and set the weight of that 50. Cross Entropy is used as classification loss and its weight set as
4.3.1 UMT IS EFFECTIVE ON VGG-SOUND AND KINETICS-400.
UMT vs Other Late-Fusion Methods. The late-fusion architecture is commonly used for multi- modal classification tasks (Wang al. 2020 Peng et al. 2022). In late-fusion architecture, the fea- tures are extracted from different modalities by the corresponding encoders, and then the head layer applied to output predictions. We compare different heads, including linear layer, MLP, and atten- tion layer. In UMT, we use a simple linear layer as the multi-modal head. We also conduct another experiment, which adds extra uni-modal linear heads to receive the uni-modal features and gen- erating additional losses to joint optimize the model, namely Auxiliary-CEloss. Auxiliary-CElos gives all losses equal weights, while G- Blending reweights the losses according to the overfitting scneralization-ratie (OGR) (Wang al. 2020). OGM-GE (Peng al. 2022 controls the optimiza- tion of each modality by online gradient modulation. However, both OGM-GE and G- G-Blending are complex implement. As shown in Table UMT outperforms other methods.
8

---

Under review as a conference paper at ICLR 2023
Table 7: Evaluation on the encoders trained by Table 8:
Self-Distillation
UMT VGG-
naive multi-modal training and UMT.
Sound.
VGG-Sound
Kinetics-400
Method
Test Acc
Methods.
RGB
Audio
RGB
Audio
Baseline
49.5
Uni Train
23.2
45.2
74.1
23,5
Self-Distill (label)
49.7
MM Baseline
15.9
18.3
72.9
18.3
Self-Distill (feature)
49.9
UMT
24.4
45.9
74.6
21.6
UMT
53.5
Table 9: Comparison Uni-Modal Ensemble with Table 10:
Comparison between Uni-Modal En- semble and balanced multi-modal learning algo- rithm (Wu al. 2022) on ModelNet40. * means
other joint training methods on UCF101.
Method
Test Acc
the result comes from Wu et al. (2022).
Linear-
82.3
Method
Test Acc
MLP-Head
80.0
Atten-Head
74.2
multi-modal (vanilla)
90.09 0.58*
Aux-CELoss.
81.3
+RUBi
90.45 = 0.58
G-Blending
83.0
+random
91.36 0.10*
OGM-GE
84.0
+guided
91.37 = 0.28
UME (ours)
86.8
UME (ours)
91.92 + 0.14
UMT AVSlowFast. Audio- SlowFast representative intermediate fusion method. We compare UMT with AVSlowFast in Kinetics-400. As Table ( shows, under different RGB encoders, UMT consistently exceeds AVSlowFast, although cannot reproduce their results due to the dynamics of Kinetics-400 (Appendix(A..) A.III
Ablation Study of UMT. We first evaluate the encoders of UMT by training linear classifiers on them to verify that UMT does improve the uni-modal feature learning. As Table 7 shows, UMT makes its encoders stand out. Benefiting from uni-modal distillation, some encoders even out- perform their uni-modal counterparts. We then compare UMT with classic self-distillation meth- ods (distillation on soft label (Hinton et al.) 2015) and feature (Romero et al. 2014)). As Table shows, naive self-distillation can only bring limited improvement, showing that UMT improves overall performance by improving the uni-modal feature learning instead knowledge distillation.
4.3.2 UNI-MODAL ENSEMBLE IN MULTI-MODAL LEARNING
In this subsection, demonstrate that Uni-Modal Ensemble is effective multi-modas datasets where modalities have strong uni-modal features, outperforming other complex methods. Even though we don't combine these uni-modal predictions in any special way, but simply average.
In UCF101, we compare Uni-Modal Ensemble with various multi-modal aate-fusion methods. As Table 9 shows, although Gradient Blending (Wang et al.) 2020) and OGM-GE (Peng et al., 2022) outperforms baseline methods, they are far worse than Un-Moode Ensemble.
In ModelNet40, the main comparing methods come from Wu et al. (2022), which uses multi- modal DNN with intermediate fusion. It proposes a balanced multi-modal algorithm which balances conditional utilization of each modality by re-balancing the optimization step. UME surpass their balanced mulli-modaa algorithm, as Table [0]
CONCLUSION
This paper analyzes the phenomenon of insufficient uni-modal feature learning in multi-modal train- ing and proves that it does hurt the overall performance. We propose to choose proper learning method from Uni-Modal Ensemble and proposed Uni-Modal Teacher according to the distribution of uni-modal and paired features and demonstrate the effectiveness of the guiding principle.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
In our paper, we detail the specific implementation steps of the methods in Sec 3.3 and also give the values of the hyper-parameters we use in detail Appendix [A.2] We also provide the codes in supplementary material, which can reproduce the results of the experiments.
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer. Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4971-4980, 2018.
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, arXiv preprint arXiv:2012.098/6, 2020.
Massih R Amini, Nicolas Usunier, and Cyril Goutte. Learning from multiple partially observed views-an application multilingual text categorization. Advances in neural information process- ing systems, 22:28-36. 2009.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Raman Arora, Poorya Mianjy, and Teodor Marinov. Stochastic optimization for multiview repre- sentation learning using partial least squares. In International Conference Machine Learning, pp. 1786-1794. PMLR, 2016.
Cristian Buciluä, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Changan Chen, Unnat Jain, Carl Schissler, Sehastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual nav- igation in 3d environments. In Proceedings of the European Conference Computer Vision (ECCV), 2020a.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020c.
Kin Wai Cheuk, Hans Anderson, Kat Agres, and Dorien Herremans. nnaudio: An on-the-fly gpu audio to spectrogram conversion toolbox using 1d convolutional neural networks. IEEE Access, 8:161981-162003, 2020.
Haytham M Fayek and Anurag Kumar. Large scale audiovisual learning of sounds with weakly labeled data. arXiv preprint arXiv:200.00/995, 2020.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 103-118, 2018.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 17966):189-1189, 2021.
10

---

Under review as a conference paper at ICLR 2023
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2827- 2836, 2016.
Jack Hessel and Lillian Lee. Does my multimodal model learn mossl interactions? it's harder to tell than you might think! arXiv preprint arXiv:2010.06572, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:/503.0255 2015.
Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang. Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 1440-1444. IEFE, 2019.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multimodal learning better than single (provably). arXiv preprint Stti::2006.04338, 2021.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longho Huang. Modality competition: What makes joint training of multi-modal network fail in deep learning (provably). arXiv preprint arXiv:2203. 12221, 2022.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.0699 06950, 2017.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal represen- tation learning. arXiv preprint arXiv:210..07502, 2021.
Zelun Luo, Jun-Ting Hsieh, Lu Jiang, Juan Carlos Niebles, and Li Fei-Fei. Graph distillation for action detection with privileged modalities. In Proceedings of the European Conference on Com- puter Vision (ECCV), pp. 166-1833 2018
Arsha Nagrani, Shan Yang. Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34, 2021.
Natalia Neverova, Christian Wolf, Graham Taylor, and Florian Nebout. Moddrop: adaptive multi- modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8):1692-1706, 2015.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In ICML, 2011.
Rameswar Panda, Chun-Fu Chen, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, and Rogerio Feris. Adamml: Adaptive multi-modal learning for efficient video recognition. arXiv preprint arXiv:2/05.05 2021.
Seong-Jin Jin Park, Ki-Sang Hong, and Seungyong Lee. Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation. Proceedings of the IEEE international conference on computer vision, pp. 4980-4989, 2017.
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly fly gradient modulation. arXiv preprint PrXXi:2203.153322 2022.
Mohammad Pezeshki, Sékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. arXiv preprint 1XXi:20111009663 2020.
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póezos. Found in translation: Learning robust representations by cyclic translations between modalities. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6892-6899, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv: 103. 000000000000..00000000000 2021.
11

---

Under review as a conference paper at ICLR 2023
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitmets: Hints for thin deep nets. arXiv preprint arXiv:1412.65500 2014.
Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, and Horst-Michael Gross. Efficient rgb-d semantic segmentation for indoor scene analysis. arXiv preprint arXiv:2011.0665 2.0-9661,. 2020.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. arXiv preprint WxXiv:2006.077000 2020.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life. 11(1-2):13-29, 2005.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint 12 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1))1999-1958, 2014.
Hao Tan and Mohit Bansal. Vokenization: Improving language understanding with contextualized, visual-grounded supervision. arXiv preprint arXiv:2010.06775. 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv
preprint arXiv:I910 19699, 2019.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12695-12705, 2020.
Nan Wu, Stanislaw Jastrzebski. Kyunghyun Cho, and Krzysztof J Geras. Characterizing and over- coming the greedy nature learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp. 24043-24055. PMLR, 2022.
Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audio- visual slowfast networks for video recognition. arXiv preprint arXiv=2001.087400 2020.
Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. 1rX15:1304.6634, 2013.
arXiv preprint
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS
A.1 DATASETS
Here, we describe the preprocessing of Kinetics-400, VGG-Sound, UCF101 and ModelNet40 in detail.
Kinetics-400 dataset (ay et al., 2017) contains over 240k videos for training and 19k for valida- tion, which we download from cydfoundation R Kinetics-400 is a commonly used dataset with 400 classes, and we mainly follow the open source preprocessing methods process that. For RGB modality, we follow the procedure of PySlowFast? which resizes the video to the short edge size of 256, and for audio modality, we follow mmaction2 ing joint training, we take consecutive 64 frames from a video with Ips of 30 and random crop the video to 224*224, and for audio inputs, we take the specgram that can be aligned in time with the clip extracted from the video. When testing, we ensemble the predictions from uniformly sampled clips with RGB and audio from video and give the final outputs, following PySlowfast.
extract specgram features. When perform-
VGG-Sound dataset (Chen et al. 2020b), which contains over 200k video clips for 309 different sound classes, also used for evaluating our method. It is an audio-visual dataset in the wild where each object that emits sound is also visible in the corresponding video clip, making it suitable for scene classification tasks. Please note that some clips in the dataset are no longer available on YouTube, and actually use about 175k videos for training and 15k for testing, but the number of classes remains the same. We design a preprocessing paradigm to improve training efficiency as follows: (1) each video is interpolated to 256 256 and saved as stacked images; (2) each audio is first converted to 16 kHz and 32-bit precision in the floating-point PCM format, then randomly cropped or tiled to a fixed duration of 10s. For video input, 32 frames are uniformly sampled from each clip before feeding to the video encoder. While for the audio input, a 1024-point discrete Fourier transform is performed using nnAudio (Cheuk et al., (020), with 64 ms frame length and 32 ms frame-shift. And we only feed the magnitude spectrogram to the audio encoder.
UCF101 dataset (Soomro et al. 2012) is an action recognition dataset with 101 action categories, including 7k videos for training and 3k for testing. And we use the rgb and flow provided by (Fe- ichtenhofer et al., 2016). For RGB, we use one image of (3 224 * 224) as the input; while for flow, we use a stack of optical flow images which contained 10 x-channel and 10 y-channel images, So its input shape is (20 224 * 224). During training, we perform random crop and random horizontal flip as the data augmentation; while testing, resize the image to 224 and do not perform data augmentation operations.
ModalNet40 is 3D object classification dataset with 9,483 training samples and 2,468 test samples. We base on the front view and the rear view of the 3D object to classify that, following Wu et al.
2022
A.2 TRAINING HYPERPARAMETERS
In this subsection, show the hyperparameters of our experiments in UCF101 and VGG-Sound in Table [I]
As for Kinetics-400's RGB modality, we totally follow the hyperparameters and settings of PySlow- As for audio modality, we modify the hyperparameters 12 to be as consistent as possible with the RGB training for further joint training. Specifically, we use the same learning rate and batch size as RGB training used.
Fast [1
As for ModelNet40, we totally follow the experimental settings of Wu et al. 2022
"hups://gthub..comi//
audio features.py/
" http://gttiooooloococc//////oo/e 8x8 R50.yaml
audiofresnet/tsn r18 64x1x1 100c kinetics400 audio feature.py
I http:://giiuub.ccmmyuaaaaaaaaae multimodal learning
13

---

Under review as a conference paper at ICLR 2023
Table 11: The Hyperparameters used in experiments for VGG-Sound and UCFI01.
Hyperparameter
Value (VGG-Sound)
Value (UCF101)
Encoder
ResNet3D (Video), 2D (Audio)
ResNet2D(Both Modalities)
Linear Head
(1024, 309)
(1024, 101)
MLP Head
(1024, 1024)
(1024, 1024)
ReLUU
ReLU
(1024, 309)
(1024, 101)
Attension Head
Attension Layer (without new parameters) linear layer
Training Epoches
20
20
LR
le-3
le-2
Batch Size
24
64
Optimizer
Adam
SGD
Scheduler
StepLR (step=10, gamma=0.1) Reducel.ROOPlatean (patience=1)
Loss Fusion
Cross Entropy for task, MSE for distillation
A.3 CAN EXISTING OPTIMIZERS SOLVE MODALITY LAZINESS?
Table 12:
Top-1 test accuracy (in %) of linear classifiers trained on frozen encoders from multi- modal late-fusion training under different optimizers and uni-modal training on VGG-Sound.
Optimizer
Multi-modal Performance
Audio Encoder
RGB Encoder
SGD
47.13
40.02
15.53
RMSprop
47.90
42.77
13.64
Adagrad
42.19
35.68
19.65
Adadelta
23.18
17.70
17.37
Adamw
49.39
42.41
15.11
Adam
49.47
43.44
15.56
Uni- Tri-Training
45.15
23.17
While the results in Table show that different multi-modal methods suffer from learning insuf- ficient uni-modal features, how about changing the optimizer? To answer this question, we different optimizers for multi-modal fute-fusion training (with linear multi-modas head), including SGD, RMSprop, Adagrad, Adadelta, Adamw and Adam. As Table 12 shows, Modality Laziness exists no matter which optimizer used.
A.4 DETAILS ON UNI-MODAL TEACHER (UMT)
In this subsection, we describe how Uni-Molda Teacher (UMT) applies on multi-modal late-fusion tasks. The overall architecture can be found in Figure
UMT in late-fusion classification. In multi-modal late-fusion architecture, modalities are first en- coded by the corresponding encoders and then mapped to the output space by a multi-modal fusion head (leftt Uni-Modal Teacher distills the pre-trained uni-modal features the correspond- ing parts in multi-modal networks in multi-modal training (Figure 4 right). Uni-modal distillation happens before fusion, so it's suitable for late-fusion multi-mode architecture. The pre-trained uni- modal features generated by inputting the data to the pre- trained uni-modal models.
UMT's weights. For VGG-Sound and Kinetics, we use 50 (both audio feature distillation and RGB feature distillation) the distillation loss's weight. We test different distillation weights VGG- Sound and Kinetics-400. As shown in Table 1 on both datasets, UMT performs well with an distillation weight of 50.
14

---

Under review as a conference paper at ICLR 2023
gm
Pm
Pm;
Pm
Pmng
Pmmz
disstill
distill
muttti
multi
Figure 4: Model architecture of naive late fusion (left) and Uni-Modal Teacher (UMT) (right). 4m. is the encoder which is supervised pre-trained uni-modal data. (m, is random initialed encoder without pre-training. Lmatti is the loss between multi-modal predictions and labels. Ldistall is the uni-modal distillation loss.
Table 13: Different distillation weights of UMT on VGG-Sound and Kinetics-400
Dataset
10
20
50
100
VGG-Soung 49.46
49.51
51.31
51,51
53.46
53.11
Kinetics-400
74.25
74.99
75.57
76.11
76.77
76.55
A.5 DROPOUT IN MULTI-MODAL TRAINING.
Here we consider the common regularizer, dropout (Srivastava et al. 2014), and a variant of it, namely modality-wise dropout, which randomly drops (with probability 1/3) the feature from one modality in each iteration. Modality dropout is akin to the ModDrop in Neverova et al. (2015). As Table [4] shows, modality-wise dropout significantly better than dropout, which implies that modality-wist wise laziness is serious and modality-wise dropout is also effective.
A.6 FINETUNING THE UNI-MODAL PRE-TRAINED ENCODERS
In this subsection, we use the uni-moda pre-trained encoders' parameters as the initialized weights in multi-modal training and randomly initialize a multi-modal linear classifier on the encoders. We set the classifier's 's learning rate as 1e 3 and try different learning rates on the encoders.
As Table I5 shows, using the uni-modal supervised pre-trained encoder's weights in multi- modal training and then fine-tuning the whole multi-modal model can bring some improvement compared to naive fusion (49.46) but is worse than UMT, which gets 53.46 accuracy. When the learning rate of encoders is large, the encoders forget some abilities to extract uni-modal features.
Ngiam al. 2011 proposes to use Bimodal Deep Autoencoder to pre-train the encoders with multiple modalities. It is direction worth exploring to address Modality Laziness of deep multi- modal models.
A.7 THE ROLE OF CROSS-MODAL INTERACTION ON DIFFERENT DATASETS
In this subsection, we conduct various experiments to further investigate the effect of cross-modal interaction and explore the benefits and harms that cross-modal interaction brings in different multi-
Table 14: Dropout in multi-modal training on VGG-Sound.
Method
Performance
Baseline
49.46
Dropout
49.83
Modal-Drop
51.37
UMT
53.46
15

---

Under review as a conference paper at ICLR 2023
Table 15:
The top-1 test accuracy of finetuning the uni-modal pre-trained encoders and linear eval- uation on finetuned encoders on VGG-Sound.
Encoder Eval Audio
Encoder LR
Top-1 Acc
RGB
le-3
50.98
43.98
21.86
le-4
49.37
44.71
21.97
le-5
50.45
45.28
23.13
le-6
50.86
45.29
23.27
0
50.95
45.15
23.17
modal tasks/datasets. We find that in different datasets, cross- cross-modal interaction has a different effect on the performance.
A.7.1 AVERAGING THE UNI-MODAL PREDICTIONS THE LINEAR CLASSIFIER TRAINED ON UNI-MODAL PRE-TRAINED ENCODERS.
In Sec 4.2, we train a multi-modal linear classifier on frozen uni-modal pre-trained encoders and compare this classifier with directly averaging uni-modal models' predictions. As Table shows, this classifier does not consistently outperform simply averaging the uni-modal predictions on all datasets. shows better performance on VGG-Sound and Kinetics-400, but worse performance on UCF101 and ModelNet40.
To further explain this phenomenon, we check and disassemble this new trained multi-modal clas- sifier on UCF101. In the late-fusion multi-modal training, the features of different modalities are concatenated first and then the multi-modal classifier receives them and output predictions. Differ- ent modalities in the classifier do not share the parameters. So we split the new trained multi-modal linear classifier into uni- uni-modal classifiers. We use the uni-modal pre-trained encoders to extract features and then the uni-modal classifiers receive the corresponding features and output predic- tions. Noting that OGM-GE (Peng et al., (002)) uses similar technique to check how well different modalities trained. As Table shows, the uni-modal classifiers from new trained mudti-modal classifiers are significantly worse than uni-modal models, implying that the multi-modal classifier trained on uni-modal pre-trained encoders suffers from serious Modality Laziness UCF101, al- though it is just a simple linear layer, resulting in worse performance than directly averaging the uni-modal predictions.
A.7.2 CLASS-LEVEL EVALCATION DIFFERENT MULTI-MODAL DATASETS
In this subsection, we compare naive late-fusion learning with averaging predictions of uni-modal models in class level. It's obvious that there are more cross-modal interactions in naive fusion.
Although naive fusion suffers from learning insufficient uni-modal features, we find in some classes in Kinetics and VGG-Sound, the accuracy of naive fusion model outperforms averaging the uni- modal models' predictions, and even outperforms the sum of the accuracy of the two uni-modal models in VGG-Sound and Kinetics-400, as shown in Table 2 and 16
However, We cannot find any class that naive fusion can exceed the sum of the accuracy of the uni- RGB model and uni-flow model in UCFI0L. We select classes in UCF101 by sorting the differences of accuracy between naive fusion and the best uni-modal model in class level and the top ten with the largest difference selected. In these classes where naive fusion has advantages, averaging the predictions can outperform naive fusion in some classes (ID:29, 67, 71), and this phenomenon is not found in VGG-Sound and Kineties. And Tabla 4 and Table 17 show, both RGB and optical flow in UCF101 can get strong performance individually. All the evidence shows that in UCF101, the uni-modal features are totally dominate and any joint training can lead to serious Modality Laziness.
The mapping betwenn class ID and class name in different datasets The correspondence be- tween and name of the selected class in VGG-Sound is: 164: People Sniggering, 303: Wood Thrush Calling, 33: Cat Meowing, 255: Sea Waves, 91: Footsteps On Snow, 4: Alligators
16

---

Under review as a conference paper at ICLR 2023
Table 16: Top-1 test accuracy of different models some classes of Kinetics. The accuracy of naive fusion model outperforms averaging the uni-modal models' predictions, and even outperforms the sum of the accuracy the uni-audio model and uni-video model.
Class ID
53
90
184
2
368
158
113
263
287
mean accuracy
Uni-Audio Uni-RGB
0
0
0
4
0
0
0
4
2
1
42
50
22
28
39
43
29
82
76
50
46.1
Avg Pred
42
50
22
28
39
29
82
78
50
46.3
Naive Fusion
56
62
32
40
45
49
35
84
86
58
54.7
Table 17: Top-1 test accuracy of different models on selected classes of UCFI01. We select the top- 10 classes according to the gap of accuracy between the multi-mode and uni-modal models. As see, uni-modal model's performance is high, meaning paired features in UCF101 rare.
Class ID
6
10
12
22
29
31
48
57
67
71
mean accuracy
Uni-RGB
74
84
76
61
67
32
78
21
75
57
62.5
Uni-Flow
60
82
58
47
61
41
64
24
78
63
57.8
Avg Pred
70
95
79
64
86
46
89
36
98
83
74.6
Naive Fusion
86
95
87
72
83
59
92
42
88
73
77.7
Crocodiles Hissing, 152: People Gargling, 127: Mynah Bird Singing, 68:
Door Slamming, 155:
People Humming.
For Kinetics-400, sort the classes alphabetically from smallest largest according to the class name, and then we can get the mapping between class name and id.
In UCF101, the mapping can be found in classInd.txt, given file of UCF101.
A.8 CROSS-MODAL INTERACTION UNI-MODAL TEACHER (UMT)
In order to verify whether the multimodal loss in UMT makes sense, we train uni-modal models by knowledge distillation to get better performance than encoders trained by UMT and then combine them by introduc- ing new multi-modal classifier on these encoders. As Table Iable|I[shows, UMT works better in multi-modal performance, although the encoders of UMT in uni- modal evaluation are worse, showing that UMT indeed benefits from cross modal interaction.
Table 18: Comparison of UMT with com- bining uni-modal models trained by distil- lation VGG-Sound.
Method
RGB
Audio
R+A
Linear CIf UMT
25.99
46.00
52.98
24.43
45,89
53.46
A.9 EXPLORING
UMT FOR MULTI-MODAL SEGMENTATION
NYU Depth V2 dataset (Silberman al. 2012) contains 1449 indoor RGB-Depth data totally and we use 40-class label setting. The number of training set and testing set is 795 and 654 respectively. All perprocessing operations are following (Seichter et 2020).
In contrast to the late fusion classification task, the RGB- Depth semantic segmentation belongs to middle fusion. The main encoder receives RGB inputs, and the depth inputs are fed into the depth encoder. At each intermediate layer, the main encoder fuses its own intermediate outputs and the depth features obtained from the depth encoder, which makes it mid-fusion task (Seichter et al.) 2020). Since features generated by each layer matter, we distill multi-scale depth feature maps using the MSE loss. For feature maps from the RGB encoder, however, since they are generated by fusing cannot distill RGB feature maps directly like depth feature maps. To mitigate this effect, we curate predictors, namely layers CNNs, aiming to facilitate the fused feature maps to predict the RGB feature maps trained by the RGB modality before distillation. The
RGB and depth modalities.
17

---

Under review as a conference paper at ICLR 2023
Fused RGB
Predicted RGB
Pre-trained RGB
Depth Feature Maps
Pre-trained Depth
Feature Maps
Feature Maps
Feature Maps
Feature Maps
Predictor
disiil
Figure 5: Distillation details of UMT for RGB (left) and depth (right) modalities in multi-modal semantic segmentation (based on ESANet).
Table Model performance comparison under UMT and ESANet on NYU-Depth RGB-Depth semantic segmentation task.
Training Setting
Initialization
ESANet
UMT
From Scratch
38.59
40.45 (+1.86)
ImageNet Pre-train
48.48
49.39 (+0.91)
full schematic diagram is presented in Figure As shown Table 19 UMT can also improve multi-modal segmentation whether the encoder is pre-trained ImageNet not.
A.10 EXPLANATIONS ON PAIRED FEATURES
We revisit the definitions of uni-modal features and paired features: uni-modal features, which can be learned by uni-modal training; paired features, which can only be learned by cross-modal interaction in joint training. Different datasets contain different proportions of these features.
this subsection, we use synthetic datasets to explain the uni-modal features and paired features in multi-modal tasks.
Understanding different types of f'aatures in multi-modal tasks by synthetic datasets. Three different multi-modal datasets are generated to help understand the uni-modal features and paired features in multi-moda tasks. The process of data generation mainly refers to Hessel & Lee (2200)
Firstly, we generate dataset where each modality can extract the features give correct predictions. We name this dataset as Dataset a. The data generation process is as follows:
1. Sample random projection P1 E R44 xd and Pg E Rddxxd from U 0.5.)
2. Sample e Rdd ~ NO, 1). Normalize to unit length
3. Sample z e Rdd No, 1). Normalize æ to unit length
4. if Jar =| S 0.1, return to the Step
5. If - > 0.1, then y = else y = 0.
6. Get the data point (Piz, P2x. y)-
7. If the amount of data generated is less than N, return the Step 3; else break
The Piæ, Paz represents two modalities of the 200, 100, 50, 5000, respectively. And we randomly split 80% of the generated data as train set, and the rest serves test set. In this dataset, uni-modal models extract useful features to give correct predictions and multi-modal joint training not necessary, as Table 20 shows (Dataset o). We name the features, that uni-modal models can learns to give correct predictions in the given task, as uni-modal features.
dataset and set d1, d2, d, N as
Secondly, generate another dataset where the model must rely both the two modalities to make correct predictions. We name this dataset as Dataset 3. The data generation process is as follows:
1. Sample random projection P Rdixd and Pa dRdd from U(-0.5, 0.5).
18

---

Under review as a conference paper at ICLR 2023
Table 20: Test Accuracy of uni-modal models and multi-modal model different synthetic datasets.
Synthetic Dataset a mainly contains uni- uni-modal features which can be learned in uni-modal training; Synthetic Dataset mainly contains paired features which need joint training to learn; Synthetic Dataset contains uni-modal features and paired features.
Dataset
Synthetic Dataset a
Synthetic Datset
Synthetic Dataset y
Uni-modal (Modality 1)
100
51.4
70.9
Uni-modal (Modality 2)
100
51.8
70.1
Multi-modal
100
92
94.4
The confusion matrix of uni-modal model in Dataset 7, In the data labeled O, each modality contains features that can give correct prediction, while in the data labeled or 2, need both modalities together to make the right predictions. Predicted
Table 21:
Actual
0
100%
1
0
57%
43%
2
45.4%
54.6%
2. Sample T1, _ N/00 1). Normalize #1,42 to unit length
3. if [x] -x22 < 0.25, return to the Step 2.
4. If -22 > 0.25, then y 1; else y O.
5. Get the data point (P1#1, P2,22, v)-
6. If the amount of data generated is less than N, return to the Step 2; else break
This multi-modal dataset is different from the first dataset, because the labels in this dataset are highly dependent on the relationship between the two modalities. As we can see Ta- ble 20 (Dataset B), the uni-modal models can only gives about 50 percent accuracy, while the multi-modal models can give about 90 percent accuracy. In binary classification tasks, 50 percent accuracy is no different from guessing. In this dataset, because labels are heavily relied the relationships of the two modalities, we must train both modalities simultaneously to extract the joint representations learn the relationship of the modalities, which beyond uni-modal fea- tures. In order to better carry out theoretical analysis, we abstract these representations into paired tasks.
features, which can only be learned from multi-modal joint training in
Finally, we generate a dataset that contains both uni-modal features and paired features. We name this dataset Dataset 7. The data generation process follows:
1. Sample random projection P1 Rd x and P2 R from U U(-0.5,0.5)
2. Sample E Rdd (0, 1). Normalize unit length
3. Sample Rdd (0, 1). Normalize to unit length
4. if z 2 0.1, get the data point (Pıæ, Pax, y 0). else, return to the step until collecting 2500 data points.
5. Sample Fı, x2 E Rdd A/0 1). Normalize æ 22 to unit length. If x1 > -0.1 or
2 -0.1, resample.
6. if [m .22 < 0.25, return to the Step 5.
7. If 22 > 0.25, then 2; else y 1.
8. Get the data point (Pi Pyz2, 3))
9. If the total amount of data generated is less than 7500, return to the Step 5; else break
In the data labeled 0, each modality contains features that can give a correct prediction, while in the data labeled or 2, need both modalities together to make the right predictions. To further
19

---

Under review as a conference paper at ICLR 2023
understand how uni-modal models give predictions in dataset that containing both uni-modal and paired features, we give the confusion matrix of the uni-modal model. As Table showss, the uni- modal model can give correct predictions for data labeled O, while for data labeled I or 2, it fails and gives a random predictions. Because for data labeled 1 or 2, we need to learn the relationship of the two modalities give the correct predictions.
shows,
In this subsection, we mainly discuss the synthetic multi-modal datasets and in Appendix A.7 we conduct various experiments real-world multi-modal datasets help understand the uni-modal features, paired features and cross-modal interaction in multi-modal training better.
Training settings on synthetics datasets. We use two layer MLP with ReLU as activation func- tion. As for hidden layer, we use 200 dimensions for multi-modal training and 100 dimensions for uni-modal training. We use SGD as the optimizer and the learning rate is 0.2. each iteration, use the whole training set compute the gradients. And we provide the code in supplement materials.
A.11 UNI-MODAL PERFORMANCE IN KINETICS-400
Kinetics-400 is dynamic dataset, because videos may be removed from YouTube. In this subsec- tion, we report the uni-modal performance of ours and Xiao et al. (2020)'s on Kinetics-400. As Table shows, we cannot reproduce their uni-modal performance and ours are lower than theirs. But we demonstrate that UMT outperforms AVSlowFast in Sec 1.3.1, which shows UMT's effec- tiveness.
Table 22: Uni-Modal Performance of ours and Xiao et al. (200
's on Kinetics-400
ours
Xiao et al.
2020
Uni-Audio
23.5
24.8
Uni-RGB (SlowFast-50)
74.9
75.6
Uni-RGB (SlowFast-100)
77.2
77.9
20

---

Under review as a conference paper at ICLR 2023
B PROOF
B. PROOF OF THEOREM 3.4
Theorem 3.4. In uni-modal ensemble, assume that the training procedure learns bmt features in modality x² and learns dmz features in modality x=a, We order the probability of uni-modal fea- tures (both x and m in decreasing order of predicting probability p. namely, P(1) Pia].-... In multi-modal training approaches, assume that the training procedure learns Kmi uni-modal features in modality xmm,, learns km2 uni-modal features in modality x=x,, , and learns kipa paired features feature predicting probability p(h1)..... ".......,.ppppp.pp. We provide three types of laziness:
with
(a. ) Quantity Laziness: kms
km2 + Kpa S min bmz).
(b.) Uni-modal Laziness: Each modality in pulti-modal training approaches performs worse than uni-modal training.
(c. Performance Laziness: Consider new testing point, for every 6 > 0, if the following inequality holds:
Σ p(h,) S 1E(
Σ
PI + A()).
VE(Bu + L,O +.aell
where A(8) = v ((pup bms- Kims + bmz - Kmz) log(1/0),, then with probability at least 1- 8, uni-modal ensemble outperform multi-modal training approaches concerning the loss on the testing point with probability.
We prove the theorem, which shows that naive joint training indeed suffers from overfitting issues, meaning that it learns less features compared to uni-modal ensemble.
Proof. We first introduce some additional notations used in the proof. We define the features trained in -nii-modal training as fi (x."),,... ...., Shnn (x="),, define the features trained in m² uni-modal training gi (xx")),.... ...., Ioaa (I. Therefore, there are in total 6ml + bmz features learned uni-modal ensemble, namely, fi (xx")) )................... Jbm (x:")) g. (z=")),... ),.... .......0 Ih (x:"2) Be- sides, We define the features trained in multi modal training approaches as fi(=")),. )...., fkm (x I1 ) Ilmna (2 ). h1 (x=H (I"",, ) hep. (x=",,""0)
When the context is clear, we omit the dependency of 2: 20 and denote them as St, Di, h, for simplicity. When the context clear, we abuse the notation r to represent arbitrary f. or h. The corresponding predicting probability of feature r is denoted p(r:). To summary, there are 6ml + bm2 features in uni-modal ensemble, km + km2 + Kpa features in multi-moda training approaches.
th
We first prove statement (a.), which claims that the number of features learned in multi-modal train- ing approaches are provably less than any of the number of features learned in uni-modal training. The proof depends on the following Lemma B.I
Lemma B.1. Assume there exists T features
If we replace one of the T features (without loss of generality, TT) with a more powerful feature r', where p(r') > p(rT), then the pre- dicting probability for each data point increases (where the probability is taken over the randomness of the training data).
We provide the proof of statements (a.): based Lemma B. We shall prove kim + kim + kpu < bml without loss of generality. Start from the features / Skm (x²)) which are common features in both multi-modal training approaches and Uni-modal training. Next step, add feature flm++1 in uni-modal approachesand 91 in multi-modal training approaches. Obviously, p(91)= > p(skmt+) due to the training priority (or multi-modal training approaches should learn fhmit1 instead of 91). Therefore, the predicting probability of multi- mudai training approaches is larger than uni-modal approaches.
Repeating the procedure by comparing 9: with faui+i and comparing h; with fkurthaa the pre- dicting probability of multi-modal training approaches is always larger than uni-modal approaches. Note that Im should be always larger than km + kmz, or the predicting probability of uni-modal
The prohahility is taken the randomness of the testing point
21

---

Under review as a conference paper at ICLR 2023
approaches would be smaller than multi-modal training approaches. At the end of the compar- ison, the predicting probability of multi-modal training approaches still larger than uni-modal approaches. This requires that uni-modal approaches should learn more features, which can be garded as uni-modal approacheslearns a features while multi-modal training approaches learns an empty feature. In conclusion, uni-modal approaches learns more features compared multi-modal training approaches, leading to 6ml kml + km2 + kipa
We next prove the statement (b.). The proof of (b.) is based (a.). We next only consider modal- ity x":, the proof for modality x= is similar. Note that the since the number of features learned in multi-modal training approaches is less than bmt. the number of features learned in xr must be less than 1m1 (Note that those features can be either paired feature or uni-modal feature, namely, Ji...., SK,, and h I.t Therefore, multi-modal training approaches learns less features com- pared to uni-modal approaches in modality "=, On the other hand, the predicting probability of features learned in multi-modal training approaches (fi..... . fkat and ......: Pig. considering only modality " for the paired feature) is less than that learned in uni-modal approaches (fi..... f...). because otherwise, uni-modal approaches will learn the features h instead of f. In conclusion, when considering only modality z= the number of features learned in proaches is less and its corresponding predicting probability is small. Therefore, each modality in multi-modal training approaches performs worse than uni-modal approaches.
training ap-
We finally prove the statement (c.). Recall that the loss - E, u(r;s) where u(re) = T(yr >
0) I[yr; < 0). Note that E(u(r:)) = ((r))) and [u(r:)] - 1. We derive that:
P
Σ u(fi)
Σ
u(gi)
Σ
w(h,)) S
Σ
u(fi)
Σ
u(gi)
VE kmnl
xe RRzel
iE Apul
EE [amil
vE(bna)
u(Si)=
Σ
u(gi)
Σ
u(hi)) 50
Ain < Kisbal
Mus tinsss
iE(kga)
=P
u(fi+ +
Σ u(gi)
Σ
u(ha))
1 +
E - E
1
2
2
Rai Shal
where E. -B(Enn Cishish u(fi) + Emere Cish hme
u(gi) Eie(s,)) u(h,)) Eie(s,)) p(hh))
p(h.)
Enmr eis p(fi) E
Hen Enacissh p(g:). Due to the training priority and the conclusion in (a.),
Σ
Σ
p(f)+ + Σ plg.).
P( PB S
1E (mm - -.bra
Therefore,
E
<
EiE p(h;)
Exe +.00m +bal Pii
18 Kpp
bml kim + bm2 kim)) log(1/8).
We next apply Hoeffding inequality on Equa-
tion B.
and derive that
P
Σ u(fi) Σ u(gi) Σ u(hs) <
Σ u(fi) EE Baal
Σ u((i)) vE((ma]
kmi)
sE[Rml
AE(Ap
< <expl- - F.2 22/88(e 18kppa + Dml kml + bm - kim2))
<8
To conclude, modal training approaches outperform uni-modal ensemble concerning the test- ing loss with probability at least - 8.
22

---

Under review as a conference paper at ICLR 2023
Compared to uni-modal ensemble, denote the additional paired feature are indexed by and the additional uni-modal feature in uni-modal ensemble are indexed by t. We have that:
P( (1(A(z) > 0) ((f())) < 00) Σ
(1()(())
> 0) - (z) < 0)) > 0)
-P [(J((1) > 0) Σ [()(t)) > 0)
Σ
Pi
Σ
p
Σ Pj
Σ :E|d
pil)
(1)
iE|c
jE(μ]
iE(c)
jE(M]
jE(0
S <exp(- 3 Pj - Σ
.E[v]
EE|0
Pj - -ie(e P: 2 u) the probability is done. Therefore, for a new data point, uni-modal ensemble can outperforms multi-modal training approaches with high probability.
Therefore, if )
Proof of Lemma B.7 We define 7|T] as the features .1...., 77--..,7--1 The proof is divided into two parts, depending on whether -ie H(;; * 0) is even or odd. We regard the term Ej I(r; * 0) as the number of effective features in r--r-- To simplify the discussion, we rescale r such that lyr| = (when r 0) or |yr| 0 (when = 0).
Case 1: When the number of effective features in "[-T] is even. (a. ) Eie yrs 2 adding Tr or r' does not alter the predicting probability, namely
y
rT +
Σ yr:
Σ iET
yr
2
+
p
y
TT
+
Σ
yr,
=0
Σ
yr's 22
EE[T-1)
<C|T-11
-E(T-
=P
y
r' +
Σ
yr:
> 0
Σ yra S
+ 2
P
y
r +
Σ
yr:
Σ yr, 22
1E[T-1]
KE|T-1j
=E[T-1]
KE|T-1]
(b. ) When the number of effective features in r[-1] is even, Eve(T-1 yr: 1
(c.
When Eierr- yrd 0. due to the assumption that p(r') > p(rT) and <(r) p(r)/c
adding r' helps increase the predicting probability compared to rT, namely
V
IT +
Σ yr 1E[T-1)
Σ T
yr
+
2
P
y
PT
+
Σ
yr
Σ iE !]
yra =0
1]
iE
iE
T-
>P
r' +
Σ yr
>
Σ
yra
=0
+
P
rr
yr;
Σ
yri
2
iE(T-1)
kE[T-1)
iE(T I]
7 1
The above inequality is derived based on the following equation:
P
PT +
Σ
yr:
>
Σ
yr,
=
+
P
PT +
Σ yr,
=0
Σ
yr: =0
11]
T|
iE|T-11
-E[T-1]
=P
yrr 0
Σ yr:
P
yrr
0
Σ
yrs =
[)]
[E[T-1]
=p(rT)= + セ [1 p(rr) - c(rr)]
- 1
(1-1c//cprr
Since we assume 1, the probability increases with probability p(rr).
Therefore, under the three conditions, adding r' increase the predicting probability more compared to rr. In summary, under case 1 (a-c), adding r' increase the predicting probability compared to rr-
23

---

Under review as a conference paper at ICLR 2023
Case 2: When the number of features in "[-T] is odd. The discussion in (b.) can be little bit more complex compared to case 1.
(a. ) If Eij urd 2 2, similar to case 1, adding rT or r' does not alter the predicting probabil-
ity, namely
P
+
Σ #E(T-1) 1)
0
Σ
2
+
P
+
Σ
0
yr's 22 2
2
ic
T-
VE
T-
SE 1|
=P
r' + Σ yr:
>
Σ yr: 2
+ P
rr +
yr;
Σ
yr
2
1E(T-1)
<E
#E|T
KET-11 1
(b. ) If Σ [T yr:
= 1: (b.1 ) If >
11 yr:
= -1:
P
PT +
Σ yri
> 0
Σ yru =1
+ 2
P
y
rT +
Σ yr: =0
Σ
yru yr==11
1E[T-1] 1]
<E|T-1]
1e|r-11 1]
<E|T-1]
=P
yrr
Σ yru = -1
+ 1 P
yrr =
Σ
yr, yn
1E/T-11
<ET-11
P
UrT
Σ yr, -1
2
<E|T-1]
1
2 p)((T))
(b.2) ) If Σ
EiE(T-1] Ur: = +1:
P
PT + Σ yr:
>
Σ yr, = 1
+ 2
P
rr + Σ yr:
= -0
iE(T-1)
1
iE TE/-11
iE[T-1]
=P
yrT 1
Σ yri =
+
yrt
Σ
yra =
T-
iE[T-1]
1
-(1 <(rr)) + ((T))
-1 -1 2c
p(rT).
Note that the probability of event (b.1) and the probability of event (b.2) satisfy the following equa- tion by Lemma B.2
Σ iE
yr: = 1 = cPP
Σ -E|T-1
yr
-
-1
(2)
Therefore, the total probability under case (b) is
1
yr = -1 + (1
1
((T)))
Σ yr; =1
2
2c
[1[T-1]
(E(T-1)
=P
yT =
[E
which is independent of p(rr). Therefore, adding IT or r' share the same predicting probability. (c.) When the number of effective features in r[-r] is odd, Eierr- yral # 0.
24

---

Under review as a conference paper at ICLR 2023
In summary, under case (a-c), adding r' do not decrease the predicting probability compared to TT-
The following lemmas used during the proof.
Lemma B.2. Consider features : -T-1, the following equation holds:
IP
yr: == clP
Σ 1E[T-1]
yr:
-
(3)
IE
Proof. can be proved to compare the events A (Eie(r-1) yr; = 1] and B = B
11
-1]. Every event in A has complementary event in B, namely,
yr; = 1 in yr:
in A
yr: in if yr: in A
yr; = 0 in yr, = in A
Comparing each event in A with its complementary event in B leads to the conclusion.
Combining case I and case 2 together leads to the final conclusion.
B.2 GENERALIZE THEOREM MORE MODALITIES
We next show that the results in Theorem 3.4 can be generalized to the regime of more modals.
Specifically, we assume a T-modal regimes, and denote the modals xi E [T]. In uni-modal pre-training approaches, let bm, denote the number of returned features in modal i. In multi-modal joint training, let km. denote the number uni-modal features for modal i, and kpn denote number of returned paired features. We derive the following Theorem B.3 for the multi-moda regimes.
Theorem B.3. Based the above notations, we provide three types laziness from three perspec- tives:
(a. Quantity Laziness:
Σ, kim + Kpa S ming (bm,).
(b. ) Uni-modal Laziness: Each modality in multi-modal training approaches performs worse than uni-modal training.
(c. Performance Laziness: Consider a new testing point, then for every 5 0, if the following inequality holds:
Σ p(h)) sElkyal
Σ
PM
A(8),
defmming&ben (bmn H+ bm;l
where A(5) 88((ipa Eylbm; km,)) log(1/5) then with probability at least 1 6,
uni-modal ensemble oulperform multi-modal training approaches concerning the loss the testing point with probability.
B.3 PROOF OF THEOREM 3.5
Theorem 3.5. Denote the paired features by hi.... hy with corresponding predicting probability
p(i)).... ((h)),...p((b))) Assume that distillation can boost the training priority by p > 0. If there exists paired features whose predicting probability exceeds the boosting probability pº, namely, the set S is not empty:
S (h+ p(h4) > +
Then UMT helps uni-modal feature
feature learning and can also learn easy-to-learn paired features.
The prohability is taken the randomness of the testing point
25

---

Under review as a conference paper at ICLR 2023
Table 23: Dataset used in Example [8.4
means the feature is larger than and means the feature is less than zero. We denote the predicting probability by and the rectified probability (due pushing force) by p'.
fi
f2
f3
92
93
h
y
p
0.20
0.10
0.05
0.15
0.08
0.02
0.28
p'
0.35(1)
0.25(1)
0.20(1)
0.32(1)
0.23(1)
0.17(1)
0.28
/
data a
+
+
+
data b
0
0
data c
+
+
0
+
+
0
data d
+
0
-1
Proof. The core of Theorem 3.5 is clarify the training priority. We revisit the notations of Theo- rem 3.4 follows without further clarification. At the end of the training, uni-modal ensemble learn bm + m2 useful features, namely, /1...., Jom 91, , ... Pour And multi-modal training approaches learn km + km2 + kpa features: fi...., fkm 91 91...... .k... hi...., many empty features e; in the model due the initialization.
hkpp We note that there are still
By distillation, the model learns the features according to the new priority. Since the set S is not empty, there exists paired features that is learned before the empty features. By distillation, the model would learn all the useful features that appear in uni-modal approaches, as well as those features in set S. Therefore, UMT outperforms uni-modal ensemblewhen there exists useful paired features.
B.4 A CONCRETE EXAMPLE TO ILLUSTRATE THEOREM 3.4
We provide concrete example to better illustrate the Modality Laziness issues. For Exam- ple [.4 we aim to show the Modality Laziness issues. For Example B.5 we aim to show the role of the pushing force.
Example B.4. Consider modality nh with features fi, f2. fa (corresponding prediction probability p 0.2, 0.1, 0.05). and modality tha with features 9l, 92- ya (corresponding prediction probability p = 0.15, 0.08, 0.02). We show the dataset in Table [3]ddd and aim to minimize the training loss to zero.
In uni-modal approaches, we learn features S1, f2 and f3 on modality x² (similarly, 91- 92. and I3 on modality =m))). Therefore, we learn features fI, f2, fax 91.92: 93 in uni-modal ensemble. In multi-modal training approaches without paired feature, we can only learn three features fI, f2. 92 due to the training priority fi > 91 is caused by modality laziness.
92 > fs 93 (decreasing order in p). This phenomenon
We next consider another paired feature h with probability p = 0.28. Under the case, multi- training approaches only learn two features i and /1- Therefore, when is not powerful enough, uni-modal ensemble outperforms multi-modal training approaches.
Example B.5. We follow the notations and dataset in Example B.4 By applying the pushing force, assume that each probability of uni-modal feature boosts 051,, which changes the training priority to fi > 91 > h > f2 > 92 > J3 > 93 (decreasing order in z/). Therefore, multi-modal training approaches (with pushing force) learns f1. h. As comparison, multi-modal training approaches (without pushing force)
only learn f1, h. Therefore, pushing force helps learn more features. We additionally remark that we only consider the training error in this example, and there might be other penalties in practice (e.g., distillation loss).
26


Reviews:
[{"review_id": "ccdGXCqRJA", "reviewer": "Reviewer_fVP6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackled the modality laziness in current multi-modal fusion methods and suggested leveraging the uni-modal feature in two ways. Depending on the importance of paired features (maybe multi-modal information), simple averaging uni-modal features or knowledge distillation from strong uni-modal models. Also, the generalization degradation is described in theoretical perspective. ", "summary_of_the_review": "Please handle the weakness in the author's response", "strength_and_weaknesses": "S1) a good motivation for the features yielded from multi-modal fusion modules and the motivation is shown experimentally.\nS2) Mathematical description for the generalization degradation\n\nW1) the detail of the uni-modal model is missed in UMT. If the uni-modal model is such a large network, UMT has benefits in terms of generalization. \nW2) Fewer baselines. Are there no works or techniques to preserve uni-modal information?\nW3) The author insists that the existing fusion modules delete the important uni-modal information somewhat. In UMT, the knowledge distillation is performed before the fusion module. Following the author's insistence, the distilled knowledge can also be corrupted after fusion. What is the reason for the design choice?\nW4) To select UMT or UME, the testing performance is required. It makes the proposed methods less mature. Is there any automatic rule for this? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b8oVPdO30X", "reviewer": "Reviewer_5mXz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with multi-modal joint training methods on deep neural networks.\nRelated to this topic, several works have reported that the best uni-modal networks outperform the multi-modal networks even though the multi-modal networks receive more information.\nThis work is focused on this topic, and the main contributions of this paper are two-fold: (1) to propose an improved methodology with experimental supports on audio-video data for sound classification (VGG-Sound), audio-video data for action recognition (UCF-101, Kinetics-400), front-rear view of object data (ModelNet40), (2) to investigate theoretical results on characteristics among uni-modal features (learned from uni-modal training), and paired features (only learned from cross-modal interaction).\n", "summary_of_the_review": "The main contribution, a method of combination of Uni-modal Ensemble and Uni-modal Teacher, seems practically utilizable and showing better performance for multi-modal joint training on deep neural networks. \nAlso, I think it is interesting idea to leverage distillation for generalized setting of multi-modal joint training. \nHowever, the writing is confusing and not enough clear to be reproducible.\n\n\n\n\n\n", "strength_and_weaknesses": "*Strength\n- To identify interesting conceptualization on uni-modal features and paired features\n- To provide experimental evidences (mostly shown in Appendices) to justify their arguments and choices on this work\n- To explain theoretical aspects to characterize modality laziness\n\n*Weaknesses\n- It seems that some terms in the paper are NOT so clear: what is the exact definition of modality laziness? Why should we utilize new term? What are the discriminating points for it? \n- This paper is hard to follow due to lack of explanation for core parts such as model architectures of uni-modal teacher, loss functions with λ_task, λ_distill, λ_distill in Algorithm 1 or the citation of reference works, which also needs for reproducibility.\n- The organization of this manuscript makes readers confusing. It seems that theoretical part (Section 3.4) and the others are totally separated. Uni-modal features and paired features also are redefined in Section 3.4. On the other hand, there is no experimental result related to Section 3.4 in main body of the manuscript (but in Appendices). \n- Even though related work Section is not bad, it would be better to compare one of the recent works [1], which deeply related to this work.\n\n[1] Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably), ICML 2022 ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NMyEb84sRyS", "reviewer": "Reviewer_DNPE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the insufficiency phenomenon of uni-modal feature learning in supervised multi-modal learning problem. The authors identify that the recent late-fusion method suffers from insufficient learning issue, called as Modality Laziness. To understand this phenomenon, the authors conduct a series of experiments and make a theoretical analysis. The authors propose two late fusion learning methods, Uni-Model Ensemble and Uni-Model Teacher, which work better in different cases according to the distribution of uni-modal and paired features. A simple guiding strategy is also provided. Experiments show the advantages of the proposed method in several multimodal datasets. \n\n--- post-rebuttal ---\\\nThanks for the authors' response. After reading the response and other reviewers' comments, my concerns are addressed. I keep my score.", "summary_of_the_review": "Overall, I think this work is well executed, with a strong motivation and good theoretical and empirical analysis. In terms of existing work, I'd like to see how this work compares with multimodal deep learning [Ngiam et al.]. ", "strength_and_weaknesses": "Strength: \n\n-The paper discovers an interesting phenomenon of Modality Laziness, and provides an in-depth understanding in both empirical and theoretical aspects. \n\n-The paper is well motivated. The analytical experiment in Table 1 really shows the severeness of existing supervised multimodal learning. \n\n-I appreciate the efforts that the authors try to explain the key concepts in an intuitive way. For example, the figure 3 is helpful. \n\n-The results compared with several existing methods are competitive. The proposed solution is simple but effective. \n\n \n\nWeakness \n\n-There is one method called “bimodal deep autoencoder” [Ngiam et al.], which can also be applied for joint feature extraction, then used for supervised learning. The discussion along this line is missing. Since  the motivation of this work is address Modality Laziness, the auto-encoder used in [Ngiam et al.] also has the capability to mitigate the insufficient learning. A comparison with this work is strongly recommended. \n\n[Ngiam et al,] Jiquan Ngiam et al., Multimodal Deep Learning, ICML 2011 \n\n-The term of “naive multimodal training” appears multiple times across the paper, and its suffers from Modality Laziness. While the formal definition of this term is unclear to me. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ATqdeV-sFns", "reviewer": "Reviewer_AaMz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\nThis paper proposes to care about the learning of unimodal features in the setting of multimodal fusion. They find out that one modality may strongly dominate the multimodal learning than other modalities on some tasks, like video classification and action recognition. This observation is consistent of previous related works, like greedy nature of multimodal DNN (Wu et al, 2022), under-optimized unimodal representations caused by strong dominated modality (Peng et al, 2022). They argue that previous works are complex to implement and are empirically inferior on some evaluation datasets (UCF101 and ModelNet40). Instead, this paper introduces Uni-Modal Teacher (UMT) to distill pretrained\nunimodal features to help learn corresponding unimodal features counterparts in the multimodal model. Furthermore, a trick is used to decide when to use the joint multimodal training (Uni-Modal Teacher, UMT) or just use the simple averaging predictions of unimodal models alone (Uni-Modal Ensemble, UME).\n", "summary_of_the_review": "See Strong&Weak", "strength_and_weaknesses": "Strong & Weak:\nS1: The writing is easy to follow. A reproducibility code is given. Better experimental results are achieved than the complex late-fusion models. \nS2: The architecture of Uni-Modal Teacher is straightforward and is empirically better than classic distillation methods including soft-label.\n\nW1: Although this paper cares about the weakness of multimodal learning and focuses on the importance of unimodal feature learning, assuming that unimodal priors are important and meaningful like previous works (Wu et al, 2022; Peng et al, 2022), it may be better to go further to design a unified, principal framework which can also subsume the settings where unimodal priors are not essential, for example on the visual question answer task. In other words, such a unified framework is agnostic to the unimodal priors and works well for both essential priors (including video classification and action recognition tasks) and meaningless priors (including visual question answer task). Perfectly, one does not need to choose which style of multimodal learning should be adopted to solve their own interest tasks at hand.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject