The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation
Authors: Zihui Xue
Zhengqi Gao
Sucheng Ren
Hang Zhao
Venue: ICLR 2023 notable top 5% (2023)
Keywords: multimodal learning
knowledge distillation

Abstract: Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications. To achieve knowledge transfer across modalities, a pretrained network from one modality is adopted as the teacher to provide supervision signals to a student network learning from the other modality. In contrast to the empirical success reported in prior works, the working mechanism of crossmodal KD remains a mystery. In this paper, we present a thorough understanding of crossmodal KD. We begin by providing two failure cases and demonstrate that KD is not a universal cure in crossmodal knowledge transfer. We then present the modality Venn diagram to understand modality relationships and the modality focusing hypothesis revealing the decisive factor in the efficacy of crossmodal KD. Experimental results on 6 multimodal datasets help justify our hypothesis, diagnose failure cases, and point directions to improve crossmodal knowledge transfer in the future.

Published as a conference paper at ICLR 2023
THE MODALITY FOCUSING HYPOTHESIS: TOWARDS UNDERSTANDING CROSSMODAL KNOWLEDGE DISTIL- LATION
Zihui Xue" Zhengqi G Sucheng Ren" Hang Zhao 1
1 The University of Texas Austin Massachusetts Institute of Technology
3 South China University of Technology 4 Tssnngua University, Shanghai Qi Zhi Institute
ABSTRACT
Crossmodal knowledge distillation (KD) extends traditional knowledge distillation to the area of multimodal learning and demonstrates great success in various applications. To achieve knowledge transfer across modalities, a pretrained network from one modality is adopted as the teacher to provide supervision signals to a student network learning from another modality. In contrast to the empirical success reported in prior works, the working mechanism of crossmodal KD remains a mystery. In this paper, we present a thorough understanding of crossmodal KD. We begin with two studies and demonstrate that KD is not universal cure in crossmodal knowledge transfer. We then present the modality Venn diagram (MVD) understand modality relationships and the modality focusing hypothesis (MFH) revealing the decisive factor in the efficacy of crossmodal KD.. Experimental results 6 multimodal datasets help justify our hypothesis, diagnose failure cases, and point directions to improve crossmodal knowledge transfer in the future.'
INTRODUCTION
Knowledge distillation (KD) an effective technique to transfer knowledge from one neural network to another (Wang & Yoon, 2021; Gou et al., 2021). Its core mechanism is teacher-student learning framework, where the student network trained to mimic the teacher through loss. The loss function, initially proposed by (Hinton et al., 2015) as the KL divergence between teacher and student soft labels, has been extended in many ways (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019; Park et al., 2019; Peng et al., 2019; Tian et al., 2019). KD has been successfully applied to various fields and demonstrates its high practical value.
The wide applicability of KD stems from its generality: any student can learn from any teacher. To be more precise, the student and teacher network may differ in several ways. Three common scenarios are: (1) model capacity difference: Many works (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019; Park et al., 2019; Peng et al., 2019) on model compression aim to learn a lightweight student matching the performance of its cumbersome teacher for deployment benefits. (2) architecture (inductive bias) difference: As an example, recent works (Touvron et al., 2021; Ren et al., 2022: Xianing et al., 2022) propose to utilize a CNN teacher to distill its inductive bias transformer student for data efficiency. (3) modality difference: KD has been extended to transfer knowledge across modalities (Gupta et al., 2016; Aytar et al., 2016; Zhao et al., 2018; Garcia et al., 2018; Thoker & Gall, 2019; Ren et al., 2021; Afouras et al., 2020; Valverde et al., 2021; Xue et al., 2021), where the teacher and student network come from different modalities. Examples include using an RGB teacher to provide supervision signals to student network taking depth images as input, and adopting an audio teacher to learn a visual student, etc.
Despite the great empirical success reported in prior works, the working mechanism of KD is still poorly understood (Gou et al., 2021). This puts the efficacy of KD into question: Is KD always
'Zihui. Zhenggi, and Sucheng contribute equally. Work is done during internship Shanghai Qi Zhi Institute. Corresponn to hangzhao@ mail, www.l.tiinguuaccommmm www.nnha.acauucccn.. Our code is available at htpp://itthh.ccm//iiinc//HHH

---

Published as a conference paper at ICLR 2023
efficient? If not, what is good indicator of KD performance? A few works (Cho & Hariharan, 2019; Tang et al., 2020; Ren et al., 2022) are in search for the answer in the context of model capacity difference and architecture difference. However, the analysis for the third scenario, KD under modality difference or formally crossmodal KD, remains an open problem. This work aims to fill this gap and for the first time provides comprehensive analysis of crossmodal KD. Our major contributions are the following:
We evaluate crossmodal KD on a few multimodal tasks and find surprisingly that teacher performance does not always positively correlate with student performance.
To explore the cause of performance mismatch in crossmodal KD, we adopt the modality Venn diagram (MVD) to understand modality relationships and formally define modality- general decisive features and modality-speciffe decisive features.
We present the modality focusing hypothesis (MFH) that provides an explanation of when crossmodal KD is effective. We hypothesize that modality- gentral decisive features are the crucial factor that determines the efficacy of crossmodal KD.
We conduct experiments on 6 multimodal datasets (i.e., synthetic Gaussian, AV-MNIST, RAVDESS, VGGSound, NYU Depth V2, and MM-IMDB). The results validate the proposed MFH and provide insights on how to improve crossmodal KDD.
2 RELATED WORK
2.1 UNIMODAL KD
KD represents a general technique that transfers information learned by a teacher network to a student network, with applications to many vision tasks (Tung & Mori, 2019; Peng et al., 2019; He et al., 2019; Liu et al.. 2019). Despite the development towards better distillation techniques or new application fields, there is limited literature (Phuong & Lampert, 2019; Cho & Hariharan, 2019; Tang al.. 2020: Ren et al., 2022: 2023) on understanding the working mechanism of KD. Specifically, Cho & Hariharan (2019) and Mirzadeh et al. (2020) investigate KD for model compression, i.e., when the student and teacher differ in model size. They point out that mismatched capacity between student and teacher network can lead to failure of KD. Ren et al. (2022) analyze KD for vision transformers and demonstrate that teacher's inductive bias matters more than its accuracy in improving performance of the transformer student. These works provide good insight into understanding KD, yet their discussions are limited to unimodality and have not touched on KD for multimodal learning.
2.2 CROSSMODAL KD
With the accessibility of the Internet and the growing availability of multimodal sensors, multimodal learning has received increasing research attention (Baltrušaitis al., 2018). Following this trend, KD has also been extended to achieve knowledge transfer from multimodal data and enjoys diverse applications, such as action recognition (Garcia et al., 2018: Luo et al., 2018; Thoker & Gall, 2019), lip reading (Ren et al., 2021; Afouras et al., 2020), and medical image segmentation (Hu et al., 2020; Li et al., 2020). Vision models are often adopted as teachers to provide supervision to student models of other modalities, e.g., sound (Aytar et al., 2016; Xue al., 2021), depth (Gupta et al., 2016; Xue et al., 2021), optical flow (Garcia et al., 2018). thermal (Kruthiventi et al., 2017), and wireless signals (Zhao et al., 2018). Although these works demonstrate potentials of crossmodal KD, they are often associated with a specific multimodal task. An in-depth analysis of crossmodal KD is notably lacking, which is the main focus of this paper.
2.3 MULTIMODAL DATA RELATIONS
There is continuous discussion on how to characterize multimodal (or multi-view) data relations. Many works (Tsai al., 2020; Lin et al., 2021: 2022) utilize the multi-view assumption (Sridharan & Kakade, 2008), which states that either view alone is sufficient for the downstream tasks. However, as suggested in (Tsai et al., 2020), when the two views of input lie in different modalities, the multi-view assumption is likely to fail. ? In the meantime, few works multimodal learning (Wang et al.,
"A detailed comparison of our proposed MVD with the multi-view assumption is presented in Appendix C.
2

---

Published as a conference paper at ICLR 2023
2016; Zhang et al., 2018; Hazarika et al., 2020; Ma et al., 2020) indicate that multimodal features can be decomposed as modality-general general features and specific features in each modality. Building upon these ideas, in this work, we present the MVD to formally characterize modality relations.
In addition, the importance of gennerl information has been identified in these works, yet with different contexts. In multi-view learning, (Lin et al., 2021; 2022) consider shared information between two views as the key to enforce cross-view consistency. To boost multimodal network performance and enhance its generalization ability, (Wang et al., 2016; Zhang et al., 2018; Hazarika et al., 2020; Ma et al., 2020) propose different ways to separate specific information. For semi-supereised multimodal learning. (Sun et al., 2020) aims maximizing the mutual information shared by all modalities. To the best of our knowledge, our work is the first to reveal the importance of modality- general information in crossmodal KD.
and modality-
ON THE EFFICACY OF CROSSMODAL KD
First, we revisit the basics of KD and introduce notations used throughout the paper. Consider a supervised K-cllas classification problem. Let fe, (x) E R and fo, (x) E RK represent the output (i.e., class probabilities) of the student and teacher networks respectively, where (0.. 02) } are learnable parameters. Without loss of generality, we limit our discussion within input data of two modalities, denoted by x and x for modality a and b, respectively. Assume that we aim to learn a student network that takes x² input. In conventional unimodal KD, the teacher network takes input from the same modality as the student network (i.e., x²). ). The objective for training the student is:
pLtask + (1 -)
where Lrask represents the cross entropy loss between the ground truth label y {0, 1, 1.... K 1} and the student prediction fo. (xx). Lud represents the KL divergence between the student prediction fo, (x*) and the teacher prediction fo, (x"). and p E [0, 1] weighs the importance of two terms Lrask and Lid (i.e,, driving the student to true labels or teacher's soft predictions).
Crossmodal KD resorts to teacher from the other modality (i.e., x") to transfer knowledge the student. Eq. (1) is still valid with a slight correction that the KL divergence term is now calculated using fo, (x*) and fe, (x*). In addition, there one variant (or special case) of crossmodal KD, where a multimodal teacher taking input from both modality and is adopted for distillation, and Led is now KL divergence term between fo. (x*) and fe, (x".
We first present a case study on the comparison of crossmodal KD with unimodal KD. Consider the special case of crossmodal KD where multimodal teacher is adopted. Intuitively, adopting a multimodal teacher, which takes both modality a and as input, can be beneficial for distillation since: (1) a multimodal network usually enjoys a higher accuracy than its unimodal counterpart (Baltrušaitis et al., 2018), and more accurate teacher ought to result in a better student; (2) the complementary depeedtt information brought by a multimodal teacher can enrich the student with ad- ditional knowledge. This idea motivates many research works (Luo et al., 2018; Hu et al., 2020; Valverde et al., 2021) to replace a unimodal teacher with a multimodal one, in an attempt to improve student performance. Despite many empirical evidence reported in prior works, in this paper, reflect on this assumption and ask the question: Is crossmodal KD always effective?
Table 1: Evaluation of unimodal KD (UM-KD) and crossmodal KD (CM-KD) on AV-MNIST and NYU Depth V2. 'Mod,' is short for modality, 'mIoU' denotes mean Intersection over Union, and A, I, RGB, D represents audio, grayscale images, RGB images and depth images, respectively.
AV-MNIST
NYU Depth V2
Teacher Acc. (%)
Student Acc. (%)
Teacher
Student mIoU/ (5)
Mod.
Mod.
Mod.
mloU (%)
Mod.
No-KD
68.36
RGB
46.36
UM-KD
A
84.57
70.10
RGB
46.36
RGB
48.00
CM-KD
I+A
91.61 (1)
69.73 (J)
RGB +
51.00 (1)
RGB
47.78 (.)
Table provides two counterexamples for the above question on AV-MNIST and NYU Depth V2 data. The goal is improve an audio student using KD on AV-MNIST and improve an RGB model
3

---

Published as a conference paper at ICLR 2023
on NYU Depth V2. From Table I, we can see that accurate multimodal network does not serve as a better teacher in these two cases. For AV-MNIST, while the audio-visual teacher itself has much higher accuracy than the unimodal teacher (i.e., +7.04%), the resulting student is worse (i.e., 00.7%% instead. Similarly, the great increase in teacher performance (i.e. +4.64%) does not translate to student improvement (i.e., -0.22%) for NYU Depth V2. These results cast doubt on the efficacy of crossmodal KD. Even with the great increase in teacher accuracy, crossmodal KD fails to outperform unimodal KD in some cases. Contradictory to the previous intuition, teacher performance seems not reflective of student performance. Inspired by this observation, our work targets on exploring the open problem: What is the fundamental factor deciding the efficacy of crossmodal KD?
4 PROPOSED APPROACH
4.1 THE MODALITY VENN DIAGRAM
To study crossmodal KD, it is critical to first establish an understanding of multimodal data. Before touching multimodal data, let us fall back and consider unimodal data. Following causal perspective (Schölkopf et al., 2012) (l.e. features cause labels), we assume that the label is determined by a subset of features in x (or x"); this subset of features are referred to as decisive features for modality a (or modality b) throughout the paper. For instance, colors of an image help identify some classes (e.g., distinguish between zebra and horse) and can be considered as decisive features.
When considering multimodal data, input features of the two modalities will have logical relations such as intersection and union. We describe the modality Venn diagram (MVD) below to characterize this relationship. Stemming from the common perception that multimodal data possess shared information and preserve information specific to each modality, MVD states that any multimodal features are composed of modality-genen general features and poddality-spcciffe features. Decisive features of the two modalities are thus composed of two parts: (1) coudality-genence decisive features and (2) modality-specifff decisive features, these two parts of decisive features work together and contribute to the final label y. Fig. left shows an example of video-audio data pair, where the camera only captures one person due to its position angle and the audio is mixed sounds of two instruments. Fig. right illustrates how we interpret these three features (i.e., modality-genere decisive, visual modality- specific decisive and audio modality -specffcc decisive) at the input level.
visual modality-quoifft
modallty-general
audio modality-pcciffee
visual input
audio input
playing
instruments
Iobely
Figure I: An input video-audio pair can be regarded as composed of modality-genere features and modality-speciff features in the visual and audio modality. For instance, the man playing violin on the right is not captured by the camera and hence its sound (marked in red) belongs to audio modelity-specifie information.
Next, we propose a formal description of MVD to capture the generating dynamics of multimodal data. Let 2 Lbb and y be the feature space of modality a, modality b, and the label space, respectively, and (x", x5, y) be a pair of data drawn from a unknown distribution P over the space X x X x y.
MVD assumes that (x", xb, y) is generated by quadruple (z zb z ".)4) E zsa zb 2 x y following the generation rule:
MVD GENERATION RULE:
x" = (z",, za =
12
z T E 2a
n0 x 2
(2)
x² gz²
- 2 E bb - z x z
where collectively, g"()) 2d x denotes an unknown generating function, if we adopt the notation E {a, b). To complete the MVD, another linear decision rule should be included.
Note that we even give preferable treatment to crossmodal KD: we take multimodal network as teacher, and this teacher achieves higher accuracy than teacher typically used in crossmodal KD.
4

---

Published as a conference paper at ICLR 2023
Specifically, the following equation:
MVD DECISION RULE: M arg max [Softmax (aattmxx(""""" [N)z = arg max [W = (3)
is assumed to hold for any (z 2bb z" ,y,, where we slightly abuse arg max[-] and here it means the index of the largest element in the argument. In essence, MVD specifies that x" generated based on a modality vector zd (generation rule), and that z" sufficient to linearly determine the label y (decision rule).
decisive feature vector z" and modality-general general decisive feature
We proceed to quantify the correlation of modality- moddality-geneall decisive features and modality-specifet decisive features. Let 2 S Id and 2 S IRd, so that 2" S Rd',, where du d + dg- We denote ratio 7 do/ (do dan + dab) E [0, which characterizes the ratio of modality- general decisive features over all decisive features. Similarly, a daa/(do + dga + dsa) and 3 ddb/ + deaa + dgb) denotes the proportion of modality-specije decisive features for modality a and b over all decisive features, respectively, and we have a + 3 +
4.2 THE MODALITY FOCUSING HYPOTHESIS
Based MVD, we now revisit our observation in Sec. 3 (i.e., teacher accuracy is not key indicator of student performance) and provide explanations. First, teacher performance is decided by both modality-general decisive and modality-specific decisive features modality a. terms of student performance, although modality-specie teacher, they can not instruct the student since the student only sees modality On the other hand, modality- modality-general decisive features are not specific to modality and could be transferred to the student. Coming back to the example in Fig. 1, if an audio teacher provides modality-specific information (i.e., the sound colored in red), the visual student will get confused this information (i.e., playing violin) is not available in the visual modality. On the contrary, modality-genen general information can be well transferred across modalities and facilitates distillation as the audio teacher and visual student can both perceive the information about the left person playing guitar. This motivates the following modality focusing hypothesis (MFH).
decisive features in modality a are meaningful for the
The Modality Focusing Hypothesis (MFH), For crossmodal KD,, distillation performance is depen- dent on the proportion of mondality-geneneee decisive features preserved in the teacher network: with larger the student network is expected to perform better.
The hypothesis states that in crossmodal knowledge transfer, the student learns to "focus on" modality- general decisive features. Crossmodal KD is thus beneficial for the case where y is large (i.e., multimodal data share many label-relevam, information). Moreover, it accounts for our observation that teacher performance fails to correlate with student performance in some scenarios - When a is large and 7 is small, the teacher network attains high accuracy primarily based on modality-specinee information, which is not beneficial for the student's ' learning process.
To have an intuitive and quick understanding of our hypothesis, here we present two experiments with synthetic Gaussian data. More details can be found in Sec. 5.2. As shown in Fig. 2, we start from the extreme case where two modalities do not overlap, and gradually increase the proportion of modality- general decisive features until all decisive features are shared by two modalities. We observe that crossmodal KD fails to work when x" and x" share few decisive features (i.e., 7 is small) since modality -ppecifcc decisive features in modality are not perceived by the student. As
7
Ascuncyr
X
b
b
a
B
B
3
Tracker
Stukan
3
8 1=
y
a=B
00
0.2
0.6
0,5
05
1.0
y -
y=1
Figure 2: An illustration of MFH with synthetic Gaussian data. Teacher modality is x" and student modality is x². We plot the confidence interval of one standard deviation for student accuracy. With increasing y, crossmodal KD becomes more effective.
5

---

Published as a conference paper at ICLR 2023
Teacher No
Accomeyy
X2
X
X
a
x
X
7
X
B 00 y=1
0.8, B
00 01
02
0.5
0.5
055
06
03
7 - 0.2
Figure 3: With increasing a (i.e., decreasing 7), the teacher improves its prediction accuracy but the student network fails benefit from KD. See the caption of Fig. 2 for more explanations.
Y gradually increases, crossmodal KD becomes more effective. For the where all decisive features possess in both modalities, the student gains from teacher's knowledge and outperforms its baseline by 2.1%. Note that the teacher accuracy does not vary much during this process, yet student performance differs greatly.
Fig. 3 illustrates the reverse process where modality-specifte decisive features in modality a gradually dominate. With increasing a, the teacher gradually improves since receives more codaalty-speccfffe decisive features for prediction. However, the student network fails benefit from the improved teacher and performs slightly worse instead. Clearly, teacher performance is not reflective of student performance this case. These two of experiments help demonstrate that teacher accuracy does not faithfully reflect the effectiveness of crossmodal KD and lend support to our proposed hypothesis.
Apart from the two intuitive examples, below we provide a theoretical guarantee of MFH in an analytically tractable case, linear binary classification. Formally, considering an infinitesimal learning rate which turns the training into continuous gradient flow defined on a time parameter & [0, +oo) (Phuong & Lampert, 2019). If n data are available, which are collectively denoted as Z E IRdu xn, we have the following theorem to bound the training distillation loss with y,
Theorem 1. (Crossmodal KD in linear binary classification). Without loss of generality, we assume (+) X + y and fe,(.) (-) X + y. Suppose max [| ZZ [I(Z=Ze.))=ll S X always holds for both u a or b, and g" are identity functions. If there exists (e, 6) such that
Pr [|| Z Z b. Z--1'z' S (1- 21
(4)
Then, with an initialization at t satisfying Pelis (0,()) S g. we have, at least probability 8:
Rdit (0, (t +00)) S n 1-e- In
where XX2 + 1)(1 - 7) and Reis (0.) is the empirical risk defined by KL divergence (corresponding to Eq. (1) when 0):
a (0 x
1 - (0,
R (t) Σ - 0 x²)
- o x()
(6)
(0f x
1 1 (0f
See Appendix for the omitting proof, several important remarks, and future improvements.
4.3 IMPLICATIONS
We have presented MVD and MFH. Equipped with this new perspective of crossmodal KD, we discuss their implications and practical applications in this section.
Implication. For crossmodal KD, consider two teachers with identical architectures and similar performance: Teacher (a) makes predictions primarily based modality -enerrall decisive features while teacher (b) relies more on modality- specific decisive features. We expect that the student taught by teacher (a) yields better performance than that by teacher (b).
The implication above provides us with ways to validate MFH. It also points directions to improve crossmodal KD We can train teacher network that focuses
modality -eeeerralggge decisive

---

Published as a conference paper at ICLR 2023
features for prediction. Compared with regularly -trinee teacher, the new teacher is more modality- general (i.e., has a larger 7) and thus tailored for crossmodal knowledge transfer.
Note that: Firstly, identical architectures and performance of the two teachers stated here for a fair comparison. Similar performance of the two teachers translate to similar ability to extract decisive features for prediction, and thus the only difference lies in the amount of modality-genens decisive features. This design excludes other factors and helps justify that the performance difference stems from y. In fact, we observe that even with inferior accuracy than teacher (b), teacher (a) still demonstrates better crossmodal KD performance in experiments. Secondly, the main focus of this paper to present the MFH and validate with theoretical analysis, synthetic experiments, and evidence from experiments conducted on real-world multimodal data. Contrary to the common belief, we detach the influence of teacher performance in crossmodal KD and point out that modality-general decisive features are the key. Developing methods to separate modality-gene general/specific decisive features from real-world multimodal data beyond the scope of this paper and left as future work.
EXPERIMENTAL RESULTS
5.1 EXPERIMENTAL SETUP
To justify our MFH, we conduct experiments on 6 multimodal datasets (synthetic Gaussian, AV- MNIST, RAVDESS, VGGSound, NYU Depth V2, and MM-IMDB) that cover a diverse combination of modalities including images, video, audio and text. In essence, we design approaches to obtain teachers of different y and perform crossmodal KD to validate the implication presented in Sec. 4.3.
We consider four different ways to derive teacher network that attends more or fewer modality- general decisive features than a regularly-trained teacher: (1) For synthetic Gaussian data, since the multimodal data generation mechanism is known, we train a modality- geeeral teacher on data with only ponallty-ggnered decisive features preserved and other channels removed; (2) For NYU Depth V2, we notice that RGB images and depth images share inherent similarities and possess identical dimensions, allowing them to be processed using a single network. Therefore, we design a modality-general teacher (i.e., has a larger y than a regularly-traine -trinnd teacher) by following the training approach in (Girdhar et al., 2022); (3) For MM- data, follow the approach in (Xue et al., 2021) to obtain a multimodal teacher which is more modality-specific (i.e., has a smaller 7) than the regularly-traines teacher; (4) For the other datasets, feature importance (Breiman, 2001; Wojtas & Chen, 2020) to rank all feature channels according to decisive information. With a sorted list of all features, we train
design an approach based on
the amount of modality-geners modality-gened teacher by only keeping feature channels with large salience values and a modality- specific teacher by keeping features with small values. summary, wide range of approaches and tasks are considered to justify MFH. See Appendix B for detailed setups and some results.
5.2 SYNTIETIC GAUSSIAN
Table 2:
Results on synthetic Gaussian data. Compared with a regular teacher, the modality-general teacher has downgraded performance yet leads to student with increasing accuracy.
Regular Teacher
genneal Teacher
Student Acc. (%)
Acc. (%)
Acc. (%)
No-KD
Regular-KD
Modality-general-KC
0.25
89.70
1.0
64,84 (L)
59.29
59,46
61.01 (t)
0.50
89.62
1.0
73.41 ())
67.64
67.
70.25 (f)
0.75
89.70
1.0
79.41 (L)
76.46
76.53
77.70 (?)
We extend the Gaussian example in (Lopez-Paz et al., 2015) to multimodal scenario (see Ap- pendix B.2 for details). To validate MFH, we train two teacher networks: (1) a regular teacher is trained on data with all input feature channels; (2) modality-general teacher is trained on data with uoddaliy-ggeeera decisive features preserved and other channels removed, thus this teacher has 7 = 1. We experiment with different data generation methods, and the regular teachers have different ? correspondingly. Crossmodal KD results obtained by these two teachers are presented in Table 2. We observe considerable performance degradation (larger than -10% accuracy loss) of the modality-
7

---

Published as a conference paper at ICLR 2023
general teacher than the regular teacher, it only relies modalittygeeeeeee decisive features and discards modality-specce -ppeciffc decisive features for prediction. However, the modality-general teacher still facilitates crossmodal KD and leads to improved student (~ +2% accuracy improvement compared with regular crossmodal KD). The results align well with MFH stating that a teacher with more emphasis modality-geneaa decisive features (i.e., has larger 7)) yields better student.
5.3 NYU DEPTH V2
We revisit the example of NYU Depth V2 in Sec. 3. We adopt teacher network that takes depth images as input to transfer knowledge to an RGB student. Both student and teacher network architectures are implemented as DeepLab V3+ (Chen et al., 2018). As described in Sec. 5.1, besides training a regular teacher, we follow (Girdhar et al., 2022) and train a teacher that learns predict labels for the two modalities with identical parameters. To be specific, training batch contains both RGB and depth images, and the teacher network is trained to output predictions given either RGB or depth images as imput. As such, the resulting teacher is assumed extract more modality-gentt general features for decision (i.e., has a larger y than a regular teacher) since it needs to process both modalities in an identical way during training.
As shown in Table 3, regular crossmodal KD does not bring many advantages: the student achieves a similar mloU compared with the No-KD baseline. Therefore, one might easily blame the failure of crossmodal KD on teacher accuracy and assume that crossmodal KD is not effective because the depth teacher itself yields poor performance (i.e,, has an mloU of 37.33%) By nature its training approach, the modality geeneal teacher is forced to extract more modality-generee decisive features for prediction rather than rely on spech-specific features as it also takes RGB images as input. While do not observe difference in teacher performance, the modality-general teacher turns out to be better choice for crossmodal KD: its student mloU improves from 46.36% to 47. .3%%. The results indicate that our MFH has the potential to diagnose crossmodal KD failures and lead improvement.
Table Results on NYU Depth V2 semantic segmentation. Teacher modality is depth images and student modality is RGB images. Compared with regular crossmodal KD, using a modality- teacher leads to better distillation performance.
Teacher Modality
Student
mIoU (%)
Modality
mloU (%)
No-KD Regular-KD KD Modality-generall KD
RGB
46.36
D
37.33 37.47
RGB
46.89
RGB /D
RGB
47.93 (1)
Nullified
32
MAP (50
Modality general
26
Regular
No-KD
24
20
80
Figure 4: The class activation maps of regular teacher (middle) and a modalittygeeeere teacher (right) VGGSound. A regular teacher attends to all decisive features (i.e., the visual objects) while a modality-general one focuses on modality-general decisive features (i.e., the area of vocalization).
leature dimession rutio
Figure 5: With more feature channels get- ting nullified, student performance starts to increase in the beginning and then decrease; the process aligns well with MVD.
5.4 RAVDESS AND VGGSOUND
Besides the approaches presented above, we design a permutation- based method to sort the features according the salience of modality-generee decisive features, which allows us to obtain teacher
8

---

Published as a conference paper at ICLR 2023
networks that possess different amount y. See Appendix B.4 for the detailed algorithm flow. We apply this approach to RAVDESS (Livingstone & Russo, 2018) and VGGSound (Chen et al., 2020a). RAVDESS is an audio-visual dataset containing 1.440 emotional utterances with 8 different emotion classes. Teacher modality is audio and student modality is images. The student and teacher network follow the unimodal network design in (Xue al., 2021); VGGSound is large-scale audio-visual event classification dataset including over 200,000 video and 310 classes. We consider two setups: (1) adopting an audio teacher and a video student; (2) using a video teacher for distillation an audio student. We also experiment with two architectures, ResNet- 18 and ResNet-50 (He et al., 2016) to be the teacher and student network backbone. In our algorithm, we sort feature channels according to
Table 4: Results on RAVDESS Emotion Recognition and VGGSound Event Classification. A, I and denotes audio, images and video, respectively. We report student test accuracy (%) for RAVDESS and mean Average Precision (%) for VGGSound. With identical model architecture, a modality- general teacher improves regular KD while -peeiit teacher leads to downgraded performance.
Task
T Mod.
S Mod.
Regular
-sdeificciiic
Random
Mod.-general
RAVDESS
A
T
77.22
42.66 (L)
64.75
78.28 (1)
VGGSound (RN-18)
A
V
30.62
24.98 (L)
28.99
31.88 (1)
VGGSound (RN-50)
V 38.78
28.65 (L)
35.95
39.81 (1)
VGGSound (RN-50)
V A 58.87
31.28 (L)
56.98
59.46 (1)
the salience of modality-generd decisive features. A modality-general teacher is derived by nullifying r% feature channels with smallest salience values. Similarly, a modallty-speeifee teacher obtained by nullifying the 7% fcature channels with largest salience values and a random teacher is derived by randomly nullifying r% channels. The feature nullifying ratio 7% is a hyperparameter and we experiment with different values of While the exact value of for different teacher networks is unknown, we know that y increases in the order of modality-specifee random and modality- -enerall teacher. As shown in Table 4, crossmodal KD performance aligns well with this order. For RAVDESS emotion recognition, a modality-specific teacher results in significantly downgraded student (i.e., accuracy drops from 77.22% to 42.66%) while modality- teacher improves regular KD by 1.06%. Similarly, for VGGSound event classification, modality-geneert crossmodal KD improves the video student performance from 30.62% to 31.88% (for ResNet-18) and from 38. 78% to 39.81% % (for ResNet-50). These results demonstrate the critical role of modality-generee decisive features in crossmodal KD.
Moreover, to provide an intuitive understanding of how modality-general teacher differs from the regular teacher, we present visualization results on VGGSound data in Fig. 4. We see that a regular video teacher utilizes all decisive features for classification, and attends to the visual objects (i.e., the Saxophone and the baby). On the contrary, a poodality-generee teacher focuses more on information available in both the visual and audio modality, thus the area of vocalization get most activated.
Finally, we vary feature nullifying ratio 7% for VGGSound data and plot the student performance curve along with r%. From Fig. 5, we observe that there exists sweet spot for modality-genedt KD. As r% increases, the student performance improves in the beginning. The improvement indicates that non modality-general general decisive features in the teacher are gradually discarded, which in turn results in a better student. Later, after all non modality-generul decisive information are discarded, the feature nullifying process starts hinder student performance modality generaa decisive features get nullified as well. The modality Venn diagram corresponding to this process is depicted in the upper figure. The observed performance curve aligns well with our understanding on MVD.
CONCLUSION AND FUTURE WORK
In this work, we present a thorough investigation of crossmodal KD. The proposed MVD and MFH characterize multimodal data relationships and reveal that modality-general decisive features are the key in crossmodal KD. We present theoretical analysis and conduct various experiments to justify MFH. We hope MFH shed light on applications of crossmodal KD and will raise interest for general understanding of multimodal learning as well. Future work includes: (i) deriving more profound theoretical analysis of crossmodal KD, (ii) differentiating modality- general/peccific decisive features for real-world data, and (iii) improving multimodal fusion robustness based on MVD.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
The authors would like to thank Hangyu Lin (HKUST) and Zikai Xiong (MIT) for providing crucial steps in proving Theorem and the anonymous reviewers for valuable feedbacks.
REFERENCES
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Asr is all you need: Cross-modal distillation for lip reading. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2143-2141 IEEE, 2020.
John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio González. Gated multimodal units for information fusion. arXiv preprint arXiv:1702.01992, 2017
2017.
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. Advances in neural information processing systems, 29, 2016.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Puttern Analysis and Machine Intelligence, 41(2): 423-443 443,443222 2018.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Liang-CCiehh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818. 2018.
Xiaokang Chen, Zeng. Bi-directional cross-modality feature propagation with rgb-d semantic segmentation. In European Conference on Computer Vision, pp. 561-577. Springer, 2020b.
Lin, Jingho Wang, Wayne Wu, Chen Qian, Hongsheng Li, and Gang gate for
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4794-4802, 2019.
Zhengqi Gao, Sucheng Ren, Zihui Xue, Siting Li, and Hang Zhao. Training-free robust multimodal jacobbiaan regularization. arXiv preprint arXiv:2204.02485 2022.
learning via sample-wise jacobian
Nuno C Gareia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In European Conference on Computer Vision, pp. 103-118, 2018.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. arXiv preprint arXiv:2201.055 08337, 2022.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6): 1090:9899999991 2021.
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 2827-2836. 2016.
Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. Misa: Modality-invarianl and- specific representations for multimodal sentiment analysis. In Proceedings of the 28th ACM International Conference Multimedia, pp. 1122-1131 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Tong He, Chunhua Shen, Zhi Tian, Dong Gong, Changming Sun, and Youliang Yan. Knowledge adaptation for efficient semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 578-587177 2019.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.023: 020331, 2(7), 2015.
Minhao Hu, Matthis Maillard, Ya Zhang, Tommaso Ciceri, Giammarco La Barbera, Isabelle Bloch, and Pietro Gori. Knowledge distillation from multi-modal to mono-modal segmentation networks. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 772-781. Springer, 2020.
Srinivas SS Kruthiventi, Pratyush Sahay, and Rajesh Biswal. Low-light pedestrian detection from rgb images using multi-modal knowledge distillation. In 2017 IEEE International Conference on Image Processing (ICIP), pp- 4207 4211. IEEE, 2017.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, ((4):541-554) 1989.
Kang Li, Lequan Yu, Shujun Wang, and Pheng-Ann Heng. Towards cross-modality medical image segmentation with online mutual knowledge distillation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 775-783, 2020.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng. Completer: Incomplete multi-view clustering via contrastive prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11174-1 1774-11188 1183, 2021.
Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2604-2613, 2019.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess); A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)cc0196391, 2018.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:1571.03643 2015.
Zelun Luo, Jun-Ting Hsieh, Lu Jiang, Juan Carlos Niebles, and Li Fei-Fei. Graph distillation for action detection with privileged modalities. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 166-183, 2018.
Fei Ma, Wei Zhang, Yang Li, Shao-Lun Huang, and Lin Zhang. Learning better representations for audio-visual emotion recognition with common information. Applied Sciences, 10(20):7239, 2020.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp- 5191-5198, 2020.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, 2012.

---

Published as a conference paper at ICLR 2023
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Baoyun Peng. Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5007-5016, 2019.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International Conference on Machine Learning, pp. 5142-5111115 5151. PMLR, 2019.
Sucheng Ren, Yong Du Jianming Lv, Guoqiang Han, and Shengfeng He. Learning from the master:
Distilling cross-modal advanced knowledge for lip reading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 13325-13333, June 2021.
Sucheng Ren, Zhengqi Gao, Tianyu Hua, Zihui Xue, Yonglong Tian, Shengfeng He, and Hang Zhao. Co-advise: Cross inductive bias distillation. Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 16773- 2022.
Sucheng Ren, Fangyun Wei, Zheng Zhang, and Han Hu. Tinymim: An empirical study of distilling mim pre-trained models. arXiv preprini arXiv:2301.01296, 2023.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint (0XKv:/206.6442 2012.
Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi learning. 2008.
Xinwei Sun, Yilun Xu, Peng Cao. Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou Wang. Tegm: An information-theene theoretic framework for cun-suueee multi malti-modality learning. In European conference computer vision, pp. 171-188. Springer, 2020.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03332 2020.
Fida Mohammad Thoker and Juergen Gall. Cross-modss modal knowledge distillation for action recognition. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 6-10. IEEE, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910. 10699, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Self-super learning from a multi-view perspective. arXiv preprint arXiv:2006.05576 2020.
Frederick Tung and Greg Mori. Similarity-preservvii knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1365-1374, 2019.
Francisco Rivera Valverde, Juana Valeria Hurtado, and Abhinav Valada. There is more than meets the eye: Self superviedd multi-objee detection and tracking with sound by distilling multimodal knowl- edge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11612-11633 1621, 2021.
Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: a multilayer approach for multimodal fusion. In Proceedings of the European Conference Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Jinghua Wang, Zhenhua Wang, Dacheng Tao. Simon See, and Gang Wang. Learning common and specific features for rgb-d semantic segmentation with deconvolutional networks. In European Conference Computer Vision, pp. 664-679, Springer, 2016.
12

---

Published as a conference paper at ICLR 2023
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual in- telligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. Deep multimodal fusion by channel exchanging. Advances in Neural Information Processing Systems, 33:4835-4845, 2020.
Maksymilian Wojtas and Ke Chen. Feature importance ranking for deep learning. Advances in Neural Information Processing Systems, 33:5105-5114, 2020.
Chen Xianing, Cao Qiong, Zhong Yujie, Zhang Jing, Gao Shenghua, and Tao Dacheng. Dearkd: Data- efficient early knowledge distillation for vision transformers. arXiv preprint arXiv:2204. 12997, 2022.
Zihui Xue, Sucheng Ren, Zhengqi Gao, and Hang Zhao. Multimodal knowledge expansion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 854-863, 2021.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: /6/2.03925 2016.
Wei Zhang, Weixi Gu, Fei Ma, Shiguang Ni, Lin Zhang, and Shao-Lun Huang. Multimodal emotion recognition by extracting common and modality-specific information. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, pp. 396-397, 2018.
Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Tor- ralba, and Dina Katabi. Through- of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 7356-7365, 2018
Through-wall wall human pose estimation using radio signals. In Proceedings
13

---

Published as a conference paper at ICLR 2023
A PROOF OF THEOREM
Lemma 1, Consider a function l(b, a):
l(b,a) - (a) (b)) - Ino(a)] (1 - o(a)) [ln(1 - ((b) - In(1 - o(a))
7
where o() represents the sigmaid functionn. When |a - 61 S E,
max ((b,a)
In -
(8)
1 1-
Proof. We only need to prove the following two things: (b) a) I I( + e, a). and (ii) ((a c, a) S
I-e-r -1 In --e--- For (i), we imagine a is fixed, and calculate the derivative of I(b, a) w.r.t b.
yielding:
01
ch ca
(1 + 6") ea)(1 +eb) = (b)) - ((a)
ab
Thus, when varying b in the range [a - + tl, the value of ((b, a) first decreases and then increases. It implies ((b, a) S U(a + a).
To prove (ii), we first consider the function:
4 (a) l(a a,
= -a [In (a + Ino - (1 - o(a)) [In[1 o( c)) In - In)1
=o(a) In 1 (a) In c)
(a)
1- 1
1- ((a)
-a( e In 1- (e)
1-o (a) hn e (ato)
(10)
1
1
-
l+e
-- 1+e-
1
1
+ ln
1
1+c-(a+)) (atc)
1
1+1
-f 1+t
+c-1 In
where in the last line, we denote
= a
Now, let us calculate the derivative of 4 w.r.t.
0i
e
0t
(1+
1+t
c-1-t + e
(11)
(1+
+t (t+e(--t)t (1+ +t)
Obviously, the denominator always larger than zero. After expanding the numerator, we see that <1(1-c)-1 I, achieves its maximum. Therefore, according to a In 4, we conclude when / = T+e-er
the maximum value of I, (a) is - - In -- achieved at = <((1-<)=1 I+c-et- we could define function (a) I(a c, a) and verify that the maximum value of (_ is - - Furthermore, after some simplifications, these two expressions, i.e., max 2 (a)
Similarly,
-c_
and max i_(a), could be proved identical.
Corollary 1. For two vectors E 1 and b E R", if bll then
L, (b, = Σ I(b a;) S n( -- - -
(12)
14

---

Published as a conference paper at ICLR 2023
The corollary is straightforward if noticing (a, - byl 1 ||a-
fa-bll bll I holds for any 1,2, ... n.
Lemma 2. If" max
||, |ZzZ*>
(Z S À always holds for both b. assume
I 7 Z Z
S (1 7,,, then we have:
| (Z
Za S x (2² +)((1 -7)
(13)
Proof. For conciseness, we temporarily denote A 7 and B - To begin with, we notice:
BBBBB") (BB"")"AA" A²)AA" (B" (BB') B A-
(14)
Then, we have:
B
(BB")- BAA
A $ [[
- ||I| A
B Bl
|A
< X BBBB
||A
S [BB
(1- . VIIAATI
(X + 1)(1-
(15)
Now, we are ready to prove Theorem 1. For reader's convenience, we re-state Theorem 1 in below.
Theorem A.1. (Crossmodal KD in linear binary classification). Without loss of generality, we assume to (-) X + y and fo, (-) : Xb y.. Suppose max (|="Z" [(Z"""2''-11 S always holds for both u = a or b, and g-) are identity functions. there exists (e,d) such that
Pr [[ Z Z Z"1l S (1 (1-7)e] 2 8-6
-))
(16)
Then, with an initialization at t = satisfying Riis (0,())) S we have, at least probability 1- ô:
R (0,(t = +oo))= s n² I-E-c
(17)
where e- = (22 + 1)(1 - 7)e and Re (0,) is the empirical risk defined by KL. divergence:
00. x - [L - (07 x+]]
1 (0. x!)
R (0.(t))
-o x
(18)
(0 x
1- (0,
Proof.
First, under the given conditions, if we could prove:
2-1l < (1-7)c => Relis (0,(t +oo)) S nl 1-e-c- 1 1-hn In e (19)
Then, adding the outer probability bracket doesn't 't alter the conclusion, and the theorem is proved. Therefore, in the following, we will focus on proving Eq. (19). The proof consists of two parts. In the first part, we will show that there exists an 0 such that Ieli (0;) is bounded. Then, in the second part, we will prove that if the training process is long enough (i.e., + -0)), then the final distillation risk Rdis (t,(t = +C)) is further bounded by R.lis
Without loss of generality, we assume the trained teacher weight ||0,1| = 1 since in linear binary classification, scaling the weight doesn't 't affect the final prediction. Now, let consider:
0- = (Z'Z"TT) - 7T 0,
(20)
Then we have:
ZZ TT; 0;
-za.
(222 T 20.T
$ I (Z'Zh z 21-
(21)
S A (A² + +)(117))
15

---

Published as a conference paper at ICLR 2023
where in the last line, we have used Lemma 2. Then using Corollary 1, we obtain
e*
(0;)) = Inn (Z ( 03 n - ml
(22)
1
1-e
where e" e== = (2² 1)(1 - 7)e. Now, the remaining effort is to prove that if time is sufficiently long (i.c., f - + +00), the trained loss (,,()) (t)) will be smaller than R(((())))
To begin with, we apply Theorem A.2 and Corollary A.1 from (Phuong & Lampert, 2019): For any sublevel set θ = {0,: 1i (0,) S q). there exists > 0 such that:
Rddis (0.) Rdi (0:) -
V Rrli (0:)1l²
(23)
Compared to the original Corollary A. of (Phuong & Lampert, 2019), slight modification is done for it to suit our case. Proving the existence of с is obvious by noticing that the left-hand side of Eq. (50) of (Phuong & Lampert, 2019) is relaxed to R((((;)) instead of 0 in our case. Next, noticing Eq. (8) and (53) of (Phuong & Lampert, 2019), we have:
dR (0.(1) -1| R (0.) S -2c (0.) + 2c 2e (0%)
(R
(24)
dt
which could be simplified as (Phuong & Lampert, 2019):
RRdss R& (0:) S -2c [R I=((((0))) (0:)
(25)
=> {Iu [ectis Reth (0:)}} } < -2c
Integrating over [0, yiclds:
R (0. S T (0. e 2nr
relii (0:) + -2or
(26)
Thus, as - +00, Rdin (0,(T +o0) will be upper-bounded by (O:). The above equation also indicates the convergence speed.
Remarks. Firstly, the above theorem implies that the training distillation loss is upper-bounded by a monotonic function with respect to 7. When y increases from 0 to 1, the upper bound gradually decreases. Further combining the above theorem with Rademacher complexity, we could even obtain a bound on the generalization eтror. Nevertheless, the key component in the generalization bound is already shown here. As such, readers should be alerted that our derived theorem is primitive, that our main contribution in this paper are MVD and MFH, and that complete theoretical analysis itself could be a standalone work. Secondly, we emphasize that proving our theorem, the MVD decision rule shown in Eq. (3) is not utilized. In the context of our theorem, Eq. (3) states that there exist O, and 0g. which could make the corresponding student and teacher network achieve zero generalization error However, our focus in the proved theorem is merely the training distillation loss, neither involving the generalization error, nor the cross entropy loss between the student prediction and the true label, so this MVD decision rule is not used. However, the decision rule is essential to fully characterize the MYD model as it states the cedaality-genere and modalty-specfic decisive features are sufficient to determine the label, and we expect will be of use when proving the generalization error. Finally, with the above two remarks on our insufficiency, it would be of great interest to extend our theorem to the generalization error with linear (or even non-linear) g g((-) and p o in K-class classification. In nutshell, we hope our proposed MVD could work as powerful model being utilized to theoretically prove generalization error of crossmodal KD.
B EXPERIMENTAL SETUP AND MORE RESULTS
B.1 CASE STUDIES IN SEC. 3
In this section, we describe implementation details and provide more results for the two case studies in Sec. 3. AV-MNIST (Vielzeuf et al., 2018) is an audio-visual dataset created by pairing audio and image features. The two modalities are MNIST images with 75% energy removed by principal
However, such a O might able achieve when doing crossmodal KD by using Eq. (1.
16

---

Published as a conference paper at ICLR 2023
component analysis and audio spectrograms with random natural noise injected. There are 50,000 pairs for training, 5,000 pairs for validation and 10,000 pairs for testing. Following (Vielzeuf et al., 2018; Gao et al., 2022), adopt a 6-layer CNN as the audio teacher network. The audio student network is implemented as a 3-layer CNN, and the multimodal teacher is late fusion network. The multimodal teacher uses LeNet5 (LeCun et al., 1989) the image backbone and 5-layer CNN as the audio backbone; Audio and image features are then concatenated and passed to fully-connected layers for the final prediction. Recall that p Eq. (1) the main text controls the relative importance of the two loss terms when training the student network. We experiment with both = 0 and p = 0.5, and repeat the experiments for 10 times. We have provided the results of p = 0.5 in Table in the main text, and more detailed version can be found in Table 5 below.
Table 5: Evaluation of unimodal KD (UM-KD) and crossmodal KD (CM-KD) on AV-MNIST.
Teacher
Student
Mod.
Acc. (%)
Mod.
Acc. (%)
p
0
p 0.5
No-KD
A
68.36
0.79
68.36 1: 0.79
UM-KD
84.57
A
69.86
0.70
70.10 1: 0.50
CM-KD
T+A
91.61
69.46 + 1.12 A
69.73 + 0.73
From the table, can that crossmodal KD does not have advantages over unimodal KD for both values of p. We hypothesize that the proportion of modality-general decisive information is small in this dataset since a multimodal data pair is assembled by randomly pairing image with audio that belongs to the same class. Thus the two modalities are not naturally correlated and there may be little modality-general information. MFH provides a plausible explanation for this failure case of crossmodal KD.
NYU Depth V2 (Nathan Silberman & Fergus, 2012) contains 1.449 aligned RGB and depth images with 40-class labels, where 795 images are used for training and 654 images are for testing. is set to 0.5. We implement two model architectures for the multimodal teacher: (1) Channel Exchanging Networks (CEN) (Wang et al., 2020) and (2) Separation-ande -aggonnnnnn-----atiiiiie Gate (SA-Gate) (Chen et al., 2020b). The unimodal teacher and student are adopted as the RGB branch of the corresponding multimodal network. The results are shown in Table 6, part of which corresponds to the right section of Table the main text.
Table 6: Evaluation of unimodal KD (UM-KD) and crossmodal KD (CM-KD) on NYU Depth V2.
CEN
SA-Gate
Teacher
Student
Teacher
Student
Mod.
mloU
Mod.
mIoU
Mod.
mloU
Mod.
mIoU
No-KD
RGB
45.69
RGB
46.36
UM-KD
RGB
45.69
RGB
46.23
RGB
46.36
RGS
48.00
CM-KD
RGB
51.14
46.70
RGB
51.00
47.78
Table demonstrates that crossmodal KD is not effective in both cases. The great advantages in teacher performance does not enhance student performance. Adopting CEN as the multimodal teacher seems better than SA-Gate, but the improvement compared with unimodal KD is still marginal (i.e. from 46.23% to 46.70%) According to MFH, different teacher networks utilize different amount of modality-generd decisive features for prediction, which results in different distillation performance. We hypothesize that CEN has a larger ? than SA-Gate due to their model design: CEN shares all parameters for the RGB and depth input except for Batch Normalization layer while SA-Gate has separate encoders for the two modalities. This indicates that CEN is more modality- -eeterl than SA-Gate, and this may further account for their performance differences. There may be other factors lying behind, and one future direction is to develop methods to compare existing model architectures to find teacher architecture that best suits crossmodal KD.
17

---

Published as a conference paper at ICLR 2023
B.2 SYNTHETIC GAUSSIAN
Assume two vectors x" E R and x" E IR compose one multimodal data pair (x", x²).. We select subset of input features as decisive features, denoted by x* E R We assume that x" exist in both x" and x², and denote the corresponding decisive feature index set of x (x*) J1 (J2). The separating hyperplanes are denoted by E TR. Formally, we generate one feature-label pair (x": x², y) by:
Na),
y 1(6,xx) > 0)
x NO,,aa,
x t XX1
(27)
x La),
x² t XJ2
As depicted in MVD, modality generalllgggealaeneee decisive features are decisive features shared by two modalities and thus indexed by J2 J1 represents the index set of decisive featsrre from both modalities. Therefore, a = 1 - [JrUJa] 8 = JeuJal and = [JrnJa] By changing J1 and J2, we can generate multimodal data with different inherent characteristics (i.e., different a, 3, and 7). We consider two settings: (1) varying (Fig. in the main paper). Let d = 25, d2 = 50 and = 20, gradually increase [J O J2l from 0 to 10, with step size of 2 and perform KD on every step. Consequently, 7 takes the value of [0, 01 0.25, 0.43, 0.67, 1]. and = 3 1=> (2) varying a (Fig. in the main text). Let di d2 = 50 and d = [J U J2l increase from 10 to 50, with step size of 10. Thus, takes the value of [0, 0.5, 0.67, 0.75, 0.80, 0.83]. We set 3 to be 0 through the process, and y = 1 a.
[J2]
[Ja]
Following (Lopez-Paz et al., 2015). the teacher and the student are both implemented logistic regression models, and we use 200 samples for training and 1,000 samples for testing. ö is sampled from the standard normal distribution. p in Eq. in the main text is set as 0.5. Results are averaged over 10 runs.
B.3 MM-IMDB
MM-IMDB (Arevalo et al., 2017) is the largest publicly available multimodal dataset for genre prediction on movies. contains 25,959 movie titles and posters that belong to 27 movie genres. We pick two movie genres (i.e. drama and comedy) for multi-label classification. There are 15,552 data for training, 2,608 for validation, and 7,799 for testing. We adopt the same pre-processing method as in (Arevalo et al., 2017) extract image and text features. We consider the special case of crossmodal KD, where the teacher is a multimodal network that takes both images and text as input and student is unimodal text network. The unimodal and multimodal architecture are identical the one in (I (Liang et al., 2021). As described in Sec. 5. we experiment with two teacher networks that have the same architecture but differ y: (1) We regularly train a multimodal network with labels; (2) Following (Xue et al., 2021), we train a multimodal network that receives pseudo labels from an unimodal image network. The second teacher only has access to pseudo labels from the image modality and leans towards the image modality when giving predictions. In other words, it is more modality-specific (i.e., has a smaller 7) than the regular teacher. We randomly split training data with the ratio 50%:50% and use the first half to train the unimodal teacher and the general multimodal teacher. The other part of data is used for training the student network, and we set p
Table 7: Results on MM-IMDB movie genre classification. T and represent text and images, respectively. With identical architecture and similar accuracy, a modality-specine teacher leads to worse crossmodal KD performance than a regular teacher since utilizes fewer modality-general decisive features for prediction.
Teacher
Student
Mod.
micro
macro
Mod.
micro Fl (%)
macro
Fl (%)
FI (%)
(%)
UM-KD
T
61.76
61.13
T
61.04
56.44
Modality-seciffc-Ke Regular-KD
T+I
62.01
61.08
T
61.77
57.26
T+I
61.01
60.37
65.09
63.31 (1)
18

---

Published as a conference paper at ICLR 2023
Table 7 shows the teacher and student performance for both unimodal KD and crossmodal KD. We select three teacher models that have similar performance on test data, and use them for distillation to detach the influence of teacher performance. Clearly, the three teachers transfer different knowledge to the student. The unimodal teacher comes from the image modality and the modality-speeiffe multimodal teacher is also biased towards the image modality due to its training strategy. Finally, the information compared with the previous
regular multimodal teacher adopts more modality- two teachers (i.e., has a larger 7). As can be seen from the table, results in the best unimodal text student, which helps verify our proposed MFH.
B.4 RAVDESS AND VGGSOUND
We first present permutation-base approach to rank given multimodal features according to the amount of modality -eenrral decisive information available in each feature channel. This approach offers an alternative way to obtain teachers of different 7 and helps validate the proposed MFH. As described in Sec. 5.4, once we have sorted list for all feature channels, we can derive a modality- general (or coodality-speciffee teacher by nullifying the top 1% smallest (or largest) channels during the distillation process.
The major steps of our proposed feature ranking approaches are demonstrated in Algorithm 1. The input Algorithm are X" E Rnxds, X R_xds and Y E R", representing n paired features from modality and b, and n target labels, respectively. The output is salience vector p E Id for modality-speciiff decisive features in modality a, where its i-th entry p. E [0, 1] reflects the salience of the i-th feature dimension. A larger salience value indicates more modality-geneeee decisive feature channel.
Algorithm 1 Modality-General Decisive Feature Ranking
Input: multimodal features (X" " E Baxd,, X Raxds Y Y
Output: salience vector p E Rdd for features of modality
1: Jointly train two unimodal networks fø; and føz using the following loss:
min L Dis((fo, (X"), So, (X*)) 01.02
CE(Y, fo. (X*)) + CE(YY fo, (X*))
(28)
Dist(; -) denotes distance loss (e.g., mean squared error) and C'E' denotes cross entropy Calculate the salience for the d-th feature dimension
2:
for = to d1 do
3:
P; =
4: for m = 1 to M do
Repeat permutation M times for better stability
5: permute the i-th column of X yielding X
6: Pt Pt + M x Dis (X²), foz (X*))
7: end for
8: end for
9: Perform normalization: p
P max: pa
10
Since input-level features contain much label-irrelevant noise, our Algorithm is designed following trace-back thought starting from the output level. Namely, we drive two unimodal networks the state of "feature alignment" at the output level using Eq. (28), and then use permutation to identify which input feature dimension has a larger impact to state. Those more influential to the state (i.e., large distance in step 6) will be assigned larger salience value.
Table 8: Test Accuracy (%) on RAVDESS emotion recognition. Teacher modality is audio and student modality images. Modality-genent crossmodal KD demonstrates best performance for all feature nullifying dimension ratio %%.
(%)
Regular
Modality-specife
Random
Modallit-geenene
77.22
75.26 ())
77.24
78.12 (1)
77.22
42.66 ())
64.75
78.28 (1)
77.22
12.00 (L)
63.40
77.82 (1)
19

---

Published as a conference paper at ICLR 2023
In step 1, we jointly train two unimodal networks fo; and fo; that respectively take unimodal data X and X as input. The first loss term in Eq. (28) aims to align feature spaces learned by the two networks, and the remaining loss terms ensure that learned features are essential for a correct prediction. We believe that this training strategy aligns three sources of decisive features at the output level. In step 2, we follow the idea of permutation feature importance (Breiman, 2001) to trace back modality-general decisive features at the input level. For the i-th dimension in X we randomly permute X along this dimension and obtain a permuted X in step 5. Next, we calculate the distance between fo: (X") and fa: (X+) in step 6. A large distance indicates that the i-th dimension largely influences the state of "feature alignment". Consequently, we are able quantify the proportion decisive featsiveeeattues in each input feature channel and use the salience vector p to represent it. We repeat the permutation process for M times and average the distance value for good stability. Finally, p is normalized to [0, 1j4 in step 9.
of modality-genertti
Note that: (1) The focus of this paper is to propose and validate MFH. Algorithm presents an approach to rank features and allows us to derive teachers of different 7 to justify the MFH implication. Developing methods that can separate modality-general decisive features and modality- specific decisive features is challenging problem worth deep investigation and is left as future work. (2) Algorithm 1 is not limited feature ranking at the input level. X" and X can be features extracted from middle layers of the neural network as well. In such case, output p reflects the salience for each middle-layer feature channel. (3) Algorithm I could be equally applied to rank modality- decisive features for modality b as long as we permute X
Next, we present results by applying this feature ranking approach to two multimodal applications, L.., RAVDESS emotion recognition and VGGSound event classification.
The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) (Livingstone & Russo, 2018) contains videos and audios of 24 professional actors vocalizing two lexically-matehee statements. For modality a (i.e., teacher modality), we adopt Kaiser best sampling and take mel- frequency cepstral coefficients (MFCCs) features from corresponding audio. For modality b (i.e., student modality), we uniformly sample single- frame images every 0.5 second from each video. We randomly split image-audio pairs, and have 7.943 data for training. 2,364 data for validation and 1,001 data for testing. Similar to (Xue et al., 2021), the teacher and student architecture are 3-layer CNNs followed by fully-connected layers. We set p in Eq. (1) in the main text as 0 (i.e., only use Led for distillation) to fully observe the teacher's influence on student performance. We report results with three feature nullifying ratio r% in Table 8, which is detailed version of Table 4 in the main text. Results are averaged over 5 runs.
As shown in the table, with more feature channels getting nullified (i.e., increasing 7%), the random and modality -specii version both suffer from a heavy performance degradation. On the contrary, a modality- general teacher still attains satisfactory distillation performance and outperforms regular KD even when the feature nullifying ratio goes to 75%. This demonstrates the efficacy of our proposed feature ranking method as well as the practical value of MFH.
Table 9: mean Average Precision (%) on VG- GSound event classification. Teacher modal- ity is audio and student modality is video.
MAP 196)
Mola Arganca
P
0
0.5
30
Regular
30.62
30.70
1000
2000
3000
4000
5000
-sdaiityspeccffic
24.98 (L)
28.14 (L)
Number Pemmutation Times
Random 28,99
29.56
Modality-general
31.88 (1)
31.98 (1)
Figure 6: Crossmodal KD peprformance with vary- ing permutation number M.
VGGSound is a large-scale audio-visual correspondent dataset. We randomly choose 100 class from its 310 classes and obtain 56,614 audio-video pairs for training and 4,501 audio-video pairs for testing. Videos clips and audio spectrograms are taken as input features, respectively. The audio network is implemented as a ResNet-18 / ResNet-50 backbone followed by linear layers and the video network is the same architecture with 2D convolution replaced by 3D convolution. For Table 4 in the main paper, we set in Eq. (1) to be 0 and experiment with both ResNet- 18 and ResNet-50. In
20

---

Published as a conference paper at ICLR 2023
Table 9, report results of both p p= and p 0.5 with the ResNet- 18 backbone. The conclusion consistent: a coddality-geeera teacher improves student performance while a modality-speeiinn teacher results in performance degradation. These results help validate proposed MFH.
In Algorithm we repeat permutation M times for a better estimation of each feature dimension's salience value. Fig. 6 provides an analysis on the number of permutation times M. As M increases, we have a more accurate estimation of p. so modality-gemtt genirrl KD gradually improves and finally reaches plateau.
C COMPARISON OF MVD WITH THE MULTI-VIEW ASSUMPTION
Below we compare MVD with the multi-view assumption (Sridharan & Kakade, 2008). We adopt the same notations used in the main text here for the multi-view learning paradigm: assume the input variable is partitioned into two different views x" and x², and there is a target variable of interest. The multi-view assumption states that either view alone is sufficient predict the target label accurately. As illustrated in Figure 7 (a), all task-relevant information is assumed to lie in the shared regions between x" and x².
Despite its wide use in unimodal self-supervised learning, the multi-view assumption is likely to fail when x" and x" are from different modalities (Tsai al., 2020). In fact, whether the multi-view assumption holds is largely dependent on downstream tasks and modalities involved. On one hand, for tasks such as image classification, either the image itself or its caption sufficient derive the target label. On the other hand, for tasks such MM-IMDB movie genre classification, using the movie poster (image modality) alone may not convey which category this movie belongs to. Movie descriptions (text modality -peeiiiii information) using the depth image alone is not sufficient for RGB-D semantic segmentation as there are regions in the image that have the same depth values but correspond different semantic labels. The multi-view assumption thus does not hold for many challenging multimodal tasks.
also needed to infer the target label y. Similarly,
Our proposed MVD generalizes the multi-view assumption by taking modality-specifie decisive features into consideration as well. It states that task-relevant information com- posed of three parts: (1) modality-
Task-relevant information
x
general decisive features; (2) decisive features specific to modality decisive features specific to modality 6, as illustrated in Figure 7 (b). Con- sequently, the multi-view assumption can be considered
and (3)
multi-view assumption
(b) modality Venn diagram
Figure 7: Illustration of (a) the multi-view assumption and (b) our modality Venn diagram.
special case of MVD when a = B = 0 and y = 1.
21


Reviews:
[{"review_id": "_56CcmOUhx", "reviewer": "Reviewer_i93D", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper gives some new insight into the multi-modal knowledge distillation community. Specifically, teacher accuracy does not necessarily indicate student performance, while modality-general features are the key in crossmodal KD. With the explanation of the modality Venn diagram, such a claim is straightforward and easy to understand. The authors also do some controlled experiments and try to support their analysis.\n", "summary_of_the_review": "I recommend accepting the paper (rating 6). I liked the hypothesis proposed by the authors. Although this paper has some drawbacks, the contribution is enough. Ratings can be improved further if the authors could solve my questions.\n\n----------- UPDATE I ----------- \n\nI thank the authors for their response to my concerns, especially about the **modality Venn diagram** and more experimental details. After reading the response, I consistently consider the proposed hypothesis is well-motivated. \n\nI hope the authors could highlight the details about the $\\alpha,\\beta,$ and $\\gamma$ in each experiment (e.g., bold, color) for better illustration.", "strength_and_weaknesses": "Pros:\n1. The proposed Modality Focusing Hypothesis is significant and might raise some interest in controllable cross-model distillation. Rich experiments verify this claim.\n2. The organization of the paper is good in general.\n\nCons:\n1. Although I admit and like the Modality Focusing Hypothesis, this hypothesis conflicts with the general multi-view assumption, i.e., each view could bring extra useful downstream information [1]. Different views always have some specific information and we always find that learning a multi-modal model is better than every single modality (CLIP), at least in most datasets. Why this phenomenon is different in knowledge distillation? One might argue that view-specific information contains much redundancy [1]. I'd like to see some explanations about this point.\n2. It is not clear to me that how to control the $\\gamma$, e.g., how to calculate $\\gamma$ in Table 2? What's the rule? Also, how to quantize this parameter during other experiments? \n3. For MM-IMDB data, why there is more modality-specific information?\n4. The modality Venn diagram is not new enough, as there are plenty of works that use this to analyze the effectiveness of multi-modal learning [1-3]. Specifically, [2] also claims that modality-general information is essential. The author is encouraged to change their claim that treats this as a contribution and do a better job by clarifying the differences between these relevant works and the proposed method.\n\n[1] Self-supervised Learning from a Multi-view Perspective, ICLR'21\n\n[2] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'22\n\n[3] COMPLETER: Incomplete Multi-view Clustering via Contrastive Prediction, CVPR'21", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KcJRn3vrz7", "reviewer": "Reviewer_Syoj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the topics of crossmodal knowledge distillation (KD), to transfer knowledge across modalities. To facilitate better understanding of crossmodal KD, the paper proposed a hypothesis that modality-general decisive features are the\ncrucial factor that determines the efficacy of crossmodal KD.", "summary_of_the_review": " The paper is overall well-motivated and the hypothesis is refreshing to me, however, I have some questions listed in the weaknesses part,", "strength_and_weaknesses": "Strength:\n(a) The author hypothesized that for crossmodal KD, distillation performance depends on the proportion of modality-general decisive features preserved in the teacher network, this hypothesis is refreshing to me.\n(b) The verification experiments in Figure 2 and 3 are convincing to me.\nWeaknesses:\n(a) In some cases,  modality general decisive features and modality-specific decisive features of audio and video modalities could be imbalanced, e.g. on VGGSound Event dataset, there could be more audio decisive features than visual features feature, How would the proposed method handle this?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CE5GV_KgCp", "reviewer": "Reviewer_vsTN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors investigated the key factors that could improve the performance of student model from a teacher model in a cross-modal knowledge distillation set. Based on authors investigation authors proposed a hypothesis that high performance of a teacher model does not always bring high performance student model, and the performance of the student model is determined by the modality-general decisive features. Based on the hypothesis, authors defined index of the modality-general decisive feature, and provided theoretical analysis and experimental evidence to prove their hypothesis.", "summary_of_the_review": "Authors investigated the key factors in determining when the student model could benefit from a teacher model in cross-modal knowledge distillation. Based on authors investigation, authors proposed a hypothesis with a new concept as modality-general decisive features which is essential in training a teacher model. However, it seems that how to determine what are the modality-general decisive features is not a easy task in real applications.", "strength_and_weaknesses": "The main strength: authors proposed the concept of modality-general decisive features for cross modal knowledge distillation, and proposed a hypothesis that the key factor for improving the student model performance is that the teacher model is trained based on the modality-general decisive features. Authors provided theoretical and experimental analysis to support their hypothesis. This hypothesis could provide new insight for the reason of when a student model could perform well in cross-modal knowledge distillation.\n\nMain weakness: the way to decide what kind of features are modality-general decisive features are difficult to control and search. The advantage of deep learning is to automatically learn the \"useful\" features for improving the performance, while authors strategy of determining the modality-general decisive features is heuristic, and usually difficult to decide. In particular, how to decide the modality-general decisive features itself is a difficult problem.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%