Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation
Authors: Chengming Hu
Haolun Wu
Xuan Li
Chen Ma
Xi Chen
Boyu Wang
Jun Yan
Xue Liu
Venue: ICLR 2024 poster (2024)
Keywords: Knowledge Distillation
Bilevel Optimization
Re-weighting
Primary area: general machine learning (i.e., none of the above)

Abstract: Knowledge distillation aims to train a compact student network using soft supervision from a larger teacher network and hard supervision from ground truths. However, determining an optimal knowledge fusion ratio that balances these supervisory signals remains challenging. Prior methods generally resort to a constant or heuristic-based fusion ratio, which often falls short of a proper balance. In this study, we introduce a novel adaptive method for learning a sample-wise knowledge fusion ratio, exploiting both the correctness of teacher and student, as well as how well the student mimics the teacher on each sample. Our method naturally leads to the \textit{intra-sample} trilateral geometric relations among the student prediction ($\mathcal{S}$), teacher prediction ($\mathcal{T}$), and ground truth ($\mathcal{G}$). To counterbalance the impact of outliers, we further extend to the \textit{inter-sample} relations, incorporating the teacher's global average prediction ($\mathcal{\bar{T}})$ for samples within the same class.  A simple neural network then learns the implicit mapping from the intra- and inter-sample relations to an adaptive, sample-wise knowledge fusion ratio in a bilevel-optimization manner. Our approach provides a simple, practical, and adaptable solution for knowledge distillation that can be employed across various architectures and model sizes. Extensive experiments demonstrate consistent improvements over other loss re-weighting methods on image classification, attack detection, and click-through rate prediction.

Published as a conference paper at ICLR 2024
LESS OR MORE FROM TEACHER: EXPLOITING TRILAT- ERAL GEOMETRY FOR KNOWLEDGE DISTILLATION
Chengming Hull2g Haolun W Xuan Li Chen Ma², Xi Chen', Jun Yan',
Boyu Wang', Xue Liu
(chengming hu, haolun, wu, xuan. 112] @mail aammil.mmigll... ca. chenma@cityus edu ok, xi.
weseeen1@@mmmil..ccmmm ca, jun. caancooncodia.cc. bwange esd. uwo. ca, xueliuêcs www.mmmlll.......
MMGGill University, "Mila- Quebec AI Institute, *City University of Hong Kong,
*Concordia University, Weettern University
ABSTRACT
Knowledge distillation aims to train compact student network using soft super- vision from a larger teacher network and hard supervision from ground truths. However, determining optimal knowledge fusion ratio that balances these super- visory signals remains challenging. Prior methods generally resort to a constant couristic-based fusion ratio, which often falls short of proper balance. In this study, we introduce a novel adaptive method for learning a sample-wise knowledge fusion ratio, exploiting both the correctness of teacher and student, well how well the student mimics the teacher on each sample. Our method naturally leads to the intra-sample trilateral geometric relations among the student prediction (S), teacher prediction (7), and ground truth (G). To counterbalance the impact of outliers, we further extend the inter-sample relations, incorporating the teacher's global average prediction (T) for samples within the same class. A simple neural network then learns the implicit mapping from the intra- and inter-sampl rela- tions to an adaptive, sample-wise knowledge fusion ratio in bilevel-optimizztion manner. Our approach provides simple, practical, and adaptable solution for knowledge distillation that can be employed across various architectures and model sizes. Extensive experiments demonstrate consistent improvements over other loss re-weighting methods image classification, attack detection, and click-through rate prediction.
INTRODUCTION
Knowledge distillation (KD) (Hinton al. 2015) widely used machine learning technique that aims to transfer the informative knowledge from cumbersome model (i.e., teacher) to lightweight model (i.e., student). The student is trained by both imitating the teacher's behavior and minimizing the difference between its own predictions and the ground truths. This is achieved by optimizing convex combination of two losses: = L& + a) LGT where a E [0, 1] the knowledge fusion ratio balancing the trade-off between the two different supervision signals.
Determining the knowledge fusion ratio a critical for training. The most straightforward method is to pre-set an identical value for all training samples (Hinton et al., 2015 Huang et al.) 2022 Clark et al. 2019: Romero et al.) 2014: Park al. 2019) Lassance et al. 2020). Other works, such as ANL-KD (Clark et al. 2019) and FitNet (Romero al. 2014), gradually decrease from to through an annealing factor. Recent studies (Lukasik et al. [02 Zhou et al., 2021 Lu et al. 202T)
imply that uniform knowledge fusion ratio across all samples is sub-optimal and cannot well capture the nuanced dynamics of the knowledge transfer process, thus designing the knowledge fusion ratio
in more grained manner. For instance, ADA-KD (Lukasik al. (021) assigns higher to a class if the teacher has higher correctness on that class. WLS-KD (Zhou et al. 2021
takes
Equal contribution with random order. To whom the correspondence should be addressed. Our source code can be found here

---

Published as a conference paper at ICLR 2024
Samples in D: Where the Teacher Predicts Correctly
Samples in 12: Where the Teacher Prodicts Incorraclyy
Top-1 Accuracy of Sludent
Aocuracy Studeni
0.8
0.3
0.6
0.6
0.4
0.4
0.0
9-
57> 9h
94
Groups based on ST discrepancy in ascending order Figure 1: Motivation experiment CIFAR-100 with ResNee-34 teacher and ResNet- 18 student.
Groups s7discrepancy sscending
The student is trained with varying knowledge fusion ratio values (a). Data is first partitioned into D (where the teacher predicts correctly) and D' (incorrect predictions), and further categorized into five equalized groups based on the student-teacher prediction discrepancies (ST), respectively. Our claim that determining greatly depends on ST and the correctness of teacher predictions.
both the teacher's and student's s correctness into consideration, and o increased if the teacher outperforms the student on sample, otherwise decreased. RW-KD (Lu et al. 2021) analyzes the same information as WLS-KD yet employs a meta-learning method to learn the sample-wise a.
However, existing methods largely ignore the discrepancy between the student's prediction (S) and the teacher's prediction (7), denoted as ST, when determining a. We argue that this oversight significant, making the student imitate the teacher lies at the heart of KD; thus intuitively, the ST discrepancy should offer valuable insights into balancing the two supervisory signals. Empirical results CIFAR- 100 (Krizhevsky (209) further verify our argument. The details of the motivation experiment are demonstrated at the end of this section. Derived from our observations, we draw the following insights:
If the teacher predicts correctly, higher ST discrepancy indicates the higher learning potential from the teacher, favoring a larger a. lower discrepancy indicates less potential to learn from the teacher and value in using the ground truth, thus smaller is preferred.
If the teacher predicts incorrectly, knowledge from the teacher is misleading, and thus smaller is advisable.
Regardless of the situation, determining a proper sample-wise value a relies on only the teacher's or student's performances but also the value of ST.
Consequently, our findings suggest that the ST discrepancy offers valuable insights for determining the knowledge fusion ratio In light of the emphasized importance of student-gron ground truth (SG) and reacher-groung truth (TG) relations existing studies (Zhou et al. 2021 Lu et al. 202I). p. we propose TGeo-KD, which captures all three relations aforementioned and naturally leads to model the intra-sample Trilateral Geometry among the signals from the student (S), teacher (T), and ground truth (9). To enhance the model stability against outliers, we further incorporate the teacher's global average prediction for a given class as an additional reference, abbreviated as T. enriching the geometric relations at both intra- and inter-sample levels. Based on the insights from the motivation experiment, we also argue that learning the sample-wist by merely heuristic rules. To this end, we propose learn the fusion ratio by neural network (NN) and formulate KD as a bilevel objective that leverages the trilateral geometric information. As result, the student is influenced by tailored blend of knowledge from both the teacher and the ground truth. Our proposed TGeo-KD, methods various tasks, from image classification to click-through rate prediction. To summarize, the main contributions of our work are as follows:
is quite involved and cannot be achieved
end-to-end solution, is versatile and proves superior to other re- weightingg
We introduce TGeo-KD KD, a novel method for learning sample-witg wise knowledge fusion ratios in KD. Leveraging the trilateral geometry, our method encapsulates the geometric relations among the signals from the student, teacher, and ground truth.
We exploit the trilateral geometry at both intra-sample and inter-sample levels, mitigating the impact of outliers training samples towards a more effective knowledge fusion.
We conduct comprehensive experiments across diverse domains demonstrate the consistent superiority over other loss re-weighting methods, as well as to highlight its versatility and adaptability across different architectures and model sizes.
2

---

Published as a conference paper at ICLR 2024
ANL-KD, FitNet al. atat,
ADA-KD Lukasik 2021
WLS-KD, RWHHD Zhoo 2021 oral,
TGeo-KD (ours)
ANL-D FRNett
ADA-K
WLS- IW-aD
TGee KD (ours)
Intro- & inter-sumple
Somple-wire
O bemple
Guasi
(uamplo
pamplel
Intre some-s reiation
X
X
Intersomplee
X
X
refotion
the slea))
Full trilotero!
X
X
X
reiotion
Figure 2: A comparison between prior works and our proposed TGeo-KD, The first two blocks show the relations captured in different methods for learning the knowledge fusion ratio from a geometric view a sample. The third block shows crosscheck comparison different method attributes. The details for representing each point and computing the geometric relation are demonstrated in Sec.
Details of Motivation Experiment. We conduct our motivation experiment on CIFAR- 100 (Krizhevsky, 2009). Specifically, we partition the dataset into two subsets, D and D', where D consists of samples on which the pre-trained teacher (ResNet-34) has correct predictions, whereas D includes those with incorrect predictions. Initially, with a = 0.5, we train a student model (ResNet- 18) over 50 epochs to imbibe preliminary knowledge. We then compute the Euclidean distance between the student's s and teacher's s predicted class probabilities across all samples, designating this as the ST discrepancy. Based on ascending ST values, further split D and D' into five equalized groups 91 ~ 95 and g ~ gs. respectively. Subsequently, the student is further trained with varying a values adjusted from {0.1, 0.3, 0.5, 0.7, 0.9). yielding five distinct student models. Upon evaluating these students across all five groups and five g' groups, we obtain 25 bins for each subfigure in Fig. D which reveals that: (i) For samples in students trained with smaller values (i.e., 0.1, 0.3) outperformed on groups with lower ST discrepancies (i.e., 92), whereas larger values (i.e., 0.7, 0.9) were beneficial for groups with higher discrepancies (i.e., G1. 9s). (ii) For samples in D', smaller a (i.e.. 0.1, 0.3) demonstrated the best performance on all g groups. Our observation shows a proper sample-wise value a relies on not only the student's 's or teacher's performances but also their discrepancy, which motivates the design of our proposed method for knowledge fusion learning.
PRELIMINARY: REVISITING KNOWLEDGE FUSION RATIO KD
The vanilla KD (Hinton al. 2015) transfers knowledge from pre-trained teacher network to a student by reducing discrepancies between their predictions and aligning with the ground truth. The student learns through two losses: CKD, the Kullback -eiiber (KL) divergence (oyce 2011 I) between student and teacher predictions, and LGT the Cross-Entrops (CE) loss (Good, 1952 from the ground truth. Formally, denoting D = y:) as the data where y; is the ground truth label represented as one-hot vector for each sample, C' as the number of classes, 2f and z as the logits of the student and teacher, we formulate the two losses in a sample-wise manner as follows:
(z(L log (2///T)
C = = PKKL 2,)
Σ 1=1
(z(27/7)
yi, C = -
= Σ
log (o,(2))).
(2)
where is the softmax function and the temperature controls the softness of logits. Then the overall training objective aims to optimize the student network (parameterized by 8) through a convex combination of the two losses with sample- wise fusion ratio a;;
C = min Σ
KD - -1- C..
We present a comparison of prior knowledge fusion methods alongside our work in Fig. empha- sizing both the geometric relations captured for learning and their distinctive model attributes. Evidently, our proposed TGeo-KD overcomes the constraints observed in previous approaches, thus leading to enhanced performance.
3

---

Published as a conference paper at ICLR 2024
3 TRILATERAL GEOMETRY GUIDED KNOWLEDGE FUSION
3.1 ADAPTIVE LEARNING FOR KNOWLEDGE FUSION RATIO
To address the limitation of prior works and employing the insights from the motivation experiment as depicted in Sec. M we propose to adaptively learn the knowledge fusion ratio based on trilateral geometry within (S. T. 9) triplet using separate network. For simplifying the notation,
consistently denote == ozz") E I N and T := azz') E Rx as the prediction probabilities of the student and teacher, and G == y R the ground truth (i.e., each row is one-hot vector). Given a training sample (x,, Y;). the knowledge fusion ratio can be correspondingly modeled as ag = (A.). where fw is one NN parameterized by w. The final layer of fw employs a sigmoid activation, ensuring that a; E (0, 1). For brevity, we omit explicitly writing the sigmoid function. A; represents the unique geometric relation among S, T, and G,.
Our ultimate goal is to find the optimal sample-wise ratios a; = f (A,) that enable the student net- work parameterized by 9 to generalize well on test data. This naturally implies bilevel optimization problem (Franceschi et al. 2018) with w as the outer level variable and 0 as the inner loop variable:
Noi
min Tonner (6" (w)) 1
GT
(4)
Nval
Nini
s.t. 0 (w) argmin Triner (0,w))
(0,w)=
:
Σ (LAA) + ((1- C.T.
(5)
Nirain
On the inner level, we aim to train a student network given a fixed w by minimizing the combined loss. On the outer level, the loss function on the validation set serves as a proxy for the generalization error of w. The goal for TGeo-KD to find w to minimize the validation loss. Note that Eq. an implicit function of w as 0" depends on w.
3.2 EXPLOITING TRILATERAL GEOMETRY
For modeling the trilateral geometry of A, we propose to capture both intra-sample and inter-sample geometric relations. The details are demonstrated as follows.
Intra-sample relations. Given the 7th sample, to capture the trilateral geometry of the (S;, Tr, Gi) triplet, denoting as ATT9, we capture its three edges as outlined below:
e,- [9: S .e.h := [9. Til RC ee := [T. [TT- E R
(6)
The three edges represent the student's correctness, the teacher's correctness, and the discrepancy between the student and teacher, respectively. Previous research (Zhou et al.]202T] Lu et al. (021) has affirmed the efficacy of the first two edges in guiding the learning of a, while the third concept is original contribution. We finally represent A,79 by also incorporating the exact three vertices S,. T, G. to capture the exact probability across all classes for incorporating more information:
A [e
c
@ S ① Ta + Gil.
where @ is the concatenation operation.
Inter-sample relations. In addition to intra-sample relations, we argue that inter-sample relations are also essential for knowledge fusion learning, especially considering the impact of outliers in training samples. Qut-of-distribution samples, which are significantly different from normal training data, commonly behave outliers to challenge the generalization capability of model (Lec) ctt al., 20188) Wang et al.) 2022). In KD, the teacher network may perform poorly on these outliers, occasionally even with high absolute values of confidence margin. Therefore, blindly using the teacher's 's prediction as the supervisory signal can result in the propagation of misleading knowledge, thereby disturbing the student's 's training process.
To address this issue, we introduce inter-sample geometric relations. For each sample, we associate it with additional vertex Te E RRC representing the teacher's global average prediction for all
4

---

Published as a conference paper at ICLR 2024
samples of the class (c') that sample i belongs to. is essential to understand that while each sample linked to its respective class-specific vertex, samples within the same class refer to the same vertex Tel. Consequently,
incorporate additional triplet, (S,, Tel, 9i), to encapsulate these inter-sample relations. This is achieved by similar process as before, focusing on the three edges, as well as incorporating all the vertices as follows:
A = e ①
e
@ cD Tox @ 9:].
As such, by introducing the teacher's s average prediction at the inter-sample level, the exploitation of more supportive knowledge can be further facilitated to effectively guide the student training process, particularly in addressing outliers.
Improved distillation with trilateral geometry. Although we can fully explore the sample-wise trilateral geometry through intra- and inter-sample trilateral relations, it is still challenging to design an explicit formulation between these signals and a knowledge fusion ratio as depicted in Sec. [ We thus use simple network (-) parameterized by w, adaptively learn flexible and sample-wiss knowledge fusion ratio with the input of geometric relations. The information captured for each sample can be represented as follows:
A == TT +
STO
|e "
of + BC
09 @ 0
Ti T Gi).
(10)
where the redundant terms are removed for brevity. Through inputting A; into (.,(),, the knowledge fusion ratio og can be adaptively learned, and is optimized with in end-to-end way.
EXPERIMENTS
4.1 TASKS AND EXPERIMENT SETTINGS
Tasks and datasets. To demonstrate the broad applicability of method, we conduct extensive experiments on three different tasks. Specifically, we use CIFAR- 100 (Krizhevsky 2009) and ImageNet (Deng et al., 2009) for image classification computer vision, HIL (Pan et al. [015) for attack detection in cyber-physical systems, and Criteo Jean-Baptiste Tien, 2014) for click click-throuhh rate (CTR) prediction recommender systems. Details of datasets and task selection are shown in Appendix A. A.
Experiment settings. In the experiment setup, the temperatures (r) are set as 4.0, 1,5, 1.5, and 2.0 on the four datasets, respectively. In the vanilla KD, the pre-set fusion ratios are 0.2, 0.3, 0.1, and 0.3, respectively. Considering that the original HIL (Pan et al.) 2015) and Criteo (Jean-Baptist Tien 2014) are imbalanced, we conduct oversampling pre-processing procedure, ensuring all classes have the equal number of samples in balanced setting. We conduct the experiments on one NVIDIA RTX-3080 GPU and one RTX-3090 GPU. The detailed experiment settings can be found in Appendix A.2
the minority class of training
the data
4.2 STUDENT CLASSIFICATION PERFORMANCE
Results on CIFAR-100. We evaluate our proposed TGeo-KD method against numerous established KD baselines, as illustrated in Table To ascertain the significance of improvement, we conduct statistical t-test across five repeated runs, with t-scores being calculated based on the top-1 classifica- tion accuracy of TGeo-KD and the baseline methods. All computed t-scores surpass the threshold value lo.05.5. indicating the acceptance of the alternative hypothesis with a statistical significance level of 5.0%. This furnishes compelling evidence that TGeo-KD consistently demonstrates marked enhancement in performance.
Notably, when the teacher (ResNet-56) and student (ResNet-20) models possess relatively similar architectures, ADA-KD (Lukasik et al. 2021) is the best baseline with marginal improvement of 0.07% over the second-best baseline WLS-KI KD (Zhou et al. 2021) ). In comparison to ADA- KD LLukasik et al. 2021). TGeo-KD KD illustrates substantial advantage of 0.76%. As the architectural gap incresses, like between the student ResNet-32 and the teacher ResNet- 10, our method's 's per- formance advantage increases 0.97%, compared to the best baseline. This performance gain
5

---

Published as a conference paper at ICLR 2024
Table Top-1 classification accuracy (%) on CIFAR-I00. We re-implemented the methods denoted
by and calculated their average results (with standard deviation) over 5 repeated runs. For the remaining methods, we utilized the results provided verified by the others (Tian et al. 2020 Zhou
et al. 2021). The best performance is bold, while the second best is underlined.
Method
Sume architecture style
Different architecture styles
Teacher
WRN-40-2
ResNer-56 ResNet-20
ResNet-1 10 ResNet-32
ResNet-110
ResNet-12x4 ResNet-8x4
ResNet-32x4
ResNet-32xx4
WRN-40-2
Student
ShuffleNeVV
ShutlleNetV2
ShuflieNerV1
Teacher Student
75.61
72.34
74.31
74.31
79.42
79.42
79.42
75.61
71.98
69.06
71.14
69.06
72.50
70.50
71.82
70.50
FitNet
72.24
69.21
71.06
68.99
73.50
73.59
73.54
73.73
AT
72.77
70.55
72.31
73.44
71.73
72.73
73.32
SP
72.43
69.67
72.69
70.04
72.94
73.48
74.56
74.52
CC
72.21
69.63
71.48
69.48
72.97
71.14
71.29
71.38
VID
73.30
70.38
72.61
73.09
73.38
73.40
73.61
RKD
72.22
69.61
71.82
69.25
71.90
72.28
73.21
72.21
PKT
73.45
70.34
72.61
70.25
73.64
74.10
74,69
73.89
AB
72.38
69.47
70.98
69.53
73.17
73.55
74.31
73.34
FT
71.59
69.84
72.37
70.22
72.86
71.75
72.50
72.03
NST
72.24
69.60
71.96
69.53
73.30
74.12
74,68
74.89
CRD
74.14
71.16
73.48
71.46
75.51
75.11
75.65
76.05
Vanilla KD
73.54
70.66
73.08
70.67
73.33
74.07
74.45
74.83
ANL-KD*
72.81=025
72.13=018
72.50=021
72.28
75.07=0025
72.58
73.11=0.14
75.27
ADA-KD*
74.67-019
72.22.0211
73.19au12
22.29
75.78=0344
71.45
72.20au24
a5:49
WLS-KD
74.48
72.15
74.12
72.19
76.05
75.46
75.93
76.21
RW-KD"
73.92=022
70.33
71.78
71.24
74.86:0.29
70.45
70.69
74.15
TGeo-KD
75.43=0116
72.98=0114
75.09
73.55
77.27=0225
76.83
76.89
77.05
Table 2: Top-1 and Top-5 classification accuracy ImageNet. We re-implemented the methods denoted by" and used the author- prrvid- or author-verified results for the others (Zhou et al. 2021
Teacher: ResNet-34 - Student: ResNet-18
Teacher: ResNet-50 - Student: MobileNetV1
Method
Top-1 ACC
Top-5 ACC
Method
Top-1 ACC
Top-5 ACC
Teacher
73.31
91.42
Teacher
76.16
92.87
Student
69.75
89.07
Student
68.87
88.76
AT
71.03
90.04
AT
70.18
89.68
NST
70.29
89.53
69.88
89.50
FSP
70.58
89.61
68.89
88.71
RKD
70.40
89.78
RKD
88.32
Overhaul
71.03
90.15
Overhaul
71.33
90.33
CRD
71.17
CRD
88.94 94
Vanilla KD
70.67
90.04
Vanilla KD
70.49
89.92
ANL-KD
71.83-022
90.21:00.6
ANL-KD"
70.40
89 49.25
ADA-KD*
71.96=0.7
90.45++0.1
ADA-KD*
71.08+0244
90.17+0.16
WLS-KD
72.04
90.70
WLS-KD
71.52
90.34
RW-KD"
70.62:022
89.76=0011
RW-KD"
70.15
89.402a.19
TGeo-KD
72.89
91.80+00.4
TGeo-KD
72.46:0.14
90.95
further escalates to 1.26% when the student ResNet-20 and the teacher ResNet- 110, underscoring the advantage of our TGeo-KD particularly when dealing with the increasing architectural disparities between the student and teacher. Moreover, our technique excels in hetero-architecture KD scenarios, wherein knowledge is distilled from either ResNet-32x4 WRN-40-2 models into ShuffleNet. In these cases, our method consistently demonstrates performance enhancements of 1.37%, 0.96%, and 0.84%. respectively, compared to the strongest baseline WLS-KD (Zhou et al. (021).
Results on ImageNet. To further demonstrate the effectiveness of our approach on larger datasets, we extend our experiments to ImageNet, adhering to the setup outlined by Zhou et al. (202I). As depicted in Table 2 TGeo-KD consistently outperforms all competing baselines. Notably, compared with the strongest baseline WLS-KD (Zhou et al. 2021), TGeo-KD KD exhibits the performance improve- ment of 1.10% when the teacher (ResNet-34) and student (ResNet-18) share the same architecture KD experiment with ResNet-50 and MobileNetV1 as teacher
style. Similarly, a and student respectively, our method realizes improvement of 0.94%.

---

Published as a conference paper at ICLR 2024
Table 3: Result comparison on HILL and Criteo under Teacher (12-layer B + Student (4-layer BERT). The best performance is bold, while the second best is underlined. "f" indicates the metric value the higher the better, while "II" indicates the lower the better. Our TGeo-KD demonstrates a statistical significance for p < 0.01 compared to the strongest baseline based on the paired t-test.
Dataset
HIL
Criteo
Method
ACC (%) ft
AUC (%) ft
NLL 1
ACC (%) ft
AUC (%) ft
NLL 4
Teacher
88,19
75,23
0.94
78.15
79.08
0.77
Student
87.64
67.58
1.02
69.43
69.02
1.79
Vanilla KD
87.55
69.52:::.70
1.00
71.08
69.4222000
1.51 10.05
ANL-KD
1:023
70,01
1.00
72.71
0.30
1.08
ADA-KD
90.15 1::3.4
70.02
0.99
72.15 0..33
71.01 0.33
1.15
WLS-KD
90.05++0.28
70.70
1.01 :0.0s
75.30=20.38
75. 20:40
0.
RW-KD
89.40++045
66.03-1:..88
1.07
75.05=:044
75. +0.53
0.89
TGeo-KD
92.39
71.65
0.94
77.80
77.00
0..14
Paccrtaage of ammpee 09
percentage of semplee (41
wthou ST
of semplee
without 57
12
9,00
U.
0.5
1.0
0.0
0.2
0.4
U.
0.8
1.0
8G
0.2
0.4
0.5
0.8
1.0
0.0
0.7
0.4
Knoodedge fusior rebl
Kncetadge fusion rabro
Snoer edge Rusion retin
Snowsedge fusion ratio
(a) Teacher X, large ST (b) Teacher X, small ST (c) Teacher
large ST (d) Teacher
small ST
Figure 3: Knowledge fusion ratio distributions learned with (dark) and without (light) incorporating ST during learning a. We first partition all samples into two subsets based on the teacher's s correctness. In each subset, we sort the samples in descending order based on their ST values and select the top and bottom 20% as those with large and small discrepancies, respectively.
Results on HII. and Criteo. To illustrate the broad applicability of our proposed TGeo-KD in diverse application scenarios, we also observe the similar superiority of our method HIL for attack detection and Criteo for CTR prediction, as shown in Table 3 For instance, TGeo-KD not only relatively improves ACC and NLL over ADA-KD (Lukasik al. 202T by 2.48% and 5.05% on HIL, respectively, but also is better than the deeper teacher with the increasing ACC of 4.20%. Besides, WLS-KD (Zhou et al.) 202T) and RW-KD (Lu al., (02T) are the best methods among all the baselines on Criteo, approving the effectiveness of adopting sample-wise knowledge fusion ratio. More results on various network architectures are provided in Appendix
4.3 FUSION RATIO ANALYSIS WITH PREDICTION DISCREPANCY
To demonstrate the effectiveness of incorporating prediction discrepancy ST between the student and teacher on learning the knowledge fusion ratio a, we follow the same settings the motivation experiment in Sec. for a comprehensive analysis. We first categorize training samples based on the correctness of teacher predictions and ST on these samples. We then compare the distributions of fusion ratio learned with and without the consideration of ST, as depicted in Fig. 3 When the teacher predicts incorrectly, it may transfer misleading knowledge the student, resulting in a decline in the student's performance. By incorporating ST, our proposed TGeo-KD decreases the fusion ratio on CKD when the discrepancy is either large or small (in Fig. 3 and Fig. b)), which suggests that the student is encouraged to acquire more knowledge from the ground truths. In cases where the teacher predicts correctly, the fusion ratio typically ranges from 0.4 and 0.8 when not incorporating ST. The fusion ratio is greater when incorporating ST in situations with a significant discrepancy between the student and teacher (in Fig. ()). This suggests that the student is expected to emulate the teacher more closely, as the teacher possesses greater potential for offering valuable knowledge. When the discrepancy is smaller, the student is encouraged to rely more on the ground truth, leading to decline in the fusion ratio (in Fig. d)). On the contrary, as illustrated in Table 4 the fusion ratio steadily increases without the incorporation of ST as the training progresses, yet an insufficient potential can be learned from the teacher.
7

---

Published as a conference paper at ICLR 2024
Table 4: Comparison of average knowledge fusion ratios with and without ST during the training.
Teacher prodictions
x
X
Discrepancy
Large
Small
Large
Small
Epoch
100
200
300
400
100
200
300
400
100
200
300
400
100
200
300
400
Without ST
0.51
0.46
0.43
0.42
0.48
0.42
0.40
0.39
0.48
0.55
0.57
0.57
0.52
0.59
0.62
0.64
With ST
0.25
0.19
0.14
0.12
0.19
0.13
0.10
0.08
0.59
0.67
0.71
0.73
0.42
0.33
0.29
0.27
Percentage of samples (89
10
Percentage of semples (89
10
Percenlage of semples (%)
10
Normal
Normal
Normal
Outlier
Outler
%.00
0.0
%.0
Knowlledge fusion ratio
Knowledge fusion ratio
Knowledge fusion ratio
(a) WLS-KD
(b) RW-KD
(c) TGeo-KD
Figure 4: Knowledge fusion ratio distributions on normal samples (light) and outliers (dark).
4.4 ANALYSIS ON NORMAL SAMPLES AND OUTLIERS
Fusion ratio on normal samples and outliers. To conduct analysis on fusion ratios between normal samples and outliers, we first create outliers by adding synthetic Gaussian noise as additional training samples following the setting outlied in the studies (Hendrycks and Gimpel| 2016 Liang al. 2018 Vyas et al. 2018). the context of Gaussian noise based outliers, each RGB value for every pixel is sampled from an independent and identically distributed Gaussian distribution with a mean of 0.5 and unit variance, and each pixel value clipped to the range [0, 1J. As illustrated in Fig. 4 we compare the final fusion ratio distributions between sample-wise based baselines (i.e. WLS-KD (Zhou et al.) 2021 and RW-KD (Lu et al., (02T)) and our T'Geo-DD on CIFAR-106 1000 Compared to the two baselines, TGeo-KD reports the final fusion ratios for normal samples within the range of 0.4 and 0.6 typically, which indicates that the student can be effectively guided with the valuable supervision signals from both the ground truth and the teacher on these normal samples. Furthermore, the teacher may make incorrect predictions on outliers, which provides misleading knowledge to the student. With the consideration of ST discrepancy and inter-sample relations. "Geo-KD reports the fusion ratios below 0.3 on outliers. which suggests that the student is expected learn more informative knowledge from the ground truth, resulting in increased fusion ratio on LGT, More comparisons about the effect of ST discrepancy and inter-sample relations on outliers can be found in Appendix A.4
Prediction discrepancy during the training and testing. Fig. 5 shows the prediction discrepancies between the student and teacher normal samples and outliers during training and testing. Compared to baselines, in Fig. 5) (). our proposed TGeo-KD achieves the smallest discrepancies (i.c., 0.19 during the training and 0.26 during the testing) normal samples, indicating that the student can effectively mimic the teacher predictions through learning the knowledge distilled from the teacher. Furthermore, in Fig. b)), although ADA-KD (Lukasik et al., (021) and RW-KD (Lu et al. 2021 (i.e, the baselines without inter-sample relations) report fewer discrepancies on outliers during the training, the teacher may report poor performance on these outliers and transfer misleading knowledge the student. With the power of our inter-sample relations, TGeo-KD surpasses these aforementioned studies during the testing, which indicates that inter-sample relations can protect the student training process from being disrupted by low-quality knowledge from the teacher, especially on those outliers.
4.5 ABLATION STUDIES
Effeet of different relations captured. TGeo-KD adaptively learns the knowledge fusion ratio by leveraging both intra- and inter-sample relations. To illustrate the effectiveness of each relation, we conduct experiments to train students under different combinations. As summarized in Table both intra- and inter sample relations yield better performance than the standalone student and vanilla KD models. Specifically, when incorporate the ST relation, there is a notable increase in top- accuracy from 73.92% to 75.28%. This suggests the importance of accounting for the discrepancies between the student and teacher models. The performance improves to 76.83% when the inter-sample
8

---

Published as a conference paper at ICLR 2024
training testing
2.0
training
tesking
0.6
dstance
1.5
Exdidaan
0.4
1.0
0.5
Vonilla
Vanilla
MNLKD
ADNKD
WLS-KD
RW-KD
TGec-KD
(a) Normal samples.
(b) Outliers.
Figure 5: Prediction discrepancies on normal samples and outliers during the training and testing.
Table Top-1 classification aceuracy (%) com- Table Ablation study on relations for learning parison among different modellings of f-) on CIFAR-I00. The teacher and student are ResNet-32x4 and ShuffleNetV1.
through attention and MLP.
CIFAR-100
ImageNet
SG+T9 ATG
ACC
1-head Atten.
73.00
70.96
Student Vanilla KD
70,50
2-head
73.63
71.72
74.07
3-head Atlen.
73.87
72.08
TGeo-KD
73.92
1-layer MLP
72.69
71.74
2-yayer MLP
74.31
72.89
TGeo-KD
75.28
3-yayer MLP
73.96
72.52
TGeo-KD
76.83
relations are further captured. A deeper dive into how various representations of these intra- and inter-sample relations affect performance can be found in Appendix A.4
Comparison between attention mechanism and MLP. We assess various options for modeling the knowledge fusion ratio generator, f-),,,, including attention mechanism as suggested in (Vaswani et al. 2017) and MLPs with different numbers layers. For CIFAR-100, the teacher and student are ResNet- 110 and ResNet-32. For ImageNet, the teacher and student are ResNet-34 and ResNet- 18. Based on the results in Table 5 a 2-layer MLP is sufficient capturing the valuable information embedded in the trilateral geometry, leading superior performance across datasets. Furthermore, MI.P settings generally demonstrate higher performance compared to attention mechanisms.
RELATED WORKS
Knowledge distillation techniques have evolved from uniformly applying fusion ratios across samples (Hinton et al.] 2015 Huang et al. 2022] Clark et al. 2019
Knowledge balance in KD.
Romero et al. 2014 Park al. 2019 Lassance et 2020) more refined strategies. Methods like ANL. KD (Clark et al. 2019) and FitNet (Romero et al. 2014) use annealing factors to adjust fusion ratios. Recent advancements include ADA-KD (Lukasik al. 2021). which uses class-wise ratios, and WLS-KD (Zhou et al. 2021), which adjusts based on student's 's and teacher's performance. RW-KD (Lu et al. 2021) employs meta-learning for adaptive ratio learning. Yet, existing methods lack consideration of the comprehensive trilateral relations among the signals from the student, teacher, and ground truth during the knowledge fusion learning.
Sample relations in KD.
The exploitation of sample relations in knowledge distillation has been a key focus of numerous studies (Zagoruyko and Komodakis 2017: et al. 22166/10 Park et al.
2019 Heo et al. 2019 Tian et al. 2020). For instance, AT (Zagoruyko and Komodakis 201 7 Hu et al. 2023) introduces attention transfer to transfer the attention maps from the teacher to the student, explicitly capturing the sample-level relation. CAMT (Zhou et al., (2016) extends this idea by incorporating both spatial and channel attention transfer to enhance the student's performance. Other studies have also emphasized relational reasoning and contrastive representation as key mechanisms for better understanding and improving knowledge distillation (Park et al.) 2019 Tian et al.) (2020). Despite these ad advancements in capturing sample relations, none of these studies target on the learning of the knowledge fusion ratio.

---

Published as a conference paper at ICLR 2024
6 CONCLUSION
We propose an innovative approach named TGeo-KD for learning sample-wise knowledge fusion ratios during KD, which exploits the trilateral geometry among the signals from the student, teacher, and ground truth by modeling both intra- and inter-sample geometric relations. Across diverse domains, TGeo-KD outperforms other weightitnn methods consistently. offers a simple yet adapt- able distillation solution that fits various architectures and model sizes, thereby easing deployment complexities and resource limitations associated with deep neural networks. For the broader impact, TGeo-Ki KD is effective across a variety of domains, including image classification, attack detection, and click-through rate prediction. Future research on TGeo-KD will address its focus on inter-sample relations within classes, exploring cross-class relationships to enhance performance.
REFERENCES
Kevin Clark, Minh-TTang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc Le. BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5931-5937, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
doi: 10.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learming, pages 1568-1577. PMLR, 2018.
J. Good. Rational decisions. Journal of the Royal Statistical Society. Series (Methodological), (1):107-114, 1952.
Huifeng Guo, Ruiming Tang, Yumming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A factorization- machine based neural network for CTR prediction. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pages 1725-1731, 2017.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outto--diitributio examples in neural networks. In International Conference on Learning Representations, 2016.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference Artificial Intelligence, pages 3779-3787, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02533 2015.
Chengming Hu, Xuan Li, Dan Liu, Haolun Wu, Xi Chen, Ju Wang, and Xue Liu. Teacher-student architecture for knowledge distillation: A survey. CoRR, has/2308.04266. 2023.
Tao Huang, Shan You, Fei Wang. Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. arXiv preprint arXiv:2205./053 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456, 2015.
Olivier Chapelle Jean Baptiste Tien, joycenv. Display advertising challegge, 2014. URL https ons/ http://///ooooeettoootooottteeee
James M. Joyce. Kullback-Leibler Divergence, pages 720-722. 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Carlos Lassance, Myriam Bontonou, Ghouthi Boukli Hacene, Vincent Gripon, Jian Tang, and Antonio Ortega. Deep geometric knowledge distillation with graphs. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8484-8488 2020.
10

---

Published as a conference paper at ICLR 2024
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distributtion samples and adversarial attacks. Advances in Neural Information Processing Systems, 31:7167-7177, 2018.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distriiutioo image detection in neural networks. In International Conference on Learning Representations, 2018.
Peng Lu, Abbas Ghaddar, Ahmad Rashid, Mehdi Rezagholizadeh, Ali Ghodsi, and Philippe Langlais. RW-KD: Sample-wist wise loss terms re- re-weighting for knowledge distillation. In Findings of the Association for Computational Linguistics: EMNLP, pages 3145-3152, 2021.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. arXiv preprint arXiv:2106. 10494, 2021.
Fuyuan Lyu, Xing Tang, Huifeng Guo, Ruiming Tang, Xiuqiang He, Rui Zhang, and Xue Liu. Memorize, factorize, or be naive: Learning optimal feature interaction methods for CTR prediction. In Proceedings of 38th IEEE International Conference on Data Engineering, pages 1450-1462, 2022.
Shengyi Pan, Thomas Morris, and Uttam Adhikari. Developing a hybrid intrusion detection system using data mining for power systems. IEEE Transactions Smart Grid, pages 3104-3113, 2015.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550. 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Proceed- ings of the 8th International Conference on Learming Representations, ICLR, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 30:5998-6000 2017.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore L. Willke. Out-of-distribution detection using an ensemble of self supervised leave-out classifiers. In Proceedings of the European Conference Computer Vision (ECCV), September 2018.
Yu Wang, Jingjing Zou, Jingyang Lin, Qing Ling, Yingwei Pan, Ting Yao, and Tao Mei. Out-of- distribution detection via conditional kernel independence model. Advances in Neural Information Processing Systems, 35:364 1-36425, 2022.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In Proceedings of the 5th International Conference on Learning Representations, ICLR, 2017.
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba.
Learning deep features for discriminative localization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2921-2929, 2016.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A birs-variance tradeoff perspective. arXiv preprint arXiv:2 102.00650 2021.
11

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.I TASKS AND DATASETS
To demonstrate the effectiveness of our method TGeo-KD, we conduct extensive experiments across three diverse tasks, including image classification, attack detection, and click-through rate (CTR) prediction. The selection of the three tasks is based on their representations of diverse domains with different data natures, underlining the versatility of our proposed method. We use a classic and widely used dataset for each of the above tasks. This section provides the details for each dataset.
The open-source benchmark dataset (Krizhevsky) 2009) is widely used for image classification, which comprises 60,000 color images of 32x 32 pixels each. It is categorized into 100 classes, each containing 600 images. Each class evenly split between the training and test sets, with 500 images for training and 100 for testing. This dataset includes wide variety of objects, animals, and scenes, making it an ideal resource for developing and testing models and algorithms.
CIFAR-100.
ImageNet. ImageNet (Deng et al. 2009) is large-scale dataset commonly used for image classifi- cation, containing millions of labeled images spanning thousands of categories, such as animals and scenes, etc. There are roughly 1.2 million training images, 50K validation images, and 150K testing images.
HIL. The open-source benchmark dataset is generated from the HIL. security testbed (Pan et al. 2015), the representative transmission network with rich operations in a realistic setting, which is widely used to develop proofs-of-coneept for attack detection in industrial control systems. Specifi- cally, this dataset includes 128 multi-sourced features, including 118 synchrophasor measurements and 12 descriptive system logs. The event scenarios can be categorized as three classes: normal operation (class 0), natural fault (class 1), and malicious attack (class 2). Our task is a three-class classification identify the class of each event scenario.
As an imbalanced dataset, the normal and fault samples are randomly oversampled to balance the attack samples in the training set, while the validation and testing sets not oversampled and remain imbalanced. To avoid dominating features, data normalization is essential to be performed to improve the uniformity of features by adjusting all feature values into the range of [0, 1] (i.e., min-max normalization (Toffe and Szegedy 2015)).
Criteo. Criteo is an online advertising dataset released by Criteo AI Lab] It contains feature values and click feedback for millions of display ads and serves as benchmark for click-through rate (CTR) prediction. Each ad within the dataset is characterized by 40 attributes, providing detailed description of the data. The first attribute acts as a label, indicating whether an ad was clicked (1) or not clicked (0), which also automatically forms the two classes in this dataset. The rest attributes consist of integer features and 26 categorical features. To maintain anonymity, all feature values are hashed onto 32 bits. The dataset represents full month of Criteo's 's traffic, offering extensive, real-world insights into online advertising dynamics.
The version we used in this paper is well-known one that used in a competition on CTR prediction jointly hosted by Criteo and Kaggle 2014 Following prior works (Guo et al., 2017 Lyu et al. 2022). the continuous features are first normalized into the range of [o, 1]. Due to the imbalance of this dataset, we adopt the same strategy as HIL to conduct random oversampling to balance the two classes in this dataset.
A.2 EXPERIMENT SETTINGS
Our method TGeo-KD can remain robust performance all datasets, using different hyperparameter settings as shown in Table 7 This paper introduces three metrics for evaluating the student's generalization ability: classification accuracy (ACC), area under the ROC curve (AUC), and negative log-likelihood (NLL). In order to avoid overfitting of the student, we conduct the early stopping
https://alab.crrito.co//ssssssst http://wwwwaggggooooooomooiioiir
12

---

Published as a conference paper at ICLR 2024
Table 7: Hyperparameter settings on different datasets.
Dataset
batch size
optimizer
initial learning rate
weight decay
dim size
CIFAR-100
128
[Adam, SGD)
le-1
5e-4
128
ImageNet
256
Adam
le-1
5e-4
128
HIL
256
[Adam, SGD)
le-2
{1e-2, le-
1e-1,
32
Criteo
1024
Adam
le-3
{le-5, 1e-3, le-2)
64
Table 8: Classification accuracy (%) on DomainNet, All results are reported in terms of average classification accuracy (with standard deviation) over repeated runs.
Method
Clipart
Real
Sketch
Infograph
Painting
Quickdraw
Average
Vanilla KD
50.6+0.2
55.1
40.7
17.9+0.2
42.5
9.8+03
34.3+0.3
ANL-KD
52.5
57.010.
41.9
19.610.2
44.610.3
11.3.0.
37.8
ADA-KD
53.6102
57.9104
42.3
20.5
45.1
12.4
38.710.
WLS-KD
53.1
57.710.
42.8
21.1
45.3
12.0.0.2
38.5103
RW-KD
51.3+03
56.2+0.
41.220.2
18.8+03
43.7
10.6+0.
37. 10.2
TGeo-KD
55.2+0.
59.3+0.4
44.6
21.7400.
46.9
14.1
40.2
strategy when the ACC does not improve on the validation set within 10 successive iterations. Our computation resources include one NVIDIA RTX-3080 GPU and one RTX-3090 GPU.
A.3 GENERALIZATION ABILITY
To demonstrate the generalization ability of our method on unseen data, we conduct additional experiments in the
Detection (OOD) task using DomainNet dataset (Peng et al. 2019). DomainNet (Peng et al., 2019) is a domain adaptation dataset with six domains (including Clipart, Real, Sketch, Infograph, Painting, and Quickdraw) and 0.6 million images distributed across 345 categories. In the experiment setup, we adopt a standard cave-one-domain our student (teacher: ResNet-50 and student: ResNet-18 on the training set of the source domains and evaluate the trained student on images of the held-out target domain. As depicted Table S TGeo-Ko KDD outperform all the relevant KD-based baseline methods.
manner, and train
A.4 MORE ABLATION STUDIES
Student and teacher architectures. In the main paper, we have shown the superiority of TGeo-KD across different architectures and model sizes teachers and students on CIFAR-I01 ImageNet. Here, we provide more results the other two datasets. As shown in Table g proposed TGeo-KD consistently demonstrates strong performance across a range architectural discrepancies between the teacher and student networks.
Table Result comparison (with different gaps between students and teachers) on HIL and Criteo. "S i, j" indicates the number of layers of the student and teacher network, respectively, which serve as a proxy for the size or the number of parameters or capacity of the network. Note that our baselines are built with "S 4.7 12" in the main paper.
Dataset
HIL
Criteo
Gap (in size)
ACC (%) 1
AUC (%) f
NLL $
ACC (%) f
AUC (%) f
NLL |
S 4,7
89.33
69.48
1.02
74.37
74.09
0.97
S = 4, T =
91.72
70.78
0.96
75.51
75.21
0.84
5 = 4, T 12
92.39
71.65
0.94
77.80
77.00
0.81
S 4, T 16
88,65
70.17
1.01
74.20
74.96
1.01
S = 4,7 T
87.92
69.58
1.03
73.28
73.27
1.08
13

---

Published as a conference paper at ICLR 2024
Table 10: Sensitivity temperature т. Top-1 classification accuracy (%) is reported.
Dataset
r=0.5
r=1.0
7=1.5
r=2.0
7=2.5
7=3.0
7=3.5
7=4.0
r=4.5
7=5.0
CIFAR-100
69,80
71.51
71.65
72.30
72.07
72.15
72.09
72.98
72.53
71.22
ImageNet
71.05
71.76
72.89
72.28
72.53
71.74
72.01
71.63
71.82
71.44
HIL
91.04
92.12
92.39
91.70
91.87
91.65
92.05
91.31
91.52
91.10
Criteo
76.72
76.58
77.29
77.80
77.16
77.00
76.89
77.08
76.35
75.97
Sensitivity temperature 7. Table 10blle10come 10leel00oompp compares the top-1 classification accuracy (%) of TGeo-KD using various temperatures (i.e., in the range of [0.5, 5.0]). It can be observed that our proposed TGeo-KD maintains stable performance using different temperatures (i.e., with the variances of 0.69, 0.26, 0.42, and 0.49), which further demonstrates the consistent performance of TGeo-KD across various temperature settings.
Sensitivity to oversampling. We employ oversampling when the dataset is imbalanced, which is widely accepted practice in the research community, especially for classification tasks. To analyze the impact of oversampling on performance, we compare our TGeo-KD with five other baselines across varying imbalance ratios on the HILL and Criteo datasets. As shown in Table 11 our proposed TGeo-KD consistently outperforms all the baselines (e.g.. the improved accuracy of 2.24% and 3. 1.15% and without oversampling on HIL
over the best baseline ADA-KD (Lukasik et al. [021] with dataset, respectively), which underscores the robustness of our TGeo-KD on imbalanced datasets.
Table 11: Top-1 classification accuracy (%) on HIL and Criteo with different imbalance ratios. The imbalance ratio p is defined as the proportion between the sample sizes of the most prevalent class and the least prevalent class, where represents a balanced setting. We calculate the average results (with standard deviation) over 5 repeated runs.
Dataset
HIL.
Criteo
Method
p
0 5
w/o
p
0
w/o
Vanilla KD
87.55
81.72.1.66
72.07
71.08
67.32.
61.01 11.39
ANL-KD
87.27
81.33
71.82
72.71 10.35
68.03
62. 12.18 10.322
ADA-KD
90.15
83.49
75.23
72.15
67.74.0422
61.35
WLS-KD
90.05
82.83
74.7710.27
75.30
71.21
64.52.10.43
RW-KD
89.40
82.22.10.59
73.91 +0.58
75.05
70.58
63.90.10.38
TGeo-KD
92.39
85.65
78.38 =0.21
77.80
74.09
67.77
Representations of Trilateral Geometry. To fully exploit intra-sample and inter-sample relations, we consider multiple representations of trilateral geometry as follows:
R1: S @
To @ G. Ti
R2: e e
+ et
a e ig Seet
. R3: o oc of oe + की + SOS
T + G.
where the explanations of notations can be found in the main paper. We conduct this ablation experiment on all the datasets. As shown in Table 12] our proposed TGeo-KD can achieve the best performance by capturing both the vertex-wise and edge-wise trilateral geometric information (i.e., R3) at the intra-sample and inter-sample levels.
Effect of prediction discrepancy and inter-samples relations on outliers. To analyze the impacts of prediction discrepancy (ST) and sample relations (A s79) on outliers, we compare the Top-1 classification accuracy outliers of CIFAR-100 when learning based on different relations. As shown in Table [3] our proposed TGeo-KD achieves the best performance on outliers when consid- ering both the discrepancy and inter-sample relations. Furthermore, compared solely exploiting
14

---

Published as a conference paper at ICLR 2024
Table 12: Top-1 classification accuracy (%) and AUC (%) comparison among different representations of trilateral geometry.
Dataset
CIFAR-I00 100
ImageNet
HIL
Criteo
Representation
ACC
AUC
ACC
AUC
ACC
AUC
ACC
AUC
R1
70.32
66.24
69.87
64.52
87.52
69.82
73.87
73.39
R2
68.44
64.10
71.28
65.91
87.14
70.26
74.21
74.01
R3 72.98
68.57
72.89
67.14
92.39
71.65
77.80
77.00
Table 13: Comparison of Top-1 classification accuracy (%) on outliers among different relations. The teacher and student are ResNet-32x4 and ShuffleNetV..
SG+TG ST
ASTO
ACC
TGeo-KD
28.46
TGeo-KD
32.15
TGeo-KD
40.71
TGeo-KD
45.89
the discrepancy, inter-sample relations make a better contribution to the improved performance on outliers.
Moreover, to analyze the impact of ST and ATT on the fusion ratios for incorrectly predicted samples, Table 14 reports the mean and standard deviation of the fusion ratios across these samples when learning fusion ratio with and without the consideration of ST and STS When considering both ST and ATS TGeo- KD demonstrates the fusion ratios below 0.3 for incorrectly predicted samples. This suggests that the student acquires more knowledge from the ground truth, leading to decreased fusion ratios on LKD Furthermore, in comparison to solely exploiting s when learning fusion ratios, ST contributes more effectively the decreased fusion ratios across incorrectly predicted samples.
Table 14: Final fusion ratios on incorrectly predicted samples with and without the consideration of prediction discrepancy (ST) and inter-sample relations (A STS.
SG+TG
ST
STg
Final fusion ratio
TGeo-KD
0.58:10.29
TGeo-KD
0.3
TGeo-KD
/
0.43=10.18
TGeo-KD
0.12++0.07
Computational cost. We compare the computational cost of our proposed TGeo-KD with the baselines including vanilla KD (Hinton et al. | (2015). ANL-KD (Clark et al. 2019). ADA-KD (Lukasik et al.) 202T), and WLS-KD (Zhou et al. 2021). We run experiments
CIFAR- 100, and report the mean training time (in seconds) on a batch of 1 128 images per iteration. The student network is ResNet-32. The experiment is conducted one NVIDIA RTX-3090 GPU. As shown in Table 15
our method achieves better training efficiency.
A.5 DISCUSSION
Limitation. Although our proposed TGeo-KD a promising knowledge distillation approach, could be limited by solely considering inter-sample relations within the same class, thereby neglecting relationships across different classes that could potentially enhance performance. This can be regarded one future research direction of this work.
Broader impact. As for the broader impact, with its capability of dynamically learning a sample- wise knowledge fusion ratio across various model architectures, T'Geo-DD holds the potential to
15

---

Published as a conference paper at ICLR 2024
Table 15: Mean training time (in seconds) on a batch for CIFAR-100 per iteration.
Vanilla KD
ANL-KD
ADA-KD
WLS-KD TGeo-KD (Ours)
0.095
0.095
0.101
0.097
0.097
significantly improve the performance of knowledge distillation across a variety of domains, including image classification, attack detection, and click-through rate prediction.
16


Reviews:
[{"review_id": "SRPIb4DiMi", "reviewer": "Reviewer_Ur13", "summary": "This paper discusses the concept of knowledge distillation (KD) and introduces a new method called TGeo-KD for determining sample-wise knowledge fusion ratios in KD. The traditional approach to determining fusion ratios in KD involves using a fixed value for all samples or gradually decreasing the ratio. The authors argue that considering the discrepancy between the student's and teacher's predictions (ST discrepancy) is crucial in determining the fusion ratio. They propose TGeo-KD, which leverages the trilateral geometry among the signals from the student, teacher, and ground truth to model the fusion process. Experiments conducted on CIFAR-100 dataset demonstrate the effectiveness of TGeo-KD compared to other methods. The main contributions of TGeo-KD include its use of trilateral geometry, mitigation of outliers, and superior performance across different tasks.", "questions": "Please see the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper introduces a novel method called TGeo-KD for determining sample-wise knowledge fusion ratios in knowledge distillation. This method leverages trilateral geometry and captures the geometric relations between the signals from the student, teacher, and ground truth. By introducing this new method, the article contributes to the existing body of research in knowledge distillation.\n2.\tThe article emphasizes that TGeo-KD is versatile and adaptable to different architectures and model sizes. This implies that the proposed method can be applied to a wide range of machine learning tasks and scenarios. The ability to apply TGeo-KD to different models and architectures enhances its practical utility and makes it a valuable contribution to the field of knowledge distillation.\n3.\tTGeo-KD leverages trilateral geometry at both intra-sample and inter-sample levels, which helps in mitigating the impact of outliers in the training samples. Outliers are samples that deviate significantly from the majority of the dataset and can negatively affect the learning process. By incorporating the trilateral geometry and considering the ST discrepancy, TGeo-KD provides a robust approach that is less sensitive to outliers, resulting in more stable and reliable knowledge transfer.", "weaknesses": "Complexity and Computational Cost: The TGeo-KD method proposed in the article leverages trilateral geometry and introduces a neural network to learn the fusion ratios. This suggests that the method may have a higher computational cost compared to simpler knowledge distillation techniques. Implementing and training the neural network for determining fusion ratios may require additional computational resources and time, which could be a potential drawback, especially in resource-constrained environments.\n\nThe proposed TGeo-KD leverages trilateral geometry and captures the geometric relations between the signals from the student, teacher, and ground truth. However, I think it lacks more sufficient theoretical proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3uCcfEcuFS", "reviewer": "Reviewer_UyVS", "summary": "This paper studies a less explored topic in Knowledge Distillation that of finding the optimal parameter/weight alpha for combining the distillation loss with the standard cross entropy loss when training the student. The authors first motivate the impact of appropriately learning the weight parameter with a well-designed experiment and then they propose a practical method to estimate it using a small network that uses as input features geometric features computed from the teacher's and student's logit and the ground truth one-hot vectors. They show good results on a variety of standard KD benchmarks", "questions": "Inter sample relations are used to model outliers. But how do you define outliers in the datasets you're using for your experiments (i.e. imagenet and cifar)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. Paper is well-written and motivating example makes sense\n2. Problem tackled is not well-studied and the solution proposed is simple and effective\n3. Results carried out on standard benchmarks show that the method is effective on these benchmarks", "weaknesses": "1. My main concern is related to the impact of the proposed solution. Like most distillation papers conclusions are drawn based on CIFAR-100 experiments using same network architecture. Claims made by the authors could substantiate better if they are shown to hold on bigger datasets like ImageNet.\n2. Tasks other than classification should be considered (e.g. detection). Experiments on HIL and Criteo datasets are of low impact \n3. Important references/comparison with SOTA is insufficient. Authors should have included comparisons with Decoupled Knowledge Distillation, CVPR'22 and KNOWLEDGE DISTILLATION VIA SOFTMAX REGRESSION REPRESENTATION LEARNING, ICLR'21. From my preliminary checking, the improvements of the proposed method are not always so significant if the aforementioned methods are considered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vijrgPDPUB", "reviewer": "Reviewer_iMjj", "summary": "This paper focuses on data-wise adaptive knowledge fusion ratio in knowledge distillation. They find that the discrepancy of the predictions from teacher and student plays an important role, and they design a novel KD method based on this. Experiments show the effectiveness of their proposed method.", "questions": "- As far as I know, the computation of similarities of class probability distribution vectors often adopts the cosine similarity. Why do they use the Euclidean distance here? Are there some advantages over cosine similarity?\n- As they state that Eq. 4 is an implicit function of ω as θ* depends on ω, how to perform alternately optimization?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- They propose a novel KD method to exploit the discrepancy of the prediction logits from S and T to adaptively learn the knowledge fusion rate, offering an valuable view for KD.\n- The experiments are solid and achieve good performance.\n- More significantly, the proposed sample-wise operation introduces little additional training time, making this method useful in practice.", "weaknesses": "- How do the outliers hurt KD? As a key motivation, the corresponding empirical evidence seems lacking.\n- In addition, the teacher model indeed may underperform on the outliers. According to previous discussion, directly decreasing the $\\alpha$ also can solve this issue. Thus, why to introduce the inter-sample relations? This motivation needs to be further claified.\n- Evaluating this method on real OOD datasets, like DomainNet, will be more convicing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CJ2Mv7qVL", "reviewer": "Reviewer_aFDp", "summary": "In this paper, the authors propose an adaptive method for learning a sample-wise knowledge fusion ratio. Specifically, the proposed method captures intra-sample and inter-sample trilateral geometric relations among the student prediction, teacher prediction, and ground truth. A simple network further learns the implicit mapping from the intra- and inter-sample relations to the fusion ratios in a bilevel-optimization manner. The proposed method is validated on three various tasks: image classification on CIFAR-100 and ImageNet, attack detection on HIL, and CTR prediction on Criteo.", "questions": "1.\tWhy do the authors choose to model the relation in Euclidean space? \n2.\tGiven that this paper proposes an adaptive learning method for sample-wise fusion ratio, does the proposed method demonstrate consistent performance across different numbers and categories of samples?\n3.\tIs there a specific reason to formulate this work into bilevel optimization? Will it lead to less efficient training? The motivation is note quite clear to me.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well motivated by clearly illustrating the valuable insights of incorporating additional ST prediction discrepancy for determining the fusion ratio.  \n2.\tAlthough KD methods have been widely studied, the investigation of the knowledge learning potential on different samples has received less attention. With the learned sample-wise fusion ratio, this paper showcases that the student and teacher can provide specific knowledge learning potential for each sample based on the ST prediction discrepancy and the teacher’s correctness. \n3.\tThis paper proposes a novel and interesting method to capture relations among the student prediction, teacher prediction, and ground truth by exploiting their trilateral geometry at both intra- and inter-sample levels. \n4.\tThe experiments and analysis on three different tasks are comprehensive and demonstrate the effectiveness of the proposed method. \n5.\tThe paper is well written and easy to follow.", "weaknesses": "The paper is overall readable and proposes an innovative idea. However, there are some weaknesses that the authors can improve. \n\n1.\tThe experimental results show improved performance on the HIL and Criteo datasets. However, the paper does not clarify if this is due to the adopted sampling method. The oversampling of the minority class as a preprocessing step is mentioned, but its direct influence is not explored.\n\n2.\tIn Section 4.3, this paper analyzes the effectiveness of incorporating ST by comparing the learned fusion ratio distributions across four scenarios. With ST, the proposed TGeo-KD reduces the fusion ratio on L_kd, thereby encouraging the student to acquire more knowledge from ground truths when the teacher predictions are incorrect on samples (e.g., outliers). However, more experiments should be conducted to investigate that the exploitation of inter-sample relations on these incorrectly predicted samples does not contribute significantly to decreased fusion ratios. \n\n3.\tThe fusion ratio is learned through a neural network, which is unclear to explain how the learning process unfolds, such as which samples are assigned larger/smaller fusion ratios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)