ModReduce: A Multi-Knowledge Distillation Framework with Online Learning
Authors: Yahya Saad Abbas
Abdelhakim Walid Badawy
Mohamed Mahfouz Shahawy
Samah Hussien
Samah Ayman
Hesham Mohamed Eraqi
Cherif Salama
Venue: Submitted to ICLR 2023 (2023)
Keywords: Knowledge distillation
Deep neural networks
Model Compression
Knowledge transfer
Online Learning

Abstract: Deep neural networks have produced revolutionary results in many applications; however, the computational resources required to use such models are expensive in terms of processing power and memory space. Research has been conducted in the field of knowledge distillation, aiming to enhance the performance of smaller models. Knowledge distillation transfers knowledge from large networks into smaller ones. Literature defines three types of knowledge that can be transferred: response-based, relational-based, and feature-based.
To the best of our knowledge, only transferring one or two types of knowledge has been studied before, but transferring all three remains unexplored. In this paper, we propose ModReduce, a framework designed to transfer the three knowledge types in a unified manner using a combination of offline and online knowledge distillation. Moreover, an extensive experimental study on the effects of combining different knowledge types on student models’ generalization and overall performance has been performed. Our experiments showed that ModReduce outperforms state-of-the-art knowledge distillation methods in terms of Average Relative Improvement.

Under review as a conference paper at ICLR 2023
MODREDUCE: A MULTI- DISTILLATION FRAMEWORK WITH ONLINE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have produced revolutionary results in many applications; however, the computational resources required to use such models are expensive in terms of processing power and memory space. Research has been conducted in the field of knowledge distillation, aiming to enhance the performance of smaller models. Knowledge distillation transfers knowledge from large networks into smaller ones. Literature defines three types of knowledge that can be transferred: response-based, relational-base basee, and feature-based. To the best of our knowl- edge, only transferring one or two types of knowledge has been studied before, but transferring all three remains unexplored. In this paper, we propose ModReduce, framework designed to transfer the three knowledge types in unified manner using combination of offfine and online knowledge distillation. Moreover, an ex- tensive experimental study on the effects of combining different knowledge types on student models' generalization and overall performance has been performed. Our experiments showed that ModReduce outperforms state-of-the-arr knowledge distillation methods in terms of Average Relative Improvement.
INTRODUCTION
The term knowledge distillation was formally popularized in the work of Hinton et al. (2013). and it refers transferring knowledge from a large pre-trained model to a smaller one, aiming to retain comparable performance to the large model. Knowledge distillation has been receiving increas- ing attention from the research community due to promising results. The methods by which knowledge can be distilled vary widely based on several factors like knowledge type, the distilla- tion algorithm, and the teacher-student architecture (Gou et al., 2021). Response-based knowledge uses the large model's logits as the teacher model's knowledge. The main idea is that the student optimizes its training over the soft targets, or the softened probability distribution, produced by the teacher model instead of using discrete labels (Hinton et al. 2015). While this method showed great success, one of its major drawbacks is that disregards the knowledge a teacher model retains in its intermediate layers. This encouraged researchers introduce methods that capture the knowledge in the intermediate layers of the teacher model, feature knowledge. Feature-based algorithms focus on the features of the teacher model's intermediate layers guide the student's learning. The chal- lenge is that the teacher and the student models have different abstraction levels, which makes it one of the objectives of the distillation process to determine the best layer associations for maximum performance (Tung & Mori] 2019 Passalis et al., 2020 Kornblith et al. 2019). Relational-based methods focus on the relationships between different data instances and different activations and neurons (Gou et al. 2021). Several algorithms and methods have been introduced, focusing on dis- tilling one or two the knowledge sources from teacher model to a student model. While they show promising results, research addresses the issue of distilling the three discussed knowledge types instead of only one or two. To the best of our knowledge, ModReduce is the first work to ex- plore this area. Moreover, this goal. For online learning distillation, we have explored four different techniques for knowledge aggregation: Peer Collaborative Learning (PCL) (Wu & Gong 2021), On-the- fly Native Ensembling (ONE) (Zhu et al. 2018), Fully Connected Layers (FC), and Weighted Averaging.
explore combining offline and online distillation strategies achieve
ModReduce exhibits the following characteristics:

---

Under review as a conference paper at ICLR 2023
1. Agnostic of the offline distillation algorithm used. The platform supports different imple- mentations for each knowledge category distillation, including tate-off-the-eee algorithms.
2. Capable of executing different online learning algorithms.
3. Agnostic of the teacher and student model architectures.
Moreover, we introduce a new benchmark which is a union of the benchmarks available at CCen
al. 20211; Tian al. 2019). With this broad benchmark, several findings have been concluded.
2
RELATED WORK
Before digging deeper into different knowledge distillation schemes, it is better to explore the dif- ferent knowledge representations neural network possesses.
2.1
KNOWLEDGE SOURCES
2.1.1 RESPONSE-BASED
Response-basel knowledge is defined as the response of neural network whenever it is presented with a particular input. The most popular response-based knowledge distillation scheme in image classification tasks using "soft targets," where the aforementioned probability distribution is con- trolled using a temperature factor, T, which is used to soften the probability distribution. According to Hinton et al. (2015), these soft targets provide informative dark knowledge from the teacher model. The soft targets "Hinton" approach is the most popular approach with the best results thus far utilizing this type of knowledge. Unfortunately, this kind of knowledge is blind to the inner features in the hidden layers of the model, as it only focuses on the final outputs.
2.1.2 FEATURE-BASED
Feature-based knowledge extends on the idea of response-based knowledge and takes the outputs of intermediate layers into consideration. Accounting for the outputs of the intermediate layers is important because deep neural networks can learn features with different levels of abstraction. For instance, a deep CNN can learn abstract features like straight and curved lines the shallowest layers while detecting features with higher complexity the deeper layers (Bengio et al., 2013). This idea is useful for constructing teacher-student architectures since this type of knowledge can be used in the training of the student network. Many knowledge distillation techniques loss function that accounts for feature-based knowledge. Equation represents general form of the distillation loss function for feature-har based knowledge, where fe(x)) and (1: are the feature maps of the teacher and student models respectively; o; and d, represent the transformation function of the teacher and student models feature maps and (.) is similarity measure between the feature maps.
distillation
(ffe) = (y o.
The state-of of-t-e--ttttttttt framework in feature knowledge distillation was introduced in Chen et al. (222I) which aimed to match the semantics between the teacher and student. They introduced semantic calibration for cross-layer knowledge distillation that made better use of the intermediate knowledge by matching the semantic level of the transferred knowledge. Then they used an atten- tion mechanism automatically learn soft layer association with multiple targets, which helped the student model learning from multiple semantically matched hidden layers instead of just one fixed layer.
2.1.3 RELATION-BASED
While the response- based distillation captures the knowledge in the output layers of the teacher model, and feature- based distillation captures knowledge contained in the intermediate layers, the relational-based distillation captures the interrelations between training data examples. Several tech- niques have been proposed to capture the relations between the training data. Instance Relationship Graph (IRG) (Liu et al., 2019) introduced a methodology of knowledge distillation based on structing a graph where features are represented as vertices and relations as edges. Relational Knowl- edge Distillation (RKD) (Park al. 2019) proposed measuring the relations between training data
2

---

Under review as a conference paper at ICLR 2023
instances using distance- wastacce-wiseeaee and angle-wise losses that penalize structural differences in relations. Contrastive Representation Distillation (CRD), which is the state-of-the-ar the-art in relational knowledge distillation (Tian et al. 2019). captures important structural knowledge of the teacher network. It trains a student to capture significantly more information in the teacher's representation of the data using objective contrastive learning, which encourages the student to map similar inputs close representations, in some metric space, while mapping different inputs to distant representations.
2.2 DISTILLATION SCHEMES
In this section, we overview the existing distillation schemes in the literature, which can be divided into three schemes: offline distillation, online distillation, and self-learning distillation. We focus on the first two schemes since they are more relevant to our target architecture.
2.2.1 OFFLINE DISTILLATION
This is the most basic and popular kind of distillation. It was introduced alongside the concept of distillation by Hinton et al. (2015). This scheme transfers knowledge from pre-trained expert teacher model into a student model. The whole training process takes place in two phases; first, the training of the teacher model on the set of training samples before distillation. The second phase extracting knowledge from the teacher model and passing it to the student model. The knowledge extracted from features, responses, or relations as mentioned in section 22. Offline distillation is simple and straightforward to implement as it employs one-way knowledge transfer from a trained teacher. In addition, the student model is usually smaller in size and simpler to train. On the other hand, in this scheme, the model capacity gap always exists and cannot be avoided due to the difference in complexity between the teacher and student models. Model capacity could be defined measure of a DNN size based on the number of nodes and layers. The offline distillation methods focus on improving knowledge transfer in several aspects. One aspect is regarding the design of the student model and the knowledge type. For instance, in Romero et al.) (2014), the student model is deeper than the teacher model but it is much thinner at the same time. Hints from the inner hidden layers of the teacher model are taught to the student model guide the training process. Another aspect is the loss functions for matching features or distributions matching.
2.2.2 ONLINE DISTILLATION
In the online distillation scheme, both the teacher and student models are being trained and updated simultaneously. Online learning has been proven to improve the generalization ability network by training it simultaneously with a pool of other networks. Moreover, online learning supports heterogeneity in student networks as they can vary in architecture and size. Several techniques have been proposed for the online learning scheme, such as Peer Collaborative Learning (PCL), On-the-fly Native Ensemble (ONE), and weighted averaging. Peer Collaborative Learning (Wu & Gong 2021) integrates online ensembling and network collaboration into unified framework. PCL constructs a multi-branch network for training, in which each branch is called a peer. Random aug- mentation multiple times is performed on the inputs peers and assemble feature representations outputted from peers with an additional classifier as the peer ensemble teacher. Moreover, PCL employs the temporal mean model of each peer as the peer mean teacher collaboratively transfer knowledge among peers, which helps each peer to learn richer knowledge and facilitates optimizing a more stable model with better generalization. In ONE (Zhu et al.) 2018), training is only a sin- gle multi-branch network while simultaneously establishing a strong teacher on-the-fly to enhance the learning of the target network. The auxiliary branches share the low-level layers with the tar- get network, with each branch, together with the shared layers, acting as an individual model. The ensemble of those branches builds the teacher model. The training is performed in a closed loop fashion where the teacher aggregates knowledge from branch models on-the-fly, and this knowl- edge is distilled back to the branches to enhance the models' learning. Evaluations of ONE report enhancement of the generalization performance while maintaining the computational efficiency.
3

---

Under review as a conference paper at ICLR 2023
Teacher Model
Distill Knowledge
via Offline Learning
Response-baree
Student Model
Student Model
Student Model
Leaming
Figure 1: ModReduce Architecture
3 MODREDUCE
3.1 FRAMEWORK STRUCTURE
The structure of our framework is follows, one cumbersome teacher model that already pre- trained and has high accuracy in the specific task and three identical untrained student models. These student models are smaller in size than the teacher model. Notably, the student models under the standard training process obtain lower accuracy than the teacher model. The framework operation divided into two phases, the offline distillation phase, and the online distillation phase. Both these phases are performed consecutively at each training step. First, offline distillation is performed between the teacher model and each student model separately, using different knowledge sources for each student. Then, online distillation is performed among the student models to aggregate the knowledge gained in the offline phase. For online learning, we implemented different techniques as mentioned in section 222
3.2 OFFLINE TRAINING SETUP
This section will go through the functionality and loss calculation of each offline distillation method used our algorithm. We used Hinton for response distillation, SemCKD for feature distillation, and CRD for relational distillation.
3.2. HINTON
By "Hinton" here we refer to the vanilla KD, which depends on the loss calculated between the logit layers the teacher and student models. This method uses the outputs of the final Softmax layer of the teacher, which contains the probabilities for each class (in classification task), then applies a temperature for these probabilities to convert them into the soft targets. Equation shows how to obtain a soft target qi, where T is the softening temperature, i is the index of the class, 2; is the logit computed for class i, and is the probability of class i. If T' is greater than 1, we obtain gi values that are softened probabilities (i.e., soft targets).
T
q
Σ
3.2.2 SEMCKD
As feature knowledge distillation method, SemCKD is concerned with transferring knowledge from the intermediate layers of the teacher to the students. Moreover, it employs an attention mech- anism solve the problem of semantic mismatch caused by the difference in teacher and student
4

---

Under review as a conference paper at ICLR 2023
architectures which could lead to degradation in performance. The attention mechanism auto- matically assigns layers from the teacher model for student layers to learn from. In addition to the attention mechanism, each student layer learns from multiple layers in the teacher model to add cross-layer supervision (Chen et al. 2021).
3.2.3 CRD
As for relational, or, structural knowledge, CRD is the current state-of-the-art. The original response-based knowledge transfer proposed by Hinton al. (213)) ignores the complex inter- dependencies between the data instances, problem CRD tries to solve by leveraging a contrastive objective to capture this output correlation (Tian al. 2019).
3.3 ONLINE TRAINING SETUP
The online learning phase enhances the generalization of the student models by sharing the knowl- edge gained by the other models the cohort during the offline phase. We have explored four different online learning techniques inspired from different sources to find the best one for our goal. These four techniques are PCL, ONE, FC, and Weighted Averaging.
3.3.1 PCL
This technique was inspired by Wu & Gong 202I). In it, the students try learn collaboratively from each other by employing a temporal mean model copy as representation for each student. In the online learning phase, each student tries mimic the soft logits of the temporal mean models of its peers.
3.3.2 ONE
In this technique, inspired by Zhu et al. (2018), inputs and predictions are used to learn a weight for each student. Those weights are used to produce group output from the individual student predictions, which is then used as guide for the students to mimic.
3.3.3 FC
Similar to ONE, this technique tries to produce a better group output from individual student pre- dictions for the students to follow. However, this objective is achieved here by employing dense layer to aggregate the three individual predictions into single one.
3.3.4 WEIGHTED AVERAGING
This technique has the same objective as ONE and FC, sharing knowledge between the students by aggregating their predictions into group output that each student is penalized against. As its name suggests, we here try to learn weight for each student, with the weights being from to I and having a total sum of Such goal is achieved by having learnable weight for each student, and those weights are optimized based on students' performance.
3.4 ALGORITHM
Algorithm shows ModReduce training detailed procedure.
EXPERIMENTAL SETUP
To verify our proposed hypothesis and demonstrate the effectiveness of our novel framework, we designed flow for experimentation. The flow chart in figure 2 describes the entire rationale of each stage.
We started with original hypothesis question: "Does aggregating three knowledge sources in one loss function yield better results than distilling only one knowledge source?". If the answer was yes, then this is definitive answer to the hypothesis that offline aggregation of the three knowledge

---

Under review as a conference paper at ICLR 2023
Algorithm ModReduce Algorithm
1: Load pretrained TeacherModel
2: $ == (Response Distillation Model, Relational Distillation Model, Feature Distillation Model)
3: Create StudentModellx] (x E
4: for epoch in epochs do
5: for batch in train_data do
6: t predictions
7: for E S do
8:
StudeemMoodlll///pecccciooon = StudemtMooddllI)
9:
StudentMoode(x)) loss StudentNNooel])).ccomppee wow.:ueeeefoiieeoosssss. end for group_out
10:
wttp://www...puuuuudduuuddeeepee Vz S)
12:
for æ E S do
StudentModel( loss
cale online los(SSudeetMooddl(///peedddedeee
group_out)
StudentMoode((xx.otlll).
online weiis * StudentMoodllxx/ooniinn..no of-
fline weigh SSudentModellxx.coopptte .offline
15:
end for
16: end for
17: end for
Does aggregating knowledge sources in one loss function yield better results than disfilling only one knowledge source?
NO: Does online aggregation of 3 knowledge sources achieve higher accuracy than knowledge type?
YES: Offline aggregation of 3 losses in 1 student is the best aggregation method
NO: Distilling knowledge type is better than aggregation of knowledge sources
YES: Which online learning method is the best?
PCL
FC
ONE
W Avg
Figure 2: Experimental Flow
sources is sufficient and produces better results than distilling only one knowledge type. With that established, would not need any further experimentation. If the answer no, then need to explore whether changing the aggregation method would enhance performance. So, the question now becomes: "Does online aggregation of three knowledge sources achieve better results than only one knowledge source?" If the answer to this question was no, then that would be a definitive answer falsify our original hypothesis and clearly show that distilling only knowledge type is better than distilling three knowledge types. However, if the answer was yes, this proves our proposed hypothesis (that aggregating three knowledge sources using online learning shall produce better results than only one knowledge source). After ensuring that the obtained results using online aggregation of three knowledge sources are higher than the state-of-the-arr benchmarks, we can further investigate which online learning technique achieves the highest performance. Addressing this question, we integrated four different online learning algorithms to test the performance of our model. We experimented with Peer Collaborative Learning (PCL), Fully Connected layers (FC), On-the-fly Native Ensemble (ONE), and Weighted Averaging.
The of-te--f-thearrr methods compared against are Hinton (in response- -ased knowledge distilla- tion), SemCKD (in feature- based knowledge distillation), and CRD (in relational- knowledge distillation). We created new benchmark that combines the experiments conducted by CRD and

---

Under review as a conference paper at ICLR 2023
Table 1: Test accuracy (%) of students and teachers with similar architectures for Hinton, Sem- CKD, CRD, and ModReduce. f indicates outperforming Hinton's 'ss vanilla KD, whereas + indicates underperforming
Teacher WRN-40-2
WRN-40-2
resnet56
resnet
resnet 10
resnet32x4
vggl3
Student
WRN-16-2
WRN-40-1
resnet20
resnet20
resnet32
resnet8x4
vgg8
Teacher
75.61
75.61
72.34
74.31
74.31
79.42
74.64
Student
73.26
71.98
69.06
69.06
71.14
73.09
70.46
Hinton
75.39
74.21
71.70
70.99
73.66
74.32
73.62
SemCKD
75.10 (L)
73.11 (L)
70.91 (L)
70.95 (L)
73.47 (1)
75.55 (1)
74.08 (1)
CRD
76.12 (1)
74.91
71.72
71.35 (1)
73.65 (L)
74.97 (1)
74.39 (1)
ModReduce
75.44 (1)
74.84 (1)
71.99 (1)
72.01 (1)
74.34(1)
75.78 (T)
74.64 (1)
Table 2: Test accuracy (%) students and teachers with different architectures for Hinton, Sem- CKD, CRD, and ModReduce. 1 indicates outperforming Hinton's vanilla KD, whereas | indicates underperforming
Teacher
vgg13
resnet32xd
WRN-40-2
resnet322c
resnet3224
resmet32s4
vggl3
WRN-40L
Student
MobileNerV2
ShuffeeNecV
ShutticNetV1
ShutlleNetV2
vgal
vgg13
ShuffleNetV2
MobileNeV2
Teacher
74.64
79.42
75.61
79.42
79.42
79.42
74.64
75.61
Student
64.60
70.50
70.50
72,60
70.46
74.82
72.60
65.43
Hinton
68.72
74.59
75.45
75.73
72.48
77.21
75.89
69.02
SemCKD
68.66 (i)
77.21 (1)
76.93 (1)
78.07 (1)
75.02
79.14 (*)
76.24
69.77
CRD
69.66 (1)
75.77 (1)
76.59 (1)
16.57 (1)
73.68 (1)
77.71 ())
76.26
70.13 (1)
ModRedince
69.23 (1)
76.96 (1)
77.14
78.23 (1)
75.21 (7)
79.51 (
76.76
69.37
SemCKD on CIFAR-100. Hence, our new benchmark contains 15 different combinations of teacher and student models architectures along with the accuracy of the expert teacher model, base student model, Hinton model accuracy, SemCKD model accuracy, CRD model accuracy, and our model "ModReduce" accuracy. The results collected our new benchmark are as reported in tables] I and 2. where the bold values represent the highest achieved accuracy. Our model surpasses the state- of-the-art benchmarks in 10 out of 15 experiments. It is worth noting that the offline training of different students reported in tables I and weight for Hinton loss was added to both SemCKD and CRD students.
This done to account for the fact that SemCKD reported their results with Hinton loss added to theirs in offline training. Furthermore, CRD reported training their student with CRD and Hinton that slightly improved upon CRD alone. Therefore, preferred re-running all the experiments for Hinton, SemCKD + Hinton, and CRD Hinton as they represent the offline knowledge distillation schemes with the best reported accuracies.
RESULTS AND DISCUSSION
In all the experiments, the 15 teacher-student models combinations shown in tables [nd were used to obtain the conclusions for questions. To capture the improvement of our model over the existing knowledge distillation techniques, we utilized Average Relative Improvement (ARI); metric that first introduced by CRD (Tian al. 2019) and was later used by SemCKD (Chen et al., 2021) in reporting their results. ARI provides a measure to test whether, on average, for the of different architectures, ModReduce improved upon certain knowledge distillation technique or not.
M Acc'sodReduce Acc 100%
1 M
ARI
(3)
Acck - AAc'st..
The first question was whether direct aggregation offline losses introduced by Hinton, CRD, and SemCKD in single loss function could improve upon using each loss function independently. For that, we ran an experiment that performs an aggregated distillation by adding the loss functions of Hinton, SemCKD, and CRD. Figure shows that aggregating the three losses by simply adding the loss functions with their weights improves only upon the Hinton model. At the same time, has equivalent performance SemCKD and lower performance than CRD. From this experiment,

---

Under review as a conference paper at ICLR 2023
conclude that simply combining the loss functions of the different knowledge distillation techniques does not improve the accuracy of the student model.
Offline different kowledge sources aggregation ARI over Hinton, SemCKD CRD
SemORD
CRD
15 5.00%%
10.00%
-0.10%
8.00%
10.00%
10.00%
Figure 3: Average Relative Improvement of Combined Loss Function (Hinton + CRD + SemCKD) over each Method of them on its own
The next step was testing whether online aggregation of offline knowledge sources could improve the student's s accuracy. We used four aggregation methods create the teacher logits for the online learning step. Two of them were based upon the Peer Collaborative Learning (Wu & Gong (021) and On-the-Fly Native Ensembling (Zhu et al., 2018). The second aggregation method used fully connected trainable layer to calculate the online teacher logits. Figure4ssowss, graph of the average relative improvement of the different aggregation methods used with ModReduce. ModReduce with ONE and Weighted Averaging (WAvg) have positive ARI compared to training student using any the underlying single offline methods. Furthermore, using ModReduce with WAvg has the highest ARIs over all the underlying offline methods; those being 48. 2%, 25.50% and 17.46% over Hinton, SemCKD, and CRD, respectively.
ModReduce different Online Learning Methods
Alten
SemCi(
ARR
WHithed
Tree Linering Medrrod
Figure 4: Average Relative Improvement (ARI) of ModReduce over SOTA Methods using Different Online Learning Techniques
Tables [and 2 show detailed results for the different experimentations. ModReduce, with WAvg as the aggregation method for the online learning step, is better in 10 out of the 15 experiments. While analyzing the five experiments which ModReduce- WAvg is not the best performing, we observed that it also never the worst performing compared to the other methods. For instance, in Figure (ealu 5a despite trailing CRD, we notice that ModReduce- WAvg has an accuracy of 69. 69.23%, improving up on SemCKD.
8

---

Under review as a conference paper at ICLR 2023
On the other hand, Figure showwss shows the opposite case where ModReduce-Wan Wavi is trailing SemCKD. However, with an accuracy 76.96%, improves upon the state-of satte-of-thh-ant the-art relational knowledge distillation (CRD).
(96) vggia
MobieleetV2
(#7) Teacher: resnet32x4
=
Student: ShuffleNervi
Aecunney
Accunecy
Hinter Accunct
Haton Accuracy
DemDSD
CRB Accuncy
09:45
(Magheed Avg)
64.23
(Nhiglens Axgl
09
72
10
77
04
Accunwry
Accuricy
(a) VGGI3 teacher and MobileNetV2 student (b) Resnet32x4 teacher and ShuffleNetVi student
Figure Accuracy of student trained with ModReduce against Hinton, CRD, and SemCKD in experiments where ModReduce did not beat the Best performing method
These results, combined with the above ARI results, show that combining different offline knowl- edge students through online distillation generalizes better than a single offline knowledge distilla- tion technique; thus, on average, ModReduce- WAvg produces better student models.
6 CONCLUSION
In this work, we introduced ModReduce: novel kuotii-knowledge distillation with online learning framework. ModReduce aggregates knowledge distilled from three different sources: response- based, feature-based, and relational-based knowledge. This aggregation is performed via online learning between students, which also boosts their performance and enhances their generalization ability. Our experiments proved that using online learning knowledge sources better than combining the losses in a single student. We also showed that distilling three knowledge types is better than only using one the state-of-the-art SemCKD and CRD distillation schemes in 10 out of 15 experiments. More specifically, ModReduce outperforms SemCKD in out of experiments and outperforms CRD in 7 out of 11 experiments. Using the Average Relative Improvement (ARI) metric, ModReduce achieved 28.29% improvement over Hinton, 25.5% improvement over SemCKD, and 17. 17.64% over CRD.
aggregation method for different
types. Our results surpass
7 FUTURE WORK
Knowledge distillation is a prominent field with many research opportunities that could result in better performance and less costly architectures. Even though ModReduce has already achieved results that surpass the state-of-the-art benchmarks, wide range of potential enhancements can be conducted.
We propose investigating the effect of switching from a synchronous offline-online training scheme sequential one (offline followed by online,) a potential enhancement to be tested. We have only replicated the experiments reported in SemCKD and CRD benchmarks in work. However, other variations teacher and student model architectures can be tested for further insights. Training student with different architectures, given experiment, might also be good point to explore. Finally, we propose doing an extensive ablation study the effect of the different components of the system on the eventual result. This ablation study can test the effect of changing all variables or parameters one at a time.
the three students trained had the same architecture in any

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588):1988-1828, 2013.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang. Zhe Wang. Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7028-7036 2021.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision
2021.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025: 020331, 2(7), 2015.
Simon Kornllihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7096-7104, 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2339-2348, 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.65500 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910. 10699, 2019.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1365-1374, 2019.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10302-10310, 2021.
Xiatian Zhu, Shaogang Gong, al. Knowledge distillation by on-the-fly native ensemble, Advances in neural information processing systems, 31, 2018
10

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASET
To accomplish the abovementioned experimentation rationale, ModReduce was tested on series of classification tasks using the CIFAR-100 100 dataset, which consists of 100 classes containing 600 images each. CIFAR-100 has 500 training images and 100 testing images per class. The 100 classes grouped into 20 superclasses. Each image comes with "fine" label (the class to which it belongs) and a "coarse" label (the superclass to which belongs). CIFAR-100 is the most used dataset in the state-of-the-ar benchmarks we compared our results against, making it convenient choice for our experimentations.
A.2 PREPROCESSING
To have fair comparison, we implemented identical pre-processing main benchmarks in Chen et al. (2021): Tian et al. (2019). All images are normalized by channel means and standard deviation. Moreover, some data augmentation is also implemented, such as random cropping and horizontal flipping.
A.3 TRAINING DETAILS
Following our benchmarks, we use stochastic gradient descent with a Nesterov momentum of 0.9. We the initial learning rate 0.01 for MobileNetV2, Shuff and 0.05 for other chitectures. The number of training epochs is 240 for all models, and the learning rate is divided by 10 epochs 150, 180, and 210. We the mini-batch size 64 and the weight decay to 5 10 10-4, The hyperparameter 3 of SemCKD is set to 400, while the temperature T of Hinton KD is set to throughout our experiments.


Reviews:
[{"review_id": "olTahsidV-h", "reviewer": "Reviewer_Ggs6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel two-stage knowledge distillation approach using a teacher network and multiple students which, in essence, combines offline response-based, feature-based and structural knowledge distillation using an online scheme. The authors do so by evaluating for online schemes, namely PCL, ONE, FC, and Weighted Averaging. The authors evaluate their approach on CIFAR100 for selected teacher and student architectures. The experiments show that the combination of the three offline schemes only works well when using an appropriate online scheme, but then can provide better results compared to individual offline knowledge distillation strategies.", "summary_of_the_review": "The paper is proposes an intersting and well-motivated approach for combining available offline knowledge distillation approaches. The authors nicely make the case for the possible merit of combining respnse-based, feauture-based and relational knowledge and propose a sensible system to achieve this goal. The paper shows the relative improvements of the trained models compared to chosen offline distillation approaches, but I am lacking a clear result on the advancement of the knowledge distillation field. What are the consequences of the results? The authors state that the individual appraches (e.g. SemCKD and CRD) are the respective State-of-the-Art, but can the reader assume there are no other combinations / individual knowledge distillators that are better? Maybe this can easily be remedied in the related work section and clear statements in the results, but in the current version it is unclear to me. ", "strength_and_weaknesses": "Strengths:\n* The approach is well-motivated and is well-designed. The proposed architecture enables to reuse online schemes for updating individual students with specialized knowledge\n* The empirical evaluation uses sensible architectures and tasks, and show results for the CIFAR dataset. The results show an improvement against individual offline knowledge distillation schemes\n\nWeaknesses:\n* The paper does not make clear how the paper compares against the global State-of-the-Art of knowledge distillation and only shows relative improvements. It would be important to discuss / show this in the paper. More specifically: Did you compare to SOTA for Online KD? The results improve wrt to offline KD, but the added value for KD approaches as a whole are not discussed in depth.\n* A second point for the evaluation: It is also mentioned that two types of knowledge have been combined before, but this is not covered in the evaluation. How would your system perform against these?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YCc3_v6AnQS", "reviewer": "Reviewer_9MXc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to transfer teacher knowledge in response, feature, and relation together.", "summary_of_the_review": "n/a", "strength_and_weaknesses": "Very bad paper, no novelty.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "E9Cd5yGRLDv", "reviewer": "Reviewer_5hfu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A framework called ModReduce was proposed to distillate knowledge from a teacher model by combining response-based, feature-based, and relation-based methods. The proposed distillation process is also divided into two phases: an offline distillation followed by an online distillation. The experimental results show that combing three different distillation methods (i.e., using three types of loss each defined for a method) yields better results than that only using one or two methods.", "summary_of_the_review": "This work just combines existing distillation methods by simply using multiple losses that were proposed in previous literature at the same time, and tries some combinations of them. The novelty seems a bit insufficient. Besides, experimental results show a slight increase in accuracy on just one dataset.", "strength_and_weaknesses": "Strength:\n\n(1) The proposed framework, named ModReduce, is simple and easy to implement.\n\n(2) They show that combining three different distillation methods (response-based, feature-based, and relational-based knowledge) can improve the performance of student models \n\n(3) ModReduce outperforms existing knowledge distillation methods in terms of Average Relative Improvement (ARI) on the CIFAR-100 dataset.\n\nWeakness:\n\n(1) Although the combination of three types of distillation methods has not been investigated, this study just combines existing distillation methods by simply adding some losses that were proposed in previous literature. The novelty seems a bit insufficient to me.\n\n(2) As shown in Tables 1 and 2, the improvement of student models trained with ModReduce is not significant (less than 1%), compared to existing distillation methods.\n\n(3) Only a dataset (i.e., CIFAR-100) was used to evaluate the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AvBUEWBpd0v", "reviewer": "Reviewer_NDC7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose ModReduce, a framework to transfer the three knowledge types (knowledge on the produced output, knowledge on internal workings/intermediate layers, and knowledge on interactions between data points) from a teacher neural net model to a student neural net model by using a combination of offline and online knowledge distillation. The proposed approach is evaluated in comparison with the three mentioned types of knowledge distillation techniques on the CIFAR-100 dataset.", "summary_of_the_review": "Although the overall idea is well motivated, the single design choices for the proposed approach seem to be ad-hoc (e.g., the steps of Algorithm 1 and the diagram of Figure 2 are not motivated). There are different ways to combine the three basic knowledge distillation approaches, and it is not clear why the proposed strategy the one to go for. The experiments are rather weak, and it is not clear how and whether they generalize over other datasets. The presentation can be improved. Many plots, diagrams, figures are too bulky for the conveyed message, many of them could be removed and replaced through single sentences in the text. ", "strength_and_weaknesses": "Strengths:\n\n1. The addressed problem is interesting.\n2. The general idea of the proposed approach is well motivated.\n3. The observations with respect to the different knowledge distillation approaches are interesting. \n\nWeaknesses:\n\n1. Although the overall idea is well motivated, the single design decisions for the proposed approach seem to be ad-hoc (e.g., the steps of Algorithm 1 and the diagram of Figure 2 are not motivated). There are different ways to combine the three basic knowledge distillation approaches, why is your strategy the one to go for?\n2. The evaluation on just one dataset is rather weak, and it is not clear whether the results generalize over other datasets.\n3. The presentation can be improved. Many plots, diagrams, figures are too bulky for the conveyed message, many of them could be removed and replaced through single sentences in the text. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject