Weighted Mutual Learning with Diversity-Driven Model Compression
Authors: Miao Zhang
Li Wang
David Gonzalo Chaves Campos
Wei Huang
Chenjuan Guo
Bin Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: knowledge distillation
model compressing

Abstract: Online distillation attracts attention from the community as it simplifies the traditional two-stage knowledge distillation process into a single stage. Online distillation collaboratively trains a group of peer models, which are treated as students, and all students gain extra knowledge from each other. However, memory consumption and diversity among peers are two key challenges to the scalability and quality of online distillation. To address the two challenges, this paper presents a framework called Weighted Mutual Learning with Diversity-Driven Model Compression (WML) for online distillation. First, at the base of a hierarchical structure where peers share different parts, we leverage the structured network pruning to generate diversified peer models and reduce the memory requirements. Second, rather than taking the average of peers, this paper, for the first time, leverages a bi-level formulation to estimate the relative importance of peers with a close-form, to further boost the effectiveness of the distillation from each other. Extensive experiments show the generalization of the proposed framework, which outperforms existing online distillation methods on a variety of deep neural networks. More interesting, as a byproduct, \WML produces a series of pruned models under different model sizes in a single run, which also achieves competitive results compared with existing channel pruning methods.

Weighted Mutual Learning with Diversity-Driven Model Compression
Miao Zhang 12, Li Wang"; David Campos', Wei Huang Chenjuan Guo Bin Yang"
Haarbn Institute of Technology (Shenzhen) AAlbborg University *UNSW
"East China Normal University "Shandong First Medical University zhangmiao@hic edu..eduuccc dgcclcs a.u. we weihuang, s ccm tojguo, byang I deaaae.eeeeeeeeeeeeeeee ecnu, edu. en; -..aang-9sstuen. .uts. edu, au
Abstract
Online distillation attracts attention from the community as it simplifies the tradi- tional two-stage knowledge distillation process into single stage. Online distilla- tion collaboratively trains a group of peer models, which are treated as students, and all students gain extra knowledge from each other. However, memory con- sumption and diversity among students are two key challenges to the scalability and quality of online distillation. To address the two challenges, this paper presents a framework called Weighted Mutual Learning with Diversity-Driven Model Com- pression (WML) for online distillation. First, the base of hierarchical structure where students share different parts, we leverage the structured network pruning to generate diversified students with different models sizes, thus also helping reduce the memory requirements. Second, rather than taking the average of students, this paper, for the first time, leverages a bi-level formulation to estimate the relative importance of students with close-form, to further boost the effectiveness of the distillation from each other. Extensive experiments show the generalization of the proposed framework, which outperforms existing online distillation methods on variety of deep neural networks. More interesting, as byproduct, WML produces series of students with different model sizes in single run, which also achieves competitive results compared with existing channel pruning methods.
Introduction
Deep neural networks (DNNs) have achieved impressive success for a variety of application fields, including computer vision 277 46). natural language processing (2, time-series analyties [53), and
However, the performance of DNNs is heavily dependent on model parameters and computations, which hinders its application on intelligent edge systems [20]. To relieve this issue, various techniques have been proposed to design lightweight models, including network pruning [18). network quantization 77 and neural architecture search [43]. Knowledge distillation [25] 33] [1],
so on 6 48 54]
complementary method to further improve the performance of the lightweight models, can transfer the discriminative ability from a high-capacity yet cumbersome teacher model to a small and compact student model by encouraging the student mimic the behavior of the teacher. The vanilla knowledge distillation (25). as shown in Figure
(a). includes two stages: it first trains the cumbersome teacher model, whose soft logits that contain informative dark knowledge [25] are then transferred to a compact student model. As the teacher is fixed during the knowledge transferring, the student can only learn from the teacher once it is trained, and the vanilla knowledge distillation also called offfine distillation [5]. To further simplify the distillation process and relieve the restriction
when the larger-capacity teacher model available, the online distillation framework has been
Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

RosBlock
Logiti
ResNet1S2
Logiri
ResBlock
ResBlock
Legic2
ResBlock
ResBllok
Logit
Lagit
RevBlock
ResBlock
Legihs
(a): Vanilla Knowledge Distillation
(c): The Proposed Framework (WML)
ResNet56
Loght
Feedforward information flow
Weighted knowledge from peer
ResNet32
Lugiz
Unweighted knowledge flow
Loght
Outputs of peer /
(b): Online Distillation
Figure 1: Knowledge distillation, online distillation, and the proposed WML. (a) [25] is the first knowledge distillation, where the student ResNet32 only learn from the trained teacher R5sNet152. (b) 770 is also called deep mutual learning, where the students can learn from each other during the training. (c) is the proposed WML framework with hierarchy structure, where different colored Resblocks are pruned under different ratios to enable diversified peer models and also reduce the memory consumption. Rather than averaging the knowledge from peers, WML leverages bi-level optimization to estimate the relative importance of each peer (e.g., weight wij for peer j) in forming more informative knowledge source.
proposed 44 70 71,. which simultaneously trains a group of student models (a.ka. peers) with similar structures. As each student gain extra knowledge from each other, it is also called deep mutual learning [70]. as shown in Figure (b). Although online distillation conducts the distillation with one-phase training scheme without pretraining teacher, it has high memory consumption when jointly training several independent models. In addition, it supposed to yield more informative knowledge when diversity presents among students, while the similar structures in online distillation may hinder the students from learning more informative knowledge from peers p [6].
In order to simultaneously overcome the two aforementioned difficulties, memory consumption and diversity among peers, this paper proposes novel framework, called Weighted Mutual Learning with Diversity-Drive Driven Model Compression (WML) for online distillation, as present in Figure [] (c). Our WML considers a hicrarchical structure where each peer share different parts, so as reduce the memory consumption compared with online distillation. In addition, we leverage the structured network pruning to generate diversified peer models under different pruning ratios, which also further reduces the memory consumption. On the other hand, simply treating the peers with different capacities be equally important might neglect the diversity of all peer models, and limit the effectiveness of online distillation when learning from group of peers with different capacities p [6]. In this paper, formulate the weighted mutual learning based online distillation as a bilevel optimization problem, where the inner loop is to learn compact peers with the online distillation and the outer loop is to find suitable weights to determine a better optimization direction for knowledge distillation from group of peers. The weights are optimized by the mirror descent 3 5) and the hypergradient is analyzed in theoretical way with a close-form. Our formulation can be seen as dynamic weighting strategy for online distillation, which adaptively distills knowledge from the peer models to student. A summary of our main contributions follows.
First, this paper proposes a Weighted Mutual Learning with Diversity-Drive Model Com- pression (WML) for online distillation, where the peer models share different parts in a hierarchical structure and the structured network pruning is leveraged further generate diversified peers and reduce the memory consumption.
Second, we formulate the weighted mutual learning based online distillation as a bilevel optimization problem, and the hypergradient of optimizing the weights is derived with close-form. The dynamic weighting strategy enables the online distillation to adaptively determine the importance of each peer in distilling knowledge from group of peers.

---

Third, extensive experiments on several neural networks and three datasets verify the effectiveness of the proposed framework, which significantly outperforms existing online distillation and self-distillation methods. More interesting, a series of pruned student models with different model sizes in single run, which also achieves competitive results when compared with existing channel pruning methods.
byproduct, WML can get
2 Related work
2.1 Knowledge Distillation
Knowledge Distillation (KD) is the concept [25 that distills knowledge from larger teacher model to help the training of a smaller student model, by steering the student's ' logits towards teacher's logits, as shown in Figure D (a). FitNet 44] further extends the feature maps the learning target instead of logits. Rather than only learning from a single teacher, many recent studies 2 16 investigate learning from group of teachers, where the outputs of teachers
usually averaged guide the student training, while how to effectively integrate different types of knowledge from multiple teachers is less investigated [9] [6]. The traditional knowledge distillation is also called offline distillation since the teacher models are first trained before distillation. Differently, online distillation simultancously trains group of student models that can gain extra knowledge from each other without pretrained teacher model, so it is also called the reacher-absent knowledge distillation. Deep mutual learning (DML) i70 the first proposed online setting for knowledge distillation, which trains multiple peer models in collaborative way that each model can be seen as teacher during others' training process. KDCL [6 further ensembles the output of all students as the teacher knowledge in the online distillation setting since the ensemble is supposed to yield better generalization ability than single model []] To further reduce the training requirements, several recent works 77 enforce all students to share the same early blocks while equipped with individual auxiliary head branches. Although this paradigm can relieve the heavy memory consumption in online distillation, it suffers from the drawback of diversity as all peers share the same backbone and with a similar structure. Differently, Song and Chai [47] proposed a hierarchical structure so that the peers can share different parts in the early blocks. Self-distillation is another recently-propee
distillation paradigm, which can be regarded as a special case of online distillation [15] [66 61]. Specifically, [6]] leverages the knowledge from the deeper layers to guide the training for the shallow sections in the backbone. In this paper, we consider a hierarchical structure [47] that each peer shares different parts and gains weighted knowledge from other peers instead of the average ensemble. Different from self-distillation that reduces the "depth" to get peers, this paper leverage the channel pruning to "slim" models to get diversified peers.
2.2 Network Pruning
Network pruning is an effective way to compress overparameterized neural networks by removing parameters with minimal performance degradation (57). Based the granularity, network pruning can be divided into two types: unstructured and structured pruning, where the former individually removes parameters in network while the latter removes entire filters, channels, even layers [40]. Although the unstructured pruning can mask most parameters in network without performance drop [2], it requires specific hardware or software accelerate the sparse network's inference
time 10 19 Differently, structured pruning preserves the structural regularity and can accelerate network on commodity hardware platforms [22 32). In this paper, we consider structured pruning for generic applications. Next, the network pruning can happen after training vs. before training (a.k.a. initialization) 49 Most conventional pruning methods identify the redundant weights that are removed with least degrading the performance of the trained model. In contrast, recent works show that the randomly initialized neural network can be pruned without incurring any training [30], where SNIP 29). GraSP [50), and SynFlow 4 are three well-known metrics for the unstructured pruning at initialization. (40] further extends these metrics for structured pruning through summing scores of all parameters 0 in a filter to determine its importance. In this paper, we extend SNIP to determine the pruning ratio for each layer at initialization as 40 demonstrates it is the most effective among the three to determine the filter importance for channel pruning at initialization.
3

---

3 Methodology
This section describes our Weighted Mutual Learning with Diversity-Drive Model Compression (WML) framework for online distillation, which consists of three main parts: the overall structure and the optimization for learning group of peer models, the pruning approach to get series of peers with different model sizes, and bi-level formulation based weighted mutual learning.
3.1 Overview Structure and Loss Function
Figure depicts the overall structure of the proposed WML framework, where we take ResNet-32, which is divided into three sections in terms of depth, as an example for easy understanding. Different from the most commonly-used auxiliary heads [7]] that all peer models share the same part, we consider hierarchical structure [7 where each peer model shares different parts. The framework is similar to the binary tree, where the branches at the same levels are copies of each other, and each path is considered ratios to further enhance the diversity among peer models and also reduce the memory consumption. Different from the vanilla knowledge distillation, which requires much larger pretrained model as the teacher, the online distillation directly learns from each other or the ensembles of students [.] In this paper, consider the linear combination of knowledge from all peer models, and the weight of each peer model determined by bi-level formulation which will be discussed in Sec. 3.3
peer model. In addition,
further prune each block in WML with different
This paper focuses on the multi-clase classification problems, and = { is the training images with N samples, Y J is the corresponding labels. The output logit for i-th peer model is represented as 2,, and its corresponding cross entropy loss is Le which is computed with the and labels Y. The overall loss to train the whole framework containing M peers is defined as:
M
loss a1 Σ wrCoE(Z, Y)
ΣΣ
wi L 2g), =1
where wi indicates the importance of the i-th peer model and a is the hyperparameter to balance the supervision from labels and teachers. Σ/= wj wyKL(=,; 2z) is the weighted knowledge from peers, which is one of the main differences between our WML and DML (70 in loss function, where DML considers each peer equally important in distillation. Another alternative distillation strategy is to calculate the KL divergence between each student with the ensemble of all students [16]. while [70]
found it is not competitive as directly learning from multiple teachers.
3.2 Diversifying Peer Models with Channel Pruning
Although the hierarchical structure can proportionally reduce the memory consumption as peers share different blocks to reduce complexity, it still has higher computational cost than traditional knowledge distillation. To further reduce the memory requirements, we introduce the channel pruning, the most commonly-used paradigm in structured pruning, to our framework, which also encourage the diversity among homogenized peers. The training-free network pruning has shown its superiority in finding competitive subnetwork with extreme efficiency (29 49 50, and (40)
has demonstrates SNIP (29] is informative to determine the filter importance for channel pruning. To be more specific, the relative importance of each filter f based on SNIP can be calculated as:
ac(0)) @ 0,), where aC(0;) 0, is the SNIP score of parameter 0,- J9. This value gives an expectation of the impact on loss when the filter is pruned, and filters with smaller SNIP scores tend produce weak impact to the loss as compared the other filters in that layer
-o ((.)) Σa.es
30.
40 In this paper, we further propose a relative importance I₁ to determine the importance of each layer at initialization, which defined as:
1 N
I
I JEl
where N; the number of parameters in layer Given a pruning ratio P a model, the number of pruned parameters of each layer is then obtained by pt = x N Σ, where Ź, is a normalization of +, and N is the total number of parameters in the model, and the number of preserved filter is

---

calculated accordingly which aims to preserve more filters in more important layer. In the practical implementation, we a upper bound of the pruned ratio for each layer avoid layer collapse [50]. In this way, given different P to peers, we can get a series of peer models with different structures.
3.3 Weighted Distillation with Bi-level Formulation
The proposed framework contains M peer models whose outputs can be also combined to generate a powerful ensemble. However, the generalization capacity of the peer model is different from the model size. Intuitively, we would expect to raise the importance of model with the highest capacity in distillation, i.e., enlarging W/ when peer model contains most parameters 4 9. In this paper, we consider dynamic weighting method determine the importance of each peer, and the weighted mutual learning for online distillation can be formulated as a bilevel optimization problem:
min LOB w Y)
s.t. 0" = argming (1 a) Σ w LCE V) +
my K KL(2,2))) 2y),
(3)
where 27 is obtained by training the whole framework 9 with loss function in Eq.(T. The inner loop is to optimize the whole network 9 with the loss defined in (T. and the outer loop is to adjust the importance
make the weighted ensemble match the ground truth that it reflects the importance of each peer more precisely. We define the loss function of inner loop optimization as Li and outer loop L2 for simplicity. Since 2 is the local optimizer in the inner loop, Eq.(3) is complicated nested optimization problem [37 42 54 64
69). In this paper, we consider a common one-step 7 VeLi with learning rate y, and the gradient for the outer loop (which is also called as hypergradient) can be formulated as:
68
unroll learning paradigm [37 42 that 0
0Ls 0L2 00 (w) 0L2 OL2 02 L1
VuLa
(4)
ow
ae
Ow
dw
ae awde'
However, it is still impractical to calculate the second-orde order information for a large neural network, 8² ć1 e.g., awa The finite-difference approximation is usually leveraged to roughly approximate this term [37,, while it only considers the first two terms with the Taylor expansion. the following theorem,
show that we can exactly calculate Tudd
& without approximation in our framework.
Theorem
With one-step unroll learning paradigm, the gradeint for Wi in Eq, 3 is formulated as:
0L2
0L2 Li
OL2
OL2 0c.
Iw: Ww,f2
7 06
-7 00 00
(5)
Ow:
ow;00
Ow:
where Cu =
Y) - M
KL(zj,2))
Proof:
Based on the definition of L1 in Eq.[I. we have
0L,
0((1 - a) LM M
w; Y)
- 2j=1 EM wi K KL(zz,
Ow;
Owi
0 2
Ei wi K (zzjz
-11 (1=a))c +a
(6)
Owi
M
- (1=0)) Y) t Σ KL(z,,21).
1=1
Accordingly, &cc, awad can be calculate with autograd of L. on 0, and Theorem is then proved.
8 Theorem shows that the second-order term awad can be analyzed in closed-form without Taylor expansion approximation when C1 is with a specific structure. Now that we can simply get the gradient update model parameters 0 and the weights for peers with autograd. Since is probability simplex that EM w; 1, we use the mirror descent to update w outlines weighted mutual learning for online distillation. To be more specific, we first run several
B 5). Algorithm
5

---

Algorithm Weighted Mutual Learning (WML) for Online Distillation
1: Input: Dataset {( (n)) n= Given pruming ratios for each peer model {pI: .... PM) .MM,
2: Initialized hierarchical model 8 and peer weights w
3: Calculate the filter importance with SNIP at initialization, and prune peer models ž based on the given pruning ratios;
4: for I,.,, do
5: With the peer importance w run 7 steps of SGD update the model parameters 0 with the weighted loss function in E
6: Calculate the gradient for wh based on Eq.
7: Run one step of mirror descent and update w to get # based on Eq.
8: end for
9: output: M models with outputs f2i,...,MM the weights for peers w.
Table Top-1 accuracy (%) comparison results with self-distillation on CIFARI00.
Networks
Methods
Baseline
Model1
Model2
Model3
Model4
Ensemble
DSN 28
77.09
67.23
73.80
77.75
78.38
79.67
SD 61
77.09
67.85
74.57
78.23
78.64
79.67
ResNet18
SCAN 62
77.09
71.84
77.74
78.62
79.13
80.46
WML
77.09
71.15
75.65
78.88
79.38
80.56
DSN 28
77.68
67.87
73.80
74.54
80.27
80.67
SD 61
77.68
68.23
74.21
75.23
80.56
81.04
ResNet50
SCAN 62
77.68
73.69
78.34
80.39
80.45
81.78
WML
77.68
78.58
79.69
80.81
81.24
82.57
DSN 28
77.98
68.17
75.43
80.98
81.01
81.72
SD 61
77.98
69.45
77.29
81.17
81.23
82.03
ResNet101
SCAN 62
77.98
72.26
79.26
80.95
81.12
82,06
WML
77.98
79.60
81.16
81.14
81.46
83.03
steps of SGD based on the loss function in Eq to update model parameters with fixed w. Then
we calculate the gradient of wi based on Eq 5 and run one step of mirror descent update wt that:
w
w/ L2
1+I
E w/exp
-n
where 77 is the step size, and w is the model importance for i-th peer in k-th step. We repeat this loop until the stopping criterion meets.
4 Experiments
We perform series of experiments to evaluate the effectiveness of our framework seven convo- lutional neural networks with three image classification datasets. We first compare our framework with two self deff-distillation baselines, SD [o]] SCAN 62), and the deeply supervised net (DSN) 28 since they share similar structures as our WML. Specifically, SD and SCAN add early-exit blocks to the shallow blocks to get peer models, where SCAN contains additional attention modules. DSN shares a similar structure as SD while without knowledge distillation. After that, we compared WML with existing knowledge distillation and online distillation baselines. More interesting, as byproduct, we also compared the pruned models produced by our framework with channel pruning baselines. A series of ablation studies on dynamic weighting strategy and pruning are also discussed.
4.1 Comparison with Self-Distillation
For fair comparisons this experiment, we consider peer models our WML, and also prune peer models into similar model size classifiers in self-distillation (SD) [61), where Model1-3 are under

---

Table 2: Top-1 accuracy (%) comparison results with self-distillation on ImageNet.
Networks
Methods
Baseline
Modell
Model2
Model3
Model4
Ensemble
SD 66
71.52
52.23
61.33
68.64
72.37
72.73
MobileNetV2
WML
71.52
65.20
69.32
71.44
72.76
73.10
different pruning ratios to match Classifier1-3 self-distillation [1], and Model4 and Classifier4 are the same as the baseline network. More experiments with other pruning ratios for WML can be found in Sec Sec We adapt our framework with several ResNets (2] and also the MobileNetV2 45 where results are present in Table and T respectively.
There are two interesting pieces of evidence showing the advantage of the proposed WML on CIFAR100. Firstly, we can observe that leads to significant accuracy boosts, where our peer Model4 outperforms baseline networks with an increment of 2.29% to 3.56% on CIFAR 100 for the three networks. In addition, our peer models beat the most classifiers in the three self-distillation baselines under similar model sizes, verifying the superiority of the proposed framework over self- distillation. Although SCAN achieves better results in Modell and Model2 for ResNet18, we should notice that SCAN adds additional attention modules over SD, which has more parameters than SD and our WML. When the peer model is small, the additional attention module that contains relatively numerous parameters plays a more important role to improve the performance, while whose advantage decreases with the peer models become larger. Secondly, compared with self-distillation baselines, which usually encounter significant performance drops in the shallower classifiers, our peer models can maintain similar performance with moderate network pruning, especially on the large networks. Taking the Model2 on ResNet50 as an example, whose model size is only 50% as the baseline, WML obtains 79.69% accuracy, while SD only achieved 74.21% accuracy. SCAN added additional attention modules to further improve self-distillation performance, achieving 78.34% accuracy while still worse than our WML. The above results demonstrate that reducing the width is a better way to get more competitive peers than the self-distillation paradigm using the shallow models. The comparison results on ImageNet in Table 2 again verify the advantage of the proposed framework, where our WML also outperforms the seff-distillation baseline under different model sizes on the ImageNet dataset. As shown in the last column of Table [ which is the ensemble of the four peer models, our WML outperforms the three baselines by large margins. This superiority comes from two aspects. One is that our weighted mutual learning can boost all peers' performance, thus enhancing the ensemble. The other is that our optimized weights can capture the peers' importance more precisely.
4.2 Comparison with Online Distillation
compares the proposed WML with several online distillation methods, including RKD[4], CTSL-MKT(41], DML (70), ONE (1],, and self-distillation (SD) [61 on CIFARIO, CIFAR 100, and ImageNet. In these online distillation methods, we consider four peer models our WML for fair comparisons, and all reported results for baselines can also be found in 663] We can see that knowledge distillation and online distillation both outperform the directly trained baseline model with large margin, showing the effectiveness of knowledge distillation. More interesting, different from traditional knowledge distillation which requires a powerful and large teacher model, online distillation methods can learn more meaningful information from peer models and gain performance improvements without a teacher. These comparison results again verified the effectiveness of the proposed framework, which outperforms most online distillation methods on different datasets and neural networks.
4.3 Comparison with Channel Pruning
In our WML, we leverage channel pruning to generate peer models with different pruning ratios at initialization. As byproduct, we can get series of pruned models with different FLOPs as peer models. this subsection, we compared the performance of the pruned models generated by our WML with existing channel pruning methods. For fair comparisons, we conduct experiments with ResNet32 and ResNet56 on CIFAR 10, and MobileNetV2 on ImageNet, which are the common setting to compare with existing works (63). One clear advantage of WML over existing channel
7

---

Table 3: Top-1 accuracy (%)
comparison results with ciline-distillation
Datasets
Models
Baseline
KD
RKD[4]
DML MKT[4]]
ONE(71]
SD
WML
R8sNet18
94.25
94.67
94.98
95.33
95.19
95.56
95.87
95.97
CIFARIO
ResNet50
94.69
94.56
95.23
95.76
95.73
95.85
96.01
96.17
ResNet18
77.09
77.79
76.43
77.46
77.54
77.87
78.64
79.38
CIFAR100
ResNet50
77.42
79.33
78.02
78.52
78.31
78.52
80.56
81.24
ImageNet
MobilcNetV2
71.52
72.23
72.46
72.29
72.20
72.37
72.76
Table 4: Comparison with channel pruning methods for ResNet-32 and ResNet-56 on CIFARIO
Models
Method
Baseline Acc
Pruned Acc
Acc Drop4
FLOPs Drop.
LCCLL 8
92.33%
90.74%
1.59%
31.2%
SFP 22
92.63%
92.08%
0.55%
41.5%
GFP40 [3
93.51% %
92.86%
0.65%
40.1%
ResNet32
GFP50 38
93.51%
92.64%
0.87%
50.1%
PSeratch [52
93.18%
92.18%
1.00%
50.0%
FPGM 24
92.63%
92.31% %
0.32%
41.5%
WML30
92.63%
92.99%
%
24.4%
WML40
92.63%
92.23%
0.40%
45.4%
WML50
92.63%
92.10%
0.53%
50.0%
PFEC [1
93.04%
93.06%
-0.02%
27.6%
LCCL 8
94.35%
92.81%
1.54%
37.9%
HRank 35
93.26%
93.52%
-0.26%
29.3%
NISP 58
93.26%
93.01 %
0.25%
35.5%
AMC 23
92.80%
91.90%
0.90%
50.0%
ResNet56
GFP40 38
93.68%
93.54%
0.14%
40.2%
GAL 36
93.26%
93.38 %
-0.12%
37.6%
WML30
93.26%
93.93%
-0.67%
24.7%
WML40
93.26%
93.46%
-0.20%
41.7%
WML50
93.26%
92.68%
0.58%
49.0%
pruning methods that we can get series of pruned models with different FLOPS in single run. Table Land compared our pruned peer models with existing baselines CIFARIO and ImageNet. We make the peer models toward (30%, 40%, 50%) pruning ratios on FLOPs for ResNet32 and ResNet56 CIFARIO, and (30%, 50%, 70% for MobileNetV2 on ImageNet.
As shown in Table there are two interesting findings. First, WML obtains higher accuracy than all channel pruning competitors on CIFAR10 after pruning similar FLOPs. For example, the model with 40% pruned FLOPs by WML only has 0.40% accuracy drop for ResNet32 and even 0.20% performance increase for ResNet56, which both outperform the competitors on CIFAR 10. Second, our pruned models with moderate pruning achieve better performance than the baseline model while with much fewer FL.OPs. This benefit comes from the online distillation we find the SNIP baseline in [40]. which adopts the same pruning method as us while without knowledge distillation, obtains worse results than those claborately designed pruning methods. However, combined with online distillation, our pruned model can achieve better performance than elaborately pruned models. straightforward direction for future work is to apply those competitive pruning methods into framework for further performance improvements.
Table presents the comparison results of our pruned peer models on ImageNet, which also achieves competitive results compared with other SOTA channel pruning methods under different FLOPs constraints. For example, when the FL.OPs are reduced by about 30%, WML30 only has 0.08% % Topl accuracy drop, which better than MetaPruning's 0.60%, AMC's 1.00%, and DMCP's 10% the similar pruning ratio. A similar advantage also be found when pruning 50% and 70% FLOPs.

---

Table 5: Comparison with channel pruning methods for MobileNetV2 on ImageNet
Models
Method
Baseline Acc
Pruned Acc
Top1 Dropt
FLOPs Drop!
MetaPruning30
39
71.80%
71.20%
0.60%
27.7%
MetaPruning50
39
71.80%
68.20%
3.60%
53.3%
MetaPruning70
39
71.80%
63.80%
8.20%
71.0%
AMC [23
71.80%
70.80%
1.00%
29.7%
MobileNetV2
DMCP
17
72.30%
72.20%
0.10%
29.7%
Group 38
75.74%
69.16%
6.58%
50.0%
WML30
71.52%
71.44%
0.08%
29.8%
WML.50
71.52%
69.32%
2.20%
52.9%
WML70
71.52%
65.20%
6.32%
63.1%
84
84
Ensemble Model4
Ensemble Model4
0.3
Weight WWeigh
83
83
Importance
Weight 3 Weight
Accuracy
Accuracy
82
0.25
81
80
0.01
0.05
0.5
Epoch
100
150
0%
10%
30%
70%
(a)
Pruning ratio
(b) Step size 1
Track during distillation
Figure 2: Ablation studies pruning ratio, step size v and peer importance.
4.4 Ablation Studies
In this section, first conduct ablation studies to investigate whether the dynamic weighting strategy and peer pruning help improve the performance of online distillation. Then, we show increasing the number peers gradually enhance the performance of the proposed framework.
Figure 2 (a) presents how the performance of Model4 and ensemble change with different pruning levels. As shown, when slightly pruning peers (only prune less than 10% parameters for Modell-3), WML can get better results than the unpruned framework in terms of both Model4 and ensemble performance. An underline reason that slightly pruning rarely decreases the performance of peer models, while the diversity it brings can further enhance the distillation. However, when the pruning ratio for Modell-3), we can see clear performance drop since the pruning damages the performance of Modell-3 seriously, which also further affects the distillation for Model4. Figure (b) presents how the dynamic weighting strategy works in our WML, where the pruning ratio is set as Sec 4. As shown, the first data point in Figure (b) indicates that we follow deep mutual learning 770 that remove the w in Eq. D.. and the remaining points are with different to update w. We can see clear performance boost from unweighted distillation weighted distillation, which verifies the effectiveness of our dynamic weighting strategy. which is also robust to the step size In Figure 2 [6]] (c) track the change of weights for each peer model. As shown, the
high (the last data point in the Figure 2 (a) prunes more than 70% parameters
(c), we prune Model 1-3 with 75%, 55% and 20% parameters match with Classifier and the four lines in Figure (c) weight of Model4 and Model4 consistently increase since they contains much more parameters and with better generalization ability. In contrast, the weights for the other two models both decrease, and Modell with the least parameters descents the fastest. This phenomenon also verifies our weighting strategy adaptively adjust the importance of peers in distillation.
The main experiments show good results of the proposed framework with four peers. We also verify the superiority of WML with different number of peers in Figure 3, For M = 2, we leave one peer unpruned and the other is pruned with 50%, and for M = 4, 8, we leave one peer unpruned and the remained peer models are pruned gradually from 30% to 70%. As show in Figure WML scale well with more peers although gain decrease with the number peers increase. The phenomenon verifies that more knowledge from peers can further enhance the performance of the students.

---

Ensemble Model.
Accuracy
82
Model.
Ac Accuracy
84
Accuracy
84
Model
80
82
80
78
78
8
Num of peers
Num of peers
Num of peers
(a) ResNet18
(b) ResNet50
(c) ResNet101
Figure 3: Performance with the number of peers under three different networks.
5
Conclusion and Future Work
This paper proposes new framework for online distillation, called Weighted Mutual Learning with Diversity-Drive Model Compression (WML), which introduces the structured network pruning to obtain diversified peer models, and bi-level formulation is further leveraged to estimate the relative importance peers in forming a powerful teacher during the online distillation. The WML innovatively combines network pruning and online distillation enhance performance and compress models simultaneously. Extensive experimental results have verified the proposed framework, which leads to consistent accuracy improvements various neural networks and datasets under different model sizes. This paper considers a simple and efficient training- free channel pruning method to get the peer models while which is not as competitive as recent training-basse channel pruning methods. Combining more advanced channel pruning methods with online distillation for further performance and efficiency boost and extending WML feature distillation or other applications. e.g., object detection, natural language processing, and architecture search, are among directions for future work.
Acknowledgments and Disclosure of Funding
This work was partially supported by Independent Research Fund Denmark under agreements 8022- 00246B and 8048-00038B, the VILLUM FONDEN under agreement 34328, and the Innovation Fund Denmark centre, DIREC. This work was also in collaboration with Digital Research Centre Denmark DIREC, supported by Innovation Fund Denmark.
References
[1] Z. Allen-Zhu and Y. Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep lcarning. arXiv preprint arXiv:2012.098// 2020.
[2] B. Alshemali and J. Kalita. Improving the reliability of deep neural networks in nlp: A review. Knowledge- Based Systems, 191:105210, 2020.
[3] A. Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters. 113):167-175... 2003.
[4] D. Chen, J.-P. Mei, C. Wang, Y. Feng, and C. Chen. Online knowledge distillation with diverse peers. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 3430-3437, 2020.
[5] S. Chen, K. Crammer, H. He, Roth, and W. Su. Weighted training cross-task learning. ICLR, 2022.
[6] Y. Chen, T. Yang, X. Zhang, G. Meng, X. Xiao, and J. Sun. Detnas: Backhone search for object detection. In Advances in Neural Information Processing Systems, pages 6642-6652, 2019.
M. Courbariaux, Y. Bengio, and J.-P. David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances neural information processing systems, 28, 2015.
[8] X. Dong, J. Huang, Y. Yang, and S. Yan. More less: more complicated network with less inference complexity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5840-5848, 2017.
10

---

[9] S. Du, S. You, X. Li, J. Wu, F. Wang, C. Qian, and C. Zhang, Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems, 33:12345- 12355, 2020.
[10] E. Elsen, Dukhan, T. Gale, and K. Simonyan. Fast sparse convnets. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14629-14638, 2020.
[11] G. Fang, J. Song, X. Wang, C. Shen, X. Wang, and M. Song. Contrastive model inversion for data-frec knowledge distillation. arXiv preprint arXiv:2105,00 08584, 2021.
[12] J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
[13] Y. Ge, C. L. Choi, X. Zhang, P. Zhao, F. Zhu, R. Zhao, and H. Li. Self-distillation with batch knowledge ensembling improves imagenet classification. arXiv preprint TrXIv:2104.13295 2021.
[14] Y. Gong. Y. Yu, G. Mittal, G. Mori, and M. Chen. Muse: Feature self-distillation with mutual information and self-information. arXiv preprint 10.626660,2666, 2021.
[15] J. Gou, B. Yu, S. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6) 1799-1898,,,, 2021.
[16] Q Guo, X. Wang, Y. Wu, Yu, Liang. Hu, and P. Luo. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11020-11029, 2020.
[17] S. Guo, Y. Wang. Q. Li, and J. Yan. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1547, 2020.
[18] S. Han, J. Pool, J. Tran, and W. Dally. Learning hoth weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
[19] S. Han, X. Liu, H. Mao, Pu, A. Pedram, M. Horowitz, and W. J. Dally. Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243 254,, 2016.
[20] S. Han, H. Mao, and J. Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. In ICLR, 2016.
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[22] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang. Soft filter pruning for accelerating deep convolutional neural networks. In IJCAI International Joint Conference on Artificial Intelligence, 2018
[23] Y. He, Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 784 800, 2018.
[24] Y. Hc, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3340-4349, 2019.
[25] G. Hinton, O. Vinyals. J. Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503. 02533., 2015.
[26] M. Ji, S. Shin, S. Hwang, G. Park, and I.-C. Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10664-1067, 2021.
[27] A. Krizhevsky, I. Sutskever, and E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[28] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In Artificial intelligence and statisticy, pages 562-570. PMLR, 2015.
[29] N. Lee, T. Ajanthan, and P. Tor. Snip: Single-shov network pruning based on connection sensitivity. In International Conference Learning Representations, 2019.

---

[30] N. Lee. T. Ajanthan, S. Gould, and P. H. Tor. A signal propagation perspective for pruning neural networks at initialization. In International Conference on Learning Representations. 2020.
[31] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In ICLR, 2017.
[32] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. In ICLR, 2017.
[33] X. Li. J. Wu, H. Fang. Y. Liao, F. Wang, and C. Qian. Local correlation consistency for knowledge distillation. In European Conference on Computer Vision, pages 18-33. Springer, 2020.
[34] Z. Li, X. Li, L. Yang, J. Yang, and Z. Pan. Student helping teacher: Teacher evolution via self self-knowledge distillation, arXiv preprint 1rX1:22110.0032,, 2021.
[35] M. Lin, R. Ji, Y. Wang, Y. Zhang. B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 1529-1538, 2020.
[36] S. Lin, R. Ji, C. Yan, B. Zhang. I.. Cao, Q. Ye, F. Huang, and D. Doermann. Towards optimal structured can pruning via generative adversarial learnnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2790-27999 2019.
[37] H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. ICLR, 2019.
[38] L.. Liu, S. Zhang, 7. Kuang, A. Zhou, J.-H. Xue, X. Wang, Y. Chen, W. Yang. Q. Liao, and W. Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021-7032. PMLR, 2021.
[39] Z. Liu, H. Mu, X. Zhang. Z. Guo, X. Yang. K.-T. Cheng, and J. Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3296-3305, 2019.
[40] E. S. Lubana and R. P. Dick. A gradient flow framework for analyzing network pruning. In ICLR, 2021.
[41] W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[42] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pages 113-124, 2019.
[43] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang. Li, X. Chen, and X. Wang. A comprehensive survey of neural architecture search: Challenges and solutions. arXiv preprint arXiv:2006.0290: 2020.
[44] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. ICLR, 2015.
[45] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenccks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[46] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[47] G. Song and W. Chai. Collaborative learning for deep neural networks. Advances in neural information processing systems, 31, 2018.
[48] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. Interna- tional Conference on Machine Learning, pages 6105-61 14, 2019.
[49] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Syxtems, 33, 2020.
[50] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[51] L. Wang. X. Li, Y. Liao, Z. Jiang. J. Wu, F. Wang, C. Qian, and S. Liu. Head: Hetero-assists distillation for heterogeneous object detectors. In European Conference on Computer Vision, pages 11--311. Springer. 2022.
12

---

[52] Y. Wang. X. Zhang, L. Xie. J. Zhou, H. Su, B. Zhang, and X. Hu. Pruning from scratch. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 12273-12280. 2020.
[53] Z. Wang. W. Yan, and T. Oates. Time series classification from scratch with deep neural networks: A strong haseline. In 2017 International joint conference neural networks (IJCNV), pages 1578-1585. TEEE, 2017.
[54] X. Wu, D. Zhang, C. Guo, C. He. B. Yang. and C. S. Jensen. Autocts: Automated correlated time series forecasting. Proc. VLDB Endow., 15(4)9911993,, 2021.
[55] T.-B. Xu and C.-L. Liu. Data-distortion guided self-distillation for deep neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3565-5572, 2019.
[56] C. Yang, I., Xie, C. Su, and A. L.. Yuille. Snapshot distillation: Teacher-student optimization in one generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 2859-2868, 2019.
[57] H. You. C. Xu, Y. Fu, Y. Wang. X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. In ICLR, 2020.
[58] R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194-9203, 2018
[59] S. Yun, J. Park, K. Lee, and J. Shin. Regularizing class-wise predictions via eef-knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 13876-13885, 2020.
[60] S. Zagoruyko and N. Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. ICI.R, 2017.
[61] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[62] L. Zhang, Z. Tan, J. Song. J. Chen, C. Bao, and K. Ma. Scan: A scalable neural networks framework towards compact and efficient models. Advances Neural Information Processing Systems, 32, 2019.
[63] L. Zhang, Y. Shi, 7. Shi, K. Ma, and C. Bao. Task-oriented feature distillation. Advances in Neural Information Processing Systems, 13:44719147771, 2020.
[64] M. Zhang, H. Li, S. Pan, X. Chang, Z. Ge, and S. Su. Differentiable neural architecture search in equivalent space with exploration enhancement. Advances in Neural Information Processing Systems, 33: 13341- 13341-13351, 2020.
[65] M. Zhang. H. Li, S. Pan, X. Chang, and S. Su. Overcoming multi-model forgetting in onc-shot with diversity maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7809-7818, 2020.
[66] M. Zhang, H. Li, S. Pan, X. Chang. C. Zhou, Z. Ge, and S. Su. One-shot neural architecture search: Maximising diversity to overcome catastrophic forgetting, IEEE Trunsactions on Pattenn Analysis and Machine Intelligence, 43(9):2921- 2935, 2020.
[67] M. Zhang, H. Li, S. Pan, T. Liu, and S. Su. One-shot neural architecture search via novelty driven sampling. In Proceedings of the Twetty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20. International Joint Conferences on Artificial Intelligence Organization, 2020.
[68] Zhang, S. Su, S. Pan, X. Chang, E, Abbasnejad, and R. Haffari. idarts:
Differentiable architecture search with stochastic implicit gradients. In International Conference on Machine Learning. PMLR, 2021.
[69] M. Zhang, Pan, Chang, S. Su, J. Hu, G. R. Haffari, and B. Yang. Balenas: Differentiable architecture scarch via the bayesian learning rulc. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11871-11888 18800 2022.
[70] Y. Zhang, Xiang, T. M. Hospedales, and H. Lu. Deep mutual learning. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 4320-4328, 2018
[71] X. Zhu, Gong, al. Knowledge distillation by on-the-fly fly native ensemble. Advances neural information processing systems, 31, 2018
13

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report bars (e.g.. with respect the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [Yes]
(h) Did you mention the license of the assets? [Yes]
(c) Did you include any
either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're 'o''re
ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "kW9nKKiwNBO", "reviewer": "Reviewer_GDJC", "summary": "This paper presents a new online distillation technique enabled by a novel bi-level formulation that is based on a closed form expression. The bi-level formulation results in a weighting of the peers according to importance thus enabling optimized training of the entire network of peers unlike prior works that depended on approximations. The proposed method enables systematic pruning of the networks which results in significant improvement over the state of the art in terms of accuracy given a certain number of flops.", "questions": "Having a closed form expression is of course impressive and handy, but in most cases a Taylor approximation provides a powerful approximation. So what accounts for the great improvement over the state of the art? ", "limitations": "The authors have pointed out that to maintain a bounded investigation they have refrained from incorporating conventional pruning techniques which they say they will adopt in future work to get even better results.\nThat is the one discussion of limitations that I could find.\n\nThere is no mention of social impact, positive or negative. I would suggest that the proposed fine-grained optimization will have only positive social impact because it will help reduce complexity in a systematic manner, thus more energy efficient and compact devices can be constructed which is a net positive for society.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Originality and Significance: The paper's proposed closed form approach is a clear advancement of the state of the art and is thus both original and significant.\n2. Quality: The proposed closed form approach provides a compact and systematically tunable framework that achieves much better than the state of the art performance. The quality is therefore high.\n3. Clarity: The paper is very well written. It provides a broad overview and then covers the state of the art in an insightful way and motivates the proposed technique. The technique is itself well described and the results are presented clearly as are the conclusions. The clarity is high.\n\nWeaknesses\n1. Having a closed form expression is of course impressive and handy, but in most cases a Taylor approximation provides a powerful approximation. So what accounts for the great improvement over the state of the art? \n2. How exactly the closed form expression is  being used needs to be explained better.\n3. The above are minor weaknesses. The overall quality of the paper far outweighs these \"weaknesses.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L6i02NTUJN", "reviewer": "Reviewer_MvTh", "summary": "This paper introduces the structured network pruning and bi-level formulation to online distillation and shows its effectiveness in performance and efficiency. On the one hand, this paper proposes to prune unimportant channels in peer models, reducing the memory requirements. On the other hand, this method leverages a new formulation to estimate the relative importance of peer models, and improve the effectiveness of online distillation. Experimental results show that the proposed method outperforms a variety of methods in knowledge distillation and channel pruning. ", "questions": "- How does the proposed method compare with the newly proposed methods in knowledge distillation and channel pruning?\n\n- How does channel pruning contributes to the diversity of peer models? \n\n- Why choose SNIP as the pruning method? Does it perform much better than others?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:  \n\n- The paper is very clearly written and easy to understand.  \n\n- The proposed method reduces the memory cost of online knowledge distillation and maintains competitive performance on a variety of deep neural networks.\n\n- This paper makes a comprehensive ablation study on the proposed method.   \n\n-----------\nCons:  \n\n- This paper proposes to have improved the diversity of peer models but does not analyze how the proposed method impacts the diversity of all peer models deeply. It is necessary to discuss it in the experiment section.   \n\n- To my understanding, SNIP is an unstructured pruning method that prunes out the unimportant connections. I think the authors should consider other competitive structured pruning methods rather than just borrowing SNIP for structured pruning.  \n\n- Most of the selected competitors have been proposed for a long time and have a better alternative. The superiority presented in the experiments is not very convincing. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BcP3Dqsv7yt", "reviewer": "Reviewer_1K3n", "summary": "The paper proposed an online-distillation framework called Weighted Mutual Learning with Diversity-Driven Model Compression (WML). The authors Introduce a hierarchical structure in which peer networks share different parts to reduce memory consumption. A structured pruning method based on SNIP is adopted for the diversity of peer students. They use a dynamic weighting method to determine the importance of the pruned peers and formulate it as a bi-level problem. ", "questions": "--- The contributions are the memory-friendliness and diversity of student peers; they respectively benefit from a hierarchical structure where peers share different parts and structure pruning. As mentioned in the original paper, the hierarchical architecture is adopted from [1] and the pruning criterion is SNIP[2]. Neither of these points is original, and the innovation of this paper needs more clarification.\n\n-- Though self-distillation can be regarded as a special case of online distillation, it is different from the mutual learning method in this paper since it only uses its own structure as a teacher. Instead of self-distillation, the comparison with online distillation using multiple peers should be of more concern. In this paper, only DML[3] and ONE[4] are compared without more updated methods. This makes a claim in line 66 \"significantly outperforms existing online distillation and self-distillation methods\" less convincing. I think Table 3 should be expanded and highlighted with more comparisons with the latest online distillation approaches.\n\n-- The authors mention the use of structured pruning to enhance diversity, but due to the use of a partially shared hierarchical structure, diversity is limited compared to other online methods that can use completely different networks. It seems to me that the advantage of structural diversity here is only in comparison with self-distillation methods or [1].\n\n-- The diagram in Figure 1 (b) is ok, but it could easily be confused with self-distillation. It would be more representative of online distillation if it were changed to different peer networks.\n\n[1] G. Song and W. Chai. Collaborative learning for deep neural networks. NeurIPS, 2018.\n[2] N. Lee, T. Ajanthan, and P. Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.\n[3] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu. Deep mutual learning. CVPR, 2018.\n[4] X. Zhu, S. Gong, et al. Knowledge distillation by on-the-fly native ensemble. NeurIPS, 2018.", "limitations": "Yes，it's oK.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-- The paper is clearly presented, and the main ideas are easy to follow.\n\n-- The two difficulties of online distillation: memory consumption and model diversity, as the proposed framework tries to solve, are important and meaningful. The authors conduct several experiments to prove the effectiveness.\n\n-- Structured network pruning is used to generate peer networks in the hierarchical structure. This combines pruning and online distillation. And to dynamically decide the weight of each peer rather than take the average, the authors formulate it as a bi-level optimization problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "3jlVbiChviV", "reviewer": "Reviewer_WYtc", "summary": "This paper tackle the problem of online distillation using weighted mutual learning, where structured network pruning is leveraged to reduce the memory consumption. Extensive experiments on several neural networks and three datasets show this method outperforms existing online distillation and self-distillation methods.", "questions": "Q1: There is nothing in the supplementary material. Did the authors submit a wrong file?\n\nTypo:\n\nTable 1 ResNet 101->WML->Model 1: \"79.6\" -> \"79.60\".", "limitations": "See weaknesses.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea is interesting and can be applied in many tasks.\n2. Experimental setups are proper and there are sufficient experiments.\n\n\nWeaknesses:\n1. The motivation of the method is not described clearly. For example, why do you propose to use several peer models to share different parts in a hierarchical structure in Figure 1? If the motivation does not described clearly, the method seems to be incremental.\n2. The improvements are marginal. In Table 1,2,3, most of the accuracy gaps between WML and SCAN/SD are within 1%, which is marginal. Did the authors run the experiments for several different times and take the means and standard variances?\n3. Tables need to be improved. More detailed descriptions should be added in the table caption. For example, the values in Table 1 and Table 3 are accuracy (%) right? The authors can clearly mention that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept