Unified Wisdom: Harnessing Collaborative Learning to Improve Efficacy of Knowledge Distillation
Authors: Atharva Abhijit Tambat
Durga S
Ganesh Ramakrishnan
Pradeep Shenoy
Venue: Accepted by TMLR ()

Abstract: Knowledge distillation (KD), which involves training a smaller student model to approximate the predictions of a larger teacher model is useful in striking a balance between model accuracy and computational constraints. However, KD has been found to be ineffective when the teacher and student models have a significant capacity gap. In this work, we address this issue via “meta-collaborative distillation” (MC-Distil), where students of varying capacities collaborate during distillation. Using a “coordinator” network (C-Net), MC-Distil enables mutual learning among students as a meta-learning task. Our insight is that C-Net learns from each student’s performance and training instance characteristics, allowing students of different capacities to improve together. Our method enhances student accuracy for all students, surpassing state-of-the-art baselines, including multi-step distillation, consensus enforcement, and teacher re-training. We achieve average gains of 2.5% on CIFAR-100 and 2% on Tiny ImageNet datasets, consistently across diverse student sizes, teacher sizes, and architectures. Notably, larger students benefiting through meta-collaboration with smaller students is a novel idea. MC-Distil excels in training superior student models under real-world conditions such as label noise and domain adaptation. Our approach also yields consistent improvements on the MS COCO object detection benchmark and introduces only a modest 5% computational overhead during training, with no additional cost at inference.

Published in Transactions on Machine Learning Research (08/2025)
Unified Wisdom:
Harnessing Collaborative Learning to Improve Efficacy of Knowledge Distillation
Atharva Abhijit Tambat"
athorvutambattOcseiittbbb
Deportment of Computer Science and Engineering Indian Institute of Technology, Bombay
Durga Sivasubramanian"
durgas@cse. wwt.::s.s..............
Department of Computer Science and Engineering Indian Institute of Technology, Bombay
Ganesh Ramakrishnan
gonesh@cse..ülu
Department of Computer Science and Engineering Indian Institute of Technology, Bombay
Pradeep Shenoy
shenoypradeep mrtnnopprudeep@ggggge..cm
Google Mind
Reviewed on OpenReview:
Abstract
Knowledge distillation (KD), which involves training smaller student model to approximate the predictions of larger teacher model is useful in striking balance between model accuracy and computational constraints. However, KD has been found to be ineffective when the teacher and student models have significant capacity gap. In this work, we address this issue via "meta-collaborative distillation" (MC-DISTIL), where students of varying capacities collaborate during distillation. Using a "coordinator" network (C-NeT), MC-DISTIL enables mutual learning among students as meta-learning task. Our insight is that C-NET learns from each student's performance and training instance characteristics, allowing students of different capacities to improve together. Our method enhances student accuracy for all students, surpassing state-of-tho-ar baselines, including multi-step distillation, consensus en foforcemnn,, and teacher re-training. We achieve average gains of 2.5% on CIFAR-100 and 2% on Tiny ImageNet datasets, consistently across diverse student sizes, teacher sizes, and architectures. Notably, larger students benefiting through meta-collaboration with smaller students is novel idea. MC-DISTIL excels in training superior student models under real-world conditions such label noise and domain adaptation. Our approach also yields consistent improvements on the MS COCO object detection benchmark and introduces only modest 5% computational overhead during training, with no additional cost at inference.
1 Introduction
Although modern deep learning methods can effectively learn from large datasets with figh-dimensional features, their associated compute and memory requirements often exceed the constraints faced by practical applications. To address this, Knowledge distillation (KD) (Hinton al.) 2015)) trains a smaller student model to approximate the higher quality predictions of a larger teacher model. These distilled student models often outperform supervised models of similar capacity; however, such gains limited by the choice of teacher and student. In particular, larger teacher-student capacity gaps are observed to produce poorer quality student models @Cho & Hariharan 2019
"Equal contribution.

---

Published in Transactions on Machine Learning Research (08/2025)
Previous work on this challenge use altered teacher (Cho & Hariharan 2019 Li & Jin 2022 introduce
additional "assistant" models of intermediate learning capacities between teacher and student model (Mirzadeh et al. 2020 Son et al.] 2021). An interesting idea in this literature is the use of multiple teachers for a given student, providing the student with a multi-grain view of the relationship between instance input and label. However, the use of inferior or intermediate-quaali teacher models in these approaches potentially limit the quality of information available to the student. We believe, instcad, that students should have access to high-quality teacher signals, but focus more on instances that are "learnable" given their model capacity (see e.g.. Mindermann et al. (2022) for supervised learning). To this, we add the following insight: we could obtain finer, multi-grain picture of instance hardness by observing the performance of students of different capacities (as opposed to generating and utilizing multiple teachers), which could then be used for better distillation. These observations lead us to the following problem statement: Can students of different. capacities learn from each other in a collaborative distillation framework, thereby understanding how to prioritize relevant "Icarnable" information and improving distillation outcomes?
Approach: We build MC-DISTIL- -eela- collaborative distillation where students of different capacitics are codistilled from single teacher. We propose to use an auxiliary meta-network for aggregating information across students, and using it to modulate each student's learning on a per-instance basis. We term this meta- learning approach towards joint multi- student distillation as mela-collaboration As shown in Figure [ the auxiliary net- work C-NET modulates the training loss of each student through an instance- dependent reweighting of the teacher and cross-entropy losses. C-NET removes the need for intermediate teachers, and stu- dents have access to high-quality teacher signals that they can emulate on a per- instance basis. We set C-NET'S objective as the pooled student accuracy on a sepa- rate, held-out dataset, setting up nested optimization that we approximate using an efficient alternating update algorithm (see Section g for details). A key advantage of MC-DISTIL is that it produces a spectrum of student models with varying capacities all with improved generalization capabilities, which can be deployed contextually at test time to suit specific application requirements. Furthermore, the effectiveness of C-Net strengthens as more students participate in the distillation process, enabling richer collaborative gains. As we demonstrale, this framework not only benefits from pooling knowledge across diverse students but also from a fine-grained, adaptive modulation of the distillation process, where training signals are dynamically tailored to both the capacity of each student and the difficulty of individual instances.
(
Teacher (T)
(8)
8:
Cs
[ah, 9,)
(8%
8
Cx,
(arg, (8))
y (%)
5
Ds.
[ox,, (xx)
training loss
C-NET
Cs. (yy 2) + (2 ()))
Figure 1: MC-DISTIL Training Workflow. Losses are weighted according to the parameters obtained from C-NET which are then used to train the individual students.
Pooling information across students: Figure 2 shows that adding more students consistently improves performance over vanilla KD, as C-NET leverages diverse capacities to better model instance difficulty. We use subsets of ResNet10-xxs,, ResNet10-xs, ResNet10-s, and ResNet10-m (He et al. 2016 Kag et al. 2023). For ResNet.10xxs,, higher-capacity models are added incrementally, while for Res 10 smaller ones are introduced. Experiments are conducted on the CIFAR-100 dataset (Krizhevsky 2009 with 1e and
ResNet18 as teachers (Figure 2a 2b) Complete results are provided in Section 4.7
Modulating distillation at student & instance gramularity: Figure 3 shows the relative weight assigned in MC-DISTIL to training instances as function of instance hardness (æ-axis) and student capacity (different curves). We use the label likelihood gap between (eacher and largest student as instance hardness (x-axis) and the total weight on loss components (a + 3) as instance weight (y-axis). Experiment details are as above.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Res.Net10-xxs
ResNet 10-m
ResNet10-xos ResNet10-xs
Abs % Gain over KD
1,7
ResNet10-s
4
1.6
3
Relative weights
2
1.5
1
1.4
2
3
4
2
3
4
No. of students
No. of students
(a) ResNet10
(b) ResNet18
7.5
Teacher
Teacher
Instance Hardness
Figure 2: Influence of student cohort size on distillation
Figure 3: C-NET prioritizes training data based on student capacity and instance hardness. See Sec- tion 4.5 for details.
gains using meta-collaboration. See Section 4.7 for details.
As seen in Figure g C-NET implicitly learns instance hardness, encouraging weaker students focus on easier examples while stronger students learn from a broader range of data. Since instance hardness isn't an explicit input, C-NET learns implicit mapping of instance difficulty. We further inspect how the choice of loss component weights and 3 (see Equation 2 helps MC-DISTIL improve KD in Section 4.5
We extensively evaluate MC-DISTIL across diverse student teacher architectures and model sizes (Sec- tion ).1.3), demonstrating consistent gains over SOTA benchmarks. MC-DISTIL improves the performance of each student, highlighting the interesting finding that smaller student models can help improve larger students in collaborative distillation. Finally, MC-DISTI, yields improved generalization and robustness (Section
2 Related works
Knowledge Distillation (KD) In supervised learning, Knowledge Distillation (KD) Hinton et al. 2015 is valuable method where 'student' model learns by mimicking a pre-trained 'teacher' model, rather than solely relying on labeled data. The success of KD depends on factors like teacher model accuracy and student model capacity (Menon ct al. 2021 Recent rescarch (Harutyunyan et al. 2023) explores efficacy relation to supervision complexity. Using early-stopped teacher models has shown promise in improving student training (Cho & Hariharan, 2019). though it requires iterative distillation. Another approach (Liu al.) 2020) employs multiple teacher networks with intermediate knowledge transfer. The strategic blending of loss components (Sivasubramanian et al. 2023) has recently improved KD, particularly in scenarios with significant representation gaps between teacher and student models. The challenge of suboptimal KD performance caused by significant capacity disparities between student and teacher models was addressed by strategically choosing points to learn from the teacher model in Kag et al. (2023). Other innovations such as "Teacher Assistants' (TAs) or intermediate models have been introduced (Mirzadeh et al. 2020), with further enhancements achieved through stochastic techniques such as Dense Gradient Knowledge Distillation (DGKD) (Son al., 2021), which involves the simultaneous training of intermediates with occasional model dropout. Recently Li & Jin (2022) showed that the gap between online and offline KD could be bridged using shadow head on top of the backbone of the teacher model. The shadow head is trained to perform bidirectional distillation. These works indicate knowledge transfer to the smaller models, resulting in their improvement by the presence of the intermediale bigger models; however the generalization of the larger models has also been shown to improve from the knowledge of smaller models [Mindermann et al. (022)) and bidirectional learning improve KD [Li & Jin 2022)). Therefore, inspired by these bidirectional signals, we present an approach to train multiple student models simultaneously and communicate vital information via coordinator network.
3

---

Published in Transactions on Machine Learning Research (08/2025)
Online knowledge distillation: In the absence of a strong pre-trained teacher, online knowledge distillation often involves collaborative model efforts to acquire soft labels. Chen et al. (2020) aggregate predictions from peer models using an attention- baaed mechanism to obtain soft targets. Wu & Gong (021)) employ an m-branch model treating cach branch as a peer model, constructing soft distillation targets from weighted logits of these branches and an exponentially moving-averaged m-branch model. Guo et al. (2020) propose using pooled student logits with varying learning capacities as soft distillation targets. Additionally, Du et al. (2023) create curriculum using variance in predictions of multiple students with varying sparsity levels as a measure of task complexity, Zhaag et al. 2018) use KL-divergence loss between every student pair to improve consensus among student models. The primary focus in these works lies either in constructing logits due to the absence of a pre-trained teacher model or explicit synchronization of the student models. In contrast, our approach centers on influencing the learning palterns of the models participating in the collaboralive process via signals from peer models.
Instancc-Specific and Meta-Lcarning: Instanco-specific learning has been greatly explored to improve training in the presence of noise. Saxena et al. (2019); Algan & Ulusoy (2021) Vyas et al. (2020) propose to learn instance-specific temperature smoothing parameters to account for potential label noise. In the context of knowledge distillation, Zhao et al. (2021) have demonstrated the advantages of learning instance-level sequences (or curricula) for training samples. A similar instance- srstance-wiseeeee weighing scheme has been proposed to improve distillation in semi-supervised settings [Tiopoulls et al. 2022) Recent contributions such as that in Ren et al. (2018); ; Shu et al. (2019); Raghu et al. (2020); Jain & Shenoy (2022) employ meta-learning based on validation sets to acquire instance-specific weights, enhancing robustness. We introduce novel approach, MC-DISTIL that involves the udilization of meta-learning based on volidation sets to facilitate collaborative learning process among maltiple models. To the best of our knowledge, this has not been previoushy explored.
3
MC-Distil: Meta-collaborative distillation
3.1 Standard Knowledge Distillation
In supervised learning, one is concerned with training a classifier y = fo using data D D= ((.;, yi) e (I,...,m)). Here, (I,, v1) E X' y denote pairings of inputs x, along with their corresponding labels ye and 0 denote the model parameters. Typically, the labels y are categorical, and practitioners have found that such labels are often inadequate in capturing nuances in the data. A popular mitigation is to incorporate more muanced 'soft labels', or distributions over labels, as the target for supervision instead of categorical labels. In particular, Knowledge distillation (KD) (Finton et al., 2015) uses the logits from a pre-trained model (the 'teacher' model) as soft labels for training a classifier, in addition to the standard categorical labels.
Suppose a pre-trained model (teacher) generates logits denoted by y(T) T(x). Then a new model (student) could be trained using a "teacher matching" objective that involves minimizing the KL-divergence between y(T) and the student's logits y( The KD objective is follows:
C. (-=((-M))) A)cce + () D
Here, lcc ()( is supervised learning loss matching (S) to the true labels y, and led 7 (K is the matching loss. Typically, H is the standard cross-entropy loss. The hyperpa- rameters T, control the softening of the KL--ivergence term, and the relative contributions of the two loss components.
3.2 MC-Distil: Multi Student Knowledge Distillation
Successful application KD often depends on the quality of the pre-trained teacher model (see e.g.: MMenon ct al. 2021)): ), and the representational gap between the teacher and student models. To address the capacity gap issue, often several Teacher Assistants (TA) or intermediate models are introduced (Mirzadeh et al. 2020
Son et al. 2021 ). The success of the such multi-teacher approaches suggests that supervisory inputs from
4

---

Published in Transactions on Machine Learning Research (08/2025)
teacher models of different sizes enrich the information available to the student. We leverage this insight in completely different setup where single teacher is simultaneously distilled into multiple cooperating student models. Our primary thesis is that the diversity in student capacities and learning behaviors can be leveraged to tailor the distillation process more effectively for each student.
Specifically, we propose to learn adaptive, instance-wise distillation losses using a coordinator network, denoted as ge (referred to as C-NET). This network modulates the contribution of each training sample to the student models based on their individual learning dynamics and their relationship with the teacher. Formally, we define set of student models S {S) | e (1. ... })), single pre-trained teacher model T, and a shared coordinator network Ip. The overall training objective for MC-DISTIL is then defined as:
outer-level
argmin
Σ
CN-Ner argmin Lo, (0,, v)
(2)
f
/=1
inner-level
where (onis...., (uai) (B1i) .. ,Ba) - go( = (aji,' ,gnn), B3 (B31, B3m)
and Lxz
Σ
(I)HH())))))
B +
. 11))
(..D:)ED
3.2.1 Coordinator network Io (C-Net):
Our key innovation in Equation 2 is to use an additional "coordinator network", which we call C-NET (Figure to address two key challenges, viz., a) reducing the number of learnable parameters and b) introducing interaction among the student models. The C-NET is a learnt function B,) = (o(),,, where A; [a1...... [.d] B, = [Brr,..., 3kx]] represent the loss mixing parameters for the i students on the input z4- Therefore, the combined BA for the entire dataset would result in matrix of size k, which can incur significant memory overhead as the training dataset grows. Thus, to compactly represent the loss mixing C-NET parameterized by ò E Φ. Moreover, the C-NET is centralized mechanism, it acts as a shared interface across students and training instances, cnabling the coordinator to incorporate the relative learning dynamics of models with different capacities. This allows C-NET to assign more informed, adaptive weights to each student based on instance-speciftc difficulty and student model behavior. The training procedure for the C-NET described in the following section.
parameters
Training the C-Net:
We train the 96 (C-NEr) using separate validation set of data V (<*),)]) E (1....,)) The objective is to learn & such that the resulting student models generalize well-i.c. -.c. they achieve low prediction error the validation data. Specifically, we minimize the average cross-entropy loss across all I student models on the validation set:
LC-NET
H (5)) b
H (+),)))
(3)
kmm
km
1=
Here, :):-)) denotes the cross-entropy loss between the predicted label from student j and the ground truth. Crucially, Le-Ner depends on only implicitly via the student parameters 0j, which themselves are optimized using the loss mixing weights generated by go(-) (see Equation 2). In other words, our proposal defines a bi-level optimization that encompasses both the C-NET and classifier parameters. Thus, the optimal values depend on the optimal choice of & and vice versa.
For most modern deep learning models, solving the inner optimization in Equation in closed form is intractable due to the non-convexity and scale of the models. To address this, we adopt an iterative strategy to approximate the solution to the bilevel optimization problem. Rather than fully optimizing the inner
5

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm 1 The MC-DISTIL approach: learning student ...... Sk. Training data D, Validation data V. teacher T and C-NET 9é-
Hyperparameters: Temmeeraurr, ti w. learning rates for k students, 7/2 learning rate for C-NET, L epoch interval for C-NET update
Initialize student model parameters with and C-NET with de 2: for E {0, ..... T do
3: for j E {1..... do
4: Update 05t1 by Equation
5: end for
1%L = then
(x". SampleMsmiBatch(()
Compute Ca-Ner using fa", v²) and (0f+1) described in Equation
Update Øl+1+1 by Eq. Equation
10: end if
11: end for
objective before updating the outer parameters, we employ an alternating stochastic gradient descent (SGD) approach. This involves interleaving updales of the student model parameters and the C-NET. Specifically, we perform few gradient steps on the student parameters using the training set and the weights predicted by the C-NET, followed by an update of the C-NET parameters based on validation loss (Equation 5p. This alternating scheme provides a practical and efficient means of optimizing the overall objective. The update steps are summarized as follows:
0,41
03
Iil Vot H(, (5,) u. + 1gglll x V (27 4J E (1,..,,
(4)
6 11
72
of (2. ( =
1/2
(1, ) * +V 0.
(5)
km
Here, denole 0++1_ (0L+11, ,0+1k the collection of updated student model parameters. The learning rates for the student models and the C-NET are represented by -.. . and 172, respectively. We use the notations L8rbi = at and 9 lg = 3 to remind our readers that as and As output from the C-NET (g(.)) parameterised by ø. The update step for the C-NET is similar to the standard meta-learning objectives it uses the updated student model parameters. We present the complete algorithm MC-DISTHL in Algorithm [ Since training C-NET adds to the cost of training, we propose to update C-NET only after L epochs. In Appendix A show theoretically that our method converges to the optima of both the validation and training loss functions under some mild conditions.
4 Experiments
We conduct experiments on a variely lasks. First, we compare our method with other knowledge distillation baselines in terms of image classification. We experiment with different settings varying architecture and datasets. Also, apply method to the object detection task.
4.1 Image Classification
For image classification, we evaluate across diverse datasets CIFAR-100 Krizhevsky 2009), Tiny Im- ageNet (Le & Yang) 2015) and ImageNet-1K (Russakovsky et al. 2015 In addition, we consider iWildCam (Beery al. (020), Tiny ImageNettCC (Hendrycks Dietterich 2019). and Clothing- 1M (Xiao al.J a (2015) as part of our further analysis. Dataset details, including splits, input dimensions, and augmentations, appear in Appendix B.T with training configurations in Appendix B.3

---

Published in Transactions on Machine Learning Research (08/2025)
CIFAR-100 Test Accuracies
Teacher
Student
KD
TAKD
DGKD
RMC
DML
SHAKE
MetaDistil
Ours
ResNet10-xxs
31.85
33.45
34.39
35.31
34.07
33.41
33.56
34.92
36.19
ResNeet
42.75
44.87
44.97
47.11
45.18
44.16
42.12
46.01
47.57
ResNot10-1
72.20
ResNet10-s
52.48
55.38
56.16
57.02
53.74
55.13
56.50
57.20
58.36
ResNetlO-m
64.28
66.93
66.66
66.06
68.80
68.28
69.42
10-xet
31.85
33.95
34.98
34.85
33.64
33.54
33.70
34.06
35.97
10sNe
42.75
44.87
45.64
46.68
42.45
44.70
44.20
46.32
47.53
ResNet10
75.18
ResNet10-s
52.48
55.56
56.51
56.84
53.64
55.29
56.59
57.78
58.10
ResNet10-m
64.28
67.27
66.58
66.25
68.63
68.89
69.21
R0sNet
31.85
33.56
34.26
34.26
33.77
34.02
32.32
34.40
35.94
RosNet10-xs
42.75
45.02
15.27
47.33
45. 14
44.10
43.82
46
46.99
ResNet18
76.99
ResNet10-s
52.48
55.73
55.
56.70
54.03
541822
56.82
57.40
57.50
ResNet10-m
64.28
66
66.04
65.94
68.12
68.43
68.45
ResNet
31.85
34.46
35.64
34.46
33.76
36.10
ResNet10-xs
42.75
44.94
45.92
47.21
42.78
44.45
45.80
46.43
47.12
ResNet33
79.47
ResNet1O-s
52.48
54.73
56.17
57.12
53.58
55.28
56.20
56.91
57.67
ResNet10-m
64.28
66.52
65.58
66.88
68.96
68.09
68.24
Tiny
ImaggeNNt
Test Accuracies
10-xss
13.76
13.53
13.81
14.34
13.69
13.95
14.70
14.78
15.48
RusNet10-xs
18.56
19.19
19.22
20.54
19.04
19.57
19.92
20.13
21.68
ResNet10-1
41.25
ResNet10
24.56
25.95
26.35
27.24
25.86
26.20
27.00
27.06
29.74
ResNet10-m
33.47
34. 63
33.72
35.34
36.66
36.02
38.26
xes
13.76
13.80
14.01
14.52
13.83
14.01
14.20
14.19
15.36
ResNet10-xs
18.56
19.48
19.09
21.21
19.28
19. 78
19.87
20.13
21.78
ResNet10
44.04
ResNet10~s
24.56
26.95
25.58
26.99
26.18
27.04
27.79
27.06
29.82
ResNet10-m
33.47
35.50
34.78
35.79
38.01
36.02
38.54
ResNet10-xxs
13.76
14.12
14.53
14.08
15.26
10-Net
18.56
19.78
19.54
19.75
18.24
21.36
ResNet18
47.94
ResNot10~s
24.56
26.30
26.17
27.42
25.08
25.92
27.92
27.32
30.36
ResNetl0-m
33.47
35.08
33.37
36.77
38.60
36.08
39.11
ResNet10-xxs
13.76
14.43
13.47
14.58
13.78
14..0
14.67
13.96
15.24
RusNet10-xs
18.56
19.72
18.33
20.84
19.28
19.29
19.76
20.93
21.82
ResNet34
50.10
ResNetlU-s
24.56
27.05
24.96
27.80
25.99
26.81
28.05
27.
29.59
ResNet10-m
33.47
35.94
33.58
35.88
37.97
36.88
38.34
Table 1:
Comprehensive comparison of methods across datasets. For each teacher model, we perform knowledge distillation with group of student models. MC-DIsTH. (Ours, last column) substantially improves the average accuracy on unseen test data compared to other distillation basclincs, especially those designed for multi-student setups. The highest accuracies are highlighted in bold.
4.1.1 Model Architecture and Training
To demonstrate the utility of our method across groups of different model sizes we experiment with a group of ResNet (He et al. 2016) models and several recent larger models. We use the ResNet32 model C-NET with the classification head changed to output weighting parameters. We present results of varying C-NET size in Section 4.8 We use ResNet10-xxxs, ResNet10-xxs, ResNet ResNet10-s and ResNet10-m (Kg el al. 2023) models as the student models. These models are simultaneously trained with either ResNet 10-1, ResNet10, ResNet18 or ResNet34 models as a teacher model. In Appendix [.2 we present details these modcls. We present experiment results on these combinations Table Henceforth, in tables, denotes that for basclines TAKD (Mirzadeh et al. 2020) and DGKD (Son et al. (2021). the initial distillation step coincides with standard KD from the teacher to the largest student, and is therefore not reported separately.

---

Published in Transactions on Machine Learning Research (08/2025)
CIFAR-100 Test Accuracies
Teacher
Student
CE
KD
TAKD
DGKD
RMC
DML
SHAKE
MetaDistil
Ours
RusNett-8od
71.12
72.62
74.26
74.45
73.89
13.37
73.96
73.12
75.38
MobileNet-V2x2
69.52
71.78
72.07
72:27
71.23
72.46
72.17
72.33
73.05
ResNet-32xd
80.10
WideReaNet-10x2
72.79
73.34
73.72
74.12
75.19
73.87
71.22
71.10
75.52
ShaffleNet V2
73.73
75.33
76.52
75.98
76.12
76.74
77.97
ResNet-Sxi
71.12
72.77
73.82
74.63
73.84
74.12
74.31
74.49
76.73
MobileNet-V2ry
69.52
72.69
72.69
71.61
70.90
73.97
74.68
74:20
74.36
WideResNet-10x2
77.60
WideResNet-4ung
72.90
73.01
73.72
74.67
75.44
ShuflleNet V2
73.73
75.85
76.72
77.87
78.08
78.17
77.94
Tiny
ImageNet Test Accuracies
ResNet-Sxd
37.16
37.23
38.83
39.52
36.76
38.86
38.12
40.46
41.89
MobileNet-V2x2
17.68
49.89
49.89
48.2
48.07
18.21
19.79
19.85
49.95
ResNet-32xi
50.20
WideReNNet-16x2
39.11
39.17
41.66
41.77
39.77
11.35
41.98
42.12
43.88
ShuffleNet-V2
47.76
50.44
49.46
49.24
49.46
50.62
52.38
Table 2: Comprehensive comparison of methods when training models with larger learning capacity. Here again MC-DISTIL (Ours, last column) substantially improves the test accuracy compared to other distillation baselines even in this sctting. The highest accuracics are highlighted in bold.
To illustrate the utility of our method in larger vision models we perform knowledge distillation with ResNet-32x4 as the leacher and ResNet-8x4, ShuflleNet-V2 (Ma el. al. 2018), WideResNNet-1622 (Zagoruyko & Komodakis 2016) and MobileNet- V2x2 (Sandler et al. 2018) the group of student models. We also perform knowledge distillation with WideResNeet40xx as a teacher model and ResNet-8x4, ShuffleNet-V2, Wed-4ees and MobileNel-V2x2 as a student model group in the large vision model setting. The results of these experiments are presented in Table
4.1.2 Baselines
In our comparative analysis, we assess the performance of our method against selection of recent works in the field of knowledge distillation, alongside standard knowledge distillation and Empirical Risk Minimization (ERM) Cross-Eio Entropy based training (CE). Specifically,
compare against TAKD (Mirzadeh et al. 2020 and DGKD (Son et al. 2021) basclines involving multiple students or intermediate models, RMC(Du et al. 2023) uses variance in predictions of students of levels of sparsity as a measure of task complexity for each instance, DML (Zhang et al. 2018) forces collaboration among the models by introducing a KL-divergence loss across different models and SHAKE (Li & Jin 2022) which introduces pseudo teacher that allows bidirectional learning during KD. We also compare MC-DISTIL against one more bascline that involves distilling knowledge to each of the students independently using a network architecturall, similar to C-NET. We refer to this baseline the MetaDistil. This baseline is similar to AMAL (Sivasubramanian et al. 2023 the strategic mixing loss components are achieved via the C-NET optimization. Further, in instance dependent label-noise selting we compare against L2R (Ren et al., 2018) and MWN (Shu et al.] 2019) reweighing techniques created for eliminating label noise by using bi-level optimization and MCD (Gal & Ghahramani 2016) which uses dropout to model uncertainties. Please refer to Appendix B.4 for more information on these methods.
4.1.3 Improving efficacy of Knowledge Distillation
In Table [ we present results from experiments conducted CIFAR-100 and Tiny ImageNet datasets, exploring scenarios with a significant capacity gap between teacher and student models. We start with 10- as the teacher and go on to continue increasing learning capacity of the teacher model and perform knowledge distillation with ResNet10, ResNet18, and ResNet34. On both the datasets, MC-DISTIL, by virtue of mnta-collaboration, achieves the best performance among the basclines showing accuracy gains of up 4% on both the datasets compared KD. The gains are much more pronounced on the larger models in the student pool for the Tiny ImageNet dataset owing to the increased difficulty in classifying it whereas for
8

---

Published in Transactions on Machine Learning Research (08/2025)
ImageNet-1K Test Accuracies
Teacher
Student
CE
KD
DGKD
SHAKE
MetaDistil
MC-Distil
ResNet14
22.28
19.00
20.92
21.15
22.84
23.79
ResNer20
26.87
22.10 10
24.31
24.48
26.44
27.94
10sNet101
81.68
ResNet32
31.04
26.83
29.34
29.26
31.56
33.07
ResNet56
37.56
33.38
34.82
37.90
39.73
Table 3:
Comprehensive comparison ImageNeet-1K test accuracies of student models distilled from a ResNet101 teacher using different knowledge distillation techniques. The proposed method (MC-DISTIL) consistently outperforms baseline approaches across all student architectures, Bold indicates the highest accuracies.
CIFAR-100 the gains are pretty uniform across the student models. These gains are consistent across a wide range of student and teacher capacities. MC-DISTIL improves all of the student models' performances as compared to the baselines, thereby showing that joint distillation of knowledge to a student set is beneficial for both smaller and larger students. We analyze the effect of introducing student models one by one in the presence of C-NET in Section 4.7
MC-Distil remains competitive even in scenarios with small capacity difference. As illustrated in Table 2 MC-DISTIL maintains its competitive advantage over KD, even when the student model closely matches the size of the teachers, achieving gains of up to 3% relative to KD. These improvements can be attributed to two key factors: (i) the reweighing of loss terms and (ii) meta- collaboration. The reweighting strategy aligns the learning focus with the model's capacity, emphasizing examples that are more suitable for the student. This targeted weighting helps explain the performance improvement observed with MetaDistil over standard KD, particularly for students like 'ReNNet-m'. However, just this reweighing not sufficient for students such as "essNetxxss' and 'Resnet-s" in the case of larger teachers. This is where MC-DISTIL' 's ability to leverage C-NET as a communication channel among student models is useful in enhancing knowledge transfer from the teacher model. While the benefits of information flow from intermediate models to smaller ones, as demonstrated in previous studics (Son ct al.) 2021 Mirzadeh et al., 2020), are well-ostablished for improving final performance, the reverse scenario remains under-explored. The gains reported in Tables) and / indicate that larger models can also benefit from information exchange with smaller models, akin standard supervised settings (Mindermann al. 2022).
MC-Distil outperforms bidirectional learning methods in KD. We compare against two KD methods viz. SHAKE's & Jin 2022) and DML (Zhang et al. 2018). both aim to enhance teacher outputs to improve KD's effectiveness. SHAKE learns a pseudo-teacher (a few additional learnable layers on the teacher's backbone) to adapt according to the student model. This performs relatively well for larger students but struggles to accommodate the specific needs of smaller students in Table 1 contrast, MC-DISTIL carefully controls the knowledge transfer from the teacher model, proving more beneficial for smaller students. On the other hand, DML's forceful explicit collaboration among student cohorts results in inferior performances. This is because aligning outputs restricts larger models from learning to their full capacity and burdens smaller models with the task of matching the outputs of their larger counterparts, similar to traditional KD. In Section 4.10 we show that such collaboration can be added to our setup and if carefully reweighted can lead to improved performances.
4.1.4 MC-Distil in extreme teacher student gap settings
To evaluate the resilience of MC-DISTIL under a substantial teacher student capacity gap, we conduct. knowledge distillation the ImageNet-1K dataset using high-capacity ResNet101 teacher model, which is significantly larger than the student models ResNet14, ResNet20, ResNet32, and ResNet56. Due to computational constraints, we compare only against the most competitive baselines: DGKD, SHAKE, and MetaDistil. The results for the same are given in Table 3
Given ImageNet's complexity and the large teacher-student disparity, KD (Knowledge Distillation) under- performs even compared to standard Entrroy (CE) training, aligning with prior findings from Cho & Hariharan 2019 The alternative methods designed to mitigate poor transfer (such DGKD and SHAKE
9

---

Published in Transactions on Machine Learning Research (08/2025)
MS COCO
Test Average Precision Values
Metric
Teacher
Student
CE
KD
TAKD
DGKD
RMC
SHAKE
MetaDistil
Ours
RN18
28.04
31.49
33.91
34.06
32.18
35.13
36.51
38.20
mAP
RN50
46.10
RN34
31.10
37.11
36.06
39.79
40.69
42.70
RN18
45.45
51.12
51.45
51.73
51.44
50.91
51.70
52.90
AP50
RN50
60.00
RN34
51.71
57.41
56.61
58.37
58.36
58.49
RN18
28.90
32.86
34.04
35.06
36.57
37.07
39.63
41.59
AP75
RN50
50.20
RN34
32.34
38.78
42.44
41.97
44.41
46.48
Table 4: Comparison of mAP, AP50, and AP75 on MS COCO Dataset [Lin et al. 2014) for Faster R- CNN with ResNet50-FPN teacher distilled into ResNet18-FPN and ResNet31-FPN students using various distillation methods. MC-DIsTH. (Ours, last column) consistently improves performance across all metrics. Bold indicates the highest accuracies.
provide some improvements over KD, they still fail surpass CE, with the exception of MetaDistill which only matches CE performance. Despite these challenges, MC-DISTIL achieves a consistent 1-2% improvement over CE, demonstrating its effectiveness even in extreme teacher-student gap scenarios.
4.2 Object Detection
For object detection, evaluate the MS COCO benchmark (Lin al. 2014), using the standard metrics mAP, AP50, and AP75. The distillation baselines are the same in the image classification experiments KD (Hinton et al.) 2015), TAKD Mirzadeh et al. 2020). DGKD (Son el. al. 2021), RMC (Du et al. 2023), , and SHAKE ÚLi & Jin 2022) with the KD loss adapted to object detection similar to described in Chen et al. (2017). Datasct details, including splits, input dimensions, and augmentations, appear in
Appendix B.1 with training configurations ions in Appendix B3
4.2.1
Model Architecture and Training
We employ Faster R-CNN Ren et al. 2015) as the base detector, with a high-capacity ResNe150-FPN leacher distilled into lower-capacity 1esNeet8-FPP.. and ResNet34-FPN students. We use the ResNet34 model as C-Net with the classification head changed to output weighting parameters. All models are initialized from ImageNet pretrained weights and fine-tuned following standard detection pipelines. Appendix B.2 presents details of these models.
4.2.2 Boosting Detection Performance with MC-Distil
Table 4 reports detection performance on MS COCO. Across all metrics-mAP AP50, and AP75-MC-DisTI.
achieves the highest performance, surpassing both traditional KD and advanced distillation basclines such TAKD, DGKD, and RMC. Notably, MC-DisTUL provides an absolute gain of up to 4-5 mAP points over KD, with larger improvements stricter thresholds like AP75, underscoring its ability to transfer fine- ine-grained localization knowledge effectively. While SHAKE performs well on ResNet34-FPN, it underperforms on the smaller Res.et18-FFP. due to its pseudo-teacher bias, whereas MC-DISTIL adapts the teacher's ' knowledge to both small and mid-sized students through meta-collaboration. The consistent. performance boost across loU thresholds highlights that our approach better aligns detection outputs between teacher and students, resulting in more precise bounding box predictions.
4.3 MC-Distil trains models with better generalisation
We perform knowledge distillation on the challenging iWildCam dataset (Beery et al. 2020). The challenge arises from variations in the capturing environment, resulting in a demanding test set (see Appendix B.1] Table shows the performance of various knowledge distillation methods based multi-student setups,
10

---

Published in Transactions on Machine Learning Research (08/2025)
CaildCam Test Accuracies
Teacher
Student
CE
KD
DGKD
DML
SHAKE
MetaDistil
MC-Distil
ResNet10-xxs
75.40
75.92
76.17
76.12
76.72
76.85
76.96
ResNet10-xs
77.30
77.37
77.86
77.50
78.30
78.27
78.82
10-1
89.05
ResNet10-s
79.97
80.42
80.30
80.51
81.79
81.94
82.80
ResNer10-m
84.77
84.94
85.80
86.91
86.52
87.35
ResNet10-xxs
75.40
75.85
76.20
76.17
76.25
76.82
76.94
ResNet10-xs
77.30
77.45
77.65
77.72
78.38
78.04
78.97
ResNet10
91.86
ResNet10-s
79.97
80.49
80.58
80,61
82.19
81.73
82.75
ResNet10-m
84.77
85.07
85.28
86.34
86.44
87.49
Table 5: Results of knowledge distillation on the challenging iWildCam Dataset (Beery et al, 2020
We
compare MC-DISTIL against standard and advanced distillation baselines, observing substantial improvements in test accuracy across all student models. The highest accuracies are highlighted in bold.
Tiny Imagenet-C (OOD test set) Accuracies
Student
CE
KD
MCD
DGKD
DML
SHAKE
MetaDistil
MC-Distil
Corruption level 1 Accuracies
ResNet10-xxs
11.25
11.38
10.61
10.77
09.99
11.47
11.02
12.05
ResNet10-xs
14.68
15.82
14.67
15.43
15.51
16.56
16.02
16.73
ResNet10-
20.88
21.97
21.94
23.18
20.29
22.58
23.97
24.88
ResNet10-m
26.91
28.56
27.08
29.56
28.09
30.40
30.43
31.00
Corruption
level 2
Accuracies
Res.ett 10-xx8 10.65
10.79
09.54
10.42
09.44
10.88
10.52
11.30
ResNet10-xs 13.75
14.60
13.72
14.60
14.56
15.42
14.78
15.52
ResNet10-s 19.28
20.64
20.46
21.26
18.74
20.74
21.93
23.04
ResNet10-m 24.76
25.87
24.36
26.97
26.20
27.94
27.39
28.67
Table Comprehensive table presenting an assessment of model (trained with Tiny ImageNet) robustness by conducting inference on the Tiny ImageNet-C Dataset (Hendrycks & Dietterich 2019) across different degrees of image corruption, alongside results for all baseline models. The columns indicate the method employed for training the model the Tiny Imagenet Dataset, utilizing ResNet18 the teacher model. Corruption level 2 reflects a higher severity of corruption than level 1. The highest accuracies are highlighted in bold.
including MC-DisTu. Across both weak (ResNet10-1) and strong (ResNet10) teachers, and for wide range of student model capacities from extremely small (ResNet10-xxs) to moderately sized (ResNett00mm) MC-DISTIL consistently achieves superior test accuracies.
The consistent performance of MC-DisTu. highlights its robustness leveraging instance-level importance through soft selection, enabling better generalization complex, real-world scenarios such as iWildCam. This reinforces the effectiveness of our method not only in standard benchmarks but also under real-world distribution shifts and limited model capacities.
Using models trained with ResNet18 as the teacher on the Tiny ImageNet datasct (reported in Table 1 we classify Tiny ImageNet-C (Heudrycks & Dietterich 2019). Table presents the accuracies obtained by models trained with different KD methods across two levels of corruption (details in Appendix B.I). Appendix C.2 we present the complete table with results obtained across all five corruption levels. Notably, MC- DISTIL produces models that demonstrate robustness and superior generalization, even when the test data distribution differs from the training set.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Student
CE
KD
MCD
MWN
L2R
MetaDistil
MC-Distil
Instance CIFA 100
Accuracies
RN10xxxS
23.69
23.67
24.33
24.14
24.88
26.48
27.07
RN10xxs
28.97
28.55
29.81
28.66
29.04
29.59
31.88
RN1Oxs
38.04
38.19
38.88
37.34
39.39
42.57
42.18
RNIOs
48.40
49.22
49.71
45.84
48.14
52.67
53.26
RN10m
59.42
60.14
61.62
55.74
60.90
60.64
61.61
Clothing- 1M A Accuracies
RN10xxxS
43.56
44.58
45.65
41.72
45.72
46.08
46.43
RN10xxs
51.67
51.28
52.22
48.74
51.60
52.88
52.58
RN10xs
57.14
57.16
57.51
53.49
56.70
57.06
58.28
RN10s
61.00
62.02
61.59
48.74
60.34
61.31
62.27
RN10m
64.83
66.55
64.86
61.21
63.45
63.64
64.92
Table Assessment of performances of de-noising and distillation methods on instance-specific noisy datasets. Results are reported for multiple student models distilled from a RosNet18 teacher on Instance CTFAR-100 (Xia et al. 2020) (lop) and Clothing-IMM (Xiao al.) 2015) (bottom). Our method (MC-DISTIL) consistently achieves the best performance, outperforming 4 out of 5 student models on both datasets.
4.4 KD in Instance Dependent Label-Noise
We now evaluate MC-DISTIL under a setting where instance-dependent label noise is present in the labels. We illustrate this experiment. on two datasets, namely: Instance C-FAR-100 (Xia et al. 2020) and Clothing- 1M (Xiao et al. 2015) where the labels inherit noise due the data generation process against randomly flipping the labels. The results
reported in Table 7 Since MC-DISTIL employs reweighting scheme, C-NET learns weights to reduce the impact of the noise. For the inputs with high noise, it is able to assign low weight to CE loss and rely more heavily on the distillation term, thereby outperforming other methods. Furthermore, the joint distillation (by communication via C-NET) of different-sized student models provides additional valuable information to combat noise effectively.
4.5 Why MC-Distil uses instance and loss component specific weights
We expand on the analysis presented Figure 3 to understand how C-NET as- signs weights to individual loss compo- nents. Figure 4al shows how as, param- eter controlling learning from the hard label and Figure 4b shows how A8, param- eter controlling learning from the teacher model (see Equation 2 changes with in- stance hardness for models with differ- ent learning capacity. We define instance hardness as the teacher largest-student likelihood gap because larger likelihood gap indicates that the student struggles to match the teacher's confidence, implying higher instance difficulty. For the smaller student models, learning majorly happens via hard labels, whereas larger models pro- fer learning from the teacher's nuanced soft labels. KD improves the label stu-
ResNatl
ResNett ResNett
ResNett
6.755
0.750
0.725
0.700
6.15
-73
-30
-25
3.0
25
5.
75
20.2
1.5
2.5
10.0
Inslance Harchess
Instance Hardness
Instance hardness
(b) 8 Instance hardness
Figure 4: Plot between student model errors and weighting param- eters during knowledge distillation is performed using Re as the teacher model on the CIFAR-100 Datasct (Krizhevsky 2009). It illustrates how MC-DISTIL adaptively adjusts the loss component weights to guide students varying capacities
12

---

Published in Transactions on Machine Learning Research (08/2025)
ResNetlO-xor xxs
ResNet1O-xs
ResNet10-s
ResNet10-m
Abs % Gain over KD
3
1
2
2
No. of students
No. students
No. of students
No. of students
(a) Adding larger students with ResNet10 model as teacher
(b) Adding larger smu- dents with ResNet18 model teacher
(e) Adding smaller stu- dents with ResNet10 model as teacher
(d) Adding smaller stu- dents with ResNet18 model as teacher
Figure 5: Analysis of the effect of introducing student model cohorts for different learning capacities in meta-collaboration setting.
dent model's performance by offering nuanced soft labels targets. However, when the student model cannot encode these muanced soft labels, their introduction can worsen the student model's performance Cho & Hariharan 2019). Therefore, choosing a favorable ground truth is also important along with focusing on the "learnable" points. MC-DISTI. achieves this by using two sets of weighing parameters as and Aas, and carefully tuning it according to student model capacity. For the smaller models, MC-DISTIL carefully introduces soft teacher labels of only the casy points, and thus MC-DISTIL improves the smaller modcl's performance over traditional KD.
4.6 Scalability of MC-Distil with very large student cohort
As demonstrated in Section 4.1.4 MC-
DISTIL scales effectively to ImageNet-1K,, large-scale dataset with over 1.2 mil- lion training images and 1,000 classes. To further demonstrate scalability, we ex- tend this analysis to six student models of increasing capacity the CIFAR-100 dataset. As shown in the Table g MC- DISTIL consistently outperforms both cross-entropy (CF) training and standard knowledge distillation (KD) across all stu- dent sizes from very compact models like ResNet10-xxx to full-capacity ResNet10. For instance, the smallest model (ResNett10xxxs) improves from 31.85% (CE) to 35.94% (MC-DISTIL), while even the largest student (ResNet10) benefits, reaching 77.2% accuracy compared to 76.73% with KD. This consistent improvement highlights MC-DISTIL'S robustness across a wide range of model capacities.
Teacher
Student
CE
KD
MC-Distil
ResNet10-xxs xxx
31.85
33.56
35.94
ResNet10-xs
42.75
45.02
46.99
ResNet10-xs
52.48
55.73
57.50
ResNet18
76.99
ResNet10-m
64.28
66.42
68.45
ResNet10-1
72.20
73.85
74.55
ResNet10
75.18
76.77 76.73
77.20
Table 8:
Performance of MC-DIsTI, the number of student
cohorts increases on CIFAR-100 Datasct (Krizhevsky 2009).
4.7 Ablation: Changing size of student cohort
To investigate the impact of introducing additional students in the presence of C-NET, we conducted an experiment in which we incrementally introduced student models, one at a time. We present the results of these experiments in Figure This experiment conducted in two distinct settings: one in which each subsequent addition involved student with larger learning capacity. This is presented in Figure 5p and 5p. The other setting is in which each additional introduced student is of smaller learning capacity as shown in Figure 5p and B.. In both settings, we perform knowledge distillation with two teachers viz., ResNet10 and 18. We note that across different teachers, introducing additional students improves the performances of all the students participating the training process. This is much more pronounced the setting where larger students are added.
13

---

Published in Transactions on Machine Learning Research (08/2025)
C-Net
Teacher
Student
ResNet10-1
ResNet10
ResNet18
ResNet33
ResNet10-xxs
36.13
34.95
36.00
36.38
ResNet10-xs
47.40
46.88
47.19
47.69
ResNet10
75.18
ResNet1O-s
56.93
57.48
57.70
58.24
10-N
68.65
67.95
67.91
69.11
ResNet10-xxs
36.12
34.48
35.28
36.14
ResNet10-xs
47.
46.44
46.98
47.68
ResNet18
76.99
ResNet10-s
57.12
57.82
57.65
58.21
RcsNet.10-m
69.72
69.72
68.96
69.62
Res.Ne110-xxs
35.93
34.71
35.65
36.13
ResNet10-xs
47.31
46.44
46.42
47.52
ResNet34
79.47
ResNet10-s 57.64
57.30
56.89
57.70
ResNet10-m 69.28
68.20
67.54
68.49
10sxet
35.88
34.35
35.76
35.90
ResNet 47.28
45.80
46.59
47.42
ResNett0-1
41.25
ResNet10-s
57.51
56.61
56.49
57.74
ResNet10-m
68.86
67.94
67.49
68.54
Table 9: The table reports test accuracies for various student-teacher- C-NET combinations, analyzing the impact of C-NET size model performance over CIFAR-100 Dataset.
4.8 Ablation: Varying the size of C-Net
Table presents analysis of Lest accuracies on the CIFAR-100 dataset for different student archilect under the influence of various sizes of the coordinater network (C-NET). We observe that, in the majority of instances, there are negligible alterations in the test accuracy of student models when it is trained with coordinator networks (C-NET) of varying sizes. The lack of significant changes suggests that the choice of C-NET size has minimal impact on the student model's ability to capture and generalize from the teacher's knowledge. This stability across different C-NET sizes indicates robustness the knowledge distillation process, emphasizing that the coordination mechanism implemented by C-NET is effective across range of architectural scales.
4.9
Ablation: Changing size of the validation data
We allocate 10% of the training data as a validation set, as detailed in Ap- pendix B.I However, to demonstrate that MC-DISTIL is not heavily dependent on large validation sets, we conduct ad- ditional experiments with reduced vali- dation set sizes sessssfccclllllly 2% and 5% of the training data. The corresponding test accuracics are reported in Table 10 where ResNet18 is used as the teacher model for distillation on the CIFAR- 100 datasct. As shown, MC-DISTHL main- tains stable and competitive performance across all student models even with signif-
Teacher
Validation Set Size
Model
Acc.
Student
2%
5%
10%
ResNet10-xos
36.05
36.22
36.54
ResNet
47.23
47.53
47.72
ResNet18
76.99
ResNet10-s
57.73
57.91
58.00
ResNet10-m
68.93
68.95
69.01
Table 10: Performance of MC-DISTIL under varying validation dataset sizes, highlighting its sensitivity and overall robustness to validation data availability. Knowledge Distillation is per- formed using ResNet18 the teacher model on the CIFAR-100 Datasct KKizhevsky 2009).
14

---

Published in Transactions on Machine Learning Research (08/2025)
icantly smaller validation sets. This indicates that the method is robust and can effectively learn meaningful instance-level weights without requiring requiring supervision from large validation corpora. The performance gains reported in Tables ! and 2 can therefore be achieved with minimal validation overhead, making MC-DISTIL practical and scalable for real-world applications with limited validation data.
4.10 Incorporating online distillation techniques in MC-Distil
Consensus across students: To further encourage information sharing across students, we propose an additional term in the loss function (Eqquation 2). This term minimizes the KL divergence between each student's logits and consensus representation of logits across students (a representation we call the "Pooled Student") "). For training instance (æ; y) E D such that the true label is c, i.e., vlc = 1, first define the PooledStudent logits for each label d:
(P max (y(S)) g (1)) =l
(6)
min (y (5)))) ... (bx) (1), otherwise
This is similar to MinLogit introduced in Guo al. 20200) Using the PooledStudent logits, the loss function in Equation is updated to:
(8,) Ois H yv .h) Bus (L
(8,) Wi
(
KL(V, (5,) :1i (PS) Vj (1....,A, .A.
ThyT
Correspondingls, the C-NET out- puts are extended to be (A, (x(x). Putting it all together, MC- DISTIL minimizes the total loss over all students, í.e., L. = E, L.y. We investigate the impact of incorporat- ing loss component derived from the pooled student, introduced in Equation 6 in Table 11 The test results labeled MC-DISTILPPS corre- spond to the model trained with the pooled student-based loss compo- nent. It is noteworthy that integrat- ing this new logit proves beneficial in enhancing the performance gains achieved through meta-collaborative learning when applied in a controlled manner, as suggested in Equation 6 This approach doesn't enforce ex- plicit collaboration among the students; rather, it serves as an additional source of knowledge. Some students, particularly those finding learning directly from the teacher challenging, may leverage this supplementary knowledge, contributing to improved performance.
Teacher
Student
MC-Distil
MC-Distil-PS
ResNet10-xxxs
36.29
36.44
ResNet10-xs
47.14
47.65
ResNet10-1
ResNet10-s
57.37
57.55
ResNet10-m
68.47
68.51
ResNett
36.57
36.78
10-Net
47.55
48.03
ResNet10
1esNett
57.59
58.20
RosNetl0-m
69.03
69.50
We present the analysis of the effect of adding loss compo- nent based on the pooled student to MC-DISTIL. Here MC-DISTI-PS represents the test result obtained with model trained with pooled student-based loss component.
Table 11:
4.11
Computational Cost Analysis of MC-Distil
Compared to baselines such TAKD (Mirzadeh al. 2020 DGKD [Son et al. 2021 and DML (Zhang et al.
2018 -which require multiple student models during training MC-DISTIL introduces an additional C-NET. For student pool consisting of ResNet10-xxx ResNet ResNet and ResNet10-m with ResNet34 as the leacher model and ResNet10-1 as the C-NET, the additional overhead is only 5% in terms of total FLOPs. Furthermore, as demonstrated in Section 4.8) the performance MC-Distil is not highly sensitive the size of the C-NET, allowing for further optimization of this component. Importantly, both the C-NET and the student cohort are utilized only during training. At inference time, MC-DISTIL requires ensemble multi-student framework and thus incurs additional cost over traditional KD.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Method
GFLOPs Batch
TFLOPs
Epoch
Time / Epoch (s)
TAKD
01.49
00.58
20.06
DGKD
04.77
01.86
24.63
RMC SHAKE
04.74
01.85
12.49
18.63
07.26
14.39
MetaDistil
10.29
04.01
09.10
MC-Distil
02.47
00.97
11.90
Table 12: Computational cost, reported as GFLOPs per batch and TFLOPs per epoch, along with training time (in seconds) per epoch for training the full student cohort.
Table 12 presents a summary of the computational cost in terms of GFLOPs per batch and TFLOPs per epoch, as well as the time required to train all student models for one epoch. These measurements provide a fair comparison of the total training overhead associated with each approach. Details of the profiling setup and assumptions are provided in Appendix B.6]
As shown in the table, methods such TAKD and DGKD require the sequential training of multiple models. This leads to inherent bottleneck in wall-clock time, since the overall duration is the sum of each intermediate stage. In contrast, MC-Distil trains all students simultaneously, avoiding this cumulative delay.
Although SHAKE and MetaDistil achieve per-epoch training times comparable to our method, they incur a larger total computational cost because they require a separate auxiliary model for each student. For instance, SHAKE uses an individual proxy teacher per student, while MetaDistil trains a separate meta-network for re-weighting the loss. When extended to train multiple students, the overhead of these approaches scales linearly with the number of students, making them significantly more expensive in aggregate compared to MC-Distil, which jointly optimizes the student cohort with shared C-NET.
While RMC is computationally lighter than SHAKE and MetaDistil, it is outperformed by MC-Distil in terms of final model accuracy. Table provides detailed comparison across CIFAR- 100 and Tiny ImageNet datasets, showing consistent gains in accuracy across all student sizes. This illustrates that MC-Distil achieves a more favorable radeoff between efficiency and effectiveness. Further details about the calculation methodology of the FLOPS are mentioned in Appendix B.6
5 Conclusion
We introduced MC-DISTIL, a novel knowledge distillation framework grounded in meta-collaboration, or Barmingg-o-oollabooaae Unlike conventional distillation techniques that treat each student model indepen- dently enforce rigid alignment with the teacher, MC-DISTIL facilitates joint learning among cohort of students via dedicated coordinator network, C-NET. This network dynamically modulates the flow of gradient signals during training, enabling each student to learn not only from the teacher but also from the training behavior and representatio ations its peers. As a result, each student model evolves in a collaborative yet personalized fashion, free from hard constraints to mimic either the teacher or other students.
This flexible coordination fosters the emergence of diverse, hirh-performing student models that are well-suitod to range of deployment needs with varying compute and memory budgets. MC-DIsTH. enables the training of full student pool in single pass, reducing overhead without sacrificing quality. The learned diversity also contributes to robustness, allowing practitioners to select from multiple viable models depending on specific task requirements or real-world constraints.
Our extensive ovaluations across CIFAR-100, Tiny ImageNet, and MS-COCO benchmarks demonstrate that MC-DISTIL consistently surpasses both classical and advanced distillation baselines, including KD, TAKD, DGKD, SHAKE, and RMC. Notably, MC-DISTH maintains its edge in the presence of data noise and under domain shifts, highlighting its robustness and generalization capabilities. By aligning students with both teacher knowledge and peer insights, MC-DISTIL paves the way for scalable, adaptable, and more resilient distillation strategies suitable for practical deployment.
16

---

Published in Transactions on Machine Learning Research (08/2025)
References
Görkem Algan and Iika Ulusoy. Meta soft label generation for noisy labels. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7142 7148. IEEE, 2021.
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. arXiv preprint
arXiv:2004.10344, 2020.
Defang Chen, Jian-Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. Proceedings of the AAAI Conference on Artificial Intelligence, 34(004))3430-3443, Apr. 2020. doi: 10.1609/aaai. v34.0.4..4. URL https://ojs. aaai orooorog/ooo ptrg/index.cm http///AAA/aattccl///////////// http://aaaaaa.aaaaaa//aa/aaa//aa
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/ http:/ooeeinnnnnonissso..ooo. cc/papr www.ffflleppppeeee 200/2/2hhcc822222eea100000006663
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEF/CVE International Conference on Computer Vision pp. 4794-4802, 2019.
Mengnan Du, Subhabrata Mukherjee, Yu Cheng, Milad Shokouhi, Xia Hu, and Ahmed Hassan Awadallah. Robustness challenges in model distillation and pruming for natural language understanding, In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1778, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2023.eacl-main. 129. URL httts:://cclathology) wwwpoog...rr////0 2023. eacl mainn. 129
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, mlr
pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL Batss//v8/gaa115 html.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11020-11029, 2020.
HrayT Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and Sanjiv Kumar. Supervision complexity and its role in knowledge distillation. In The Eleventh International Conference on Learning Representations, 2023. URL https:// http:///wwreiieeeeeeefrrrrrrrr wett:fforuuidd/ju//y///YW.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL https://openrevier
net/foruumii--BJ2tt..co
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
Fotis Iloooouls, Vasilis Kontonis, Cenk Baykal, Gaurav Menghani, Khoa Trinh, and Erik Vee. Weighted distillation with unlabeled examples. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openeevieww.ettt
forum (iimmidd-i MAVVVEUVA
Nishant Jain and Pradeop Shenoy. Selective classification using a robust meta-learning approach, 2022.
Anil Kag, Durmus Alp Emre Acar, Aditya Gangrade, and Venkatesh Saligrama. Scaffolding student instill knowledge. In The Eleventh International Conference on Learning Representations, 2023. URL
17

---

Published in Transactions on Machine Learning Research (08/2025)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint or Xix::112.6690/ 2014.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Ya Le and Xnan Yang. Tiny imagenet visual recognition challenge. CS 281N, 7(7):3, 2015.
Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge transfer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: wttp://eeoooooo/////roooropoeee
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europeon Conference on Computer Vision, pp. 740-755. Springer, 2014.
pyramid networks for object detection, 2017. URL pctps://arxxxv..og//b////// 03144 Tsung- Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocom- puting, 415:106 113, 2020.
llya Loshehilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint drXiv:1711.05101, 2017.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient. cmn architecture design. In Proceedings of the European conference computer vision (ECCV), pp. 2018.
Julien Mairal. Stochastic majorivatiion-miiiizaaaooo algorithms for large-scale optimization. In NewrIPS, 2013.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pp. 7632-7642. 7642. PMLR, 18 24 Jul 2021.
Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 5191-5198, 2020.
Aniruddh Raghu, Maithra Raghu, Simon Kornblith, David Duvenaud, and Geoffrey Hinton. Teaching with commentaries. In International Conference on Learning Representations, 2020.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pp. 4334-4343, 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time ob- ject detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://proceedee wwwplings.nurriis.comm hewprppp.c/ppppr...ee//ppp///p/p 72526b111777040b002222a331333333
18

---

Published in Transactions on Machine Learning Research (08/2025)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211 252, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiell Chen. Mobilenetv2: Inverted rosiduals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pollern recognilion, pp. 4510 45200, 2018.
Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste. Data parameters: new family of parameters for learning differentiable curriculum. NewrIPS, 2019.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. weig-weittt:-et::: Learning an explicit mapping for sample weighting, Advances in neural information processing systems, 32, 2019.
Durga Sivasubramanian, Ayush Maheshwari, AP Prathosh, Pradeep Shenoy, and Ganesh Ramakrishnan. Adaptive mixing of auxiliary losses in supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1855-9863) 2023.
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9395 9404, 2021.
Nidhi Vyas, Shreyas Saxena, and Thomas Voice. Learning soft labels via meta learning, 2020.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp. 196-202. Springer, 1992.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10302 10310, May 2021. doi: 10.1609 10.1609/aaai.v331 URL https://ojs. aaai wrt//rdex..../oooooooooooge.eeee index. php/ AAAI/ /ttpaaarrieeeeeee/////3//
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33:7597-7666 2020.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691 1099,,99, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXie preprint arXiv:1605.0m 000110005,,1,, 2016.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 1320-4328 2018.
Haoran Zhao, Xin Sun, Junyu Dong, Zihe Dong, and Qiong Li. Knowledge distillation via instanco-level sequence learning. Knowledge- Based Systems, 233, 2021.
Xiao Zhou, WEIZHONG ZHANG, Zonghao Chen, SHIZHE DIAO, and Tong Zhang. Efficient neural network training via forward and backward propagation sparsification. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https://openreviae http:////pppeeeeeeeeeeeee/////aa
19

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
A Convergence
In this section, we lay out the complete proof of theorems. Our proof technique is inspired partly by the prior literature (Shu et al. 2019).
Lemma 1 Suppose the mela loss function is Lipschilz smooth with constant L,, and C-NEr(-) is differential with 8- 6-bounded gradient and twice differential with its Hessian bounded by B, and the loss function L have p-bounded gradients with respect to training/metadato. metadata. Then the gradient of @ with respect to meta loss is Lipschitz continuous.
Proof From equation Equation
B
m H (Bi j=1
m km Σ Σ
CC-NET
((S)) yf)
H (ffo(
(8)
km
The gradient of o with respect to meta loss be written
k
V CCN
Vo,
H (fo, (x)).
Vol,
k
j=1
Vo, 11 (fe, (zf),
h
V H (v. (8,) .w. + 1/ +7² VocKL(s) (8) .v. (11
V
[1)
(9)
Let GH Vo, (fo, (x))
Vg. (5))
H(,
v.) and G Vo, Hfo, (27), 72 Vog KI (81)
Then again taking gradients on both sides w.r.t o.
* n Σ VoCli v + G Vagg v.Ck' * Vogg + tckt Vagh
VoLc.Ne
(10)
k n
For the first term on the right-hand side, we have that,
V VOG * v
V
VoVe, * Vo H (v:
(S,) v.)
(2i))
1=1
Let [i]
V 0f Hy: (8,) vi) + D + (Ly (() * V H (v(
(8) ((
(5)) .23)
(11)
Since
V3, H (fo,
s I. Ho (5)) w)) s kL الاا s p. Voggs
s 8 and
|
Vo H (fo vi) V. [b] * Voy (1.
- - k,g (8) (i7)))))))
n
1=1
0r
(85)
r Σ
L*
*0
s 1
(12)
t=1
20

---

Published in Transactions on Machine Learning Research (08/2025)
Similarly,
V VOOkL vookl
L
(13)
And for the second term on the right-hand side in Equation
10 have,
GG
v 2
V
- B 8²
(14)
Similarly,
|G Vig - B 6²
(15)
Let 7/1 maxx())."" ,n/)- Thus combining Equations 12 13 and 15
CC-NNell |l s 2 (L8 + 7 + +
(L8
(16)
Define Lv = 2* (Lo + B), based on Lagrange mean value theorem, we have:
(er VLC.Ner(d22) Lvllón d2l1, 00
(17)
Theorem Suppose the loss function é is Lipschitz smooth with constant L, and C-NET(-) differential with 5-bounded gradient and twice differentiol with its Hessian bounded by B, and the loss function C have p-bounded gradients conceruing
Lel the learning rales m/,,... ,nt sulisfies vii = mi, JVj E (1.....,k)) for some >> O, such that T < 1, and 12 St S N is monotone descent sequence, v/ = min D. - for some > 0, such that avT 2 L, and Ets1 v S (n= & co. Then MC-DISTH. can achieve TB[[] (O
Se in (11e²) sleps. More specifionlly,
BII CO S 0 VT USIST
min
(18)
where C is some constant independent of the convergence process, a is the variance of drawing uniformly mini- samples at random.
Proof
From Equation
5
12
15
V re L 0'+1) 8' - (0*
(19)
1=1
This be written as,
++1 = 0' - 1 (0
(20)
Since the mini-batch E, is drawn uniformly from the entire data set, we can rewrite the update equation as:
- V WoLLNNN (0 41 (6')) c(1)
(21)
where g(0 = (O
(0 Note that g(t) are i.i.d random variables with finite variance since E, are drawn i.idd with a finite number of samples. Furthermore, B[g(2) = 0, since samples are drawn uniformly at random. Observe that
LC-Ner (0+t2(() LoNEE (0th((() = (C
- (CNer()
21

---

Published in Transactions on Machine Learning Research (08/2025)
+ (C - LC.NEr (0++1
(22)
By Lipschitz smoothness of meta loss function, we have
(0x2 (Cx (g)+1)) S V (o 01t (
0 2 (0 -
I
(23)
According to the update rule, we have:
( @r* (0+tl) =
Σ
-all1] [1]
V Hy; (8))
(1))) v.) + 1 7² Vogii KL (8)) 18: ((
n2
7/1
n
gor [ Vo H H(! (1) v.) + [b) 72 VoL.KI (L
(Sa) -y6 )
n
This follows from Equation
1 Now, under the assumptions:
H (5) ui) s. |[ogn .1 (T)) p.
V (fo, (z (.,)) sp, go(:) 11, m = ma 1/
obtain the following bound:
( (0 s 72) + kL(T/(1 (1 + (1+ +2))² P²
|| (0)+ Ca.Ner
||C
2
( 1
n/ (1
(24)
2
By Lipschitz continuity of VLC.Ner according to Lemma we can obtain the following:
CcNu((""!( (0)) (
0
(25)
From Equation 21
Le. ( (o)t)))))) - LC
S (e (e + 5(00)
L(n))²
VoLCNm (0')) + g(0) |
--(n2² n/
L(n))²
10th
2
2
T(75)²
1 - (né --((((2)))
{(1))
(26)
2
Thus Equation 22 satifies,
(0 (6) 2 (1+ 73)
(1
I1
+7)
2
L(vb)²
1/3
L(ng)2
( 112 -
(e
(27)
2
22

---

Published in Transactions on Machine Learning Research (08/2025)
Rearranging the terms, obtain
( 1/2 L(nh)²)
V CEr. (ot))) (0 - CC. (()+)
2
1+ 1
+ km/p (1 72)
2
L(nb)² 2
+
15 (v/ - (VC (69),
(28)
2
Summing up the above inequalities and rearranging the terms, we obtain
L(n2)²
"0 & (C-Ner(O'(d (C
1/2
2
T
p (1 (1+ ++7) (1+
1+ 1
T L(nb)²
+
10
1|
2
2
1=1
1=1
Σ (n/ E(m - L(nb)²) (CC c+
S (0 + Σ kn/p2(1+ (1 (1+ 1(1+
L(m2)
||
2
2
T
Σ (T/ - (8
(29)
Taking expectations with respect to g(N) on both sides, we obtain:
2(nb)²
Σ
Ee(N)| V (0 - (O'
1/2
T
knf 2 + 72) (1 1+ 1/ +7
T
L(b))2
(30)
since Bean (0(++ = 0 and [|||<(0||]] S 02, where o² is the variance of g(0), Furthermore, we can deduce that
Σr
1/2
m(n))2 2
BE(N))
V (O
min E V (0
ΣT=
1(ng12
T
1
(0'")) +
ΣT 1=
(2n²) - L(n5)2)
Lo² 2
(16)2
1=1
1
(0
+ + x2)) + Lo Σ (n6))2
+
2im T/
7'
T
<
1
(0(0)) + | + Lo
> (nb))
Trp
23

---

Published in Transactions on Machine Learning Research (08/2025)
(O
(1 + L(1 +2)) + Lo
+
Tmé
mb
(0t)))
m. oVT)
T
ovT
a + m
7)
+ 12 + Lo mint c
L OoT
(C
L(1 (1++
Loe VT = 0
(31)
cVT
CVT
The third inequlity holds for Ex (2nh2 - L(nb)²) S Σ t=l m. Therefore, we can conclude that our algorithm can always achieve minosesT E CC-NET (9) >')1 I 0 in steps, and this finishes our proof of Theorem
Lemma 2 (Lemmaa A.5 in MMairal that the series > an diverges, the series E anbn converyes, and there exists K > such that 1b - bnl S Kan- Then the sequences (bn)agı converges to 0.
2013)) ) Let (an)as1, (ba)agı be taso non-negative real sequences such
Theorem Suppose the loss function C is Lipschitz smooth with constant L, and C-NET(-) is differential with a 5-bounded gradient and twice differential with Hessian bounded by B, and the loss function C have p-bounded gradients with respect to training/ mmela dala. Let the learning rales n/,,... ," sutisfies vy = min(1, VVj E (1...., ,k} for some > O, such that T < 1, and 1/2, S t N is monotone descent sequence, n/= = min
L' ( T for some <> 0, such that ovT 2 L, and Ex= vb co, Ei=(1)) S co. Then
lim VL,, (0), o²) -0. t00
(32)
Proof
From Equation 4 we have,
Cs, (0), x-) 1 Σ Gamll * (v))))
(8t) .b) + S 1 x
f1.....
(33)
It is easy to conclude that mil, satisfy Ei गpं = =0, ا (1r < co. From Equation
8
Gil H (v: (8,) v.) + g0,b) KL vi)) ej ,)
(34)
Since the mini-batch V, is drawn uniformly at random, we can rewrite the update equation as:
g 1-+1= = 0f vi
G V Hv. (8s) vi)
+ g [j] * KL (5:) vi ( + V.
(j.........)
0 m 1j V Ls, (0) 0²) w.
(35)
where (0() = V Ls, (0), 0')l,, - Ls, (0). d¹). Note that (1 is i..d. random variable with finite variance, since V, are drawn i.id. with a finite number of samples. Furthermore, E[b(*) O, since samples are drawn uniformly at random, and B( _ 2
24

---

Published in Transactions on Machine Learning Research (08/2025)
The objective function Cs, (ff, Ø²) can be easily checked to be Lipschitz-smooth with constant L, and have p- b-uundddd gradients with respect to training data. Observe that,
Ls, 1,001100-9 Ls,, (0) = { (0) ,0th) - Lss, (0, ,003 } + (0ftt - L.s, L.s, (03:03) (36)
For the first term,
Ls, (0ft), , - -Ls, (0ftt1.05 00) 1 Σ
- - H (v, (8)) v.)
n
+ (0 +++1 (or115) x+² * KL( (L (5:) )
V VoDet (b) t1 - -0 5(15) 2
1
100+1 1--1--0'1 02'13 H1 (1( (5:) v))
2
+ (
g (5]) * 22² kT. y3
(5:) (T)
VV (oog [i]. - V (0') + 1
)( 2
10 H Hv! (5)) v.) + - -2((111)+ KL(b, (v)))
2
1
(0')) 100) + 5(1²)²
(IVaLeno(0
15 1
2
+ 2 (o (1)) (00
H(! (5,) v.) -6 [,] - (g, x KL() (3) m.
(o')) + 10 + 5(15)²
(Iaaesr (0')) 11 + 1
* H H(. 15,) v.) Σ ViV (0')) + )
+ (
6(1))² (IT
(')1 11 + +2( ( }} ) KL(( (8)) (1 (37)
+
2
For the second term from Equation 36
(s, .0) Lss (0) S Vo, (0, 0'), 00. -8)) I(ri)2 110 0,11
(Vo, Ls, (0), 8¹), 7/' 0²) + v.l) L(mi)²
1 Ls, (0), 0") + v.l1
nn --(1i L(n)) )2
| IVLs, 11 + L(md) )2
||e:(0)
2
2
-0t - =(br)) Ls, (0), 0'), (00))
(38)
Combining Equations 37 and 38 we can rewrite Equation 36 as,
Cs, (0,1 10041.0""
,0 - (0,
(V Vege. (b) - - V ( + 500
6(12)² +
|Vo (0')) + 1 + 2(V ( (5) v:)
2
+ (Vegg.1))
25

---

Published in Transactions on Machine Learning Research (08/2025)
6(1))² +
(100CcNr(0011 + |1 + 2(V (6 +7² (l( w.th))
2
- -(1t nf 2
L(m))*)
IVLLs, (ff. の0 1 1 [ - (vf' - (Ls) 100)
(39)
Taking expectation on both sides and since E|g(0)] = 0, Elpb(0) = 0, we have
[
B|Cs, (0ft))/(++)) - B|Cs, (09, Φ¹)l s E
(V Fa [il. - ((
0(12)²
1|
H4, (8) .11)
2
n
(V oo [j], -2 V
V CET @ (0')))
n
8(12)²
V ( (0)))1 + +1|5+ 1| + + * KL (8)) .y: (/ }
2
L(Ti) V (0), 8' |[]] E (( 12)
(0,, (0),
12) +
(40)
2
Summing up the above inequalities over t 1,.... 0o in both sides, we obtain,
bIll (0*h
n
KT. Vi
1=1
I(nD)²
E|| Css (0 2'>1 + 111
[E[Cs,(0), 6'1
lim E[Cs, (0), d')] T-x
6(m))
1
E V (0rt (o))) E
|1 (1 (5)) :x)ll + 7² |KL (86) (,'"",,((")) m)))
2
11
i=1
30
L(nR)²
2
1p²
E[Cs, (0) 0
+ +
8
8(1))²
2 M +00) + soc
(41)
2
2
The last inequality holds since Ei ( < 2i=0 (n) x0, 12 and
<o,
w.)l s M
and 1 E [ (8,) , (T)))|| < M for limited number of samples' loss is bounded. Thus, we have
Σ T EE V Ls, (0) .ل
(5,) .9) III bill (o (d)))
+ Σ 2
E
KL (vi (5,) ,0 IIV bf (0+ 00
(T)
(42)
Since
E
[ (8,)
V LC (0)+1(61)1
5 Mpp Σ
1/ S 00
(43)
1=1
1=1
and
E
1 Σ
(v. (8))
P (0++ (6)))
< ~0
Σ 2 < x
(44)
n
1=1
26

---

Published in Transactions on Machine Learning Research (08/2025)
This implies that Σ B (0j, 0') co.
By Lemma 2 to substantiate lim [ (fj, 3')1 = since Ei=o n/ = =c, therefore we only need to show,
B (0) , E Vss (0) o') S CrHl
(45)
for some constant C. Based on the inequality:
[[Iaa] + [[aa) - |[b|)| N ||e + b11l1a b11
(46)
we then have:
E|ll EII (0)+1 .0+1)11 Ls, 01) = (BIIV (0) ,1*1
- EII V ( (11l (0) |a2l| + VL), (0), ||2))
VCs ( |VCs, (0), d')1l2 VLs, (0 0 (0-1))11 1)2 + LVs,, (0, 26)l
< E
VV ,0th1 VLs, (0), )1) VL2 (0) / (8) x 122
< E
(IVLs (0 +1 (011)112 - V (0) (0)1)) (0,)) | +
|| VCs, (09, 62)112)
- 21
1 (0); [12]
s 2 E
[[Vss, (8), )¹) + u. V..Va (6C)
< 22
VILVCS, (8 0t) 1 + VO + {(
< 2
BlI(VCs, 0-) + B 1 +
- 2
BlI( (0), + + E EIll V 0 (0)) + ||
< 22pppnn 2222+22pp
(47)
According the above incquality, we can conclude that our algorithm can achieve
lim E[I (0), 0²) 1 =0 100
(48)
The proof is completed.
B
Training Details
B.1 Dataset Details
We conduct experiments using the following real-world datasets to showcase the effectiveness of our approach,
CIFAR-100 [Krizhevsky 2009). The dataset consists of a total of 60K examples, distributed across 100 distinct classes. Each example in this dataset comprises images with a resolution of 32 x 32 Specifically, the training set encompasses 50,000 examples, while the remaining 10K serve as the testing set. In our experimental setup, approximately 5K examples are allocated for use as a validation for our C-NET. For the other baseline models, these validation examples are utilized for hyper-parameter tuning.
27

---

Published in Transactions on Machine Learning Research (08/2025)
ImageNet-1K Ruussakovsyy et al., 2015 The dataset comprises over 1.2 million training images and 50,000 validation images, spanning 1,000 object categories. Each image is high-resolution and varies in size, but is typically resized to 224 x 224 x 3 during preprocessing. In our experiments, we follow the standard ImageNet- split. A subset of 50K images from the training set is held out as a validation set for tuning the parameters of the coordinator network (C-NET). For the remaining baselines, this same subset used exclusively for hyperparameter tuning.
Tiny ImageNet Le & Yang 2015). The dataset is derived from the extensive ImageNet-IK dataset Russakovsky et al., 2015). This dataset encompasses a total of 100K images, which have been downsampled to a 64 x 64 resolution, representing a subset of 200 classes mirroring those in the ImageNet- 1K dataset, with each class containing precisely 500 images. As part of our experimental protocol, we have set aside an independent validation set comprising 10K examples, which is utilized for both our proposed method and the baseline models.
Clothing-IM (Xiao et al. 2015). The dataset consists of 10,00,000 noisy label training examples spread acToss 14 class labels of clothing items taken from various online shopping websites. Each example consists of image with resolution 224 x 224 3. The validation and test set consists of 14,313 and 10,526 examples respectively. The dataset also contains 47,560 clean examples, but we do not use that portion for our experiments. It is a datasel with noisy labels, since the data is collected from several online shopping websites and include many mis-labelled samples.
Instance CIFAR-100 (ia et al. 2020) The dataset contains manually corrupted version (with an error rate 7) of the CIFAR-100 (Krizhevsky) 2009) dataset. This is done by first constructing transition matrix T(x), where (x) is the probability that the corrupted label is j, given the true label is i; while keeping the overall error rate as т. Then, the labels are flipped according to the defined transition probabilties. In our experiments, the training uses 50K examples with noisy labels, corrupted by the above method, while the remaining 10K examples, which serve as testing set have no label noise. Approximately 5K noisy examples are allocated for use as a validation set for our C-NET. The overall label flip rate т, used in our experiments was 0.2.
iWildCam-2020 (Beery et al. 2020) The dataset is set of collection of images collected using the heat trap and motion activated cameras for the better understanding of wildlife and biodiversity. There is so wide diversity in lighting, camera angle, backdrop, vegetation, color, and relative animal frequencies between different camera traps, and also the number of animals of particular species varies a lot, hence the data about different spices also vary similarly. In experimentation we trained the models using approx 100k training, 12k validation and 12k test images across 216 classes.
Tiny ImageNet-C (Hendrycks & Dietterich, 2019). The dataset is a downsampled version of the ImageNet- dataset, which consists of 15 diverse corruption types applied to the ImageNet-1K (Russakovsky et al. 2015 validation dataset. These corruptions have been drawn from 4 main categories noise, blur, weather, and digital. Fach corruption has 5 severity levels, 1 being the least severe and 5 being the most severe. In our experiments, the models trained on Tiny ImageNet (Le & Yang 2015) dataset, with train-validation split of 90K and 10K examples respectively, are lested on Tiny ImageNet-CC for testing robustness gains.
MS COCO (Lin et al. 2014 The dataset contains over 330K images, with more than 200K labeled and 80K annotated with object segmentation masks, spanning 80 object categories. Each image typically includes multiple objects in complex scenes with rich contextual information. In our experimental setup, we use the standard training-validation split, selecting a subset of 5K images from the training set as a validation set for tuning C-NET, For all other baselines, this validation set is used solely for hyperparameter optimization.
For all the image classification datasets, we use the data augmentation methods, using the torchvision's 's transforms module. We use RandomCrop, RandomResizzddCopp RandomSizedCrop, RandomHorizontall Normalize, and ColorJitter for our purpose. Augmentation mothods were applied on both datascts, over all experiments, baselines, overall models.
28

---

Published in Transactions on Machine Learning Research (08/2025)
Basic block
CIFAR- 100
Tiny ImageNet
Architecture
Filters
Repeats
MACs
Params
MACs
Params
ResNet10-xxs
[8, 8, 16, 16]
[1, 1, 1,
2 M
13 K
M
ResNet10-xs
88 16, 16, 32]
[1, 1, 1,
3 M
28 K
M
31 K
ResNet
88 16, 32, 64]
[1, 1,1,,11
4 M
84 K
90
ResNeet10-m
[16, 32, 64, 128]
[1, 1,,11
16 M
320
M
333 K
ResNet10-1
[32, 64, 128, 256]
[1, 1,
64 M
1.25 M
255 M
1.28 M
ResNet10
[64, 128, 256, 512]
[1, 1, 1]
253 M
4.92 M
1013 M
5 M
ResNet18
[64, 128, 256, 512]
[2, 2, 2]
555 M
11.22 M
2221 M
11.27 M
ResNet34
[64, 128, 256, 512]
33 6, 3]
1159 M
21.32 M
4637 M
21.38 M
Table 13: Comparision of newly introduced smaller ResNet with the standard ResNet models
B.2 Model Details
Image Classification Models In Section 4.1 introduce several smaller ResNet models, specifically ResNet10-xxoxs, ResNet10-xoxs. ResNet10-xs, ResNet10-s, ResNet10-m, and ResNet10-1. These models are adopted from Kag et al. (2023), and in Table 13 we compare their architectural details with well-cstablished ResNet models like ResNet10, ResNet18, and ResNet34. Similar to the standard ResNet models, these newer, more compact versions also employ the traditional 'BasicBlock' as their fundamental building block. The architectural structure consists of convolutional block, four stages of residual blocks, an adaptive average pooling layer, a convolutional block, and classifier layer. The sole variation among the various capacity variants within this experimental setup is the number of filters in each stage and the residual block. Additionally, Table 13 provides information about the number of parameters and multiply-addition (MAC) operations for each model for the datasets CIFAR-100 and Tiny-ImageNet.
Object. Detection Models For the object detection experiments in Section 4.2 employ Faster R- CNN Ren et al.) 2015) as the base detector, with a high-capacity ResNet50-FPN as the teacher, and Res-Nell8FPPN and Res VeL34-FPN as the student models. These are equipped with Feature Pyramid Networks (FPNs) (Lin et al. 2017) that enhance multi-scale feature learning. The C-Net used in this setting is a ResNet34 model, modified to output weighting parameters in place of classification logits. All models are initialized from prreeenet-retrrainee weights and are fine-tuned on the MS COCO dataset following standard detection pipelines. Unlike the plain ResNet architectures used for image classification, Faster R-CNN incorporates Region Proposal Network (RPN) and detection head top of ResNet-FPN backbone. The ResNet-FPN backbonc outputs multi-scale feature maps, enabling detection of objects at various sizes. These features are processed by the RPN to generate region proposals, which are then classified and refined by the ROI head.
B.3 Sopper-arrmmters
For all the KD baselines listed in Section 4.1.2 and MC-DISTIL, we use temperature == employ the SGD optimizer train the student models and ADAM optimizer (Kingma & Ba 2014) to train C-NET. We use batch size of 400 for all datasets. We train the student models for 300 epochs CIFAR- 100, Instance CIFAR-100, Clothing- 1M, Tiny ImageNet and Tiny ImageNet-C datasets; and for 100 epochs iWildCam datasot, ImageNet and MS COCO while updating C-NET every 20 epoch (i.e., L = 20). We use cosine annealing (Loshchilov & Hutter 2017) as the learning rate schedule for training the student models. We warm start each student model by first training it using the cross-entropy loss without using the teacher model for all KD basclines and MC-DISTIL. For all datascts, we perform a grid search on 0.1, 0.05 for the learning rate, on le 5, 5, le for the weight decay, and 0.65, 0.75, 0.85, 0.95 for the momentum. For the C-NET training
learning rate of le and set weight decay to le 4.
29

---

Published in Transactions on Machine Learning Research (08/2025)
B.4 Baselines
We compare MC-DISTIL with the following SOTA Knowledge distillation baselines:
Teacher Assistant Knowledge Distillation (TAKD) (Mizzadeh et al., 2020) This approach introduces multi-step distillation process that leverages intermediate-levell teachers to facilitate the efficient transfer of knowledge from a large pre-trained teacher network to more compact student model. To realize this, we employ a teacher network identical to our own and enlist fellow student models with higher learning capacities to serve as intermediale models in this knowledge distillation process.
Densely Guided Knowledge Distillation (DGKD) (Son et al. 2021 Much like TAKD, this approach employs several intermediate models; however, it distinguishes itself by training the final student model through single distillation step. In addition to the teacher KL divergence loss, the training objective for the final student model incorporates the KL divergence loss obtained from the pre-trained intermediate models.
Robust Model Compression (RMC) (Du el al. 2023). It uses multiple students with various levels of sparsity and interprets the variance in their predictions for each instance as a measure of task complexity. Subsequently, it refines the teacher predictions based on this complexity metric, resulting in a more robust knowledge distillation process. In our experiments, we employ students with diverse learning capacities as a substitute for models with different levels of sparsity, achieving similar benefits.
Deep Mutual Information (DML) (Zhang et al. 2018 It is collaborative lcarning paradigm that leverages the interactions among multiple neural networks to enhance their collective performance. In DML, a group of neural networks, often referred to as student models, collaboratively learns by exchanging information with each other during training. This is achieved by using the predictions or feature representations of one model to guide the learning process of others. The collaborative nature of DML allows the models to exploit the diverse knowledge learned by their peers, fostering a mutual learning dynamic.
SHAdow KnowlEdge transfer framework (SHAKE) (Li & Jin 2022) It is novel knowledge distillation framework designed to enhance the transfer of knowledge from teacher model to student model in deep learning. Introduced to address the challenges associated with large capacity gaps between teacher and student models, SHAKE employs a pscudo-teacher concept. This pscudo-teacher consists of a set of additional learnable layers attached to the backbone of the teacher model. What sets SHAKE apart is its ability to dynamically adapt to the student model during training.
In experiments where we train with noisy datascts, we compare against additional two bilevel optimisation- based reweighting schemes:
Learning to Reweight (L2R) (Ren et al. 2018 It is machine learning paradigm that focuses training models to dynamically assign weights to different instances in datasct during the lcarning process. The fundamental idea behind L2R is to adaptively adjust the significance of individual data points, emphasizing more informative instances and de-emphasizing noisy or less relevant ones. This approach is particularly valuable in scenarios where the training data may be imbalanced or subject to noise, helping models prioritize the most crucial samples for better generalization.
Meta- Beita-Weigtt-Ntt (MWN) (Shu et al., 2019 Like L2R, MWN also learns weights assigned to individual samples based on their specific characteristics or importance. However, this explicit mapping of sample weights is learned through meta-learning framework. Meta- Meta-Weight Weight-Net aims to address challenges related to imbalanced or noisy datasets by explicitly learning a mapping function that adapts sample weights during the training process. By doing so, the model can assign higher importance to informative or challenging samples and reduce the impact of less relevant ones.
B.5 Compute Resources
We our experiments on a mixture of GPUS viz. A100s and RTX 2080 as our experiments don't need any sophisticated modern GPUs. MC-DISTIL memory requirements are slightly higher that traditional KD as we train student cohorts simultancously. The additional memory requirements depend the number of students used and size of cohorts
30

---

Published in Transactions on Machine Learning Research (08/2025)
KD
1.53e-05
TAKD
1.53e-05
0,0005
DGKD
L.53e-05
1.53e-05
0.0006
RMC
4.78c-05
0.99
0.9992
DML
1.53e-05
0.978
0.1156
SHAKE
4.78e-05
0.115
0.64714
0.9855
0.0222
0.0416
META DISTI.
1.53e-05
1.53e-05
0.0005
0.5896
4.58e-05
1.53e-05
0.001
MC-DISTH.
1.53e-05
1.53e-05
-53e-05
7.63e-05
1.53e-05
1.33e-05
0.00015
1.53e-05
CE
KD
TAKD
DGKD
RMC
DML
SHAKE
METADISTIL
Table 14: Pairwise significance p- p-values using Wilcoxon signed rank test
B.6 Timing Calculation Methodology
The values presented in Table 12 are basod on profiling a full epoch of training and are intended offer a lair estimate of the computational budget required to train an entire cohort of student models. The following principles were applied:
A standard hcuristic was used to estimate computational cost: a model's forward pass contributes 1x its theoretical FLOPs, while a full training step comprising both forward and backward passes is estimated to consume 3x the forward FLOPs, following Zhou et al. (2021).
The profiling was performed on the full student cohort consisting of ResNet10-xxs, ResNet10-xs, ResNet10-s, and ResNet10-m, with ResNet34 used the teacher. All reported FLOOPs are normalized a per-epoch basis. Auxiliary components MetaDissil uses a ResNet32 meta-network (similar to AMAL); and MC-Distil uses KesNee33-bbsse C-NET (C-Net). RMC relies on the cohort itself for difficulty estimation.
configured follows: SHAKE uses ResNet18 proxy teacher per student;
For sequential distillation methods such as TAKD (Teacher + TA, + TA2 ...) and DGKD (Teacher + TAs + TA), total per-epoch time and TFLOPs are calculated as the cunulative sum across each training stage. This approach captures the actual total training effort.
For methods that train each student independently-sneed TFLOPs are computed as the sum of each student-specific run. However, the reported time per epoch reflects the slowest student model (R)sNet10-m), representing an optimistic scenario under ideal hardware parallelism.
SHAKE, MetaDistil, and RMC-the total
In contrast, MC-Distil trains all students simultaneously single run, reusing shared components such as the C-NET, thereby significantly reducing both wall-clock time and aggregate computational cost.
C Additional Experiments
We have released the code at the following URL: htttgr///gtthuoooo/////rrrraaaat
C.1 Standard deviation and statistical significance results:
In Table 14, we show the p-valucs of onc-tailed Wilcoxon signed-rank test Wilcoxon| (9922) performed on every single possible pair of knowledge distillation strategies determine whether there is significant statistical difference between the strategies in each pair, across all datasets. Our mull hypothesis is that there is no difference between the knowledge distillation strategies pair. From the results, it is evident that MC-DISTI. significantly outperforms other baselines at < 0.05.
Table 15 shows the standard deviation results over three training runs on CIFAR-100 and Tiny ImageNet datasets. The results show that the MC-DISTIL has the least standard deviation compared two of the most competitive baselines viz. DGKD and MetaDistil. We note that DGKD has large standard deviations owing to not having principled way of introducing student cohort models.
31

---

Published in Transactions on Machine Learning Research (08/2025)
CIFAR100 Test Accuracies
Teacher
Student
CE
KD
DGKD
MetaDistil
MC-Distil
ResNet10-xxs
32.41 + 0.517
35.23 + 1.132
34.37 + 0.453
35.47 + 0.737
36.23 = 0.247
ResNet1O-xs
42.42 = 0.724
45.62 0.722
44.89 1.775
46.86 0.836
47.74 0.244
ResNet10
75.18
ResNet10-s
52.70 + 0.384
55.64 + 0.106
56.25 1 0.545
57.25 1: 0.535
58.42 + 0.275
ResNet10-m
64.19 + 0.142
67.05 0.191
67.27 1 0.577
68.75 0.185
69.45 - 0.427
ResNet10-xxx
32.41 + 0.517
35.19 1.415
34.14 + 1.259
35.26 1: 0.745
36.03 + 0.442
ResNet10-xs
42.42 + 0.724
45.49 0.412
46.18 + 1.055
46.23 + 0.420
47.24 + 0.459
ResNet18
76.99
ResNet100s
52.70 + 0.384
55.49 0.315
56.07 + 0.578
57.26
0.131
57.63 + 0.176
ResNet10-m
64.19 + 0.142
66.42 + 0.095
66.73 + 0.548
68.02 + 0.490
68.52 = 0.237
Tiny-Image Sony-ImageNer
Net Test Accuracies
ResNet10-xxs
13.68 = 0.460
14.40 + 0.526
14.20 + 0.475
14.47 + 0.597
15.33 = 0.521
ResNet10-xs
18.78 + 0.231
20.20 0.624
20.90 1 0.351
20.85 1 0.692
22.07 + 0.285
ResNet10
44.04
ResNet10-s
24.86 + 0.362
27.83 0.779
27.59 1 0.605
28.07 1 0.890
29.62 1 0.263
ResNet10-m
33.93 + 0.493
35.86 0.332
35.82 + 0.398
37.64 + 1.452
38.46 + 0.068
ResNet10-xxs
13.68 = 0.460
14.52 0.368
14.42 + 0.481
14.68 0.595
15.26 0.105
ResNetl0-xs
18.78 + 0.231
20.28 1 0.500
20.58 1 0.931
20.95 1: 1.038
21.36 1 0.360
ResNet18
47.94
ResNet10-s
21.96 + 0.362
27.26 0.874
27.82 1 0.378
28.45 0.983
30.36 1 0.006
ResNet10-m
33.93 = 0.493
35.9 = 0.724
35.96 + 0.612
37.82 + 1.562
39.11 = 0.295
Table 15: Standard deviation results for CIFAR-100 and Tiny ImageNet datasets for 3 runs. We compare MC-DIsTH. with two most competitive baselines.
C.2
Robustness of MC-Distil trained models
For the models trained with Resnet18 serving as the teacher model on the Tiny ImageNet dataset in Table we present their performance on Tiny ImageNel-C (Hendrycks & Dielterich 2019 (see Appendix B.1 in Table 16 incorporating all baseline models and diverse corruption levels. We assess the resilience of models whose perfromance was reported in Table | when confronted with data distribution mismatches. Notably, across various corruption levels and teacher-student combinations, MC-DISTIL demonstrates robust performance, even though it not explicitly tailored for handling distributional shifts between training and test datasets. Specifically, at lower corruption levels, models trained with MC-DISTIL exhibit a performance advantage of 1-2% compared to other baselines. Even at higher corruption levels, MC-DISTIL mainlains competitiveness with the baselines.
D Limitations & Social Impact
We have presented a novel meta-colee collaboratvee distillation- wwhree students of different capacities are codistilled from single teacher. While the initial results wide range of student & teacher architectures and model sizes are encouraging, more work is needed to understand under what conditions such adaptive mixing optimizations for metacollaborative distillation can provide gains. Our experiments are proof-of-concept on widely used benchmarking datasets; future work will explore whether similar gains can be realized in real-world applications as well. Our proposal involves bi-level optimization objective (for which we provide cheap approximate meta-learning preocedure)- the tradeoff of increased training time vs increased accuracy may depend on the specific application context.
Our work is a fairly general-purpose optimization procedure for improving knowledge transfer from a teacher model to a very small student model and therefore tremendously improving the inference time. This could yield incremental improvements a broad range of ML applications, and result benefit those applications. We do not anticipate any inherent risk of negative social impact from our work, over and above the risk inherent in the MT. application itself.
32

---

Published in Transactions on Machine Learning Research (08/202))
Tiny ImageNet-C Test Accuracies
Student
CE
KD
MCD
DGKD
DML
SKAKE
MetaDistil
MC-Distil
Corruption level Accuracies
RN10-xx8
11.25
11.38
10.61
10.77
09.99
11.47
11.02
12.05
RN10-xs
14.68
15.82
14.67
15.43
15.51
16.56
16.02
16.73
RNTO-s
20.88
21.97
21.94
23.18
20.29
22.58
23.97
24.88
RN10-m
26.91
28.56
27.08
29.56
28.09
30.40
30.43
31.00
Corruption level Accuracies
RN10-xxs
10.65
10.79
09.54
10.42
09.44
10.88
10.52
11.30
RN10-xs
13.75
14.60
13.72
14.60
14.56
15.42
14.78
15.52
RN10-s
19.28
20.64
20.46
21.26
18.74
20.74
21.93
23.04
RN10-m
24.76
25.87
24.36
26.97
26.20
27.94
27.39
28.67
Corruption level Accuracies
RN10-XXS
08.66
08.78
07.54
08.93
08.46
09.29
08.58
09.42
RN10-xs
10.94
11.54
11.08
11.82
11.96
12.19
11.78
12.58
RN10-s
15.68
16.85
16.71
16.95
14.76
16.95
17.65
18.79
RN10-m
19.70
21.07
18.79
22.20
21.45
22.23
20.83
22.63
Corruption level Accuracies
RN10-XXs
04.92
04.66
04.38
05.52
05.05
05.57
04.60
05.87
RN10-xs
06.77
06.96
05.88
07.54
06.89
06.57
07.35
07.39
RN10-s
09.41
09.92
10.21
09.42
08.35
10.29
09.77
10.80
RN10-m
11.12
12.28
10.74
12.74
12.87
12.44
11.54
12.03
Corruption level Accuracies
RN10-xxs
03.58
03.15
03.44
03.98
03.66
04.29
03.33
04.40
RN 10-xs
05.14
05.18
04.34
06.23
05.15
05.05
05.47
05.53
RN10-s
06.94
07.24
07.33
07.07
06.19
07.93
07.15
07.38
RN10-m
08.16
09.09
07.86
09.73
09.46
09.48
08.44
08.80
Table 16: Comprehensive table presenting assessment of model (trained with Tiny ImageNNt in Table robustness by conducting inference on the Tiny ImageNet-C dataset. (Hendrycks & Dietterich 2019 across increasing degrees of image corruption (from level 1 to 5), alongside results for all baseline models. The columns indicate the method employed for training the model on the Tiny ImageNet dataset, utilizing ResNet 18 as the teacher model. The highest accuracies are highlighted in bold.
33


Reviews:
[{"review_id": "xGpdo0tskt", "reviewer": "Reviewer_x9rt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Innovative use of C-NET for meta-collaboration, effectively addressing capacity gaps in KD.\n2. Comprehensive experiments across varied architectures, datasets, and scenarios, with statistically significant gains.\n3. Demonstrates robustness in extreme teacher-student gaps and noisy data settings.\n\nWeaknesses:\n1. Limited discussion on computational trade-offs of bi-level optimization and C-NET training.\n2. Proof-of-concept focused on benchmark datasets; real-world applicability remains untested.\n3. Scalability of MC-Distil with very large student cohorts or datasets is underexplored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces \"Meta-Collaborative Distillation\" (MC-Distil), a novel approach to knowledge distillation (KD) where multiple student models of varying capacities collaboratively learn from a single teacher model. A coordinator network (C-NET) facilitates this by adaptively weighting training losses based on instance hardness and student capacity, leading to improved accuracy across diverse datasets (e.g., CIFAR100, Tiny-ImageNet, ImageNet). Key contributions include:\n\n1. Enhanced student model performance, with average accuracy gains of 2.5% on CIFAR100 and 2% on Tiny-ImageNet.\n2. Novel insight that larger students benefit from collaboration with smaller ones, a departure from traditional multi-teacher approaches.\n3. Robustness to real-world challenges like label noise and domain adaptation, outperforming state-of-the-art baselines (e.g., TAKD, DGKD, SHAKE).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Quantify computational costs (e.g., training time, memory usage) of MC-Distil compared to baselines to clarify trade-offs.\n2.  Expand evaluation to real-world datasets or applications to validate practical utility.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2gzzVr7oYJ", "reviewer": "Reviewer_b4Eo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method is well-motivated and innovative particularly the coordinator network concept.\n2. The paper is well written and easy to follow.\n\nWeaknesses:\n\nI have a two major concerns:\n\n1. There is no clear ablation on the impact of C-Net. For example, what if one was to use independent learnable loss weights for each student individually, rather than generating them through the proposed C-Net? Or use weights proportional to the capacity of the student? There needs to be clarity on the impact of the C-Net module. As an additional example, the authors mention that a naive solution of using a n × k matrix would be prohibitive as training samples increase. Would it be possible to try out this method on CIFAR100 only, where the training set is quite small? Just as a comparison of how good such a strategy can be regardless of the tradeoff.\n\n2. The experimental analysis is a bit limited. Results are limited to only classification, mainly with TinyImageNet and CIFAR100. Similarly, the networks are mainly limited to various ResNet variants. It would be interesting to see the performance beyond classification (such as object detection on MSCOCO in various other works [1,2,3,4]) and different networks including the cases where teacher and student are different architectures [2]. Without detailed experimentation, the effectiveness of the method is in question.\n\n[1] Chen, P., Liu, S., Zhao, H., Jia, J.: Distilling knowledge via knowledge review. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5008–5017 (2021)\n\n[2] Jin, Y., Wang, J., Lin, D.: Multi-level logit distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24276–24285 (2023)\n\n[3] Lin, S., Xie, H., Wang, B., Yu, K., Chang, X., Liang, X., Wang, G.: Knowledge distillation via the target-aware transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10915–10924 (2022)\n\n[4] Miles, R., Mikolajczyk, K.: Understanding the role of the projector in knowledge distillation. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence (AAAI-24)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a collaborative knowledge distillation approach where multiple student models of varying capacities learn simultaneously from a single teacher, coordinated by an auxiliary \"coordinator network\" (C-Net). The key innovation is that C-Net dynamically reweights loss components for each student on a per-instance basis, allowing weaker students to focus on easier examples while stronger students tackle harder cases. Notably, the framework demonstrates that smaller students can actually help improve larger students through this collaborative process.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please consider the weaknesses. I would recommend that the authors consult the cited works [1,2,3,4] and consider reporting on the various benchmarks and evaluation settings used in those works. I would additionally request detailed ablations on the C-Net and potential variants to understand its effectiveness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zZyZq9wPyI", "reviewer": "Reviewer_qANo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* Novel Approach: The concept of meta-collaborative distillation using a C-NET to modulate learning for a cohort of diverse students is innovative. It moves beyond traditional single-student distillation or multi-teacher setups.\n* Addressing Capacity Gaps: The method specifically targets and shows strong performance in scenarios with large teacher-student capacity gaps, a known limitation in KD.\n* Improved Performance: MC-DISTIL demonstrates consistent and significant average accuracy gains over state-of-the-art baselines on benchmark datasets (e.g., 2.5% on CIFAR100, 2% on TinyImageNet). All students in the cohort, regardless of size, show near consistent  improvement.\n* Implicit Instance Hardness Learning: The C-NET implicitly learns to differentiate instance hardness and tailors learning for students accordingly (e.g., weaker students focus on easier examples).\n\n**Weaknesses**\n- Computational Cost: Training multiple students simultaneously along with an additional C-NET introduces increased computational overhead in terms of training time and memory compared to standard KD. While the paper mentions an \"efficient alternating update algorithm\" and updating C-NET less frequently, a more explicit comparison of computational costs against baselines is missing.\n\n- ImageNet Results Interpretation: While the ImageNet results in Table 3 demonstrate MC-DISTIL's effectiveness in extreme teacher-student gap settings (e.g., ResNet101 to ResNet14/20/32/56), the choice of student models makes the improvements less directly comparable to more traditional distillation scenarios with full models such as ResNet101 teacher and ResNet50, ResNet34 students. The reported student accuracies are also relatively low for ImageNet, though this is acknowledged as a consequence of the large gap and even CE performing poorly.\n\n**Typos and Minor Corrections**\n1. Location: Page 4, Section 3.1 (\"Standard Knowledge Distillation\"), Sentence 2 \n\n    Original Text: \"Typically, the labels y are cardinal, and practitioners have found that such labels are often inadequate in capturing nuances in the data.\" \n\n    Suggested Correction: The term \"cardinal\" here might be less precise. For classification, labels are typically \"categorical.\" \"Ordinal\" would apply if there's an inherent order to the categories. \"Cardinal\" refers to the count of elements in a set (e.g., the number of unique classes). A more accurate phrasing might be: \"Typically, in classification, labels y are categorical...\"\n\n2. Location: Page 5, Section 3.2.1, First paragraph \n    Original Text: \"...as the C-NETiS a centralized mechanism...\" \n\n    Typo: C-NETiS\n\n     Suggested Fix: C-NET is\n3. Location: Page 5, Section 3.2.1, First paragraph \n\n    Original Text: \"...represent the loss mixing parameters for the k students on the input xi​.Therefore, the ...\" \n\n    Typo: input $x_i$.Therefore\n\n    Suggested Fix: input $x_i$. Therefore\n\n4. Location: Page 9, Section 4.5, Second paragraph \n\n    Original Text: \"...choosing a favorable ground truth is also important along with focusing on the \"learnable\" points.\" \n\n    Formatting Suggestion: \"learnable\" -> ``learnable'' (i.e., using LaTeX-style double backticks and double apostrophes for opening and closing quotes).\n\n5. Location: Page 10, Section 4.7 (\"KD in Instance Dependent Label-Noise\"), First paragraph \n\n    Original Text: \"...instance CIFAR-100 (Xia et al., 2020a) and Clothing-1M Xiao et al. (2015b) where the labels inherit noise...\" \n\n    Citation Style Suggestion: For \"Xiao et al. (2015b)\", consider using \\citet (or the appropriate command for the paper's citation package) if the goal is to have the author names as part of the sentence text, e.g., \"Clothing-1M by Xiao et al. (2015b)\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Meta-Collaborative Distillation (MC-DISTIL), a knowledge distillation framework designed to address distillation when a significant capacity gap exists between teacher and student models. The core idea is to facilitate collaborative learning among a cohort of student models with varying capacities, all learning from a single teacher. This collaboration is orchestrated by a Coordinator Network (C-NET), which functions as a meta-learner. C-NET modulates the training loss for each student on an instance-by-instance basis by reweighting the standard cross-entropy loss and the teacher-matching KL divergence loss. The C-NET learns from each student's performance and training instance characteristics, enabling students of different capacities to improve together. The authors demonstrate that MC-DISTIL achieves significant accuracy gains on datasets like CIFAR100 and TinyImageNet across diverse student and teacher architectures, outperforming existing state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Computational Cost Analysis:\n\n    Provide a more explicit discussion or table comparing the computational cost (e.g., training time, FLOPs relative to training the smallest student model with cross-entropy, or relative to standard KD) of MC-DISTIL against key baseline methods. This would help readers better understand the trade-offs involved.\n\n2. ImageNet Experimentation:\n\n    The authors are encouraged to provide results for MC-DISTIL on ImageNet with more traditional teacher-student pairings, such as ResNet101 to ResNet50 or ResNet101 to ResNet34. This would offer a clearer picture of its benefits in commonly studied large-scale distillation scenarios.\n\n3. Terminology Clarification:\n\n    Revise the statement on page 4, section 3.1 regarding labels. Instead of \"Typically, the labels y are cardinal,\" consider \"Typically, in classification, labels y are categorical, often numerically encoded\". Ordinal labels could also be mentioned as a specific type of categorical label.\n\n4. Address Typos:", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)