Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity
Authors: Seonghoon Yu
Dongjun Nam
Dina Katabi
Jeany Son
Venue: NeurIPS 2025 poster ()
Keywords: knowledge distillation
knowledge augmentation
angular diversity
Primary area: deep_learning

Abstract: Knowledge Distillation (KD) aims to train a lightweight student model by transferring knowledge from a large, high-capacity teacher.
Recent studies have shown that leveraging diverse teacher perspectives can significantly improve distillation performance; however, achieving such diversity typically requires multiple teacher networks, leading to high computational costs. In this work, we propose a novel cost-efficient  knowledge augmentation method for KD that generates diverse multi-views by attaching multiple branches to a single teacher. To ensure meaningful semantic variation across multi-views, we introduce two angular diversity objectives: 1) $\textit{constrained inter-angle diversify loss}$, which maximizes angles between augmented views while preserving proximity to the original teacher output, and 2) $\textit{intra-angle diversify loss}$, which encourages an even distribution of views around the original output. The ensembled knowledge from these angularly diverse views, along with the original teacher, is distilled into the student. We further theoretically demonstrate that our objectives increase the diversity among ensemble members and thereby reduce the upper bound of the ensemble's expected loss, leading to more effective distillation. Experimental results show that our method surpasses an existing knowledge augmentation method across diverse configurations. Moreover, the proposed method is compatible with other KD frameworks in a plug-and-play fashion, providing consistent improvements in generalization performance.

Single-Teacher View Augmentation: Boosting
Knowledge Distillation via Angular Diversity
Seonghoon Yu!
Dongiun Nam 2:
Dina Katabi²
Jeany Son?
GIST *POSTECH
*MIT CSAIL
seonghoon@gm.c gist. ac. kr
dina@csail mit edu
(june6423, jaata,,,ejaann jeany)@postech.aco acc
Abstract
Knowledge Distillation (KD) aims to train lightweight student model by transfer- ring knowledge from large, high-capacity teacher. Recent studies have shown that leveraging diverse teacher perspectives can significantly improve distillation performance; however, achieving such diversity typically requires multiple teacher networks, leading high computational costs. In this work, propose novel cost-efficient knowledge augmentation method for KD that generates diverse multi- views by attaching multiple branches single teacher. To ensure meaningful semantic variation across multi-views, we introduce two angular diversity objec- tives: 1) constrained inter-angle diversify loss, which maximizes angles between augmented views while preserving proximity to the original teacher output, and 2) intra-angle diversify loss, which encourages even distribution views around the original output. The ensembled knowledge from these angularly diverse views, along with the original teacher, is distilled into the student. We further theoretically demonstrate that our objectives increase the diversity among ensemble members and thereby reduce the upper bound of the ensemble's expected loss, leading more effective distillation. Experimental results show that our method surpasses an existing knowledge augmentation method across diverse configurations. Morcover, the proposed method is compatible with other KD frameworks in plug-and-play fashion, providing consistent improvements generalization performance.
Introduction
Knowledge Distillation (KD) has emerged as a powerful paradigm for model compression, aiming to transfer knowledge from large, high-capacity teacher network to compact student model. By guiding the student not only through hard labels but also via richer supervisory signals, such as softened class probabilities [19, 62, 22], intermediate features [6, 15, 39], attention maps [58], KD enables the student to learn nuanced decision boundaries and fine- -gained representations. This approach significantly improves the performance of lightweight models, making it highly suitable for deployment in resource-constrainee environments, such as mobile devices [7], embedded systems [63], and IoT platforms [1].
Building upon this foundation, recent studies have introduced multi-teacher distillation [3, 20, 45, 36], where the student learns from the collective knowledge of multiple teachers, offering richer and more diverse supervisory signal. By aggregating complementary insights from distinct teacher models, these approaches significantly enhance the student's generalization capabilities. However, the benefits of multi-teacher frameworks come with increased computational and memory costs due to the need to train and maintain multiple large models. Moreover, the diversity in such frameworks often arises from initializing identical model architectures with different random seeds (as in ensemble distillation [3]), which inherently limits diversity due shared structural biases [38].
Equul contributions
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

о: D:Teacterioo logt D Teacner Feat. OiAugmentod logt. Augmanied
Constrained Inter-angle (AA Diversify Intra-angle (A) Diversify
Lineel
JEBUT ... Deaun
Inter
O Auginentidd Anguir margir
O: Tercher knowledge
Pushing
Angular
Pushing
Samper
Pushing
Pulling
Pushing
Teacher
margin
(a) Teacher Knowledge Augmentation
(b) Maximizing Inter-, Intra-angles between augmented views
The illustration of our angularly diverse knowledge augmentation: (a) Multi-views are generated from a single pre-trained teacher using multiple pathways, and (b) these augmented outputs are then optimized with two complementary angular objectives maximize their intra- and inter- angular diversity.
Figure 1:
Recently, TeKAP [20] was introduced as knowledge augmentation strategy that simulates multi- perspective supervision by adding multiple stochastic perturbations into features in a single teacher model. This approach reduces the computational cost of need train multiple teacher networks. However, the diversity it induces is entirely driven by random noise, offering limited control over the semantic structure or informativeness of the augmented outputs. This raises the question of whether more structured and controllable augmentation can lead to more effective knowledge transfer, which is direction we explore in this work.
distillation by avoiding the
In this work, we propose a lightweight knowledge augmentation approach that generates diverse perspectives from a single teacher by attaching multiple pathways (Fig. la), thereby eliminating the overhead of training multiple teacher networks. Unlike a noise-based method [20], our approach enables explicit control over the diversity of augmented views while preserving semantic consistency. To ensure attached linear branches produce complementary and informative knowledge, we introduce two novel learning objectives (Fig. 1b) that maximize angular diversity among the augmented outputs. First, the constrained inter-angle diversity loss maximizes the angular separation among all augmented views while constraining them within a margin around the original teacher output to maintain alignment with the source knowledge. Second, the intra-angle diversity loss encourages an even spread of the augmented views centered on the teacher's 's original view, promoting broad coverage of the local knowledge space. Together, these objectives generate a set semantically rich and angularly diverse teacher views. These views are aggregated with the original teacher output to provide a stronger and more informative supervisory signal for student distillation. In addition, significantly reduces computational cost by eliminating the need for training multiple teacher models.
Furthermore, we theoretically demonstrate that our proposed objectives increase ensemble diver- sity [38], which in turn tightens the upper bound on the expected loss, an effect directly linked to improved distillation performance [34]. Empirically, our approach achieves state-of-the-ae perfor- mance on standard knowledge distillation benchmarks under various teacher-student configurations. Moreover, it consistently enhances student generalization when integrated as a plug-and- aud--and-lla module into existing distillation frameworks.
Our contributions can be summarized as follows:
We propose Angular-KD, novel knowledge augmentation framework that generates multiple diverse views from a single teacher by attaching lightweight, learnable linear branches. This eliminates the need to train and store multi multi-teacherree models.
To explicitly control and enhance the diversity of the augmented views, we introduce two angular objectives: 1) a constrained inter-angle diversity loss that encourages angular separation while preserving alignment with the original teacher prediction, and an intra-angle diversity loss that enforces a uniform spread around the teacher output.
We provide both theoretical analysis and empirical evidence that the proposed angular diversity objectives lead to more diverse and structured augmented views, which consistently improve student performance across wide range of distillation settings.
2 Method
In this section, we present Angular- KD, lightweight yet effective knowledge augmentation frame- work that generates diverse supervisory signals from single pre-trained teacher. The core idea is to

---

append multiple lightweight linear branches to the teacher, each generating distinct logits or features that serve as diverse views for distillation (Fig. la). To ensure these augmented outputs provide complementary supervision, we optimize them by maximizing both inter- and intra-angular diversity, while constraining their output within the target class boundaries (Fig. 1b). This encourages the generation of rich knowledge without incurring the costs of training multiple teacher models.
We begin by describing the detailed design of single-teacher view augmentation heads, which generate multiple augmented views from the original teacher knowledge (Sec. 2. We then introduce the inter- and intra-angulae diversity objectives, which explicitly optimize these views achieve the desired angular diversity (Sec. 2.2). Finally, we describe how the student is distilled with both the augmented and original teacher knowledge in an and-to-end manner (Sec. 2.3).
2.1 Single-teacher View Augmentation Heads
To generate diverse views of the teacher's s knowledge using a single network, we attach multiple linear transformation heads, referred to as view augmentation heads, to the teacher. To encourage structural diversity at initialization, each head is initialized using orthogonal initialization [43], allowing them to explore distinct representational directions. This reduces redundancy among learned features and leads to more diverse and informative representations early in training. Additionally, we apply dropout [46] with varying probabilities to the input features of each head, introducing stochastic variation and further encouraging semantic diversity among the views. The architectural design of the view augmentation heads is described in detail below.
Teacher Knowledge Extraction. We utilize two forms of teacher knowledge: the final feature representation and the logit probability. To enable multi-view augmentation, we first extract the teacher knowledge both feature and logit levels using the teacher network. Given an input sample x, the teacher network first produces the final feature vector F = f(x; (() Rdr where f(= ) denotes the teacher's feature extractor, parameterized by ST, and dr is the dimensionality of the feature representation. The corresponding logit probabilities z" are then computed by applying a classification layer WT E RCdr to the extracted features: z = <W""p""/rr E TRC where a.)) is the softmax function, C is the number of classes, and - is a temperature scaling factor to control the smoothness of the distribution.
View Augmentation Heads. To generate N diverse views from the extracted teacher knowledge, we attach set of N linear transformation heads to the teacher. These view augmentation heads are independently applied at the feature and logit levels, producing alternative representations from single teacher source. We first attach N number of multiple feature-level augmentation heads {or J with orthogonal initialization [43] to promote diversity across views. To further encourage semantic variation in views, we apply a dropout mask M, with dropout probability m, to the input feature before passing it through each head. The i-th augmented feature is then computed as:
F (FF),, where o, (FF*) BN(W (M, @ F²)) E Rdr, i (I.....NN) .N, (I) where BN(() is a BatchNorm layer, W Rddxdr xdr is linear layer, and @ denotes the element- wise product. Augmented features F are derived from a unique pair of dropout masks M, and feature-level heads di, enabling the generation of diverse views for distillation.
Each augmented feature is further transformed into logit probability Z E RCC over C classes. This is achieved by passing the augmented feature F through logit-level augmentation head v.. consisting of linear layer W E [xdr,, followed by a softmax a(-) with a temperature r
Z tha (F,)) where v/s (F = ( FA ///r3) E RC, i (1,...,N) N).
J The resulting augmented features (FA) ) and logits (Z Ji=1 are jointly optimized maximize angular diversity by updating the parameters of the view augmentation heads { and ) using our proposed angular diversity loss functions (see Sec. 2.2). These augmented view pairs d [F. N are also distilled into the student model alongside the original teacher pair (FT, ?
as detailed in Sec. 2.3.
2.2 Angular Losses for Learning Diverse Representations
While orthogonal initialization and dropout introduce initial diversity the view augmentation heads, they are insufficient to ensure meaningful separation between the generated views. To explicitly
3

---

encourage representational diversity across views during training, we introduce two angular diversity loss functions: Constrained inter-angle diversity loss and Intra-angle diversity loss.
Constrained Inter-angle Diversity Loss. The constrained inter-angular diversity loss Lime is designed to maximize the angular separation among the augmented knowledge by minimizing their cosine similarity, thereby capturing the absolute deviation in representation space (Fig. 1b). We adopt cosine similarity due to its numerical stability compared directly optimizing the arccosine [1 []. However, excessive angular deviation may cause augmented views to drift toward non-target class boundaries. To mitigate this, we introduce a constraint that encourages each augmented view to remain within a learnable angular margin from the teacher's view. Once all augmented views satisfy this constraint, a diversity term is activated to further maximize inter-view separation. Formally,
N
exp [minnI, 7 +3/1/70)
N
N
Caug inter
log
(3)
ΣnEN, exp (o%/70)
jyts
Construint term
Diversity term
cos F F(A) (feature-level) = cos(ZT, (ZT, ) (logit-level),
cos (FA FA) cos (Z,) 7
(feature-level) (logit-level),
where
$1
cos(u, v)=u-v/llull ||]l, is learnable angular margin, 7 is temperature parameter, Mi denotes the set of negatives for the i-th augmented samples from other batches, and I (.) is an indicator function activated when the specified condition holds.
The loss can be applied either to augmented features FA or to logits Z by computing their angular similarity, encouraging diversity at the representational or predictive level, respectively. To prevent semantic drift, the constraint term penalizes augmented views that fall outside the angular margin, ef- fectively pulling them closer to the teacher representation. Incorporating negatives further encourages discriminative alignment. Once all augmented views reside within the angular margins, the diversity term maximizes inter-view angular discrepancy enhance diversity across the augmented views.
Intra-angle Diversity Loss. While inter-angle loss encourages view separation in the absolute representation space, it does not account for how each view deviates from the teacher. To ensure that the augmented views are evenly distributed around the teacher's knowledge, we introduce the intra-angle diversity loss, Lang This objective minimizes the pairwise cosine similarity among the offset vectors { A ")== which quantify the directional deviation of each augmented view from the teacher representation. Specifically, each offset vector is computed the difference between the teacher representation and the corresponding augmented instance, i.c., AT = R R where R (F,Z) Z} denotes either the feature (F) or logit-level (Z) representation. Formally, defined as:
N
N
FT F.1 (feature-level) ZT
Cang intra
13 where S = cos( T-4 AJ-
AA-A), A.-
Z (logit-level).
Minimizing Aimr encourages structurally diverse offset directions, complementing the inter-angle objective by shaping a more structurally balanced variation around the teacher and promoting diverse, informative knowledge augmentation.
Overall Loss Function for View Augmentation In addition to two angular diversity objectives, we supervise each augmented logit with the ground-truth one-hot labels y = {v1,- ............ ..e} IRC with C' classes, as follows: La = Σi= | CrossEntropy(y, z This supervision acts as a regularization signal to prevent the augmented predictions from diverging too far from the true semantic target. The overall loss function applied to the augmented knowledge is defined as: _ gug + L + Came We refer to feature-level augmentation as the setting where the angular diversity loss _aug is applied exclusively to the outputs of the feature-level augmentation heads, while logit-level augmentation applies the loss solely to the outputs of the logit-level heads. Our full model incorporates both logit- and feature-level augmentation, enforcing angular diversity at both the feature- and logit-level representations. This encourages consistency and diversity across both the representational and predictive spaces, leading more robust and generalizable learning.

---

2.3 Distillation with Augmented Knowledge
To distill the augmented teacher knowledge into the student, we construct a (N+1)-wa ensemble by averaging the original teacher output with its N augmented variants. The resulting ensemble representation is denoted as F or z depending on whether feature- or logit-level outputs are used. This combined representation serves supervision for the student model. For feature-level distillation, we employ a Contrastive Representation Distillation (CRD) [50] loss, Ldistill to align the student's &s features with the ensembled teacher representation. For logit-level distillation, we employ a Kullback-Leibler (KI.) divergence loss [19], Liogi KL. E ) to align the student's logit z with the ensembled logits aggregated from the teacher and augmented views. Additionally, we apply a standard cross-entropy loss Cdistill CrossEntropy((y zo) to directly supervise the student with ground-truth labels. The overall distillation loss is defined as Ldistilll feat. + L + Cdistil where either or both of the feature- and logit-level terms can be included depending on the chosen distillation setting. The augmentation and distillation losses are applied to the teacher and student, respectively, enabling joint end-to -nd training of the entire model.
3 Theoretical Analysis
To theoretically justify the performance benefits of our approach, build upon existing analyzes of ensemble diversity [38], which demonstrate that increased diversity among ensemble members reduces the upper bound on the expected ensemble loss. In this section, we formally analyze (I) how our inter- and intra-angle diversity losses relate to the ensemble diversity, and (2) how this amplified diversity tightens the upper bound on the ensemble's ' expected loss, thereby improving student performance.
Effect of Intra-, Inter-angular Losses on Ensemble Diversity. Let [ N denote the set of logits used in the ensemble, where Zo - is the original teacher output and Z; Z are the logits from the i-th augmented view for i = -,,... N. The ensemble diversity metric ID(-) [49, 37], defined over the ensemble members {Z, 1.2 and dataset D. can be expressed in two equivalent forms, depending on either the pairwise similarity st (defined in Eq. (3)) between normalized logits or the angular similarity si) (defined in Eq. (4)) between offset vectors, as follows:
E E
N
Z 2 - EN 22 I1 Is [ (5a)
D((Z)) ( V.V.
Z: max Z₁
]]
(x T / -1 11
(5b)
The full derivation is provided in the supplementary material. These two equations indicate that ensemble diversity grows as the cosine similarities st and &t decrease. Notably, the inter-angle diversity loss (Eq. (3)) explicitly minimizes st,, while the intra-angle diversity loss (Eq. (4)) targets at. Together, these losses reduce the similarity terms in the ensemble diversity metric D, thereby increasing diversity and tightening the upper bound on the ensemble's expected loss, shown below.
A
Upper Bound on Expected Loss Ensemble. Similar to the analysis in [38], the expected loss ((xy)= [C()] of the ensemble logit E over the dataset D is bounded by
E (xx))>> [
< E (xx))~
BY [C())]] ]
KD((Z)
(6)
where is a cross-entropy loss and 0 is constant. This bound shows that increasing the ensemble diversity DD lowers the upper bound on the ensemble's 's expected loss. Furthermore, this bound is also tightened by an improvement in the average quality of the individual views. By maxi- mizing angular diversity, our method lowers this bound, strengthening the ensemble supervision and improving the student performance [34]. The detailed derivation is also included in our supplementary.
4 Experiments
4.1 Experimental Setting
Dataset and Metric. We conduct experiments on various KD benchmark datasets: 1) CIFAR- 100 [24], 100-class image classification dataset containing 50,000 training and 10,000 validation images of size 32 x 32, 2) ImageNet [10]: a large-scale classification dataset with 1,000 categories, approximately 1.28 million training and 50,000 validation images, each of size x24x224 and 3)
5

---

Table Results on CIFAR-100. Top-1 test accuracy (%) of student models across various teacher-
student configurations compared with SoTA augmentation method [20]. Augmentations are applied at three different levels: logit-level augmentation with logit distillation augmentation with feature distillation (CRD), and logit- and feature-level augmentation with logit and feature distillation (KD CRD).
Same Architecture
Different Architecture
Arch, of Teacher Student
RN32x4
VGG13
WideRN 40
RN32x4
RN32x4
WideRN 40
RN8x4
VGG8
WideRN-10-1
WideRN-40-2
WideRN-16-2
RN8x4
Acc. of Teacher
79.42
74.64
75.61
79.42
79.42
75.61
Student
72.50
70.36
71.98
75.61
73.26
72.50
Logit-level augmentation
Logit Distil. [19] who
73.33
72.98
73.54
77.70
74.70
73.97
TcKAP [20]
74.79
74.00
73.80
77.97
75.08
75.09
Angular- (ours)
76.08
74.57
74.86
78.68
76.22
76.27
Feature-level augmeniation
Feat. Distil. [50] w/o nng
75:51
73.94
74.14
78.15
75.65
75:24
w/ TeKAP [20]
75.65
74.10
74.21
78.05
75.42
75.65
Angular-KD KD (ours)
75.82
74.55
74.91
78.51
76.19
76.20
Logit- and feature-level augmentation
Combined Distil.
75.46
74.29
74.38
78.01
75.86
75.57
TeKAP [20]
75.98
74.42
74.41
78.68
75.62
75.90
w/ Angular- KD (ours)
76.46
74.76
75.00
78.94
76.42
76.29
Table 2: Plug-and-Play Results on CIFAR-100. Comparison with a SoTA augmentation method [20]
for KD. Three scenarios are reported: the original SoTA KD method without any augmentation, augmented with TeKAP, and augmented with our Angular-KI KD.
Same Architecture
Different Architecture
Arch. of Teacher Student
RN32x4
VGG13
WideRN 402
RN32x4
RN32x4
WideRN
RN8
VGGS
WideRN-40-1
WideRN-40-2
WideRN-16-2
RN8x4
Teacher
79.42
74.64
75.61
79.42
79.42
75.61
Acc. of Student
72.50
70.36
71.98
75.61
73.26
72.50
DKD [62] wño aug
76.32
74,68
74.81
78.46
75.70
75.56
TcKAP [20]
76.65
74.55
73.83
78.64
75.28
76.22
w/ Angular-KT KD (ours)
76.51
74.76
74,89
78.99
76.05
76.14
ReviewKD [6] wdo ang
75.63
74.10
75.09
78.96
76.11
74.34
w/ TeKAP [20]
75.32
74.29
75.15
79.55
76.36
75.59
w/ Angular-KD (ours)
(ours)
75.78
74.12
75.45
78.28
76.40
75.97
MLKD [22] aug
77.08
75.18
75.35
79.26
76.52
77.33
w/ TeKAP [20)
77.04
75.37
75.31
78.72
76.46
77.28
w/ Angular-KD KD (ours)
77.28
75.63
75.37
79.52
76.60
77.45
Imbalanced CIFAR-100, following prior works [50, 20], where 43 classes out of 100 CIFAR classes
are selected, and each class is limited to 50 training samples. The imbalanced classes can found in our supplementary. 4) STL-10 [8], a 10-class image classification dataset with an image size of 96 96, comprising 5,000 training images and 8,000 test images. 5) Tiny ImageNet [10], a 200-class subset of ImageNet, with each class containing 500 training images, 50 validation images, and 50 test images with size of 64 x 64. For the evaluation metric, we use top-1 classification accuracy (%).
Implementation Details. For view augmentation heads (Sec. 2.1), we generate N = 5 augmented views, apply dropout with probabilities (0.2, 0.25,0.3. 0.35, 0.4). and use softmax temperature of 7 = 4. For constrained inter-angle diversity loss (Sec. 2.2), the learnable margin y is initialized to 0.2, and the contrastive temperature is set C 0.07. For the ensembling (Sec. 2.3), we uniform weights across all ensemble members. Training starts with a 30-epoch warm-up phase where only the view augmentation heads are trained to ensure stability. Subsequently, the student model is trained for 240 epochs on a single RT' 2080 Ti GPU using SGD optimizer, and a batch size of 64. The learning rate starts at 0.01 and is decayed by a factor of 10 epochs 150, 180, and 210.
4.2 Main Result
Results on CIFAR-100. Table reports a comparison between the proposed Angular- RD and and the prior augmentation method [20], on CIFAR-100. We evaluate three augmentation levels: (1)

---

Table 3: Results on ImageNet. The logit-level augmenta- tions are applied for both Angular-KI KD and TeKAP under a ResNet34 teacher and ResNet18 student.
Table 4: Results on Binary Segmen- tation on the Carbana Image Masking dataset.
Metric
Teacher
Student
w/o aug
TeKAP
Ours
Methods
Dice Loss
IoU
Top-1 Ace.
73.31
69.75
70.41
70.67
71.07
Naíve KD
0.0218
94.83
Top.5 Acc.
91.42
89.07
89.88
89.92
90.39
+ Ours
0.0208
95.72
Table 5: Results imbalanced CIFAR-100. "Imbalanced set" includes 43 under-sampled classes, "Full ser" indicates total 100 classes with extra balanced 57 categories.
Table 6: Transferability Results. The distilled
knowledge on CIFAR-100 is transferred to STL- 10 and TinylmageNet. The classifier for each
dataset is trained on top of a frozen student.
Class type
w/o aug
TeKAP
Ours
Dataset
w/o aug
TeKAP
Ours
Imbalanced set
33.23
34.98
35.66
STL-10
68.01
68.71
70.23
Full set
60.74
61.18
61.84
TinyImageNet
31.17
31.54
32.97
logit-level augmentation with logit distillation (KD [19]), (2) feature-level augmentation with feature distillation (CRD [50]), and (3) their combination. Evaluations are conducted across multiple teacher- student configurations under two scenarios: same-architcctree, where both teacher and student use the same network, and cross-archiectuue, where different backbones are employed. In the table, we denote ResNet [17] and WideResNet [57] as RN and WideRN, respectively. Across all configurations, Angular-KD consistently outperforms TeKAP by substantial margins, highlighting the effectiveness of angular-diversity-bss augmentations over TeKAP's s random noise perturbations.
Plug-and-Play Results on CIFAR-100. Table presents the distillation performance of our Angular-KDD when integrated as a plug-and-play module on top of SoTA KD methods [62, 22, 6]. compared to TeKAP [20], under both same architecture and different architecture settings. For most methods, adding Angular-KI ID yields consistent performance gains, and it outperforms TeKAP in most settings, demonstrating its effectiveness and generalization in generating valuable supervisory signals for the student through our angularly diverse augmentation.
Results on ImageNet. To validate the scalability, we evaluate Angular-KD on the ImageNet [10] validation set using ResNet34 teacher and ResNet18 student. Table 3 reports Top-1 and Top-5 accuracies (%) for both Angular-KD and TeKAP under logit-level augmentations with naive KD [19] Our method achieves the best results compared to the unaugmented one and TeKAP, confirming its
robustness on large-scale datasets. The detailed implementations are presented in the supplementary.
Results on Binary Segmentation. To further validate that our approach is effective beyond the image classification task, we conduct a binary segmentation task on the Carvana Image Masking dataset [44], which includes 5,088 training images and 100,064 test images. Using a U-Net-32 teacher and a U-Net- 16 student, we compare the augmented KD with our Angular- KD against the unaugmented KD baseline. As shown in Table 4, our approach achieves substantial improvements over the baseline across two metrics (Dice loss [35] and IoU), demonstrating its task generality beyond image classification.
imbalanced 100 dataset, where 43 of the 100 classes are undersampled to 50 training images (See supplementary for details). The student is assessed two test settings: (1) the 47-class imbalanced subset and (2) the full 100- class set including 53 balanced classes. All models use a WideRN-40-2 teacher and a WideRN-16-2 student, with both feature- and logit-level augmentations. As shown in Table 5, Angular- achieves the highest accuracy in both settings, highlighting strong robustness in imbalanced scenarios.
Results on Imbalanced CIFAR-100. We assess robustness
Transferability Results on STL-10 and Tiny ImagemmaeNee. To validate that our Angular- KD leads student to a more general model, we evaluate the 100 distilled student (via unaugmented, TeKAP, or Ang on S1L-10 and TinylmageNet. For this experiment, we train a classifier for each transferred dataset on top of the frozen student, and use a WideRN-40-2 teacher and WideRN- 16-2 student. In Table 6, a trained student via Angular-KI achieves the best performance, underscoring that angularly diversified distillation produces more generalizable representation.
7

---

Table Advantages over multi-teacher approaches. The best results are highlighted in bold and second best are underlined. i indicates Ensemble Distillation [3].
WideRN 2rater
WideRN 2i
VGGS
Metric
Mulit-teocher Methous
Single-teccher Aug.
Mulli-tucher Methody
Swwle-teacherr Aug.
EDI
TAKD [36]
DGKD [45]
TeKAP
Ours
EDI
TAKD [66]
DGKD [45]
TeKAP
Ours
76.31
75.04
76.24
76.20
76.33
24.67
74.40
74.76
Params (M)
6.69
2.26
2.40
47.31
20.31
FLOPs (M)
1645
797
797
329
329
1427
497
285
247
Table 8: Ablation on each proposed method. We report accuracy and the ensemble diversity metric (from our theoretical analysis) to assess the impact of each objective.
THKAP
72
Constrained Inter-angle Diversity Loss
Intra-angle Diversity Loss
Ensemble Diversity
68
Acc.
75.46
64
76.16
11.522
76.28
11.617
50%
76.46
11.633
Figure 2: Few-Shot Results.
Advantages over Multi-teacher Approach. To highlight our advantages over multi-teache distil- lation methods [3, 36, 45], we compare KD accuracy and resource efficiency Table We consider two configurations: (1) WideRN setting, where Angular-Ki KD and TeKAP employ single WideRN- 40-2 teacher and a WideRN-16- student, while TAKD [36] and DGKD [45] each train four teachers (WideRN-40-2, -34-2, -28-2, -22-2), and Ensemble Distillation [3] uses five WideRN-40-2 teachers; and (2) VGG setting, where Angular-KI and TP use single VGG13 teacher and a VGG8 student, while TAKD [36] and DGKD [45] train two teachers (VGG13, VGG11), and Ensemble Dis- tillation [3] uses five VGG13 teachers. We report the total number of teacher parameters and FLOPs, which reveal the substantially higher resource cost of multi-teacher methods. Notably, Angular-KD achieves higher accuracy than all baselines while requiring only a single teacher, demonstrating superior efficiency.
Few-shot Results. We further evaluate Angular- KD under limited-data scenarios by randomly sampling 25% 50%, and 75% of the CIFAR- 100 training set compared with unaugmented KD + CRD and SoTA augmentation method, namely TeKAP [20], as shown in Fig. 2. All compared methods use a WideRN-40-2 teacher and a WideRN-16-1 student. Across all sampling ratios, Angular- KD achieves the best average performance over three trials (error bars are shown in the figure). These results indicate that our angular- diversity augmentations provide more informative supervisory signals that help the student generalize even when training data are scarce.
4.3 Ablation Study
We conduct an ablation study CIFAR-100 using ResNet32 and ResNet8 x as the teacher and the student, respectively.
Effect of Each Proposed Method. In Table 8, we analyze the contribution of each proposed method in our Angular-KI KD by reporting accuracy and ensemble diversity (defined in Sec. 3). Beginning with the unaugmented KD CRD baseline, we then sequentially introduce our angular diversification objectives with augmentation heads. The constrained inter-angle diversity loss alone increases both accuracy and ensemble diversity, indicating that separating views from one another beneficial. Our intra-angle diversity loss also drives up ensemble diversity, thereby improving performance. When combining both angular-diversin objectives, we achieve the best accuracy and diversity, confirming that inter- and antra-nguular objectives work together to strengthen distillation.
Effect of the Number of Augmentations. Table 9 explores the effect of varying the number
of augmentations (N) on performance. Notably, even single augmentation (N = 1) yields improvement over the unaugmented KD CRD baseline, denoted as "woo" in the table. We obtain

---

Table Ablation study on the number of augmentations. "w/o aug" refers baseline with logit
(KD) and feature (CRD) distillation without augmentation. denotes teacher's 's Params and FLOPs.
of Augmentations (N)
Metric
w/o aug
2
3
5
6
Acc.
75.46
75.87 +0.39
75.85 +0.39
76.25 +0.79
76.44 +0.98
76.46 +1.00
76.37 +0.91
Params (M)
7.434
+0.092
+0.184
+0.276
+0.368
+0.460
+0.552
FLOPs (M)
1085.629t
+0.092
+0.185
+0.277
+0.370
+0.462
+0.554
Table 10: Ablation study within view augmentation head.
Table 11: Ablation study within the constrained inter- angle diversify loss.
Orth. Init.
Dropout
Acc.
(a) Constraint, diversity terms
(b) Margin y
17
Constraint
Diversity
Acc.
Margin y
Acc.
76.31
76.25
0.1
76.34
76.35
/
76.29
0.2
76.46
76.46
/
76.46
0.3
76.31
the best results at N = 5. Importantly, increasing N incurs marginal computational overhead, as each augmentation adds only a lightweight head with negligible impact, espcially on overall FLOPs.
Ablation Study within View Augmentation Heads. We conduct ablation to isolate the individ- ual effects orthogonal initialization [43] and input dropout [46] on our view augmentation heads, as shown in Table 10. Initializing each head with orthogonal weights increases accuracy, demonstrating that starting in distinct directions leads to more diverse views. Applying dropout before each head further boosts the results. Combining both orthogonal initialization and dropout yields accuracy gains, confirming that these techniques work synergistically to diversify the augmented knowledge.
Ablation Study within Constrained Inter-angle Diversity Loss. We analyze the effects of the constraint and diversity terms, as well as the choice of angular margin in our constrained inter-angle diversity loss. Table la shows that combining both constraint and diversity terms yields the highest accuracy, confirming the effectiveness of our objective design. Table ll further reveals that setting an angular margin 7 0.2 yields the highest results, offering the best trade-off between promoting diversity and preserving class boundaries.
I-SNE Visualization. We visualize the t-SNE embeddings of the original teacher logit and aug- mented logits produced by our method (Fig. 3a) compared to those from SoTA augmentation method [20] (Fig. 3b). For clarity, we randomly sample 10 out of the 100 classes and assign a unique color to each category. We observe that augmented views persed, particularly for the gray and red classes, while those of TeKAP are clustered. Furthermore, TeKAP's augmented views overlap heavily the purple and red class regions, unlike ours.
more evenly dis-
(a) Ours
(b) TeKAP
Figure 3: The t-SNE visualization teacher (x) and augmented logits (o) from Angular-KD compared to TeKAP.
5 Related Work
Knowledge Distillation. Knoweldge Distillation (KD), first introduced by Hinton et al. [19], train a compact student to mimic the softmax outputs of a large teacher. Since then, 1) lased-based approaches [62, 29, 47, 59, 61, 28] have been developed by modifying logit probabilities, egg. using adaptive temperature [14], or predicting sample-wise temperature [29]. Parallel to this, 2) feature- based approaches [2, 18, 30, 40, 32, 55, 5, 15] transfer intermediate representations directly, such as feature maps [42], and spatial attention maps [58], enabling more fine- fine-grained guidance. To further boost knowledge transfer, multi-teacher distillations [56, 13, 54] aggregate diverse views

---

from multiple independently trained teachers. While effective, these methods demand significant computational resources. In this work, we replace the array of teachers with lightweight linear branches, drastically reducing the overhead of acquiring diverse knowledge.
Diversity and Ensemble Learning. Ensemble methods boost performance by encouraging diversity among their models. Classic Bagging [4] and Boosting [12] do via resampling or reweighting, while modern deep ensembles use random initializations [27, 52] or stochastic optimization [33, 60, 533. In contrast, we generate varied predictors through lightweight single-linear branches combined with our angular diversity losses. Furthermore, a range of diversity metrics [37, 23] (i.e., ambiguity [25], orthogonality [26], disagreement [25]) has been proposed. Especially, Ortega et al. [38] formalize a generic diversity measure linked to the generalization error bound. We show that our objectives increase this measure, thereby tightening the theoretical bound and enhancing distillation [34].
Teacher Knowledge Augmentation. TeKAP [20] first tackles the high cost of multi- Malti-teacher distillation by augmenting angle-teacher knowledge into diverse views via random perturbations. While this reduces resource demands, its reliance on undirected, stochastic noise limits the overall diversity. In contrast, our method explicitly maximizes angular separation among generated views via two novel angular-diversity losses, ensuring more complementary, less-redundant augmentations.
Angle-based Learning Objective. A series of methods [48, 21, 51] have shown that imposing angular margins on the classifier's hypersphere largely boosts inter-class separation and enforces intra- class compactness. SphereFace [31] replaces softmax logit with cos(m0) to enforce multiplicative angular margin; CosFace [51] applies an additive margin m via cos(f) and ArcFace [1 1] directly adds an angular offset cos(@ + m) optimize geodesic separability. Unlike these methods, our angular-diversity losses work on multiple augmented views of a single teacher, not on different classes, harness angular maximization for knowledge diversity rather than class maas-n enforcement.
Discussions
Limitations. A key limitation of our approach is that the augmentations are inherently constrained by the original teacher's knowledge, making it difficult to introduce fundamentally new semantic information. While we achieve multi-view diversity through inter- and intra-angular variations, our method does not capture the broader semantic richness that could be offered by multiple distinct teacher models. Moreover, although angular augmentation computationally more efficient than multi-teacher approaches, generating multiple views still incurs non- negligggie training- time over- head. In future work, we plan mitigate this limitation by incorporating novel semantic signals beyond those of the original teacher, while keeping additional costs minimal.
Social Impacts. Our method, Angular-KI MDD improves the efficiency and accessibility knowledge distillation by eliminating the need for multiple large teacher models. This may facilitate the deployment compact, high-performing models in low-resource environments such as edge devices, mobile platforms, and underserved regions. The reduced computational demand may also contribute to lowering the environmental impact of large- training and reducing the overall carbon footprint. However, there are also potential societal risks. As all augmentations teacher, any biases or blind spots present in the teacher model may be transferred or even amplified in the student. This raises concerns related to fairness and representational harm, particularly in sensitive domains such as hiring, finance, healthcare, or surveillance. Furthermore, since angular perturbations not semantically grounded, the approach may limit interpretability and increase susceptibility to adversarial manipulation.
generated from single
7 Conclusion
In this paper, we present a novel KD augmentation method that generates angularly diverse views from a single teacher using linear layers with two novel angular diversity learning objectives, where valuable findings are provided: (1) Theoretically, angular diversity in augmented views leads to high ensemble diversity, reducing student emrrr, and (2) empirically, our angularly diverse augmentation achieves the best performance across various KD benchmarks, and delivers additional gains when incorporated as plug-am aade--nddplay with existing KD approaches.
10

---

This work was supported by the IITP grants (RS-2019-1119196, RS-2022- (1220926) funded by MSIT and the GIST-MIT Research Collaboration grant funded by GIST, Korea.
Acknowledgments.
References
[1] Mahmoud Abbasi, Amin Shahraki, Javier Prieto, Angélica González Arrieta, and Juan M. Corchado. Unleashing the potential of knowledge distillation for iot traffic classification. IEEE Transactions on Machine Learning in Communications and Networking (TMLCN), 2:221-239, 2024.
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. In International Conference on Learning Representations (ICLR), 2023.
[4] Leo Breiman. Bagging predictors. Machine Learning, 24:123-144 1-2--140..10 1996.
[5] Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[6] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia, Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2021.
[7] Kwanghee Choi, Martin Kersner, Jacob Morton, and Buru Chang. Temporal knowledge distillation for on-device audio classification. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[8] Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature lcarning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
[9] Tri Dao. Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey, Knowledge distillation as semipara- metric inference. International Conference on Learning Representations (ICLR), 2021.
[10] Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[11] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[12] Yoav Freund and Robert E Schapire. Experiments with boosting algorithm. In International Conference on Machine Learning (ICML), 1996.
[13] Takashi Fukuda, Masayuki Suzuki, Gakuto Kurata, Samuel Thomas, Jia Cui, and Bhuvana Ramabhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, 2017.
[14] Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofei He, and Deng Cai. Reducing the teacher-student gap via spherical knowledge disitllation. arXiv preprint 2702:201 2020.
[15] Ziyao Guo, Haonan Yan. Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
[16] Shayan Mohajer Hamidi, Xizhen Deng, Renhao Tan, Linfeng Ye, and Ahmed Hussein Salamah. How train the teacher model for effective knowledge distillation. European Conference on Computer Vision (ECCV), 2024.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[18] Bycongho Hco, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A compre- hensive overhaul of feature distillation. In Proceedings of the IEEESCVF International Conference on Computer Vision (ICCV), 2019.
[19] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXi::1003.0253) 2015.

---

[20] Md Imtiaz. Hossain, Sharmen Akhter, Choong Seon Hong, and Eui-Nam Huh. Single teacher. multiple perspectives: Teacher knowledge augmentation for enhanced knowledge distillation. In International Conference on Learning Representations (ICLR), 2025.
[21] Yuge Huang, Yuhan Wang. Ying Tai, Xiaoming Liu, Pengcheng Shen, Shaoxin Li, Jilin Li, and Feiyue Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[22] Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[23] Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? Advances in Neural Information Processing Systems (NeurIPS), 2017.
[24] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[25] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in Neural Information Processing Systems (NeurIPS), 1994.
[26] Ludmila I Kuncheva and Christopher Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning. 51:181-207, 2003.
[27] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[28] Zheng Li, Ying Huang, Defang Chen, Tianren Luo, Ning Cai, and 7higeng Pan. Online knowledge distillation via multi-branch diversity enhancement. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2020.
[29] Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023.
[30] Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang- Knowledge distillation via the target-aware transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[31] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[32] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[33] Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep leaming. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[34] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. statistical perspective on distillation. International Conference on Machine Learning (ICML), 2021.
[35] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In International Conference on 3D Vision (3DV), 2016.
[36] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
[37] Ambroise Odonnat, Vasilii Feofanov, and levgen Redko. Leveraging ensemble diversity for robust self- training in the presence of sample selection bias. In International Conference on Artificial Intelligence and Statistics (NeurIPS) Workshops, 2024.
[38] Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural network ensembles. In International Conference Artificial Intelligence and Statistics (AISTATS), 2022.
12

---

[39] Wonpyo Park. Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[40] Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2019.
[41] Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International conference machine learning (ICML), 2019.
[42] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. International Conference on Learning Representations (ICLR), 2015.
[43] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of lcarning in deep linear neural networks. In International Conference on Learning Representations (ICLR), 2014.
[44] Brian Shaler. DanGill, Maggie, Mark McDonald, Patricia, and Will Cukierski. Carvana image masking challenge, 2017.
[45] Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the JEEE/CVF International Conference on Computer Vision (CVPR). 2021.
[46] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting The Journal of Machine Learning Research (JMLR), 15999 1958, 2014.
[47] Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[48] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[49] Ryan Theisen, Hyunsuk Kim, Yaoqing Yang, Liam Hodgkinson, and Michael W Mahoney. When are ensembles really effective? In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[50] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations (ICLR), 2020.
[51] Hao Wang. Yitong Wang. Zheng Zhou, Xing Ji. Dihong Gong. Jingchao Zhou, Zhifeng Li. and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[52] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations (ICLR).
[53] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems (NeurIPS). 2020.
[54] Meng-Chiel Chieh Wu, Ching-Te Te Chiu, and Kun-Hsuan Wu. Multi-teacher knowledge distillation for compressed video action recognition on deep neural networks. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
[55] Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[56] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks, In International Conference on Knowledge Discovery and Data Mining (KKD), 2017.
[57] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint ar5v::605..07146. 2016.
13

---

[58] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. International Conference on Learning Representations (ICLR), 2017.
[59] Rongzhi Zhang, Jiaming Shen. Tianqi Liu, Jialu Liu, Michael Bendersky, Mare Najork, and Chao Zhang. not blindly imitate the teacher: Using perturbed loss for knowledge distillation, arXiv preprint arXiv:2305.05012 2023.
[60] Ruqi Zhang, Chunyuan Jianyi Zhang. Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representations (ICLR), 2019.
[61] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[62] Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[63] Qi Zhao, Shuchang Lyu, Zhiwei Zhang, Ting-Bing Xu, and Guangliang Cheng. Embedded knowledge distillation in depth-leve dynamic neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions list in Sec. introduction are detailed in Sec. Theoretical Analysis and Sec. Experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Sec. 6 and the supplementary
material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: We provide short, brief proofs in Sec. and detailed explanations are offered in the supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We explain experimental settings in Sec. 4 and more detailed settings in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Codes are included in the supplementary material Zip file. We use datasets that are publicly available. We will release our code on GitHub in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httpssccccpbcccccccceec.c cerollccggiidee gugggdiesss des/ CooeeSubm ssi assonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all necessary details understand the results in Sec. 4 and the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars in the supplementary material and Fig. 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the information in Sec. Implementation Details, Table and Table 7.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttp:///nurips... cc/ cc/public/Btieee www.dieieneeeeeeeeeee
Answer: [Yes]
Justification: This work conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss social impacts in Sec. 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite all assets and mention the license and terms of usage.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in wwwrrsswtthhhe.....
package should be provided. For popular datasets, paperswi thcode. com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We are releasing our code as new asset, fully documented on GitHub, to complement the documentation in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions NeurlPS Code of Ethics and the
and locations, and we expect authors to adhere
guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity Supplementary materials -
Overview of Supplementary Materials
We provide the table of contents for the supplementary materials below:
A. Additional Results
A.1. Additional Plug-and-Play Results with a Multi-teacher Method
A.2. Additional Transferability Results on STL-10 and TinylmageNet
A.3. Comparisons on Training Time
A.4. Experiments Statistical Significance
B. Analysis on Augmented Views
B.I. Accuracy for each Augmented View
B.2. Effect of Inter- and Inter- angles in Ensemble Diversity and Accuracy
B.3. Correlation Matrix between Augmented Views
C. Additional Implementation Details
D. Data Augmentation v.s. Our View Augmentation
E. Additional Theoretical Analysis
E.1. Generalized Ensemble Diversity beyond a Single Class
E.2. Ensemble Diversity is Proportional to Logit Variance
E.3. Two Equivalent Forms of Ensemble Diversity
E.4. Better Teacher, Better Student
A Additional Results
A.1 Additional flug-and-Play Results with Multi-teacher Method
In Table 10, we report the KD performance when integrating our Angular- KD into the multi- teacher approach, TAKD [36]. compared with when adapting the SoTA KD augmentation method, TeKAP [20]. In this setup, TAKD uses additional teacher assistants (WideRN-22-1 or WideRN-22- 2) with the primary teacher network (WideRNN40-2) for multi-teacher distillation. We apply our Angular-KD KD to all transfer steps: 1) the primary teacher to the assistant, and 2) the assistant to the student. The augmented TAKD with our methods shows superior performance over an unaugmented TAKD and an augmented one with TeKAP, highlighting that introducing angularly diversified views into each teacher assistant yields stronger distillation than random perturbations TeKAP.
Table 10: Plug-and-Pin Results with TAKD on CIFAR-100, The WideRN-40-2 teacher is used at all assistant settings.
Teacher Assistant
WideRN-22-2
WideRN-22-1
Student
WideRN-16-2
WideRN-40-1
WideRN-16-2
WideRN-40-1
TAKD [36]
75.02
72,73
72,56
71.19
TcKAP [20]
76.02
74.02
73.02
72.47
w/ Angular-Ki KD (ours)
76.27
75.28
75.02
74.33
21

---

A.2 Additional Transferability Results on STL-10 and TinyImageNet
We extend our transferability experiments on both STL,100 and TinylmageNet using 1) a ResNet32 x teacher and ResNet8 x 4 student and 2) a VGG13 teacher and a VGG8 student (results for WideRN-40-2 teacher and a WideRN-16-2 student appear in Table 4 of the main paper). We adopt augmentations at both feature and logit levels. For each method (the unaugmented one, the augmented with TeKAP, and (ngular-KD), we freeze the student model distilled from CIFAR-1O0 and train a linear classifier each downstream dataset. As shown in Table 11, Angular- KD again achieves the highest accuracy, demonstrating that its benefits carry over to different model backbones, underscoring its broad applicability.
Table 11: Transferability Results. The distilled knowledge on CIFAR-100 is transferred STL-10 and TinylmageNet. The classifier for both datasets is trained on top of a frozen student. The "WRN" indicates WeteResNet.
RN8 x 4qnverr
VGGI 3 Fowweer
VGG8
WRN-40 2rwwee
WRN-16 16-2
Dataset
w/o aug
TeKAP
Ours
w/o aug
TeKAP
Ours
w/o aug
TeKAP
Ours
STL-10
70.66
71.70 34.46
73.25
67.11 51.92
67.53
68.43
68.01
68.71
70.23
TinyImugeNet
33.37
36.71
31.96
32.43
31.17
3134
32.97
A.3
Comparisons on Training Time
To validate the efficiency of our approach, we compare the training time of our methods with 1) nafve KD + CRD without augmentation, 2) the augmented one by TeKAP, and 3) multi-teacher methods (e.g., Ensemble distillation [3]), including computational demands (FLOPs and Params; reported in Table 7) in Table. This experiment is conducted with a WideResNer40-22 teacher and a WideResNet-166 student on 2080 Ti GPU over 50,000 training images and a total of 240 epochs. We observe that our multiple branches incur only marginal increase in both training time and computation cost (FLOPs and Params), since the branches are composed of lightweight linear heads.
Table 12: Comparisons on Training Time. We report training time during epoch over 50,000 CIFAR training images with computational demands (Parameters and FLOPs).
w/o aug
TeKAP
Ours
Ensemble Distillation [3]
Training Time (8)
27
32
35
47
Params (M)
2.26
2.26
2.40
11.28
FLOPs (M)
329.02
329.02
329.19
1645. 10
A.4 Experiments for Statistical Significance
Table 13 presents the results of a 3-trial experiment to evaluate the stability of our methods on CIFAR-100 under each of augmentation level (feature- logit-, and their combined-levels), using a R4 teacher and a ResNet8 x4 student (Table 13a) and a ResNet32 x4 teacher and a WideRessNt40x22 student (Table 13b), compared to TeKAP. We report accuracy for each trial and then compute the average (Avg.) and the standard deviation (Std.) across runs. Our methods consistently outperform TeKAP average performance across all settings. This consistency indicates that the observed improvements are statistically significant and not a result of random variation.
Table 13: Experiments for Significal Significance. We report 3-trial results on CIFAR-100 with the average (Avg.) and the standard deviation (Std.)
Logit level Distill.
Leve! Distill.
Logit, Feature- level Distill.
Metri
w/o aug
TeKAP
Ours
w/o aug
TeKAP
Ours
w/o aug
TeKAP
Ours
Acc. (Top-1)
73.33
15.84
76.08
75.51
75.82.0.15
75.94:0.11
75,46
75.89.1006
20.3
(a) A ResNet32x4 teacher and a ResNet8 44 student.
Logit-leve! Distill.
Feature-teve! Distill.
Logit, Feature-level Distill.
Metric
w/o aug
TeKAP
Ours
w/o aug
TeKAP
Ours
w/o aug
TeKAP
Ours
Acc. (Top-1)
77.70
77.90 77.99=019
78.57=0133
78.15
77.97+021
78.56.0.20
78.01
77.9620.0
78.62=000
(b) A ResNe1122x teacher and a WideResNer40x2 student.
22

---

B Analysis on Augmented Views
B.1 Accuracy for each Augmented View
In Table 14, report Top-1 accuracy (%) on 100 for each augmented view and for the final ensemble between the original teacher and all augmented views, with a WideRN-40-2 teacher and a set N for both
WideRN-16: 16-2 student. In this experiment, of our five angularly diversified views performs worse than the original teacher and shows relatively large performance variation. However, when ensembling all five augmented views with the teacher, the overall accuracy improves beyond that of the teacher alone. In contrast, ensembling the teacher with AP's five random perturbations actually lowers performance compared to using the teacher by itself. In addition, our student model also achieves better accuracy than TeKAP. The result shows that our views provide more complementary information, leading the highest KD accuracy.
method and TeKAP [20]. Each
Table 14: Accuracy for each Augmented View. We report the accuracy of the final ensemble with the original teacher and all augmented views (denoted as "Ensemble Acc."). The ResNet32 x teacher and ResNet8 x 4 student are used. "Student KD Acc." indicates the accuracy of the distilled student model. f denotes the original score reported in the TeKAP paper, while other scores are our reproduced results based on the authors' code.
Augmented Views N
Ensemble Acc.
Student KD Ace.
Methods
Teacher
2
3
4
TcKAP [20]
79.42
79.34
79.38
79.41
79.48
79.45
79.38
7 75.998
Angular-KI KD (ours)
79.42
79.23
79.30
79.42
79.34
79.19
79.51
76.46
B.2 Effect of Inter- and Inter-angles in Ensemble Diversity, Expected Loss, and Accuracy
In this work, claim that angular diversity directly leads improved knowledge distillation, based on the theoretical analysis presented in the main paper. This effect is achieved by (1) increasing ensemble diversity and (2) consequently lowering the upper bound of the ensemble expected loss. To support this key claim, we report the KD performance, average inter- and intra-angle statistics, measured ensemble diversity, and ensemble expected loss between ensembled logits and ground- truth label in Table 15, As our angular diversity objectives are added, both the mean inter- inter-angle and intra-angle increase. This, in turn, raises the measured ensemble diversity and lowers the measured ensemble's 's expected loss. Consequently, the KD accuracy improves.
We further compare these results with those of the state-of-the-an art KD augmentation method, TeKAP, in Table 16. Notably, TeKAP achieves a higher intra-angle, but all other metrics are inferior to ours. Although TeKAP's random perturbation can boost intra-angle by scattering views randomly around the teacher, it cannot safely scale up noise to increase inter-angle without risking drift into non-target classes. As result, its inter-angle, ensemble diversity, and expected-loss reductions all fall short of ours. In contrast, our method simultaneously increases both inter- and intra-angle, tightens the ensemble's 's expected-loss bound, and improves KD accuracy, demonstrating a more robust and effective augmentation strategy.
Table 15: Effect of Inter- and Intra-angles in Ensemble Diversity and Accuracy with each component. The reported inter- and intra-angles are averaged value over all augmented views.
Constrained Inter-angle Diversity Loss
Intra-angle Diversity Loss
Student KD Acc.
Angles
Ensemble Diversity
Ensemble Loss
Inter
Intra
75.46
16.
11.23
11.40
11,522
0.810
76.28
14.09"
16.85
11.617
0.809
76.46
20.86
25.38
11.633
0.806
23

---

Table 16: Effect Inter- and Intra-angles in Ensemble Diversity and Accuracy compared to
TeKAP. The reported inter- and intra- intra-angles are averaged value over all augmented views.
Student KD Acc.
Angles
Ensemble Diversity
Ensemble Loss +
Methods
Inter
Intra
TcKAP [20]
75.98
4.35
35.64
6.65
0.853
Angular-KD (ours)
76,46
20.86
25.38
11.633
0.806
B.3 Correlation Matrix between Augmented Views
Fig. 4 shows a comparison of the pairwise co- sine similarity matrices (i.e., correlation ma- trices) between our angularly augmented log- its and TeKAP's perturbed logits. For clarity, both methods use the same number of augmen- tations (N = 5). For Angular-KD (Fig. 4a), similarities range widely (e.g.. 0.786 between view 1 and view 5), revealing substantial di- versity. In contrast, TeKAP's randomly per- turbed logits (Fig. 4b) are almost identical, with all pairwise similarities around 0.997. These results demonstrate our angular diversification produces more varied soft logits, providing richer, non-redundant supervision for the stu- dent, thereby improving value for distillation.
1.000
0.901
0.958
0.990
0. 0.785
V.
1 1.000
0.9997
0.997
0.997
0.987
V43
991
1.000
0.987
0.972
0.042
V/3
0. 0.997
1.000
0.997
0.997
0.997
0.987
1,000
V/
0.997
1.000
0.987
0.997
1.000
0.706
0.842
00,872
1.000
V.
0:997
0.999
0.997
0.997
1.000
V/
V/2
V.5
V4
0/5
V/
V/
1/
V/3
(a) Ours
(b) TeKAP
Figure 4: Correlation Matrix between augmented views (Vi,..., V/5) compared to TeKAP.
C Additional Implementation Details
ImageNet Experiments. We elaborate on the dataset and implementation details for ImageNet experiments, reported Table For the dataset, ImageNet [10] is a large-scale classification dataset with 1,000 categories, approximately 1.28 million training and 50,000 validation images, each of size 224x224. Its large-scale and high-diversity make ImageNet an ideal testbed for evaluating the scalability and robustness of knowledge distillation methods. For the implementation details, we first begin with a 10-epoch warm-up training, where only the view-augmentation heads are trained for stability. Then, like previous KD approaches [20, 47]. we follow the standard Py training schedule but extend it by 20 additional epochs, decay the learning rate by 0.1x at epochs 40, 70, and 100, and use a batch size of 512.
Play-and-Play Experiments. To integrate our Angular-KD into the SoTA KD methods, i.e. DKD [62] and MLKD [22], we follow their configurations and details of the official code of the corresponding methods. We also perform an initial warm-up training exclusively on the view- augmentation heads during 30 epochs. For the multi-teacher approach, TAKD, we follow a two- stage procedure: first, we train the primary teacher network and distill its knowledge into a single assistant network with augmentations; next, we distill the assistant's knowledge to the student with augmentations.
Imbalanced CIFAR-100 Experiments. To construct the imbalanced CIFAR-100 dataset, we employ the same class selection protocol as TeK [20]. In detail, we select 43 classes as imbalanced classes among a total of 100 classes. For each of these classes, we choose the first 50 training samples in sequence (without shuffling) for reprodueibility and discard the rest. The remaining 57 classes keep their full set of 500 training images. This yields imbalanced distribution for evaluating KD methods under severe class imbalance.
Few-shot Experiments. create the limited-data scenarios, we randomly sample 25%, 50% and 75% of CIFAR- 100 training data. For reproducibility, we select the first 25%, 50%, and 75% samples in sequence (without shuffling). This deterministic selection ensures reproducibility while simulating sho-shot learning scenarios.
24

---

D Data Augmentation v.s. Our View Augmentation
Traditional data augmentation methods are commonly used generate positive views. To evaluate our approach, which generates views via linear pathways, we compare it against the conventional augmentation technique. For experiments, we apply standard data augmentations (random flip, color jiterring, and random rotation) to generate multiple views and ensemble them for student distillation, similar to our view augmentations. Table 17 shows the CIFAR-I00 accuracy for our view-augmented logits v.s. the augmeeted logits, where ResNet32 x teacher and a ResNet8 x4 student are used. We observe substantial performance drop when multiple views are generated via data augmentation. To analyze this degradation, we check the accuracy of each augmented view in Table 18. Each augmented view by data augmentation shows significantly lower accuracy due to the noise caused by the data perturbation, leading to decline in overall performance. This highlights the effectiveness of our approach in producing consistent and diverse representations without computational overhead.
Table 17: KD Performance.
Table 18: Accuracy for each Augmented View.
Methods
Acc.
Augmented Views
Teacher Acc.
Methods
Baseline
75.46
2
3
4
5
Data aug.
66.91
Data aug.
74.52
74.62
74.56
74.41
74.30
79.42
w/ View aug. 76.46
View aug.
79.23
79.30
79.42
79.34
79.19
Furthermore, our view-augmentation provides two major benefits over the data augmetnation: I) Efficiency (1 forward pass vs. N forward passes); our method produces multi-views with a single forward pass, whereas data augmentation typically requires forward passes for N augmented inputs, leading to significantly higher computational cost, and 2) Evenly dispersed representations; by explicitly maximizing the angular diversity in the representation space, our method ensures that augmented views are evenly dispersed and exhibit minimal redundancy. In contrast, heuristic data augmentation influences embeddings only indirectly, often resulting in redundant representations or ones that drift away from the teacher's original view, making difficult to achieve consistent and balanced dispersion in the representation space.
E
Additional Theoretical Analysis
In this section, we supply the full derivations to support our theoretical claims in Sec. 3. Theoretical Analysis of the main manuscript. We begin by deriving the upper bound of the expected ensemble loss using the ensemble diversity measure [38] and then explicitly connect these our inter- and intra-angle diversity objectives.
Following the analysis in [38], we first show that the expected loss E(x.y)>D[((() of the ensemble logit E (i.e., o for all ensemble members {Z. J. over the dataset D is bounded follows:
E [ < Exxyy> E [(Z)) K Diineratte (f No)
(13)
where L. is a KL divergence loss and K > O is a constant. Teeennrrliied is the same as the ensemble diversity D that appears in the main paper (the reason why we call this as generalized diversity are present in Sec. I) This bound implies that increasing the generalized ensemble diversity ID Deeneralizedd lowers the upper bound of the expected ensemble loss. We also show that the generalized ensemble diversity can be expressed in terms of the total variance of logits across classes (see Sec. E. for the full formulation):
Beeneallizd ( (Z.) Ex
V.
Z:
]]
(14)
i=
max Z
Here, we define the total variance by V over all C' classes as V(z] Varioal [z] = >> where
Var[2*]
j=m
Furthermore, we show that this formulation is approximately proportional to the expected total variance of the logits (see Sec. E.2 for detailed derivation):
DGeneralizedl ( (Z, J O))
B(xx))DD
1= max Z ]] xc E ((xx)) VVN
Z,
(V. [
(15)
25

---

The original ensemble diversity metric in [38] measures diversity only on logit 2 for the ground-truth class We generalize this to include all C logits Z; = [=]..... 2f"], computing diversity across every class logit. This full-spectrum diversity measure appears in both our main paper and supplemental This implies that increasing expected total variance raises the generalized ensemble diversity, resulting decrease of the upper bound the expected ensemble loss.
Based on the expected total variance of logits in Eq. (36), we derive two mathematically equivalent formulations of ensemble diversity in terms of angular similarity. Specifically, show that the ensemble diversity measure Deencrllizd (-) can be expressed in the following two equivalent forms: Inter-angle form (Eq. (16a)) uses the pairwise cosine similarity si (defined in Eq. (3) inter- angle similarity) between each pair of augmented logits Z; 2) Intra-angle form (Eq. (16b)) uses the cosine similarity stj (defined Eq. (4) as intra-angle similarity) between the difference vectors
A A Z" Z," as follows:
(
E Z2
|Z/I 12, |=]]]
(16a)
Deesentiied ((2))==)
VN i=0
Zz max Z.
Inter-angle Formulation
A T-4
|| (
(16h)
Intra-ungle Formulation
where the detailed derivation is provided in Sec. E.3:
E.1 Generalized Ensemble Diversity beyond a Single Class
proofs. Compared to the original single-class version [38], the generalized metric introduces constant scaling K because it sums over C' classes instead of one target class. Below, we detail the derivation of this generalized ensemble diversity.
Original Ensemble Diversity. The classic ensemble diversity [38] is defined as follows:
Doriginal ( (Z: [xxxy)>
Var: max
where 2 [Z.]%,
(17)
2
and [Z.]a the logit the target class y the vector Z, among C' classes. This diversity focuses solely on the ground-truth class logit z.
Generalized Ensemble Diversity.
We extend this to capture variability across all C' classes as:
DeGneralized [Z ( = Exxy>D
Z: max Z;
[]
(18)
where max denotes the maximal logit value over class indices j I,.... C. This formulation captures diversity across every output logit rather than focusing solely on the target class. This full-spectrum diversity metric underpins our thcoretical analysis in both the main manuscript and the supplementary. The complete derivation follows below.
Detailed derivation. The original diversity metric [38] is derived from the cross-entropy loss in their paper, so it only measures variability the ground-trute -tuth class logit. In contrast, our generalized diversity is based on the KL-divergence loss between softmax distributions, capturing variability across all class logits rather than just the target class. For the derivation, we begin with the Taylor expansion of the logarithmic function log(æ) around a fixed point > 0, incorporating the second-order remainder term:
log = log a + (x - a)
(x - a)), & (x, 2ga
(19)
Here, Ç lies between æ and a. This expansion will allow us to approximate log terms arising in the KL--divergence expressions. Applying Eq. (19) with x : 3 centering at E. [2] EN
[=]
>
y
for arbitrary ensemble member class index j, ground-truth label Y [y .... with yd
26

---

and ensembling ol with uniform weights over N ensemble units for specific j-th category
logit, we have:
R.V. - +
log( y
= log(
(2)
E.
ol
( (2) - E. ol
(20)
y
EN
oo
By taking the expectation on both sides and multiplying through by y², we obtain:
y E ool log( ( ]]
E.N ol
- 3d log(
EN
( E =
(21)
25 25220T
Rearranging terms,
y2
y log(
= oo
log(
-EN
ENN
(z)
EN oo
(22)
B.N. [
[=]]]
2
2/
Next, observe that since & E (x, a), it follows that & max. max; derive the inequality:
Using this bound, we
y3 (2 } log(
y
y log
S E,
E.
(2f Eo
(23)
E
2/
2(max; 2f)²
Summing over all classes j E (1.....(C, C). we obtain:
yd
yd ) y log(
y
log
N.N
E
2 mmax, 2² (2) EN o[ (24)
EN
2
j=1
Recalling the definition of KL-divergence KL(YY be rewritten as:
j=1 y² log vi the above incquality can z!
ys
KL(Y|E (Z)) S E [KL(Y [Z)] - E.N
(o) E o=?...
(25)
2(mx 2f
Taking expectation over data distribution (x, ~ on both sides leads to:
(x KL EN [
y
N
[z)]
Exx,y)
ENN
(2)
(26)
(max z
j=1
We now recall the definition of the generalized diversity measure:
Deeneralized ((Z, ((Z,),o)) = E Exxxy)
V.N 1=0
Z:
]]
Ex
Var,
]]
(27)
max Z.
max
j=1
1 (max: 2/132 (2f
(xy))>
E.N.
E.N -
(28)
Since nd > O. the last term in Eq. (26) corresponds to the summation of the generalied diversity measure DeGeerraizzed (Z, )o) in Eq. (27) over all class indices Vj E .1....,C), (we insert this constant sum into constant factor K, resulting in Doriginal = K Deeneratizaa))) Therefore, maximizing the diversity DGenenalizea ( increases the negative term on the right-hand side of Eq. (26), which tightens the upper bound and effectively reduces the expected KI.-divergence loss of the ensemble.
27

---

Relations between Expected Ensemble Loss and Generalized Diversity. diversity work [38] shows that the expected ensemble loss of the ensemble logit E = [2] is bounded
The original ensemble
by:
(x,y)> ((E [.I) < (x.)))
E [c().)] Driginal ( (Z, N /i=o);
(29)
where C is a cross-entropy loss. Because our generalized diversity Denerllied differs from the original one by constant factor K, we can re-write this bound as:
E(x,y)=D ( (Z.)) - ( [(())]] K Generalie ( (2, (Z)))))
(30)
which is Eq. (6) in the main paper. implies that increasing the ensemble diversity DD tightens the upper bound on the ensemble's expected loss, thereby delivering strong supervisory signal to the student and improving KD performance, as demonstrated in [34]. Empirically, Table 16 and Table 15 demonstrate that higher ensemble diversity indeed correlates with lower measured expected ensemble loss, confirming this theoretical relationship.
E.2 Ensemble Diversity is Proportional to Logit Variance
In Eq. (36), we demonstrate that the generalized ensemble diversity is function of the total logit variance. Below, we offer the detailed derivation, showing that the generalized ensemble diversity is scaled total logit variance. We start from the definition of the generalized diversity measure:
Z
Deenrallizd (Z.) Exxxy>D
V.N ==0
(31)
max Ze
Var, [=]]]
1] (xy) 2/
-xxx))>D
Var
(32)
max;
(max; 2
To analyze the relationship between diversity and total logit variance, we model af across all ensemble units {i]--- as a Gaussian variable with mean μı o and variance o². Since z/ represents a logit probability for a specific category j. is bounded above by 1. Based on this assumption, approximate the maximum value over individual unit as:
max of R min(E + 1)
(33)
Which approximates the maximum values of 2 across by capturing deviations above the mean under the Gaussian assumption, while the minimum operator ensures the value remains bounded by 1. Thus, we can rewrite the generalized diversity measure as:
Var,
D D(IZ) = E (x
B(xxy)
(34)
maxx =3)²
C koo,, 1)21
To show that diversity increases monotonically with variance, it suffices to prove that the following function is monotonically increasing a.
f(6) = min(p + ko, 1)2 where p [0, 1]. 120 0.
(35)
We consider two cases:
Case 1: If min(p + ko, 1) = 1, then f(o) = 0²,
which is clearly increasing in Ø.
Case 2: If mmin(p- + ko, 1) = p + ko, then:
2po
f(o) k f'(a)
(p + ko)3 2
Since p E 10.1 2 and the denominator always non-negative, (p + ko)s 2 0, we have (a) 2 0. Therefore, in both cases, f(a) is non-decreasing function of a. Thus, increasing the thereby supporting our claim that
variance o² results an increased diversity measure ID ((Z,) 1N ensemble diversity is proportional to logit variance.
28

---

E.3 Two Equivalent Forms of Ensemble Diversity
In this section, we show that the generalized ensemble diversity metric [eGerraaiied (-) can be written in two mathematically equivalent forms: 1) Inter-angle form (Eq. (16a)) and 2) Intra-angle form (Eq. (16b)). We start from the total variance of all ensemble logits [ for two form derivations, since the ensemble diversity is proportional to the total logit variance, as follows and proved in See E.2:
Generalized ((2))/))) = (xxxy>D
V.N i=
Z;
]]
x E
((xx))D
V.V. 1=0
Z
].
(36)
max Z.
derivation
Inter-angle Formulation of Ensemble Diversity. To derive the inter-angle form (Eq. (16a)) of diversity from the total logit variance, Le.,
focus solely the augmented logits (Z, J.V omitting the teacher (i = 0) since our angular objectives apply to these views. We then begin with a definition of variance derive the inter-angle form (Eq. (16a) from the logit variance, i.e., V. Z
N
Zi
E
N 1=1
7Z
|
|2.112
(37)
1 |Z
Σ 11 N
(38)
N
N
N
N
|2
Z, Zj
(39)
N
N2 (=1 }=1
N Σ
|
Zt////////coo(//2 2₁)
(40)
N
N2
j=1
1
12
||
|ZA
(41)
N
N2
1=1
N
Z
Z 14,
11
(42)
angle Formulation Eq (16a)
where st] = cos (Z; Z,). cos(u, v)= u-v u-v/||ulllvvl]] and V denotes a total variance over all C' classes. The inter-angle formulation of the ensemble diversity makes it clear that reducing each pairwise similarity st increases the total variance of logits and therefore raises the ensemble diversity. Our constrained inter-angle diversity loss explicitly minimizes set so it naturally enhances the ensemble diversity DD. In Table 15, we empirically observe that adding the inter-angle diversity loss both raises the average inter-angle (i.e., lowers stt) and increases the ensemble diversity measure.
Intra-angle Formulation of Ensemble Diversity. We can express VAN [Z.] depending on s using the property that > A = 0 (we assume that the average of augmented logits can approximate the original teacher z N Zi). We also start from the definition of variance for the derivation of inter-angle formulation (Eq. (16b)):
29

---

V.V
[Z
= E
|
| i=
|1
(43)
1
1
12:11
Z
(44)
N
N 1
N
N
N
Σ
|2
Z
Z
(45)
N
N 1= 1=1
1=1
|2/12
2
Σ
Z
Zj
+
1
Z Zz
(46)
N
N² 1=
N2
j=1
1
1
IZ
1/
(:
Z,)
(47)
N
N
i=1
1
A
11
(48)
N
M
N 3
N
N
AT
A
A
43-4
(49)
1=1
1=1
N
N
A3-4
(50)
AJ-
(51)
N
N N
N
A4 [1A
(52)
[A
! +
(53)
Intra-angle Formulation Eq (16b)
Since lowering each st raises the variance of the logits, our intra-angle diversity loss, which minimizes si). directly increases both variance and, consequently, the ensemble diversity. Table 15 empirically confirms this: adding intra-angle diversity loss not only boosts the mean intra-angle between views but also elevates the measured ensemble diversity.
We validate our two core theoretical claims: 1) our angular-diversity objectives boost ensemble diversity, as derived in Sec. E.3, and 2) increased diversity tightens the ensemble loss bound, as proven in Sec. E. Empirically, Table 15 shows that our inter- and intra-angle diversity losses both raise the measured ensemble diversity and lowers the measured ensemble expected loss, directly translating into stronger KD performance.
E.4 Better Teacher, Better Student
In this section, we leverage prior theoretical results [34, 41, 9, 16] to connect improvements in the teacher's expected loss (in our case, the ensemble's expected loss) directly to the reductions in the student's is expected loss. Specifically, in [34], they derive the student's generalization error (measured as the discrepancy between its KD loss (LxD) and the KI loss (Lxx.) to a Bayes-optimal soft label Y can be bounded by two terms: 1) the variance of the KD loss across the data, and 2) the teacher's approximation error to the ideal soft label (captured by an MSE term). Formally,
E(xyy) (Zx 2²) LKL YY, )
VONYYDD [ZD 2 + -xx-(x [MMS (Z" Y)]
S
Stadent Generalization Error (KD less GT loss)
Variance of Distillation Loss
Teacher Approximate Loss
where Z and Z are the output logits of the teacher and student, respectively, Y soft ideal ground-truth label (the Bayes-optimal target distribution), and k is positive constant value.
30

---

Employing knowledge distillation could reduce the first term by providing the student with consistent teacher signal, which lowers the variability of the KD loss across the dataset D. The second term is decreased by using an ensemble: generating diverse views (i.e. ensemble) produces a closer approximation to the ideal soft target (the Bayes-optimal target distribution), reducing the teacher's approximation error. Together, these effects tighten the upper bound on the student's generalization error, confirming that stronger teacher signals (via our angular-Ki RD augmentations) directly improve the student's s performance.
31


Reviews:
[{"review_id": "tIY8Ef6PGc", "reviewer": "Reviewer_tUi1", "summary": "This study proposes inter-angle and intra-angle losses to derive high-diversity perspectives from a single teacher model. Both losses operate to train linear layers to expand the diversity of outputs for the same sample in the feature and logit domains. Through this, the teacher model applies transfer learning to guide the student's learning direction by delivering multi-view supervision for the same data.", "questions": "1. Why does the proposed method induce better view-augmentation compared to traditional data augmentation techniques?\n2. Can the diversity achieved through view-augmentation be quantitatively demonstrated?\n3. Which dataset was used to derive the results presented in Table 7?\n4. Given that the diversity of multi-views may vary across classes, can this be accounted for in the method?", "limitations": "1. The novelty of the proposed method is somewhat lacking.\n2. Additional analytical experiments are insufficient. Comparisons of representation variants based on diversity margins or the number of views ($N$) are needed.\n3. The number of training samples in CIFAR-100 should be reviewed for accuracy.\n4. The subcaptions (a) and (b) in Table 9 are meaningless and lack context.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n1. The writing is exceptionally clear and well-structured, making the content highly accessible and easy to follow.\n2. The approach expands the student model's representation coverage by generating multi-view (akin to negative views) from a single teacher model for the same sample.\n3. By pushing the similarity between feature and vector spaces, the method secures flexibility in diversity expansion.\n4. The mathematical approach of the proposed method is clearly presented and well-organized for easy comprehension.\n\n$\\textbf{Weaknesses}$\n1. The novelty of the proposed method is somewhat limited, as it applies the concept of negative pairs from existing contrastive learning to multi-representations of the same sample.\n2. The accuracy improvement of the proposed method is not convincingly superior compared to existing methods.", "was_revised": true, "final_justification": "Thank you for your thorough and considerate responses to the reviewer’s comments. I believe this manuscript holds sufficient scholarly merit to be recognized for its research value.", "rebuttal": "Dear reviewer tUi1,\n \nThank you for your time and efforts in reviewing our manuscript. We appreciate your recognition of our paper’s strengths, particularly its clear writing, well-structured mathematical approach. Below, we would like to provide clarifications and address the concerns and questions you raised.\n\n---\n\n**[W1, L1] Limited Novelty on mimicking the concept of negative pairs from contrastive learning.**\n\nWe would like to clarify that the novelty of our approach lies in promoting a broadened coverage of augmented views via intra- and inter-angular diversity maximization. Although the inter-angle loss, particularly the constraint term in Eq. (3), may superficially resemble conventional contrastive learning, our core formulation, especially the diversity term, is fundamentally different. Rather than treating each view as a negative pair and pulling it indiscriminately from the teacher view, we encourage angular separation among all pairs of multi-views, while preserving their alignment with the teacher. This promotes diversity without sacrificing representational consistency. Intra-angle diversity loss further ensures an even distribution across views around the teacher output. We believe that this fundamental distinction sets our formulation and ideas apart from conventional contrastive learning objectives, which focus on negative separation.\n\nMoreover, our work presents two novel findings: 1) Angularly diversified knowledge from a single teacher can serve as, and even outperform multi-teacher networks (Table 5), and 2) Higher angular diversity theoretically and empirically improves distillation effectiveness by increasing ensemble diversity and lowering ensemble loss (Tables 15 and 16).\n\nTo the best of our knowledge, these analyses on the effect of angular diversity are not covered in existing knowledge distillation and ensemble learning studies, underscoring the value of our contributions.\n\n\n---\n\n**[W2] Minimal accuracy improvement.**\n\nWe believe that our improvements are substantial and meaningful, particularly given the strength of the underlying baselines. To better quantify this, we compare the average improvements of our method and the current SoTA augmentation method, TeKAP, relative to the non-augmented baselines across all 18 settings in Table 1. As shown in the “Average Gains” column of the following table, our augmentation method yields an average gain of +1.02pp, whereas TeKAP achieves only +0.36pp from baseline methods.\n\n\n| Method          | Average Gains |\n| --------------- |:------------:| \n| TeKAP      |       +0.36   |      \n| Ours           |     **+1.02**    | \n\n\nOur average gain is 2.86 times larger than TeKAP (+1.02pp vs. +0.36pp). We believe this consistent performance gain across diverse settings demonstrates the practical effectiveness of our approach.\n\n---\n\n**[Q1] Why is our view augmentation better than the data augmentation?**\n\nThanks for your insightful question. We believe that our view-augmentation provides two major benefits over the data augmentation.\n1) Efficiency (1 forward pass vs. N forward passes); our method produces multi-views with a single forward pass, whereas data augmentation typically requires N forward passes for N augmented inputs, leading to significantly higher computational cost.\n2) Evenly dispersed representations; by explicitly maximizing the angular diversity in the representation space, our method ensures that augmented views are evenly dispersed and exhibit minimal redundancy.  In contrast, heuristic data augmentation influences embeddings only indirectly, often resulting in redundant representations or ones that drift away from the teacher’s original view, making it difficult to achieve consistent and balanced dispersion in the representation space.\n\nTo further investigate this, we applied standard data augmentations (random flip, color jitter, and random rotation) to generate multiple views and ensemble them for student distillation, similar to our view augmentations.\nThe table below shows the CIFAR-100 accuracy for our view-augmented logits vs. the data-augmented logits, where a ResNet-32x4 teacher and a ResNet-8x4 student are used.\n\n\n| Method          | Top-1 Accuracy |\n| --------------- |:------------:| \n| Baseline     |       75.46   |    \n| w/ Data augmentation     |     66.91     |   \n| w/ View augmentation (ours)           |     **76.46**    | \n \nWe observe a substantial performance drop when multiple views are generated via data augmentation. \nTo analyze this degradation, we check the accuracy of each augmented view in the table below. \n\n|  View index | 0 | 1  | 2  | 3 | 4 | Teacher acc\n| --------------- |:------------:|  :------------:|  :------------:|  :------------:|  :------------:| :------------:| \n| Data augmentation     |     74.52     |   74.62 | 74.56 | 74.41 |  74.30 | 79.42 |\n| View augmentation (ours)           |     79.23    | 79.30 | 79.42 | 79.34  | 79.19 | 79.42 |\n\nEach augmented view by data augmentation shows significantly lower accuracy due to the noise caused by the data perturbation, leading the overall performance to decline.\n\nWe believe that this highlights the effectiveness of our approach in producing consistent and diverse representations without computational overhead.\nWe will include these results and a detailed comparison with the data‑augmentation approach in the final version if accepted.\n\n\n---\n\n**[Q2] How can the diversity introduced by view augmentation be measured quantitatively?**\n\nYes, we quantify diversity using the ensemble-diversity metric [35]. As shown in the supplementary materials (Table 16), our method achieves 11.633, whereas TeKAP obtains 6.650, demonstrating a substantially higher degree of diversity.\nTable 6 further shows that each of our proposed loss functions contributes to increasing this diversity metric.\nFinally, in Figure 4 of the supplementary, we visualize the correlation matrix (i.e., similarity matrix) between augmented views compared to TeKAP; our approach yields significantly lower similarity between views than TeKAP, further validating that our methods produce more diverse augmented logits.\n\n\n---\n\n**[Q3] Which dataset was used for Table 7?**\n\nThank you for catching this omission.\nTable 7 presents results on CIFAR-100.\nWe will update the manuscript to indicate this clearly.\n\n---\n\n**[Q4] Can our method control the diversity at the class level?**\n\nThank you for your valuable suggestion. Yes. We do so as follows: \n\nFirst, we compute per-class variance to quantify each class's diversity by measuring the variance over all samples of each class in the table below. As you mentioned, we observe that the diversity varies across classes.\n\n| Class index | 0 | 1 | 2 | ... | 98 | 99 |\n| --------------- |:------------:|  :------------:|  :------------:|  :------------:|  :------------:| :------------:| \n| Variance      |    86.96      |   126.83   | 214.36 | ... | 190.20 | 212.10 |\n\n\nThen, we normalize these variances via the mean value over those of all classes to obtain class-wise weights. Finally, we apply these weights to the inter-angle and intra-angle diversity losses. Formally,\n \n\n$$L^{aug}=\\frac{1}{C}\\sum_{c=1}^{C}w_c(L^{inter}_c + L^{intra}_c) + L^{GT},$$\n\nwhere C is the number of classes, L^{inter}\\_{c} is the constrained inter-angle diversity loss on the sample x with class c, L^{intra}\\_{c} is the intra-angle diversity loss on the sample x with class c, and L^{GT} is the ground-truth loss.\nWith this implementation, classes with larger diversity receive larger loss weights, further promoting their views to spread (reducing overlapping). \nThe table below reports the performance of this class-wise re-weighting on CIFAR-100.\n\n| Method | Top-1 Accuracy |\n| --------------- |:------------:| \n| KD + CRD w/o aug | 75.46 | \n|w/ Class-wise re-weighting | 76.04 | \n| w/ Ours | 76.46 |\n\n\nWe observe that this yields no significant improvement in performance. \nHowever, since we have confirmed that diversity indeed varies across classes, we will leverage this insight in future work to explore its potential.\n\n---\n\n**[L2] Additional analytical experiments based on diversity margins or the number of views.**\n\nThank you for the suggestion. As noted in the main submission, we already report Top-1 accuracy ablations for the angular margin (Table 9b) and the number of views (Table 7), but we agree that further analysis would be valuable for our paper. \nBelow, we present extended ablations over additional values of angular margin and the number of views.\n\n| # Views | 1 | 2| 3| 4| 5 | 6| 7 | 8 | 9 |\n| --------------- |:------------:|:------------:|:------------:|:------------:|:------------:|:------------:|:------------:| :------------:| :------------:|\n| Top-1 accuracy  | 75.87 | 75.85  | 76.25 | 76.44 | **76.46** | 76.37 | 76.26 | 76.20 | 76.10 | \n\n| Angular Margin 𝛾 | 0.1 | 0.15 | 0.2  | 0.25 | 0.3 | \n| --------------- |:------------:| :------------:| :------------:| :------------:| :------------:|\n| Top-1 accuracy | 76.34 | 76.29 | **76.46** | 76.20 | 76.31 | \n\nWe will include these additional results in the final version, if accepted.\n\n---\n\n**[L3] Review of CIFAR-100 training sample count.**\n\nAs described in Section 4.1 (line 210), CIFAR-100 contains 50,000 training and 10,000 validation images across 100 classes. We will revise the manuscript to ensure this is more clearly stated where CIFAR-100 is mentioned.\n\n\n---\n\n**[L4] Meaningless subsections (a) and (b) in Table 9.**\n\nThank you for pointing this out. We will update subsections in Table 9 to: “(a) Effect of diversity and constraint terms” and “(b) Effect of angular margin 𝛾” to ensure each subtable is clearly presented."}, {"review_id": "ANM6bnXwv0", "reviewer": "Reviewer_TLMU", "summary": "This paper proposes the Angular-KD framework, which generates diverse multi-view knowledge by attaching multiple lightweight linear branches to a single teacher model. It introduces a constrained inter-angle diversity loss and an intra-angle diversity loss. Experimental results show that Angular-KD outperforms existing knowledge augmentation methods across multiple knowledge distillation benchmarks and can be integrated as a plug-and-play module with other KD frameworks.", "questions": "Although the proposed method generates diverse views by attaching multiple branches, it still relies on a single teacher model. This means that if the teacher model itself has biases or limitations, these biases may be passed on to the student model. In contrast, multi-teacher methods can provide a broader perspective through multiple independently trained teacher models. In the comparative experiments, the paper compares the performance with multi-teacher distillation methods, showing that this method outperforms them. This result raises the question of whether this method remains leading when the complexity of both the teacher and student models increases. The paper lacks test results on larger-scale models.", "limitations": "Yes. The authors acknowledge that this method relies on the teacher's representation and thus cannot introduce truly novel semantics beyond the teacher's knowledge. While it outperforms multi-teacher methods, it lacks the ability to capture independent and complementary signals from multiple teachers. Relying on a single teacher may propagate teacher biases, which could have potential implications in privacy-sensitive or surveillance-related applications.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Streangth:\n\n1.\tThe method presents a novel angle for KD augment and is compatible with other knowledge distillation frameworks, designed as a plug-and-play solution.\n\n2.\tExperiments were conducted on multiple datasets and in imbalanced scenarios, achieving good performance.\n\n3.\tThe paper provides theoretical analysis and implementation code, enhancing the credibility and practicality of the research.\n\nWeakness:\n\n1.\tThe comparison is limited to only one data augmentation method, lacking comparisons with other advanced methods.\n\n2.\tThe comparison with multi-teacher methods is limited and not up-to-date, such as TAKD published in 2023, which may not be the latest multi-teacher method.\n\n3.\tThe visualization analysis in this paper is relatively limited, only providing t-SNE plots to show the distribution of the teacher and augmented views. It fails to further demonstrate the model's performance across different perspectives through other visualization methods (e.g., feature space distribution or decision boundaries).", "was_revised": true, "final_justification": "The author responses have addressed most of my concerns. However, the concept of enforcing angular diversity for optimization has already been explored in other works. Therefore, I have decided to retain my score as borderline accept.", "rebuttal": "Dear reviewer TLMU,\n\nThank you for your time and efforts in reviewing our manuscript. We appreciate your recognition of our paper’s strengths, particularly our novel and compatible methods, comprehensive experiments on multiple datasets, and our theoretical analysis and codes. Below, we would like to provide clarifications and address the concerns and questions you raised.\n\n---\n\n**[W1] The comparison is limited to only one data augmentation method, lacking comparisons with other advanced methods.**\n\nThank you for the reviewer’s valuable comments. We understand that limiting the comparison to only TeKAP may not sufficiently demonstrate the general effectiveness of our approach across diverse knowledge distillation (KD) paradigms.\n\nTo address this concern, we emphasize that our method is **framework-agnostic** and designed to be **plug-and-play**, making it compatible with a wide range of KD frameworks. In our experiments, we applied our method on top of several representative and diverse KD frameworks, including DKD [57], MLKD [20] and TAKD [33] in addition to conventional KD [17] and CRD [45], and observed consistent improvements over their baseline performances (see Table 2 and Table 11).\n\nAdditionally, the table below shows further plug-and-play results from applying our methods to Review-KD [6], in comparison with TeKAP. Once again, our method achieves consistent improvements, and this suggests that our approach can complement, rather than replace, existing KD methods.\n\n\n$$\n\\\\begin{array}{lcc}\n\\\\textbf{Teacher} & \\\\text{ResNet-32}\\\\times\\\\text{4} & \\\\text{WideResNet-40-2} & \\\\text{ResNet-32}\\\\times\\\\text{4}    \\\\\\\\\n\\\\textbf{Student} & \\\\text{ResNet-8}\\\\times\\\\text{4} & \\\\text{WideResNet-40-1}  & \\\\text{WideResNet-16-2} \\\\\\\\ \\\\hline\n\\\\textcolor{gray}{\\\\text{Review-KD w/o aug}} & \\\\textcolor{gray}{\\\\text{75.63}} & \\\\textcolor{gray}{\\\\text{75.09}} & \\\\textcolor{gray}{\\\\text{76.11}} \\\\\\\\ \n\\\\text{w/ TeKAP} & 75.32 &  75.15  &76.36     \\\\\\\\\n\\\\text{w/ Angular-KD (ours)} & \\\\mathbf{75.78} & \\\\mathbf{75.45} & \\\\mathbf{76.40}  \\\\\\\\\n\\\\end{array}\n$$\n\nWe will revise the manuscript to clarify this point and better highlight the broad compatibility and effectiveness of our approach across various KD frameworks.\n\n---\n\n\n**[W2] Compared multi-teacher methods are not up-to-date.**\n\nWe acknowledge that the compared multi-teacher methods (TAKD [33], published in 2023; and DGKD [40], published in 2021) may not appear very recent. However, to the best of our knowledge, they remain the latest and SoTA multi-teacher KD methods available to date. If any newer approaches in this category emerge, we will incorporate them into our revision accordingly.\n\n---\n\n**[W3] Additional Visualization.**\n\nThank you for your helpful suggestion. \nWe analyzed two visualizations for our augmented views and the original view, compared to those of TeKAP: 1) UMAP embeddings, and 2) the decision boundaries.\nAlthough we cannot add figures during rebuttal due to the NeurIPS 2025 rebuttal policy, we observe the following: 1) In UMAP, TeKAP's augmented logits substantially overlap with the teacher's logit, whereas our method yields more dispersed views, aligning with TeKAP's lower ensemble diversity scores (Table 16, supplementary); and 2) In decision boundary plots, our augmented views produce clearly more discriminative boundaries, as our higher accuracy of each view than TeKAP's each view (Table 14, supplementary).\n\n\nWe will include additional visualizations of UMAP and decision boundaries, along with a detailed implementation process, in the final version, if accepted.\n\n---\n\n**[Q1] Results on larger-scale models.**\n\nWe appreciate the reviewer’s thoughtful recommendation. To assess scalability, in the following table, we conduct experiments with larger teacher and student configurations: 1) a ResNet-110 teacher and a ResNet-32 student, and 2) a ResNet-110 teacher and a ResNet-20 student.\n\n\n$$\n\\\\begin{array}{lcc}\n\\\\textbf{Teacher} & \\\\text{ResNet-110} & \\\\text{ResNet-110} \\\\\\\\\n\\\\textbf{Student} & \\\\text{ResNet-32} & \\\\text{ResNet-20} \\\\\\\\ \\\\hline\n\\\\textcolor{gray}{\\\\text{Na\\\\'ive KD and CRD w/o aug}} & \\\\textcolor{gray}{\\\\text{73.75}} & \\\\textcolor{gray}{\\\\text{71.56}} \\\\\\\\\n\\\\text{w/ TeKAP} & 73.86 & 72.02     \\\\\\\\\n\\\\text{w/ Angular-KD (ours)} & \\\\mathbf{74.89} & \\\\mathbf{72.52}  \\\\\\\\\n\\\\end{array}\n$$\n\nAcross all settings, our methods yield consistent improvements and outperform TeKAP (a SoTA knowledge augmentation method), indicating that our approach remains effective at larger model scales. We will include these results in the final version.\n\n\nFurthermore, in Table 14 of the supplementary, integrating our approach into the multi-teacher framework (TAKD [33]), which has high complexity due to the multi-teacher network, improves distillation performance, underscoring the robustness of our method under high-complexity methods."}, {"review_id": "bymu4aeggh", "reviewer": "Reviewer_SW6m", "summary": "This paper proposes Angular-KD, a novel knowledge distillation approach that enhances a single teacher model with lightweight linear branches to generate multiple diverse views. These views are optimized using two angular diversity objectives—constrained inter-angle and intra-angle losses—to increase semantic variance while preserving alignment with the original teacher. The approach is theoretically motivated and empirically validated across several benchmarks, showing improvements over prior work such as TeKAP.", "questions": "See Weaknesses.", "limitations": "The experiments are limited to image classification. Can it works on other tasks?", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The idea of generating diverse teacher views using angular diversity is novel and provides a unique perspective on knowledge augmentation.\n2. The proposed method can be easily integrated into existing KD frameworks.\n3. The paper is well-structured and logically coherent.\n\nWeaknesses\n1. The intra-angle diversify loss encourages an even distribution of views around the original output. However, the similarity between different views and the original image may vary, making this assumption unreasonable.\n2. The theoretical analysis cannot match the algorithm well. The main result—Upper Bound on Expected Loss of Ensemble in Equation (6)—has significant issues. The first term on the right-hand side of the bound increases with the number of augmented views N, implying that more views can lead to worse performance. However, there is no proof provided to show that the bound is tighter as N increases.\n3. Adding multiple branches to the teacher model increases much computational cost. However, the paper does not provide a comparison of the training time between the teacher and student models.\n4. All the coefficients of three loss are 1 and is it reasonable. Can you conduct an ablation for the coefficients ?\n5. The experiments are limited to image classification. It is necessary to include additional experiments on image segmentation as well as results on the ImageNet-1K dataset.", "was_revised": true, "final_justification": "Thank you for your constructive and insightful comments on my manuscript. I have carefully considered your feedback, and I am pleased to inform you that your responses have addressed the majority of my concerns. Your suggestions have significantly clarified several aspects of the paper, and I now feel more confident in its contribution to the field.\n\nAs a result, I have revised my assessment and have decided to increase the score accordingly.", "rebuttal": "Dear reviewer SW6m,\n\nThank you for your time and efforts in reviewing our manuscript. We appreciate your recognition of our paper’s strengths, particularly our novel approach using angular diversity, the compatibility of our methods, and a well-structured paper. Below, we would like to provide clarifications and address the concerns and limitations you raised.\n\n---\n\n**[W1] The assumption that intra-angle diversity loss promotes an even spread of views is unreasonable.**\n\nWe acknowledge the concern that varying similarities between different views and the teacher output could challenge the assumption of an even spread.\n However, in our empirical analysis (Table 15 in the supplementary), we observe that the intra-angle diversity loss significantly increases pairwise intra-angles between views.\n We interpret larger intra-angles as indicating that the views are more evenly distributed around the teacher representation, i.e., they are less clustered in a single direction, as also illustrated in Figure 1b. We believe this supports our claim that the intra-angle diversity loss encourages directional diversity among the views.\n\nFurthermore, we clarify that in this work, the term “even spread” is used in the sense of promoting representational diversity and reducing directional redundancy among views, rather than enforcing equal similarity to the teacher output. We will revise the final version to make this interpretation more explicit and avoid potential confusion.\n\n\n---\n\n\n**[W2] The first term on the right-hand side of the bound increases with the number of augmented views N.**\n\nWe would like to clarify that the first term on the right-hand side of the expected ensemble loss represents the mean of the losses across all augmented views, not their sum. Therefore, this term does not increase with the number of views N; rather, it reflects the average quality of individual views (e.g., each view’s cross-entropy loss with respect to the ground-truth label).\n\nWe will revise the explanation around Eq. (6) in the final version to clearly distinguish this averaging behavior and avoid any potential ambiguity.\n\n\n---\n\n\n**[W3] The training time comparisons.**\n\nThank you for your great suggestions. We agree that including a comparison of training time would be valuable. The following table compares the training time of our methods with 1) naive KD + CRD without augmentation, 2) the augmented one by TeKAP (a SoTA knowledge augmentation method), and 3) multi-teacher methods (e.g., Ensemble distillation [3]), including computational demands (FLOPs and Params; reported in Table 5). This experiment is conducted with a WideResNet40-2 teacher and a WideResNet-16-2 student on a 2080 Ti GPU over 50,000 training images and a total of 240 epochs.\n\n| Method          | Training time (1 epoch) | Params (M) | FLOPs (M) |\n| --------------- |:------------:| :------------:| :------------:| \n|  w/o aug |     27s     | 2.26 | 329.02 |\n| TeKAP      |   32s    |  2.26  | 329.02 |  \n| Ours           |    35s    |  2.40 | 329.19 |\n| Ensemble distillation | 47s |  11.28 | 1645.10 |\n\nWe observe that our multiple branches incur only a marginal increase in both training time and computation cost (FLOPs and Params), since the branches are composed of lightweight linear heads. In the revision, we will include these results to clearly present the lightweight nature of our approach. \n\n---\n\n**[W4] Ablations on the loss coefficients.**\n\nYes, we appreciate your valuable suggestion. The table below reports an ablation study over the three loss coefficients. \n\n| Coefficients (Inter:Intra:GT)          | Top-1 Accuracy |\n| --------------- |:------------:| \n| 0.5:1:1 |    76.03      |\n|  1:0.5:1     |   75.99    |  \n|  1:1:0.5     |    75.91   |\n|  0.5:0.5:1     |   75.74    |    \n| 1:1:1 (reported)         |     **76.46**   |\n\nWe observe that setting all coefficients to 1 yields the best performance. \nIn contrast, reducing the coefficients for both inter‑ and intra‑angle diversity losses to 0.5 results in a substantial decrease in performance, suggesting that these two angular losses are critical.\nWe will include this result in the final version to highlight the effect of each loss component.\n\n---\n\n**[W5, L1] Experiments on other tasks and ImageNet-1k.**\n\nWe appreciate the reviewer’s interest in evaluating broader applicability. However, we would like to clarify that results on ImageNet-1K are already included in our main manuscript—specifically in Table 10 of the supplementary material, where our approach outperforms the naive KD method without augmentation and a SoTA knowledge augmentation method, TeKAP. \n\nTo further validate that our approach is effective in other tasks, we are currently conducting object detection or image segmentation experiments and exploring training recipes optimized for our environments. Given the time required for thorough evaluation, we kindly ask whether it would be acceptable to share the results during the reviewer–author discussion phase. We will update the results as soon as the experiments are complete."}, {"review_id": "GpemzLGPqs", "reviewer": "Reviewer_KCJt", "summary": "This paper presents a new knowledge distillation method.  The idea is motivated from recent work showing that using multiple teacher networks for training a single student network leads to improved accuracy.  However, using multiple teacher networks is computationally expensive.  The new idea is to add \"view augmentation heads\" to a single teacher network to yield a similar effect to having multiple teacher networks with only a modest increase in computational cost.  View augmentation heads are linear transformations of either the final feature layer or the logit layer of the teacher network.  These new linear transformation layers are trained to produce new features or logits that exhibit diversity by using two new losses: an inter- and an intra-angle diversity loss.  The effect is similar to having multiple independent teacher networks since the augmentation heads yield diverse features and logits that are not too far from the original teacher's features and logits.  Experiments are conducted on the CIFAR-100, STL-10 and TinyImageNet datasets using various teacher-student network pairs.  Modest improvements are shown over a competing view augmentation approach.", "questions": "More discussion of how this paper's improvement to knowledge distillation compares with the many other papers on knowledge distillation improvements, not just multiple teacher approaches, would be helpful.\n\nWhy weren't all the experiments in Table 1 averaged over 3 or more runs?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The main idea of the paper, simulating multiple teacher networks using low-cost view augmentation heads, is well motivated and makes sense.\n\nThe paper is written clearly and is relatively easy to follow.\n\nThe proposed method is novel and includes new loss functions and a novel approach to view augmentation for improving knowledge distillation.\n\nThe theoretical results showing that increased ensemble diversity lowers the upper bound on the ensemble loss, is helpful for understanding how the view augmentation heads improve accuracy.\n\nWhy don't the experiments, especially in Table 1 average over multiple runs and include standard deviations?  This has been done in many past knowledge distillation papers.  The appendix A.4 does show one experiment that is averaged over 3 runs, but it would be even better to do this on all or most of the experiments.\n\nThe paper does not give a good sense of how the proposed view augmentation approach fits in with the large amount of previous work on knowledge distillation in terms of accuracy and computational cost.  There have been many proposed modifications to the original knowledge distillation scheme of Hinton et al. 2015, each of which typically improves accuracy a modest amount.  Since this paper only compares with TeKAP, it is unclear how it compares to the hundreds (thousands?) of other modifications to knowledge distillation that have been proposed.\n\nMinor typo:  On line 314, \"Hiton\" should be \"Hinton\".", "was_revised": true, "final_justification": "The authors did a good job of addressing my concerns by providing standard deviations for results over multiple runs and by including more comparisons to other KD methods.  I think the paper is above the bar for acceptance to NeurIPS.", "rebuttal": "Dear reviewer KCJt,\n\nThank you for your time and efforts in reviewing our manuscript. We greatly appreciate your recognition of the strengths of our work, including the well-motivated and novel methodology, the easy-to-follow presentation, and the theoretical results that help explain how our approach improves accuracy.\n\n---\n\n**[W1, Q2] Request for results averaged over three runs.**\n\nWe fully agree that averaging over multiple random seeds provides a more reliable evaluation. Our initial submission did not include three-run average results in Table 1 due to computational constraints (Table 1 spans 18 settings across architectures, and our experiments were conducted on limited resources of 2080 Ti GPUs). In the table below, we additionally report three-run averaged results under two representative configurations from Table 1: \n(1) a ResNet-32x4 teacher -> a ResNet-8x4 student, and \n(2) a ResNet32x4 teacher -> a WideResNet-40-2 student. \nThese are presented alongside results from TeKAP, a SoTA knowledge augmentation method.\n\n\n$$\n\\\\begin{array}{lcc}\n\\\\textbf{Teacher} & \\\\text{ResNet-32}\\\\times\\\\text{4} & \\\\text{ResNet-32}\\\\times\\\\text{4} \\\\\\\\\n\\\\textbf{Student} & \\\\text{ResNet-8}\\\\times\\\\text{4} & \\\\text{WideResNet-40-2} \\\\\\\\ \\\\hline\n\\\\textcolor{gray}{\\\\text{Logit distil. (KD) w/o aug}} & \\\\textcolor{gray}{\\\\text{73.33}} & \\\\textcolor{gray}{\\\\text{77.70}} \\\\\\\\ \n\\\\text{w/ TeKAP} & 75.84\\\\,\\\\pm 0.156 & 77.99\\\\,\\\\pm 0.087     \\\\\\\\\n\\\\text{w/ Angular-KD (ours)} & \\\\mathbf{76.08}\\\\,\\\\pm 0.217 & \\\\mathbf{78.57}\\\\,\\\\pm 0.125  \\\\\\\\ \\hline\n\\\\textcolor{gray}{\\\\text{Feature distil. (CRD) w/o aug}} & \\\\textcolor{gray}{\\\\text{75.51}} & \\\\textcolor{gray}{\\\\text{78.15}} \\\\\\\\ \n\\\\text{w/ TeKAP} & 75.82\\\\,\\\\pm 0.153& 77.97\\\\,\\\\pm 0.207     \\\\\\\\\n\\\\text{w/ Angular-KD (ours)} & \\\\mathbf{75.94}\\\\,\\\\pm 0.105 & \\\\mathbf{78.56}\\\\,\\\\pm 0.195  \\\\\\\\ \\hline\n\\\\textcolor{gray}{\\\\text{Logit +  Feat. distil. (KD + CRD) w/o aug}} & \\\\textcolor{gray}{\\\\text{75.46}} & \\\\textcolor{gray}{\\\\text{78.01}} \\\\\\\\ \n\\\\text{w/ TeKAP} & 75.89\\\\,\\\\pm 0.061& 77.96\\\\,\\\\pm 0.299     \\\\\\\\\n\\\\text{w/ Angular-KD (ours)} & \\\\mathbf{76.39}\\\\,\\\\pm 0.168 & \\\\mathbf{78.62}\\\\,\\\\pm 0.050  \\\\\\\\ \\hline\n\\\\end{array}\n$$\n\nWe will include the full results of Table 1 across all settings with three-run averages in the final version, if accepted.\n\n---\n\n**[W2, Q2] More comparisons with other KD methods beyond TeKAP.**\n\nThank you for the reviewer’s valuable comments. We understand that limiting the comparison to only TeKAP may not sufficiently demonstrate the general effectiveness of our approach across diverse knowledge distillation (KD) paradigms.\n\nTo address this concern, we emphasize that our method is **framework-agnostic** and designed to be **plug-and-play**, making it compatible with a wide range of KD frameworks. In our experiments, we applied our method on top of several representative and diverse KD frameworks, including DKD [57], MLKD [20] and TAKD [33] in addition to conventional KD [17] and CRD [45], and observed consistent improvements over their baseline performances (see Table 2 and Table 11).\n\nAdditionally, the table below shows further plug-and-play results from applying our methods to Review-KD [6], in comparison with TeKAP. Once again, our method achieves consistent improvements, and this suggests that our approach can complement, rather than replace, existing KD methods.\n\n$$\n\\\\begin{array}{lcc}\n\\\\textbf{Teacher} & \\\\text{ResNet-32}\\\\times\\\\text{4} & \\\\text{WideResNet-40-2} & \\\\text{ResNet-32}\\\\times\\\\text{4}    \\\\\\\\\n\\\\textbf{Student} & \\\\text{ResNet-8}\\\\times\\\\text{4} & \\\\text{WideResNet-40-1}  & \\\\text{WideResNet-16-2} \\\\\\\\ \\\\hline\n\\\\textcolor{gray}{\\\\text{Review-KD w/o aug}} & \\\\textcolor{gray}{\\\\text{75.63}} & \\\\textcolor{gray}{\\\\text{75.09}} & \\\\textcolor{gray}{\\\\text{76.11}} \\\\\\\\ \n\\\\text{w/ TeKAP} & 75.32 &  75.15  &76.36     \\\\\\\\\n\\\\text{w/ Angular-KD (ours)} & \\\\mathbf{75.78} & \\\\mathbf{75.45} & \\\\mathbf{76.40}  \\\\\\\\\n\\\\end{array}\n$$\n\nWe will revise the manuscript to clarify this point and better highlight the broad compatibility and effectiveness of our approach across various KD frameworks.\n\n---\n\n**[W3] Typo on line 314.**\n\nThank you for pointing out the typo. We will correct “Hiton” to “Hinton” on line 314 in the final version."}]

Author rebuttal:

Meta-review: This paper presents a new knowledge distillation method, which uses multiple cheap augmentation views of the teacher for student to learn from. Compared with the most relevant work, the views are learned instead of created with random perturbation. The method also uses additional loss terms to enforce the views to be diverse. Experiments are conducted on the CIFAR-100, STL-10 and TinyImageNet datasets using various teacher-student network pairs, and modest improvements are shown over previous multi-teacher approaches. The authors shall incorporate the reviewers comments in the final version.
Decision: Accept (poster)