Title: Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity
Venue: NeurIPS 2025
Accepted: yes
Keywords: knowledge distillation

knowledge augmentation

angular diversity
Abstract: Knowledge Distillation (KD) aims to train a lightweight student model by transferring knowledge from a large, high-capacity teacher.
Recent studies have shown that leveraging diverse teacher perspectives can significantly improve distillation performance; however, achieving such diversity typically requires multiple teacher networks, leading to high computational costs. In this work, we propose a novel cost-efficient  knowledge augmentation method for KD that generates diverse multi-views by attaching multiple branches to a single teacher. To ensure meaningful semantic variation across multi-views, we introduce two angular diversity objectives: 1) $\textit{constrained inter-angle diversify loss}$, which maximizes angles between augmented views while preserving proximity to the original teacher output, and 2) $\textit{intra-angle diversify loss}$, which encourages an even distribution of views around the original output. The ensembled knowledge from these angularly diverse views, along with the original teacher, is distilled into the student. We further theoretically demonstrate that our objectives increase the diversity among ensemble members and thereby reduce the upper bound of the ensemble's expected loss, leading to more effective distillation. Experimental results show that our method surpasses an existing knowledge augmentation method across diverse configurations. Moreover, the proposed method is compatible with other KD frameworks in a plug-and-play fashion, providing consistent improvements in generalization performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new knowledge distillation method.  The idea is motivated from recent work showing that using multiple teacher networks for training a single student network leads to improved accuracy.  However, using multiple teacher networks is computationally expensive.  The new idea is to add \"view augmentation heads\" to a single teacher network to yield a similar effect to having multiple teacher networks with only a modest increase in computational cost.  View augmentation heads are linear transformations of either the final feature layer or the logit layer of the teacher network.  These new linear transformation layers are trained to produce new features or logits that exhibit diversity by using two new losses: an inter- and an intra-angle diversity loss.  The effect is similar to having multiple independent teacher networks since the augmentation heads yield diverse features and logits that are not too far from the original teacher's features and logits.  Experiments are conducted on the CIFAR-100, STL-10 and TinyImageNet datasets using various teacher-student network pairs.  Modest improvements are shown over a competing view augmentation approach.\n\nSTRENGTHS AND WEAKNESSES:\nThe main idea of the paper, simulating multiple teacher networks using low-cost view augmentation heads, is well motivated and makes sense.\n\nThe paper is written clearly and is relatively easy to follow.\n\nThe proposed method is novel and includes new loss functions and a novel approach to view augmentation for improving knowledge distillation.\n\nThe theoretical results showing that increased ensemble diversity lowers the upper bound on the ensemble loss, is helpful for understanding how the view augmentation heads improve accuracy.\n\nWhy don't the experiments, especially in Table 1 average over multiple runs and include standard deviations?  This has been done in many past knowledge distillation papers.  The appendix A.4 does show one experiment that is averaged over 3 runs, but it would be even better to do this on all or most of the experiments.\n\nThe paper does not give a good sense of how the proposed view augmentation approach fits in with the large amount of previous work on knowledge distillation in terms of accuracy and computational cost.  There have been many proposed modifications to the original knowledge distillation scheme of Hinton et al. 2015, each of which typically improves accuracy a modest amount.  Since this paper only compares with TeKAP, it is unclear how it compares to the hundreds (thousands?) of other modifications to knowledge distillation that have been proposed.\n\nMinor typo:  On line 314, \"Hiton\" should be \"Hinton\".\n\nQUESTIONS:\nMore discussion of how this paper's improvement to knowledge distillation compares with the many other papers on knowledge distillation improvements, not just multiple teacher approaches, would be helpful.\n\nWhy weren't all the experiments in Table 1 averaged over 3 or more runs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Angular-KD, a novel knowledge distillation approach that enhances a single teacher model with lightweight linear branches to generate multiple diverse views. These views are optimized using two angular diversity objectives—constrained inter-angle and intra-angle losses—to increase semantic variance while preserving alignment with the original teacher. The approach is theoretically motivated and empirically validated across several benchmarks, showing improvements over prior work such as TeKAP.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The idea of generating diverse teacher views using angular diversity is novel and provides a unique perspective on knowledge augmentation.\n2. The proposed method can be easily integrated into existing KD frameworks.\n3. The paper is well-structured and logically coherent.\n\nWeaknesses\n1. The intra-angle diversify loss encourages an even distribution of views around the original output. However, the similarity between different views and the original image may vary, making this assumption unreasonable.\n2. The theoretical analysis cannot match the algorithm well. The main result—Upper Bound on Expected Loss of Ensemble in Equation (6)—has significant issues. The first term on the right-hand side of the bound increases with the number of augmented views N, implying that more views can lead to worse performance. However, there is no proof provided to show that the bound is tighter as N increases.\n3. Adding multiple branches to the teacher model increases much computational cost. However, the paper does not provide a comparison of the training time between the teacher and student models.\n4. All the coefficients of three loss are 1 and is it reasonable. Can you conduct an ablation for the coefficients ?\n5. The experiments are limited to image classification. It is necessary to include additional experiments on image segmentation as well as results on the ImageNet-1K dataset.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe experiments are limited to image classification. Can it works on other tasks?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Angular-KD framework, which generates diverse multi-view knowledge by attaching multiple lightweight linear branches to a single teacher model. It introduces a constrained inter-angle diversity loss and an intra-angle diversity loss. Experimental results show that Angular-KD outperforms existing knowledge augmentation methods across multiple knowledge distillation benchmarks and can be integrated as a plug-and-play module with other KD frameworks.\n\nSTRENGTHS AND WEAKNESSES:\nStreangth:\n\n1.\tThe method presents a novel angle for KD augment and is compatible with other knowledge distillation frameworks, designed as a plug-and-play solution.\n\n2.\tExperiments were conducted on multiple datasets and in imbalanced scenarios, achieving good performance.\n\n3.\tThe paper provides theoretical analysis and implementation code, enhancing the credibility and practicality of the research.\n\nWeakness:\n\n1.\tThe comparison is limited to only one data augmentation method, lacking comparisons with other advanced methods.\n\n2.\tThe comparison with multi-teacher methods is limited and not up-to-date, such as TAKD published in 2023, which may not be the latest multi-teacher method.\n\n3.\tThe visualization analysis in this paper is relatively limited, only providing t-SNE plots to show the distribution of the teacher and augmented views. It fails to further demonstrate the model's performance across different perspectives through other visualization methods (e.g., feature space distribution or decision boundaries).\n\nQUESTIONS:\nAlthough the proposed method generates diverse views by attaching multiple branches, it still relies on a single teacher model. This means that if the teacher model itself has biases or limitations, these biases may be passed on to the student model. In contrast, multi-teacher methods can provide a broader perspective through multiple independently trained teacher models. In the comparative experiments, the paper compares the performance with multi-teacher distillation methods, showing that this method outperforms them. This result raises the question of whether this method remains leading when the complexity of both the teacher and student models increases. The paper lacks test results on larger-scale models.\n\nLIMITATIONS:\nYes. The authors acknowledge that this method relies on the teacher's representation and thus cannot introduce truly novel semantics beyond the teacher's knowledge. While it outperforms multi-teacher methods, it lacks the ability to capture independent and complementary signals from multiple teachers. Relying on a single teacher may propagate teacher biases, which could have potential implications in privacy-sensitive or surveillance-related applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study proposes inter-angle and intra-angle losses to derive high-diversity perspectives from a single teacher model. Both losses operate to train linear layers to expand the diversity of outputs for the same sample in the feature and logit domains. Through this, the teacher model applies transfer learning to guide the student's learning direction by delivering multi-view supervision for the same data.\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{Strengths}$\n1. The writing is exceptionally clear and well-structured, making the content highly accessible and easy to follow.\n2. The approach expands the student model's representation coverage by generating multi-view (akin to negative views) from a single teacher model for the same sample.\n3. By pushing the similarity between feature and vector spaces, the method secures flexibility in diversity expansion.\n4. The mathematical approach of the proposed method is clearly presented and well-organized for easy comprehension.\n\n$\\textbf{Weaknesses}$\n1. The novelty of the proposed method is somewhat limited, as it applies the concept of negative pairs from existing contrastive learning to multi-representations of the same sample.\n2. The accuracy improvement of the proposed method is not convincingly superior compared to existing methods.\n\nQUESTIONS:\n1. Why does the proposed method induce better view-augmentation compared to traditional data augmentation techniques?\n2. Can the diversity achieved through view-augmentation be quantitatively demonstrated?\n3. Which dataset was used to derive the results presented in Table 7?\n4. Given that the diversity of multi-views may vary across classes, can this be accounted for in the method?\n\nLIMITATIONS:\n1. The novelty of the proposed method is somewhat lacking.\n2. Additional analytical experiments are insufficient. Comparisons of representation variants based on diversity margins or the number of views ($N$) are needed.\n3. The number of training samples in CIFAR-100 should be reviewed for accuracy.\n4. The subcaptions (a) and (b) in Table 9 are meaningless and lack context.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}