Right Time to Learn: Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation

Authors: Guanglong Sun, Hongwei Yan, Liyuan Wang, Qian Li, Bo Lei, Yi Zhong
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Knowledge Distillation, Brain-inspired Al, Machine Learning, Spacing effect

Abstract:
Knowledge distillation (KD) is a powerful strategy for training deep neural networks (DNNs). While it was originally proposed to train a more compact “student” model from a large “teacher” model, many recent efforts have focused on adapting it as an effective way to promote generalization of the model itself, such as online KD and self KD. Here, we propose an easy-to-use and compatible strategy named Spaced KD to improve the effectiveness of both online KD and self KD, in which the student model distills knowledge from a teacher model trained with a space interval ahead. This strategy is inspired by a prominent theory named spacing effect in the field of biological learning and memory, positing that appropriate intervals between learning trials can significantly enhance learning performance. We provide an in-depth theoretical and empirical analysis showing that the benefits of the proposed spacing effect in KD stem from seeking a flat minima during stochastic gradient descent (SGD). We perform extensive experiments to demonstrate the effectiveness of our Spaced KD in improving the learning performance of DNNs (e.g., the additional performance gain is up to 2.31% and 3.34% on Tiny-ImageNet over online KD and self KD, respectively).

Decision: reject
Reviewer scores: 5, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730248145814

thread_type:
official_review

<thread_object>
creator: reviewer_v8iu

<content>
summary:
This paper introduces Spaced Knowledge Distillation (Spaced KD), a bio-inspired approach to knowledge distillation that incorporates spacing intervals in the training process to enhance model generalization. Inspired by the "spacing effect" observed in biological learning, Spaced KD involves training a teacher model a few steps ahead of the student model and periodically transferring knowledge at intervals. This technique allows the student model to find flatter minima in the loss landscape, which leads to better generalization and improved robustness to noise. Extensive experiments demonstrate that Spaced KD yields superior performance across multiple datasets and architectures compared to traditional KD methods without additional training costs.

strengths:
1) Novelty: The paper introduces a fresh approach by integrating the bio-inspired “spacing effect” into knowledge distillation (KD), which is relatively unexplored in existing literature.

2) Strong Theoretical Foundation: The authors provide a well-structured theoretical analysis linking Spaced KD to flat minima, supporting claims about enhanced generalization. The use of Hessian trace analysis to demonstrate why Spaced KD leads to better generalization adds rigor and robustness to the claims.

3) Comprehensive Experiments: The experimental results are extensive, spanning multiple datasets (CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-1K) and network architectures (ResNet variants, DeiT-Tiny, PiT-Tiny). These experiments show consistent improvements with Spaced KD, reinforcing the method’s general applicability.

4) Practicality: Spaced KD is straightforward to implement, making it accessible for a broad range of applications without requiring additional training time or major architectural modifications. This ease of integration enhances its appeal for both academia and industry.

5) Robustness and Generalization Gains: Spaced KD demonstrates superior robustness to noise and out-of-distribution data, an increasingly valued property in modern AI. This robustness adds significant value, especially for real-world applications where models encounter varied data distributions.

weaknesses:
1) Incremental Novelty: While Spaced KD introduces a creative application of the spacing effect to knowledge distillation, it lacks a fundamentally new mechanism within the KD framework. This incremental contribution may limit its overall impact, as the method primarily builds on existing KD techniques without significantly advancing the underlying theory.

2) Interval Sensitivity and Adaptability: The success of Spaced KD hinges on selecting an effective interval for knowledge transfer. However, the paper does not explore adaptive strategies for dynamically tuning this interval, which may constrain Spaced KD’s versatility across diverse tasks and datasets with varying training schedules.

3) Applicability Across Model Sizes and Architectures: Spaced KD is demonstrated only with identical teacher and student models of the same size, without addressing model compression. It remains uncertain how this approach would perform if applied to different model sizes or architectures, such as a transformer teacher model paired with a CNN student. The potential to generalize Spaced KD for cross-architecture or cross-size distillation remains unexplored.

questions:
1) Applicability Across Model Sizes and Architectures: How does Spaced KD perform when the teacher and student differ in size or architecture? Specifically, can it support scenarios where the teacher is larger or has a different architecture (e.g., a transformer teacher with a CNN student), or does it rely on identical model sizes?

2) Comparison with Gradient-based Methods: How does Spaced KD differ from traditional gradient-based methods where gradients are updated after each batch? Given that Spaced KD introduces intervals between teacher and student updates, is this approach essentially a modified gradient update, or does it provide additional benefits that distinguish it from standard gradient optimization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631547411

thread_type:
official_review

<thread_object>
creator: reviewer_bfli

<content>
summary:
This paper studies a variant of Knowledge Distillation, designed with Space Interval learning, inspired by the spacing effects observed in biological learning. 

Design: The Space Interval method is applied to Online KD, where the teacher model is trained a few intervals (s iterations) ahead of the student. The teacher’s weights are then frozen and used to train the student, who lags behind. The hypothesis is that, by updating the teacher less frequently, it provides more stable guidance to the student. 

Theoretical Analysis: The authors use the trace of the Hessian matrix as a measure of the loss landscape’s flatness, where a smaller Hessian trace indicates a flatter, and thus potentially more generalizable, landscape. They show that the loss landscape of the new paradigm has flatter Hessian trace ompared to the traditional Online KD, suggesting improved generalization. 

Experimental Results: The proposed method outperforms the traditional Online KD on CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-1K datasets by approximately 1–2% on average with three ResNet variants. The authors also provide experiments demonstrating robustness improvements over online and self-KD counterparts, along with ablation studies on learning rate, batch size, and loss functions.

strengths:
Overall, the paper is easy to follow. The idea seems simple to implement and appears interesting as inspired by biological learning processes. Empirically, it achieves a few percentage points of improvement over existing online methods, as demonstrated in the experimental sections.

weaknesses:
The results are overall encouraging, there are few points to be clarified with the authors in term of bio-inspiration, the generality of the method, the settings used across experiments and the seemingly conflicts between theoretical and empirical results as in Questions section.

questions:
1. While the bio-inspiration is interesting, the learning mechanism applied here differs from the spacing effects observed in biology. In biological learning, spacing effects enhance learning and memory retention of a subject when study sessions or practice trials are spread across intervals of time. However, in this application, the teacher is trained at intervals while the student learns continuously which seems the design is not well-aligned with the biological inspiration. Could the authors provide deeper details on this point to show the bio-inspiration is valid.

2. In the online setting, most experiments use a teacher with the same architecture as the student. To demonstrate the generality of the method, could the authors conducting experiments where the teacher’s architecture is significantly larger than the student's?

3. As shown in Fig. 3, the period of applying Space KD is critical. Are all experiments using the best setting or different experiments has different periods? Could the authors provide details on the optimal settings used across the experiments in the paper and explain how these settings were selected?

4. Is there a conflict between the theoretical analysis and the empirical results? While the theoretical analysis is independent of the period during which Space KD is applied, the empirical results indicate that it is only useful in the last 10 epochs of training? Could the authors provides also the only online KD baseline's results in Fig. 3 for the comparision?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661962591

thread_type:
official_review

<thread_object>
creator: reviewer_5j6i

<content>
summary:
The paper introduces a new method for online knowledge distillation that uses the teacher model trained ahead of the student at each epoch, resulting in a flatter loss landscape.

strengths:
The paper is well-written and easy to follow.

The experiments show the effective improvement of the proposed method over vanilla KD.

weaknesses:
1) I did not get the connection between the proposed method and the spacing effect in biological learning mentioned in Section 3.2. The teacher has different weights from the student. Why does training for several additional epochs cause a spacing effect?

2) The theoretical part is not robust. The proposed method assumes that the teacher trained ahead is close to the local minimizer $\phi^*$, so why not use a well-trained model instead?

3) There is a lack of comparison with other KD methods like [1,2]

[1] Self-Distillation from the Last Mini-Batch for Consistency Regularization.

[2] Self-Knowledge Distillation with Progressive Refinement of Targets.

questions:
1) What are the numerical values of x and y in Figure 2a?

2) As 0.5 epochs mean training on half of the dataset, would this cause a bias towards that half of the dataset? Is the training dataset shuffled?

3) How can the proposed method be applied to self-distillation in [3], where the teacher is the deepest layer of models and the students are the shallow layers?

4) What dataset was used for Figure 3, and which settings were used for the experiments?



[3] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730758037356

thread_type:
official_review

<thread_object>
creator: reviewer_ac8y

<content>
summary:
In this work, the authors perform a scheme on the lines of online and self KD to promote generalization of the model. They propose SpacedKD which is inspired from spacing effect in biological learning which highlights that study sessions across time is critical for memory formation. They argue that this spacing effect can be generalized to a kind of ensemble learning at the temporal scale. They implement this spaced KD in both online and self KD setups. For online KD, they just train the teacher in advanced for a certain number of steps termed as spaced interval. Then they just freeze this teacher and distill this knowledge to the student using the same data. They further provide a theoritical justification of its better generalization than online KD by analyzing the Hessian Matrix of the loss function of student in both the scenarios by arguing that if the student model converges to a local minimizer at step t of SGD then expected value of trace of hessian matrix using Spaced KD is lesser.  Furthermore, they provide experimental results to verify their claim on the CIFAR-100, ImageNet-100 and Tiny-ImageNet datasets for both self and online KD where they show significant gains.

strengths:
The connection identified to the spacing effect in biological learning is quite interesting and provide intuitive backing for the method. The analysis of hessian of loss being lesser for spaced KD is also interesting and it being converging to a flatter local minima via SGD. 
The results show prominent gains in performance which is again promising. Furthermore, the authors have also provided analysis of using different value of the interval various architectures along with the different architecture of teacher student pairs (inclduing different architectures for teacher). Also, the analysis in presence of various corruptions further backs the use case of their method.

weaknesses:
Not exactly clear from the paper what is the exact difference between spaced and online KD setup. Also, why is the description of implementing spaced KD for self distillation not provided. I understand that it is pretty straightforward to adapt it but for completeness it should be there and there might be some smal changes that would be required into a naive adaptation that helps improving the result. 

Regarding results, even though the authors have mentioned in one of the figure captions the reason for not providing results on ViT architecture, I feel that it is not justified. To test its applicability it should be tested on state of the architectures and large scale datasets. The major results revolve around CIFAR-100, Tiny-ImageNet and ImageNet-100 architectures and only one result is provided in the ImageNet-1k (and that too with a ResNet-18 architecture) leading to speculations that it might not give this sort of gains on ImageNet-1k or other large scale settings. Also, I dont understand what the Appendix A.5 results want to convey.

questions:
Please check the weakness section.  Appendix A.5 is confusing. Does it imply that distilling from a teacher trained s steps ahead is not useful? But then this is the method right that first you should update teacher for s steps and then using that distill to student for those s steps. Also it would have been very useful to provide a psuedo code comparison for online and spaced KD so as to what exactly is the differennce. Overall I feel the presentation of the paper could have been improved especially the emipirical section. 

Anyhow, right now I am going with the marginally below acceptance but I would be happy to reconsider my rating based on the justifications provided in the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731985035281

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response

comment:
Dear reviewers,

We sincerely appreciate your great efforts in reviewing our manuscript and all your valuable feedback. It is encouraging to see many positive comments, especially noting that our method is "interesting" (Reviewers Ac8Y, bFLi), "easy to follow" (Reviewer 5J6i, bFLi, v8iU) and "effective" (Reviewers 5J6i, bFLi, v8iU). We have provided a point-to-point response to all comments, summaized the corresponding changes, and uploaded a revised version of our manuscript. Our responses and revisions can be summarized as follows:

(1) We have added more explanations to the connections between the proposed Spaced KD and biological spacing effect.

(2) We have added more explanations to the theoretical analysis of offline KD, online KD and Spaced KD, demonstrating the superiority of our proposal.

(3) We have added more explanations to the implementation of online KD, self KD and their spaced version, with the coressonding pseudo code.

(4) We have added more in-depth and extensive experiments across a variety of network architectures and benchmark datasets, and the proposed Spaced KD achieves consistent improvements.

We believe that the quality of the current manuscript has been largely improved. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732610155625

thread_type:
official_comment

<thread_object>
creator: area_chair_qfve

<content>
title:
Encouragement to Actively Participate in the Discussion Phase

comment:
Dear Reviewers,

Thank you for your valuable contributions to the review process so far. As we enter the discussion phase, I encourage you to actively engage with the authors and your fellow reviewers. This is a critical opportunity to clarify any open questions, address potential misunderstandings, and ensure that all perspectives are thoroughly considered.

Your thoughtful input during this stage is greatly appreciated and is essential for maintaining the rigor and fairness of the review process.

Thank you for your efforts and dedication.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings
This paper introduces Spaced Knowledge Distillation (Spaced KD), a novel approach inspired by the biological "spacing effect." The proposed method introduces intervals between teacher and student training updates to enhance the generalization performance of knowledge distillation (KD). Key contributions include:
A spacing interval mechanism applied to online and self-distillation settings.
A theoretical justification using Hessian trace analysis to link the method to flatter minima, supporting better generalization.
Empirical validation on multiple datasets and architectures, showing performance gains over traditional KD methods.

(b) Strengths of the Paper
Interesting Bio-inspiration: The application of the spacing effect to KD is innovative and provides an intuitive motivation for the method.
Comprehensive Empirical Analysis: The paper includes extensive experiments across CIFAR-100, Tiny-ImageNet, ImageNet-100, and ImageNet-1k with various architectures, demonstrating performance improvements.
Theoretical Contributions: The authors provide a theoretical explanation of the mechanism using Hessian trace analysis, linking Spaced KD to flatter loss landscapes.
Practical Simplicity: Spaced KD is easy to implement, requiring minimal changes to existing KD pipelines.

(c) Weaknesses of the Paper
Incremental Contribution: The method primarily adapts an existing concept (spacing effect) without significantly advancing the theoretical or methodological understanding of KD.
Limited Scalability Validation: The performance gains on large-scale datasets (e.g., ImageNet-1k) and transformer-based architectures (e.g., ViT) were marginal and less compelling.
Unconvincing Biological Connection: The claimed link between the biological spacing effect and the proposed KD method is weakly justified and does not align with the mechanisms observed in biological learning.
Inconsistent Results: Improvements over state-of-the-art methods like PSKD and DLB are minimal, and the lack of adaptive interval tuning limits generalizability across diverse settings.
Weak Baseline Comparisons: The paper fails to benchmark against more recent or advanced KD methods and does not adequately explore scenarios involving different teacher-student architectures.

(d) Reasons for Rejection
Lack of Substantial Novelty: While the bio-inspiration is interesting, the contribution is primarily incremental, adapting a known concept without introducing significant theoretical or methodological advancements.
Limited Impact on Large-Scale Settings: The method's limited effectiveness on larger datasets and advanced architectures undermines its broader applicability and scalability.
Weak Empirical Justifications: The minimal improvements over existing KD methods and lack of robust comparisons with state-of-the-art baselines weaken the empirical claims.
Unresolved Reviewer Concerns: Despite the authors’ efforts during the rebuttal, key issues such as the biological connection, baseline completeness, and scalability remained inadequately addressed.