Right Time to Learn: Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
Authors: Guanglong Sun
Hongwei Yan
Liyuan Wang
Qian Li
Bo Lei
Yi Zhong
Venue: ICML 2025 poster (2025)
Keywords: Knowledge Distillation
Brain-inspired Al
Machine Learning
Spacing effect
Primary area: general_machine_learning->supervised_learning
TL;DR: We draw inspirations from the spacing effect of biological learning and propose a new paradigm of knowledge distillation to improve generalization.

Abstract: Knowledge distillation (KD) is a powerful strategy for training deep neural networks (DNNs). While it was originally proposed to train a more compact “student” model from a large “teacher” model, many recent efforts have focused on adapting it as an effective way to promote generalization of the model itself, such as online KD and self KD. Here, we propose an easy-to-use and compatible strategy named Spaced KD to improve the effectiveness of both online KD and self KD, in which the student model distills knowledge from a teacher model trained with a space interval ahead. This strategy is inspired by a prominent theory named spacing effect in the field of biological learning and memory, positing that appropriate intervals between learning trials can significantly enhance learning performance. We provide an in-depth theoretical and empirical analysis showing that the benefits of the proposed spacing effect in KD stem from seeking a flat minima during stochastic gradient descent (SGD). We perform extensive experiments to demonstrate the effectiveness of our Spaced KD in improving the learning performance of DNNs (e.g., the additional performance gain is up to 2.31% and 3.34% on Tiny-ImageNet over online KD and self KD, respectively). Our codes have been released on github~\url{https://github.com/SunGL001/Spaced-KD}.

Right Time to Learn: Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
Guanglong Sun 123
Hongwei Yan 123 Liyuan Wang*
Qian Li!
Bo Lei
Yi Zhong
12
Abstract
Knowledge distillation (KD) is powerful strat- egy for training deep neural networks (DNNs), Al- though it was originally proposed to train more compact "student" model from a large "teacher" model, many recent efforts have focused on adapt- ing it to promote generalization of the model it- self, such as online KD and self KD, Here, we propose accessible and compatible strategy named Spaced KD to improve the effectiveness of both online KD and self KD, in which the student model distills knowledge from a teacher model trained with a space interval ahead. This strategy is inspired by a prominent theory named spacing effect in biological learning and memory, posit- ing that appropriate intervals between learning trials can significantly enhance learning perfor- mance. With both theoretical and empirical anal- yses, we demonstrate that the benefits of the pro- posed Spaced KD stem from convergence to a flat- ter loss landscape during stochastic gradient de- scent (SGD). We perform extensive experiments to validate the effectiveness of Spaced KD in im- proving the learning performance of DNNs (e.g.. the performance gain is up to 2.31 % and 3.34% on Tiny-ImageNet over online KD and self KD, respectively). Our codes have been released on github https: htttp:://gthuut ggthub...../ggg//gg com/ SunGLO01/ Spaced-KD.
online KD
T
w/ Ours
self KD
w/ Ours
Teacher
Student
KD
Training time
Figure I. Diagram of Spaced KD. In online KD, the teacher and student are two individual networks. In self KD, we follow the prior work (Zhang et al., 2019) that distills knowledge from the deepest layer to the shallower layers of the same network. In Spaced KD, we train the teacher with a controllable space interval steps ahead and then distill its knowledge to the student network.
(DNNs) (Gou et al., 2021; Wang & Yoon, 2021). Despite its extensive applications to construct a more compact "stu- dent" model from a converged large "teacher" model (aka offline KD), there have been many recent efforts using KD to promote generalization of the model itself, such as line KD (Zhang et al., 201 8; Zhu et al., 2018; Chen et al., 2020) and self KD (Zhang et al., 2019; Mobahi et al., 2020), Specifically, online KD simplifies the KD process by train- ing the teacher and the student simultancously. while self KD involves using the same network as both teacher and student. However, these paradigms improve learning performance, how to design sirable KD paradigm in terms of generalization remains an open question.
only moderately de-
1. Introduction
Knowledge distillation (KD) is a powerful technique to transfer knowledge between deep neural networks
Compared to DNNs, biological neural networks (BNNs) are advantageous in learning and generalization with specialized adaptation mechanisms and effective learning procedures. In particular, it is commonly recognized that extending the interval between individual learning events considerably enhance the learning performance, known as the spacing effect (Ebbinghaus, 2013; Smolen et al., 2016). This high- lights the benefits of spaced study sessions for improving the efficiency of learning compared to continuous sessions, and has been described across a wide range of species from in-
"Equal contribution 'School of Life Sciences, IDG/MeGovern Institute for Brain Research, Tsinghua University, Beijing, China "Tsinghua-Peking Joint Center for Life Sciences 'Beijing Academy of Artificial Intelligence, Beijing, China 'Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Beijing, China. Correspondence to: Liyuan Wang <wly2033@ (atpyy222@@snnngaa.commmm tsinghua.cdu.cn>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Brain-inspired spacing effect that improves knowledge distillation.
vertebrates to humans (Beck et al., 2000; Pagani et al., 2009; Menzel al., 2001; Anderson al., 2008; Bello-Medina et al., 2013; Medin, 1974; Robbins & Bush, 1973).Taking human learning as an example, the spacing effect could enhance skill and motor learning (Donovan & Radosevich, 1999; Shea et al., 2000), classroom education (Gluckman et al., 2014; Roediger & Byrne, 2008; Sobel et al., 201 10, and the generalization of conceptual knowledge in chil- dren (Vlach, 2014).
DML (Zhang et al., 2018) implements bidirectional distilla- tion between peer networks. For self KD, the same network is used as both teacher and student (Zhang et al., 2019; Das & Sanghavi, 2023; Mobahi et al., 2020; Zhang & Sabuncu, 2020; Yang et al., 2019; Lee et al., 2019). In this paper, the self KD we refer to is the distillation between different layers within the same network (Zhang et al., 2019; Yan et al., 2024; Zhai et al., 2019). However, existing methods for online KD) and self KD often fail to effectively utilize high-capacity teachers over time, making it an intriguing topic to further explore the relationships between teacher and student models in these environments.
Inspired by biological learning, we propose to incorporate such spacing effect into KD (referred to as Spaced KD, see Fig. 1) as a general strategy to promote the generalization of DNNs (see Fig. 2). We first provide an in-depth the- oretical analysis of the potential benefits of Spaced KD. Compared regular KD strategies, the proposed Spaced KD helps DNNs find flat minima during stochastic gra- dient descent (SGD) (Sutskever et al., 2013), which has proven to be closely related to generalization. We then perform extensive experiments to demonstrate the effective- ness of Spaced KD, across various benchmark datasets and network architectures. The proposed Spaced KD achieves strong performance gains (e.g., up to 2.3 1 % and 3.34% on Tiny-ImageNet over regular KD methods of online KD and self KD, respectively) without additional training costs. We further demonstrate the robustness of the space interval, the critical period of the spacing effect, and its plug-in nature to a broad range of advanced KD methods.
Adaptive Distillation. Recent studies have found that the difference in model capacity between a much larger teacher network and a much smaller student network can limit dis- tillation gains (Liu et al., 2020a; Cho & Hariharan, 2019; Liu et al., 2020b). Current efforts to address this gap fall into two main categories: training paradigms (Gao et al.. 2018) and architectural adaptation (Kang et al., 2020; Gu & Tresp, 2020). For instance, ESKD (Cho & Hariharan, 2019) suggests stopping the training of the teacher early, while ATKD (Mirzadeh et al., 2020a) employs a medium-sized teacher assistant for sequential distillation. SHAKE (Li & Jin, 2022) introduces a shadow head as a proxy teacher for bidirectional distillation with students. However, existing methods usually implement adaptive distillation by adjust- ing sttudetrtttdentt architecture from spatial level. In con- trast, Spaced KD provides an architecture- and algorithm- agnostic way to improve KD from a temporal level.
Our contributions can be summarized as follows: (1) We draw inspirations from the paradigm of biological learning and propose to incorporate its spacing effect to improve online KD and self KD; (2) We theoretically analyze the potential benefits of the proposed spacing effect in terms of generalization, connecting it with the flatness of loss landscape; and (3) We conduct extensive experiments to demonstrate the effectiveness and generality of the proposed spacing effect across a variety of benchmark datasets, net- work architectures, and baseline methods.
Flatness of Loss Landscape. The loss landscape around parameterized solution has attracted great research atten- tion (Keskar et al., 2016; Hochreiter & Schmidhuber, 1994; Izmailov et al., 2018; Dinh et al., 2017; He et al., 2019). A prevailing hypothesis posits that the flatness of minima following network convergence significantly influences its generalization capabilities (Keskar et al., 2016). In general, flatter minima is associated with a lower generalization error, which provides greater resilience against perturba- tions along the loss landscape. This hypothesis has been empirically validated by studies such as (He al., 2019). Advanced advancements have leveraged KD techniques to boost model generalization (Zhang et al., 2018; Zhao et al., 2023; Zhang et al., 2019). Despite these remarkable ad- vances, remains a challenging endeavor to fully under- stand the impact of KD on generalization, especially in assessing the quality of knowledge transfer and the efficacy of teacher-student architectures.
2. Related Work
Knowledge Distillation (KD). Representative avenues of KD can be generally classified into offline KD, online KD, and self KD, based on whether the teacher model is pre- trained and remains unchanged during the training process. Offline KD involves one-way knowledge transfer in two- phase training procedure. It primarily focuses on optimizing various aspects of knowledge transfer, such as designing the knowledge itself (Hinton et al., 2015; Adriana et al., 2015), and refining loss functions for feature matching or distri- bution alignment (Huang & Wang, 2017: Asif et al., 2019; Mirzadeh et al., 2020b). In contrast, online KD simplifies the KD process by training both teacher and student simul- taneously and often outperforms offline KD. For instance,
3. Preliminaries
In this section, we first present the problem setup and some necessary preliminaries of KD. Then we describe the spac-
2

---

Brain-inspired spacing effeet that improves knowledge distillation.
ing effect in biological learning and discuss how it may inspire the design of KD.
2022c; Wu & Gong, 2021). But to maintain a valid knowl- edge gap between student and teacher, they are often ini- tialized using different random seeds in practice. Besides, the training process of teacher network could also be in- tervened by auxiliary loss from students through reverse distillation (Li & Jin, 2022; Qian et al., 2022; Shi et al., 2021).
3.1. Problem Setup
We describe the problem setup with supervised learning of classification tasks as an example. Given N training samples
Dusin = (z., }:) where Ij E Rd and yi E R the neural network model fa(-) : R& + R with parameters 0 E R is optimized by minimizing the empirical risk over Dumin and evaluated over the test dataset Dres-- Using the SGD optimizer (Sutskever et al., 2013), fo(.) is updated for each mini-batch of training data B. {(T,.31) PuranDeezz I S (1, .......
Self KD. As an alternate approach a pre-trained teacher, self KD utilizes the hidden information within the student network to guide its learning process. Instead of relying on a large external model, self KD achieves multiple knowledge alignments by introducing auxiliary blocks or creating dif- ferent representations of the same encoded data. For block- wise network, fo = =fe o fo, o.... fe., that is composed of m consecutive modules, the whole network fo is regarded teacher while shallower blocks fo.... fo, o... faz (1 S k < m) are students. Following the common set- ting (Zhang et al., 2019), 0 is updated with multiple task supervision and cross-layer distillation, which in fact can be formulated in terms of I, ((eacher) in Eq. 3 and (KD) Eq. 2 (see Alg. 3 in Appendix A.10):
ac+1= = 0c-
Σ (OL.))). B
11
(1)
where (,) = hask fo (z:), 31) is a task-specific supervi- sion loss. 17 and B = Iel denote the learning rate and batch size, respectively. KD supports various kinds of interac- tion between multiple neural networks. The teacher-student framework we refer to here consists by default of teacher network (-))) and a student network (o).). where the flow of knowledge transfer is often one- direeircttion: the learning of ff is guided by the output of g. but not vice versa. The loss of weighted sum
student network f in KD is bi-component of task-specific and distillation loss (lask and (KD), where hyperparameter a controls the impact of teacher guidance:
oc+1 = 0:
B FET.
V
L
Qescher) (0)
+
0) L.
LIKT (0
(4)
L,MO (0, Φ) (1 alla (fo(=:); mi) alkn(Jo(es)) go(()) (2) In many applications, the teacher network g is often differ- ent from and much larger than the student network to obtain a more compact model. Meanwhile, there is an increasing number of efforts to implement KD to improve generaliza- tion for one particular architecture, where the teacher and student may share a common framework but differ in the random seeds for initialization. Some KD methods even treat different parts within one single network as teacher and student. Below we describe two representative methods:
3.2. Spacing Effect in Biological Learning
Originally discovered by (Ebbinghaus, 2013), the biological spacing effect highlights that the distribution of study ses- sions across time is critical for memory formation. Then, its functions have been widely demonstrated in various animals and even humans (see Sec. 1). Many cognitive computing models have proposed the concept of spaced learning and described its dynamics, positing an optimal inter-trial inter- val during memory formation (Landauer, 1969; Peterson, 1966; Wickelgren, 1972). These studies motivate us to fur- ther investigate if a proper space interval could benefit KD of possible data variability across training batches. Here we provide more detailed explanations of the interdisciplinary connections:
Online KD Though traditional KD assumes the teacher network g as a pre-trained and powerful model, there ex- ist scenarios where obtaining such a teacher is costly or impractical. Online KD is proposed to learn from an on- the-fly teacher network, allowing for dynamic adaptation during student training. In online KD, the updating of g is aligned with f for every mini-batch B, with Ie (see Alg. in Appendix A.10) 1:
In machine learning, KD aims to optimize the parameters of a student network with the help of teacher network by regularizing their outputs to be consistent in response to similar inputs. As shown in a pioneering theoretical analy- sis (Allen-Zhu & Li, 2020), KD shares a similar mechanism with ensemble learning (EL) in improving generalization from the training set to the test set. In particular, online KD performs this mechanism at temporal scales, and self KD can be seen as a special case of online KD. In comparison, the biological spacing effect can also be generalized to a kind of EL at temporal scales, as the brain network processes similar inputs with a certain time interval and updates its synaptic weights based on previous synaptic weights, which allows for stronger learning performance at test time (Pagani
Φ++1 Φt-
7
V o I. (scocher) (
B
IEI
(3)
= Φi-
7
V Palaa(So (2:),,
B
IEI
The design of an online teacher is quite demand-oriented, it could be simply copy of the student network (Li et al.,
For clarity, we use the same notation v, B and lask to describe the training of g and f. although they may select different training al algorithms and hyperparameter values in practice.
3

---

Brain-inspired spacing effeet that improves knowledge distillation.
al., 2009; Smolen et al., 2016).
4.2. Theoretical Analysis
The proposed Spaced KD draws inspirations from the bi- ological spacing effect and capitalizes on the underlying connections between KD and EL. It incorporates a space interval between teacher and student to improve generaliza- tion. In particular, we hypothesize that an optimal interval may exist between the learning paces of teacher and student in DNNs, as in BNNs.
To understand why Spaced KD might provide better gener- alization than online KD*, analyze the Hessian matrix of the loss function for the student network in both scenarios. The Hessian matrix plays crucial role in understanding the curvature of the loss landscape. In literature, various metrics related to the Hessian matrix have been adopted to evaluate the flatness of a loss minimum after training con- vergence, reflecting the generalization ability of the trained model (Krizhevsky al., 2009; Blanc al., 2020; Damian et al., 2021; Zhou et al., 2020). Here we choose the Hes- sian trace as a representative for convenience. A smaller Hessian trace indicates flatter loss landscape, which has also been proved to be related to the upper bound of test set generalization error.
4. Spaced KD
In this section, we describe how Spaced KD is implemented into online KD and self KD, and include a pseudo code for each in Appendix 10 We then theoretically analyze the benefit of the proposed spacing effect in improving generalization.
Setup. For simplicity we set the dimension of class space as = 1, and the extension of 1 is straightforward Let the mean square error (MSE) be the task-specific loss. The KD loss characterizes the distance between two distributions 0 and y: lunk (Ú. y) = kkD y) = 2 (0 - -y)
4.1. Incorporating Spacing Effect into KD
By applying spaced learning in the pipeline of KD, more precisely in the context of online KD, we implement pro- cess of alternate learning between teacher and student. The teacher network updates itself several steps in advance, and then it helps the student network train on the same set of batches. Formally, we define a hyperparameter Space Inter- val denoted as to represent the gap between the teacher's ss and student's learning pace. Spaced KD described as follows (see Fig. 1):
Hessian Matrix. For KD loss at the i-th data sample that follows Eq. 2. the Hessian matrix at a point of student fo(+) with respect to its teacher (-)-) can be calculated as the second-derivative of the empirical risk L (KD)) o) =
N EN L. (IKD)( Φ). It could be casily verified that:
H²) Văr. (KD) (0, N
Velole:)Vols((:)) + 86.,0,)
1. First, we train the teacher (-) for steps (from B, to Bi++-1) according to the learning rule in Eq. 3, obtaining an advanced teacher Boct. (.) identical to that of online KD.
(6)
where (() (1oo)) - v1) a(fo(x;)
go (z:)), and in fact VOL. = B(i, (),,8,))Vol Vefo((;; At arbitrary time stamp during the supervised training process, the teacher model's parameters for student 0, in online KD is di. In Spaced KD it should be Ok(I) with k(t) = [[t/s]]s where [.] denotes ceiling operation. No- tice that for online KD, the loss function constantly changes due the update of the teacher, but when we focus on the loss curve for particular Φi the differentiability of I. (KD) are preserved, allowing us to continue the discussion.
2. Then, we freeze the parameters +t+s of our teacher g, and start to transfer knowledge from it to the student fo, (.) that lags behind over the same batches of training data B
01.. 01
1l
Σ E
Vol
(KD) (0,, 014s);
(5)
B
where L (KD) is the same as Eq. 2 but using fixed teacher parameters dc+s-
Detinition 4.1 (Local linearization.). Let l be local mini- mizer of loss function w.r.t fo (-), we call the local lineariza- tion of fe(-) at 0 around 0* as: fe = (x) + (0 0*, Vo (x)).
Intrinsically, Spaced KD is a special case of online KD. The main difference that sets Spaced KD apart from online KD the less frequent updates of the teacher network, which provides a relatively stable learning standard for the student network and potentially contributes to its better generaliza- tion ability than the online setting. In practice, we initialize the teacher in Spaced KD using the same random seed as the student. To take a closer look, we theoretically illus- trate the impact of the proposed spacing effect on KD with step-by-step by-tep mathematical derivations in the next section.
For both teacher and student networks, this linearized model in Def. 4.1 provides an applicable approximation of the local dynamic behavior around converged point. We denote o² and 0* as the local minimizer of teacher and student, respectively. Without loss of generality, we assume that
2For all theoretical analysis and conclusions in this section, we treat self KD as special case of online KD since they share the same teacher-student relations. In the later Sec. 5, our experiments empirically support this argument as they behave similarly.
4

---

Brain-inspired spacing effect that improves knowledge distillation.
RIE-CICO
R18-T200
spacing effect (1%)
R10L-C100
R50-C100
850-1200
spacing effect (%)
g
Strength of memory trace
Increased of
Increase of
2-
0.0
1.0
1.5
2.0 max
0.0
0.5 1.0
1.5
2.0 max
Inter-stimulus intervals (a) Cognitive model
Interval (epochs)
Interval (epochs)
(b) Spaedd KD
(c) Polyfit of (b)
Figure 2. Alignment of spaced learning in BNNs and DNNs. (a) Computational cognitive model of spaced learning, modified from (Lan- dauer, 1969). (b) Overall performance Spaced KD from different networks and benchmarks. R18: ResNet-18; R50: ResNet-50; R101: ResNet-101; C100: CIFAR-100; T200: Tiny-ImageNet. (c) Quadratic polynomial fitting of all performance from (b).
after enough learning steps, Vx,, I6- (o-(x;) = fo- (a;) Vi which means both models follow the over- wver-parameterized setting so that their training set accuracy eventually become 100%. Therefore, when the student network f.(-) converges to local minimizer 0" in both online KD and Spaced KD, its corresponding teacher network (-)-) should also be close to o²
The comparison between the Hessian trace for Spaced KD and online KD finally settles in the difference between spaced but advanced teacher and frequently updated teacher. Detailed proof of Theorem 4.4 are provided in Appendix A.I with the help of Lemma 4.3, indicating a flat- ter loss landscape and thus potentially better generalization ability for the student network of Spaced KD.
B(i, 0" 0) (1 a) (x,) - v.) (x.)) go (z.))
a o.; Vege- (z,)) adø" Vogo- (z.),
The above analysis reveals key distinctions between Spaced KD, offline KD, and online KD. Spaced KD guides the student f with a well-defined trajectory estab- lished by the teacher that is slightly ahead in training (Shi et al., 2021; Rezagholizadeh et al., 2021), thereby ensur- ing low errors along such informative direction to improve generalization. With ideal condition where g and con- verge to the same local minima, offline KD and Spaced KD should perform identically best. However, this ideal condi- tion hardly exists in practice, especially given the nature of over-pan parameterization in advanced DNNs and the complex- ity of real-world data distributions. These two challenges result in a highly non-conver loss landscape of both g and f with a large number of local minima. Therefore, using a well well-trained teacher in offline KD tends to be sub-optimal since y and f can easily converge to different local minima with SGD. In comparison, the limitation of online KD lies in its narrow, constant interval between g and f. restricting the exploration of informative directions. By maintaining an appropriate spaced interval, Spaced KD allows for broader explorations and encourages convergence more desir- able region of the loss landscape, empirically validated in the following section.
Discussion.
where Aø Φ- .0 3 directly reflects the difference in the teacher model updating between online KD and Spaced KD. We then demonstrate how the combination of mini-batch training and space interval affects the role of the teacher model under the KD framework.
Definition 4.2 (Teacher model gap). For a teacher model gx(+) trained with SGD using the updating rule in Eq. 3, we define current prediction eтror over training dataset as the performance gap between @ and loss minima o" u(Ø)=
N 1 A Ve9e--1n (1ra)l
At a training step t close to convergence (a global time stamp) of the student model, considering the randomness brought by mini-batch sampling, we denote 24 = [())) for online KD, and (*(()) E [u (d*())] for Spaced KD (with space interval s) as the parameter gap of their corre- sponding teacher models, respectively.
Lemma 4.3 (Lower risk of spaced teacher). (x(()) S 4n-
Proof. It is straightforward that the teacher with dx(e) in Spaced KD is an advanced model which has undergone several updating iterations ahead of the student at step t. Namely, by definition i < k(t)= = ([t/s])s S st+a s. Thus, given the fact that SGD eventually selects a loss minima with linear stability (Wu et al.), i.e., (eachee)))) (1)- S E (tecche))) (0e) around s", we have Uz(t) I Ur.
5. Experiment
In this section, we first describe experimental setups and then present experimental results.
Theorem 4.4. the student model fo(-) converges to local
minimizer 0² at step i of SGD, let H! (0*) and H(S) oz (0')
5.1. Experimental Setups
be the Hessian of online KD and Spaced KD, then
We evaluate the proposed spacing effect on both ResNet-based architectures (He et al., 2016) such as
Benchmark.
E(T (0 )) (T (0) (0'))
5

---

Brain-inspired spacing effeet that improves knowledge distillation.
ResNet-18, 18, ResNet-50 and ResNet-101, and transformer- based architectures (Dosovitskiy et al., 2020) such as DeïT- Tiny (Touvron et al., 2021) and PiT-Tiny (Heo et al., 2021). We consider four commonly used image classifi- cation datasets: CIFAR-100 (Krizhevsky et al., 2009), Tiny- ImageNet, ImageNet-100, and ImageNet- IK (Russakovsky et al., 2015). CIFAR-100 is a well-known image classifica- tion dataset of 100 classes and the image size is 32 32. Tiny-ImageNet consists of 200 classes and the image size is 64 x 64. ImageNet-100 and ImageNet-I K contain 100 and 1000 classes of images, respectively, and the image size is 224 x 224.
Table I. Overall performance of online KD (%). Here are the
results for online KD with an interval of 1.5 epochs. The perfor- mance of different intervals can be seen in Fig. and Tab. 6.
indicates Spaced KD's 's performance gain w.r.l online KD.
Dataset
Network
w/o KD
w/o Ours
Ours
A
ResNet-18
68.12
71.05
72.43
+1.38
ResNet-50
69.62
71.85
73.77
+1.92
CIFAR-100 100
ResNet-101
70.04
72.03
73.22
+1.19
DeiI-Tiny
64.77
65.67
67.30
+1.63
PiT-Tiny
73.45
74.14
75.55
+1,41
ResNet-18
$3.08
59.19
60.75
+1.56
ResNet-50
56.41
60,99
63.30
+2.31
Imageeet
ResNet-101
56.99
61.29
63.76
+2.47
DeiT-Tiny
50.23
51.82
54.20
+2.38
Implementation. For ResNet-based architectures, we use an SGD optimizer (Sutskever et al., 2013) with 0.9 momen- tum, 128 batch size, 80 epochs, and constant learning rate of 0.01. For KD-related hyperparameters (Zhang et al., 2019), we use a distillation temperature of 3.0, a feature loss coefficient of 0.03, and a KL-Divergence loss weight of 0.3. For transformer- based architectures, we use an Adam optimizer (Loshchilov & Hutter, 2017a) of batch size 128 and epoch number 300 (warm-up for 20 epochs). Besides, cosine learning rate decay policy (Loshchilov & Hutter, 2017b) is utilized with initial learning rate 5e - 4 and final 6,-6, following the training pipeline of previous works (Liu et al., 2021; Li et al., 2022h; Sun et al., 2024).
Prf:Tiny
57.89
58.25
60.25
+2.00
ResNet-18
77.82
78.73
80.39
+1.66
ResNet-50
77.95
79.78
82.43
+2.65
ImageNet-100
DeiT-Tiny
70.52
70.72
73.34
+2.62
PiT-Timy
76.10
76.60
78.34
+1.74
Table 2. Overall performance of self KD (%). Here are the results for self KD with an interval of 4.0 epochs. A indicates Spaced KD's performance gain w.r.t self KD
Dataset
Network
w/o KI)
w/o Ours
w/ Ours
R8sNet-18
68.12
73.29
75.73
+2.44
CIFAR-100
ResNet-50
69.62
75.73
78.73
+3.00
ResNet-101
70.04
76.16
79.24
+3.08
Deit- Deit
64.77
65.24
68.26
+3.02
For Spaced KD, we manually control sparse interval s in terms of epochs, which is proportional to the total num- ber of samples in the training set (e.g., 0.5 denotes half of the training set). To avoid potential bias, the train- ing set is shuffled and both teacher and student receive the same data flow. In online KD, the teacher employs the same network architecture as the student if not spec- ified, distilling both response-based (Hinton et al., 2015) and feature-based (Adriana et al.. 2015) knowledge. In self KD, the teacher is the deepest layer of the network and the students are the shallow layers along with auxiliary classifiers (Zhang et al., 2019). Specifically, ResNet-based architectures consist of 4 blocks so 3 students correspond to the three shallower blocks. The number of students for trans- formers depends on the network depth, namely, 11 our setup. Auxiliary alignment layers and classifier heads are utilized to unify the dimensions of feature and logit vectors produced by students from different depths for distillation. Unless otherwise specified, all results are averaged over three repeats.
ResNet-18 53.08
61.08
62,83
+1.75
ResNet-50
56.41
63.58
65.80
+2,22
Tiny-ImageNet
ResNet-101
56.99
63.35
66.79
+3,44
Deit- Deit-Tiny
50.17
49.73
53.59
+3.86
ResNet-18 18
77.82
76,21
79.27
+3.06
ImageNet-100
Deit- Deit-Tiny
69.52
70.50
73.46
+2.96
enhancement of accuracy is 2.14% on average, with moder- ate variations from minimum of 1.19% on ResNet- 101 / CIFAR- to maximum of 3 3.44% on ResNet-101 / Tiny- ImageNet. For the larger dataset ImageNet- our Spaced KD improves the performance for ResNet-18 and ViT net- works by up to 5.08% (see Tab. 7, Tab. 8 of Appendix A.4).
Teacher-Student Gap. Considering that capacity gaps between teacher and student for their different architec- tures or training progress would affect distillation gains (see Sec. 2), we further evaluate various teacher-student pairs across model sizes and architectures, and Spaced KD remains effective in all cases (see Tab, 9 and Tab. 10 in Appendix A.5). Interestingly, if we train the teacher ahead of the student by s steps the beginning and then distill its knowledge to the student maintaining a constant training gap, there is no significant improvement over the online KD (see Tab. 11). This indicates the particular strength of Spaced KD, which applies in the later stage rather than the early stage.
5.2. Effectiveness and Generality of Spacing Effect
Overall Performance. Our proposed Spaced KD outper- forms traditional online KD (see Tab. 1) and self KD (see Tab. 2) across different datasets and networks. The per- formance of different intervals can be seen in Fig. 2 and Tab. 6. Compared to vanilla online KD and self KD, the
6

---

Brain-inspired spacing effeet that improves knowledge distillation.
KD in all comparisons. This property also largely avoids the computational cost and complexity of model optimization imposed by the new hyperparameter.
Table 3. Performance of Spaced KD on ResNet-18/ CIFAR- using different loss functions.
Loss Function
w/o
w/ 8 1.5
A
69.54
69.30
-0.24
Critical Period of Spaced KD. In order to better under- stand the underlying mechanisms of Spaced KD, we em- pirically investigate the critical period of implementing the proposed spacing effect. As shown in Fig. 3, we control the start time of spaced distillation throughout the training process, and discover that initiating Spaced KD in the later stage of training is more beneficial than the early stage for performance improvements of the student network. This suggests that in KD, not only the interval between learning sessions but also the timing of spaced learning are impor- tant. Unlike previous understandings that attribute the KD efficacy to the knowledge capacity gap between the teacher and the student (where Spaced KD should be more effective in the early stage of training, out novel direction for KD research from a temporal per- spective. Specifically, the "right time to learn" is critical for the student, and the teacher could influence the student's s convergence to a better solution by intervening during the later training stage.
Smooth L1
68.96
69.45
+0.49
MSE (Mean)
69.34
69.45
+0.11
MSE (Sum)
71.05
72.43
+1.38
Cross- Pross--Etropy
70.38
72.04
+1.66
Table Performance of Spaced KD on ResNet-18 CIFAR-100 using more recent KD methods.
Dutase//Moeel
Method
TSB
CTKD
LSKD
w/o KD
67.65
67.86
67.94
CIFAR-100ResNee188
w/ KD
71.70
69.41
70.74
Ours
72.82
71.12
71.76
wlo KD
51.90
53.31
52.31
CIFAR-100DeiTTTmmy
w/ KD
52.63
54.20
52.93
Sec. 2), our results point
w/ Ours
55,47
54.72
53.83
w/o KD
55.21
53.03
54.05
Iiny-ImageNe///ssNe--888
w/ KD
59.92
58.78
59.30
w/ Ours
61.65
60.32
60.28
w/o KD
40.29
40.82
39.65
TinylImageNNet/eiiTTiin
w/ KD
40.13
41.22
41.14
w/ Ours
43.36
41.60
41.48
Spaced KD online KD
online KD
Spaced KD
Different KD Losses. To evaluate generality, we imple- ment Spaced KD with representative loss functions, such as LI, smooth LI, MSE (reduction-mean)) MSE (reduc- tion=sum), and cross-entropy. As shown in Tab. 3, Spaced KD applies to different loss functions with consistent im- provements.
start start=10
60
start
ResNet:18
Final accusecy (96)
CIFAR-100
ResNes- 50
1100
ResNes. 101
Baseline
72$
72
70
50-40
00.70
70
10
(a) Period wihh Spaced KD (epachs)
(b) Period Spaced tepachs)
Different KD Methods, We combine our Spaced KD with other more advanced KD methods, including (1) TSB (Li et al., 2022a) constructs superior "teachers" with temporal accumulator and spatial integrator; (2) CTKD (Li et al., 2023) controls the task difficulty level during the student's learning career through dynamic and learnable tempera- ture; (3) LSKD (Sun et al., 2024) employs plug-and-play Z-score pre- pre-prccess of logit standardization before applying softmax and KL divergence. As shown Tab. 4, Spaced KD brings significant improvements to a wide range of KD methods. The above results suggest that the benefits of Spaced KD arise from the fundamental properties of pa- rameter optimization in deep learning, consistent with our theoretical analysis in Sec. 4.4.
Figure 3. Impact of different initiating times of Spaced KD (s 1.5). which introduced (a) for constant 10 training epochs (b) till the end of training.
Learning Rate and Batch Size. As described in previ- ous works, the learning rate and batch size influence the endpoint curvature and the whole trajectory (Frankle et al.; Lewkowycz et al., 2020; Xie et al., 2020). The learning rate corresponds to the parameters' updating step length, and batch size would affect the total number of updating iterations which directly relates to the choice of space inter- val s. Therefore, we further validate the impact of learning rate and batch size. As shown in Fig. 5 of Appendix A. we summarize the results: (i) Spaced KD proves effective w.r.t naive online KD and self KD across different learning rates; (ii) Spaced KD exhibits its advantages when training with a relatively large batch size (greater than 64). These observations also align with previous research (Jastrzebski et al., 2019; Wu et al.) regarding a small batch size limiting
5.3. Extended Analysis of Spacing Effect
Sensitivity of Space Interval. Through extensive inves- tigation (see Fig. 2 and Tab. 6 in Appendix A.2), the space interval s is relatively insensitive and s = 1.5 results in con- sistently strong improvements. Therefore, we selected it as the default choice to obtain the performance of our Spaced
7

---

Brain-inspired spacing effeet that improves knowledge distillation.
the maximum spectral norm along the convergence path found by SGD from the beginning of training.
model's generalization ability to multiple transformations that create out-of-distriuution images. Specifically, we apply 6 representative operations of image corruption (Michaelis et al., 2019) to the images of the CIFAR- 100 test set. The test accuracy at noise intensity 1.0 is recorded in Tab. 5 and results of other intensity levels can be found in Tab. 12 of Appendix A.8. It is clear that in most cases with different corruption types and network architectures, our proposed Spaced KD helps the student network resist noise attacks, which reflects its superior robustness to unseen inference situations. Besides, we test robust accuracy using a represen- tative adversarial attack method called BIM (Kurakin et al., 2017), and our Spaced KD is more robust across different ar- chitectures (see Tab. 13 in Appendix A.9). The above results empirically offer evidence for the generalization promotion brought by the spacing effect.
5.4. Generalization of Spaced KD
Flat Minima. To verify whether Spaced KD could con- verge to flat minima, we conduct experiments to ob- serve the model robustness that reflects the flatness of loss landscape around convergence, following previous works (Zhang et al., 2018; 2019). We first train ResNet- 18/50/101 networks on CIFAR- 100 with traditional online KD (w))) and our Spaced KD (w/1.5, 5, the interval is 1.5 epochs), Then Gaussian noise added the parameters of those models to evaluate their training loss and accuracy over the training set at various perturbation levels, which are plotted in Fig. 4. The results show that the model trained with Spaced KD maintains a higher accuracy and lower loss deviations than naive KD under gradient noise level. Fur- thermore, after applying this interference, the training loss of the independent model significantly increases, whereas the loss of the Spaced KD model rises much less. These results suggest that the model with Spaced KD has found a much wider minima, which is likely to result in better generalization performance.
6. Conclusion
In this paper, we present Spaced Knowledge Distillation (Spaced KD), a bio-inspired strategy that is simple yet ef- fective for improving online KD and self KD. We theoreti- cally demonstrate that the spaced teacher helps the student model converge to flatter local minima via SGD, resulting in better generalization. With extensive experiments, Spaced KD achieves significant performance gains across variety of benchmark datasets, network architectures and baseline methods, providing innovative insights into the learning paradigm of KD from a temporal perspective. Since we also reveal a possible critical period of spacing effect and provide its potential theoretical implications in DNNs, our findings may offer computational inspirations for neuroscience. By exploring more effective spaced learning paradigms and investigating detailed neural mechanisms, our work is ex- pected to facilitate a deeper understanding of both biological learning and machine learning.
Thining aarugrcc (8)
100
wio
0.016
w1.5
0.012
85
Traning loss
0.008
80
0.004
0.000
0.001
0.012
0.003
0.004
0.005
0.006
0.007
0.000
0.000
0.001
0.002
0,003
0.004
0,00g
0.006
0.807
0.008
Standard deviation of noise
Standard deviatio noise
R ResNet-1a
Training loss aoge
100
100066
Training antiinncccuyyyy
0.012
95
0.008
80
0.004
Although our approach has achieved remarkable improve- ments, it also has potential limitations: Our results suggest relatively insensitive optimal interval (s 1.5) for Spaced KD), yet remain under-explored its theoretical foundation and an adaptive strategy for determining it. Additionally, our results indicate that the timing of Spaced KD is important. The effectiveness of adaptive adjusting the space interval and the timing of distillation remains be validated and analyzed in subsequent research. In future work, we would actively explore the application of such spacing effect for broader range of scenarios, such as curriculum learning, continual learning, and reinforcement learning.
0.000
0.301
0.022
0.003
0.004
0.005
206
0.7
988
0.000
0.999
0.901
0.002
0.903
0.00
0.055
0.088
0.977
0.988
Standard devistion noise
Standard deviaton noise
[b] 5e
Training dosss g
Traningaccuany accumcyy (90)
100
8..066
98
9.008
80
0.004
0.00
0.001
002
0.005
0.004
0,77
1888
0.000
0.000
0.007
0.002
0.003
0.004
0.00
0.007
one
Standard deviation of noise
Standard deviation of noise
(c) Re1
Figure 4. Impact of Gaussian noise performance.
A Acknowledgements
This work was supported by the ST12030-Major Projects (20227D0204900 to Y.Z.), the NSFC Projects
Noise Robustness. In addition to manipulating network parameters, we conduct an extra experiment to evaluate the
8

---

Brain-inspired spacing effeet that improves knowledge distillation.
Table Comparison accuracy under image corruption attack (%). A indicates Spaced KD's performance gain w.r.t online KD. The intensity of noise is 1.0 and the results of other intensities (i.e., 3.0, 5.0) can be seen in Tab. 12 of Appendix. A.8.
ResNet-11
ResNet-50
ResNet-101
Method
Attack
w/o Ours
w/ Ours
A
w/o Ours
w/ Ours
A
w/o Ours
w/ Ours
A
impulse repulse.nnciee sees
52.92
54.15
1.23
55.41
57.19
1.78
55.66
57.17
1.51
zoom.blur
66.45
67.43
0.98
66.86
68.53
1.67
66.20
66.53
0.33
9new
57.28
59.05
1.77
59.19
59,55
0.36
57.82
58,38
0.56
Online KD
Irost
57.55
59.64
2.09
60.16
59.93
-0.23
58.57
59.63
1.06
jpag compression
41.12
40.23
-0.89
42.22
42.65
0.43
42.69
43.70
1.01
brightness
67.01
69.06
2,05
68.31
69.15
0.84
66.82
67.45
0.63
impulse.noise
50.65
60.57
9.92
62,18
71.57
9.39
59.33
68,78
9.45
zoom.blur
64,44
68.60
4.16
68.03
72.13
4.10
66.09
71.16
5.07
Self KD
snew
61.30
66.14
4.84
64.72
69.65
4.93
64.03
68.76
4.73
frost 63.96
67.80
3.84
66.36
71.81
5.45
66.73
70.21
3.48
jpeg compressior.
30.99
34.67
3.68
34.44
35.34
0.90
33.64
34.76
1.12
brightness 73.18
75.92
2.74
74.91
79.19
4.28
75.10
78.91
3.81
(Nos. 62406160 to L.W., 32021002 to Y.Z.), and National Science and Technology Major Project (2022ZD01 163013 to B.L.). L.W. is also supported by the Postdoctoral Fellow- ship Program of CPSF under Grant Number GZB20230350 and the Shuimu Tsinghua Scholar.
Bello-Medina, P. C., Sánchez-Carrasco, L., González- Ornelas, N. R., Jeffery, K. J., and Ramírez-Amaya, V. Differential effects of spaced massed training in long- term object-identity and object-location recognition mem- ory. Behavioural Brain Research, 250: 102-1 113, 2013.
Blanc, G., Gupta, N., Valiant, G., and Valiant, P. Implicit reg- ularization for deep neural networks driven by ornstein- uhlenbeck like process. In Conference on Learning The- ory, pp. 483-513. PMLR, 2020.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, D., Mci, J.-P.. Wang, C., Feng. Y., and Chen, C. On- line knowledge distillation with diverse peers. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3430-3437 2020.
References
Adriana, R., Nicolas, B., Ebrahimi, K. S., Antoine, C., Carlo, G., and Yoshua, B. Fitnets: Hints for thin deep nets. International Conference on Learning Representations, 2 33:1,2015
Cho, J. H. and Hariharan, B. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, October 2019.
Damian, A., Ma, T., and Lee, D. Label noise sgd prov- ably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449-27445 2021.
Allen-Zhu, Z. and Li, Y. Towards understanding ensem- ble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:20/2.098 2020.
Das, R. and Sanghavi, S. Understanding self-distillation in the presence of label noise. In International Conference on Machine Learning, pp. 7102-7140. PMLR, 2023.
Anderson, M. J.. Jablonski, S. A., and Klimas, D. B. Spaced initial stimulus familiarization enhances novelty prefer- ence in long-evans rats. Behavioural Processes, 78(3): 481-486, 2008
Dinh, L., Pascanu, R., Bengio, S.,, and Bengio, Y. Sharp min- ima can generalize for deep nets. In International Con- ference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Asif, U., Tang, J., and Harrer, S. Ensemble knowledge distillation for learning improved and efficient networks. arXiv preprint arXiv: 1XXv:1909.000 09097 2019.
Donovan, J. J. and Radosevich, D. J. A meta-analytic review of the distribution of practice effect: Now you see it, now you don't. Journal of Applied Psychology, 84(5):795, 1999.
Beck, C., Schroeder, B., and Davis, R. L. Learning per- formance of normal and mutantdrosophila after repeated conditioning trials with discrete stimuli. Journal of Neu- roscience, 20(8):2944 2953, 2000.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.,
9

---

Brain-inspired spacing effeet that improves knowledge distillation.
Heigold, G., Gelly, S., al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1199 2020.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 054000 2018.
Ebbinghaus, H. Memory: A contribution to experimental psychology. Annals of Neurosciences, 20(4):155, 2013.
Jastrzebski, S., Kenton, Z., Ballas, N., Fischer, A., Ben- gio, Y., and Storkey, A. J. On the relation between the sharpest directions of dnn loss and the sgd step length. In International Conference on Learning Representations, 2019.
Frankle, J., Schwab, D. J., and Morcos, A. S. The early phase of neural network training. In International Con- ference on Learning Representations.
Furlanello. T., Lipton, Z., Tschannen, M., Itti, L., and Anandkumar, A. Born again neural networks. In Interna- tional Conference on Machine Learning, pp. 1607-1616. PMLR, 2018.
Kang, M., Mun, J., and Han, B. Towards oracle knowledge distillation with neural architecture search. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4411, 2020.
Gao, M., Shen, Y., Li, Q. Yan, J., Wan, L., Lin, D., Loy, C. C., and Tang. An embarrassingly simple approach for knowledge distillation. arXiv preprint arXiv:1812.018899 2018.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang. T. P. On large- training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: (00.66699.888. 2016.
Gluckman, M., Vlach, H. A., and Sandhofer, C. M. Spacing simultaneously promotes multiple forms of learning in children's s science curriculum. Applied Cognitive Psy- chology, 28(2):266-25) 277223,,,,, 2014.
Kim, K., Ji, B., Yoon, D., and Hwang, S. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 6567-6576, 2021.
Gou, J., Yu, B., Maybank, J., and Tao, D. Knowledge distillation: A survey. International Journal of Computer Vision, 12966) 1819, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Gu, and Tresp, Search for better students to learn distilled knowledge. arXiv preprint 1611, 2020.
Kurakin, A., Goodfellow, I., and Bengio, S. Adversarial examples in the physical world, 2017. URL https: / sarxiv..rg//bee. 107.007.000 02533.
Hc, H., Huang, G., and Yuan, Y. Asymmetric valleys: Be- yond sharp and flat local minima. Advances Neural Information Processing Systems, 32, 2019.
Landauer, T. K. Reinforcement as consolidation. Psycho- logical Review, 76(1):82, 1969.
Lee, H., Hwang, S. J., and Shin, J. Rethinking data augmen- tation: Self-supervision and self-distillation. 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Lewkowyez, A., Bahri, Y., Dyer, E., Sohl-Dickstein, J., and Gur-Ari, G. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022:. 022118 2020.
Hco, B.. Yun, S., Han, D., Chun, S., Choe, J., and Oh, S. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 1193-119453 2021.
Li, C., Wang, Z., and Qi, H. Online knowledge distilla- tion by comporal-ppatie boosting. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 197-206, 2022a.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503. 02531,, 2015.
Li, K., Yu, R., Wang, Z., Yuan, L., Song, G., and Chen, J. Locality guidance for improving vision transformers on tiny datasets. In European Conference on Computer Vision, pp. 110-127. Springer, 2022b.
Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. Advances Neural Information Processing Systems, 1994.
Huang, Z. and Wang, N. Like what you like: Knowl- edge distill via neuron selectivity transfer. arXiv preprint arXiv: 1707.0707 2017.
Li, L. and Jin, Z. Shadow knowledge distillation: Bridging offline and online knowledge transfer. Advances in Neural Information Processing Systems, 35:635 649, 2022.
10

---

Brain-inspired spacing effeet that improves knowledge distillation.
Li, S., Lin, M., Wang. Y., Wu, Y., Tian, Y., Shao, L.,, and Ji, R. Distilling powerful student model via online knowl- edge distillation. IEEE Transactions on Neural Networks and Learning Systems, 34(11):8743-875) 2022c.
Mobahi, H., Farajtabar, M., and Bartlett, P. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Li, Z., Li, X., Yang, L.,, Zhao, B., Song, R., Luo, L., Li, J., and Yang, J. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504-1512, 2023.
Pagani, M. R., Oishi, K., Gelb, B. D., and Zhong, Y. The phosphatase shp2 regulates the spacing effect for long- term memory induction. Cell, 139(1):186-198, 2009,
Peterson, L. R. Short-term verbal memory and learning. Psychological Review, 73(3):193, 1966.
Liu, Y., Jia, X., Tan, M., Vemulapalli, R., Zhu, Y., Green, B., and Wang, X. Search to distill: Pearls are everywhere but not the eyes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020a.
Qian, B., Wang, Y., Yin, H., Hong, R., and Wang. M. Switch- able online knowledge distillation. In European Confer- ence on Computer Vision, pp. 449444444444 466. Springer, 2022.
Liu, Y., Jia, X., Tan, M., Vemulapalli, R., Zhu, Y., Green, B., and Wang, X. Search distill: Pearls are everywhere but not the eyes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7539- 7548, 2020b.
Rezagholizadeh, M., Jafari, A., Salad, P., Sharma, P., Pasand, A. S., and Ghodsi, A. Pro-kd: Progressive distillation by following the footsteps of the teacher. arXiv preprint arXiv:2 10.08332, 2021.
Robbins, D. and Bush, T. Memory in great apes. Journal of Experimental Psychology, 97(3):344, 1973.
Liu, Y. Sangineto, F., Bi, W., Sebe, N., Lepri, B., and Nadai, M. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing 2021.
Roediger, H. L. and Byrne, J. Learning and memory: A comprehensive reference (vol. 2). Cognitive Psychology of Memory, 2008.
Systems, 34:23818-255
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv: 2017a.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition chal- lenge. International Journal of Computer Vision, 115:
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts, 2017b.
211 211-252,22 2015.
Medin, D. The comparative study of memory. Journal of Human Evolution, 3(6):455-463, 1974.
Shea, C. H., Lai, Q.. Black, C.,, and Park, J.-H. Spacing prac- tice sessions across days benefits the learning of motor skills. Human Movement Science, 19(5):3377700, 2000.
Menzel, R., Manz, G., Menzel, R. and Greggers, U. Massed and spaced learning in honeybees: the role of cs, us, the intertrial interval, and the test interval. Learning & Memory, 8(4):198-2)) 1900,,,,--...... 2001.
Shen, Y., Xu, L., Yang, Y., Li, Y., and Guo, Y. Self- distillation from the last mini-batch for consistency regu- larization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11943- 11952, 2022.
Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bring- mann, O., Ecker, A. S., Bethge, M., and Brendel, W. Benchmarking robustness in object detection: Au- tonomous driving when winter is coming, arXiv preprint arXiv:1907.07484 2019.
Shi, W., Song, Y., Zhou, H., Li, B., and Li, L. Follow your path: a progressive method for knowledge distilla- tion. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III 21, pp. 596-611. Springer, 2021.
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.
2020a.
Smolen, P., Zhang, Y., and Byrne, J. H. The right time to learn: mechanisms and optimization of spaced learning. Nature Reviews Neuroscience, 17(2):77-88, 2016.
Mirzadeh, S. I., Farajtabar, M., Li, A., Levine, N., Mat- sukawa, A., and Ghasemzadeh, H. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5191-5198, 2020b.
Sobel, H. S., Cepeda, N. J., and Kapler, I. V. Spacing effects in real-world classroom vocabulary learning. Applied Cognitive Psychology, 25(5):763-767, 2011.

---

Brain-inspired spacing effeet that improves knowledge distillation.
Sun, S., Ren, W., Li, J., Wang, R., and Cao, X. Logit standardization in knowledge distillation. arXiv preprint arXiv:2403.01427, 2024.
Zhang, L.., Song, J., Gao, A., Chen, J., Bao, C., and Ma, K. Be your teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2019.
Sutskever, I., Martens, J., Dahl, G., and Hinton, On the importance of initialization and momentum in deep learn- ing. In International Conference on Machine Learning, pp. 1119-11444 PMLR, 2013.
Zhang, Y., Xiang, T., Hospedales, T. M., and Lu, H. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320 4328, 2018
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- & distillation through attention. In International Con- ference Machine Learning, pp. 10347-10357. PMLR, 2021.
Zhang, Z. and Sabuncu, M. Self-distillation instance- specific label smoothing. Advances in Neural Information Processing Systems,
2195, 2020.
Vlach, H. A. The spacing effect in children's generalization of knowledge: Allowing children time to forget promotes their ability to learn. Child Development Perspectives, 8 (3):163-168, 2014.
Zhao, B., Cui, Q. Song. R., and Liang, J. Dot: A distillation- oriented trainer. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 6189-6198. 2023.
Wang, I.. and Yoon, K.-J. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions Pattern Analysis and Machine Intelligence, 4446)33048,3068, 2021.
Zhou, P., Feng. J., Ma, C., Xiong, C., Hoi, C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285 2020.
Wickelgren, W. A. Trace resistance and the decay of long- term memory. Journal of Mathematical Psychology, 9(4): 418-455, 1972.
Zhu, X., Gong, S., et al. Knowledge distillation by on-the- fly native ensemble. Advances in Neural Information Processing Systems, 31, 2018.
Wu, G. and Gong, S. Peer collaborative learning for online knowledge distillation. Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pp. 10302- 10310, 2021.
Wu, L.,, Wang, M., and Su, W. The alignment property of SGD noise and how it helps select flat minima: A stability analysis. pp. 4680-4693.
Xie, Z., Sato, I., and Sugiyama, M. A diffusion the- ory for deep learning dynamics: Stochastic gradient de- scent exponentially favors flat minima. arXiv preprint arXiv:2002.03499 034900 2020.
Yan, H., Wang, L., Ma, K., and Zhong, Y. Orchestrate latent expertise: Advancing online continual learning with multi-level supervision and reverse self disttilllaiin. arXiv preprint arXiv:2404.00 004/7, /7, 2024.
Yang, C., Xie, I.., Su, C., and Yuille, A. L.. Snapshot distilla- tion: Teacher-student optimization in one generation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2859-2868, 2019.
Zhai, M., Chen, L.,, Tung, F., He, J., Nawhal, M., and Mori, G. Lifelong gan: Continual learning for conditional im- age generation. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 2769- 2019.
12

---

Brain-inspired spacing effeet that improves knowledge distillation.
A. Appendix
A.1. Proof of Theorem 4.4
Proof. For a general KD loss, we have the trace of its Hessian matrix at global minimizer 0":
N
1
Tr(H)(00) =
|I ff- (zx))l B(i, 0" .00.0000)T T)Tr (x.)))
N
(8)
N
1
e (z,) aso Vags (z, Tr (v))
For online KD and Spaced KD, the expectation their Hessian trace should be:
N Ao V (c))
(TTTHHOO = -off
a N
(9)
(T = E, Volo (z:)
E
Vola (2,) (V
(10)
By Lemma 4.3, 1 N E; B|I V Voga. (z:)1] S El
の V Voge (z;))]-
N
D Va Vāfe- (x:))]
E [o Vog (z:)Tr (VāJo-
Substituting the above inequality into Eq. and Eq. 10 completes the proof.
A.2. Performance of Different Intervals for Online KD
Table 6, Overall performance of different intervals for Fig. 2 and Tab.
Dataset
Network
Baseline
w/o
w/0.5
w/1.0
w/1.5
w/2.0
w/max
ResNet-18
68.12
71.05
72.02
72.03
72.43
72.18
72.22
ResNet-50
69.62
71.85
73.39
73.25
73.77
73.28
73.35
CIFAR-100
ResNet-101
70.04
72.03
73.11
73.22
72.91
73.22
74.01
DeiT-Tiny
64.77
65.67
66.03
66.22
67.30
66.45
65.69
PiT-Timy
73.45
74.14
75.55
75.50
75.27
75.12
74.07
ResNet-18
53.08
59.19
59.62
59.68
60.75
59.52
59.34
ResNet-50
56.41
60.99
62.13
62.27
63.30
62.55
62.47
ImageNe
ResNet-101
56.99
61,29
62.70
62.64
63.76
62.80
63.10
DeiT-Tiny 50.23
51.82
54.20
53.55
52.92
53.48
52.21
PiT-Tiny
57.89
58.25
59.45
59.77
60.25
59.75
58.23
A.3. Implementation of SOTA methods with Spaced KD
For traditional KD methods (BAN (Furlanello al., 2018), TAKD (Mirzadeh et al., 2020a)) and online KD methods (DML (Zhang et al., 2018) and SHAKE (Li & Jin, 2022)), we preserve their basic training frameworks for reproducing results in w/o KD (raw ResNet-18 18 training) and KD (ResNet-18 with the corresponding method) columns and delay the students' supervised learning and distillation by a space interval of 1.5 epochs for w/ Ours. For self KD methods (DLB (Shen et al., 2022) and PSKD (Kim et al., 2021)), we initiate a student network identical to the teacher. We train the teacher model utilizing PSKD or DLB, and the student model trained either online or spaced style with an interval of
13

---

Brain-inspired spacing effeet that improves knowledge distillation.
1.5 epochs. Specifically, the results w/o KD of PSKD and DLB in Tab. 4 are the performance of the teacher model, w/ the performance of online students, and w/ Ours corresponds spaced students. Because follow the exact training pipeline (including learning rate scheduler, optimizer, and dataset transformation, etc) of those works when reproducing their results, which is different from that of Tab. and Tab. 2, the baselines without KD may be different.
A.4. Performance of Spaced KD on ImageNet-1k
Table Performance of ResNet-18 ImageNet- Ik Dataset (space interval 1.5 epochs).
Epoch (epoch)
20
40
60
80
online KD
44.35
45.94
46.62
48.59
online KD w/ Ours
49.28
51.35
52.36
53.67
self KD self KD w/ Ours
44.87
46.78
47.78
47.99
47.03
49.94
50.22
51.57
Table 8. Performance Deit-Tiny on ImageNet- Ik Dataset (space interval 1.5 epochs).
Epoch (epoch)
100
200
300
online KD online KD w/ Ours
58.18
65.93
72.04
58.47
66.54
72.34
self KD self KD w/ Ours
58.81
66.37
72.39
60.82
67.27
73.69
A.5. Performance of Spaced KD on Different Teacher-Student Architectures
Table 9. Overall performance of student networks distilled from different teachers CIFAR-100. We use ResNet- 18 as the student.
Teacher
Baseline
Online KD
Spaced KD
ResNet-18x2
69.40
71.77
72.77
Width
ResNet-18x4
70.75
72.17
73.11
ResNet-18x8
70.77
72.03
73.52
ResNet-50
69.21
72.18
73.49
Depth
ResNet-101
69.54
71.61
73.04
DeiT-Tiny
64.65
78.61
79.38
Architecture
PiT-Tiny
73.78
77.13
78.77
Table 10. Comparison of Spaced KD and offline KD from different teacher-student pairs CIFAR-100. We use ResNet-18 the student.
Teacher
Offline KD
Spaced KD
ResNet-18x2
72.53
72.77
Size
ResNet-18xx
72.83
73.11
ResNet-18
73.04
73.52
DeiT-Tiny
78.80
79.38
Architecture
Pit-Tiny
78.50
78.77
14

---

Brain-inspired spacing effeet that improves knowledge distillation.
A.6. Performance of Student Distilled from Constant Ahead Teacher
Table 11. Performance of ResNet-18 on CIFAR-100 distilled from trained teacher with constant 8 step ahead. There are no significant improvements the online KD.
Interval (epoch)
0
0.5
1.5
2.5
CIFAR-100
71.05
70.67
70.85
70.69
71.04
70.78
Here we consider a naive baseline of implementing the proposed spacing effect. Specifically, we first train the teacher model for 8 steps and then transfers knowledge to the student model at each step during the following training time. other words, the teacher model keeps constant s steps ahead of the student model. However, such a naive baseline exhibits no significant improvement
online KD (see Table 11), consistent with empirical analysis (see Fig. 3) and theoretical analysis (see Sec. 4.2): The teacher model of Spaced KD can provide stable informative direction for optimizing the student model after each steps, whereas the teacher model of the naive baseline fails in this purpose due to its ongoing changes when optimizing the student model. Such different effects also suggest that the implementation of spacing effect is highly non-trivial and requires specialized design as our Spaced KD.
A.7. Performance of Spaced KD using different learning rate and batch size
Accuracy (%)
75
72
w/0.5
w/o
Accuracy (%)
70
w/1.0
70
w/1.5
65
w/2.5
68
0.001
0.005
0.01
0.05
0.1
66
0.
32
64
128
256
512
(a)
Learning rate
(b) Batch size
Figure Hyperparameter validation for Spaced KD. Accuracy of different learning rate (a) and batch size (b) gradient intervals.
A.8. Performance of Spaced KD on Different Image Corruption Attacks
Here we visualize 6 representative image corruption operations (Michaelis al., 2019) applied to the images from the CIFAR-10K 100 dataset (Krizhevsky et al., 2009) to assess models' robustness and generalization ability in Fig. 6. The accuracy under adversarial attacks with more noise intensity levels is listed in Tab. 12.
15

---

Brain-inspired spacing effeet that improves knowledge distillation.
impulse_naie
zoom_blur blur
frost
peggcoopresoion
brightness
intens ity 1.0
original
Intensity 3.0
intens
Figure 6. Image corrppiio operation. We choose 6 representative image corruption operations with different severity (1.0, 3.0. 5.0) and visualized images come from the CIFAR-100 test set.
A.9. Performance of Spaced KD after Adversarial Attack
A.10. Pseudo Code of Online KD, Self KD and Spaced KD
Algorithm Training Algorithm of Online KD
Require: student fo, teacher Pø. dataset Prriis KD loss weight epoch number E Ensure: train both teacher and student using online knowledge distillation
E do 2: for (24, 3/) Durain do
for
3: Update teacher <- ạb V (
4: Update student + 0 - Vo (fox;) (xx)) + (1 adlus (fo 9:)]
5: end for
6: end for
16

---

end for Brain-inspired spacing effeet that improves knowledge distillation.
Algorithm 2 Training Algorithm of Online KD with Spaced KD
Require: student fe, teacher Sø. dataset Duain KD loss weight epoch number E, space interval Ensure: train both teacher and student using spaced knowledge distillation
Initialize data index set: R <- y
2: for S E do
3: for (2,, 3/) Durii do
4: R +- R {i}
5: Update teacher Φ -
Volus v.)
6:
[R] = 8 then
7: for j E R do
8:
Retrieve (zy, M5) from Duain
9:
Update student / + Ve (o) + (1 (fo
end for 11: end if
10:
12: Clear index set: R f- 0
13: end for
14:
end for
Algorithm 3 Training Algorithm of Self KD
Require: network fe fer ..... fa.. consisting of m blocks, dataset D KD loss weight epoch number E Ensure: train So by distilling logits from the last block to the shallower blocks
for S e do
2: for (In; 344) D irain do
3: Calculate loss I, lusk (Jo (fo()))))))
4:
for 1 < k < do
5:
L L + alkD (fo, ... for (x:). fo():)) + ai)uss .... for (x)).
6: end for
7: Update network +-6 8 - VoL
8: end for
9:
Table 12, Comparison of accuracy under image corruption attack (%).
indicates Spaced KD's 's increased performance based on online
KD. The results of 1.0 intensity be in Tab.
R8
ResNet-50
ResNet-101
Attack
Noise Intensity
w/o Ours
w/ Ours
w/o Ours
w/ Ours
A
w/o Ours
w/ Ours
A
3.0
34.19
35.33
1.14
35.41
36.53
1.12
37.56
38.16
0.60
impulse noise
12.54
12.04
-0.50
10.49
10.57
0.08
12.08
11.39
-0.69
3.0
64.73
65.29
0.56
65.00 15.04
66.45
1.41
64.5
64.98
0.48
zoom_blur
61.02
61.53
61.36
62.67
1.31
61.32
62.18
0.86
44.48
45.42
46.91
47.17
0.26
45.87
1.37
5.0
28.60
29.48
0.88
30.09
29.71
-0.38
30.09
30.75
0.66
42.40
43.10
0.70
44.87
44.69
-0.18
45.10
45.28
0.18
frost
5.0
37.80
39.47
1.67
39.26
39.97
0.71
41.24
40.59
-0.65
3.0
33.23
32.32
-0.91
33.05
33,99
0.94
34.80
35.63
0.83
jpeg.compression
5.0
20.75
21.32
0.57
20.29
20.86
0.57
21.55
22.29
0.74
3.0
62.77
64.68
1.91
64.48
64,63
0.15
62.90
64.01
1.11
bri brightness
5.0
54.11
54.56
0.45
55.34
55,46
0.12
$4.47
55.71
1.24
17

---

Brain-inspired spacing effeet that improves knowledge distillation.
Table 13. Performance of Spaced KD CIFAR- 100 after adversarial attack called BIM (Kurakin et al., 2017). Spaced KD is robust than online KD.
Network
ResNet-18
ResNet-50
ResNet-101
w/o
31.33
31.32
31.70
w/1.5
31.44
31.70
33.69
A
+0.11
+0.38
+1.99
Algorithm 4 Training Algorithm of Self KD with Spaced KD
Require: network So = So. fo.. consisting of m blocks, dataset Derins KD loss weight a, epoch number E, space
interval
Ensure: train fo by distilling logits from the last block shallower blocks in spaced manner
1: Initialize data index set: R <-
2: for ISe S E do
3: for (I,, y4) Derain do
4: R + RU (i)
5: Calculate loss L = lusk (fo (x:), Vi)
6: Update network 0 +- 0 - V
7: if [ = then
8:
for j E R do
9:
Retrieve (Iy, from Duain Caleulate loss T/' lank (fo(1,),
m do for L' L' + afk o... for (2z). (1 a)lunk(fo, .... for (2,).,)))
12:
13:
end for
14:
Update network 0 + VoL' end for
15:
16: end
Clear index set: R
end for
19:
end for
18


Reviews:
[{"review_id": "ys2ME6Ng01", "reviewer": "Reviewer_bqdc", "summary": "This paper proposes a space KD strategy, which is inspired by spacing effect in biological learning and memory. Overall, the experiments verified the effectiveness when comparing the proposed space KD with self KD(Zhang et al., 2019). However, there are many KD methods been proposed in recent years, it missing comparisons with SOTA KD methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We provide a point-to-point response as follows. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.\n\n**Q1: Review, comparison, and compatibility experiment on latest KD methods.**\n\nFollowing your suggestion, we have included more recent KD methods. Here we discuss some representative ones: TSB [1] constructs superior \"teachers\" with temporal accumulator and spatial integrator. CTKD [2] controls the task difficulty level during the student’s learning career through a dynamic and learnable temperature. LSKD [3] employs a plug-and-play Z-score pre-process of logit standardization before applying softmax and KL divergence.\n\nAs shown in the following table, our Spaced KD can be combined with these methods and provide significant improvements under two datasets and two architectures.\n\n|Dataset/Model|Method|TSB [1]|CTKD [2]|LSKD [3]|\n|-|-|-|-|-|\n|CIFAR-100/ResNet-18|w/o KD|67.65|67.86|67.94|\n| |w/ KD|71.70|69.41|70.74|\n| |w/ Ours|**72.82**|**71.12**|**71.76**|\n|CIFAR-100/DeiT-Tiny|w/o KD|51.90|53.31|52.31|\n| |w/ KD|52.63|54.20|52.93|\n| |w/ Ours|**55.47**|**54.72**|**53.83**|\n|Tiny-IN/ResNet-18|w/o KD|55.21|53.03|54.05|\n| |w/ KD|59.92|58.78|59.30|\n| |w/ Ours|**61.65**|**60.32**|**60.28**|\n| Tiny-IN/DeiT-Tiny|w/o KD|40.29|40.82|39.65|\n| |w/ KD|40.13|41.22|41.14|\n| |w/ Ours|**43.36**|**41.60**|**41.48**|\n\n[1] Online Knowledge Distillation by Temporal-Spatial Boosting. WACV, 2022.\n[2] Curriculum Temperature for Knowledge Distillation. AAAI, 2023.\n[3] Logit Standardization in Knowledge Distillation. CVPR, 2024.\n\n**Q2: Selection of intervals for self KD and online KD.**\n\nWe theoretically demonstrate that there exists a desirable temporal interval to improve generalization of online KD (as well as self KD) and empirically investigate the selection of specific values (see online KD in Table 6, and self KD in the following table). We find that the desirable temporal interval for a given KD paradigm is relatively stable across different network architectures and benchmark datasets. \nSince the SGD-induced variability of online KD (targeting the entire network) is larger than that of self KD (targeting a few network blocks), the latter requires a larger temporal interval to aggregate such variability to obtain an appropriate teacher-student gap. We therefore select s=1.5 for online KD and s=4.0 for self KD as the default implementation.\n\nTemporal interval of self KD:\n|Interval (epochs)|0|1|2|3|4|8|\n|-|-|-|-|-|-|-|\n|CIFAR-100/ResNet-18|73.25|74.27|75.15|74.30|75.73|76.41|\n|CIFAR-100/ResNet-50|75.73|76.67| 79.27|79.89|79.43|79.44|\n|CIFAR-100/ResNet-101|76.16|75.97| 79.02|79.27|79.24|79.64|\n\n**Q3 & Q4: More discussion of ``Teacher-Student Gap'' in online KD and its failure analysis.**\n\nWe agree that the capacity gap between teacher and student is critical to the online KD performance. In previous studies of (online) KD, the teacher-student gap is regulated in various dimensions, such as the training data, network architecture, innate randomness (random initialization and SGD), etc. In this work, our main motivation lies in improving generalization of the model itself with **identical** network architectures, random initialization and data sources between teacher and student. We therefore characterize the teacher-student gap in online KD into the interplay of the SGD-induced variability and the temporal interval that aggregates such variability. \n\nTheoretically, we demonstrate that a proper temporal interval between teacher and student (i.e., a proper teacher-student gap) helps the model to find flatter local minima using SGD. This is because the teacher that is slightly ahead in training provides a well-defined trajectory, ensuring low errors along so-called informative direction to improve generalization. In contrast, the naive SGD only ensures low errors in random directions around the convergence point, which limits the \"radius\" of loss flatness (see Sec.4.2).\n\nEmpirically, we perform extensive experiments to validate our proposal in online KD across a range of datasets and architectures. Aligned with our theoretical analysis, the proposed Spaced KD is most effective only when the temporal interval is set to an appropriate value (Fig.2 and Table 6). When the temporal interval is too large, the spaced version of online KD is closer to offline KD and the improvement is compromised. This is because the too large teacher-student gap makes them converge to different local minima, which violates the assumption of our theoretical analysis. \n\nIn the absence of differences in other spatial elements that produce the teacher-student gap, the effectiveness of our Spaced KD is regulated by the temporal interval in SGD. We believe that the effectiveness of our Spaced KD will be affected if the teacher-student gap produced by such spatial elements is too large, which is out of scope of the current focus.", "claims_and_evidence": "Overall the claims are well supported. However, whether the space KD is compatible with other SOTA KD methods is not well analyzed.", "methods_and_evaluation_criteria": "The evaluation criteria is reasonable", "theoretical_claims": "The theoretical basis is the  spacing effect in biological learning and memory. I think it is correct.", "experimental_designs_or_analyses": "1. Missing comparisons with SOTA KD methods\n\n2. Missing experiments on whether the propose strategy is compatible with SOTA KD methods.", "relation_to_broader_scientific_literature": "No problems with this part.", "essential_references_not_discussed": "Missing review on latest KD methods.", "other_strengths_and_weaknesses": "1. Missing comparisons with SOTA KD methods\n\n2. Missing experiments on whether the propose strategy is compatible with SOTA KD methods.\n\n3. Missing review on latest KD methods.\n\n4. In online KD, the knowledge distillation performance will be affected by the capacity gap. The discussion in 'Teacher-Student Gap' is shallow. It is not clear when and why the Spaced KD is not effective in online KD.\n\n5. Missing discussions on the selection of Intervals for self KD and online KD. \n\n6. Missing failure analysis. It is not clear when the spaced KD not works.", "other_comments_or_suggestions": "See [Other Strengths And Weaknesses]", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "guiNYbMG2C", "reviewer": "Reviewer_C5Ts", "summary": "The paper introduces Spaced Knowledge Distillation (Spaced KD), a novel method drawing on the biological spacing effect to enhance generalization in online and self-knowledge distillation by inserting intervals between teacher and student training steps. It makes notable contributions: firstly, the bio-inspired strategy of adding temporal intervals between teacher updates and student distillation, which results in flatter loss landscapes ; secondly, the theoretical analysis that shows Spaced KD converges to flatter minima through Hessian trace analysis; and lastly, the empirical validation that reveals significant performance improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We provide a point-to-point response as follows. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.\n\n**Q1: ImageNet-1K results are less comprehensive in Table 7.**\n\nWe would respectfully point out that the ImageNet-1K results are presented in both Table 7 and Table 8, including different KD paradigms (online KD and self KD) and network architectures (ResNet-18 and Deit-Tiny). Our Spaced KD provides improvements in all cases.\n\n**Q2: Evaluation of more recent KD methods with more scenarios.**\n\nAs shown in the response to **Reviewer bqdc's Q1**, our proposed Spaced KD brings significant improvements to a range of more recent KD methods under two datasets and two architectures. We will add these results in the final version.\n\n**Q3: The proof (Appendix A.1) assumes over-parameterized models and linearized dynamics, which may not fully capture real-world DNN training.**\n\nWe would respectfully argue that state-of-the-art DNNs are often **over-parameterized** to improve generalization. This property results in multiple local minima with similar training errors but different testing errors, often reflected in the flatness of loss landscape. We therefore adopt the over-parameterization assumption to analyze how to converge to a flatter loss landscape.\n\nAlso, we would point out that we assume **local linearization** around the convergence point, rather than global linearization. As shown in previous work [1], SGD eventually selects a loss minima with linear stability (i.e., low errors with moderate linear disturbance), therefore ensuring flat loss landscape and improving generalization. \nWe will add more explanations to these two assumptions in the final version.\n\n[1] The alignment property of SGD noise and how it helps select flat minima: A stability analysis. NeurIPS, 2022.\n\n**Q4: Some details (e.g., hyperparameters for transformer training) are missing.**\n\nFor transformer training, we adopt the well-established training pipeline tailored for the benchmark datasets [1-3]. We will add more detailed descriptions in the final version.\n\n[1] Efficient Training of Visual Transformers with Small Datasets, NeurIPS. 2021.\n[2] Locality Guidance for Improving Vision Transformers on Tiny Datasets. ECCV, 2022.\n[3] Logit Standardization in Knowledge Distillation. CVPR, 2024.\n\n**Q5: One related work [1] is not cited.**\n\nThanks. We have conceptually analyzed this work and empirically demonstrated Spaced KD's plug-in benefit on it (see response to **Reviewer bqdc's Q1**).\n\n**Q6: The pseudo-code (Appendix A.10) lacks implementation details (e.g., gradient accumulation for interval s).**\n\nWe would clarify that our proposed Spaced KD inherently avoids gradient accumulation through its dual-loop design. The hyperparameter s (i.e., temporal interval) operates as a temporal decoupler rather than a gradient accumulation window. With the outer loop, the teacher model updates its parameters immediately after each batch (Algorithm 2, line 5), with no gradient retention. With the inner loop (when $|\\mathcal{R}=s$), the teacher’s parameters remain frozen while the student updates using the cached samples, ensuring no backward passes occur through the teacher in this phase. This design strictly segregates the gradient flows: the teacher’s gradients are computed and applied instantly in the outer loop, while the student’s gradients are confined to the inner loop without cross-interval persistence. We will add more explanations to make this clearer.\n\n**Q7: The \"temporal interval\" is more precise than \"space interval\".**\n\nFollowing your suggestion, we will modify our expression for better clarity.\n\n**Q8: Why is the optimal interval s = 1.5 epochs (Page 7)? Is this dataset-dependent? A theoretical justification is missing.**\n\nWe theoretically demonstrate that there exists a desirable temporal interval to improve generalization of online KD (as well as self KD) and empirically investigate the selection of specific values (see online KD in Table 6, and self KD in the response to **Reviewer bqdc's Q2**). We find that the desirable temporal interval and its effectiveness is related to the strength of SGD-induced variability (target of KD paradigms, learning rate, batchsize, etc.), and is relatively **insensitive** to different network architectures and benchmark datasets. \n\nSince the SGD-induced variability of online KD (targeting the entire network) is larger than that of self KD (targeting a few network blocks), the former requires a smaller temporal interval to aggregate such variability to obtain an appropriate teacher-student gap. We therefore select s=1.5 for online KD and s=4.0 for self KD as the default implementation. We further investigate the impact of learning rate and batch size in Fig.5, which together with the temporal interval affect the overall performance.", "claims_and_evidence": "Performance improvements (Tables 1–2) are backed by extensive experiments.  \nFlat minima hypothesis is validated via noise robustness tests (Fig. 4, Page 8) and Hessian analysis (Sec. 4.2).", "methods_and_evaluation_criteria": "Methods: Spaced KD introduces a temporal interval  s  (e.g., 1.5 epochs) between teacher updates and student distillation, compatible with existing KD frameworks (Algorithms 1–3, Appendix).  \nEvaluation: Standard benchmarks (CIFAR-100, Tiny-ImageNet) and metrics (test accuracy) are appropriate. However, ImageNet-1K results are less comprehensive (Table 7, Page 14).  The architectures validated in the experiments mainly include variants of ResNet, DeiT and PiT. Including more diverse architectures, such as WRN and VGG, would help to verify the effectiveness of the method. Additionally, It is mentioned( Page 2, Line 69 ) that it has a plug - in effect for a wide range of self - distillation and online distillation methods. However, the improvements over existing methods (Table 4) do not follow the latest methods, and the validated architectures/datasets are only ResNet - 18/CIFAR - 100.", "theoretical_claims": "Theorem 4.4 (Page 4) links Spaced KD to flatter minima via Hessian trace analysis. The proof (Appendix A.1) assumes over-parameterized models and linearized dynamics, which may not fully capture real-world DNN training.", "experimental_designs_or_analyses": "Strengths: Ablation studies on interval sensitivity (Fig. 2, Page 6) and critical timing (Fig. 3, Page 7) are thorough.  \nWeakness: The baseline comparison lacks state-of-the-art methods (e.g., contrastive distillation). Tables 1–2 focus on older baselines (e.g., BAN, DML).", "relation_to_broader_scientific_literature": "Connects KD with biological spacing effect (Page 1, Lines 30–40), leveraging prior work on flat minima (Keskar et al., 2016) and online/self-KD (Zhang et al., 2018).", "essential_references_not_discussed": "Online Knowledge Distillation by Temporal-Spatial Boosting\nC. Li, Z. Wang and H. Qi, \"Online Knowledge Distillation by Temporal-Spatial Boosting,\" doi: 10.1109/WACV51458.2022.00354. \nA recent work on rehearsal-based KD with temporal intervals, which shares conceptual similarities but is not cited.", "other_strengths_and_weaknesses": "Originality: Novel integration of neuroscience principles into KD.  \nClarity: Well-structured, but the pseudo-code (Appendix A.10) lacks implementation details (e.g., gradient accumulation for interval s).", "other_comments_or_suggestions": "Clarity: The term \"space interval\" (Page 4) could be confused with spatial intervals; \"temporal interval\" is more precise.", "questions_for_authors": "Why is the optimal interval s = 1.5 epochs (Page 7)? Is this dataset-dependent? A theoretical justification is missing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdSTwBC0Vv", "reviewer": "Reviewer_rhLU", "summary": "This paper, inspired by the spacing effect, proposes Spaced KD, which distills the student model by a teacher who pretrains s-steps ahead of the student. This paper demonstrates theoretically that Spaced KD produces flatter loss landscapes, and proves experimentally the superior performance of Spaced KD in both online KD and self KD scenarios.\n\n## update after rebuttal\n\nFurther responses from the authors show that their method leads to a slight increase in training time without other additional overhead, and can be combined with other KD methods to improve performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We provide a point-to-point response as follows. We hope you may consider this a sufficient reason to raise the score. If you have any further questions, please let us know.\n\n**Q1: The additional overhead that Spaced KD incurs compared to vanilla KD.**\n\nThe computational and parameter overhead of our Spaced KD is essentially **identical**  the counterparts of online KD and self KD. Regardless of whether or not Spaced KD is introduced, the entire model (both teacher and student) employs the same network architecture and is trained with the same total number of epochs. In practice, since we need to train the teacher s epochs in advance and wait for the student to follow, this results in a slight delay in the runtime (around 30%) but does **not** increase the computational overhead (i.e., the waiting teacher is frozen, not computing).\n\n**Q2: Lower update frequency of the teacher model.**\n\nOur claim \"less frequent\" means that, the teacher is distilled every s epochs in Spaced KD, rather than every iteration in online KD. We will clarify this in the final version.\n\n**Q3: How Spaced KD applies to self KD and the additional parameter/training overhead.**\n\nIn self KD, the deepest layer (as the teacher) transfers knowledge to the shallow layers (as the student) at each training time step [1]. In its spaced version, we first train the model using the cross-entropy loss between the deepest layer's output and the ground-truth label for $s$ steps. We then train the model using the standard self KD loss [1] between the deepest layer's output and each shallow layer's output for $s$ steps. Therefore, our spaced version of self KD does **not** store additional parameters and does **not** increase the training overhead (i.e., the total number of epochs is not changed). We will make this clearer.\n\n[1] Be your own teacher: Improve the performance of convolutional neural networks via self distillation. ICCV, 2019.\n\n**Q4: The accuracy of the teacher model and its comparison to the student model.**\n\nWe would point out that in all experiments, \"w/o KD\" denotes the teacher's performance, \"w/o Ours\" and \"w/ KD\" denote the student's performance after online KD or self KD without our spacing effect, while \"w/ Ours\" denotes the student's performance after online KD or self KD with our spacing effect (referred to as the Spaced KD). \nThis is because existing online KD and self KD methods often employ (two copies of) the same network as both teacher and student to improve generalization of the model itself. Therefore, the student's performance will exceed the teacher's performance if the online KD and self KD methods work well.\n\n**Q5: In Table 5, the authors should provide a comparison with other distillation methods.**\n\nIn addition to online KD in Table 5, we add more experiments of **self KD** with different corruption types and network architectures. As shown in the following table, Spaced KD can also largely improve the robustness of self KD in generalizing to different noisy scenarios.\n\n|Attack|ResNet18 w/o Ours|ResNet18 w/ Ours|ResNet50 w/o Ours|ResNet50 w/ Ours|ResNet101 w/o Ours|ResNet101 w/ Ours|\n|-|-|-|-|-|-|-|\n|impulse_noise|50.65|**60.57**|62.18|**71.57**|59.33|**68.78**|\n|zoom_bhr|64.44|**68.60**|68.03|**72.13**|66.09|**71.16**|\n|snow|61.30|**66.14**|64.72|**69.65**|64.03|**68.76**|\n|frost|63.96|**67.80**|66.36|**71.81**|66.73|**70.21**|\n|jpeg_compression|30.99|**34.67**|34.44|**35.34**|33.64|**34.76**|\n|brightness|73.18|**75.92**|74.91|**79.19**|75.10|**78.91**|\n\n**Q6 & Q7: Include citations and comparisons to more recent KD methods.**\n\nFollowing your suggestion, we have included more recent KD methods, especially those for 2023-2024. For example, TSB [1] constructs superior \"teachers\" with temporal accumulator and spatial integrator. CTKD [2] controls the task difficulty level during the student’s learning career through a dynamic and learnable temperature. LSKD [3] employs a plug-and-play Z-score pre-process of logit standardization before applying softmax and KL divergence.\n\nAs shown in the following table, our Spaced KD can be combined with these methods and provide significant improvements under two datasets and two architectures.\n\n|Dataset/Model|Method|TSB [1]|CTKD [2]|LSKD [3]|\n|-|-|-|-|-|\n|CIFAR-100/ResNet-18|w/o KD|67.65|67.86|67.94|\n| |w/ KD|71.70|69.41|70.74|\n| |w/ Ours|**72.82**|**71.12**|**71.76**|\n|CIFAR-100/DeiT-Tiny|w/o KD|51.90|53.31|52.31|\n| |w/ KD|52.63|54.20|52.93|\n| |w/ Ours|**55.47**|**54.72**|**53.83**|\n|Tiny-IN/ResNet-18|w/o KD|55.21|53.03|54.05|\n| |w/ KD|59.92|58.78|59.30|\n| |w/ Ours|**61.65**|**60.32**|**60.28**|\n| Tiny-IN/DeiT-Tiny|w/o KD|40.29|40.82|39.65|\n| |w/ KD|40.13|41.22|41.14|\n| |w/ Ours|**43.36**|**41.60**|**41.48**|\n\n[1] Online Knowledge Distillation by Temporal-Spatial Boosting. WACV, 2022. (requested by Reviewer C5Ts)\n[2] Curriculum Temperature for Knowledge Distillation. AAAI, 2023.\n[3] Logit Standardization in Knowledge Distillation. CVPR, 2024.", "claims_and_evidence": "The authors claim that Spaced KD has a lower update frequency of the teacher model compared to online KD. However, Spaced KD updates the teacher model s steps ahead of the student model, as shown in Figure 1 and Algorithm 2, and updates the teacher model s times per advance, which does not reduce the number of updates to the teacher model. I would like the author to explain this.", "methods_and_evaluation_criteria": "Spaced KD is easy to understand and implement, and experimental results show that Spaced KD further improves distillation performance in both online and self KD.\n\nHowever, I am still confused about how Spaced KD applies to self KD. [1] uses the final output to guide the intermediate output for self-distillation, so the teacher and student are the same model but with different capacities. This self-distillation is designed to reduce distillation overhead. If Spaced KD is used in self-distillation, is it necessary to additionally store the parameters of the teacher model, which is updated s steps ahead during training? How much would this increase the training overhead?\n\n[1] Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV. 2019.", "theoretical_claims": "There is no obvious error in the theoretical analysis presented in this paper.", "experimental_designs_or_analyses": "1. I suggest that the authors include the additional overhead that Spaced KD incurs compared to vanilla KD, such as memory and time during training.\n2. It is recommended that the authors provide the accuracy of the teacher model in their experiments to show the performance gap between Spaced KD trained students and teachers. In other words, is it possible that Spaced KD-trained students can outperform teachers? As far as I know, some distillation methods have been able to enable their students to perform beyond the teacher model.\n3. In Table 5, the authors should provide a comparison with other distillation methods.\n4. The authors achieved further performance gains by using Spaced KD on other KD methods in Table 4. However, I consider these methods to be too dated and suggest that the authors use the most recent KD methods as a baseline to further demonstrate the generalizability of their method.", "relation_to_broader_scientific_literature": "The method proposed in this paper is simple and easy to implement, and can be better combined with previous distillation methods to further improve performance.", "essential_references_not_discussed": "This paper presents preliminaries for understanding its method and cites relevant papers in Section 3. However, I would still recommend including citations and comparisons to more recent KD literature (years 2023 & 2024).", "other_strengths_and_weaknesses": "I believe that the method proposed in this paper can contribute to the development of bio-inspired KD algorithms, and the experimental results demonstrate the superior performance of Spaced KD. However, I am concerned about the overhead of Spaced KD during training, which, while easy to implement, has the potential to incur significant additional memory/time overhead, thus affecting distillation efficiency, especially for self-distillation with the goal of efficient distillation.", "other_comments_or_suggestions": "It is recommended that the authors elaborate further on the details and overhead of Spaced KD.", "questions_for_authors": "I have no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Spaced Knowledge Distillation inspired by the "spacing effect" in biological learning, where spaced learning intervals improve learning and generalization. The authors provide theoretical analysis suggesting spaced knowledge distillation helps DNNs find flatter loss minima during training, which is linked to better generalization. Experiments across multiple benchmarks and architectures demonstrate that the proposed method yields notable performance gains without additional training costs.

Reviewers generally acknowledged the originality of integrating spacing effect into knowledge distillation and found the theoretical motivation, which connects the spacing effect to achieving flatter loss minima for better generalization, plausible. Initial experimental results were also recognized for demonstrating performance improvements over standard online distillation and self-distillation baselines. Despite the appreciated novelty, a recurring concern raised by the reviewers was the missing comparison with broader SOTA in knowledge distillation; reviewers pointed out missing comparisons, compatibility tests, and literature review concerning more recent and advanced KD methods beyond the basic online/self KD paradigms initially compared against. Another area of concern involved the practical overhead of Spaced KD, with specific questions about potential increases in memory usage and training time, particularly questioning its efficiency within self KD.

The authors provided a comprehensive rebuttal and presented new experimental results demonstrating that Spaced KD is compatible with and provides significant performance boosts when integrated with several recent SOTA KD methods (including TSB, CTKD, LSKD, and DTSKD). They clarified that the method incurs minimal additional computational or memory overhead, explaining the slight runtime delay and detailing its application to self KD without extra parameter storage. Additionally, they offered justifications for the interval selection process based on SGD variability, elaborated on their theoretical framework and assumptions, and clarified technical points like the meaning of update frequency.

This rebuttal successfully addressed the primary concerns for reviewers who engaged with it, leading to positive final assessments and recommendations leaning towards acceptance, although some still noted a desire for broader SOTA comparisons. In terms of review scores, the paper is borderline. Nevertheless, weighing the novel neuroscience connection and the strong rebuttal, while also noting the varying reviewer engagement post-rebuttal, I believe this submission can be a useful addition to the conference and recommend acceptance.
Decision: Accept (poster)