Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate
Authors: Can Jin
Tong Che
Hongwu Peng
Yiyuan Li
Dimitris N. Metaxas
Marco Pavone
Venue: NeurIPS 2024 poster (2024)
Keywords: Generalization
Regularization
Learning from Teaching
Primary area: optimization_for_deep_networks
TL;DR: We propose Learning from Teaching (LoT), a novel regularization technique for deep neural networks to enhance generalization.

Abstract: Generalization remains a central challenge in machine learning. In this work, we propose *Learning from Teaching* (**LoT**), a novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability to capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners are trained by the main model and, in turn, provide feedback to help the main model capture more generalizable and imitable correlations. Our experimental results across several domains, including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction of LoT brings significant benefits compared to training models on the original dataset. The results suggest the effectiveness and efficiency of LoT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code is available at https://github.com/jincan333/LoT.

Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate
Can Jin
Tong Che
Hongwu Peng
Yiyuan List
Dimitris N. Metaxas
Marco Pavone
'Rutgers University Nvidia Research *University Connecticut University of North Carolina Chapel Hill Stanford University can. jin@rutgers.ern edu, tongeOnvidia.co com
Abstract
Generalization remains central challenge in machine learning. In this work, we propose Learning from Teaching (LoT), novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners trained the main model and, in turn, provide feedback help the main model capture more generalizable and imitable correlations. Our experimental results across several domains. including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction LOT brings significant benefits compared training models on the original dataset. The results suggest the effectiveness and efficiency of LOT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code available at ptttps://githnn com/ jincan333LLLT
Introduction
Improving the generalization performance of models unseen data is major challenge in machine 79 [07]. Despite its significant advances, identifying the most generalizable model within the space of potential models remains challenging. Existing deep learning ap- proaches focus on crafting the hypothesis spaces where prediction eтrors are optimized using training
learning [6] 7
73
data 333 69 71. These spaces are shaped by inductive biases [33 77 embedded in the neural
architectures which include implicit assumptions about the data [[] 25 155), objective functions
(notably regularizers) 20 68 [03), and learning methodologies [14 72 87].
In this paper, to enhance generalization, we use the methodology of regularization
337
51
88
which prioritizes specific regions in the hypothesis spaces. Regularization techniques often involve employing auxiliary losses or regularizers (80 alongside the primary task losses. For instance,
LI regularization [2 [3 encourages sparsity within models [16] 40
54
57]. Other regularization
techniques include model averaging 44 [02].
dropout techniques [37 67 and additional
"'Equal contribution, 'Equal contribution, *Equal advising, <utgejin@rtgers..clu>> Tong Che niviii.comm>>
Correspondence to:
Can
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

optimization components 43 63 104].
Due to its effectiveness and simplicity, regularization is critical in modern machine learning techniques for achieving better generalization [37 109
We aim to answer the research question: Among all possible models fitting the training data, which ones are inherently generalizable? A common belief in cognitive science is that human intelligence development involves distilling information and filtering out extraneous details to discern 'simple' correlations among few selected relevant abstract variables [8] 94]. This approach leads to the formation of correlations through simple patterns 2 56 at the right scales. However, identifying simple correlations deep learning remains challenging, mostly due to not being easy to identify the right scale of the problem. Studies in emergent languages suggest that the more structured language is, the more efficiently can be transmitted to message receivers [1(5] Inspired by this finding, we propose defining simple and generalizable correlations at the right scales, as those that can be readily imitated by other learners, provided they possess suitable inductive biases.
Based on this definition, we propose a novel regularization approach, Learning from Teaching (LoT). ) The core of LoT to compute measure of 'imitability" for the main model learn data correlations at the correct scales. By adding this measure to the objective function and optimizing it during training, we encourage the teacher model to refine its learned multiscale correlations, making them more accessible through teaching, which in turn leads to better generalization. LOT computes this measure by jointly training the main model as the 'teacher' with one or more auxiliary 'student' models. The student models strive to distill and assimilate the correlations acquired by the teacher model. Thus, the learning performance of the student defines the measure of imitability of the teacher, which is then used as the LoT regularizer.
We conduct comprehensive experiments using LoT improve the Reinforcement Learning (RL) formulation, as well as in Natural Language Processing (NLP) and Computer Vision (CV) applica- tions. In RL, the experimental results demonstrate that LOT attains an average normalized reward enhancement of 44% on four Atari games. In language modeling tasks, LOT achieves significant perplexity reductions on the Penn Tree Bank (64) and WikiText- 103 [63] Notably, LoT enhances the supervised fine-tuning performance of LLaMA 196 977 models on GSM8K [19] and MATH (35] In image classification tasks, LoT achieves accuracy gains of 1.99% and 0.83% on CIFAR-100 49
and ImageNet- IK [23 respectively.
2 Methodology
2.1
Generalizable and Spurious Correlations
Given a dataset D (1),,)))." ....................... (Xx,, Hn)} generated from data-generating distribution D, there are infinitely many continuous functions I such that f(x) = y for all (x, y) E D. Therefore, finding the f that precisely models the true generalizable correlation between and y is challenging, especially with real-world data like natural images, which are complex and multiscale. In such sce- narios, neural network may compute incorrect (according to the ground-truth relationship between variables) yet perfect (in the empirical data distribution) correlations that explain the relationship between and y 32 773. This phenomenon is particularly evident when is entirely noise- based and independent of x, but the neural network still fits to x perfectly [07). This process, often called brute-foree memorization [3, involves the network creating intricate computational strategies to encode all (x, y) pairs in the samples. Consequently, correlations established in this way are spurious, originating from sampling noise in the data rather than ground-truth relationships.
But how do humans distinguish generalizable correlations from spurious ones? Instead of relying on brute-force memorization to establish input-output correspondences. humans naturally focus on understanding high-level concepts within the input data, selectively ignoring irrelevant details [8] 94 This approach leads to the formation of correlations through simple, comprehensible patterns 2 Empirical evidence in emergent languages also suggests that the more compositional a language is, the more learners will use it (56.
We can, therefore, define the distinctions between generalizable and spurious correlations. First, generalizable correlations are simple and comprehensible, exhibiting lower Kolmogorov Complex- ity [3 58 90]. Second, while there only one ground-truth correlation for dataset, the number of spurious correlations can be massive. These two major distinctions lead to the following hypothesis.

---

Generalizable correlations should be more easily imitable by learners compared to spurious correlations. Specifically, assume TG and Ts are two teacher models that capture the generalizable correlation and spurious correlation from dataset, respectively. We have student learners Sg and Sş that separately imitate To and Ts:
Hypothesis:
From effectiveness perspective, the final training and test losses of learner So after training are typically lower than those of learner Ss.
From an efficiency perspective, during training, the test losses of learner Sg decrease more rapidly than those of Ss.
This hypothesis emphasizes that generalizable correlations inherent in data are not only more in- terpretable but also more readily imitable. It suggests that the inherent simplicity and uniqueness of generalizable correlations make them more attainable and recognizable for learning algorithms, in contrast to the complex and abundant nature of spurious correlations derived from noise. In the following present novel approach.
2.2 Learning from Teaching Regularization
Building upon the Hypothesis, we propose that the ease of imitation of the teacher model by student models can serve as proxy for the generalizability of learned representations. By measuring the 'imitability" of the teacher model in the learning process, we can infer the generalizability of it. A teacher that is easier imitate implies higher generalization. We then design a novel regularization approach that involves training a teacher model T' alongside student models S to imitate T, subsequently measuring the imitability of the teacher during training. We maximize imitability by incorporating it as an additional loss during the training of the teacher T. This imitability loss is termed the Learning from Teaching regularizer (LoT regularizer). By doing so, T is optimized to be teacher that is easier to imitate and, thus, possesses superior generalization compared to models without the LOT regularizer. We refer to this class of regularization methods as 'Learning from Teaching Regularization' (LOT). ). LOT aligns with the broader concept of regularization in machine learning, where the goal is to promote generalizable representations and prevent overfitting.
Although LoT can be applied to supervised, unsupervised. and reinforcement learning, we begin our discussion with supervised learning. We train network Te. parameterized by 0. as the main model, which also serves as the teacher model. Additionally, we train set of K networks S; i 1.2..... K, as the student model:]] The total set of parameters of the K networks denoted by Φb Given training dataset DD = {(x1, y/),,.. (Xn, }n)), we train T and S to model p(ylx), denoted as (x)x)) and ps (y)x,, respectively. Additionally, LOT includes a predefined imitability metric (.a(+)) (.S())) (-)). Intuitively, Ms, measures the difference between S and T"'s predictions on the same input (occasionally denoted as μ henceforth for convenience). There are many possible choices for the metric μu, such as the 1,2 loss between the hidden representations of specific layer. In our experiments, we choose {x()) = (p.((p [|pe (y) x)). which is the KL-divergence [21). to quantify the distribution similarity between S and T.
We first train the teacher model. The objective function of the teacher combines the regular task loss with the additional LoT regularizer R(O) (defined in Equation 3). For example, in supervised learning, we can use the negative log- likeliihldoo loss for the regular task loss, and the objective function can be written as:
(0)
1
Σ
logp((u(xx,) + R(0),
[Dr
(X..).)D:
where [D,] is the number of samples in the dataset D,.
To train the student networks and enhance information diversity, we require an independent unlabelled This dataset can be identical to D,, or generated either by
dataset, denoted as DD- {x1, ... xm_) a generative model trained on D: or through alternative augmentation methods (c.g. synthetic data
generation). This unlabelled dataset constitutes the environment for the student networks to follow the prediction of the teacher and, therefore, explores and generalizes beyond the original training data.
For convenience, S is referred single student learner henceforth.
3

---

Specifically, the student networks' goal is to imitate the correlations acquired by the teacher network during the training process. The training loss for students can be written as:
(,(
(x);
D2
where [Da] the number of samples in the unlabelled dataset D.. The loss function Lg encourages the student networks to learn from the teacher network by minimizing the difference between their
predictions, as measured by the metric (x).
The feedback from all students St constitutes the LoT regularizer:
R(O)=
D
XED,
where X; 2 O represents the coefficient weight of the i-th student S,, with Σ = The a can be either learnable parameter or fixed, such as 1. Essentially, the LoT regularizer measures the imitability of the teacher. The regularization coefficient a controls the trade-off between the original task learning objective of T and the feedback from the students.
The detailed procedure of LOT for supervised and unsupervised learning is outlined in Algorithm T and LoT regularization for RI (using PPO an example) outlined in Algorithm The teacher T and student S; networks are initialized differently to ensure they learn diverse features and representations. In both algorithms, the teacher and student networks iteratively learn from each other, with the students imitating the teacher's correlations and the teacher incorporating the students' feedback into the learning process.
Algorithm 1 Learning from Teaching Regularization
1: Input: Dataset D,, D,, Regularization Coefficient 0, Student Steps Ratio N > 0
2: Initialize teacher network T parameterized by 0 and student networks St, i 1,2, ... K, parameterized by
3: repeat
4: Sample batch of data B, D,, CD2
5: Compute R(0) a
[B.| ExEB, Eim1 (iHle,s(xx) Compute (0) -1.1 E(xWEE, log (ogpe(ylx)) + R(0)
6:
7:
Update 0 using gradient (o)
8:
for i = to N do
9:
Sample 8, D.
10:
Compute (.())= = B. ExEB, Σ (x,
11:
Update student networks' parameters using loss gradient VoL.(D)
12:
end for
13:
until 7' converges
2.3
Discussion
The works most related LOT are knowledge distillation (KD) 29 36
and ease-of-teaching
56] in emergent languages. However, LoT differs significantly from these approaches. In KD, teacher model containing task-specific knowledge transmits this knowledge to student model (often smaller than the teacher), with the primary focus on the student's 's performance post-distillation. Conversely, in LoT, both the teacher and student models may lack or possess different task-specific knowledge. Generalization is improved through joint training, incorporating additional signals from student feedback. emergent languages, Li and Bowling (56) propose that structured language is easier to teach to other agents than less structured ones, achieving higher task success rates with less training. Additionally, Chaabouni et al. [T] identify strong positive correlation between language transmission efficiency to new message receivers and the degree of compositionality (structuredness)) of the language. In LOT, we focus on tasks distinct from emergent languages, finding that generalizable correlations are easier to imitate. Under our Hypothesis, design novel

---

VT-8 Deceptive Student
VIT-B Sophisticated Student
VITL Deceptive Student
VITLL Sophisticated Studert
Trein KL KLSS
Test
0.2
10
2
Total Steps (k]
Total Stops (k)
Figure 1: Training and test KL-divergence losses of student models in LoT using ViT-B/16 and ViT-L/16 on CIFAR-100 with different teacher models. The sophisticated students achieve lower losses than the deceptive students given the same computational budget.
LoT regularizer and algorithm enhance the generalization of deep neural networks, extending the case-of-teaching concept to supervised. unsupervised, and reinforcement learning. In parallel work, Ning et al. 774 proposes Learning by Teaching (LbT), which utilizes teacher and student models to generate answers as training samples for the teacher model. However, the regularization method in LOT fundamentally distinct from that Ning et [74]
Experiments
We first validate Hypothesis in Section 3.1 Subsequently. assess the performance of LoT across several tasks: Atari games (Section 3.2) language modeling (Section (.3),, and image classification (Section 3.4h. We compare LoT to Teacher-only baseline, wherein the regularization coefficient a in R())) set to 0, thereby blocking the student feedback. Unless specified otherwise, employ only student model. Except for the Atari games where the student learn from the offline samples of the teacher, we set N = to manage computation (we study the impact of N in Section 3.6). Moreover, we study the computational efficiency and effects of hyperparameters of LOT in Sections 3.5 3.6
3.1 Generalizable Correlations are Easier to Imitate than Spurious Correlations.
In Hypothesis, learners presumed more readily imitate generalizable correlations than spurious ones. To investigate this, we design experiments involving two distinct teacher models: sophisticated teacher and a deceptive teacher. The sophisticated teacher effectively captures generalizable correlations, while the deceptive teacher primarily learns spurious correlations. We an identical student model to learn from both teachers separately, monitoring the ttudent-teacher KL divergence during training and testing. The student that learns easier-to-imitate correlations is expected to exhibit lower training and test KL losses with fewer training steps.
employ the ViT-B/16 and ViT-L/16 architectures 24 for both the teachers and students. The sophisticated teachers are trained on the full CIFAR- 100 49 training set for 10,000 steps to achieve optimal convergence. The deceptive teachers, using the same hyperparameters and training steps as the sophisticated teachers, trained a random subset of 2, 560 images from the CIFAR-100 training set, leading to over-fitting. Consequently, the sophisticated teachers are expected to exhibit better generalization ability (their test accuracy surpasses that of the deceptive teachers by 14%).
The two student models referred to as the sophisticated student and the deceptive student, share identical hyperparameters and initializations. They are trained to imitate the correlations from their respective teachers on the full CIFAR- 100 training set. The teacher models are kept frozen during training of the students, with the objective (g) defined as follows:
(.(
1
Σ
4KL (x).pp//)l
D.
XED.
where D, represents the full training set of CIFAR-100.
We present the training and test losses in Figure and make the following observations:
5

---

BeandRider
Graviar
UpNDown
Breakout
Teacher-angy only
900
Teach only
3,00
Teacher-anty
600
Teacher--nee only LoT
LoT
800
700
oT
E
LoT
Eptede Re
400
400
500
2000
400
300
200
200
1003
100
9
4
12 Step (Milioo)
16
20
0
4
12
16
8
12
15
20
0
8
12
16
20
Step (Millon)
Step (Million)
Step (Millin)
Figure 2: The episodic return of the teacher agent in LoT and the Teacher-only on four Atari games (averaged over ten runs). LoT demonstrates return gains over Teacher-only on all games.
Given the same computational budget, the sophisticated students achieve lower final KL losses on both the training and test sets compared to the deceptive students. This suggests that the student can more effectively imitate the prediction distribution of teacher that captures generalizable correlations.
The deceptive students require more training steps to achieve the same training and test student-teacher KI. losses as the sophisticated students. This indicates that learners tend to grasp spurious correlations much more slowly than generalizable correlations.
These results suggest that generalizable correlations easier to imitate than spurious In LoT, we expect the teacher model to master generalizable correlations by incorporating feedback from students via the LOT regularizer.
3.2 Atari Games
We conduct experiments four Atari games, namely BeamRider, Breakout, UpNDown, and Gravitar, following the implementation in Huang et al. identical hyperparameters. All agents are trained using Proximal Policy Optimization (PPO) [83). While the teacher agents interact with the game environment, the student agents are trained on the most recent 10,240 samples generated by the teacher agents, ensuring that LoT and Teacher-only experience the same environmental interactions. We use different values for various games and set N efficiently imitate the teacher. More details provided in Appendix D
Both the LoT and Teacher-only agents have
The empirical results are presented Figure 2 and we make the following observations:
LoT improves the agent return compared the Teacher-only version with 20 million teacher training steps. Specifically, LoT achieves (63.14%, 9. 9.79%, 66.48%, 35.70% normalized return enhancements on [BeamRider, Gravitar, UpNDown, Breakout).
The performance gain of LoT becomes more prominent as the training progresses (from 15 million to 20 million steps).
These results suggest that LoT is an effective approach for enhancing the generalization of RL agents, as requires no additional environmental interactions while delivering significant performance gains,
3.3 Language Modeling
Language modeling a widely acknowledged NLP task, and regularization techniques have been demonstrated to significantly enhance performance in this domain [106]. To examine the impact of LoT language modeling, we conduct experiments in two scenarios: unsupervised language pretraining and supervised fine-tuning.
3.3.1 Unsupervised Language Pretraining
We conduct experiments of LoT and Teacher-only using LSTM 39
AWD-LSTM [66]. and
Transformer-XI. [22 for teacher and student on Penn Tree Bank (PTB) 64 and WikiText- 103
65
We follow the implementations outlined in Dai et al. (22, Merity et al. [66] Zaremba et al. [06
In LOT, we utilize different coefficients a for various architectures and benchmarks to control
LoT regularizer. To ensure fair comparison, maintain the same total number of training steps

---

Table 1: The test perplexity of the teacher model in LoT and the baseline on PTB and WikiText- 103. Results are averaged over three runs. LOT achieves consistent perplexity reduction over different choices of architectures and benchmarks.
Dataset
Teacher
Student
Teacher #Param.
Teacher-only
LoT
LSTM AWD-LSTM
LSTM AWD-ISTM
82.75 + 0.36
71.72 0.34
PTB
24M
38.69 + 0.37
53.31
0.56
Transformer XL-B
Transformer
151M
23.72 1 0.41
21.65
00.38
WikiText-103
Transformer-Xe XL-L
Transformer-XLe
257M
18.50 + 0.25
16.47 0.23
(with teacher and student training steps accumulated) for LoT and the Teacher-only setup. Please refer to Appendix D for more implementation details.
From the empirical results presented in Table [ observe that LoT achieves notable perplexity (PPL) gains across various architectures and benchmarks under the same number of learning steps as Teacher-only. Specifically, LOT achieves least points PPL gains across all settings, and 11.03 gain for LSTM on PTB. It indicates that LOT can be effectively applied to both LSTM and Transformer architectures in language pretraining.
3.3.2 Supervised Fine-tuning
Furthermore, to evaluate the effectiveness of LoT in fine-tuning pretrained large language models (LLMs), we conduct supervised fin-tuning (SFT) experiments using LLaMA-1 [96] and LLaMA-
2 197
two mathematical reasoning benchmarks: GSM8K [99 and MATH 35
We compare LoT in-context learning (ICL) 9 and SFT. Following Touvron al. 197 the number of in-context examples is 8 for GSM8K and 4 for MATH. The SFT configuration follows Yue et al. [05), and we fine-tune the LLaMA models for four epochs. In LOT, the teacher and student models share the same architecture for simplicity. The models are trained for two epochs in LOT to match total training steps in SFT for fair comparison. All other configurations are consistent with those used in SFT. More implementation details are described in Appendix D
We measure the accuracy of greedy decod- ing results in Table 2 and we observe that LoT enhances reasoning abilities on all archi- tecture and dataset choices. This indicates the competence of LoT in improving the fine-tuning performance with a computational cost compa- rable to SFT.
Table 2:
The accuracy of the teacher model in LOT and the baseline on GSM8K and MATH, Re- sults averaged over three runs.
Setting
GSM8K
MATH
LLaMA-1 7B -B-IICL
10.69 1 0.87
2.84 0.25
LLaMA-1 7
34.39 1 1.28
4.78
1 0.23
LLaMA-1 7B LoT
36.42 1 1.46
5.39
0.28
3.4 Image Classification
LLaMA-2 7B ICL
14.62 + 0.96
2.46 + 0.25
ILaMA-2 7B S
39.81 + 1.34
5.79 0.31
To investigate the effects of LoT computer vi- sion tasks, we apply LoT to image classification by conducting experiments using ResNets (34). MobileNetV2 [81, ViT [24], and Swin [61 ar-
LLaMA-2 LOT
41.87 1.62
6.28
0.22
chitectures pretrained ImageNet-IK and ImageNet-21
24 as teacher and student models. We
choose CIFAR-100 100 [49] and ImageNet- K as the downstream datasets. The total training steps for
LoT and the Teacher-only approach are the same for fair comparison. Further implementation details are provided in Appendix D We conclude the following observations from results in Table 3
LoT achieves accuracy gains across various architectures and datasets without additional computational costs. For example, LoT improves test accuracy by almost 2 points using a ResNet-18 18 teacher and a ResNet- 50 student on CIFAR- 100 after pretrained on ImageNet- IK. Similarly, on the larger-scaled ImageNet dataset ImageNet-21KK, LoT still obtains nearly point improvement using ViT-B/16 as the teacher and VIT-L/16 as the student.
.
The generalization of teacher models can be effectively enhanced by students of larger sizes. For instance, ResNet-50, ViT-L/16, and Swin-I. students can enhance the performance of ResNet- 18, ViT-B/16, and Swin-B teachers, respectively. Similarly, small student models
7

---

can also enhance the generalization performance of larger teacher models using LoT For example, a MobileNetV2 student improves the performance RestNet- 18 and ResNet-50 by more than 1 point on CIFAR-100 with a much smaller model size. Similar results appear on the ViT-L/16 teacher and ViT-B/16 student combination in the ImageNet-IN task.
For transformer-bont
models, employing different architectures for teachers and students achieves better performance than sharing the same architecture. For example, when applying ViT-B/16 student, a ViT-L/16 teacher achieves 0.27% more accuracy than using ViT-L/16 student. This suggests that using different architectures for teacher and student increases information diversity, which contributes to enhanced generalization for teacher models [84]
These experimental results demonstrate the effectiivness of LoT in enhancing the generalization of pretrained CNN-based and Transformer-based vision models in image classification.
3.5 Analysis of Computational Cost and Efficiency
For supervised and unsupervised tasks, LoT involves training teacher models alongside student models outlined in Algorithm D Compared Teacher- only, the potential limitation of LoT is that it re- quires additional computation and memory for the student models. Therefore, in our results in Sec-
94
Test Accuracy
92
Mw
90
ViT-B Teacher-only
tion 3 maintain the same total training steps be- tween LOT (accumulated for the teacher and student) and Teacher-only and demonstrate that LoT achieves better generalization performance under the same number of updates. In this regard, we show the accuracy of image classification between LoT and Teacher-only using VIT models with respect the total training steps in Figure 3 We note that LoT achieves better test accuracy than Teacher-only in both ViT-B/16 and ViT-L/16 with fewer total training steps. Moreover, we demonstrate that LOT remains effective even when the student model smaller than the teacher model in Table which further reduces the computation cost compared to Teacher-only in the same total training steps and accommodates different student model choices with resource constraints. We provide more results regards efficiency of LOT in Appendix H
VIT-B LoT
88
ViT-L Teacher-only
VIT-L LoT
86
0
2
4
8
10
Total Steps (k)
Figure 3: Test accuracy of teacher mod-
els in LoT and Teacher-only using ViT-B/I6 and ViT-L/16 on CIFAR-100. LoT achieves higher test accuracy with fewer training steps.
In RL tasks, only the teacher model interacts with the environment to collect samples, and the student can learn from the teacher samples exclusively (please refer Appendix G for the algorithm of
Table 3: The test accuracy of the teacher model for various teacher-studen model combinations in LoT and the baseline. Results averaged over three runs. LoT consistently enhances test performance in all model choices and datasets.
Pretrained
Downstream
Teacher
Student
Image Size
Teacher/Student #Param.
Teacher-only
LoT
ResNet-11
MohilaNerV2
12M/4M
0.36
ResNet-18
ResNet-18
224"
12M 12M
81.14 + 0.58
82.89
0.25
ImageNet-
CIF
ResNet-18
83.13
ResNet-50
MobileNetV2
224"
26M 4M
84.09 + 0.32
85.38
0.44
ReaNet-50
ResNot-13
2243
26M 12M
81.09 - 0.32
85.77
0.19
ResNet-50
ResNei-50
224
26M 26M
84.09 1 0.32
86.04
01.38
VIT-B/16
VIT-B/16
3841
86M/ 86M
91.37 - 0.31
93.17 :: 01.55
ImageNNet21K CIFAR-100
B/16
/
91.57
0.44
VIT-1/16
VIT-R116
3842
307M/ / 86M
93.44 + 0.28
94,29
0.33
VIT.1/16
93.44
VIT-B/16
V/1-
3844
86M 86M
83.97 1: 0.11
84.54
01.15
VIT-B/16
VIT./16
384
86M 307M
81.97 4 0.11
84,80
0.08
ViT-L/16
VIT-B/16
384
307M 86M
85.15
85.92
0.09
ImageNee-21K ImageNes- IK
VIT-L/16
VIT-L/16
3847
30TM 307M
85.15 0.17
85.65
0.11
Swin-B
Swin-B
86.37
0.15
Swin
Swin. L.
3842
88M 197M
88.37 1. 0.06
86.73
0.14
Swin-L
Swin-B
88M
87.27
0..12
Swin-L.
Swin.L.
384
197M 197M
87.27 1. 0.11
$7,59
0.09

---

Table 4: Performance comparison of Teacher-only, BAN and LoT on CIFAR-100. LoT achieves superior performance to Teacher-only and BAN.
Dataset
Teacher
Student
Teacher-only
BAN (Student)
LoT (Teacher)
CIFAR-100
ResNet-18
ResNet-18
81.14
82.08
82.89
CIFAR-100
ResNet-50
ResNet-50
84.09
84.73
86.04
CIFAR-100
VIT-B/16
ViT-B/16
91.57
92.44
93.17
CIFAR-100
ViT-L/16
ViT-L/16
93.44
93.82
94.18
84
82
Teacher-only
Teacher-only
80
LoT
LoT
E 78
74
72
0.0
0.2
0.4
0.6
0.8
1.0 Alpha
1.2
1.4
1.6
2
5
6
N
Figure 4: Effects of regularization coefficient (left) and student steps ratio N (right). 1 is the best value to achieve the lowest test perplexity of the teacher model, and moderate student steps ratio N such as 4 and 5 benefit the teacher model the most.
PPO-version LoT). Therefore, LoT introduces negligible computation costs since sample collections are more rssource-intensin than fitting the agent network to the samples in RL. For instance, in our Atari games experiments, the training time LOT (606 minutes) is comparable to the Teacher-only setting (597 minutes) on single NVIDIA A6000 GPU.
3.6 Additional Investigation
Comparison to KD. To investigate the effect of LoT compared to other student-teacher learning paradigms, we compare LoT to the born-again networks (BAN) bascline [29]. In BAN, we select the checkpoint with the best performance of the Teacher-only model as the (frozen) teacher and distill its knowledge into a student model with an identical architecture. Equal weights are assigned to the hard loss (from the dataset) and soft loss (from the teacher) to train the student model [36). All other configurations remain consistent with LoT. The results in Table Table (indicate that LoT achieves superior performance than BAN with strong feedback model, further indicating the significance of the interactive learning process in LOT.
Effect of regularization coefficient a. The strength of regularization plays a crucial role in the overall training effect [50]. To investigate the effects of LoT the generalization of the teacher model, we perform experiments on PTB using the LSTM architecture for both teacher and student models. The configuration follows Section 3.3 except that gradually increase the value of in LOT from 0 to 1.7 and examine the test PPL. the teacher model. The results are presented in Figure4[[cft We observe that the performance of the teacher model improves rapidly as a increases from 0 to 1, and when the value exceeds this point, the performance of the teacher begins to decline. This observation suggests that moderate feedback from the student most beneficial for the teacher, but an excessively strong signal can hinder the teacher's 's learning process. Similar effects of large values have been noted in joint teacher-student training knowledge distillation 75
Effect of student steps ratio N. To demonstrate the importance of the student steps ratio in LoT, we conduct additional experiments by training LSTM teacher and student models on PTB using various values of N. The empirical results presented teacher benefits most from moderate N value, such as 4 or 5. This finding suggests that achieving balanced ratio between teacher and student model updates crucial for optimal performance. When N is too low, the student may not sufficiently learn from the teacher, thereby reducing the quality of the feedback it provides. Conversely, if N too high, the student may overfit the teacher's errors, resulting in less effective imitability measurement.
Figure 4 (right) indicate that the

---

4 Conclusion
Identifying generalizable multiscale correlations from the vast space of possible correlations remains significant challenge in machine learning. Inspired by cognitive science beliefs about human intelligence, we have shown experimentally that generalizable correlations are more imitable by other learners. In particular, generalizable correlations by teaching student models and exploiting their feedback. We conducted comprehensive experiments across various learning tasks and neural architectures. The results demonstrate that our proposed regularizer enhances model performance effectively and efficiently. In conclusion, our proposed LoT regularization offers promising new approach improve the gener- alization of neural networks by leveraging the learning process of student models and incorporating their feedback to refine the teacher model.
introduced novel regularization method, LOT, which identifies
5
Acknowledgments
Metaxas is partially supported by research grants from NSF: 2310966, 2235405, 2212301, 2003874, 1951890, AFOSR 23RT0630, and NIH 127661.
References
[1] Samira Abnar, Mostafa Dehghani, and Willem Zuidema. Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006.00555: 2020.
[2] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julics
A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 233-242. PMLR, 06-11 Aug 2017. URL URLhttps: http:::///ppooeeeiiie .mr. mlr. pivesssv70/arrittteo
[3] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances in neural information processing systems, 27, 2014.
[4] Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26, 2013.
[5] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2019. URL httpps://ppeerevieeee net/forum did
[6] Peter Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 1996.
[7] Eric Baum and David Haussler. What size net gives valid generalization? Advances in neural information processing systems, 1, 1988.
[8] Leo Breiman and Nong Shang. Born again trees. University of California, Berkeley, Berkeley, CA, Technical Report, 1(2),, 1996.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[10] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
10

---

[11] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4427-4442, Online, July 2020. URL
Association for Computational Linguistics. doi: 116653//1/2022.cl/mmaaaa https://aclanthologg. org/ org/2020 maaimma..00
[12] Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International Conference on Learning Representations. 2022. htt..tttt:://peereeeee wtt.://ttpe//pppeeeeeeeee//eeeee http://rrrrn/AAAUUUEDDIIVVII
[13] Satrajit Chatterjee. Learning and memorization. In Jennifer Dy and Andreas Krause, edi- tors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 755-763. PMLR, 10-15 Jul 2018. URL www.ggssssr.ss.s.ss..mmmmmeeeeee
https://procceddings.u mlr. press wtt.:ppeeeeeeeseeeeeeeeeeeeeeeee h
[14] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Staristical Mechanics: Theory and Experiment, 2019 (12):124018, 2019.
[15] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5008-5017, 2021.
[16] Minsu Cho, Ameya Joshi, and Chinmay Hegde. Espn:
Extremely sparse pruned networks, In 2021 IEEE Data Science and Learning Workshop (DSLWW, pages 1-8. IEEF, 2021.
[17] Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent compositional commu- nication learning from raw visual input. In Internationall Conference on Learning Representa- tions, 2018. URL httptt:ts//pppreeeee
[18] Sahil Chopra, Michael Henry Tessler, and Noah Goodman. The first crank the cultural ratchet: Learning and transmitting concepts through language. In CogSci, pages 226-232, 2019.
[19] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
[20] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. L2 regularization for learning kernels. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligncce, UAI '09, page 109-116, 116, Arlington, Virginia, USA, 2009. AUAI Press. ISBN 9780974903958.
[21] I. Csiszar. I-Divergence Geometry of Probability Distributions and Minimization Problems. The Annals of Probability, 3(1): 158, 1975. doi: 10. 10.1214/app/111769965 URL
https:///oi.. http:///oooorr////// 1214 /5514//0//17799955///
[22] Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov, Transformer-XI. Attentive language models beyond a fixed-length context. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages Association for Computational Linguistics. doi: 10.18653/v1/P19-2222 1285-
Florence, Italy, July 2019.
URL https:/7
aclanthology. www./ogg----. 1285
[23] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 245. leee, 2009.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: arXiv:2010.I1 2010. 11929, 2020.

---

[25] Stéphane d' Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, pages 2286-2296. PMLR, 2021.
[26] Jakob Foerster, loannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
[27] Nicholas Frosst and Geoffrey E. Hinton. Distilling neural network into soft decision tree. In Tarek R. Besold and Oliver Kutz, editors, Proceedings of the First International Workshop Comprehensibility and Explanation in Al and ML 2017 co-located with 16th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, November 16th and 17th, 2017, volume 2071 1 of CEUR Workshop Proceedings. CEUR-WS.org. 2017. URI. htt.hhtps:// ceurr ws. http///1117111/Exx/ 2aa.AIAAA_111_111111111 _apper_pp______ 3. pdf
[28] Tommaso Furlanello, Jiaping Zhao, Andrew M Saxe, Laurent Itti, and Bosco S Tjan. Active long term memory networks. arXiv preprint arXiv: 1rXivv:606..00 02355, 2016.
[29] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anand- kumar. Born again neural networks. In Jennifer Dy and Andreas Krause, editors, Pro- ceedings of the 35th International Conference on Machine Learning. volume 80 of Pro- ceedings of Machine Learning Research, pages 1607-1616. PMLR, 10-15 Jul 2018. URL https://prooeedinggs mlr wwtps///////rrlleeeeerreee
[30] Lukas Galke, Yoav Ram, and Limor Raviv. What makes a language easy to deep-learn? arXiv preprint arXiv:2302. 12239, 2023.
[31] Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint
2023.
[32] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[33] David Haussler. Quantifying inductive bias: Ai learning algorithms and valiant's learning framework. Artificial intelligence, 36(2):177-221, 1988.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[35] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL
attp://arxic: org/ abs 1503, 02531
[37] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhut- dinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: / /207.050, 2012.
[38] Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
[39] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9 1735-778) 1997.
[40] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 1288):00822-1100 2021.
[41] Arthur E Hoerl and Robert W Kennard, Ridge regression: Biased estimation for nonorthogonal problems. Techmometrics, 12(1):55-67, 1970.
12

---

[42] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mchta, and João G.M. Araújo. Cleanrl: High-qualits single-file file implementations of deep reinforcement learning algorithms. Jourmal of Machine Learning Research, 23(274):
1-18, 2022. URL hmtt::/jmmm wttg hwtg/appers/v232/11
[43] Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448-456 Lille, France, 07-09 Jul 2015. PMLR. URL https://proceeding mlr wwwps/eessspppee / v37 htt/vv////ffff /iifffff1 html
[44] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization, In 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876-885, Association For Uncertainty in Artificial Intelliennce (AUAI), 2018.
[45] Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity. IEEE Transactions on Evolutionary Computation, 10, 2001.
[46] Simon Kirby, Hannah Cornish, and Kenny Smith. Cumulative cultural evolution in the labora- tory: An experimental approach to the origins of structure in human language. Proceedings of the National Academy of Sciences, (05(31):10681-1000
2008
[47] Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution of language. Current opinion in neurobiology, 28: 18:1088114 114, 2014.
[48] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pages 2962-2967, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18655/11/D11-13221 URL https: httpss://accaaaeee accannthloln org org/D17-1321
[49] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[51] Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv preprini arXiv: 10686, 2017.
[52] Alex Labach, Hojjat Salehinejad, and Shahrokh Valaee. Survey of dropout methods for deep neural networks. arXiv preprint arXiv: 13310, 2019.
[53] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations, 2017. URL httl:ttts///opreeiiieeeee httpreenrreeeeeeeee/foierreee nett//forun1dd-Hkk333L.... id=Hk8N3Sclg
[54] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NET- WORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Confer- ence on Learning Representations, 2019. URL https://gpenrrrviee.eee///o wwtp:/iwwwiitettouuuneuuaa B1VZqjAcYX
[55] David Lewis. Convention: A philosophical study. John Wiley & Sons, 2008.
[56] Fushan Li and Michael Bowling- Fase-of-teaching and language structure from emergent communication. Advances in neural information processing systems, 32, 2019.
[57] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations, 2017. URL https://openreviee http:://ppeneeeeeeee.eeeeoorrrop metf/foorrr///1i1-rro00000....
13

---

[58] Ming Li, Paul Vitányi, et al. An introduction to Kolmogorov complexity and its applications, volume Springer, 2008.
[59] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2447, 2018. 2773081.
doi: 10.1 (0.1109/PPM...0017.
[60] Zachary C. Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31 1-10311-111111 jum 2018. ISSN 1542-7730. doi: 10.1 1445/3336388.32113335
URL https: org/ 10. 1145/3236386... 3241340
[61] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[62] David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distilla- tion and privileged information. arXiv preprint arXiv:/511.036433 2015.
[63] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http:://p/pperrrreeen http://ppanrrreee..eeffforrr
id=Bkg6RiCqy7
[64] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2)331--330, 1993. URL httpt:ss//allaaathlllogg/ org/
[65] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL
https://openreviiee net/ httt//oooummiddyyy//aarree
[66] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. In International Conference on Learning Representations, 2018.
[67] Poorya Mianjy and Raman Arora. On convergence and generalization of dropout training. Advances in Neural Information Processing Systems, 33:21151 11511-216611 21161, 2020.
[68] Charles A Micchelli, Massimiliano Pontil, and Peter Bartlett. Learning the kernel function via regularization. Journal of machine learning research, 6(7), 2005.
[69] Ryszard S Michalski. A theory and methodology of inductive learning. In Machine learning. pages 83-134. Elsevier, 1983.
[70] Tom M Mitchell. The need for biases in learning generalizations. 1980.
[71] Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In International Conference on Learning Representations, 2021. URL httts::/ nett:/oorumndd-guutriHitt http::///aaaaareeeeeeeeeeeeeeorr
[72] Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro. Path-sgd: Path- noth-normllized optimization in deep neural networks. Advances in neural information processing systems, 28, 2015.
[73] Behnam Neyshabur, Srinadh Bhojanapalli,, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
[74] Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew B Blaschko, Guohao Dai, Huazhong Yang. and Yu Wang. Can Ilms learn by teaching? a preliminary study. arXiv preprint 2XXi:22406.14622 2024.
[75] Dae Young Park, Moon-Hyun Cha, Changwook Jeong, Daesin Kim, and Bohyung Han. Learning fridenl teacher networks for knowledge distillation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. htttpp::://peeeeeeeeereee http://www....ttoorrreeoooooooo.
14

---

[76] Dmitry Pechyony and Vladimir Vapnik. On the theory of learnining with privileged information. Advances in neural information processing systems, 23, 2010.
[77] Ben Poole, Jascha Sohl-Dickstein, and Surya Ganguli. Analyzing noise in autoencoders and deep networks. arXiv preprint arXiv:1406. 1831, 2014.
[78] Yi Ren, Samuel Lavoie, Michael Galkin, Danica J Sutherland, and Aaron C Courville. Improv- ing compositional generalization using iterated learning and simplicial embeddings. Advances in Neural Information Processing Systems, 36, 2024.
[79] David Rumelhart, Geoffrey E Hinton, Ronald Williams, et al. Learning internal representa- tions by error propagation, 1985.
[80] Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell, Policy distillation. arXiv preprint arXiv:1511.06299. 2015.
[81] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[82] Craig Saunders, Alexander Gammerman, and Volodya Vovk. Ridge regression learning algorithm in dual variables. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML '98, page 515-521, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 1558605568.
[83] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.0666 1707.063777 2017.
[84] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, 2016.
[85] Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[86] Kenny Smith, Simon Kirby, and Henry Brighton. Iterated learning: A framework for the emergence of language. Artificial life, 9(4):371-386 2003.
[87] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-28) 22788, 2018
[88] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1958 2014. URL http://jppml http://mmmoorrr//ppp./e. arg/papers/
115/srivastava14a html
[89]
Ilya Sutskever. An observation on generalization. https://simon: http://imiooos...beeee...eee edt/tylku.stsssssaeae takks/ i ilya- sutskever opp.-oopnni---e 2023 04 2023.
[90] Ilya Sutskever. An observation on generalization.
Large Language Models
and Transformers Workshop, 2023. URL https: http:///iimmns...eeelleee.. www.eedlllaaaae
-outtskevee-opeaiii... 20-3-08-14
[91] Sarah Tan, Rich Caruana, Giles Hooker, and Yin Lou. Distillandd-cmppar:: Auditing black- box models using transparent model distillation. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 303-310, 2018.
[92] R Tibshirani. The lasso method for variable selection in the cox model. Statistics in medicine, 16(4):385- 335, February 1997. ISSN 0277-6715.
[93] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
15

---

[94] Michael Tomasello. The human adaptation for culture. Anmual review of anthropology, 28(1): 509-529, 1999.
[95] Hugo Touvron, Matthieu Cord, Matthij Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficienn image transformers via distillation through attention. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347-10357. PMLR, 18-24 Jul 2021. URL ptttpp://prroceedinggs mlr press/ touvron21a.hcr html
[96] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[97] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09220 2023.
[98] Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville. Iterated learning for emergent systematicity in vqa. In International Conference on Learning Repre- sentations.
[99] Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville, Iterated learning for emergent systematicity in (vqa). In International Conference on Learning Representations, 2021. URL httptttt//peeereeeese htw.aarrriiiii.teeoouuuuono net//ooumnidd-d........ pot.rrruurriidddddoot1111111
[100] Li Wan, Matthew Zeiler, Sixin Zhang. Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1058-1066, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL
mlr peress/v28/sse hhml
[101] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's trans- formers: State-of-the-art natural language processing. arXiv preprini (eti::1910.07777 2019.
[102] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-23998. PMLR, 2022.
[103] Chuanguang Yang, Zhulin An, Helong Zhou, Linhang Cai, Xiang Zhi, Jiwen Wu, Yongjun Xu, and Qian Zhang. Mixskd: Self-knowledge distillation from mixup for image recognition. In European Conference on Computer Vision, pages 534-551. Springer, 2022.
[104] Kai Yu, Wei Xu, and Yihong Gong. Deep learning with kernel regularization for visual recognition. Advances in neural information processing systems, 21, 2008
[105] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
[106] Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization.
arXiv preprint arXiv::409.1 2329, 2014.
[107] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
[108] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEFF/CVE Conference on computer vision and pattern recognition, pages 11953-11962, 2022.
16

---

[109] Zhenzhu Zheng and Xi Peng. gelf-guidance: Improve deep neural network generalization via knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3203-3212, January 2022.
17

---

A Ethics and Social Impacts
In this work, we propose a regularization method to improve the generalization of deep neural networks. Our work focuses technical contributions to deep learning and Al. Therefore, the potential social impacts AI in general apply to our work, including fake information, toxic content, fairness concerns, and misuse of AI. For example, toxic content like hate speech can lead to data contamination and therefore have harmful impacts on society, which has been observed in large-scale pretrained models. By employing method, such harmful behavior can potentially be amplified.
B
Related Works
B.1 Regularization in Deep Learning
Regularization serves as a primary strategy to improve generalization capabilities and mitigate over-fitting [52]] Various regularization techniques exist for deep neural networks. One of the earliest and most straightforward approaches to regularization involves constraining the model's capacity by adding penalty function to the original objective function. Techniques such LI regularization [1 92
93], L2 regularization [20 50 [2). and weight decay [50) 104) fall into this
category. Introducing noise [8
(77 the system can also judiciously enhance generalizability and
prevent over-fitting, Dropout 37 88 100) is a widely used regularization technique that randomly
drops certain neural network connections during training.
B.2
Student-Teacher Learning Paradigms
B.2.1 Knowledge Distillation
Knowledge distillation (KI) is technique that transfers knowledge from teacher model to student model by training the student to imitate the teacher's outputs [36.. This approach is widely applied in
areas such model compression, transparency, and interpretability [8 10 27 36, 60 [9]. Model
compression is often motivated by resource constraints. Pioneering works include Buciluă et al. [10], which compresses ensemble neural networks into a single network, and Ba and Caruana 3 which improves shallow neural network accuracy by mimicking deep networks. KD is also applied in various
domains, including deep reinforcement learning 80 continual learning 28
59 85 and learning
privileged information theory [62 [7].. The dark knowledge method (36) further develops KD, where student model aims to fully match the output distribution of the teacher. Intuitively, distillation is effective because the teacher's 's output distribution over classes provides a more informative training signal than one-hot label. Additionally, in born-again networks (BAN) (29, the teacher and student have identical neural architecture and model sizes, but the student can surprisingly surpass the teacher's s accuracy.
B.2.2 Language Emergence
In a cooperative environment, agents can learn emergent languages for communication solve specific tasks. The emergence of such communication protocols is extensively studied in the context of multi-agent referential games [26] 55). In these games, one agent is required to describe its observations to another agent, which is then tasked with deducing the initial agent's 's observations (53). The majority of methods employed to learn discrete communication protocols between agents utilize RL 26 99]. Compositionality is desirable feature in the language used by agents, as it facilitates flawless generalization for previously unseen combinations of attributes [T 17 However, the community still lacks strong research indicating what general conditions are necessary or sufficient for compositional language emergence. Chaabouni et [I 12 Galke al. [30, Li and Bowling (56) postulate that compositional languages are more straightforward to learn.
C Motivation and Insights of Our Method
The concept of Learning from Teaching originates in cognitive psychology and linguistics, particularly within the iterated learning theory of language emergence 45 86). This theory posits that the generalizable nature of languages arises from the iterative learning process across generations in
18

---

society. The core hypothesis is that generalizable language is inherently easier to teach and learn which aligns with our main hypothesis.
[56 78 98
In the Al community, recent research has aimed to employ iterated learning to enhance the general- ization of emergent languages and language acquisition among artificial learners. For example, some studies have used iterated learning to improve the generalization of emergent languages between Al agents [56] 78), while others have applied it to address generalization challenges in tasks like compositional Visual Question Answering (VQA) [98]. LOT shares the same motivation as this line of research. Our primary contribution extends the concept of "ease-of-teaching [56) from language learning to a broader range of machine learning tasks, including supervised, unsupervised learning, and reinforcement learning.
LOT functions as a regularizer, similar to other commonly used regularizers like the 1.2 regularizer. The 1.2 regularizer is effective because it encourages neural networks learn simpler correlations, thereby avoiding overfitting. It is widely accepted that correlations with lower Kolmogorov complexity are more generalizable if they can perfectly explain complex dataset. This aligns with the idea that "generalization equals optimal compression." as discussed by Ilya Sutskever [89]. Essentially, this notion adapts Occam's 's Razor to the field of AI. Our key insight that the "case-of-teaching" metric serves as an effective regularizer beyond language emergence tasks.
Consider an intuitive example: Student A learns math by rote memorization, while Student B understands the core concepts and only memorizes essential rules, deducing the rest when needed. Both approaches can perform similarly on simple problem sets. However, data complexity increases, Student A's burden grows significantly, while Student B's understanding- remains manageable. Consequently, Student B's knowledge is easier teach to another student, it involves less complexity. Therefore, teachability (or imitability) can serve as a proxy for complexity.
approach
D Implementation Details.
Atari Games. We perform experiments on four Atari games, namely Beam-Rider, Breakout, UpNDown, and Gravitar, following the implementation outlined in [42]. We set the regularization coefficient to 0.5 for BeamRider, Breakout, and UpNDown, and to 0.1 for Gravitar. The other hyperparameters remain consistent across all four games. We use of 5. For all agents, the optimizer employed is Adam, with an initial learning rate of 0.00025. The teacher agent is trained for a total of 20,000,000 timesteps. The temperature used in the KL loss is set to 1. The experiments are implemented on the NVIDIA A6000 48GB GPUs.
Language Modeling. In the training-from--eraach experiments, we use the Transformer-XL ar- chitecture following Dai et al. [22]. the LSTM architecture following Zaremba al. [106), and the AWD-LSTM architecture following Merity et al. (66]. For supervised fine-tuning experiments with LLaMA-1 and LLaMA-2, we employ the hyperparameters described in Yue et al. [105] and use the HuggingFace Transformers library [00]. The hyperparameters for LOT are detailed in Table 5 The experiments for LSTM and AWD-LSTM are implemented on one single NVIDIA A100 40GB GPU. The Transformer-Xi XL. and LLaMA of LoT are trained on 4 and 8 NVIDIA A100 40GB GPUs, respectively.
Model
Dutaset
a
N
Optimizer
Learning Rate
Training Epochs/Steps
Temperature
LSTM
FTB
1.0
SGD
30
30 Epochs
1.5
AWD-LSTM
PTB
1.0
ASCID
30
250 Esocchs
1.5
Transforer-XL.
WikiText 103
0.1
ADAM
0.01
60,000 Steps
Transfomer-XL-L
WikiText-103
0.1
ADAM
0.01
150,000 Steps
LLaMA-1 7B
GSMSK
0.01
ADAMW
10
Epochs
LLaMA-1 7B
MATH
0.01
ADAMW
x 10-5
Epochs
LI 7B
GSMSK
0.01
AIAAMW
2 10-5
2 Epochss
LLaMA-2 7B
MATH
0.01
ADAMW
2 10
Epochs
Table 5: Hyperparameters for Language Modeling.
Image Classification. For CNN experiments, we use the ImageNet- 1K pretrained architectures MobileNetV2 and ResNets, which can be downloaded from the official PyTorch Model Zod 2
For
19

---

VIT and Swin experiments, we follow the implementations described in Dosovitskiy et al. [24] and Liu et al. 6 using the official ImageNet- IK or ImageNet-21K, pretrained weights downloaded from and The optimal hyperparameters for LOT are obtained through grid research. The detailed hyperparameters are illustrated in T The experiments for MobileNetV2 and ResNets are implemented on one single NVIDIA A A100 40GB GPU. The ViT and Swin experiments are implemented on 4 NVIDIA A100 40GB GPUs.
Model
Dataset
Optimizer
Learning Rate
Training Epochs/Steps
Temperature
MobileNetV2
CIFAR-100
1.0
SGD
0.02
30 Epochs
1.5
ResNet-18
CIFAR-100
SGD
0.02
Fpochs
1.3
ResNet-50
CIFAR-100
1.0
SGD
0.02
Epochs
1.5
ViT-B/16
CIFAR-100
1.0
SGD
0.02
5,000 Steps
1.5
VIT-L/16
CIFAR-100
1.0
SGD
0.02
5,000 Steps
1.5
VIT-B/16
ImageNet- IK
1.0
SGD
0.03
10,000 Steps
1.5
VIT-L/16
ImageNet-1k IK
1.0
SGD
0.03
10,000 Steps
1.5
Swin-B
ImageNet-1k IK
0.5
ADAMW
10-5
15 Epochs Epochs
1.5
Swin-L
Im
0.5
ADAMW
2 10
1.5
Table 6: Hyperparameters for Image Classification.
E
Scalability Analysis
From extensive results shown in Section LoT proves to be widely applicable across various domains, including reinforcement learning (Section B.2), unsupervised learning (Section (..3). and supervised learning (Section (34). It can be effectively applied to different architectures such as CNN-based (Table3). LSTM- (Table D. and mransforner-based(Tbblll
models. LOT works well on both small datasets like PTB (Table T and CIFAR-100 (Table (3), and large datasets such as WikiTText- 103 (T and ImageNet (Table 3). is also suitable for both small models like ResNets (Table 3 and large models like ViT (Table and LLaMA (Table (2. Additionally, LoT compatible with existing regularization methods such as weight decay and dropout. In our experiments with ResNets, weight decay was applied both LoT and Teacher-only setups, In the experiments with Transformer-Xi VIT, and Swin, dropout is applied to both LoT and Teacher-only setups.
F Limitation
potential limitation of LoT lies in the additional computational and memory costs required for training the student models. However, as demonstrated in Section|..5 LoT achieves better general- ization with fewer training steps compared Teacher-only models, and the flexibility in choosing student models can accommodate varying resource constraints. In RL, the additional computational costs introduced by LoT are negligible, as sample collection more resource-intensiv than fitting the agent networks to the samples, as discussed in Section 3.5 Moreover, in real-world settings, inference cost is more critical than training cost. The superior generalization achieved by LoT offers significant benefits during inference without introducing additional inference costs.
G Algorithm for the PPO-version of Our Method
The LoT algorithm for Proximal Policy Optimization (PPO) is illustrated in Algorithm In experiments, the teacher's sampled data B, is continuously added to the student sample collections D., Meanwhile, the most recent samples from D, are used to formulate the student training batch B, to ensure a high quality of its training dataset.
H Additional Results
Computational Efficiency. To further demonstrate the computational efficiency and superiority of LoT, we conduct experiments using LSTM on PTB and ViT-B/16 on CIFAR- with varying training epochs and steps, while keeping other configurations the same as Section 3.3 and Sec- tion 3.4. The results presented in jablessemmnstrral that given equivalent computational budgets, LoT consistently outperforms the Teacher-only model across various datasets and architectures, even when the Teacher-only model trains for twice the number of epochs and steps. This further highlights
htttp:://gthhbb.cc///oooeeeeeere httpaaaarrccciiiiioooooaaaeeeeee tranef former
http::///ggttabb..coom/cteestsss Transforrrr
20

---

Algorithm 2 Learning from Teaching for PPO
1: Input: Regularization Coefficient 0, Student Steps Ratio N >
2: Initialize teacher network parameterized by and student networks S = 1,2, ... K, parameterized by Φ. 3: Initialize replay buffer D, = 0
4: repeat
5: Sample minibatch B, by running T in simulator, add B, to D.
6:
Sample batch of data B, C D,
7:
Compute R(O) [3 ExEB, Ei AeHle, (x)
a
8:
Compute (0) using the PPO loss on minibatch B,
9:
Update 0 using gradient (O)
10:
Fit value network for PPO on minibatch B,
11:
for = 1 to do
12:
Sample B, D. Compute (Ø) -. ExEB, E (x,+(x)
BB.
Update student networks' parameters using loss gradient VoL.(d)
15:
end for
until T converges
ResNet-118 18 Deceptive Student
ResNit-11 Sophisticated Student
ResNet-30 Deceptive Student
ResNett-50 Sophistcated Student
0.4
0.2
Total Fpochs hs
Total Epochs
Figure 5: Training and test KL-divergence losses of student models in LoT using ResNet-18 and ResNet-50 CIFAR-100 with different teacher models.
LoT's effectiveness in improving the teacher model's generalization while maintaining enhanced computational efficiency.
Table 7: Performance of the teacher model in LoT and Teacher-only on image classification. The hyperparameters the same the corresponding experiments in the paper.
Dataset
Teacher
Student
Total Train Epochs/Steps
Teacher-only
LoT
CIFAR-100
VIT-B/16
B/16
10,000 steps
91.57
93.17
CIFAR-100
VIT-B/16
VIT-B/16
15,000 steps
91.74
93.23
CIFAR-100
VIT-B/16
VIT-B/16
20,000 steps
91.82
93.40
PTB
LSTM
LSTM
60 epochs
82.75
71.72
PTB
L.STM
LSTM
90 epochs
82.48
71.22
PTB
LSTM
LSTM
120 epochs
82.42
70.67
Additional Evidence for Hypothesis. We provide additional experimental results to validate our hypothesis using ResNet-50 and ResNet-1s as both the teacher and student models on CIFAR-100. following the same methodology described in Section 3.1 but with different model architectures. The training and test KL-divergence of the sophisticated and deceptive students are shown in Figure 5 We observe that the sophisticated students achieve lower final KL losses compared to the deceptive students with fewer training epochs, which further supports our hypothesis
Out-of-distribation Performance. We conduct additional experiments by fine-tuning models on ImageNet- IK and evaluating them on ImageNet-R and ImageNet-Sketch using ViT-B/16 and ViT-L/16 models to investigate the out-of-distribution robustness of LoT. The results, shown in
21

---

Table 8 demonstrate that LoT also brings performance improvements on these datasets, indicating the robustness of LoT across broader set of scenarios.
Additional Comparison KD. Table we show that LoT outperforms the distillation method BAN. To provide stronger validation of LoT"s effectiveness, we conduct additional experiments using ResNet-50 and ViT-B/16 on CIFAR-100. We compare LOT distillation methods such as BAN, DKD [08]. and ReviewKD KD [1S], with the teacher weights in these methods being the best checkpoint of Teacher-only. The results, shown in Table indicate that LoT achieves better performance than these distillation baselines, further underscoring the effectiveness of the unique interactive learning process of LOT.
Table 9: Performance of LoT, BAN, Re- Table 8: Performance of LoT and Teacher-only viewKD, DKD on CIFAR100. ImageNet-R and ImageNet-Sketchh
Method
Teacher
Student
Accuracy
Dataset
Teacher
Student
Teacher-only LoT
Teacher-only
ResNet-50
N/A
ImageNet-R
ViT-B/16
ViT-B/16
49.11 52.27
BAN
ResNet-50
ResNet-
84.73
ImageNet-R
VIT-B/16
VIT-L/16
49.11 $4.08
ReviewKD
ResNet-50
ResNet-50
85.31
ImageNet-R
VIT-L/16
VTT-B/16
54.42/ 58.18
ResNet-50
ResNet-50
85.17
ImageNet-R
VIT-L/16
VIT-L/16
/4.42/57.79
LoT
ResNet-50
ResNet-50
86.04
ImageNet-Sketch
ViT-B/16
VII-B/16
38.85 41.46
Teacher-only
VIT-B/16
N/A
91.57
ImageNet-Skellen -keekthh
VIT-B/16
V1T-1/16
38.85 42.89
VIT-B/16
ImageNet-Sketee
VIT-L/16
VIT-B/16
43.83 47.61
ReviewKD
VIT-B/16
V/T-B/16
92.73
ImageNet- Sketch
VIT-1/16
VIT-1/16
43.83 45.91
DKD)
VIT-B/16
VTT-/116
92.82
LoT
VIT-B/16
VIT-B/16
Results on Validation Datasets. We provide additional results on the official validation datasets for PTB and WikiText- 103 in Table These results demonstrate that LoT consistently outperforms the Teacher-only approach on both the validation and test datasets for PTB and WikiText- further validating the effectiveness of LoT.
Table 10: Test/Validation perplexity of LoT and Teacher-only on the official test/validation datasets.
Dataset
Teacher
Student
Teacher-only (Valid)
Teacher-only (Test)
LoT (Valid)
LoT (Test)
FTB
LSTM
LSTM
86.02
82.75
73.98
71.72
AWD-1.STM
AWD.LSTM
Wikitext-103
Transfomeer XL-B
Transfenner-X4 XL-B
24.68
23.72
22.24
21.65
Wikitext-101
Transformer-o
Transformer-X0
Performance Student Models. We present the results for the student models in Table [T] Our observations indicate that when the student and teacher models share the same architecture. the student models can achieve performance levels comparable those of the teacher models. While the performance of the student models improves under LoT,, it is important to highlight that LoT is primarily designed to enhance the generalization capabilities of the teacher model.
Detialed Computation Cost. We provide detailed comparison of the computational budget for LOT and Teacher-only in Table 12 Our analysis shows that LoT uses the same number of CPU cores as Teacher-only, with GPU usage being 12% 55% higher. Despite this, LoT exhibits lower training times compared to Teacher-only (except in RL tasks) when subjected to the same total training epochs/steps, while still achieving significant performance improvements.
Table 11: The performance of student models in LoT language modeling and image classification.
Task
Dataset
Teacher
Student
Teacher-only
LoT (Teacher)
LoT (Student)
Language Modeling
PTB
LSTM
LSTM
82.75
71.72
73.33
Language Modeling
WikiText 103
Transformer XL.-L
Transformer XL-L
18.50
16.47
16.89
Image Classification
CIFAR100
ResNet-50
ResNet-18
84.09
Image Classification
CIFAR100
ResNet 50
ResNet 50
84.09
86.04
85.72
Image Classification
ImageNet-1K
ViT-B/16
ViT-B/16
Image Classification
ImageNet- IK
B/T.
N/T.
91.57
93.25
93,89
22

---

Table 12: Computational resources, memory usage, and training time of LoT and Teacher-only.
Teaierr
GRI Maateraselllll
Traknng
Pefferaae (Teu Berroelll
ITea Wnw.br-wdlll.t
Standerd betwerk Stunder Nowert
30M
PONEKA AMRS 4808 ort.
E203 yOU
1017 Htd
Leaez care
pp: 71.1
NYERA aKB LEU
214438
abc
Rocho--231 Rushap-18
00xpoch
NYEER Aii aKB CEU
12,633 GB
54.1 1808
VIFUIS/ VIF-01
205 10p
NYEGA
23,1
2899//1675
asc (MU
Ablation Metrics in LoT Regularizer. We conduct experiments with different metrics for the "imitability" measurement, such as L.2 loss. However, find that using KL-divergence achieves better performance compared to L.2 loss. The results of utilizing L2 loss for the LOT regularizer with ViT-B/16 and ViT-L/16 CIFAR-100 using L2 loss for the LoT regularizer also brings performance improvements, further indicating the effectiveness of LOT regularization.
presented in Table [3] These results show that
Table 13: Performance of using L2 loss for the LOT regularizer on CIFAR100.
Dataset
Teacher
Student
Teacher-only
LoT (KI-Divergence)
LoT (L2)
CIFARIO0
ViT-B/16
ViT-B/16
91.57
93.17
92.77
CIFAR 100
VIT-B/16
VIT-I/16
91.57
93.25
92.94
CIFAR100
ViT-L/16
VIT-B/16
93.44
94.29
94.12
CIFAR100
VIT-L/16
VIT-L/16
93.44
94.18
94.05
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Abstract and Introduction (Section 1 in this paper reflect the contributions
our method. The strong results in our experiments further reflect our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The potential limitation of LOT lies the additional computational and memory costs required for training the student models. However, demonstrate that LoT achieves better generalization with fewer training steps compared to Teacher-only models and the flexibility in choosing student models can accommodate varying resource constraints. We provide discussions in Section 3,5 and Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The implementation details of this paper is fully illustrated in Appendix and we make an extensive effort to ensure the reproducibility of the results in this paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide the code of this paper in the additional supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
descoeuuuus//oo 7/nips cc/ wtc/publiic/ cubbic/guuidss gww.///ieeee/eee// Codeeoobiissseee SanSSuubissss isssinn ssionnoiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting and details are introduced in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All our main results are averaged over multiple runs and the error bar are provided our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources details are provided in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The paper conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The socictal impacts of this paper is discussed in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release new models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly popular datasets and models and obtain the license of using LLaMA models. We credit the license and term of use in the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include license in the code in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "hwl5VEATkl", "reviewer": "Reviewer_JrP1", "summary": "The paper introduces Learning from Teaching (LOT), a regularization technique to enhance the generalization capabilities of deep neural networks. LOT uses separate student models trained by inmate the prediction of the teacher model to provide feedback, promoting the capture of generalizable and imitable correlations. The effectiveness of LOT is demonstrated through significant performance improvements in tasks across Computer Vision, Natural Language Processing, and Reinforcement Learning, showcasing better generalization with fewer training steps compared to traditional methods.", "questions": "1. I want to ask what is the performance of those NLP model or CV model on the validation set of trained datasets? Since it is also a type of generalization. It could be used to show whether the LOT is helpful for preventing over fitting.\n2. For \"The total training steps for LOT and the Teacher-only approach is the same for fair comparison\", my understanding is that the LOT only train half number of epochs comparing to the teacher-only approach? (If we only train with one student model)\n3. The paper only reports the performance the teacher model, what is performance for those students model?", "limitations": "I think the paper adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Originality:\nLOT is a novel regularization technique. This approach is unique in its hypothesis that generalizable correlations are easier to imitate, leading to the design of a regularization method that leverages the feedback from student models to improve the main model's generalization. This represents a creative combination of existing ideas from cognitive science and machine learning, applied in an innovative way to enhance neural network performance.\n\nQuality:\nThe quality of the research is good. The paper have rigorous experimental validation across multiple domains, including Computer Vision, Natural Language Processing, and Reinforcement Learning. The methodology is well-detailed, with clear formulations and algorithms provided for implementing LOT in various learning contexts.\n\nClarity:\nThe paper is well-written and organized.  The introduction and motivation for LOT are clearly articulated, providing a strong rationale for the proposed approach. The methodology section is thorough, with detailed explanations and pseudocode for the LOT algorithm. The experimental results are presented with clear figures and tables, effectively illustrating the benefits of LOT.", "weaknesses": "1. The paper primarily compares LOT to a baseline teacher-only model. It will be the best to Include more comparisons with other regularization methods, such as dropout, batch normalization, or other recent advances in knowledge distillation, would provide a stronger validation of LOT's relative effectiveness, since it is possible that LOT could be replaced by combination of other much similar regularization.\n\n2. Although the paper mentions the computational efficiency of LOT, a more thorough analysis of the scalability and computational cost, especially for larger models and datasets, would be beneficial. Providing detailed benchmarks on computational resources, memory usage and time required for training with LOT compared to other methods would help clarify its practical applicability in real-world scenarios.\n\n3. The paper uses a fixed set of student models for feedback. Exploring the impact of different types of student models with varying capacities and architectures could provide deeper insights into the robustness and versatility of LOT.\n\n4.  While the empirical results are strong, the theoretical foundations behind why generalizable correlations are easier to imitate could be elaborated further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of the novelty and significant improvements brought by LoT, as well as the clearly articulated presentation. Below, we provide detailed responses.\n\n > W1: It will be the best to Include more comparisons with other regularization methods, or other recent advances in knowledge distillation.\n\nAs detailed in Appendix D, LoT is orthogonal to existing regularization methods such as weight decay, dropout, batch normalization, and layer normalization. For instance, in Table 1, we applied dropout to Transformer-XL models following [9]. Additionally, in Table 3, the teacher and student models are equipped with regularizations other than LoT, specifically: ViT (layer normalization, dropout, weight decay) as per the setups in [10]. LoT demonstrates effectiveness across all these settings, thereby adding value to the existing pool of regularization methods.\n\nRegarding knowledge distillation methods, we have shown that LoT outperforms the distillation method BAN [13] (Table 4). To provide stronger validation of LoT’s effectiveness, we conducted additional experiments using ResNet-50 and ViT-B/16 on CIFAR-100. We compared LoT to distillation methods such as BAN [13], DKD [11], and ReviewKD [12], with the teacher weights in these methods being the best checkpoint of Teacher-only. The results, shown in Table 9 of the PDF in GR, indicate that LoT achieves better performance than these distillation baselines, further underscoring the effectiveness of the unique interactive learning process of LoT.\n\n> W2: a more thorough analysis of the scalability and computational cost would be beneficial.\n\nWe have provided a detailed comparison of the computational budget for LoT and Teacher-only in Table 12 in the PDF. Our analysis shows that LoT uses the same number of CPU cores as Teacher-only, with GPU usage being 12% to 55% higher. Despite this, LoT exhibits lower training times compared to Teacher-only (except in RL tasks) when subjected to the same total training epochs/steps, while still achieving significant performance improvements.\n\n> W3: Exploring the impact of different types of student models could provide deeper insights into the robustness and versatility of LOT.\n\nWe appreciate your suggestion. In fact, our findings demonstrate that LoT remains effective across a wide range of student model architectures and capacities, as detailed in Section 3.4 and Table 3 of our paper. Key observations include:\n\n- **Enhanced Teacher Performance with Weaker Students:** For instance, a MobileNetV2 student model significantly boosts the performance of stronger models like ResNet-18 and ResNet-50 by over 1% on CIFAR-100, despite its smaller size and lower capacity.\n- **Improved Generalization with Stronger Students:** Incorporating a ResNet-50 student with a ResNet-18 teacher in LoT enhances the teacher's performance by 1.99%.\n- **Benefits of Diverse Architectures in Transformer-Based Models:** For transformer-based models such as ViTs and Swins, utilizing different architectures for the teacher and student yields better performance than using the same architecture. For example, with a ViT-B/16 student, a ViT-L/16 teacher achieves 0.27% higher accuracy compared to a ViT-L/16 student. \n\n> W4: the theoretical foundations behind why generalizable correlations are easier to imitate could be elaborated further.\n\nTo provide a more comprehensive understanding, we refer you to the theoretical foundations of LoT as discussed in the fields of cognitive sciences and evolutionary linguistics in reference [GR1]. For a detailed mathematical theoretical framework, please refer to [GR3].\n\n> Q1: what is the performance of those NLP model or CV model on the validation set of trained datasets?\n\nIn our experiments, we did not employ a separate validation dataset from the training dataset. We found that LoT performs effectively without the need for hyperparameter tuning on a validation set. This is particularly advantageous given the substantial resource demands associated with hyperparameter tuning on large datasets. Consequently, we believe the test performance remains reliable even without a separate validation set.\n\nTo address your concern, we provided additional results on the official validation datasets for PTB and WikiText-103 in Table 10 of the PDF. These results demonstrate that LoT consistently outperforms the Teacher-only approach on both the validation and test datasets for PTB and WikiText-103, further validating the effectiveness of LoT.\n\n> Q2: my understanding is that the LOT only train half number of epochs comparing to the teacher-only approach? \n\nYes, your understanding is correct. To maintain an equivalent total number of training steps (teacher + student) as the Teacher-only setting, LoT is trained for half the number of epochs or steps compared to the Teacher-only approach (with the exception of RL tasks). Our hypothesis, supported by results presented in Figure 1 and Figure 3 of our paper, as well as Figure 5 and Table 12 in the PDF, demonstrates that under the same total training steps, LoT converges faster and achieves better final performance compared to the Teacher-only approach.\n\n> Q3:  what is performance for those students model?\n\n**We would like to emphasize that LoT is designed primarily to enhance the generalization capabilities of the teacher model.** However, we also provided results for the student models in our experiments, as shown in Table 11 of the PDF. Our observations indicate that when the student and teacher share the same architecture, the student models can achieve performance comparable to that of the teacher models.\n\n**References:**\n\n[9] Transformer-XL: Attentive language models beyond a fixed-length context. ACL 2019\n\n[10] An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2020\n\n[11] Decoupled Knowledge Distillation. CVPR 2022\n\n[12] Distilling Knowledge via Knowledge Review. CVPR 2021\n\n[13] Born again neural networks. ICML 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VhiLi3Aizv", "reviewer": "Reviewer_NPF4", "summary": "This paper proposes the LOT (Learning from Teaching) regularization technique, which employs auxiliary student learners to help the main model capture these more generalizable correlations. The authors hypothesize that generalizable correlations are expected to be easier to imitate, and LOT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The results suggest the effectiveness and efficiency of LOT in identifying generalizable information.", "questions": "Please see the Weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "(i) This work proposes a concept that generalizable correlations are expected to be easier to imitate.\n\n(ii) This work proposes Learning from Teaching (LOT), a novel regularization technique for deep neural networks to enhance generalization. It is to compute a measure of ‘imitability’ for the main model to learn data correlations at the correct scales.\n\n(iii) Multiple experiments demonstrate the effectiveness of LOT", "weaknesses": "(i) The core argument of this article is that \"generalizable correlations are expected to be easier to imitate\", but previous studies have shown that more complex data and correlations contain richer information, where models need to obtain better accuracy to learn, making them more helpful for generalization. These two views seem to be contradictory. At the same time, the author constructed their hypothesis based on human cognition, but the relevant theoretical basis is lacking, which makes its reliability questionable.\n\n(ii) This distillation is similar to meta-learning, but its support for generalization is confused. As we all know, meta-learning performs great generalization by obtaining general knowledge by distilling task-specific knowledge and then using it to complete various tasks. But in this paper, as the author said, \"both the teacher and student models may lack or possess different task-specific knowledge\", it is difficult to understand why it improves generalization. I think it seems to be \"joint learning\", but this learning mechanism and optimization details are missing. I hope the author can further provide LOT's insight.\n\n(iii) The description of the training process in this article is a bit hard to follow. For example, the author mentioned that generalization is improved through \"joint learning\", but in Section 2.2, the teacher-learner and student-learner are optimized alternately. In the introduction, the author mentioned using the student learner as an auxiliary task to improve generalization, but it is not clear how to capture the generalization connection, what this generalization connection is, and how to guide the model learning. More details may be better.\n\nIn summary, my concerns focus on the credibility of the author's motivation and the soundness of the method. If these issues can be addressed, I will be happy to improve my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the novelty of our regularization method and acknowledging that our experiments demonstrate the effectiveness and efficiency of LoT in identifying generalizable information. Below, we provide detailed responses.\n\n> W1: previous studies have shown that more complex data and correlations contain richer information, where models need to obtain better accuracy to learn, making them more helpful for generalization. These two views seem to be contradictory.\n\nThank you for pointing out the potential confusion. We appreciate the opportunity to clarify our hypothesis. Our hypothesis may be better expressed as follows: \"Given a dataset D with rich enough information, if there are two learned correlations A and B that both perfectly explain D, but A is easier to imitate than B, then A is more likely to be generalizable than B.\"\n\nWe agree that complex data containing richer information can indeed aid generalization. However, it's important to note that the complexity of the dataset and the simplicity of the learned correlations are independent factors. To clarify:\n\n1. **Why does high complexity in the dataset help?** Real-world cases are inherently complex, so the dataset should be complex enough to capture this complexity. Otherwise, models might rely on shortcuts that don't work in real scenarios. For instance, memorizing the results of a simple addition rule, like c = a + b when a and b are within ten, is easier than understanding the rule of addition itself.\n\n2. **Why are generalizable correlations easier to imitate given a complex dataset?** Intelligence involves finding simple, teachable rules and correlations within complex real-world data. Learning the rules of addition is simpler than memorizing 1 million addition results. Please refer to [GR2] for further explanation.\n\nIn other words, AI can only emerge when an artificial learner effectively understands simple correlations, rules, and concepts of low Kolmogorov complexity that can successfully explain complex datasets. This idea, widely accepted by the deep learning community [14], can be seen as an implementation of Occam’s Razor, a principle dating back to the 14th century.\n\n\n> W1-2: the author constructed their hypothesis based on human cognition, but the relevant theoretical basis is lacking, which makes its reliability questionable. This distillation is similar to meta-learning, but its support for generalization is confused. As the author said, \"both the teacher and student models may lack or possess different task-specific knowledge\", it is difficult to understand why it improves generalization. I think it seems to be \"joint learning\", but this learning mechanism and optimization details are missing. I hope the author can further provide LOT's insight.\n\nThank you for your insightful feedback.  For a comprehensive review of related ideas and theories in psychology and evolutionary linguistics, please refer to [GR1] and the references therein. Additionally, [GR2] provides further insights into the concept of Learning from Teaching (LoT).\n\nIt is worth noting that our method is orthogonal to the meta-learning paradigm, which involves a random distribution of tasks in the meta-learning setup. The learning paradigm most closely related to our method is iterated learning, where “ease-of-teaching” is widely accepted as an important concept [15]. Our contribution can be viewed as a generalization of the iterated learning idea to broader machine learning tasks.\n\nFurthermore, please refer to [GR3] for a discussion on the challenges of developing comprehensive mathematical theories in this context.\n\n\n\n> W3: The description of the training process in this article is a bit hard to follow. The author mentioned that generalization is improved through \"joint learning\", but in Section 2.2, the teacher-learner and student-learner are optimized alternately. In the introduction, the author mentioned using the student learner as an auxiliary task to improve generalization, but it is not clear how to capture the generalization connection, what this generalization connection is, and how to guide the model learning. More details may be better.\n\nThank you for your feedback. We apologize for any confusion regarding the description of the training process. \n\nTo clarify, in LoT, the teacher and student are optimized alternately. The student learns from the teacher (Eq. 2), and the teacher is optimized based on both the regular task loss and the LoT regularizer, which is the feedback from the student (Eq. 1). The detailed procedure of LoT is outlined in Algorithm 1 in the paper.\n\nThe generalization connection is embodied in the LoT regularizer. According to our hypothesis, a smaller imitation loss indicates a more generalizable teacher model. By incorporating the LoT regularizer as an additional loss to the teacher, the optimization process encourages the teacher to achieve a smaller imitation loss. Consequently, the generalization of the teacher model is enhanced compared to models that do not employ LoT.\n\n**References:**\n\n[14] Ilya Sutskever, An Observation on Generalization, Simons Institute, 2023  \n[15] Ease-of-Teaching and Language Structure from Emergent Communication, ICLR 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TPxNHylV5Z", "reviewer": "Reviewer_DocJ", "summary": "The authors proposed a novel regularization approach called Learning from Teaching (LoT) to enhance generalization. The hypothesis that simple correlations are generalizable is the main question for this work. Through a teacher student approach, the authors are capable of provide to the main model more generalizable and imitable", "questions": "- The hypothesis on Section 2.1 have been proposed before? I better discussion on that will help the overall reading and to better position the paper when compared to other approaches.\n- For all the experiments N (student steps ratio) was 1? If not the authors compared LoT with other methods for the same number of steps? If not the superior number of steps can partially be the cause of the performance gains. \n- Regarding the choice of metrics the authors chose KL-divergence, did the authors experimented with other losses? \n- Not sure if it is feasible but benchmark LoT on ImageNet-R [1] and ImageNet-Sketch [2] will be interesting for a broader set of results.\n\n[1] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR pp. 8340–8349, 2021. \n\n[2] Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning ro- bust global representations by penalizing local predictive power. In NeurIPS, pp. 10506–10518, 2019.", "limitations": "- The current set of baselines can be expanded. Methods like [3] and [4] can provide a better understanding on how well LoT is positioned. \n\n[3] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021 \n\n[4]  Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang, Decoupled Knowledge Distillation, CVPR, 2022.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The LoT computation measurement of ‘imitability’ through the student, later used as the regularizer, is the paper main contribution. \n- Comprehensive experimental section with a broad set of experiments, such as RL, Fine-tuning and Image classification.", "weaknesses": "- I would like to see further evidences of the proposed hypothesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the novelty of our method and the comprehensiveness of our experiments.\n\n> W1: further evidences of the proposed hypothesis.\n\nThank you for your request for further evidence supporting our hypothesis.\n\nWe have provided additional experimental results to validate our hypothesis using ResNet-50 and ResNet-18 as both the teacher and student models on CIFAR-100, following the same methodology described in Section 3.1, but with different model architectures. The training and test KL-divergence of the sophisticated and deceptive students are shown in Figure 5 of the PDF in the General Responses. We observe that the sophisticated students achieve lower final KL losses compared to the deceptive students with fewer training epochs, which further supports our hypothesis.\n\nDue to the time constraints of the rebuttal process, we have included additional results for two models on CIFAR-100. We promise to provide more extensive results in the final version of our paper.\n\n> Q1: The hypothesis on Section 2.1 have been proposed before? I better discussion on that will help the overall reading and to better position the paper when compared to other approaches.\n\nThank you for your insightful feedback. We agree that providing more context and discussion on the motivations and background of our hypothesis is crucial for better positioning our paper.\n\nFor in-depth discussions, please refer to [GR1] and [GR2].\n\nThe hypothesis presented in Section 2.1, in the context of human language acquisition and emergence, is a widely accepted concept in cognitive sciences and linguistics [17, 18, 19, 20]. In the AI community, similar hypotheses have been proposed concerning artificial language emergence [16]. However, to the best of our knowledge, we are the first to propose that this principle is widely applicable across different domains and can be used to deduce a practical regularizer.\n\nWe will incorporate these discussions to better position our paper and provide a clearer comparison with other approaches.\n\n> Q2: For all the experiments N (student steps ratio) was 1? If not the authors compared LoT with other methods for the same number of steps? If not the superior number of steps can partially be the cause of the performance gains.\n\nThank you for your question. \n\nYes, in all our experiments \\(N = 1\\), ensuring that the total training steps (Teacher + Student) for LoT are the same as for the Teacher-only approach. This setup maintains an equal training steps budget for a fair comparison. Therefore, the performance gains observed are not due to additional training steps, as both LoT and Teacher-only utilize the same total number of steps.\n\nAdditionally, we present results in Table 7 of our paper where we further increased the training steps for both LoT and Teacher-only. These results show that further increasing the training steps does not significantly improve performance, which further underscores the effectiveness of LoT in achieving performance gains under the same training steps as the Teacher-only approach.\n\n> Q3: Regarding the choice of metrics the authors chose KL-divergence, did the authors experimented with other losses?\n\nThank you for your question regarding the choice of metrics.\n\nWe have experimented with different metrics for the “imitability” measurement, such as L2 loss. However, we found that using KL-divergence achieves better performance compared to L2 loss. The results of utilizing L2 loss for the LoT regularizer with ViT-B/16 and ViT-L/16 on CIFAR-100 are presented in Table 13 in the PDF. These results show that using L2 loss for the LoT regularizer also brings performance improvements, further indicating the effectiveness of LoT regularization.\n\n> Q4: LoT on ImageNet-R [1] and ImageNet-Sketch [2] will be interesting for a broader set of results.\n\nThank you for your suggestion.\n\nWe conducted additional experiments by fine-tuning models on ImageNet-1K and evaluating them on ImageNet-R and ImageNet-Sketch using ViT-B/16 and ViT-L/16 models to investigate the out-of-distribution robustness of LoT. The results, shown in Table 8 of the PDF, demonstrate that LoT also brings performance improvements to these datasets, indicating the robustness of LoT across a broader set of scenarios.\n\n> Q5: The current set of baselines can be expanded. Methods like [3] and [4] can provide a better understanding on how well LoT is positioned.\n\nThank you for your suggestion regarding expanding the set of baselines.\n\nIn our paper, we have already demonstrated that LoT achieves better performance than the distillation method BAN [23] (Table 4). To further validate LoT’s effectiveness, we conducted additional experiments using ResNet-50 and ViT-B/16 on CIFAR-100, comparing LoT to distillation methods such as BAN [23], DKD [21], and ReviewKD [22]. The teacher weights utilized in these methods were the best checkpoint for Teacher-only. The results, shown in Table 9 of the PDF, indicate that LoT outperforms these distillation baselines, underscoring the effectiveness of LoT's unique interactive learning process.\nSure, here are the references in markdown format with each on a different row:\n\n**References:**\n\n[16] Ease-of-Teaching and Language Structure from Emergent Communication, ICLR 2019  \n[17] Iterated Learning: A Framework for the Emergence of Language, Artificial Life, 2003  \n[18] Iterated Learning and the Evolution of Language, Current Opinion in NeuroBiology, 2014  \n[19] Cumulative Cultural Evolution in the Laboratory: An Experimental Approach to the Origins of Structure in Human Language, PNAS, 2008  \n[20] Spontaneous Evolution of Linguistic Structure: An Iterated Learning Model of the Emergence of Regularity and Irregularity, IEEE Transactions on Evolutionary Computation, 2001  \n[21] Decoupled Knowledge Distillation, CVPR 2022  \n[22] Distilling Knowledge via Knowledge Review, CVPR 2021  \n[23] Born Again Neural Networks, ICML 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Highlighted General Response**

We sincerely appreciate all reviewers’ time and efforts in reviewing our paper. We also thank all reviewers for the insightful and constructive suggestions, which helped a lot in further improving our paper. In addition to our point-by-point responses below, we provide the following highlighted general responses.

**[GR1] Motivations of LoT and Related Theories in Cognitive Sciences and Linguistics:**

The concept of Learning from Teaching (LoT) originates in cognitive psychology and linguistics, particularly within the iterated learning theory of language emergence [4, 5, 6, 7]. This theory posits that the generalizable nature of languages arises from the iterative learning process across generations in a society. The core hypothesis is that a generalizable language is inherently easier to teach and learn [4, 5,6,7], which aligns with our main hypothesis.

In the AI community, recent research has aimed to employ iterated learning to enhance the generalization of emergent languages and language acquisition among artificial learners. For example, some studies have used iterated learning to improve the generalization of emergent languages between AI agents [1, 3], while others have applied it to address generalization challenges in tasks like compositional Visual Question Answering (VQA) [2].

LoT shares the same motivation as this line of research. Our primary contribution extends the concept of “ease-of-teaching” [1] from language learning to a broader range of machine learning tasks, including supervised, unsupervised learning, and reinforcement learning. We appreciate the reviewers' feedback and will incorporate these discussions into the related works and motivation sections.

**[GR2] Core Insights on Why LoT Can Improve Generalization:**

LoT functions as a regularizer, similar to other commonly used regularizers like the L2 regularizer. The L2 regularizer is effective because it encourages neural networks to learn simpler correlations, thereby avoiding overfitting. It is widely accepted that correlations with lower Kolmogorov complexity are more generalizable if they can perfectly explain a complex dataset. This aligns with the idea that "generalization equals optimal compression," as discussed by Ilya Sutskever [8]. Essentially, this notion adapts Occam’s Razor to the field of AI. Our key insight is that the "ease-of-teaching" metric serves as an effective proxy for the uncomputable Kolmogorov complexity, thus leading to a good regularizer. 

Consider an intuitive example: Student A learns math by rote memorization, while Student B understands the core concepts and only memorizes essential rules, deducing the rest when needed. Both approaches can perform similarly on simple problem sets. However, as data complexity increases, Student A's burden grows significantly, while Student B's understanding-based approach remains manageable. Consequently, Student B's knowledge is easier to teach to another student, as it involves less complexity.

**[GR3] Mathematical Theory and Foundations for LoT:**

While LoT is grounded in theoretical findings from linguistics and cognitive science, we do not expect a comprehensive formal theory to emerge due to the hypothesis's dependence on specific data. In practical deep learning tasks, generalizable correlations tend to be simple enough for human comprehension. However, distinguishing between "natural" datasets and artificially complex ones is challenging.

Consider the task of learning arithmetic correlations between integers. If we have a dataset of triplets (a, b, c), there are two ground truth correlations: c = a + b and c = (a + b) mod $10^{99}$. If the dataset isn't large enough, these two correlations are indistinguishable from each other. However, our claim is that c = a + b is easier to imitate than c = (a + b) mod $10^{99}$, because the former has lower Kolmogorov complexity [8]. This claim is supported by cognitive evidence, but formalizing these intuitions mathematically—proving something holds for one correlation and not the other—is very challenging.

**References:**

[1] Ease-of-Teaching and Language Structure from Emergent Communication, ICLR 2019  
[2] Iterated Learning for Emergent Systematicity in VQA, ICLR 2020  
[3] Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings, NeurIPS 2023  
[4] Iterated Learning: A Framework for the Emergence of Language, Artificial Life, 2003  
[5] Iterated Learning and the Evolution of Language, Current Opinion in NeuroBiology, 2014  
[6] Cumulative Cultural Evolution in the Laboratory: An Experimental Approach to the Origins of Structure in Human Language, PNAS, 2008  
[7] Spontaneous Evolution of Linguistic Structure: An Iterated Learning Model of the Emergence of Regularity and Irregularity, IEEE Transactions on Evolutionary Computation, 2001  
[8] Ilya Sutskever, An Observation on Generalization, Simons Institute, 2023

Meta-review: This paper introduces LOT (Learning from Teaching), a regularization technique that uses auxiliary student learners to help the main model capture more generalizable correlations. The key hypothesis is that generalizable correlations are easier to imitate, and LOT leverages this by employing auxiliary learners to enhance the main model's generalization.

All reviewers found the proposed approach effective, supported by strong experimental results. However, the primary concern was the **validity of the hypothesis** on which this method is based. During the response period, the authors provided additional experimental evidence to validate their hypothesis, which satisfactorily addressed the reviewers' concerns.

Given the effectiveness of the approach and the comprehensive response, the paper is considered above the acceptance threshold. For the final version, it is suggested that the authors de-emphasize the hypothesis and its motivations from Cognitive Science or Linguistics. Instead, the focus should be on the demonstrated superiority of the proposed method itself, as highlighted in the ablation studies provided during the response period.
Decision: Accept (poster)