Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate

Authors: jin  can
tong  che
hongwu  peng
yiyuan  li
dimitris n metaxas
marco  pavone
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:42:32

Abstract: Generalization remains a central challenge in machine learning. In this work, we propose *Learning from Teaching* (**LoT**), a novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability to capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners are trained by the main model and, in turn, provide feedback to help the main model capture more generalizable and imitable correlations. Our experimental results across several domains, including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction of LoT brings significant benefits compared to training models on the original dataset. The results suggest the effectiveness and efficiency of LoT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code is available at https://github.com/jincan333/LoT.

Main paper: 
 introduction: 
 Improving the generalization performance of models on unseen data is a major challenge in machine learning [6, 7, 73, 79, 107] . Despite its significant advances, identifying the most generalizable model within the vast space of potential models remains challenging. Existing deep learning approaches focus on crafting the hypothesis spaces where prediction errors are optimized using training data [33, 69, 71] . These spaces are shaped by inductive biases [33, 70] embedded in the neural architectures which include implicit assumptions about the data [1, 25, 95] , objective functions (notably regularizers) [20, 68, 103] , and learning methodologies [14, 72, 87] .In this paper, to enhance generalization, we use the methodology of regularization [37, 51, 88] , which prioritizes specific regions in the hypothesis spaces. Regularization techniques often involve employing auxiliary losses or regularizers [20, 38, 103] alongside the primary task losses. For instance, L1 regularization [41, 92, 93] encourages sparsity within models [16, 40, 54, 57] . Other regularization techniques include model averaging [44, 102] , dropout techniques [37, 67, 100] , and additional 0 * Equal contribution, † Equal contribution, ‡ Equal advising, Correspondence to: Can Jin <can.jin@rutgers.edu>, Tong Che <tongc@nvidia.com>.optimization components [43, 63, 104] . Due to its effectiveness and simplicity, regularization is critical in modern machine learning techniques for achieving better generalization [37, 109] .We aim to answer the research question: Among all possible models fitting the training data, which ones are inherently generalizable? A common belief in cognitive science is that human intelligence development involves distilling information and filtering out extraneous details to discern 'simple' correlations among a few selected relevant abstract variables [18, 94] . This approach leads to the formation of correlations through simple patterns [2, 56] at the right scales. However, identifying simple correlations in deep learning remains challenging, mostly due to not being easy to identify the right scale of the problem. Studies in emergent languages suggest that the more structured a language is, the more efficiently it can be transmitted to message receivers [11, 56] . Inspired by this finding, we propose defining simple and generalizable correlations at the right scales, as those that can be readily imitated by other learners, provided they possess suitable inductive biases.Based on this definition, we propose a novel regularization approach, Learning from Teaching (LOT ). The core of LOT is to compute a measure of 'imitability' for the main model to learn data correlations at the correct scales. By adding this measure to the objective function and optimizing it during training, we encourage the teacher model to refine its learned multiscale correlations, making them more accessible through teaching, which in turn leads to better generalization. LOT computes this measure by jointly training the main model as the 'teacher' with one or more auxiliary 'student' models. The student models strive to distill and assimilate the correlations acquired by the teacher model. Thus, the learning performance of the student defines the measure of imitability of the teacher, which is then used as the LOT regularizer.We conduct comprehensive experiments using LOT to improve the Reinforcement Learning (RL) formulation, as well as in Natural Language Processing (NLP) and Computer Vision (CV) applications. In RL, the experimental results demonstrate that LOT attains an average normalized reward enhancement of 44% on four Atari games. In language modeling tasks, LOT achieves significant perplexity reductions on the Penn Tree Bank [64] and WikiText-103 [65] . Notably, LOT enhances the supervised fine-tuning performance of LLaMA [96, 97] models on GSM8K [19] and MATH [35] . In image classification tasks, LOT achieves accuracy gains of 1.99% and 0.83% on CIFAR-100 [49] and ImageNet-1K [23] , respectively. 
generalizable and spurious correlations: 
 Given a dataset D = {(x 1 , y 1 ), • • • , (x n , y n )} generated from a data-generating distribution D, there are infinitely many continuous functions f such that f (x) = y for all (x, y) ∈ D. Therefore, finding the f that precisely models the true generalizable correlation between x and y is challenging, especially with real-world data like natural images, which are complex and multiscale. In such scenarios, a neural network may compute incorrect (according to the ground-truth relationship between variables) yet perfect (in the empirical data distribution) correlations that explain the relationship between x and y [32, 73] . This phenomenon is particularly evident when y is entirely noise-based and independent of x, but the neural network still fits y to x perfectly [73, 107] . This process, often called brute-force memorization [2, 13] , involves the network creating intricate computational strategies to encode all (x, y) pairs in the samples. Consequently, correlations established in this way are spurious, originating from sampling noise in the data rather than ground-truth relationships.But how do humans distinguish generalizable correlations from spurious ones? Instead of relying on brute-force memorization to establish input-output correspondences, humans naturally focus on understanding high-level concepts within the input data, selectively ignoring irrelevant details [18, 94] . This approach leads to the formation of correlations through simple, comprehensible patterns [2, 11] . Empirical evidence in emergent languages also suggests that the more compositional a language is, the more learners will use it [11, 56] .We can, therefore, define the distinctions between generalizable and spurious correlations. First, generalizable correlations are simple and comprehensible, exhibiting lower Kolmogorov Complexity [31, 58, 90] . Second, while there is only one ground-truth correlation for a dataset, the number of spurious correlations can be massive. These two major distinctions lead to the following hypothesis.Hypothesis: Generalizable correlations should be more easily imitable by learners compared to spurious correlations. Specifically, assume T G and T S are two teacher models that capture the generalizable correlation and spurious correlation from a dataset, respectively. We have student learners S G and S S that separately imitate T G and T S :• From an effectiveness perspective, the final training and test losses of learner S G after training are typically lower than those of learner S S .• From an efficiency perspective, during training, the test losses of learner S G decrease more rapidly than those of S S .This hypothesis emphasizes that generalizable correlations inherent in data are not only more interpretable but also more readily imitable. It suggests that the inherent simplicity and uniqueness of generalizable correlations make them more attainable and recognizable for learning algorithms, in contrast to the complex and abundant nature of spurious correlations derived from noise. In the following we present our novel approach.Building upon the Hypothesis, we propose that the ease of imitation of the teacher model by student models can serve as a proxy for the generalizability of learned representations. By measuring the 'imitability' of the teacher model in the learning process, we can infer the generalizability of it. A teacher that is easier to imitate implies higher generalization. We then design a novel regularization approach that involves training a teacher model T alongside student models S to imitate T , subsequently measuring the imitability of the teacher during training. We maximize imitability by incorporating it as an additional loss during the training of the teacher T . This imitability loss is termed the Learning from Teaching regularizer (LOT regularizer). By doing so, T is optimized to be a teacher that is easier to imitate and, thus, possesses superior generalization compared to models without the LOT regularizer. We refer to this class of regularization methods as 'Learning from Teaching Regularization' (LOT ). LOT aligns with the broader concept of regularization in machine learning, where the goal is to promote generalizable representations and prevent overfitting.Although LOT can be applied to supervised, unsupervised, and reinforcement learning, we begin our discussion with supervised learning. We train a network T θ , parameterized by θ, as the main model, which also serves as the teacher model. Additionally, we train a set of K networks S i , i = 1, 2, • • • , K, as the student models 1 . The total set of parameters of the K networks is denoted by ϕ. Given a training dataset D t = {(x 1 , y 1 ), • • • , (x n , y n )}, we train T and S to model p(y|x), denoted as p t (y|x) and p s (y|x), respectively. Additionally, LOT includes a predefined imitability metric µ s,t (•) = µ(S(•), T (•)). Intuitively, µ s,t measures the difference between S and T 's predictions on the same input (occasionally denoted as µ henceforth for convenience). There are many possible choices for the metric µ, such as the L 2 loss between the hidden representations of a specific layer.In our experiments, we choose µ(x) = µ KL (p s (y|x)||p t (y|x)), which is the KL-divergence [21] , to quantify the distribution similarity between S and T .We first train the teacher model. The objective function of the teacher combines the regular task loss with the additional LOT regularizer R(θ) (defined in Equation 3). For example, in supervised learning, we can use the negative log-likelihood loss for the regular task loss, and the objective function can be written as:EQUATIONwhere |D t | is the number of samples in the dataset D t .To train the student networks and enhance information diversity, we require an independent unlabelled dataset, denoted asD s = {x 1 , • • • , x m }.This dataset can be identical to D t , or generated either by a generative model trained on D t or through alternative augmentation methods (e.g. synthetic data generation). This unlabelled dataset constitutes the environment for the student networks to follow the prediction of the teacher and, therefore, explores and generalizes beyond the original training data.Specifically, the student networks' goal is to imitate the correlations acquired by the teacher network during the training process. The training loss for students can be written as:EQUATIONwhere |D s | is the number of samples in the unlabelled dataset D s . The loss function L s encourages the student networks to learn from the teacher network by minimizing the difference between their predictions, as measured by the metric µ s,t (x).The feedback from all students S i constitutes the LOT regularizer:EQUATIONwhere λ i ≥ 0 represents the coefficient weight of the i-th student S i , withK i=1 λ i = 1.The λ i can be either a learnable parameter or fixed, such as 1 K . Essentially, the LOT regularizer measures the imitability of the teacher. The regularization coefficient α controls the trade-off between the original task learning objective of T and the feedback from the students.The detailed procedure of LOT for supervised and unsupervised learning is outlined in Algorithm 1, and LOT regularization for RL (using PPO as an example) is outlined in Algorithm 2. The teacher T and student S i networks are initialized differently to ensure they learn diverse features and representations. In both algorithms, the teacher and student networks iteratively learn from each other, with the students imitating the teacher's correlations and the teacher incorporating the students' feedback into the learning process. EQUATIONUpdate student networks' parameters ϕ using loss gradient ∇ ϕ Ls (ϕ)12:end for 13: until T convergesThe works most related to LOT are knowledge distillation (KD) [29, 36] and ease-of-teaching [11, 56] in emergent languages. However, LOT differs significantly from these approaches. In KD, a teacher model containing task-specific knowledge transmits this knowledge to a student model (often smaller than the teacher), with the primary focus on the student's performance post-distillation. Conversely, in LOT , both the teacher and student models may lack or possess different task-specific knowledge. Generalization is improved through joint training, incorporating additional signals from student feedback. In emergent languages, Li and Bowling [56] propose that structured language is easier to teach to other agents than less structured ones, achieving higher task success rates with less training. Additionally, Chaabouni et al. [11] identify a strong positive correlation between language transmission efficiency to new message receivers and the degree of compositionality (structuredness) of the language. In LOT , we focus on tasks distinct from emergent languages, finding that generalizable correlations are easier to imitate. Under our Hypothesis, we design a novel LOT regularizer and algorithm to enhance the generalization of deep neural networks, extending the ease-of-teaching concept to supervised, unsupervised, and reinforcement learning. In parallel work, Ning et al. [74] proposes Learning by Teaching (LbT), which utilizes teacher and student models to generate answers as training samples for the teacher model. However, the regularization method in LOT is fundamentally distinct from that in Ning et al. [74] . 
experiments: 
 We first validate our Hypothesis in Section 3.1. Subsequently, we assess the performance of LOT across several tasks: Atari games (Section 3.2), language modeling (Section 3.3), and image classification (Section 3.4). We compare LOT to a Teacher-only baseline, wherein the regularization coefficient α in R(θ) is set to 0, thereby blocking the student feedback. Unless specified otherwise, we employ only one student model. Except for the Atari games where the student can learn from the offline samples of the teacher, we set N = 1 to manage computation (we study the impact of N in Section 3.6). Moreover, we study the computational efficiency and effects of hyperparameters of LOT in Sections 3.5 and 3.6.In our Hypothesis, learners are presumed to more readily imitate generalizable correlations than spurious ones. To investigate this, we design experiments involving two distinct teacher models: a sophisticated teacher and a deceptive teacher. The sophisticated teacher effectively captures generalizable correlations, while the deceptive teacher primarily learns spurious correlations. We use an identical student model to learn from both teachers separately, monitoring the student-teacher KL divergence during training and testing. The student that learns easier-to-imitate correlations is expected to exhibit lower training and test KL losses with fewer training steps.We employ the ViT-B/16 and ViT-L/16 architectures [24] for both the teachers and students. The sophisticated teachers are trained on the full CIFAR-100 [49] training set for 10,000 steps to achieve optimal convergence. The deceptive teachers, using the same hyperparameters and training steps as the sophisticated teachers, are trained on a random subset of 2, 560 images from the CIFAR-100 training set, leading to over-fitting. Consequently, the sophisticated teachers are expected to exhibit better generalization ability (their test accuracy surpasses that of the deceptive teachers by 14%).The two student models referred to as the sophisticated student and the deceptive student, share identical hyperparameters and initializations. They are trained to imitate the correlations from their respective teachers on the full CIFAR-100 training set. The teacher models are kept frozen during the training of the students, with the objective L s (ϕ) defined as follows:EQUATIONwhere D s represents the full training set of CIFAR-100.We present the training and test losses in Figure 1 and make the following observations: • Given the same computational budget, the sophisticated students achieve lower final KL losses on both the training and test sets compared to the deceptive students. This suggests that the student can more effectively imitate the prediction distribution of a teacher that captures generalizable correlations.• The deceptive students require more training steps to achieve the same training and test student-teacher KL losses as the sophisticated students. This indicates that learners tend to grasp spurious correlations much more slowly than generalizable correlations.These results suggest that generalizable correlations are easier to imitate than spurious ones. In LOT , we expect the teacher model to master generalizable correlations by incorporating feedback from students via the LOT regularizer.We conduct experiments on four Atari games, namely BeamRider, Breakout, UpNDown, and Gravitar, following the implementation in Huang et al. [42] . Both the LOT and Teacher-only agents have identical hyperparameters. All agents are trained using Proximal Policy Optimization (PPO) [83] .While the teacher agents interact with the game environment, the student agents are trained on the most recent 10,240 samples generated by the teacher agents, ensuring that LOT and Teacher-only experience the same environmental interactions. We use different α values for various games and set N = 5 to efficiently imitate the teacher. More details are provided in Appendix D.The empirical results are presented in Figure 2 , and we make the following observations:• LOT improves the agent return compared to the Teacher-only version with 20 million teacher training steps. Specifically, LOT achieves {63.14%, 9.79%, 66.48%, 35.70%} normalized return enhancements on {BeamRider, Gravitar, UpNDown, Breakout}.• The performance gain of LOT becomes more prominent as the training progresses (from 15 million to 20 million steps).These results suggest that LOT is an effective approach for enhancing the generalization of RL agents, as it requires no additional environmental interactions while delivering significant performance gains.Language modeling is a widely acknowledged NLP task, and regularization techniques have been demonstrated to significantly enhance performance in this domain [106] . To examine the impact of LOT on language modeling, we conduct experiments in two scenarios: unsupervised language pretraining and supervised fine-tuning.We conduct experiments of LOT and Teacher-only using LSTM [39] , AWD-LSTM [66] , and Transformer-XL [22] for teacher and student on Penn Tree Bank (PTB) [64] and WikiText-103 [65] .We follow the implementations outlined in Dai et al. [22] , Merity et al. [66] , Zaremba et al. [106] .In LOT , we utilize different coefficients α for various architectures and benchmarks to control the LOT regularizer. To ensure a fair comparison, we maintain the same total number of training steps Furthermore, to evaluate the effectiveness of LOT in fine-tuning pretrained large language models (LLMs), we conduct supervised fine-tuning (SFT) experiments using LLaMA-1 [96] and LLaMA-2 [97] on two mathematical reasoning benchmarks: GSM8K [19] and MATH [35] .We compare LOT to in-context learning (ICL) [9] and SFT. Following Touvron et al. [97] , the number of in-context examples is 8 for GSM8K and 4 for MATH. The SFT configuration follows Yue et al. [105] , and we fine-tune the LLaMA models for four epochs. In LOT , the teacher and student models share the same architecture for simplicity. The models are trained for two epochs in LOT to match the total training steps in SFT for fair comparison. All other configurations are consistent with those used in SFT. More implementation details are described in Appendix D. We measure the accuracy of greedy decoding results in Table 2 , and we observe that LOT enhances reasoning abilities on all architecture and dataset choices. This indicates the competence of LOT in improving the fine-tuning performance with a computational cost comparable to SFT.To investigate the effects of LOT on computer vision tasks, we apply LOT to image classification by conducting experiments using ResNets [34] , MobileNetV2 [81] , ViT [24] , and Swin [61] architectures pretrained on ImageNet-1K and ImageNet-21K [23] as teacher and student models. We choose CIFAR-100 [49] and ImageNet-1K as the downstream datasets. The total training steps for LOT and the Teacher-only approach are the same for a fair comparison. Further implementation details are provided in Appendix D. We conclude the following observations from results in PPO-version LOT ). Therefore, LOT introduces negligible computation costs since sample collections are more resource-intensive than fitting the agent network to the samples in RL. For instance, in our Atari games experiments, the training time of LOT (606 minutes) is comparable to the Teacher-only setting (597 minutes) on a single NVIDIA A6000 GPU.Comparison to KD. To investigate the effect of LOT compared to other student-teacher learning paradigms, we compare LOT to the born-again networks (BAN) baseline [29] . In BAN, we select the checkpoint with the best performance of the Teacher-only model as the (frozen) teacher and distill its knowledge into a student model with an identical architecture. Equal weights are assigned to the hard loss (from the dataset) and soft loss (from the teacher) to train the student model [36] . All other configurations remain consistent with LOT . The results in Table 4 indicate that LOT achieves superior performance than BAN with a strong feedback model, further indicating the significance of the interactive learning process in LOT .Effect of regularization coefficient α. The strength of regularization plays a crucial role in the overall training effect [50] . To investigate the effects of LOT on the generalization of the teacher model, we perform experiments on PTB using the LSTM architecture for both teacher and student models. The configuration follows Section 3.3, except that we gradually increase the value of α in LOT from 0 to 1.7 and examine the test PPL of the teacher model. The results are presented in Figure 4 (left). We observe that the performance of the teacher model improves rapidly as α increases from 0 to 1, and when the value exceeds this point, the performance of the teacher begins to decline. This observation suggests that moderate feedback from the student is most beneficial for the teacher, but an excessively strong signal can hinder the teacher's learning process. Similar effects of large α values have been noted in joint teacher-student training in knowledge distillation [75] .Effect of student steps ratio N . To demonstrate the importance of the student steps ratio N in LOT , we conduct additional experiments by training LSTM teacher and student models on PTB using various values of N . The empirical results presented in Figure 4 (right) indicate that the teacher benefits most from a moderate N value, such as 4 or 5. This finding suggests that achieving a balanced ratio between teacher and student model updates is crucial for optimal performance. When N is too low, the student may not sufficiently learn from the teacher, thereby reducing the quality of the feedback it provides. Conversely, if N is too high, the student may overfit the teacher's errors, resulting in less effective imitability measurement.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
conclusion: 
 Identifying generalizable multiscale correlations from the vast space of possible correlations remains a significant challenge in machine learning. Inspired by cognitive science beliefs about human intelligence, we have shown experimentally that generalizable correlations are more imitable by other learners. In particular, we introduced a novel regularization method, LOT , which identifies generalizable correlations by teaching student models and exploiting their feedback. We conducted comprehensive experiments across various learning tasks and neural architectures. The results demonstrate that our proposed regularizer enhances model performance effectively and efficiently. In conclusion, our proposed LOT regularization offers a promising new approach to improve the generalization of neural networks by leveraging the learning process of student models and incorporating their feedback to refine the teacher model.[109] Zhenzhu Zheng and Xi Peng. Self-guidance: Improve deep neural network generalization via knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3203-3212, January 2022.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: The implementation details of this paper is fully illustrated in Appendix D and we make an extensive effort to ensure the reproducibility of the results in this paper. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not release new models or datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use publicly popular datasets and models and obtain the license of using LLaMA models. We credit the license and term of use in the code in the supplementary material.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this work, we propose a regularization method to improve the generalization of deep neural networks. Our work focuses on technical contributions to deep learning and AI. Therefore, the potential social impacts of AI in general apply to our work, including fake information, toxic content, fairness concerns, and misuse of AI. For example, toxic content like hate speech can lead to data contamination and therefore have harmful impacts on society, which has been observed in large-scale pretrained models. By employing our method, such harmful behavior can potentially be amplified.Regularization serves as a primary strategy to improve generalization capabilities and mitigate over-fitting [52] . which compresses ensemble neural networks into a single network, and Ba and Caruana [3] , which improves shallow neural network accuracy by mimicking deep networks. KD is also applied in various domains, including deep reinforcement learning [80] , continual learning [28, 59, 85] , and learning privileged information theory [62, 76] . The dark knowledge method [36] further develops KD, where a student model aims to fully match the output distribution of the teacher. Intuitively, distillation is effective because the teacher's output distribution over classes provides a more informative training signal than a one-hot label. Additionally, in born-again networks (BAN) [29] , the teacher and student have identical neural architecture and model sizes, but the student can surprisingly surpass the teacher's accuracy.In a cooperative environment, agents can learn emergent languages for communication to solve specific tasks. The emergence of such communication protocols is extensively studied in the context of multi-agent referential games [26, 55] . In these games, one agent is required to describe its observations to another agent, which is then tasked with deducing the initial agent's observations [53] .The majority of methods employed to learn discrete communication protocols between agents utilize RL [26, 99] . Compositionality is a desirable feature in the language used by agents, as it facilitates flawless generalization for previously unseen combinations of attributes [5, 11, 17, 48] . However, the community still lacks strong research indicating what general conditions are necessary or sufficient for compositional language emergence. Chaabouni et al. [11, 12] , Galke et al. [30] , Li and Bowling [56] postulate that compositional languages are more straightforward to learn.The concept of Learning from Teaching originates in cognitive psychology and linguistics, particularly within the iterated learning theory of language emergence [45] [46] [47] 86] . This theory posits that the generalizable nature of languages arises from the iterative learning process across generations in a society. The core hypothesis is that a generalizable language is inherently easier to teach and learn [56, 78, 98] , which aligns with our main hypothesis.In the AI community, recent research has aimed to employ iterated learning to enhance the generalization of emergent languages and language acquisition among artificial learners. For example, some studies have used iterated learning to improve the generalization of emergent languages between AI agents [56, 78] , while others have applied it to address generalization challenges in tasks like compositional Visual Question Answering (VQA) [98] . LOT shares the same motivation as this line of research. Our primary contribution extends the concept of "ease-of-teaching" [56] from language learning to a broader range of machine learning tasks, including supervised, unsupervised learning, and reinforcement learning.LOT functions as a regularizer, similar to other commonly used regularizers like the L2 regularizer. The L2 regularizer is effective because it encourages neural networks to learn simpler correlations, thereby avoiding overfitting. It is widely accepted that correlations with lower Kolmogorov complexity are more generalizable if they can perfectly explain a complex dataset. This aligns with the idea that "generalization equals optimal compression," as discussed by Ilya Sutskever [89] . Essentially, this notion adapts Occam's Razor to the field of AI. Our key insight is that the "ease-of-teaching" metric serves as an effective regularizer beyond language emergence tasks.Consider an intuitive example: Student A learns math by rote memorization, while Student B understands the core concepts and only memorizes essential rules, deducing the rest when needed.Both approaches can perform similarly on simple problem sets. However, as data complexity increases, Student A's burden grows significantly, while Student B's understanding-based approach remains manageable. Consequently, Student B's knowledge is easier to teach to another student, as it involves less complexity. Therefore, teachability (or imitability) can serve as a proxy for complexity.Atari Games. We perform experiments on four Atari games, namely Beam-Rider, Breakout, UpNDown, and Gravitar, following the implementation outlined in [42] . We set the regularization coefficient α to 0.5 for BeamRider, Breakout, and UpNDown, and to 0.1 for Gravitar. The other hyperparameters remain consistent across all four games. We use N of 5. For all agents, the optimizer employed is Adam, with an initial learning rate of 0.00025. The teacher agent is trained for a total of 20,000,000 timesteps. The temperature used in the KL loss is set to 1. The experiments are implemented on the NVIDIA A6000 48GB GPUs.Language Modeling. In the training-from-scratch experiments, we use the Transformer-XL architecture following Dai et al. [22] , the LSTM architecture following Zaremba et al. [106] , and the AWD-LSTM architecture following Merity et al. [66] . For supervised fine-tuning experiments with LLaMA-1 and LLaMA-2, we employ the hyperparameters described in Yue et al. [105] and use the HuggingFace Transformers library [101] . The hyperparameters for LOT are detailed in From our extensive results shown in Section 3, LOT proves to be widely applicable across various domains, including reinforcement learning (Section 3.2), unsupervised learning (Section 3.3), and supervised learning (Section 3.4). It can be effectively applied to different architectures such as CNN-based (Table 3 ), LSTM-based (Table 1 ), and Transformer-based(Table 1 ) models. LOT works well on both small datasets like PTB (Table 1 ) and CIFAR-100 (Table 3 ), and large datasets such as WikiText-103 (Table 1 ) and ImageNet (Table 3 ). It is also suitable for both small models like ResNets (Table 3 ) and large models like ViT (Table 3 ) and LLaMA (Table 2 ). Additionally, LOT is compatible with existing regularization methods such as weight decay and dropout. In our experiments with ResNets, weight decay was applied to both LOT and Teacher-only setups. In the experiments with Transformer-XL, ViT, and Swin, dropout is applied to both LOT and Teacher-only setups.A potential limitation of LOT lies in the additional computational and memory costs required for training the student models. However, as demonstrated in Section 3.5, LOT achieves better generalization with fewer training steps compared to Teacher-only models, and the flexibility in choosing student models can accommodate varying resource constraints. In RL, the additional computational costs introduced by LOT are negligible, as sample collection is more resource-intensive than fitting the agent networks to the samples, as discussed in Section 3.5. Moreover, in real-world settings, inference cost is more critical than training cost. The superior generalization achieved by LOT offers significant benefits during inference without introducing additional inference costs.The LOT algorithm for Proximal Policy Optimization (PPO) is illustrated in Algorithm 2. In our experiments, the teacher's sampled data B t is continuously added to the student sample collections D s . Meanwhile, the most recent samples from D s are used to formulate the student training batch B s to ensure a high quality of its training dataset.Computational Efficiency. To further demonstrate the computational efficiency and superiority of LOT , we conduct experiments using LSTM on PTB and ViT-B/16 on CIFAR-100 with varying training epochs and steps, while keeping other configurations the same as in Section 3.3 and Section 3.4. The results presented in Table 7 demonstrate that given equivalent computational budgets, LOT consistently outperforms the Teacher-only model across various datasets and architectures, even when the Teacher-only model trains for twice the number of epochs and steps. This further highlights ComputeEQUATIONCompute Lt (θ) using the PPO loss on minibatch B t 9:Update θ using gradient ∇ θ Lt (θ)10:Fit value network for PPO on minibatch B t 11:for i = 1 to N do 12:Sample B s ⊂ D s 13:ComputeEQUATIONUpdate student networks' parameters ϕ using loss gradient ∇ ϕ Ls (ϕ) LOT 's effectiveness in improving the teacher model's generalization while maintaining enhanced computational efficiency. Additional Evidence for Hypothesis. We provide additional experimental results to validate our hypothesis using ResNet-50 and ResNet-18 as both the teacher and student models on CIFAR-100, following the same methodology described in Section 3.1, but with different model architectures.The training and test KL-divergence of the sophisticated and deceptive students are shown in Figure 5 . We observe that the sophisticated students achieve lower final KL losses compared to the deceptive students with fewer training epochs, which further supports our hypothesis Out-of-distribution Performance. We conduct additional experiments by fine-tuning models on ImageNet-1K and evaluating them on ImageNet-R and ImageNet-Sketch using ViT-B/16 and ViT-L/16 models to investigate the out-of-distribution robustness of LOT . The results, shown in Table 8 , demonstrate that LOT also brings performance improvements on these datasets, indicating the robustness of LOT across a broader set of scenarios.Additional Comparison to KD. In Table 4 , we show that LOT outperforms the distillation method BAN. To provide stronger validation of LOT 's effectiveness, we conduct additional experiments using ResNet-50 and ViT-B/16 on CIFAR-100. We compare LOT to distillation methods such as BAN, DKD [108] , and ReviewKD [15] , with the teacher weights in these methods being the best checkpoint of Teacher-only. The results, shown in Table 9 , indicate that LOT achieves better performance than these distillation baselines, further underscoring the effectiveness of the unique interactive learning process of LOT .Table 8 : Performance of LOT and Teacher-only on ImageNet-R and ImageNet-Sketch.Ablation of Metrics in LOT Regularizer. We conduct experiments with different metrics for the "imitability" measurement, such as L2 loss. However, we find that using KL-divergence achieves better performance compared to L2 loss. The results of utilizing L2 loss for the LOT regularizer with ViT-B/16 and ViT-L/16 on CIFAR-100 are presented in Table 13 . These results show that using L2 loss for the LOT regularizer also brings performance improvements, further indicating the effectiveness of LOT regularization. Justification: The potential limitation of LOT lies in the additional computational and memory costs required for training the student models. However, we demonstrate that LOT achieves better generalization with fewer training steps compared to Teacher-only models and the flexibility in choosing student models can accommodate varying resource constraints. We provide discussions in Section 3.5 and Appendix F. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.For convenience, S is referred to as a single student learner henceforth.https://pytorch.org/vision/stable/models.htmlhttps://github.com/google-research/vision_transformerhttps://github.com/microsoft/Swin-Transformer 


Decision: This paper introduces LOT (Learning from Teaching), a regularization technique that uses auxiliary student learners to help the main model capture more generalizable correlations. The key hypothesis is that generalizable correlations are easier to imitate, and LOT leverages this by employing auxiliary learners to enhance the main model's generalization.

All reviewers found the proposed approach effective, supported by strong experimental results. However, the primary concern was the **validity of the hypothesis** on which this method is based. During the response period, the authors provided additional experimental evidence to validate their hypothesis, which satisfactorily addressed the reviewers' concerns.

Given the effectiveness of the approach and the comprehensive response, the paper is considered above the acceptance threshold. For the final version, it is suggested that the authors de-emphasize the hypothesis and its motivations from Cognitive Science or Linguistics. Instead, the focus should be on the demonstrated superiority of the proposed method itself, as highlighted in the ablation studies provided during the response period.

Reviews: review: paper_summary: paper_summary: The paper introduces Learning from Teaching (LOT), a regularization technique to enhance the generalization capabilities of deep neural networks. LOT uses separate student models trained by inmate the prediction of the teacher model to provide feedback, promoting the capture of generalizable and imitable correlations. The effectiveness of LOT is demonstrated through significant performance improvements in tasks across Computer Vision, Natural Language Processing, and Reinforcement Learning, showcasing better generalization with fewer training steps compared to traditional methods.
strength_weakness: strength_weakness: 1. The paper primarily compares LOT to a baseline teacher-only model. It will be the best to Include more comparisons with other regularization methods, such as dropout, batch normalization, or other recent advances in knowledge distillation, would provide a stronger validation of LOT's relative effectiveness, since it is possible that LOT could be replaced by combination of other much similar regularization.

2. Although the paper mentions the computational efficiency of LOT, a more thorough analysis of the scalability and computational cost, especially for larger models and datasets, would be beneficial. Providing detailed benchmarks on computational resources, memory usage and time required for training with LOT compared to other methods would help clarify its practical applicability in real-world scenarios.

3. The paper uses a fixed set of student models for feedback. Exploring the impact of different types of student models with varying capacities and architectures could provide deeper insights into the robustness and versatility of LOT.

4.  While the empirical results are strong, the theoretical foundations behind why generalizable correlations are easier to imitate could be elaborated further.
questions: questions: 1. I want to ask what is the performance of those NLP model or CV model on the validation set of trained datasets? Since it is also a type of generalization. It could be used to show whether the LOT is helpful for preventing over fitting.
2. For "The total training steps for LOT and the Teacher-only approach is the same for fair comparison", my understanding is that the LOT only train half number of epochs comparing to the teacher-only approach? (If we only train with one student model)
3. The paper only reports the performance the teacher model, what is performance for those students model?
limitations: limitations: I think the paper adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes the LOT (Learning from Teaching) regularization technique, which employs auxiliary student learners to help the main model capture these more generalizable correlations. The authors hypothesize that generalizable correlations are expected to be easier to imitate, and LOT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The results suggest the effectiveness and efficiency of LOT in identifying generalizable information.
strength_weakness: strength_weakness: (i) The core argument of this article is that "generalizable correlations are expected to be easier to imitate", but previous studies have shown that more complex data and correlations contain richer information, where models need to obtain better accuracy to learn, making them more helpful for generalization. These two views seem to be contradictory. At the same time, the author constructed their hypothesis based on human cognition, but the relevant theoretical basis is lacking, which makes its reliability questionable.

(ii) This distillation is similar to meta-learning, but its support for generalization is confused. As we all know, meta-learning performs great generalization by obtaining general knowledge by distilling task-specific knowledge and then using it to complete various tasks. But in this paper, as the author said, "both the teacher and student models may lack or possess different task-specific knowledge", it is difficult to understand why it improves generalization. I think it seems to be "joint learning", but this learning mechanism and optimization details are missing. I hope the author can further provide LOT's insight.

(iii) The description of the training process in this article is a bit hard to follow. For example, the author mentioned that generalization is improved through "joint learning", but in Section 2.2, the teacher-learner and student-learner are optimized alternately. In the introduction, the author mentioned using the student learner as an auxiliary task to improve generalization, but it is not clear how to capture the generalization connection, what this generalization connection is, and how to guide the model learning. More details may be better.

In summary, my concerns focus on the credibility of the author's motivation and the soundness of the method. If these issues can be addressed, I will be happy to improve my score.
questions: questions: Please see the Weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: The authors proposed a novel regularization approach called Learning from Teaching (LoT) to enhance generalization. The hypothesis that simple correlations are generalizable is the main question for this work. Through a teacher student approach, the authors are capable of provide to the main model more generalizable and imitable
strength_weakness: strength_weakness: - I would like to see further evidences of the proposed hypothesis.
questions: questions: - The hypothesis on Section 2.1 have been proposed before? I better discussion on that will help the overall reading and to better position the paper when compared to other approaches.
- For all the experiments N (student steps ratio) was 1? If not the authors compared LoT with other methods for the same number of steps? If not the superior number of steps can partially be the cause of the performance gains. 
- Regarding the choice of metrics the authors chose KL-divergence, did the authors experimented with other losses? 
- Not sure if it is feasible but benchmark LoT on ImageNet-R [1] and ImageNet-Sketch [2] will be interesting for a broader set of results.

[1] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In CVPR pp. 8340–8349, 2021. 

[2] Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning ro- bust global representations by penalizing local predictive power. In NeurIPS, pp. 10506–10518, 2019.
limitations: limitations: - The current set of baselines can be expanded. Methods like [3] and [4] can provide a better understanding on how well LoT is positioned. 

[3] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021 

[4]  Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, Jiajun Liang, Decoupled Knowledge Distillation, CVPR, 2022.

Comments: title: Thank you for the feedback and for raising our score!
Thank you for the feedback and for raising our score!
Thank you for the feedback and for raising our score!
Official Comment by Reviewer NPF4
comment: We sincerely thank the reviewer for the feedback and the score adjustment. We appreciate the reviewer's continued interest in our paper. We are committed to incorporating the new results and discussions in the revised version to further enhance the quality and contribution of our work.
We sincerely thank the reviewer for the feedback and the score adjustment. We are pleased to hear that our response satisfactorily addressed the reviewer’s question. In the revised version of our paper, we will incorporate the new results and discussions to further enhance the quality and impact of our work.
I think the authors well addressed my question. I will increase my score from 5 to 6.
I appreciate the authors' rebuttal. I raised my score from 5 to 6 in response to the authors' rebuttal. I will also discuss with the other fellow reviewers and/or AC.
We thank the reviewer for the thoughtful feedback and score adjustment! We are pleased to know that some of your confusion has been resolved.

In response to your concern about the reliability of our algorithm's insights, we have provided evidence in Figure 1 of our paper to illustrate that "Generalizable correlations are more easily imitable by learners compared to spurious correlations." Additional supporting evidence is detailed in Figure 5 of the PDF in the General Response and in the discussion of Q1 in the rebuttal to Reviewer DocJ. Our hypothesis extends the concept of "ease-of-teaching" [24] from language learning to a broader range of machine learning tasks and demonstrates effectiveness in several tasks and setups.

The evidence for "ease-of-teaching" in language emergence is further supported by [24], which empirically shows that "compositional language is easier to teach than a less structured language." Additionally, [25] indicates that "more compositional languages are easier to learn for new agents, including those that are architecturally different from the ones that evolved the language." We recommend referring to these works for more evidence regarding the connections between teachability (imitability) and generalization.

We acknowledge that developing a comprehensive theoretical foundation for this hypothesis is indeed a challenging task, particularly in real-world contexts, and may be beyond the scope of the current paper. However, we recognize the importance of this aspect and are committed to exploring it in future work.

We will include these discussions and analyses to support our hypothesis in the revised version. We genuinely appreciate your constructive suggestions and are committed to further improving the quality and contribution of our work.

[24] Ease-of-Teaching and Language Structure from Emergent Communication, NeurIPS 2019  
[25] Compositionality and Generalization in Emergent Languages, ACL 2020
Thanks to the author for the feedback, which solved some of my confusion, but the reliability of the algorithm's insights is still not guaranteed since it basically relies on a strong Hypothesis. It would be better if a theoretical analysis could be provided. Despite this, I am still willing to revise my score from 4->5, and hope that the author can further polish and improve this work.