Transferring Inductive Biases through Knowledge Distillation
Authors: Samira Abnar
Mostafa Dehghani
Willem H. Zuidema
Venue:  (2021)
Keywords: Knowledge Distillation
Inductive Biases
Analyzing and Understanding Neural Networks
Recurrent Inductive Bias

Abstract: Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing, and efficiently adapting inductive biases is not necessarily straightforward. Inductive biases of a model affect its generalisation behaviour and influence the solution it converges to from different aspects. In this paper, we investigate the power of knowledge distillation in transferring the effects of inductive biases of a teacher model to a student model, when they have different architectures.
We consider different families of models: LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios with linguistics and vision applications, where having the right inductive biases is critical. We train our models in different setups: no knowledge distillation, self-distillation, and distillation using a teacher with a better inductive bias for the task at hand. We show that in the later setup, compared to no distillation and self-distillation, we can not only improve the performance of the students, but also the solutions they converge become similar to their teachers with respect to a wide range of properties, including different task-specific performance metrics, per sample behaviour of the models, representational similarity and how the representational space of the models evolve during training, performance on out-of-distribution datasets, confidence calibration, and finally whether the converged solutions fall within the same basins of attractions.

Under review as a conference paper at ICLR 2021
TRANSFERRING INDUCTIVE BIASES THROUGH KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are limiting factor, or where training data not perfectly representative of the conditions at test time. However, defining, designing, and efficiently adapting inductive biases not necessarily straightforward. Induc- tive biases of model affect its generalization behavior and influence the solution it converges to from different aspects. In this paper, we investigate the power of knowledge distillation in transferring the effects of inductive biases of teacher model student model, when they have different architectures. We consider different families of models: LSTMs Transformers and CNNs MLPs, in the context tasks and scenarios with linguistics and vision applications, where having the right inductive biases is critical. We train our models in different setups: no knowledge distillation, self-distillation, and distillation using a teacher with a better inductive bias for the task hand. We show that in the later setup, compared to no distillation and self-distillation, we can not only improve the performance of the students, but also the solutions they converge become similar their teachers with respect to wide range of properties, including different task-speciff perfor- mance metries, per sample behavior of the models, representational similarity and how the representational space of the models evolve during training. performance on -iiitiitiitiiittononn datasets, confidence calibration, and finally whether the converged solutions fall within the same basins of attractions.
INTRODUCTION
Inductive biases are the characteristics of learning algorithms that influence their generalization behavior, independent of data. They are one of the main driving forces to push learning algorithms toward particular solutions (Mitchel] 1980). Having the right inductive biases is especially important for obtaining high performance when data or compute a limiting factor, or when training data not perfectly representative of the conditions at test time. Moreover, in the absence of strong inductive biases, a model can be equally attracted to several local minima on the loss surface; and the converged solution can be arbitrarily affected by random variations like the initial state the order of training examples (Sutskever et al. 2013) McCoy et al. 2020 Dodge et al. 2020
There are different ways to inject inductive biases into learning algorithms, for instance through architectural choices, the objective function, the curriculum, or the optimization regime. In this paper, we exploit the power of Knowledge Distillation (KD) to transfer the effect of inductive biases between neural networks. KD refers to the process of transferring knowledge from teacher model student model. where the logits from the teacher are used to train the student. KD is best known as effective method for model compression (Buciluă et al. 2006 Hinton et al. 2015 Sanh et al. 2019) which allows taking advantage of a huge number of parameters during training while having an efficient smaller model during inference.
The advantage of KD goes beyond model compression and it can be used to combine strengths of different learning algorithms (Kuncoro et al. 2019 2020 Different algorithms vary in terms of the computational/memory efficiency at Inferenegmfeerncce inference or the inductive biases for learning particular patterns. This makes them better at solving certain problems and worse at others, i.e., there is "one size fits all" learning algorithm. Hence, it is important to explore the potential KD for finding better trade-offs.

---

Under review as a conference paper at ICLR 2021
(a) MLP
(b) CNN
(c) CNN MLP
Figure I: Training paths of different models on the Translated MNIST task. Different points represent the state of the model at different epochs. from the initial state to the convergence. The visualization is based 2D projection of the representational similarity of the activations from the penultimate layer for the examples from the validation set, ie. Translated MNIST (more details Appendix B
The question that we ask in this paper is: "In KD, are the preferences of the teacher that are rooted in its inductive biases, also reffected in its dark knowledge" and can they thus be transferred to the student?". We are interested in cases where the student model can realize functions that are realizable by the teacher, i.e., the student model is efficient with respect to the teacher model (Cohen cct al. 2016), while the teacher has a preference inductive bias so that the desired solutions are easily learnable for the teacher (Seung et al. 1991).
We consider two scenarios where the teacher and the student are neural networks with heterogeneous architectures, hence have different inductive biases. We train the models, both independently and using KD, on tasks for which having the right inductive biases crucial. In the first test case, we study RNNs vs. Transformers (Vaswani et al., 2017), the subject-vett agreement prediction task (Linzen et al., (016)) In this task, we use LST TSMM (Hochreiter & Schmidhuber) 1997) as the most widely used RNN variant. L.TTMs are shown to perform better than vanilla Transformers in this task and their superior performance is attributed to their so-called "recurrent" inductive bias (Tran et al., 2018). First, we identify the sources of the recurrent inductive bias of LSTMs: sequentiality, memory bottleneck, and recursion, and design experiments to show the benefits of each. Then, we show that through distilling knowledge of LSTMs to Transformers, the solutions that the Transformer models learn become more similar to the solution learned by LSTMs.
In the second test case, we study CNNs vs. MLPs, the context of the MNIST-C (Corrupted MNIST) benchmark (Mu & Gilmer 2019), which is designed to measure -ut----distributiio robustness of models. We train our models on MNIST and evaluate them on the Translated/Scaled MNIST. The particular form of parameter sharing in CNNs combined with the pooling mechanism makes them equivariant to these kinds of transformations (Goodfellww et al. 2016), which leads to better generalization in these scenarios compared to MLPs.
In our experiments and analysis on these two test case from a wide range of perspectives, when trained in different setups including (1) when trained without KD, but directly from the data, (2) when trained with KD using a teacher with a similar architecture to the student, i.e. self-distillation. and (3) when trained with KD using a teacher with different architecture that has stronger inductive biases that suit the task, compared to the student.
M we compare the behavior different models,
As the first step, in setup (1), i.e., no KD, we demonstrate how inductive biases arising from different architectural choices affect the generalization behavior of the models we study ($2. and $3.1). We show that the models with more suitable inductive biases not only have better accuracy but also the solutions they converge to is a better solution in terms of other metrics. We also show that different instances of the model with stronger inductive biases have less variance terms of all the metrics.
Then, we apply KD to train the models and contrast the behavior of models trained with the setups (2) and (3) with the models trained with setup (1), i.e. with KD vs. without KD. We show that regardless of the properties of the teacher, KD is powerful technique in which the teacher model drives the student toward a particular set of solutions that is more restricted compared to the set of possible solutions that a student can converge to when it learns directly from data ($
and
Appendix
Dark knowledge refers to the information encoded in the output logits of neural network Hinton et al. 2015 'The codes the input pipelines, models, analysis, and the details of the hyper- hyper- parameters used our experiments available at wtttttss//// /ttttss///NNNMMTRRR
2

---

Under review as a conference paper at ICLR 2021
Next, as the main contribution of our paper over previous works that study KD, we contrast the behavior of models trained with setup (3) with the models trained with setups (1) and (2):
We show the performance of the student models in setup (3) increases, not only on in- distribution test sets ($2.2), but also on out-of-distribution data ($3.2). We demonstrate that this happens when the teacher has the right inductive bias and not necessarily otherwise, i... setup (2).
In setup (3), besides performance, show that, the solution that a student model converges to shares similar characteristics with the solution of its teacher. For instance in terms of confidence calibration 2.2 and 3.2). and per sample behaviour of the model (Appendix [E)
We demonstrate that although the student model is merely exposed to the final logits of the teacher, the structure of the latent space of the student model becomes similar to the teacher, i.e. relational similarity of the internal representations from the student and its teacher increases (52.2 and (3.).
As an example, in our second test case (MNIST-C), when training an MLP model with KD using a CNN teacher. the student model explores the solution space in ways more similar to its teacher. Figure [ visualizes and compares the path that an MLP takes during training (Figure La], compared to CNN (Figure Tbp. The CNN model explores the surface in completely different manner than the MIP, while the path of a student MLP distilled from the CNN model as the teacher (FigureTc) is more similar to the CNN.
DISTILLING LSTMS INTO TRANSFORMERS
LSTMs and Transformers are the basic building blocks of many state-of-the-art models for sequence modeling and natural language processing. Transformers are an expressive class of models that do extremely well on many tasks where the training data is adequate in quantity (Devlin et al. 2019 Keskar et al., 2019; Radford et al., 2019 Brown et al. 2020). Several studies, however, have shown that LSTMs can perform better than Transformers on tasks requiring sensitivity to (linguistic) structure, especially when the data is limited (Tran al. 2018 Dehghani et al.) 2019).
We chose the subject-verb agreement prediction task, introduced by Linzen al. (016), as the test case, as it yields a meaningful difference between LSTMs and Transformers (Tran et al. 2018). We compare these two families of models and conduct experiments emphasize the differences between them when trained independently and through KD.
Recurrent Inductive Bias. Among sequence modeling architectures, models with recursion are in particular powerful for natural language processing due to their adequacy to model hierarchical structures (Linzen et al. 2016). The recursion in model can be implemented in various ways, like in Recurrent Neural Networks (Elman 1990). Recursive Neural Networks (Socher et al. 2010 Le & Zuidema 2014) and Universal Transformers (Dehghani et al. 2019 Hao et al. 2019). While theo- retically, both recurrent neural networks (RNNs) and Transformers can deal with finite hierarchical structures, empirical results indicate the superiority of RNNs over Transformers (Tran et al. 2018
Dehghani et al.] 2019 Hahn 2020).
In the literature (Sutskever et al. [013 Dehghani et al. 2019). the inductive bias of RNNs is referred to as the recurrent inductive bias. Here, we distinguish between three main sources of this bias:
1. Sequentiality: There is an inherent notion of order in the architecture that forces the model to access the next tokens in the input one by one and process them sequentially.
2. Memory bottleneck: The model has no direct access to the past tokens and has to compress all the information from the past in a hidden state, which is accessible when processing a new token.
3. Recursion: The model recursively applies the same function on the varying input at every step.
Transformers (aaswani et al. 2017). ). in contrast, process the input in parallel. Although weak notion of order is encoded by positional embeddings, no explicit assumption is made in the connectivity structure of the architecture. Moreover, they have a global receptive field and can access all tokens through self-attention. Finally, standard Transformers are not recursive. However, the standard Transformer can be modified to have an architecture with specifications that are similar to RNNs. We provide empirical results to demonstrate the benefits of these different sources of inductive biases of RNNs. For this purpose, we design experiments with variants of Transformers in which we attempt to approximate some of the RNNs' assumptions.
3

---

Under review as a conference paper at ICLR 2021
Table Performance (mean:tstd over 4 trials) of different LSTM and Transformer models trained independently with the LM objective.
Model
Perplexity 4
D D-Aceuracy 1
A-Accuracy 1
Transformer
$7.50 1 0.1199
0,9417 + 0.0017
0.9191 0.0018
Small Transformer
55.31 + 0,0847
0.9467 + 0.0012
0.9261 + 0.0020
LSTM
56.68 + 0.0906
0.9510 + 0.0012
0.9400 + 0.0024
Small LSTM
58.05 = 0.1141
0.9491 = 0.0006
0.9366 = 0.0015
Table 2:
Performance (me over 4 trials) of different LSTM and Transformer models trained independently with the classification objective.
Model
Fr Accuracyyyy t
D Accuracy
A A-Aceuracy
Transformer
0,954 + 0.0016
0.901 = 0.0037
0.717 + 0,0244
Transformer-seg
0.964 + 0.0010
0.909 0.0037
0.742 + 0.0121
Universal rriivrrsa/ranssfommrrsee
0.969 + 0.0004
0.932 0.0055
0.806 0.0153
LSTM
0.977 1 0.0001
0.970 0.0003
0.928 0.0007
Task and Models. We study the performance of LSTMs and variants of Transformers on the task of predicting camber--ggremeer between subjects and verbs in English sentences. We investigate the quality of the solutions they converge to when they are trained independently and when they are trained through distillation. We use the subject-verb agreement dataset of Linzen al. (2016), for which the size of the training set is 121k examples and the size of the test set is ~m. Succeeding at this task is strong indicator that model can learn syntactic structures and is therefore proposed by Linzen et al. 62016) as proxy for assessing the ability of models to capture hierarchical structure in natural language. It is shown that RNNs have better inductive biases to learn this compared to standard Transformers (Tran et al.) 2018 Dehghami et al. 2019). In this task, examples are grouped into different levels of difficulty based on the number of "agreement attractors' and distance between the verb and its subject. Hence, we report both micro accuracy (І- (ucurrcryyy) and macro accuracy over different groups in terms of distance ID-Aceuracy and numbers of attractors (A-An AAAAcuurccy..
Similar to Tran et al. (2018), we follow two setups: 1) when the learning objective is next word prediction, i.e., language modeling (LM); 2) when we directly optimize for predicting the verb number, singular or plural, i.e., classification. In the LM setup, we look at the probabilities predicted when the target of the prediction the verb of interest, and see whether the probability of the correct form of the verb is higher than the other form (singular plural). In the classification setup. the input to the model is a sentence up to the position of the verb of interest and the model predicts whether the verb at that position is singular or plural.
In the LM setup, we employ two unidirectional LSTMs with different sizes, LSTM and Small LSTM, and two Transformers, Transformer and Small Transformer. In this setup, corresponding LSTMs and Transformers have roughly the same number of parameters. In the classification setup we compare the following models: (1) standard unidirectional LSTM (sequentiality + memory bottleneck + recursion) (2) Transformer: Transformer encoder with class token (CLS) for classi- fication, BERT (Devlin et al. (019)) style. (3) Transformer-ser Transformer encoder with future masking where the classification is done using the representation of the last toker H (sequentiality) (4)
seq: Universal Transformer (Dehghani et al.) 2019) encoder, in which the parameters are shared in depth, with future masking (sequentiality + recursion). Appendix provides more details on the architectures.
2.1 THE IMPORTANCE OF RECURRENT INDUCTIVE BIAS
In this section, we report results without distillation that illustrate the merits of the recurrent inductive bias. Table [] shows the performance of the models when trained with the LM objective. A first important observation, in line with the results of Tran et al. (2018). is that LSTMs achieve better accuracy on the subject-verb agreement task compared to Transformers. Even for instances of Transformer language models that achieve better (lower) perplexity, the accuracy on this task is worse compared to LSTM instances.
Agreement attractors intervening nouns with different number than the number of the subject, E.g., given the inpur
"The keys to the cabinet (is?/are) /are?)) "" the word "cabinet" is agreement attractor.
Note that future tokens masked out by default when using a transformer in the decoder mode, e.g., LM setup.
4

---

Under review as a conference paper at ICLR 2021
Teacher:
Data (No distillation)
LSTM
UTransformer-sog
Teansformer-scg
Transfomerr
0.98
0.06
0.05
Accurcy
0.97
0.04
0.96
0.03
Trandiemet
Transfonme--cc
UTanntormen-cc
LSTM
Transtornee
Trensiermer-sc
UTunatemmer--c
LSTM
(a) Accuracy t
(b) Expected Calibration Error +
Figure 3: Performance std over 4 trials) of models with different inductive biases trained independently using KD with different teachers.
Small LSTM
ESTM
Since both models achieve good scores on the training set (Appendix D), this suggests that LSTMs better capture relevant patterns, such as the hierarchical structure of the input, which leads to better generalization on this task.
A- A-curacy
0.93
4
Transformor
Figure 2 iillustrates the accuracy versus perplex- ity of several instances of each model, in the LM setup. Note that although perplexity is an indi- cator of how well the model is optimized given the objective function, the accuracy is the metric that matters and shows models' generalization in the subject-verb agreement task. Later, we will show how using KD the behavior of Trans- formers changes in terms of accuracy versus perplexity and become more similar to LSTM teachers.
Transformer
59
57
56
Perplexity
Figure 2: A-Accuracy vs perplexity (high to low from left to right) for language models of different architec- tures and sizes.
There is another interesting observation in Figure 2 In this plot, for each model, we have two different settings: large and small variants, as measured by the number of trainable parameters. More parameters for model, given fixed architecture, means richer hypothesis spaces. We can see that while for the LSTMs, increasing the size of the model results better performance, for the Transformers increasing the number of parameters results in worse performance. This aligns with the vlas-nariance trade-off argument that when using a model with weaker biases for the task at hand, in this case Transformers. if we fix the amount of data, richer hypothesis spaces may hurt the generalization because they increase variance. In contrast, adding more capacity leads to better accuracy in LSTMs as their stronger inductive biases control the generalization error.
In Table we show the results models trained on the classification objective. We compare LSTM with variants of Transformers with different inductive biases. The table shows that similar to the LM results, LSTM achieves the best performance. Interestingly, comparing all four models, we find that the performance steadily increases as more aspects of the recurrent inductive bias are included. This is illustrated in Figure Ba with the filled circles on the black, dashed line (no distillation).
As another indicator of the quality of the solutions that different models converged in the classifica- tion setup, we look into their confidence calibration. Confidence calibration captures how well the likelihood (confidence) of the prediction of the model predicts its accuracy (Guo et al. 2017). For well-calibrated model, if we bin the confidence scores and compute the accuracy for each bin, the accuracies are perfectly correlated with the confidence values. The Expected Calibration Error (ECE) is computed the distance between the calibration curve of the model and the perfect calibration curve (DeGroot& & Fienberg 1983). In Figure 3b we plot the ECE (Guo et al. 2017) of the models in the classification setup, with the filled circles on the black dashed line (no distillation). In line with the trends in the performances these models, the expected calibration error decreases as we move from Transformer toward LSTM.
5

---

Under review as a conference paper at ICLR 2021
Table 3: Performance (meanisst over trials) of different LSTM and Transformer models with LM objective when we apply pure distillation with r
Teacher Model
Student Model
LSTM
Small L.TM
Transformer
Small Transformer
Accuracy T
0,9275 0.0017
0.9310 = 0.0013
0.9083
0.0044
0.9257 0.0027
LSTM
perplexity
59.45 + 0.0191
60.92 + 0.0185
60.01 + 0.0328
58.65 0.0036
Accuracy f
0,9224 + 0.0024
0.9272 + 0.0026
0.8985 0.0057
0,9157 + 0.0020
Small LSTM
perplexity
62.52 1 0.1071
63.44 1: 0.0272
63.45 1 0.0644
61.62 - 0.0619
Accuracy
0.92% 0.0029
0,9316 + 0.0012
0.8956 0.0018
0,9195 0.0015
Transformer
perplexity
57.03 = 0.0092
59.09 0.0126
57.67 0.0091
56.64 0.0352
Accuracy
0.9201 0.0018
0.9233 1 0.0011
0.8827 0.0027
0,9131 0.0014
Small Transformer
perplexity
57.84= 0.0269
59.73 + 0.0166
58.44 + 0.0354
57.16 0.0087
Table 4: 14- Accuracyy t (mean-Lstd
over 4 trials) of different LSTM and Transformer models with classification objective when apply pure distillation with
Teacher Model
Student Model
Transformer
Transformer-seq
UTransformer--sq
LSTM
Transformer
0.9555 1 0.0013
0.9556
0.0006
0,9571 + 0.0027
0.9596 1 0.0008
Transformer-seq
0.9599 1 0.0006
0,9629 + 0.0008
0,9679 1 0.0005
0.9720 1 0.0017
U'raansformer-seq
0.961 + 0.0006
0.9635
0.0004
0.9688 0.0008
0.9748 + 0.0003
LSTM
0.9682 + 0.0002
0.9690 0.0011
0.9741 + 0.0004
0.9759 + 0.0001
In summary, the results from this section support the importance of recurrence for solving this task (Tran et al., [2018) Dehghani et al. (219)). Additionally, as shown in Figures BaarreesBBaaand [ajjand[3]] we find a decreasing trend the variance of the models, i.e., adding more inductive biases to the models decreases their variance. This is empirical evidence that supports the relation between variance of the solutions a model converges to and its inductive biases.
2.2 TRANSFERRING THE EFFECT OF RECURRENT INDUCTIVE BIAS
In this section, we show distilling knowledge from LSTM to Transformer can close the gap between their performance by pushing the Trans- former to converge to solutions more similar to LSTM's.
Small LSTM
L.STM
tescheri))
1eauhern2
0.94
Accaracy
0.93
Transfrnerr
Table Table summarize the distillation results, when the training objective is language modeling and classification respectively. A first general observation is that, for these tasks and se- tups, distilling a model into an identical model could result in decrease in the performance. Note that whether self-distillation results in im- proved performance could potentially depend many different factors such as the architecture of the model, optimization algorithm, and de- tails of the distillation process (Furlancllo et al.) 2018 Mobahi al. 2020). Despite no significant changes in the performance with self-distillation, we can improve the performance of the Transformers through distillation from LSTM teachers.
Trandommer snidencki
Transforme
39
58
07
56
Perplexity 1
Figure 4: AA Accuracy 1 vs perplexity 4 (high to low from left right) for student Transformer with LM ob- jective. In this figure the triangle marks indicate Trans- former models, and cireles indicate the LSTM teachers used to train the student Transformers.
To check whether this improvement due the transfer of the effect of inductive biases through distillation and whether distillation helps students to converge to solutions similar to their teachers, we run series of analyses. In Figure we see how teacher LSTMs pull student Transformers toward solutions with higher accuracy on the subject-verb agreement task in the LM setup. This happens when the perplexity of the student Transformer is higher (worse) than the independent Transformer.
Figure 3 also shows the effects of distillation each of the four models we study in the classification setup. In Transformer- based models, we get the most significant improvement both in accuracy and ECE when the teacher is LSTM. As the recurrent inductive biases of the teacher get weaker, the amount of improvement in the performance student models decreases. Figure[]show shows the effect of KD the calibration, given student Transformer and an LSTM teacher.

---

Under review as a conference paper at ICLR 2021
Perfect calibration
Perfect culibration
Perfect calibration
LSLM
transdomer
LSTM TTansfoomer
accuracy
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
confidence
confidence
confidence
Figure 5: Calibration plots for independent and distilled Transformer for the classification setup. Note that since the task is binary classification, accuracy for confidences lower than 0.5 is not defined.
Transfomerr
(Transfeemm--eo
Transfommer
LSTM Transformer
LSTM Tranformerr
Small LSTM Small Transfommer
LSTM
LSTM U'ranstormer sca
Small LSTM Transforme
o?
Small Transtommer
Transformer-ss sey
LSTM
LSTM Imansfornee
(a) Language Modelling
(b) Classification
Figure 6: 2D projection of representational similarity of the activations from the penultimate layers for 1000 examples from the validation set (check Appendix B for more details). We use the notation of - b to refer to the student model distilled from teacher model
Is the improvement in calibration merely the product using soft targets? Mueller et al. (2019) shows training neural networks with soft targets (e.g. through label smoothing) results in models that are better calibrated. On the other hand, KD has a regularization effect similar to label smoothing (Yuan al. 2019 Tang et al. 2020). Given the lack of significant improvement in ECE in the self-distillation experiments (Figure 3b), it more likely that the cause of the improvement in ECE when distilling LSTMs into Transformers is beyond the label smoothing effect of KD..
To further explore and better understand the effects of KD, we compare the internal representations of these models besides their final output. FigureloJshows the 2D projection of the relational similarity of representations 5 (Laakso & Cottrell 2000) from the penultimate layer of the models (see Appendix B for details). We see that, in the LM setup, the internal representations of student Transformers that are distilled from LSTMs are structured differently compared to independent Transformers and are more similar to the L.ST models. For the classification objective, we also see that the distilled models are further away from their independent versions. This supports the idea that the effect of distillation goes beyond the output of the models and their final performances.
3 DISTILLING CNNS INTO MLPs
To evaluate the robustness of our findings on the transfer of inductive biases through KD, we performed a second case study, using different neural architectures and a different task. We use convolutional neural networks (CNN) vs. multilayer perceptrons (MLP) as two families of models with different inductive biases. CNNs topology. Sparse connectivity and parameter sharing in CNNs make them an effective and statistically efficient architecture. The particular form of parameter sharing in the convolution operation makes CNNs equivariant to translation (Goodfellow et al., 2016). Note that, we can view CNNs as MLPs with an infinitely strong prior over their weights, which says that first of all the weights for each hidden unit are identical to the weights of its neighbor with a shift in space, second, the weights out of the spatially continues receptive field assigned each hidden unit are zero
the de facto choice for processing data with grid-like like
Task and Models. We study CNNs and MLPs in the context of the CorruptedMNNIS dataset (MNIST-C) (Mu & Gilmer 2019), which aims at benchmarking out-o--disti robustness. We
Note that the relational similarity captures the similarity of structures, the absolute values.
7

---

Under review as a conference paper at ICLR 2021
Table 5: Accuracy and Expected Calibration Error (mean=std over trials) of CNN and MLP trained indepen- dently on MNIST and evaluated on MNIST, AINIST-Scaled and -rAIS--Taanslted..
(a) Accuracy
(b) Expected Calibration Error
Model
MNIST
Scaled
Translated
Model
MNIST
Scaled
Translated
CNN
0.992 + 0.0009
0,962
0.0021
0
0.0003
CNN
0.011 + 0.0006
0.060 + 0.0044
0.028 + 0,0016
MLP
0.985 = 0.0011
0.794 = 0.0154
0.373 + 0.0151
MLP
0.015 + 0.0006
0.175 = 0.0081
0.564 + 0.0091
Table
Accuracy and Expected Calibration Error (mean std over trials) of CNN and MI. trained with pure distillation with r = 5, on MNIST and evaluated on MNIST, MNIST and MNIST-Translated.
(a) Accuracy
Student Model
MNIST
Scaled
Translated
CNN
CNN
CNN
CNN
0 + 0.0004
0,990 + 0.0007
0,951 0.0046 0.955 + 0.0065
0.978
0.0003
0.976 0.0012
MLP
0.988 + 0.0005
0,985 + 0.0015
0.904 + 0.0073
0.839 + 0.0096
0,510 + 0.0148
0.395 + 0.0069
(h) Expected Calibration Error
MNIST
Scaled
Trunslated
Student Model
CNN
MLP
CNN
MLP
CNN
MLP
CNN
0.014 + 0.0004
0.013 + 0.0005
0.068 + 0.0043
0.054 + 0.0063
0.033
0.0006
0.030 + 0.0016
MLP
0.013 + 0.0004
0.015 + 0:0012
0.109 0.0053
0.155 + 0.0079
0.432 0.0136
0.555 0.0038
train the models on the original MNIST training set and evaluate them on the Translated and Scaled MNIST test sets from MNIST-C. In this scenario, the inductive biases of CNNs help them generalize better than MLPs. Our CNN architecture a stack of convolutions and pooling layers. Combining convolution and pooling over spatial regions results in invariance to translation. To have CNNs that can learn to be invariant to other transformations like changes in the scale, we can use cross-channel pooling (Goodfellow et al. 2013). where we pool over separately parametrized convolutions that have learned to detect different transformed versions of the same underlying features. Our MLP is simply stack of fully molle-connected layers. More details on the architectures are in Appendix H
3.1 THE IMPORTANCE OF TRANSLATION EQUIVARIANCE.
Table 5 presents the accuracy and ECE of CNNs and MLPs when trained independently. All models are trained on the original MNIST training set and tested on the Scaled and Translated sets from MNIST-C. Even though CNNs' accuracy and ECE on the original MNIST test set are only slightly better than MLPs (.992 vs .855, there is a rather large gap between their performances on the Scaled (.962 vs. .794) and Translated (.981 vs. .373) test sets. This is expected since the inductive biases of CNNs make them suitable for these types of generalizations. Moreover, the variance of the results from the CNNs is much less compared to MLPs. This is due to the fact that different instances of model with stronger inductive biases are more likely to converge to solutions that belong to the same basin in the loss landscape (Neyshabur et al. 2020 (See Appendix C for more analysis).
3.2 BETTER OUT OF DISTRIBUTION GENERALIZATION WITH KD.
Table 6 that distillation from CNN into MLP improves both accuracy and ECE for all three test sets, decreasing the gap for the Scaled test set (.904 vs. .794 without KD), and much more improvement on the performance on the Translated test set (.510 vs. .373 without KD). We also see a lower variance in the performance of MLP models that are trained through KD with CNN teachers.
We further compare the results of all possible pairs of models as teachers and students, take into account different effects of KD that can potentially improve the performance of the student model. Although delf-disillattinn results in a slightly better performance in MLPs, perhaps due the regularization effect of distillation (Mobahi et al. 2020; Tang et al. 2020), the improvement in the performance of MI.Ps with an MI.P teacher is much less compared to when the teacher is a CNN. Regardless of the teacher (MLP CNN), KD results in slightly lower performances in student CNNs compared to CNNs trained independently (similar to results of an LSTM student in test case 1).
Furthermore, in Figure 7 we compare the relational similarity of the representations from penultimate layers of independently trained CNNs and MLPs as well as their distilled ones. First of all, as expected based on our assumptions about the inductive biases of these models, MLPs have more variance than CNNs. Second, distilling from a CNN to an MLP results in representations that are more similar to the representations learned by CNNs, while this is not the case with MI.s as teachers
8

---

Under review as a conference paper at ICLR 2021
CNN MLP
CNN MLP
MLP
MLN
MLP -> MLP
MLP MLP
CNN
CNN
CAN 6 CNN - CNN CNV - MIP
CNN CNN - CNN FNN MPP
CNN
CNN
MLP e MLP MLP
MLP
CNN
MLP MLP
MLP MLP
CNN
MLP - CNN
VIP CNN
(a) Scaled MNIST
(b) Translated MNIST
Figure 7: 2D projection of representational similarity of the activations from the penultimate layers for all examples from the test set (check Appendix B for more details), We use the notation of a - b to refer to the student model b distilled from teacher model o.
and CNNs as students. Morcover, for both CNNs and MLPs, self-distillation does not significantly change the representations they learn.
Finally, we compare the paths the models follow during training until they converge to a solution. To plot the training path of model, we compute the pairwise representational similarity between different stages of training the model. Figure [ illustrates the training path for an independent MLP, an independent CNN, and an MLP that is distilled from CNN. While MIP and CNN seem to have very different behavior during training. the student MLP with CNN as its teacher behaves differently than independent MLP and more similar to its teacher CNN. This is interesting, in particular, since the student model only exposed to the final solution the teacher has converged to and information about the intermediate stages of training is provided in the offline KD.
CONCLUSIONS
The no free lunch theorem states: for any learning algorithm, any improvement on performance over one class of problems balanced out by decrease in the performance over another class (Wolpert & Macready 1997). Neural networks with different architectures have different inductive biases and this is reflected their performance across different tasks. In this paper, we investigate the power of KD to enable benefiting from the advantages of different models at the same time. First, we demonstrate how inductive biases arising from different architectural choices affect the generalization behavior of the models we study. We further show that when model has the right inductive bias, can transfer its knowledge to a model that lacks the needed inductive bias and indicate that solutions only quantitatively but also qualitatively reflecting the effects of
that the student model learns the inductive biases of the teacher model.
REFERENCES
Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. Blackbox meets blackbox: Representational similarity & stability analysis of neural language models and brains. In Proceed- ings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2019. URL httph::////// /wt::://wwwww.. awwwwb.www......... org/ antttnntooooyyooo /Wl9 -120-820911111010
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, CVPR'19, 19, 2019. URL httpps://axxv..or.....
abs/1904. 05835
Rohan Anil, Gabriel Pereyra, Alexandre Tachard Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. In Proceedings of the 6th International Conference on Learning Representations, ICLR' 18, 2018. URL https 77 7oopenreview net h
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14 165, 2020.

---

Under review as a conference paper at ICLR 2021
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '06, 2006. URL /Rttps:://di http://c.......oo///....111//1/0
Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: tensor analysis. In Conference Learning Theory, pp. 698 998-728,44 2016.
Mark W. Craven and Jude W. Shavlik. Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems 8, NeurIPS'95. Cam bridge, MA. USA, 1995. MIT Press. URL /sttpa://papers..oo L htwp:/s..pppccccccpppeeeecaaacee ott.:r/eeeettttttccrrrrrrrrrrrrr
ks
pdf
Mark William Craven. Extracting Comprehensible Models from Trained Neural Networks. PhD
thesis, University of Wisconsin-Madison, 1996. URL htt.hhttss////m http:::://wwwwbbbt http://www.botttttt/tooo www...i.......... edu/
#craven/papers//hees pdf
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Sociery: Series D (The Statistician), 1983.
Mostafa Dehghami, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In Proceedings of the 7th International Conference on Learning Representations, ICLR'19, 2019. hotthhtps://// /aa.tps://axxx...o www.riiiir.rrroooooooooooo abs 11877003111
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT' 19, 2019. URL ptstts://arxive http://wwwww....//////////110.// 04805
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprini arXiv: 2002.06305, 2020. URL httpp:::///::
absp://b///00 2002. 06305
Jeffrey L. Elman. Finding structure in time. Cognitive Science, 14:179-211, 1990. URL https:
7/crl. ucsd.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, train- able neural networks. In Proceedings of the 7th International Conference on Learning Rep- resentations, ICLR'19, 19, 2019. URL http:://blpp httt:://bblp.uu--tiiee...ee www.hhtmmFrraeeeetc
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural machine translation. CoRR, abs/1 hts/1702.01802. 2017. htt./ittt.//aaxxi.orr///////////
Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. In Proceedings of the First International Workshop Comprehensibiilit and Explanation in Al and ML 2017 co-located with 16th International Conference of the Itaiian Association for Artificial Intelligence, 2017. URL /tttp:://axxi..ttt
http://b///10 .97844
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandku- Bom-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICMLI'18 18, 2018. URL httph: http:/://aaaxv.aao org abs// 04770
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016 http:
Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. arXiv preprini arXiv: 1302.4389, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, 2017. URL htI.tttss:t.. httttsss/aaaal..oor//aa///////// wtt./oogg/abb//o/ooos 04599
10

---

Under review as a conference paper at ICLR 2021
Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8, 2020.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. arXiv preprint arXiv::/04.003300 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015. URL
org. wttp://wwww...r///////////// 02531
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural computation, hew.aotoooooooooooooootttttttto 1780, 1997.
988)11735-
1.7.5
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint 19MV::990000888,, 2019.
05858
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP' 16, 2016. URL ht https:/
/www aclweb. org/ anthologhoooooooooooooooootoooot watpooggy//6111333
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,
abs/1412.6980, 2014. URL sttps://arxivv wttp:iwww..g//////12...... 6980.pdf
Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. Scalable syntax- aware language models using knowledge distillation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1472-3484, Florence, Italy, July 2019. URLhttta //Wt /ww
Linguutttsso aclweb. wwwp.///g//thhhloo..../o/o/////e /ttpa///oooooooooooooo/////11110
Association for Computational Linguistics. doi: 20.18653/1//199/ 1337.
Adhiguna Kuncoro. Lingpeng Kong. Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, and
Phil Blunsom. Syntactic structure distillation pretraining for bidirectional encoders. arXiv preprint arXiv:2005. 13482, 2020.
Aarre Laakso and Garrison Cottrell. Content and cluster analysis: Assessing representational similarity in neural systems. Philosophical Psychology, 13(1):47-76, 2000. doi: 10. 1080/ 09515080050002226 URL.
Phong Le and Willem Zuidema. The inside-outside recursive neural network model for dependency parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP'14, 14, 2014. htt.ttt::: / www. ac web. .eob.oogg/oo anthol ooyllggyllll pdf
/D14//144 4- 1081
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computationnal Linguistics, 4(0), 2016.
URL httl:tttt///sssseeeeee org/ojs/ index wttpeeepppp/aaaa/ttttteeeee////e
Xiaodong Liu, Pengeheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. ArXiv, (bs//1944.09433 2019. URL/attps://xxxvvorro abs 10t...994..000 09482
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, In Pro- ceedings of the 5th International Conference Learning Representations, ICLR 17, 2017. URL https: /tttp:///axxvv /arxivv.... org. ottg/bss////oooooooooooo//..s 2obs/16053 03983
Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. Knowledge amalgamation from heterogeneous networks by common feature learning. In Proceedings of the Sheent-Eighhtl International Joint Conference on Artificial Intelligence. 19, 2019. URL
htttps://arxiss org/
waigg/abs. abs/ 1906 10546
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent net- works. In Advances in neural information processing systems 32, NeurlPS' 19, 2019. URL
https://arxil wwwrriivoorggaaoo wwwp///////oooooooo/oo////... /ttp///077000008

---

Under review as a conference paper at ICLR 2021
R. Thomas McCoy, Robert Frank, and Tal Linzen. Does syntax need to grow trees? sources of
hierarchical inductive bias in sequennce-t--sequeeee https: wttp//aaxvvvv///a////i/ org/ abs 2001 03632
networks. CoRR, abs/2001.03632, 2020. URL
Tom M. Mitchell. The need for biases in learning generalizations. Technical report, Rutgers University,
New Brunswick. NJ, 1980. URL http::///mmmmmmmm cs www..ssc.......... wtw.ss.....uuuuuueeoeeee wwt.:edup..ccc....eee./ mldm tools/
Reaciing/NeecforBBiie pdf
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartlet, Self-distillation amplifies regularization in hilbert space. arXiv preprint (xXi::0022005/5,, 2020.
Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv: 02337, 2019.
Rafael Rodrigo Mueller, Simon Korbllith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems 32, NeurlPS' 19, 2019. URL
02629
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008. 16687, 2020.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR' 19, 2019. URL
abs/ 1904 05068
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. volume 97 of Proceedings of Machine Learning Research, pp. 5142-5151, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL UR..ttpp:
ngs. m r. wttpss/se//ppupeeeeeee
9a html
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog.
[ttp:p/.hh...aeeaaaaaaaaaaraaaan http://aas...eeeebppaae///aaaaaa 2019.
URL. https: h/wp::::/wwww... www. http://wwww.eii...aaaaaaaaaa
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019. URL
atttss://ariii: www.orgg//b.... 1910. 01108
Andrew Michael Saxe, Yamini Bansal, Joel Dapello. Madhu Advani, Artemy Kolchinsky, Bren- dan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In Proceedings of the 6th International Conference Learning Representations, ICLR' 18, 2018. URL
net wtt.ee/wwwneeeeoorddddeer... WPG A
H. S. Seung, H. Sompolinsky, and N. Tishby. Learning curves in large neural networks. In Proceedings of the Fourth Annual Workshop on Computational Learning Theory, COLT '91, 1991. URL
https: / acm. org. doi 10. 505551155115.. 114437
114647,
Richard Socher, Christopher D. Manning, and Andrew Y. Ng. Learning contin- phrase representations and syntactic parsing with recursive neural networks. In In Proceedings of the NeurlPS'J0 Deep Learning and Unsupervised Feature Learning Workshop, 2010. URL http::::/aa..saaforddte htt.://oorrr..daaaaaaaaaaeere hew.ff/fw........oooooooeeeeeeee
Suraj Srinivas and R. Venkatesh Babu. Data-free parameter pruming for deep neu- ral networks. In Proceedings of the 26th British Machine Vision Conference,
BMVC'15, 2015. Data-freet -rr Neural Seiiiiii-ii--iiii Babu/ bbbe441a00004cc00000444444433233
URL https: www. www.nteetttt...m wwsccholllaeoooo ortpppppr//pp/////// hwtps/////w..ssmatt.m.mm
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In FMNIP//JINIPP.. 2019.
12

---

Under review as a conference paper at ICLR 2021
Iya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum deep learning. In International conference on machine learning, ICML'I3 2013. URL
dl acm. org/ 3o1/10..55533 5555/3042811111111 3043064
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Yan Liu. Multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv: 1902. 1902,10461, 2019. URL https
//rxiv.org/abs//9g.. 10461
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03532, 2020.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. ArXiv, Moss/903.12136 2019. URL
hhttss://ass //rxivv org/ abs 1903. 12136
Ke Tran, Arianna Bisazza, and Christof Monz. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP' 2018. URL https: ac aww.eb. www.awwwwwww....anooooaoeeeeaeae antholog/
D18-1503
Frederick Tung and Greg Mori. Similarity knowledge distillation. ArXiv, abs/1907.99682, 2019. httt:::/aaxxi..or///bba222/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems 30, NeurlPS' 17, pp. 5998-6008, 2017. URL http:: /ttppp/://axxv..o//o/a//o 1706.03762
D. H. Wolpert and W. G. Macready. No free lunch theorems for optimization. IFFE Transactions on Evolutionary Computation, 1(1):67-82, April 1997. ISSN 1089-778X. doi: 10. (0.1109/42355585892
URL https: doi wrg/10ggg/ooooo 1109/ 4235. 585893
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: teacher-free framework. arXiv preprint arXiv: /723, 2019.
13

---

Under review as a conference paper at ICLR 2021
A KNOWLEDGE DISTILLATION IN NEURAL NETWORKS
Knowledge Distillation is a technique that transfers knowledge from one model to another (Hinton et al. [ 2015). Hinton et al. (2015) suggest that the power of KD is mostly in being able transfer the useful information that is embedded in the soft targets of the teacher model, c.g., the relation between the output classes as captured by the teacher model. This is often referred to as dark knowledge. Phuong & Lampert (2019) studies KD from a theoretical point of view in a simplified setting where the task binary classification, and teacher and student are linear models. They attribute the success of distillation to three main factors: (1) data geometry, (2) optimization bias, and (3) strong monotonicity. And more recently Tang et al. (2020), , conduct extensive analysis and identify three sources for why KD helps: (1) label smoothing, (2) example re- weighting based on teacher's 's confidence, and (3) prior knowledge of optimal output layer geometry.
The most well-known use of KD is to compress large, unwieldy model an ensemble model into a smaller model. Empirically, many people have found that bigger models are easier to train (often explained with the 'lottery ticket hypothesis" (Frankle & Carbin 2019); ): KD makes it possible to distill the knowledge in the large model into a much smaller model, and thus in some sense offer the best of both worlds (Buciluà al., 2006 Hinton al., 2015 Srinivas & Babu 2015). Distilling knowledge from a very big model or an ensemble of models with similar or heterogeneous architectures that are trained on the same different tasks into single model with much fewer parameters can lead to similar or sometimes even better performance compared to the teachers (Luo
al. 2019 Liu et al. 2019 Hinton al. 2015 Tan et al. 2019 Kim & Rush 2016).
Previous work has examined the effectiveness of KD in different settings: where the teacher is bigger than the student, but both have similar building blocks (Hinton et al. 2015 Kim & Rush
2016 Sanh et al.] 2019); where teacher and student are of similar size and architecture (Furlanello
et al. 2018 Freitag et al. 2017); or where the student and teacher have fundamentally different
architectures (Frosst & Hinton 2017 Tang et al. [019 Luo et al. [019] Ahhn et al.) 2019))
KD has also been proposed as an interpretation technique, where the knowledge of a big com- plex model is distilled into a more interpretable model (Craven & Shavlik 1995, Craven 1996 Frosst & Hinton, (2177;; Or as a method to compare the capacity and expressiveness of different models [Maheswaranathan et al., 2019; Saxe et al., 2018).
Offline Distillation In most cases, KD is applied in an offline setting, i.e., we first train the teacher network and use the trained teacher train the student, while the parameters of the teacher are fixed. This is the standard distillation process introduced by Buciluã et al. (2006); Hiinon et al. (015)). We apply this setup in our experiments since it is the most common approach. There are other possible settings for KD, e.g. online distillation, where teacher and student models are trained simultaneously.
Distillation Loss There are several different ways of computing the distillation loss: using only the output of the teacher or taking intermediate layers into account as well (Anil et al. 2018 Ahn et al.. 2019 Sun et al. 2019 Park et al. 2019 Tung & Mori 2019: Buciluã et al. 2006; Hinton et al. 2015). Potentially, using these alternative losses could lead to transferring different kinds of knowledge depending on the tasks and the configurations of the models. While it is worth doing a thorough comparison of all these techniques, in this paper we have focused on the most commonly used loss introduced by Hinton et al. (2015). which is based on the Kullback-Leible divergence between output distributions of the teacher, i.e., soft targets, and the output distributions of the student. The output distributions of the teacher and student model, Pt and Pa. are computed similarly, with Equation
[
exp(2./T) Σ, exp((2//T))
(1)
where 7 > 1 is the softmax temperature and 2 is the logits from the model.
The distillation loss is: H(P,, P.). where H is the cross entropy loss and is computed as:
H(PP. = P(())log
-PP.P.))<R(())() (x)
(2)
When KD is applied as means for model compression, it is common to compute the total loss as mixture of distillation loss and actual loss. Since, our focus in this paper is on how much the student model can learn from the teacher model, in our experiments we use pure distillation.
14

---

Under review as a conference paper at ICLR 2021
B VISUALISATION OF REPRESENTATIONAL SIMILARITY OF THE ACTIVATIONS FROM THE PENULTIMATE LAYER
To compare and visualize the state of m different models to each other (at convergence or any stage of training), we propose using representational similarity (Laakso & Cottrell 2000 Abnar et al. 2019) of the activations from their penultimate layer.
Note that representational similarity measures how similar two models learn to represent the data in terms of the global "relations" between all the data points, not local example-by-exee -xeempleeee similarity. In fact, the "direct" similarity between the activations of the penultimate layers of two models can be quite low, while having high representational similarity. This is because models can keep the relations between data points similar while embedding data into completely different representational spaces.
This is particularly useful when these models do not have the same architecture and their parameter space not directly comparable. To do so, given sample set of size n from the validation/test set (e.g. 1000 examples), we feed them to the forward pass of each model to obtain the representation from the penultimate layer of the models. Then, for each model, we calculate the similarity of the representations of all pairs from the sample set using dot product which leads to a matrix of size n. We use the samples similarity matrix associated with each model to compute the similarity between all pairs of models. To do this, we compute the dot product of the corresponding rows of these two matrices after normalization and average all the similarities of all rows, which leads to a single scalar. Given all possible pairs of models, we then have model similarity matrix of size m x m. We then apply multidimensional scaling algorithm to embed all the models a 2D space based on their similarities.
The code for projecting the representational similarity of the activations from the penultimate layer to 2D space can be found in hatp::://ANNOMMI/Z55 ZED
C DO THE DISTILLED MODELS CONVERGE TO THE SAME BASIN IN THE LOSS LANDSCAPE?
To gain better understanding of the effect of KD and in- ductive biases of the models from an optimization point of view, we looked into how different models relate in terms of the solutions they converged to in the loss landscape.
cnn-cnn
0.18
mlp-mip
0.34
To do so, inspired by the discussion in (Neyshabur et al.) 2020), we look into different pairs of models and check if their final solution belong the same flat basin of the loss landscape or they converged to completely different optima. To do so, given two models,m, and m2, we take their parameters, 0g and f2z and evaluate series of models obtained by linearly interpolating 01 and 02. with different coefficient, i.e., the parameters of model m, is computed as 02 X,0, + (1 A1) It has been shown (Neyshabur et al. 2020) that if the converged solutions of m. and m2 belong to the same flat basin of the loss landscape, the models obtained by linearly interpolating their parameters are well-behaved because they also remain in that basin. However, for two models that converge to different optima and don't share the flat basin of the loss landscape, the liner interpolations do not lead to well behave models.
0:12
0.06
0.06
0.04
0.02
Figure 8: Performance barriers between dif- ferent instances of MLPs and CNNs (with the same initialization). in terms of loss on the test.
Here, we first, compare different instances of MLPs and CNNs. We train two instances of the same architecture with the same initial state but different random seeds (which would lead to different ordering of training examples, and different dropouts). Figure 8 shows the loss on the test set (y axis) for the two trained instances, as well as models obtained by linear interpolation of the two
httpps: / /ttttittittteeeeeeeeeeeeeeeiitcc meikiii-eeaaaaa learn. org/ hetg//tabl abblee htt//moolleeaeeeeeeeeeeeeeeeeeee e// / /tt.g/geeee////eeeeeeee wttp//kleennn.aam www.mmnninnoinmmmmmmmmmmmmmm.omm MDS html
Basin refers areas parameter space where the loss function has relatively low values.
15

---

Under review as a conference paper at ICLR 2021
models with different Às (r axis). In the of MLPs, there is large barrier between the two instances, showing that these models, even with the same initialization, will converge to solutions in different basins of the loss landscape. In contrast, for CNNs, their strong inductive biases drive them to converge to the solutions in the same basin, regardless of the stochasticity of the training process. This also supports the higher variance in the results report for models with weaker inductive biases in 2.2 and $3.2
Next, we look into the effect of distillation on the diversity of the basins different instances of models converge to. Figure 2 shows the performance barriers of different pairs of MLPs (MLP#1 and MLP#2), when they are trained independently (i.e. when the teacher is data), as well as trained through KD, with an MLP and CNN model as teachers.
First all, we observe that two models, initialized similarly but with different random seeds, trained through distillation with the same teacher are likely to converge to the same area in the loss surface (plots (c) and (f)). This happens regardless of the inductive bias of the teacher and student models. Comparing the plots in the diagonal of Figure 9 we can see that for both CNN + M LP (plot and M + MPP (plot c) the performance barrier rather small in contrast to the large barrier between two independently trained MLPs (plot a). This indicates the power of KD to narrow down the search space the student model and drive it to particular set of solutions.
nip
Data
Teacher for MLPP1
=lpomippmin
nippmp-mippmp
MLP
(b)
(c)
cnn2mip.nk mlp
cnn2mp mip2mip
014
CNN
013
0.5
(e)
(f)
MLP
Teacher for MLP#2
Figure Performance barriers between different instances of MLPs with the same initialization trained independently or through knowledge distillation. Here y-axis on each subplot is the value the loss on the test set and æ-axis is the value of the interpolation coefficient, x. The rows in the figure correspond to the teacher of the instance on the left side (MLP#1) and the columns correspond to the teacher of the instance on the right side of the plots (MLP#2).
Moreover, comparing the distilled instance of a model with an independently trained instance with the same initialization and different random seeds, the first column not in the same basin, regardless of
(a), (b), and (d)),
we see that the distilled instances and independent instances the teacher but the barrier is larger (larger bump in the plots) when the teacher has a stronger inductive bias (CNN + M.P). Similarly, depicted in the second and third columns of Figure while models distilled from the same teacher seem to close in the loss surface (plots (c) and (f)), models distilled from different teachers (plot (e)) seem be further away (have larger barrier in between).
16

---

Under review as a conference paper at ICLR 2021
D PERFORMANCE SCORES ON THE TRAINING DATA
In the paper, for our first test case, we report the performance of LSTM and different Transformer models on the test set, when trained independently and with knowledge distillation. We observe that LSTMs achieve better accuracy on the test set compared to Transformers due their inductive biases. Here, we also report the performance of all the models, for both classification and LM setup, on the training set, which confirms that Transformer models have enough capacity to achieve good scores on the training data.
This solidifies the narrative that the inductive bias of LSTMs is helping with generalization and rules out, for example, the possibility that LSTMs have higher capacity trained better.
Model
Perplexity
D Accuracy f
A A-Accuracy
Transformer
29.62 + 0.10
0.956 0.001
0.936 + 0.004
Small Transformer
33.02 + 0.05
0.959 + 0.001
0.948 + 0.005
LSTM
28.92 = 0.08
0.964 + 0.003
0,955 = 0.003
Small LSTM
31.03 + 0.11
0.964 + 0.001
0.952 + 0.006
Table Performance (mean over trials) of different LSTM and Transformer models trained independently with the LM objective on the training set.
Model
Train μr- Accuracy 1
Transformer
99.57
Transformer-seq
99.57
UniversalTrannformerssss
99.66
LSTM
98.62
Table Performance (mea over trials) of different LSTM and Transformer models trained independently with the classification objective on the training set.
E PER-SAMPLE BEHAVIOUR
To compare the models with each other and better understand how distillation affects the student models, we take closer look at their per sample behavior and investigate if the errors a student model makes are more similar to its teacher's errors. Here, we look into the error overlap of the students and teachers, which reflects their similarity in terms of their behavior per data example. This similarity can be another proxy to measure the similarity of the solutions learned by the models, with and without distillation. Figures I0 IT and illusttrae the error overlap between different models as Venn diagrams when they are trained independently and when we use distillation.
In Figure 10, we observe that when the Transformer and LSTM models are trained independently, [igguresTObbaaae two independent LSTMs behave more similarly compared to two Transformers (Figures TOb and [Oa). Given a similar number of trainable parameters, i.e., similar capacity for LSTMs and Transformers, this again supports the claim that models with stronger inductive biases converge more similar solutions (Also shown in Figure Ba)
When apply KD in cross- arsstarchiee setting, with LSTM teacher and student Transformer, Figures [Od[and and Figure 10c] the student Transformer behaves more similarly to the LSTM teacher and independent LSTM. M. compared the independent version of itself. This confirms that through distillation the way the student model solves the task becomes more similar to the way the teacher model solves the task.
We have similar observations in Figures II and 12, where errors of a student MLP are less and more similar to the errors the teacher CNN compared to independently trained ML.P
17

---

Under review as a conference paper at ICLR 2021
Tesed
326
133
240
45
213
24
280
Stodent-Trandterre
LXEN
(a)
(b)
(C)
(d)
Figure 10: Error overlap for LSTM and Transformer models trained with the classification objective on SVA task. These Venn diagrams show the intersections of the sets of examples miss-classified by the models. In (a) we compare two independent LSTMs (LSTM#1 and LSTM#2) and an independent Transformer; in (b) we compare two independent Transformers (Transformer# and Transforme#22) and an independent LSTM; in (c) we compare a student Transformer and a teacher LSTM with an independent Transformer; and in (d) we compare student Transformer and teacher LSTM with an independent LSTM.
RL
Guacker
CRBB
55
740
1246
66
180
143
167
BLudes:-HH HLP
Ssudent-
XEP
CIN
(b)
(c)
(d)
Figure 11: Error overlap for CNN and ML.P models trained on MNIST and tested on Scaled-MNISS set from MNIST-C dataset. These Venn diagrams show the intersections of the sets of examples miss-classffied by the models. In (a) compare two independent CNN (CNN#1 and CNN#2) and an independent MI.P; in (h) we compare two independent MLP (MLP#1 and MLP#2) and an independent CNN: in (C) compare student MLP and teacher CNN with an independent MLP; and (d) we compare a student MLP and a teacher CNN with an independent CNN.
CHSFI
NEM1
Pascher COO
86
T44
1760
332
CAL0
4:T2
Stadent-NELP
Studeet-NtL
(a)
(b)
(C)
(d)
Figure 12: Error overlap for CNN and MLP models trained on MNIST and tested Translated-MNNI set from MNIST-C dataset. These Venn diagrams show the intersections of the sets examples miss-classified by the models. In (a) we compare two independent CNN (CNN#1 and CNN#2) and an independent MLP; in (b) we compare two independent MI.P (MLP#1 and MLP#2) and an independent CNN; in (c) we compare student MI.P and teacher CNN with an independent MIP; and in (d) we compare student MLP and teacher CNN with independent CNN.
F IMPACT OF THE QUALITY OF THE TEACHER
Here, in an ablation experiment for our second case study, we investigate the impact of the quality of the teacher in the on-distriution set on the generalization the student in the oft-t--distributiin set. To do so, given CNN the teacher and an MIP as the student, take snapshots of CNN model
18

---

Under review as a conference paper at ICLR 2021
during different stages of training teachers with different qualities (we use 9 different teachers). Using each teacher, we train an MLP student.
Figure 13 presents the quality of the different teachers based different test sets: Vanilla MNIST and Scaled MNISt cout-ofddistriuution For the CNN models that trained with ground truth labels vanilla MNIST, expected, the number of training iterations grows, the performance of the model on all the three test sets increases. In Figure [3b] we see that in general, the accuracy of the MLP students follows the same trend, i.e., better CNN teacher results in better MLP student. Given the results of an independently trained MLP from Table Sa the benefit of training an MLP via distillation for better generalization on and out of distribution sets only kicks in when we have CNN teacher with a quality more than a certain threshold.
(in-distribttion, Translated MNIST
MNIST
Translated MNIST
Scaled MNNST
Accuracy
0.8
Accuracy
0.6
500
1.000
1.500
2,000
0
500
1.000
1.500
2,000
Number of training steps
Number of training steps of the teacher
(a) CNN teachers
(b) MLP students
Figure 13: Effect of the quality of the teacher CNNs the accuracy of the student MLPs. In the left plot, points that share value on the x-axis represent quality of a CNN, with respect to different test sets: Vanilla MNIST (in-distribution). Translated MNIST (out-of-distribution) and Scaled MNIS (out-of-distribttin)) In the right plot, similar to the left plot, points with the same x-value represent the quality of a same MLP model, trained via KD using the teacher the corresponding place in the left plot, evaluated on the Vanilla, Translated, Scaled MNIST sets.
G IMPACT OF THE DATASET USED IN THE DISTILLATION STEP
In our experiments in this paper, our focus is on the setups where we use the same dataset that was used train the teacher model, to transfer its knowledge the student model.
We use this setup mainly because we want to see how effective is the distillation process to transfer the generalization behavior of the teacher in isolation, as using a different dataset in the distillation step would add another factor. In other words, during the training of the teachers and as well as the students (i.e., distillation step), we only use samples from the in-distribution set to make sure the desired generalization behavior is not apparent from the dataset used for training neither the teacher nor student models.
In this section, we extend the CNN-MLP experiments Corrupted-MNIST and look into the performance of the student model, when we use samples from the duttt--distributtin set in the distillation step for training the student.
Table presents the result an MLP student when we use different training sets the distillation step. We can that when distilling knowledge from CNN teacher that is trained on vanilla MNIST, we use translated or sealed MNIST the distillation step, the student MLPs achieve relatively high performance
the corresponding test sets, while the performance dutt---diittiibutinn set drops compared to when we use vanilla MNIST in the distillation step. To have complementary information for better comparisons, Table Tossows the accuracies of ML.s when
the other
19

---

Under review as a conference paper at ICLR 2021
Table 9: Accuracy of MLPs trained through KD with CNN teachers, where the CNN teachers trained on -idiiitibtttion (vanilla MNIST) training set, while the training set in the distillation step is either in in-distortion (first row) or -iittiibbttitttoon (second and third rows). Note that during the distillation step, the student do not have access to the ground truth labels from the training set.).
Test Dataset Translated MNIST
Distillation Dataset
MNIST
Scaled MNIST
MNIST
0.99 + 0.001
0.51 + 0.015
0.90 + 0.007
Translated MNIST
0.79 + 0.015
0.98 = 0.001
0.53 + 0.015
Scaled MNIST
0.79 + 0.016
0.30 = 0.009
0.98 = 0.001
Table 10: Accuracy of MLPs trained with ground truth labels on different splits of the Corruptd-MNNSST dataset.
Test Dataset
Training Dataset
MNIST
Translated MNIST
Scaled MNIST
MNIST Translated MNIST
0.99 = 0.001
0.37 = 0.015
0.79 + 0.015 0,42 - 0.039
0.62 - 0.011
0.98 + 0.001
Scaled MNIST
0.76 - 0.014
0.26 + 0.006
0.99 + 0.001
Table 11: Accuracy of CNNs trained with ground truth labels on different splits of the Corrupted -NNSS dataset.
Test Dataset
Training Dataset
MNIST
Translated MNIST
Scaled MNIST
MNIST Translated MNIST
0.99 + 0.001
0.98 + 0.000
0.96 + 0.002
0.99 - 0.001
0.99 + 0,001
0,97 = 0.002
Scaled MNIST
0.88 0.010
0.88 + 0.014
0.99 + 0.001
they are directly trained on each of these datasets. Interestingly, we observe that when trained through KD the performance of the student MLPs are higher or match the performance of MLPs when trained with ground truth labels, and additionally, they achieve better performance on the other datasets. For example, for the case when we use translated MNIST to train the MLPs, the accuracy of the student MLPs match the accuracy of MIPs trained with the ground labels, while the accuracies on the other two datasets (vanilla and scaled) are higher for the student MLPs trained with CNN teachers.
H DETAILED MODELS ARCHITECTURES AND TRAINING SETUP
For the subject-verb agreement task, we study Transformers and LSTMs. In the LM setup, we use two sizes for each architecture: LSTM: two-layer uni-direction LSTM, with a hidden size of 1024. Small LSTM: two-layer uni-direction LSTM, with hidden size of 512. Transformer; six-layer Transformer decoder with a hidden size of 512 and 8 heads. Small Transformer: Transformer: six-layer Transformer decoder with hidden size of 256 and 8 heads.
In the classification setup, we employ an LSTM and three variants of Transformer, where the LSTM has a two-layer with a hidden size of 256, and the Transformers have 6 layers, 8 heads, and a hidden size of 128. We use hidden size of 256 for the UniversalTrrnsfoomerssss since its parameters are shared in depth and with the same hidden size as other Transformers, it will have fewer parameters.
On the MNIST-C dataset, we study CNNs and MIPs. Our CNN has two 3 3 convolutions, followed by max- max--oooligg layer over spatial dimensions, followed by another 3 3 convolution and maxout (max-pood mmax-pooling over channel dimension) layer (Goodfellow et al.) 2013). Finally a global averaging is done over spatial dimensions, before the projection layer. The MLP model simply has three fully connected layers.
For training the independent models nential decay learning rate scheduler and for the student models in the distillation process, we use Adam optimizer with cosine decay restart (Loshchilov & Hutter 2017) learning rate scheduler. The
the Adam optimizer (Kingma & Ba 2014) with expo-
20

---

Under review as a conference paper at ICLR 2021
hyperparameters related to the regularization and learning rate schedulers tuned separately for each model/experiment. For each model, we report the set of hyper ppermearaeeters that gives the best average performance across multiple trials with different random seeds for initialization.
I
CODE
The code for all the analysis and experiments including the input pipelines, models, the details of the nyper-parameter sets used in our experiments are available at
to facilitate
the replication of all the experiments.
For the anonymized submission, we remove the links to the repository of the code from the paper and upload the code as zip file part of the supplementary materials for submission.
21


Reviews:
[{"review_id": "zp3LWFlt1n0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the oft-overlooked aspect of knowledge distillation (KD) -- why it works. The paper highlights the ability of KD for transferring not just the soft labels, but the inductive bias (assumptions inherent in the method, e.g. LSTM's notion of sequentiality, and CNN's translational invariance/equivariance) from the student so that the student exhibits, to an extent, the teacher's generalization properties as well. The paper explores doing KD between LSTMs and several versions of Transformers (with varying structural constraints) on a subject-verb-agreement dataset, and between CNNs and MLPs on MNIST and corrupted MNIST. Compared to prior work showing that better teacher performance lead to better student performance, this paper also shows that the student's performance on different aspects becomes more similar to the teacher's -- (1) if the teacher is strong on metric A and weak on metric B compared to a student on its own, the student can become stronger on A and weaker on B when distilled using the teacher; (2) if the teacher can generalize well to a separate, previously unseen dataset but the student generalizes poorly on its own, after distillation the student can generalize much better than it can possibly learn to on its own.\n\nPros:\n- Very interesting hypothesis and sheds light on the inner working of KD. (see above)\n- Interesting and novel set of experiments. Some (not all) experiments shed light on how the hypothesis seems to be true. (see above)\n- Comes up with ways to measure transferred inductive bias, by highlighting different aspects of generalization for a student and comparing with and without distillation.\n\nCons:\n- The writing is very confusing and cryptic, especially the first page until its last paragraph.\n    1. The abstract is especially not telling the readers much about what is in the paper. I personally would be confused and skip reading this paper because I thought the paper discusses \"can we distill knowledge using knowledge distillation\". Inductive bias come in many forms and is not often discussed, and it helps to use examples to tell the story directly, e.g. by mentioning the specific differences between inherent priors in CNNs/MLPs or LSTMs/Transformers in the abstract *and* the first paragraphs of the introduction.\n    1. Second page, bold \"Second\" and \"Third\" are the same thing.\n- Although after extensive thinking I believe the paper is distinct from previous KD analyses, the paper does not itself distinguish its findings enough from what is known in the literature.\n    1. Granted it is hard to distinguish the inductive bias transfer aspect of KD versus other aspects of KD, it is hard to experimentally prove it because the field does not quite know what are the aspects of KD that makes it work. But the paper does not do a good job explaining which behavior is certainly due to inductive bias transfer, rather than the behavior can possibly be caused by other hypothesis in the field, such as KD transfer \"knowledge\" of inter-class relationship, or the effect from soft labels.\n    1. Note that the ECE results don't tell readers much. People expect soft labels to help not because they make models better calibrated, but because they boost performance, and it's not clear if people think better calibration leads to better main performance. Even if people do, in Fig 3(b) the ECE improves quite a lot for Transformer student with better teachers, so it is wrong to claim \"Given the lack of significant improvement in ECE...\".\n    1. The CNN/MLP experiment only has tasks that CNNs outperform MLPs. It would make it more interesting to see a task where the MLP outperforms CNN, e.g. a made up task whose ground truth is the xor of a few pixel positions, which could be hard for CNNs while easy for MLPs.\n- Relatively small number of datasets. Just two datasets and two sets of networks is not very convincing to claim these findings generalize to other architectural changes.\n- Experiments are sometimes not apple-to-apple comparisons. And some experiments are not convincing or irrelevant.\n    1. The MNIST experiments only have two networks, and arguably CNN is absolutely better than MLP. It would make the point clearer if a worse CNN is used such that the MNIST-vanilla performance is the same as the MLP, and show improved generalization results on MNIST-C.\n    1. Figure 1 does not tell readers much, because latent representation can be both inductive bias and regular representation power, and we already know that KD can improve the student's representation power. Same for the third bullet point in page 2.\n    1. Suspect of cherry-picking results from which loss (LM or classification) to show. Figures 2,4 are experiments using the LM loss, and Figures 3,5 are using the classification loss, without giving a clear explanation why.\n\nSummary:\nGiven the interesting hypothesis and set of experiments, I think the community can benefit from this paper's findings in understanding KD so we use it more wisely, or at least generate more discussions of why KD is working. Despite the relatively unclear writing of the introduction and some experiments being unconvincing, the impact of the paper still outweighs these flaws.\n\n=============================\nUpdate\nWhile I agree in principle with Reviewer 1 that this paper has jarring flaws in writing and the rebuttal version does not adequately address it, I disagree that the writing warrants such a low score. I have seen worse papers with outrageous claims (e.g. try to claim significance with p=0.1) and I would not give those a 2. I would also disagree with R1 that there is no interesting result in this paper, because there is no prior work I know that even considers how distilled models generalize like their teacher.\nIf I were to grade this paper based on different aspects, the originality and significance would be both 9's, quality a 6 due to experiment issues and careless generalization, and clarity a 3-4 due to unclear motivation in the abstract/early intro and poor differentiation from prior work in terms of experiment design and analysis.\nThat said, the rebuttal did not change my mind that the writing probably will not be improved enough post-rebuttal, I would thus not be able to consider this a top paper despite the interesting observations.", "extra_scores": {}, "extra_text": {}}, {"review_id": "c2KLxBvGoS1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this manuscript, the authors investigate the power of KD to enable benefiting from the advantages of different models at the same time. It first talks about inductive bias can be crucial in some tasks and scenarios, and further show that when a model has the right inductive bias, we can transfer its knowledge to a model that lacks the needed inductive bias and indicate that solutions that the student model learns are not only quantitatively but also qualitatively reflecting the inductive biases of the teacher model. The paper is well written, but not easy to follow. The efforts of this study may help better learn and find suitable models for some AI issues.\n\nHere are my comments:\n1. It seems that these experiments are well planned but without more detail, it was challenging to thoroughly evaluate the proposed work. \n2.  How the large data sets will be integrated and analyzed, and what might come from the collective analyses was not described in detail. Some concern was expressed regarding whether the large amount of data generated would address the defined goals of the manuscript. More details regarding the integration and analysis of the experimental findings would help clarify this.\n3. I also wonder that how the batch effects among different data and data types will be taken care of, as the integration of multiple data types and analysis is the key to this model.\n4. I wonder why the manuscript chooses these models.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ag91KYqOKw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that knowledge distillation from a (teacher) model A with an appropriate inductive bias to a (student) model B lacking it can lead to B generalizing better than if B was trained without knowledge distillation (but not as well as A), including out-of-distribution. The authors also show that the resulting learned representations inside B, as well as the shape of the training trajectories, are more like those of A (than those of B without knowledge distillation). \n\nThis is not very surprising but is still interesting from the point of view of the understanding of the nature of inductive biases. We already knew that inductive biases (like translation invariance) can be transferred through examples (e.g. by generating data transformations such as translated images), so this paper extends that kind of idea to knowledge distillation to provide the targets for such examples.\n\nAnother nice contribution of the paper is the case study of the specific inductive biases of RNNs which transformers lack, decomposed into sequentiality, memory bottleneck and recursion. Not very surprising but the experiments confirm intuitions and expectations which is always useful.\n\nOne concern I have is 'so what?' and 'then what?'. Have the authors thought of possible way (in say, future work) to take advantage of that observation? It is not obvious, because if you already have a teacher model A with the right inductive biases for the given task, why would you care about training a student B which is going to be worse than A anyways? Just use A. In addition, unlike for the original motivation of knowledge distillation, we normally expect that B would have MORE capacity than A (because it needs to 'learn' the inductive biases, so one would expect it would not work to choose B much smaller than A, in the sense that the gain would be much smaller, and certainly not as good a model as using A).  We already knew that examples could transfer inductive biases, now we know that knowledge distillation can do it, but why would that be useful?\n\nExperiments where B is much smaller than A would be interesting, because in that case, it might be worthwhile to do the knowledge distillation from a larger but better biased A. Also, the outcome of such experiments would not be apriori obvious (we would expect a gain vs the regular B, but would it be sufficiently interesting to be worth it?).\n\nAnother question I would have liked to be studied is about what happens out-of-distribution (OOD). The paper already shows the unsurprising  result that distilling into B from A helps somewhat OOD. It would also be interesting to explore whether taking inputs outside of the training distribution of A as distillation examples when training B would increase the robustness of B OOD.\n\nMinor comments:\n\n- fig 4: caption is insufficient to understand the figure\n- the sec 3.2 sentence with 'almost closing the gap' is too strong and needs to be weakened (there is still a significant gap, with almost twice the error with B compared with A)\n- the conclusion sentence with 'demonstrate having the right inductive bias can be crucial' should be reformulated, since this is not a new demonstration (and reading it without reading the rest of the paper may give that false impression)\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8x7mxzJboFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I previously reviewed a version of this paper and unfortunately \nthe primary issues with it have not been addressed sufficiently. \nWhile some parts have changed,  I will draw on relevant portions \nof my previous review where appropriate. \n\nThis paper sets out to investigate the respective \"inductive biases\" \nof LSTM and Transformer neural networks, two dominant model families\nthat are frequently employed in applied NLP tasks.\nThey also seek to compare the \"inductive biases\" of CNNs and MLPs.\n\nThe air quotes are placed here because all generalization and thus \nany claim concerning the generalization performance of a model\nnecessarily concern (whether explicitly or implicitly) inductive biases.\n\nHowever, we do not typically need to invoke the term \"inductive bias\"\nin every single sentence in a paper just to discuss the comparative suitability\nof some models for some tasks and the comparatively poor performance.\n\nThere are times when it's beneficial not just to talk about comparative performance\nof models but to talk rigorously about inductive biases. In many settings we can\nformally characterize the bias of a hypothesis, e.g. through learning-theoretic \ncomplexity measures. However, here the term is used excessively with \nfuzzy claims made about some models having \"stronger\" or \"weaker\" inductive biases\nwithout invoking any concrete measure of the expressivity of a hypothesis class.\n\nSo the flaws with this paper are two-fold. First, I do not believe that the contribution\nis sufficiently interesting to warrant publication. Second, I do not believe that the \ncurrent exposition is suitable for publication.\n\nThroughout the authors confuse what has actually been showing in prior works \nfor what has been speculatively claimed in prior works. For example, the authors\nrefer to the better performance of LSTMs vs Transformers on a set of agreement tasks\nas an inductive bias for learning syntactic structures. \n\nThe authors show plots that simply depict performance but describe them \nas characterizing the bias-variance tradeoff (absent any discussion of variance).\nThe authors have a lengthy discussion of calibration that does not make much sense\nand parrots incorrect claims from previous papers such as the bizarre claim that label smoothing\ncalibrates classifiers (it's rather easy to see how label smoothing could lower ECE \nfor an otherwise overfit classifier but how in general it does not calibrate and can \neven decalibrate classifier. \nThe idea that calibration magically falls out of knowledge distillation\n[[[\npreviously this review had the clause:\n\"or that any of these models is \"perfectly calibrated\" (a claim they actually make)\"\nhowever, as the authors rightly point out, this was a mistake in my review,\nthe context here was defining a marginally calibrated classifier,\nnot in claiming that the KD models achieved perfect calibration.\n]]]\nis bizarre and unacceptable in a proper publication. \n\nThe knowlegde distillation experiments are interesting but the speculative interpretations\ngo far beyond what the actual experiments show (that distilling from a better performing teacher model\ngives a better performing student model, regardless of the student's architecture).\n\nIn short, this paper is not suitable for publication and must be substantially rewritten.\nThe authors need both a more compelling result and a more forceful editor.  \nThe authors train {LSTMs, Transformers, etc} for some tasks\n where they have been shown to perform better,\nand show that downstream student models benefit from better teachers, \nbut this doesn't substantiate what strikes me as a strained framing of the material\n\n======================================================================\n\nUPDATE AFTER READING THE REBUTTAL\n\n======================================================================\n\nThe rebuttal was thoughtful and detailed, and caught one careless error in my review\nand I appreciate the author's care. At the same time the rebuttal itself contained \nmany conceptual flaws and failed to alleviate many core concerns. \nNevertheless, I think it warrants a minor increase in my score from 2 to 3. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies through empirical analysis an interesting problem: distilling the (strong) inductive bias of a teacher model to the student model (of weak inductive bias). The main claim/finding is that not only the "dark knowledge" in the logits can be transferred, but also the inductive bias (e.g. recurrence in RNN and translation invariance in CNN) can be transferred to make the student model stronger. This conclusion looks not very surprising but does contribute some new ideas to the fields of both deep learning and transfer learning.

The paper receives insightful but controversial reviews. Throughout reading the lengthy rebuttal and discussions, the AC, as a neutral referee of both sides, thought that while some expressions in the discussions seem a bit urgent and strained, both reviewers and authors managed to participate in the academic debate with a professional attitude that focus only on the technical issues. These discussions are very extensive and helpful for drawing a thorough understanding of this paper, and of the important research problem.

After the public interactions with the authors, a private discussion was performed between all reviewers and the AC, and among the four reviewers, one argued for rejection, one voted for rejection, one voted for acceptance, and one argued for acceptance. The AC believed that one of the reject votes lacks enough support in the comments and thus discarded it.
However, due to the wild disagreement between the reviewers, as well as between the reviewers and the authors, the AC read the paper carefully. The AC's main points are as follows:

- The research problem is interesting, and this paper appears to be the first work that studies the inductive bias transfer problem.

- The paper has made its endeavor to try to delve into this problem, through providing with extensive empirical results and analyses.

- The biggest weakness of this paper is the experimentation approach towards quantitatively studying the inductive bias: comparing the teacher and student models through the relational similarity between the penultimate-layer representation is simply not enough to justify that the inductive bias has been distilled/transferred.

Two reasons:
+ Due to the expressiveness of neural networks, it is not hard for the student to resemble the teacher's representations; in fact, this is a quite common result repeatedly used by researchers and practitioners, even when the student is only a smaller model. While the idea of distilling inductive bias is interesting, it simply cannot be sufficiently justified by the current experimentation design.
+ Inductive bias is something encoding our prior knowledge about the learning task and is often effective during the whole training procedure, which cannot be refreshed by the training data. However, albeit the distilled student model (transformer or MLP) resembles the representations of the teacher model (RNN or CNN), it is not certain whether the "distilled inductive bias" can linger in the student model if you further fine-tune the student model to downstream tasks. That is, it is highly possible that such "distilled inductive bias" of the student model will be refreshed by the future training data. In contrast, if we directly fine-tune the teacher model to downstream tasks, their inductive bias (recurrence of RNN or translation invariance of CNN) will be retained successfully in the fine-tuned model.
Basically, through this thinking, it is not clear whether the inductive bias has be distilled. If the distilled thing is refreshed out, it is probably not the inductive bias. More experimentation or a formal quantification of inductive bias is highly necessary here.

While Reviewer #1 was a bit skeptical in the comments and discussions (regarding which I had a private discussion with him/her), some of his/her comments are reasonable and should be well addressed before this paper could be accepted:
- Be rigorous in scientific writing. While the experiments with bias-variance tradeoff and calibration are interesting and relevant, the key concepts were used with less care. It is good to expand the authors' understanding of these concepts to make sure what they actually refer to.
- Try to provide sufficient elaboration when you try to claim something. It is true that for now, in our field, there are quite a few papers claiming something very big in the title or abstract, but simply cannot fulfill their story through rigorous or sound technical study. I suggest to tone down some of the key claims such that "inductive bias can be transferred" if they are not clearly provable.

Finally, AC believes this paper studies a very interesting problem that may draw wide attention, and the paper is acceptable in a future version if the above comments are well addressed. Since this is already a resubmission (as mentioned by Reviewer #1), I'd encourage the authors to focus on the technical parts of the comments and revise the paper substantially before submitting to yet another top venue.
Decision: Reject