Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this manuscript, the authors investigate the power of KD to enable benefiting from the advantages of different models at the same time. It first talks about inductive bias can be crucial in some tasks and scenarios, and further show that when a model has the right inductive bias, we can transfer its knowledge to a model that lacks the needed inductive bias and indicate that solutions that the student model learns are not only quantitatively but also qualitatively reflecting the inductive biases of the teacher model. The paper is well written, but not easy to follow. The efforts of this study may help better learn and find suitable models for some AI issues.

Here are my comments:
1. It seems that these experiments are well planned but without more detail, it was challenging to thoroughly evaluate the proposed work. 
2.  How the large data sets will be integrated and analyzed, and what might come from the collective analyses was not described in detail. Some concern was expressed regarding whether the large amount of data generated would address the defined goals of the manuscript. More details regarding the integration and analysis of the experimental findings would help clarify this.
3. I also wonder that how the batch effects among different data and data types will be taken care of, as the integration of multiple data types and analysis is the key to this model.
4. I wonder why the manuscript chooses these models.
review_title: I think that the issues are interesting, but of limited scope.

final_score: rating: 3: Clear rejection
review_content: I previously reviewed a version of this paper and unfortunately 
the primary issues with it have not been addressed sufficiently. 
While some parts have changed,  I will draw on relevant portions 
of my previous review where appropriate. 

This paper sets out to investigate the respective "inductive biases" 
of LSTM and Transformer neural networks, two dominant model families
that are frequently employed in applied NLP tasks.
They also seek to compare the "inductive biases" of CNNs and MLPs.

The air quotes are placed here because all generalization and thus 
any claim concerning the generalization performance of a model
necessarily concern (whether explicitly or implicitly) inductive biases.

However, we do not typically need to invoke the term "inductive bias"
in every single sentence in a paper just to discuss the comparative suitability
of some models for some tasks and the comparatively poor performance.

There are times when it's beneficial not just to talk about comparative performance
of models but to talk rigorously about inductive biases. In many settings we can
formally characterize the bias of a hypothesis, e.g. through learning-theoretic 
complexity measures. However, here the term is used excessively with 
fuzzy claims made about some models having "stronger" or "weaker" inductive biases
without invoking any concrete measure of the expressivity of a hypothesis class.

So the flaws with this paper are two-fold. First, I do not believe that the contribution
is sufficiently interesting to warrant publication. Second, I do not believe that the 
current exposition is suitable for publication.

Throughout the authors confuse what has actually been showing in prior works 
for what has been speculatively claimed in prior works. For example, the authors
refer to the better performance of LSTMs vs Transformers on a set of agreement tasks
as an inductive bias for learning syntactic structures. 

The authors show plots that simply depict performance but describe them 
as characterizing the bias-variance tradeoff (absent any discussion of variance).
The authors have a lengthy discussion of calibration that does not make much sense
and parrots incorrect claims from previous papers such as the bizarre claim that label smoothing
calibrates classifiers (it's rather easy to see how label smoothing could lower ECE 
for an otherwise overfit classifier but how in general it does not calibrate and can 
even decalibrate classifier. 
The idea that calibration magically falls out of knowledge distillation
[[[
previously this review had the clause:
"or that any of these models is "perfectly calibrated" (a claim they actually make)"
however, as the authors rightly point out, this was a mistake in my review,
the context here was defining a marginally calibrated classifier,
not in claiming that the KD models achieved perfect calibration.
]]]
is bizarre and unacceptable in a proper publication. 

The knowlegde distillation experiments are interesting but the speculative interpretations
go far beyond what the actual experiments show (that distilling from a better performing teacher model
gives a better performing student model, regardless of the student's architecture).

In short, this paper is not suitable for publication and must be substantially rewritten.
The authors need both a more compelling result and a more forceful editor.  
The authors train {LSTMs, Transformers, etc} for some tasks
 where they have been shown to perform better,
and show that downstream student models benefit from better teachers, 
but this doesn't substantiate what strikes me as a strained framing of the material

======================================================================

UPDATE AFTER READING THE REBUTTAL

======================================================================

The rebuttal was thoughtful and detailed, and caught one careless error in my review
and I appreciate the author's care. At the same time the rebuttal itself contained 
many conceptual flaws and failed to alleviate many core concerns. 
Nevertheless, I think it warrants a minor increase in my score from 2 to 3. 
review_title: Some interesting but insufficient experiments with a fundamentally flawed presentation 

final_score: rating: 7: Good paper, accept
review_content: The paper investigates the oft-overlooked aspect of knowledge distillation (KD) -- why it works. The paper highlights the ability of KD for transferring not just the soft labels, but the inductive bias (assumptions inherent in the method, e.g. LSTM's notion of sequentiality, and CNN's translational invariance/equivariance) from the student so that the student exhibits, to an extent, the teacher's generalization properties as well. The paper explores doing KD between LSTMs and several versions of Transformers (with varying structural constraints) on a subject-verb-agreement dataset, and between CNNs and MLPs on MNIST and corrupted MNIST. Compared to prior work showing that better teacher performance lead to better student performance, this paper also shows that the student's performance on different aspects becomes more similar to the teacher's -- (1) if the teacher is strong on metric A and weak on metric B compared to a student on its own, the student can become stronger on A and weaker on B when distilled using the teacher; (2) if the teacher can generalize well to a separate, previously unseen dataset but the student generalizes poorly on its own, after distillation the student can generalize much better than it can possibly learn to on its own.

Pros:
- Very interesting hypothesis and sheds light on the inner working of KD. (see above)
- Interesting and novel set of experiments. Some (not all) experiments shed light on how the hypothesis seems to be true. (see above)
- Comes up with ways to measure transferred inductive bias, by highlighting different aspects of generalization for a student and comparing with and without distillation.

Cons:
- The writing is very confusing and cryptic, especially the first page until its last paragraph.
    1. The abstract is especially not telling the readers much about what is in the paper. I personally would be confused and skip reading this paper because I thought the paper discusses "can we distill knowledge using knowledge distillation". Inductive bias come in many forms and is not often discussed, and it helps to use examples to tell the story directly, e.g. by mentioning the specific differences between inherent priors in CNNs/MLPs or LSTMs/Transformers in the abstract *and* the first paragraphs of the introduction.
    1. Second page, bold "Second" and "Third" are the same thing.
- Although after extensive thinking I believe the paper is distinct from previous KD analyses, the paper does not itself distinguish its findings enough from what is known in the literature.
    1. Granted it is hard to distinguish the inductive bias transfer aspect of KD versus other aspects of KD, it is hard to experimentally prove it because the field does not quite know what are the aspects of KD that makes it work. But the paper does not do a good job explaining which behavior is certainly due to inductive bias transfer, rather than the behavior can possibly be caused by other hypothesis in the field, such as KD transfer "knowledge" of inter-class relationship, or the effect from soft labels.
    1. Note that the ECE results don't tell readers much. People expect soft labels to help not because they make models better calibrated, but because they boost performance, and it's not clear if people think better calibration leads to better main performance. Even if people do, in Fig 3(b) the ECE improves quite a lot for Transformer student with better teachers, so it is wrong to claim "Given the lack of significant improvement in ECE...".
    1. The CNN/MLP experiment only has tasks that CNNs outperform MLPs. It would make it more interesting to see a task where the MLP outperforms CNN, e.g. a made up task whose ground truth is the xor of a few pixel positions, which could be hard for CNNs while easy for MLPs.
- Relatively small number of datasets. Just two datasets and two sets of networks is not very convincing to claim these findings generalize to other architectural changes.
- Experiments are sometimes not apple-to-apple comparisons. And some experiments are not convincing or irrelevant.
    1. The MNIST experiments only have two networks, and arguably CNN is absolutely better than MLP. It would make the point clearer if a worse CNN is used such that the MNIST-vanilla performance is the same as the MLP, and show improved generalization results on MNIST-C.
    1. Figure 1 does not tell readers much, because latent representation can be both inductive bias and regular representation power, and we already know that KD can improve the student's representation power. Same for the third bullet point in page 2.
    1. Suspect of cherry-picking results from which loss (LM or classification) to show. Figures 2,4 are experiments using the LM loss, and Figures 3,5 are using the classification loss, without giving a clear explanation why.

Summary:
Given the interesting hypothesis and set of experiments, I think the community can benefit from this paper's findings in understanding KD so we use it more wisely, or at least generate more discussions of why KD is working. Despite the relatively unclear writing of the introduction and some experiments being unconvincing, the impact of the paper still outweighs these flaws.

=============================
Update
While I agree in principle with Reviewer 1 that this paper has jarring flaws in writing and the rebuttal version does not adequately address it, I disagree that the writing warrants such a low score. I have seen worse papers with outrageous claims (e.g. try to claim significance with p=0.1) and I would not give those a 2. I would also disagree with R1 that there is no interesting result in this paper, because there is no prior work I know that even considers how distilled models generalize like their teacher.
If I were to grade this paper based on different aspects, the originality and significance would be both 9's, quality a 6 due to experiment issues and careless generalization, and clarity a 3-4 due to unclear motivation in the abstract/early intro and poor differentiation from prior work in terms of experiment design and analysis.
That said, the rebuttal did not change my mind that the writing probably will not be improved enough post-rebuttal, I would thus not be able to consider this a top paper despite the interesting observations.
review_title: Great paper with jarring flaws

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper shows that knowledge distillation from a (teacher) model A with an appropriate inductive bias to a (student) model B lacking it can lead to B generalizing better than if B was trained without knowledge distillation (but not as well as A), including out-of-distribution. The authors also show that the resulting learned representations inside B, as well as the shape of the training trajectories, are more like those of A (than those of B without knowledge distillation). 

This is not very surprising but is still interesting from the point of view of the understanding of the nature of inductive biases. We already knew that inductive biases (like translation invariance) can be transferred through examples (e.g. by generating data transformations such as translated images), so this paper extends that kind of idea to knowledge distillation to provide the targets for such examples.

Another nice contribution of the paper is the case study of the specific inductive biases of RNNs which transformers lack, decomposed into sequentiality, memory bottleneck and recursion. Not very surprising but the experiments confirm intuitions and expectations which is always useful.

One concern I have is 'so what?' and 'then what?'. Have the authors thought of possible way (in say, future work) to take advantage of that observation? It is not obvious, because if you already have a teacher model A with the right inductive biases for the given task, why would you care about training a student B which is going to be worse than A anyways? Just use A. In addition, unlike for the original motivation of knowledge distillation, we normally expect that B would have MORE capacity than A (because it needs to 'learn' the inductive biases, so one would expect it would not work to choose B much smaller than A, in the sense that the gain would be much smaller, and certainly not as good a model as using A).  We already knew that examples could transfer inductive biases, now we know that knowledge distillation can do it, but why would that be useful?

Experiments where B is much smaller than A would be interesting, because in that case, it might be worthwhile to do the knowledge distillation from a larger but better biased A. Also, the outcome of such experiments would not be apriori obvious (we would expect a gain vs the regular B, but would it be sufficiently interesting to be worth it?).

Another question I would have liked to be studied is about what happens out-of-distribution (OOD). The paper already shows the unsurprising  result that distilling into B from A helps somewhat OOD. It would also be interesting to explore whether taking inputs outside of the training distribution of A as distillation examples when training B would increase the robustness of B OOD.

Minor comments:

- fig 4: caption is insufficient to understand the figure
- the sec 3.2 sentence with 'almost closing the gap' is too strong and needs to be weakened (there is still a significant gap, with almost twice the error with B compared with A)
- the conclusion sentence with 'demonstrate having the right inductive bias can be crucial' should be reformulated, since this is not a new demonstration (and reading it without reading the rest of the paper may give that false impression)


review_title: Review of 'Transferring Inductive Biases through Knowledge Distillation'

Meta-review: This paper studies through empirical analysis an interesting problem: distilling the (strong) inductive bias of a teacher model to the student model (of weak inductive bias). The main claim/finding is that not only the "dark knowledge" in the logits can be transferred, but also the inductive bias (e.g. recurrence in RNN and translation invariance in CNN) can be transferred to make the student model stronger. This conclusion looks not very surprising but does contribute some new ideas to the fields of both deep learning and transfer learning.

The paper receives insightful but controversial reviews. Throughout reading the lengthy rebuttal and discussions, the AC, as a neutral referee of both sides, thought that while some expressions in the discussions seem a bit urgent and strained, both reviewers and authors managed to participate in the academic debate with a professional attitude that focus only on the technical issues. These discussions are very extensive and helpful for drawing a thorough understanding of this paper, and of the important research problem.

After the public interactions with the authors, a private discussion was performed between all reviewers and the AC, and among the four reviewers, one argued for rejection, one voted for rejection, one voted for acceptance, and one argued for acceptance. The AC believed that one of the reject votes lacks enough support in the comments and thus discarded it.
However, due to the wild disagreement between the reviewers, as well as between the reviewers and the authors, the AC read the paper carefully. The AC's main points are as follows:

- The research problem is interesting, and this paper appears to be the first work that studies the inductive bias transfer problem.

- The paper has made its endeavor to try to delve into this problem, through providing with extensive empirical results and analyses.

- The biggest weakness of this paper is the experimentation approach towards quantitatively studying the inductive bias: comparing the teacher and student models through the relational similarity between the penultimate-layer representation is simply not enough to justify that the inductive bias has been distilled/transferred.

Two reasons:
+ Due to the expressiveness of neural networks, it is not hard for the student to resemble the teacher's representations; in fact, this is a quite common result repeatedly used by researchers and practitioners, even when the student is only a smaller model. While the idea of distilling inductive bias is interesting, it simply cannot be sufficiently justified by the current experimentation design.
+ Inductive bias is something encoding our prior knowledge about the learning task and is often effective during the whole training procedure, which cannot be refreshed by the training data. However, albeit the distilled student model (transformer or MLP) resembles the representations of the teacher model (RNN or CNN), it is not certain whether the "distilled inductive bias" can linger in the student model if you further fine-tune the student model to downstream tasks. That is, it is highly possible that such "distilled inductive bias" of the student model will be refreshed by the future training data. In contrast, if we directly fine-tune the teacher model to downstream tasks, their inductive bias (recurrence of RNN or translation invariance of CNN) will be retained successfully in the fine-tuned model.
Basically, through this thinking, it is not clear whether the inductive bias has be distilled. If the distilled thing is refreshed out, it is probably not the inductive bias. More experimentation or a formal quantification of inductive bias is highly necessary here.

While Reviewer #1 was a bit skeptical in the comments and discussions (regarding which I had a private discussion with him/her), some of his/her comments are reasonable and should be well addressed before this paper could be accepted:
- Be rigorous in scientific writing. While the experiments with bias-variance tradeoff and calibration are interesting and relevant, the key concepts were used with less care. It is good to expand the authors' understanding of these concepts to make sure what they actually refer to.
- Try to provide sufficient elaboration when you try to claim something. It is true that for now, in our field, there are quite a few papers claiming something very big in the title or abstract, but simply cannot fulfill their story through rigorous or sound technical study. I suggest to tone down some of the key claims such that "inductive bias can be transferred" if they are not clearly provable.

Finally, AC believes this paper studies a very interesting problem that may draw wide attention, and the paper is acceptable in a future version if the above comments are well addressed. Since this is already a resubmission (as mentioned by Reviewer #1), I'd encourage the authors to focus on the technical parts of the comments and revise the paper substantially before submitting to yet another top venue.
Decision: Reject