Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a framework for knowledge distillation with smaller number of parameters.The authors proposed a new parameter sharing method that allows a greater model complexity for the student model. Another contribution is that a KD-specialized initialization method named Pretraining with Teacher’s Predictions can improve the student's performance. The author combined these two methods to improve the performance of the student model on existing tasks, which has surpassed the existing knowledge distillation baseline.

The SPS method is a very natural idea of extending the student model. It expands the model complexity of the student model and improves the representation ability of the model without increasing the number of parameters. The idea of PTP is an excellent initializing method. At the same time, the teacher model's generalization of knowledge is given to students through initialization, and then fine-tuned.

The experimental part of this article is also quite sufficient. The ablation experiment illustrates the effectiveness of the two steps for the optimization of results. 
review_title: A Method to Expand and Initialize Student Model of Knowledge Distillation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a parameter-efficient KD which consists of two main parts: Shuffled Parameter Sharing (SPS) and Pretraining with Teacher’s Predictions (PTP). This work explores some new framework like parameters-sharing and shuffling in KD and obtain promising results. 


Strengths:
1. The paper is well-written and easy to follow.
2. The experiment part explores the overall performance, the effects of SPS, and PTP separately, which is clear and makes readers easy to understand the effects of each part (SPS: step1+step2, PTP) in Pea-KD.  

Weaknesses:
1. When applying SPS, the number of layers in the student model is double than the normal case, even the parameters are the same as the counterpart, I guess the FLOPs will increase or be doubled than the original one. I think for a small student model, the number of parameters should not be the only metric but also consider the FLOPs. Why this paper doesn’t discuss this? This is my main concern.
2. From Table4, we can find that the shuffling in SPS has great effects. step1+step2 would have much greater improvement than only applying step1, how about only conduct shuffling without parameters sharing (only step2 by shuffling the original parameters without step1)?  On the other hand, what is the potential reason why shuffling can enrich the model capacity?


review_title: Review of AnonReviewer3

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a distillation method for BERT. The work is based on two-fold main ideas. First, as the student model is usually smaller in the number of parameters, the model capacity is limited. The authors propose to stack the layers that share parameters to counter this limitation. Second, the authors argue that the initialization of the student model is crucial, so they propose an pre-training strategy for boosting the student's performance.

Pros:
The idea of learning good initializations for the student model is interesting.


Cons:
This paper has several writing problems which need be carefully addressed before its publication.

- As all the experiments are conducted on GLUE and only BERT models are considered in the paper, the proposed method seems to be tailored for BERT. The authors should stress this point clearly and early in the paper (in the title, abstract or introduction). Otherwise, the authors should provide some experimental results on other models or tasks, e.g., the typical image recognition task in computer vision.

- In the section of PROPOSED METHODs (section 3.1), the motivation and the main idea of the paper are introduced again. As no any new information is provided here compared to the abstraction and the introduction sections, it seems very redundant.

- The introduction of SPS in Section 3.2 is quite confusing. What do Key  and Query parameters stand for?  I have no idea what the author is talking about here. Maybe it is because I have little background in NLP and BERT. However, even so the authors are still responsible for making the paper easy to follow for readers like me.


As for the motivation and the proposed method, my comments are as follows.

- The motivation and the proposed method are somewhat problematic. Firstly, the authors argue that small student with few parameters are limited in capacity, so they propose to stack repeating layers to enlarge the model capacity. However, stacking repeating layer will introduce much more computation cost, which violates the goal of distillation.

-  Secondly, the authors propose to pre-train the student model with the teacher predictions to initialize the student. However, it is odd to view this step as pre-training as it actually adopts the teacher predictions to train the student. It is actually doing distillation! The improvement  in performance may simply come from the more training epochs.

======================

post-rebuttal:

I have read all the comments from other reviewers and replies from the authors. The revised version partially addresses my concerns so I raise the score from 4 to 5. However, my concerns about the motivation of the work still exist, so I am still slightly leaning to reject this paper.

review_title: The paper is poorly written. The motivation and the proposed method are problematic.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper claims to present an improved form of knowledge distillation which tackles the following perceived weaknesses of existing kd systems:
- student's expressive power is less than original teacher model, and
- unclear how to initialize the student model weights in a principled way

The paper proposes two methods to improve kd:
- using stacked layers, with weight sharing between the weights, and keys and queries swapped in the upper layers
   - this is proposed because it means the effective student geometry matches that of the teacher, or at least is significantly larger than without the shared layers, whilst the number of unique parameters remains small
- getting the student to predict the confidence of the teacher on each example, rather than just the softmax output of the teacher
   - the confidence is classified as either 'strong' or 'weak'
   - the student must also predict whether the teacher gets the example correct or not (which we know, since this is for training set examples, for which we have ground truth)

Strong points of the paper:
- SPS sounds novel, but I felt it was only very weakly motivated
    - in addition it sounds to me like the inference time will be similar to the original model? I feel that reducing the number of parameters in the student is not the only goal: the student model should be fast to execute.
    - the paper does admittedly not claim to provide fast inference times as a goal; nevertheless I have a hard time imagining a BERT model running on an edge device with a limited amount of resources in a reasonable time, so I think that inference time is a critical metric which I feel this paper does not consider, or measure
- PTP sounds to me only weakly novel, since typically the student will be trained to predict the full softmax output of the teacher, whereas in PTP, the student must predict whether the highest value of the softmax is high or low. I feel that little insight or motivation is given as to why this auxiliary task was used, or is better than predicting the full softmax distribution. Nor was a rigorous comparison carried out to compare using just this pre-training approach with just the softmax-matching approach, where both using appropriately tuned hyper-parameters, I feel. There was an experiment in 4.4 where the PTP task is dropped, but I got the impression that this was after tuning the hyper-parameters to rely on PTP task? The hyper-parameters would I feel need to be retuned in the absence of the PTP task I think?

I mostly like the paper. I found it interesting. However, I have two main concerns:
- the approaches used in the paper feel to me only weakly motivated. Little insight is given into why the approaches were chosen, and why they should work. For example, flipping the keys and queries was not well justified I felt. Nor was the PTP task relative to traditional softmax matching.
- the paper avoids discussing inference times, and it's unclear to me whether a distillation approach that results in few parameters but long inference times would be useful in practice?
    - of course, the underlying techniques can still be interesting from a theoretical point of view

Whilst I was reading through the paper, I wrote down some more detailed reactions:

Introduction:

"stacking the layers that share parameters" => "stacking layers that share parameters" (the former form of language is using 'that share parameters' as a filter to choose from existing layers; the latter form implies that we create new layers, which we then stack, and which we configure to have shared parameters)

3.1 Overview

So, it looks like SPS means the number of unique parameters is kept low whilst the effective model complexity is unchanged compared to original BERT. How does this affect inference speed? It sounds to me like the inference speed under SPS will be unchanged from a BERT model of equivalent model complexity? Often, a use-case of distillation is to reduce inference time, and requirements in terms of power, eg for mobile devices, or production inference. Assuming that the inference time is unchanged from the original model of equal complexity, what is the target use-case of SPS/Pea-KD?

3.2 SPS

No insight or intuition is given as to why swapping query and key is likely to be a useful thing to do. I think it would be good to provide such insight and intuition. Like, you say that it increases the expressive power, but you don't justify why you feel this is true.

For the six-layer student, I couldnt quite undersatnd why it looks like the 6-layer student has 9 layers? Please consider clarifying this point:
- why do you call it '6 layer' if it has 9 effective layers? Perhaps consider renaming it to 9-layer student?
- why do you make the 6 layer model become 9 layers, and not 12 layers? ie, following the initial recipe detailed in section 3.2, of doubling all the layers.

3.3 PTP

I feel that the column for 'PTP label' in table 1 should be on the right hand side:
- the inputs should be I feel on the left, ie 'teachers prediction correct', and 'confidence > t',
- and then we can read off the 'output' of the table as the right-hand column
- (currently I find the table hard to read)

The pretraining task itself seems interesting to me.

How do you get the student to predict this result? Do you put a linear layer on the output from the student? Why do you use a classification task (4 classes) instead of using a regression task to predict the confidence? did you try a regression task to predict the confidence? Why use a classification into 4 classes, rather than two classifiers into two classes each?

4. Experiments:

For table 2, I wasn't really sure how these numbers compare to two key baselines:
- a simple bilstm with attention, without any pretraining, and
- a BERT-base model

So, I dug these out, and here are the peabert numbers from table 2, put into the context of these two baselines (which I feel are lower and upperbound really for what we'd expect to see from PeaBERT):

     BiLSTM+Attn
     (single-task training)   BERT-base   PeaBERT1  PeaBERT2  PeaBERT3
    RTE: 51.9                     66.4        53.0     64.1       64.5
    MRPC 68.5                     88.9        81.0     82.7        85.0
    SST2 85.9                     93.5        86.9     88.2       90.4
    QNLI 77.2                     90.5        78.8      86.0       87.0

I got the BiLSTM+Attn numbers from GLUE paper, and
BERT numbers from https://arxiv.org/pdf/1910.03176.pdf

I kind of feel that I might not be the only person who might want to see the PeaBERT numbers in the context of such lower and upperbound baselines? Maybe consider adding these baselinse into table 2?

Basically, my take away from this is that PeaBERT1 is barely better than the simple BiLSTM+Attn baseline, but PeaBERT3 approaches the score of a full BERT-base model?

I kind of think the sentence "For example, DistilBERT took approximately 90 hours with eight 16GB V100 GPUs while PeaBERT took a minimum of one minute (PeaBERT1 with RTE) to a maximum of one hour (PeaBERT3 with QNLI) using just two NVIDIA T4 GPUs" should be highlighted in a table somewhere somehow perhaps, rather than buried in text? Not sure if that's a good idea, just occurred to me though.

Question: why only show results on a subset of the GLUE tasks? eg patient-kd paper shows results on additionally:
- QQP
- MNLI-m
- MNLI-mm
(whilst also showing results for: SST-2, MRPC, QNLI and RTE, as here)

It is unclear from this whether you ran against all, and only showed the four tasks that show a benefit, or whether you simply didn't have time to run on all 7 tasks. Preference to show results for all 7 tasks that Patient-KD paper reports results for.

For ablation studies, I feel it would be interesting to see an ablation study that removes each of various losses in equation 3 in turn.

Importantly, none of the experiments mention inference time, which I feel is a key metric to report for distillation?

review_title: Interesting ideas, but each idea seems only weakly motivated to me; inference times are not mentioned or measured.

Meta-review: This paper got mixed reviews. One for acceptance, one for reject and two borderline. After the rebuttal, AR2 raises the review to borderline.  AR1 gives the highest recommendation but did not provide detailed supporting evidence. Other reviews provide comment on the strength and also share the concerns. Most of the concerns concentrate on the motivation (whether the proposed method is violating the objective of knowledge distillation) and the brought additional computation overhead. Also the scope of this paper was defined wider than the actual one. The authors only did experiments for BERT but did not consider and compare with existing KD method. Overall, AC read the paper and also has the similar concerns, the novelty is limited. the brought increase in inference time is violating the KD objective and the scope of this paper was not defined clearly. The authors should improve the submission in these aspects. At its current status, AC does not recommend acceptance. 
Decision: Reject