Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tackles with a problem that when distilling a small student model using a large pre-trained transformer, the “capacity gap” between the student and the teacher often hinders the teacher from transferring good performance to the student. The HomoDistill method proposed by this paper investigates to progressively prune the teacher model into a small student model, while in the same time distilling it using the original teacher model. Experiments show that the combination of pruning and distillation shows advantage on several NLU and QA tasks.
strength_and_weaknesses: strength:
* HomoDistill illustrates advantageous performance over other distillation methods like TinyBERT.
* The combination of pruning and distillation in task-agnostic setting seems novel. 

weakness:
* A comparison with other prune & distill methods such as (Xu et al., 2021; Xia et al., 2022) would make this submission more complete.
* Please also discuss the difference between HomoDistill and [1], which presents a similar method for vision tasks, where a teacher model is slimmed into a small student model while distilling it.

[1] Slimmable Neueral Networks, ICLR 2019.
clarity,_quality,_novelty_and_reproducibility: The wrting is clear, the whole paper is easy to understand. I see the experiments are all conducted on public benchmarks.
summary_of_the_review: In general, I think HomoDistill is a novel method that combines pruning with distillation in a task-agnostic setting for NLP tasks. The experiments on NLU and QA benchmarks show the advantages of HomoDistill over other distillation methods. However, the paper lacks experimental comparison with other methods that combine pruning and distillation. Appending such comparison would make this paper more solid.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper propose HomoDistil, a method which combines knowledge distillation and pruning to obtain efficient networks.
strength_and_weaknesses: The proposed method is simple yet effective, good performance are obtained according to experiments;

The paper is easy to follow;

However, the proposed method has limited novelty, it is based on some existing methods;

It would be nice if the authors can provide a comprehensive comparison of computation cost/time between the proposed method and previous methods;

Why results of model with 66M parameters is not reported in Table 3?

Please adjust the position of Table 2, Table 3 and the paragraph between the two tables.

Some related works are suggested to be compared.

[1]. Multi-Granularity Structural Knowledge Distillation for Language Model Compression. Chang Liu, Chongyang Tao, Jiazhan Feng, Dongyan Zhao

[2]. BERT Learns to Teach: Knowledge Distillation with Meta Learning. Wangchunshu Zhou, Canwen Xu, Julian McAuley.

[3]. LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding. Hao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, Xiaolong Li.

[4]. MixKD: Towards Efficient Distillation of Large-scale Language Models. Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, Lawrence Carin.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow, the novelty is limited. No code is provided, but the experiment seems to be reproducible.
summary_of_the_review: Although the proposed method is simple and seems effective, I lean towards rejection because of the limited novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a task-agnostic knowledge distillation method for NLP models based on iterative pruning combined with distillation. A novel distillation loss is proposed, that performs differentiable matching of the latent representations. The proposed distillation method is compared to a selection of baselines, and is shown to perform well.
strength_and_weaknesses: Strengths:
* An efficient distillation method is proposed that leverages both KD and pruning
* Experiments are realistic, and show the superiority of the proposed method compared to the SOTA baselines
* Latent space mapping/alignment is done elegantly via a learned weight matrix

Weaknesses:
* Reported medians over 5 runs, no standard deviations: how robust is the method?
* Table 2 could be presented visually
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written, with minor grammatical issues. The experiments are run five times, and medians are reported: I would have appreciated also seeing means and standard deviations. The method itself combines existing technique in a new setting, thus there is novelty, albeit somewhat limited. 

Will the code be made available should the paper be accepted? This would obviously help with reproducibility.
summary_of_the_review: Overall, I think this is a useful, practical paper, showing how existing methods and ideas can be combined to achieve higher quality solutions. Compressing huge models is an important task, since many resource-constrained settings exist that currently can’t leverage the modern NLP models. The authors should clarify if the latent representation alignment via a differentiable weight matrix is their idea: this is obviously an excellent way of discovering the alignment, I have not encountered it in the literature before. However, if the idea is borrowed, the sources must be stated clearly.

See below for additional suggested corrections:

Page 4: “the loss for continual pre-training the student model on the open-domain data” -> the loss for continual pre-training of the student model on the open-domain data
Page 4: “consists both the distillation…” -> consists of both…

Medians of 5 runs are reported - what about the corresponding standard deviations? Perhaps Table 2 can be represented visually (e.g. box-and-whisker plots)?

Regarding the matching rule: you propose a differentiable weighted matching scheme, i.e. the matching is determined via training. Is my understanding correct? Has similar matching been applied in the NLP domain before? Please indicate the sources.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies task-agnostic knowledge distillation of pretrained language models. It combines iterative pruning with layer-wise knowledge distillation. The main strengths of the paper are its empirical contribution, improvements over strong baselines, and extensive analyses. The main weakness of the paper is low novelty from a methodological standpoint.
justification_for_why_not_higher_score: Methodological novelty is low since this paper is in some sense just combining iterative pruning with knowledge distillation.
justification_for_why_not_lower_score: Thorough comparison against strong baselines indicate that this approach can meaningfully improve over existing work.
Decision: Accept: poster