Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: This paper tackles the problem of distilling a faster/smaller model from multiple expert models. It demonstrations good results in a inference speedup in addition to retaining most of the original expert performances.
strengths_and_weaknesses: Strengths:
- There is a clear identification of current issues suffered by MoE-based transformer models, and this paper shows good results that tackle these issues (lower inference time while retaining performance)
- Apart from minor grammar issues paper is well written with good figures and tables that help highlight the results and explain the approach well.
- Appendix appears to be well written and documented, so reproducibility should be possible (open-sourced code would be great though)

Weaknesses:
- The benefits metric could be explained a bit more clearly. It's not immediately clear what $s_{stu}$ might refer to.
- the input representation $x$ could be clarified. Is this the input from the datasets tested on? The word representation makes it sound that the original dataset inputs were processed/formatted into some other representation.
suggested_changes: Minor:
Table 2 description can be improved by explaining what the bolds mean (usually it means best result but because of the existence of the teacher this is not exactly true).
Same for table 1, although this one is more confusing since there are multiple bolds in one column across a few models.

Grammatical (not super important! but always helpful):
"Recent study in deep learning proposed mixture-of-experts (MoE), a deep neural
network with multiple experts." -> "A recent study in deep learning proposed mixture-of-experts (MoE), a deep neural
network with multiple experts." Also perhaps this is a typo? MoE is not a deep neural net, but a ML technique.

"Inspired by human education
model" -> "Inspired by the human education
model..."
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: The authors propose a novel task of knowledge integration to obtain a dense student model which is comparable to a sparse MoE.
strengths_and_weaknesses: Strengths:
- The problem is relevant, well formulated and clearly explained.
- The experiments are quite detailed covering both vision and language MoE.
- Data/hyperparameters are mentioned for reproducibility.

Weakness:
- The formatting and structure of the paper needs to be fixed.
- In a related point, although the paper is well written, it is a bit too dense for this track which has a 2 page limit for main text.
suggested_changes: Please refer to point 1 of the weakness above.
review_title: null

Meta-review: It's been pointed out to the PCs that this is a shortened version of a published work (https://arxiv.org/abs/2201.10890). Not enough evidence was given to support that this version is sufficiently different from the original (despite the difference in length), and the submission didn't make the fact clear (didn't cite or mention the other published work). We have hence come to a conclusion to give a "invite to revise" rating. 
Decision: Invite to present