Breaking ReLU Barrier: Generalized Mixture of Experts for Dense Pretrained Models
Authors: Jaeseong Lee, seung-won hwang, Wonpyo Park, Mingi Ji


Reviewer paper summaries: The paper addresses the research area of converting a pre-trained language model (PLM) to a sparse Mixture of Experts (MoE) models, specifically for cases where the activation function is not ReLU. Their method, called G-MoEfication, is applicable to arbitrary dense models, overcoming the limitations imposed by ReLU based activation sparsity assumptions. The authors discuss the predicament of zeroing out deactivated experts while simultaneously preserving dense activation information to ensure the converted model performs as well as the original PLM. Through a sequence of well designed experiments, the authors discuss the advantages of their strategy compared to traditional MoEfication.
Summary of strengths: 1. The authors present a well articulated and mathematically sound technique to convert an arbitrary PLM to a sparse model while retaining most of the performance of the original model.
2. The paper presents a set of thorough experiments to showcase the effectiveness of G-MoEfication compared to standard MoEfication, and also discuss the influence of different parts of their method such as the choice of representative value $r$ and disentangle the effect of additional fine-tuning from MoEfication.
Summary of weaknesses: 1. In the experiments section (Section 4), the authors discuss the datasets and number of examples they choose for training the MLP for calculating the representative value $r$. However, it is not clear why the authors chose a certain number of examples for calculating the value. Does increasing the number of examples we train on lead to a better estimate of $r$? Is there a way to estimate the variance of $r$ for a specific sample size? Answers to these questions would help users better utilize their methodology.
2. In RQ5 - The authors discuss the generalizability of their method on out of distribution datasets by presenting multPL-E Java evaluation numbers. However, it would be great if they could showcase this on language tasks, which are more nuanced as compared to Code benchmarks. These results would also inform users on how the quality of data they use for calculating the value of $r$ impacts downstream results.
Comments, suggestions and typos: N/A
Overall assessment scores: 3.5