MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT
Authors: Omkar Chakradhar Thawakar, Ashmal Vayani, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Timothy Baldwin, Eric P. Xing, Fahad Khan


Reviewer paper summaries: 1. The paper addresses the research question of building a high performance SLM (Small Language Model) which is suitable for on-device processing, has lower memory footprint, and is efficient at inference time.
2. The main contribution of the authors is to use a shared FFN design for all transformer blocks within the SLM (compared to having a dedicated FFN per transformer block). This helps reduce the overall number of trainable parameters while still retaining the representative capacity of the model.
3. Through a series of evaluations on nine benchmarks, the paper shows that MobiLlama achieved favorable results compared to existing state of the art models in the sub 1B parameter category.
Summary of strengths: 1. The authors present a high performance SLM called MobiLlama-0.5B which achieves a gain of 2.4% over the next best performing model (Pythia-410m), while retaining the same overall trainable number of parameters.
2. The paper also presents a detailed efficiency comparison with other smaller language models (their baseline model $large-base$ , Llama2-7B, Phi2-2.7B) and shows that their model is more efficient compared to these baselines. Also, they compare against other post-training sparsification schemes (Ashkboos et al. 2024) and show that despite having a lesser number of sliced parameters, MobiLlama achieves a higher average performance.
3. The authors present two high performing SLM variants - MobiLlama 0.5B and 0.8B - which is a step forward in on device privacy preserving LLMs which are fully transparent and efficient in terms of inference costs.
Summary of weaknesses: 1. In Section 4, Efficiency Comparison, it’s not clear to me how the baseline models were chosen. Most (all?) of the models have a larger number of parameters, which means they would take more time in the forward pass and thereby have lesser efficiency. I might have missed something here, it would be great if the authors could clarify their choice of models here.
2. In Section 4, Efficiency Comparison, the evaluation set of HellaSwag, PIQA, Arc_C and Winogrande is different from the ones used previously. I’m curious why the same set of evaluations was not used in Table 5 vs Table 4 for example. Having a standard set of evaluations makes it easier to do an apples to apples comparison, since some tasks might be inherently easier than others.
Comments, suggestions and typos: 1. In Section 4, Baseline Comparison, line 431, the authors mention that MobiLlama achieves an average score of 34.4 over the four benchmarks. However, Table 3 shows the average score to be 35.55
Overall assessment scores: 3