MobileLLM: Optimizing Sub-billion Parameter Language Models for
  On-Device Use Cases
Authors: name: Zechun Liu; name: Changsheng Zhao; name: Forrest Iandola; name: Chen Lai; name: Yuandong Tian; name: Igor Fedorov; name: Yunyang Xiong; name: Ernie Chang; name: Yangyang Shi; name: Raghuraman Krishnamoorthi; name: Liangzhen Lai; name: Vikas Chandra
Published: 2024-02-22 18:58:55

Abstract: This paper addresses the growing need for efficient large language models
(LLMs) on mobile devices, driven by increasing cloud costs and latency
concerns. We focus on designing top-quality LLMs with fewer than a billion
parameters, a practical choice for mobile deployment. Contrary to prevailing
belief emphasizing the pivotal role of data and parameter quantity in
determining model quality, our investigation underscores the significance of
model architecture for sub-billion scale LLMs. Leveraging deep and thin
architectures, coupled with embedding sharing and grouped-query attention
mechanisms, we establish a strong baseline network denoted as MobileLLM, which
attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
state-of-the-art models. Additionally, we propose an immediate block-wise
weight sharing approach with no increase in model size and only marginal
latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
MobileLLM model family shows significant improvements compared to previous
sub-billion models on chat benchmarks, and demonstrates close correctness to
LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
for common on-device use cases.