A Survey of Small Language Models
Authors: name: Chien Van Nguyen; name: Xuan Shen; name: Ryan Aponte; name: Yu Xia; name: Samyadeep Basu; name: Zhengmian Hu; name: Jian Chen; name: Mihir Parmar; name: Sasidhar Kunapuli; name: Joe Barrow; name: Junda Wu; name: Ashish Singh; name: Yu Wang; name: Jiuxiang Gu; name: Franck Dernoncourt; name: Nesreen K. Ahmed; name: Nedim Lipka; name: Ruiyi Zhang; name: Xiang Chen; name: Tong Yu; name: Sungchul Kim; name: Hanieh Deilamsalehy; name: Namyong Park; name: Mike Rimer; name: Zhehao Zhang; name: Huanrui Yang; name: Ryan A. Rossi; name: Thien Huu Nguyen
Published: 2024-10-25 23:52:28

Abstract: Small Language Models (SLMs) have become increasingly important due to their
efficiency and performance to perform various language tasks with minimal
computational resources, making them ideal for various settings including
on-device, mobile, edge devices, among many others. In this article, we present
a comprehensive survey on SLMs, focusing on their architectures, training
techniques, and model compression techniques. We propose a novel taxonomy for
categorizing the methods used to optimize SLMs, including model compression,
pruning, and quantization techniques. We summarize the benchmark datasets that
are useful for benchmarking SLMs along with the evaluation metrics commonly
used. Additionally, we highlight key open challenges that remain to be
addressed. Our survey aims to serve as a valuable resource for researchers and
practitioners interested in developing and deploying small yet efficient
language models.