Demystifying Small Language Models for Edge Deployment
Authors: Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu
Accepted at: acl25

Demystifying Small Language Models for Edge Deployment

Zhenyan Lu†1,2, Xiang Li†1, Dongqi Cai1,3, Rongjie Yi1,
Fangming Liu2, Wei Liu4, Jian Luan4, Xiwen Zhang5,
Nicholas D. Lane3,6, Mengwei Xu1,

1Beijing University of Posts and Telecommunications, 2Peng Cheng Laboratory,
3University of Cambridge, 4MiLM Plus, Xiaomi Inc., 5Helixon Research, 6Flower Labs
Correspondence: mwx@bupt.edu.cn

Abstract

Small language models (SLMs) have emerged as a promising solution for deploying resource-constrained devices, such as smartphones and Web of Things. This work presents the first comprehensive study of over 60 SLMs such as Microsoft Phi and Google Gemma that are publicly accessible. Our findings show that state-of-the-art SLMs outperform 7B models in general tasks, proving their practical viability. However, SLMs’ in-context learning capabilities remain limited, and their efficiency has significant optimization potential. We identify key SLM optimization opportunities, including dynamic task-specific routing, model-hardware co-design, and vocabulary/KV cache compression. Overall, we expect the work to reveal an all-sided landscape of SLMs, benefiting the research community across algorithm, model, system, and hardware levels.

1 Introduction

The evolution of language models is diverging. On one hand, in the pursuit of artificial general intelligence, increasingly large language models (LLM) have been born in datacenters that host hundreds of thousands of GPUs (Kaplan et al., 2020; Xu et al., 2024c). The aim of this path is to demonstrate that machines can solve the most challenging language tasks, with the ultimate goal of advancing human civilization by pushing the boundaries of science and technology. On the other hand, there is a growing focus on small language models (SLM), designed for resource-efficient and ubiquitous deployment on scenarios such as mobile devices and robotics. The vision behind SLMs is to democratize access to machine intelligence, making it both accessible and affordable to a wider range of users. This approach seeks to make intelligence ubiquitous and practical, available to anyone, anywhere, at any time – much like the human brain, which everyone possesses.

Both LLM and SLM are important in reshaping our daily lives, yet the latter receives significantly less attention in academia. There has been very few literature exploring SLM capabilities (Lepagnol et al., 2024; Schick and Schütze, 2020; Zhou et al., 2023) or their runtime cost on devices (Li et al., 2024b; Laskaridis et al.; Xu et al., 2024b), often with limited scale or depth. In industrial, however, SLMs have already been integrated into commercial off-the-shelf (COTS) devices on a massive scale (Yuan et al., 2023; Dubiel et al., 2024). For instance, almost every popular browser has built-in access to local SLM capability, including Google Chrome (chr, 2024), Microsoft Edge (edg, 2024) and Opera (ope, 2024). At system level, the latest Google/Samsung smartphones have built-in LLM services (Gemini Nano), allowing third-party mobile apps to leverage LLM capabilities through prompts and LoRA modules (goo, 2024).

This work presents the first in-depth, systematic investigation of SLMs, thoroughly discussing their capabilities and runtime performance. The scope of this work is limited to those language models with 100M–5B parameters in decoder-only transformer architecture for their wide deployment on edge devices, which covers the range from low-end WoT/wearable gadgets like smartwatches to high-end mobile devices such as smartphones and tablets. In total, we collected 68 popular SLMs released by 24 organizations, spanning from OPT (2022.05) to Llama3.2 (2024.09). The details of those models are shown in Table 1.

We then built up an end-to-end benchmark that comprehensively evaluates the model capabilities (mainly commonsense reasoning and in-context learning) through 10 widely used datasets, as well as their runtime costs (prefill and decode speed, memory footprint, etc.) on two real development edge boards. Through such investigation, we try to
answer the following crucial questions concerning SLMs: “Can SLMs catch up to LLMs in terms of intelligence?” “What datasets are more likely to produce a highly capable SLM?” “How different SLM architecture (e.g., depth, width, atten type) and the deployment environments (quantization algorithms, hardware type, etc) impact runtime performance?”

Based on the benchmarking results, we obtain valuable insights of SLMs. Here, we summarize a few of them. (1) SLMs capabilities are fast evolving, closing the gap with Llama-7B/8B series (\S2.3). (2) Not all SLMs benefit from in-context learning (\S2.3). (3) SLMs are typically trained on way more tokens than what Chinchilla recommends (“over trained”) (\S2.4). (4) Dataset quality is more crucial than dataset size (\S2.4). (5) Model architecture has non-trivial impacts on inference speed (\S3.1). (6) Quantization gains diminish in long context (\S3.2). We will explain how these insights create opportunities for the advancement of SLMs. Overall, our findings paint a promising picture of on-device ubiquitous SLMs, yet also highlight the challenges towards resource efficiency. Specifically, the findings show strong implications for multiple stakeholders of the SLM development pipeline. We expect the work to reveal an all-sided landscape of SLM and benefit the research community, including those working on the algorithm, model, system, and hardware levels.

In summary, we make the following contributions in this work.

• We exhaustively review 68 small language models released in recent years, benchmark their capability as well as on-device cost.

• To this end, we have developed the first mobile SLM evaluation suite, which handles downloading, quantizing, deploying, and measuring the performance of SLMs across heterogeneous edge devices. A leaderboard website is created as well to advance and facilitate the SLM research.

• Through such in-depth experimental investigation combined with comprehensive literature review, we obtain valuable insights from open-sourced SLMs, fostering future light-weight SLM research. We also summarize a few potential research topics in SLM.

2 SLM Overview and Benchmarking

2.1 Collecting Popular SLMs

<table>
  <tr>
    <th>Affiliation</th>
    <th>Model name</th>
    <th>Size</th>
    <th>Date</th>
    <th>Attention</th>
    <th>Activation</th>
    <th>Open training dataset</th>
    <th>Max-context window</th>
  </tr>
  <tr>
    <td rowspan="3">Meta</td>
    <td>OPT (Facebook, 2022-05)</td>
    <td>120M</td>
    <td>2022-05</td>
    <td>MHA</td>
    <td>ReLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Gulabana (Facebook, 2023-11)</td>
    <td>720M</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Llama 1.1 (Meta, 2024-09)</td>
    <td>7.1B</td>
    <td>2024-09</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>7.2k</td>
  </tr>
  <tr>
    <td rowspan="2">BigScience</td>
    <td>Bloom (BigScience, 2022-11a)</td>
    <td>1.1B</td>
    <td>2022-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Blooms (BigScience, 2022-11b)</td>
    <td>560M</td>
    <td>2022-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td rowspan="2">EleutherAI</td>
    <td>Python (EleutherAI, 2021-05)</td>
    <td>110M</td>
    <td>2021-05</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Pythia (EleutherAI, 2021-05)</td>
    <td>110M</td>
    <td>2021-05</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Cerebras</td>
    <td>Cerebras-GPT (Cerebras, 2023-05)</td>
    <td>256M</td>
    <td>2023-05</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td rowspan="2">Microsoft</td>
    <td>Phi-1 (Microsoft, 2023-06)</td>
    <td>1.7B</td>
    <td>2023-06</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Phi-1 (Microsoft, 2023-06)</td>
    <td>1.7B</td>
    <td>2023-06</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Phi-3.5 (Microsoft, 2024-01)</td>
    <td>1.7B</td>
    <td>2024-01</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>StabilityAI</td>
    <td>StableLM-3.5 (StabilityAI, 2023-11)</td>
    <td>1.7B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>StabilityAI</td>
    <td>StableLM-3.5 (StabilityAI, 2023-11)</td>
    <td>1.7B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>StabilityAI</td>
    <td>StableLM-3.5 (StabilityAI, 2023-11)</td>
    <td>1.7B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>StabilityAI</td>
    <td>StableLM-3.5 (StabilityAI, 2023-11)</td>
    <td>1.7B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>StabilityAI</td>
    <td>StableLM-3.5 (StabilityAI, 2023-11)</td>
    <td>1.7B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Alibaba</td>
    <td>Qwen 1.0 (Alibaba, 2024-01)</td>
    <td>1.5B</td>
    <td>2024-01</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Alibaba</td>
    <td>Qwen 2.0 (Alibaba, 2024-01)</td>
    <td>1.5B</td>
    <td>2024-01</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Alibaba</td>
    <td>Qwen 2.5 (Alibaba, 2024-09)</td>
    <td>1.5B</td>
    <td>2024-09</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>MBZUAI</td>
    <td>MiniLM (MBZUAI, 2023-02)</td>
    <td>100M</td>
    <td>2023-02</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>MBZUAI</td>
    <td>LaMini-GPT (MBZUAI, 2023-06)</td>
    <td>100M</td>
    <td>2023-06</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Google</td>
    <td>Artemis (Google, 2024-01)</td>
    <td>2B</td>
    <td>2024-01</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Google</td>
    <td>Artemis (Google, 2024-01)</td>
    <td>2B</td>
    <td>2024-01</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>OpenBBM</td>
    <td>MiniLM (OpenBBM, 2024-01)</td>
    <td>100M</td>
    <td>2024-01</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>OpenBBM</td>
    <td>MiniLM (OpenBBM, 2024-01)</td>
    <td>100M</td>
    <td>2024-01</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Apple</td>
    <td>OpenELM (Apple, 2024-04)</td>
    <td>450M</td>
    <td>2024-04</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>H2O</td>
    <td>dandelion (H2O.ai, 2024)</td>
    <td>4B</td>
    <td>2024-07</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>MetaAI</td>
    <td>FastText (MetaAI, 2024)</td>
    <td>7.5B</td>
    <td>2024-07</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>HuggingFace</td>
    <td>SmallLM (HuggingFace, 2024-07)</td>
    <td>560M</td>
    <td>2024-07</td>
    <td>QGA</td>
    <td>SLU</td>
    <td>✓</td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Legato</td>
    <td>Legato (Legato, 2024-07)</td>
    <td>2.5B</td>
    <td>2024-07</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Legato</td>
    <td>Legato (Legato, 2024-07)</td>
    <td>2.5B</td>
    <td>2024-07</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Legato</td>
    <td>Legato (Legato, 2024-07)</td>
    <td>2.5B</td>
    <td>2024-07</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Legato</td>
    <td>Legato (Legato, 2024-07)</td>
    <td>2.5B</td>
    <td>2024-07</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Xiaolonghua</td>
    <td>MiniLM (Xiaolonghua, 2023-11)</td>
    <td>1B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Xiaolonghua</td>
    <td>MiniLM (Xiaolonghua, 2023-11)</td>
    <td>1B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Xiaolonghua</td>
    <td>MiniLM (Xiaolonghua, 2023-11)</td>
    <td>1B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>Xiaolonghua</td>
    <td>MiniLM (Xiaolonghua, 2023-11)</td>
    <td>1B</td>
    <td>2023-11</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
  <tr>
    <td>AMD</td>
    <td>AMD Llama (AMD, 2023-07)</td>
    <td>120M</td>
    <td>2023-07</td>
    <td>MHA</td>
    <td>GEGLU</td>
    <td></td>
    <td>2k</td>
  </tr>
</table>

Table 1: Detailed configurations of SLMs benchmarked. We mainly use the base models in experiments, with exceptions of StableLM, Phi-3/3.5, and Dolly-v2 (marked with *) that only provide the instruct version.

SLMs have gained increasing attention from both the research and industrial communities. Notably, since the end of 2023, the number of SLM models has surged significantly. To understand their capability and cost, we comprehensively collect SLMs based on the following criteria: (1) Decoder-Only Transformer Architecture. (2) Open Weights to evaluate them freely. (3) Parameter Range between 100M and 5B parameters. (4) Base Model Focus: Only base pre-trained models are included, except for cases where only instruct versions exist (e.g., Microsoft Phi, StabilityAI StableLM).

These models, detailed in Table 1, encompass a broad spectrum from both industry and academia, differing in hyperparameters and training datasets. While they share similar architectures,
Figure 1: SLM capabilities over time. Performance is the average of all tasks (except for Math). The size of the circle is proportional to the model size. Red dashed lines show the state-of-the-art model at different time, indicating the trend that SLMs are getting better over time. Instruction-tuned models are highlighted with bold borders. LLaMA-7B series models are shown in horizontal blue dashed lines for comparison.

some datasets remain closed-source, leading to performance variations across tasks, as discussed in following sections. The details about our evaluation suite are shown in Appendix A.

2.2 Evaluation Datasets and Metrics

<table>
  <tr>
    <th>Habitat</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>TextSwag (Zellers et al., 2019)</td>
    <td>Text narrative completion.</td>
  </tr>
  <tr>
    <td>TruthfulQA (Lin et al., 2022)</td>
    <td>Assesses truthfulness.</td>
  </tr>
  <tr>
    <td>Winograd (Sakaguchi et al., 2020)</td>
    <td>Evaluates pronoun resolution.</td>
  </tr>
  <tr>
    <td>CommonsenseQA (Talmor et al., 2019)</td>
    <td>Commonsense multiple-choice questions.</td>
  </tr>
  <tr>
    <td>PIQA (Bisk et al., 2020)</td>
    <td>Physical commonsense reasoning.</td>
  </tr>
  <tr>
    <td>OpenBookQA (Clark et al., 2018)</td>
    <td>Open-book science questions.</td>
  </tr>
  <tr>
    <td>ARC (Clark et al., 2019)</td>
    <td>Novel and open-ended reasoning.</td>
  </tr>
  <tr>
    <td>ARC Easy (Clark et al., 2018)</td>
    <td>Simple science questions.</td>
  </tr>
  <tr>
    <td>ARC Challenge (Clark et al., 2018)</td>
    <td>Complex science questions.</td>
  </tr>
  <tr>
    <td>MMLU (Hendrycks et al., 2021)</td>
    <td>Problem-solving across disciplines.</td>
  </tr>
</table>

Table 2: Datasets used to evaluate SLM capabilities.

We used 10 datasets as described in Table 2 to evaluate the SLM performance. Following (Gao et al., 2024), we use accuracy as the primary evaluation metric. Accuracy measures the compute log-likelihood of generating a continuation from a context. The default shown accuracy is instructed by 5 shots, as it is the most common setting in the released model.

2.3 SLM Capabilities

Figure 1 illustrates the progress of small language models (SLMs) in commonsense reasoning and problem-solving. From March 2023 to September 2024, SLM performance improved by 12.5% on average, surpassing the 7.5% improvement of LLaMA models over the same period. Notably, SLMs have outpaced LLaMA-7B series (v1–3.1), highlighting their growing potential for on-device tasks.

The Phi family, trained on closed-source datasets, leads in performance, reaching 70% average accuracy, comparable to LLaMA 3.1 (7B parameters). As of September 2024, Phi-3.5-mini (2.7B) achieves the highest accuracy, rivaling LLaMA 3.1 (8B). This advantage likely stems from careful data engineering, instruction tuning, and potential dataset overfitting (Zhang et al., 2024a). These findings suggest that SLMs are rapidly closing the gap with LLMs in general reasoning tasks.

While larger models generally perform better, exceptions exist. Qwen2-1.5B outperforms many 3B-parameter SLMs, demonstrating that smaller models can excel in specific tasks.

The gap between open-source and closed-source SLMs is narrowing, driven by high-quality datasets like DCLM and FineWeb-Edu. Notably, SmoILM (64.2%) and DCLM-1B (63.8%) achieve strong performance in commonsense reasoning, highlighting the impact of high-quality training data.

Insight#1: We draw following key insights from the evolution of SLMs: (1) From March 2023 to September 2024, SLMs exhibited significant performance improvements across various language tasks, outpacing the improvements of the LLaMA-7B/8B series. Among them, the Phi family consistently achieves state-of-the-art performance across most tasks. (2) Smaller models like Qwen 2-1.5B can excel in specific tasks despite having fewer parameters. (3) SLMs trained on open-source datasets are closing the gap, thanks to high-quality datasets.

Opportunity#1 State-of-the-art SLMs have surpassed 7B models in general tasks, demonstrating that their capabilities for real-world deployment. Moreover, the exceptional performance of certain SLMs on specific tasks highlights their potential for task-specific model routing, where different models are dynamically assigned based on task requirements to optimize efficiency and accuracy.

In-context Learning Capabilities. We evaluate various SLMs and their 2B-parameter variants (or the closest available ones) on 8 tasks, including commonsense reasoning and problem-solving. In-context learning (ICL) generally improves performance, with five-shot ICL increasing zero-shot accuracy by 2.1% on average.

However, HellaSwag and PIQA show minimal
(a) SLM in-context capabilities across tasks.

(b) Average accuracy improvement after in-context learning across different SLM model size.

Figure 2: In-context learning performance with different tasks and models. Red line in (b) highlights the trend of the average score improvement with the increase of model size.

improvement, likely due to their lower complexity compared to datasets like ARC Challenge. LaMini is the only model with a performance drop of over 2%, possibly due to overfitting, where additional context introduces noise. Gemma 2 exhibits the most significant improvement, with accuracy increasing by 4.8%. Notably, ICL effectiveness improves with model size.

Insights#2: We draw two key insights from the in-context learning capacity of SLMs: (1) Most SLMs exhibit in-context learning ability, but its effectiveness varies by task. Significant gains are observed in ARC Challenge, while HellaSwag and PIQA show minimal benefits across all models. (2) Larger models generally perform better in in-context learning, while some smaller SLMs experience performance declines.

Opportunity#2: Due to the smaller parameter size of SLMs, the effectiveness of ICL is limited. Combining ICL with supervised fine-tuning (SFT) may yield better performance (Zhu et al., 2024).

2.4 Training Datasets

We investigate how the open-sourced pre-training datasets are used in training the SLMs. Overall, we find 12 such datasets being used and show them in Table 3.

<table>
  <tr>
    <th>Subscription</th>
    <th>Model</th>
    <th>Date</th>
    <th>Tokens (B)</th>
    <th>Datasets</th>
    <th>Acc.</th>
  </tr>
  <tr>
    <td rowspan="6">&lt;1B</td>
    <td>SmallLM-360M</td>
    <td>24.07</td>
    <td>600</td>
    <td>FineWeb-</td>
    <td>0.448</td>
  </tr>
  <tr>
    <td>OpenELM-450M</td>
    <td>24.04</td>
    <td>1500</td>
    <td>RefinedWeb, The Pile, RedPajama, Dolma</td>
    <td>0.417</td>
  </tr>
  <tr>
    <td>SmallLM-135M</td>
    <td>24.07</td>
    <td>600</td>
    <td>EdFi*, StarCoder, Cosmopolis*</td>
    <td>0.416</td>
  </tr>
  <tr>
    <td>MobliLlama-0.5B</td>
    <td>24.02</td>
    <td>1259</td>
    <td>RedPajama, RefinedWeb</td>
    <td>0.405</td>
  </tr>
  <tr>
    <td>OpenELM-270M</td>
    <td>24.04</td>
    <td>1500</td>
    <td>RefinedWeb, The Pile, RedPajama, Dolma</td>
    <td>0.393</td>
  </tr>
  <tr>
    <td>Python-410M</td>
    <td>23.03</td>
    <td>300</td>
    <td>The Pile</td>
    <td>0.388</td>
  </tr>
  <tr>
    <td rowspan="3">BLOOMZ-560M</td>
    <td>22.11</td>
    <td>350</td>
    <td>WuDacCorpora</td>
    <td>0.366</td>
  </tr>
  <tr>
    <td>BL0OMZ-660M</td>
    <td>22.11</td>
    <td>350</td>
    <td>WuDacCorpora</td>
    <td>0.363</td>
  </tr>
  <tr>
    <td>OPT-125M</td>
    <td>22.05</td>
    <td>180</td>
    <td>PuBERTA, The Pile, RedShift.io Reddit</td>
    <td>0.361</td>
  </tr>
  <tr>
    <td rowspan="3">Cerebra-GPT-590M</td>
    <td>23.03</td>
    <td>12</td>
    <td>The Pile</td>
    <td>0.358</td>
  </tr>
  <tr>
    <td>OPT-125M</td>
    <td>22.05</td>
    <td>180</td>
    <td>PuBERTA, The Pile, RedShift.io Reddit</td>
    <td>0.349</td>
  </tr>
  <tr>
    <td>Python-160M</td>
    <td>23.03</td>
    <td>300</td>
    <td>The Pile</td>
    <td>0.347</td>
  </tr>
  <tr>
    <td rowspan="2">Cerebra-GPT-111M</td>
    <td>23.03</td>
    <td>2</td>
    <td>The Pile</td>
    <td>0.330</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">1B–1.4B</td>
    <td>DCLM-1.4B</td>
    <td>24.08</td>
    <td>4300</td>
    <td>DCLM baseline*</td>
    <td>0.557</td>
  </tr>
  <tr>
    <td>OpenELM-1.1B</td>
    <td>24.04</td>
    <td>1500</td>
    <td>RefinedWeb, The Pile, RedPajama, Dolma</td>
    <td>0.463</td>
  </tr>
  <tr>
    <td rowspan="10">1.5B–2B</td>
    <td>TinyLlama-1.1B</td>
    <td>23.12</td>
    <td>3000</td>
    <td>RedPajama, StarCoder</td>
    <td>0.436</td>
  </tr>
  <tr>
    <td>MobliLlama-1B</td>
    <td>24.02</td>
    <td>1259</td>
    <td>RedPajama, RefinedWeb</td>
    <td>0.434</td>
  </tr>
  <tr>
    <td>MobliLlama-1.4B</td>
    <td>23.12</td>
    <td>1300</td>
    <td>RedPajama</td>
    <td>0.428</td>
  </tr>
  <tr>
    <td>Python-1.4B</td>
    <td>23.03</td>
    <td>300</td>
    <td>The Pile</td>
    <td>0.423</td>
  </tr>
  <tr>
    <td>OPT-1.4B</td>
    <td>22.05</td>
    <td>180</td>
    <td>PuBERTA, The Pile, RedShift.io Reddit</td>
    <td>0.413</td>
  </tr>
  <tr>
    <td>Python-1B</td>
    <td>23.03</td>
    <td>300</td>
    <td>The Pile</td>
    <td>0.406</td>
  </tr>
  <tr>
    <td>Bloomz-1.1B</td>
    <td>22.11</td>
    <td>350</td>
    <td>WuDacCorpora</td>
    <td>0.394</td>
  </tr>
  <tr>
    <td>Bloomz-1B1</td>
    <td>22.11</td>
    <td>350</td>
    <td>WuDacCorpora</td>
    <td>0.384</td>
  </tr>
  <tr>
    <td>Cerebra-GPT-1.3B</td>
    <td>23.03</td>
    <td>26</td>
    <td>The Pile</td>
    <td>0.383</td>
  </tr>
  <tr>
    <td>StableLM-2-zephyr-1.6B</td>
    <td>24.01</td>
    <td>2000</td>
    <td>RefinedWeb, RedPajama</td>
    <td>0.556</td>
  </tr>
  <tr>
    <td>SmallLM-1.7B</td>
    <td>24.07</td>
    <td>1000</td>
    <td>FineWeb-, Edu*, StarCoder, Cosmopolis*</td>
    <td>0.503</td>
  </tr>
  <tr>
    <td rowspan="3">2.5B–3B</td>
    <td>StableLM-3zephyr-3B</td>
    <td>23.11</td>
    <td>400</td>
    <td>RefinedWeb, RedPajama</td>
    <td>0.582</td>
  </tr>
  <tr>
    <td>Python-2.8B</td>
    <td>23.03</td>
    <td>300</td>
    <td>The Pile</td>
    <td>0.448</td>
  </tr>
  <tr>
    <td>OPT-2.7B</td>
    <td>22.05</td>
    <td>180</td>
    <td>PuBERTA, The Pile, RedShift.io Reddit</td>
    <td>0.439</td>
  </tr>
  <tr>
    <td>Cerebra-GPT-2.7B</td>
    <td>23.03</td>
    <td>53</td>
    <td>The Pile</td>
    <td>0.405</td>
  </tr>
</table>

Table 3: Classify according to the model parameter quantity and sort in descending order according to average normalized accuracy. Acc(Avg) is the average of the accuracies of the two types of tasks, Commonsense reasoning/understanding and Problem solving. a indicates that this dataset is generated by LLM. b indicates that this dataset has been filtered by LLM.

Comparing the quality of pre-training datasets. We analyzed the quality of open-source pre-training datasets by evaluating SLM performance across models trained on them. SLMs from the past three years were grouped by parameter size (&lt;0.5B, 1B, 2B, and 3B) and ranked by average accuracy on Commonsense Reasoning/Understanding and Problem Solving tasks (Table 3). The results highlight DCLM and FineWeb-Edu as the top-performing datasets, both employing model-based data filtering. Additionally, many pre-training datasets, including StarCoder, contain coding data, despite SLMs on edge devices not prioritizing coding tasks. This inclusion is likely driven by the belief that coding data enhances reasoning ability (Zhang et al., 2024b).

The number of training tokens vs. the size of model parameters. The Chinchilla law (Hoffmann et al., 2022) suggests an optimal parameter-to-token ratio of 1:20 (e.g., a 1B model with 20B tokens). We analyzed SLMs under 4B parameters from 2022 to 2024, as shown in Figure 3(a) and found that recent models use significantly more to-
(a) The relationship between tokens and parameters. (b) The influence of training tokens on accuracy

Figure 3: The relationship between the number of training tokens, the number of model parameters, and the model accuracy. Here, the “accuracy” is averaged across all benchmarks in Table 3. (a) The relationship between the number of training tokens and model parameters size. According to scaling law (Chinchilla), that SLMs are often over-trained for better performance at deployment stage. (b) The influence of the number of training tokens on the model accuracy.

kens (typically over 1.5T) than this guideline (Figure 3(a)). Typically, larger models are trained on more tokens , but some smaller SLMs exceed larger ones (e.g., Qwen 2-0.5B with 12T tokens vs. Qwen 2-1.5B with 7T tokens), indicating over-training. This strategy aims to enhance SLM performance for resource-constrained deployment by increasing training-time FLOPs. However, over-training can lead to performance saturation (Godey et al., 2024).

The amount of training tokens vs. model accuracy. Figure 3(b) shows the relationship between the number of training tokens and the accuracy of the model. In general, there is a positive correlation between the two metrics, especially for those with less than 700B tokens. However, the correlation is weak, since the data quality often outweighs the impacts of more training tokens, especially when the training tokens exceed 1T.

Insights#3: We make two key observations in SLM training. (1) Data quality plays a crucial role in SLM capability, often outweighing data quantity and model architecture. A key trend in dataset research is model-based filtering, leading to state-of-the-art open-source pre-training datasets: FineWeb-Edu (1.3T/5.4T) (Penedo et al., 2024) and DCLM-baseline (4T) (Li et al., 2024a). (2) Recent SLMs are trained over large amount of tokens (typically >1.5T), disregarding their parameter sizes. In some cases, smaller SLMs are trained over even more data (e.g., Qwen2-0.5B at 12T tokens but Qwen2-1.5B at 7T tokens).

Opportunity#3 SLMs trained on model-based filtering datasets achieve competitive performance with those using closed datasets, significantly improving research reproducibility. Additionally, over-training can cause saturation and performance degradation. Defining an edge-optimized Chinchilla law is essential to ensure that additional tokens contribute to performance gains rather than diminishing returns.

Figure 4: Latency and memory overview.

3 SLM Runtime Cost

Setup In this section, we measure models of different sizes on robotic platform (Jetson Orin) and smartphone. We first analyze the latency and memory usage of models with different parameter sizes. Next, we assess the impact of quantization methods and hardware on model latency. Finally, we break down the latency and memory usage to identify the key factors influencing these metrics across various parts of the model. To eliminate the impact of inference engine implementations, we evaluated 20 models supported by llama.cpp, a widely recognized open-source inference engine.

We set a standard prompt length of 50 and a token generation length of 50 unless specified otherwise. To measure larger models, we applied 4-bit
<table>
  <tr>
    <th>Device Name</th>
    <th>Specifications</th>
    <th>Release Time</th>
  </tr>
  <tr>
    <td>Jetson Orin NX</td>
    <td>1024-core, 16G DRAM</td>
    <td>Feb. 2023</td>
  </tr>
  <tr>
    <td>Pixel 7Pro</td>
    <td>GoogleTensor G2, 12G RAM</td>
    <td>Oct. 2022</td>
  </tr>
  <tr>
    <td>Xiaomi 12S</td>
    <td>Snapdragon 8Gen1+, 12G RAM</td>
    <td>Jul. 2022</td>
  </tr>
  <tr>
    <td>MEIZU 18Pro</td>
    <td>Snapdragon 888, 8G RAM</td>
    <td>Mar. 2021</td>
  </tr>
</table>

Table 4: Testing devices.

quantization to all models before conducting experiments in all sections except § 3.2.1.

3.1 Cost Overview

3.1.1 Inference Latency

Figure 4 presents inference latency, including first token time and per-token decode latency, for models ranging from 0.1B to 3B parameters. Latency scales with model size across three categories: 0.1-1B, 1-2B, and 2-3B. For models of similar size but different architectures, first token time varies significantly. For example, Qwen2-0.5B’s first token time is 1.46x that of Qwen1.5-0.5B and comparable to OpenELM-1.1B, which has 2.18x the parameters. Qwen2’s architecture shares the embedding layer and LM head, allocating more parameters to attention and FFN, increasing computational cost. Notably, Pythia-1.4B has higher latency than SmoILM-1.7B, Qwen2-1.5B, and Qwen-1.8B despite being smaller. Phi-2 also exhibits 1.11x the latency of OpenELM-3B, a larger model. The prefill stage dominates on-device LLM inference due to long-context processing for personalization on edge (Xu et al., 2024a).

Decode-stage latency generally follows a linear trend with model size and is primarily memory-bound, unlike the compute-bound prefill stage. Qwen2-0.5B and Qwen1.5-0.5B show similar decode latency, while Pythia-1.4B has lower latency than larger models. Among 2-3B models, Gemma-2B, Phi-2, and OpenELM-3B show a positive correlation between latency and model size. Architectural differences impact compute-bound stages more significantly, with wider, shallower models benefiting from higher parallelism.

3.1.2 Memory Footprint

Figure 4 evaluates memory footprint using llama.cpp on Jetson for models ranging from 0.1B to 3B parameters, with memory usage between 275MB and 2456MB. Since llama.cpp allocates KV cache and compute buffer based on maximum context length, we standardized it to 2048 across all models. Memory usage generally scales linearly with model size, but some models deviate. Gemma-2B, with a vocabulary of 256,000, and Bloom-560M/Bloom-1B1, with 250,880, consume more memory than expected due to their large vocabularies. In contrast, OpenELM models use less memory than similarly sized models, benefiting from a smaller 32,000-token vocabulary (compared to the typical 50,000) and GQA instead of MHA, reducing KV cache requirements. The impact of vocabulary size on memory usage is detailed in § 3.3.2.

![First token time vs. model size and quantization methods](page_1012_120_393_246.png)
(a) First token time

![Decode latency per token vs. model size and quantization methods](page_1405_120_393_246.png)
(b) Decode latency per token

Figure 5: The relationship between the latency and quantization methods

Insights#4: We draw following insights regarding the runtime cost of SLMs on devices.
(1) Model architecture has a greater impact on inference latency than model size, especially for smaller models (<1B). The correlation is likely hardware-dependent. (2) The impacts of model architecture on inference speed is more significant at prefill stage than decode stage because the compute bound of prefill stage. (3) Memory footprint scales with model size, but vocabulary size has a disproportionate impact.

Opportunity#4: SLM architectures should align with hardware design, optimizing vocabulary size, FFN width, and layer depth for efficiency. Given different bottlenecks in prefill and decode, cloud systems adopt PD-separated clusters, while edge devices should leverage hardware heterogeneity, using NPU for prefill and CPUs for decode.

3.2 Impact of Quantization and Hardware

3.2.1 Impact of Quantization

On Jetson and similar mobile devices without low-bit hardware support, quantization improves efficiency by reducing memory access overhead. On server GPUs, it lowers inference latency through higher Tensor Core throughput for int8 operations, lower memory usage leading to higher batch size, and lower memory access overhead.

We evaluated five quantization methods for Phi-1.5 (Figure 5). Qn_K (and Qn_K_M) apply n-bit
quantization using the \( k \)-quants method, with Qn_0 denoting symmetric quantization. In the prefill stage, quantization reduces latency by at least 25% for short prompts, but the benefit diminishes with longer inputs. At a prompt length of 50, even the best-performing Q4_K_M achieves only a 13% reduction. This is because weights are shared across tokens, diluting the per-token benefit as prompt length increases. In the decode stage, quantization provides more consistent improvements, reducing latency by 17% to 75%, as weights are accessed per token, benefiting memory efficiency. Among methods, Q4_K_M consistently outperforms others, reducing latency by an average of 50%. In contrast, Q6_K and Q3_K become ineffective for long prompts, with latency matching or exceeding FP16. The inferior performance is due to irregular bit-widths, leading to higher overhead from alignment and padding.

Insights#5: We draw following insights regarding the quantization technique on SLM deployment. (1) Quantization is more effective in the bandwidth-bound decode stage than in the compute-bound prefill stage, especially when prompt length increasing. (2) Regular quantization precision enhances efficiency by avoiding extra hardware overhead.
Opportunity#5: Reducing memory access through quantization is not enough to significantly lower latency in edge deployments. Hardware designed for low-bit computation are essential.

3.2.2 Impact of Hardware
We tested Bloom-1B1 on two edge devices: Jetson Orin NX 16GB (GPU) and Meizu 18 Pro (CPU). The GPU is 40× faster than the CPU in the prefill stage but only 1.84× faster in decode. Prefill benefits from high parallelism, leveraging the GPU’s numerous computing units, while decode is sequential, limiting GPU efficiency. In the prefill stage, first token time increases linearly with prompt length, with Jetson’s advantage expanding. In decode, latency per token rises as more tokens are generated. On Meizu, latency spikes from 1 to 10 tokens due to thermal throttling, then stabilizes at high latency. Jetson, with better cooling, fluctuates only after 30 tokens. We tested Qwen1.5-1.8B on three smartphones with 60s intervals to reduce power effects. Latency scales linearly with token count. The Xiaomi 12S had the lowest latency, showcasing the efficiency of Snapdragon 8 Gen 1+. The Pixel 7 Pro followed, while the Meizu 18 Pro had the highest latency due to its older Snapdragon 888 and lower memory configuration.

Insights#6: We draw following insights of impacts of hardware. (1) GPU has greater advantage in the prefill stage. (2) Jetson maintains consistent latency due to its simpler hardware structure and better heat dissipation, whereas smartphones experience higher thermal fluctuations during long inference tasks. (3) The development of System on a Chip (SoC) generations effectively improves inference efficiency.
Opportunity#6: For smartphones, fully utilizing the heterogeneous computing power of the SoC (e.g., GPU, NPU) can significantly improve prefill efficiency. Additionally, power consumption from continuous requests or long-context processing remains a major challenge.

3.3 Latency and Memory Breakdown

3.3.1 Latency Breakdown
Figure 7 presents a breakdown analysis of Qwen 2-0.5B and Qwen 1.5-0.5B, two models of similar size but different latencies. We measured the time distribution across Embedding, Attention, FFN, and LM Head.

In the prefill stage, both models are dominated by Attention and FFN layers. In Qwen 1.5, Attention accounts for a slightly higher proportion of latency
than FFN, whereas in Qwen 2, FFN contributes significantly more due to its wider FFN layer. During the decode stage, Attention latency increases in Qwen 1.5, likely due to KV Cache growth, while FFN remains the dominant bottleneck in Qwen 2. At the operator level, mul_mat_vec_q (matrix-vector multiplication) accounts for over 80% of total latency in both prefill and decode stages. This proportion is even higher in Qwen 2-0.5B due to its wider FFN layer, further increasing computation time.

3.3.2 Memory Breakdown
As shown in Figure 9a, vocabulary size significantly influences memory consumption beyond model size. Larger vocabularies increase compute buffer memory due to the logits, sized as batch_size * sequence_length * vocabulary_size. For instance, Bloom-560M (vocabulary: 250,880) requires 492MB memory, 3.5× more than OpenELM-1.1B (vocabulary: 32,000), while Bloom-1B1 requiring 496MB memory exceeds Qwen2-1.5B (vocabulary: 151,936) by 1.6x. Models using GQA have reduced KV cache size compared to MHA. OpenELM-3B is 3.9× smaller than StableLM-zephyr-3B. At long context lengths, compute buffer and KV cache dominate memory usage. For Qwen2 series (context length: 131,072), they account for 85%, while for Qwen1.5 (context length: 32,768), they make up 87%.

Insights#7: We have following insights regarding the breakdown of SLM runtime cost. (1) Mul_mat_vec (matrix by vector multipli-
cation) is the most time-consuming operations of SLM, which constitute more than 70% end-to-end inference time. (2) Vocabulary size and Context length is crucial for model runtime memory usage.

Opportunity#7: SLMs are increasingly expanding vocabulary size to enhance performance. However, larger vocabulary increases inference latency and memory usage, necessitating compression strategies that preserve model capability. Similarly, as long-context support becomes a key trend, KV cache compression and quantization are crucial for SLMs.

Figure 8: On-device inference latency Breakdown.

(a) Layer granularity        (b) Op granularity

Figure 9: Memory Breakdown.

(a) 2048 context window.    (b) Max context window.

4 Related Work

Benchmarking SLM capability. Several public leaderboards evaluate the capabilities of LLMs, such as Open LLM Leaderboard (ope) support by Hugging Face clusters, FlagEval (fla). Some datasets have released their own leaderboards, such as SuperCLUE (Xu et al., 2023), C-Eval (cev), and MMLU (mml). These leaderboards include limited SLMs and lack a rich variety of datasets. MobileAIBench (Murthy et al., 2024) and MELTing point (Laskaridis et al., 2024) also evaluate some LLMs on device. Compared to them, we are the first dive into the SLM capability through experiments on a large number of representative SLMs.

Benchmarking SLM runtime cost. Currently, some studies have measured the inference throughput and power consumption of LLMs on various hardware devices. MELTing point focuses on the throughput and energy consumption across different hardware. MELODI (Husom et al., 2024) also proposes a framework that focuses on energy consumption of LLMs. Using its dataset, the study explores how prompt attributes, such as length and complexity, correlate with energy expenditure. Additionally, MobileAIBench evaluates the runtime cost of 3 models under 3 billion parameters after 4-bit quantization on an iPhone 14. However, these studies have measured only a limited number of SLMs, and miss crucial observations such as the influence of model architecture on runtime costs.

5 Conclusions

In this work, we conduct a comprehensive study on the capabilities and performance of small language models (100M–5B parameters). We evaluate most, if not all, open-source SLMs and analyze their re-
sults, drawing key insights to guide future research. These insights provide a clear understanding of SLM strengths and limitations, identifying areas for architectural improvements and deployment optimizations.

6 Limitations

For SLM capability evaluation, we selected 10 commonsense reasoning and problem-solving datasets, excluding math datasets due to the performance gap between SLMs and larger models in mathematical reasoning. For cost analysis, to ensure consistency and eliminate inference engine variations, we evaluated 20 models supported by llama.cpp, excluding those not compatible with it.

Acknowledgments

We thank the anonymous reviewers for their feedback on this work. This work was supported in part by the Major Key Project of PCL under Grant PCL2024A06 and PCL2022A05, in part by the Shenzhen Science and Technology Program under Grant RCJC20231211085918010, and in part by Xiaomi Open-Competition Research Program.

References

FlagEval - leaderboard.

Leaderboard | C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models.

MMLU Benchmark (Multi-task Language Understanding) | Papers With Code.

Open LLM Leaderboard 2 - a Hugging Face Space by open-llm-leaderboard.

2024. Copilot in microsoft edge. https://support.microsoft.com/en-us/topic/copilot-in-microsoft-edge-3fe6c1d4-9bd8-4492-a063-2cc6a5d01fef.

2024. Google ai edge sdk for gemini nano. https://developer.android.com/ai/aicore.

2024. Opera becomes the first major browser with built-in access to local ai models. https://press.opera.com/2024/04/03/ai-feature-drops-local-1lms/.

2024. Unlocking 7b+ language models in your browser: A deep dive with google ai edge’s mediapipe. https://research.google/blog/unlocking-7b-language-models-in-your-browser-a-deep-dive-with-google-ai-edges-mediapipe/.

Alibaba. 2023.11. Qwen 1. https://huggingface.co/alibaba/Qwen-1.

Alibaba. 2024.02a. Qwen 1.5. https://huggingface.co/alibaba/Qwen-1.5.

Alibaba. 2024.02b. Qwen 2. https://huggingface.co/alibaba/Qwen-2.

Alibaba. 2024.09. Qwen 2.5. https://qwenlm.github.io/blog/qwen2.5/.

AllenAI. 2024.04. allennai/olmo-1b-hf. https://huggingface.co/allennai/OLMo-1B-hf.

AMD. 2024.08. Llama. https://huggingface.co/amd/AMD-Llama-135m.

Apple. 2024.04. Openelm. https://huggingface.co/apple/OpenELM.

Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830–839.

Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. Cosmopedia.

BigScience. 2022.11a. bigscience/bloom-560m. https://huggingface.co/bigscience/bloom-560m.

BigScience. 2022.11b. bigscience/bloomz-1bl. https://huggingface.co/bigscience/bloomz-1bl.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence.

Cerebras. 2023.03. cerebras/cerebras-gpt-111m. https://huggingface.co/cerebras/Cerebras-GPT-111M.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. In arXiv preprint arXiv:1803.05457.

Together Computer. 2023. Redpajama: an open dataset for training large language models.
DataBricks. 2023.04. databricks/dolly-v2-3b. https://huggingface.co/databricks/dolly-v2-3b.

Mateusz Dubiel, Yasmine Barghouiti, Kristina Kudryavtseva, and Luis A Leiva. 2024. On-device query intent prediction with lightweight llms to support ubiquitous conversations. Scientific Reports, 14(1):12731.

EleutherAI. 2023.03. Eleutherai/pythia-410m. https://huggingface.co/EleutherAI/pythia-410m.

Facebook. 2022.05. facebook/opt-125m. https://huggingface.co/facebook/opt-125m.

Facebook. 2022.11. facebook/galactica-125m. https://huggingface.co/facebook/galactica-125m.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach', Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Nathan Godey, Éric de la Clergerie, and Benoît Sagot. 2024. Why do small language models underperform? studying language model saturation via the softmax bottleneck. arXiv preprint arXiv:2404.07647.

Google. 2024.02. Gemma. https://huggingface.co/google/Gemma.

Google. 2024.04. recurrentgemma. https://huggingface.co/google/recurrentGemma.

Google. 2024.07. Gemma-2. https://huggingface.co/google/Gemma-2.

H2O.ai. 2024. h2o-danube3-4b-base. https://huggingface.co/h2oai/h2o-danube3-4b-base.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

HuggingFace. 2024.07. Smollm. https://huggingface.co/huggingface/SmollM.

Erik Johannes Husom, Arda Goknil, Lwin Khin Shar, and Sagar Sen. 2024. The Price of Prompting: Profiling Energy Use in Large Language Models Inference. arXiv preprint. ArXiv:2407.16893 [cs].

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.

Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. Mobile and edge evaluation of large language models. In Workshop on Efficient Systems for Foundation Models II@ ICML2024.

Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. 2024. MELTING point: Mobile Evaluation of Language Transformers. arXiv preprint. ArXiv:2403.12844 [cs].

Pierre Lepagnol, Thomas Gerald, Sahar Ghannay, Christophe Servan, and Sophie Rosset. 2024. Small language models are good too: An empirical study of zero-shot classification. arXiv preprint arXiv:2404.11122.

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. 2024a. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Lucchioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jemite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023. Starcoder: may the source be with you!

Xiang Li, Zhenyan Lu, Dongqi Cai, Xiao Ma, and Mengwei Xu. 2024b. Large language models on mobile devices: Measurements, analysis, and insights. In Proceedings of the Workshop on Edge and Mobile Foundation Models, pages 1–6.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.

Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

M.A.P. 2024.04. Ct-llm. https://huggingface.co/m-a-p/CT-LLM-Base.

MBZUAI. 2023.04. Mbzui/lanini-gpt-774m. https://huggingface.co/MBZUAI/LaMini-GPT-774M.

MBZUAI. 2024.02. Mobillama. https://huggingface.co/mbzui/Mobillama.

Meituan. 2023.12. Mobilellama. https://huggingface.co/meituan/MobileLLMA.

Meta. 2024.09. meta-llama/llama-3.2-3b. https://huggingface.co/meta-llama/Llama-3.2-3B.

Microsoft. 2023.09a. microsoft/phi-1. https://huggingface.co/microsoft/phi-1.

Microsoft. 2023.09b. microsoft/phi-1_5. https://huggingface.co/microsoft/phi-1_5.

Microsoft. 2023.12. microsoft/phi-2. https://huggingface.co/microsoft/phi-2.

Microsoft. 2024.04. microsoft/phi-3-mini. https://huggingface.co/microsoft/phi-3-mini.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

Rithesh Murthy, Liangwei Yang, Juntao Tan, Tulika Manoj Awalgonkar, Yilun Zhou, Shelby Heinecke, Sachin Desai, Jason Wu, Ran Xu, Sarah Tan, Jianguo Zhang, Zhiwei Liu, Shirley Kokane, Zuxin Liu, Ming Zhu, Huan Wang, Caiming Xiong, and Silvio Savarese. 2024. MobileAI Bench: Benchmarking LLMs and LMMs for On-Device Use Cases. arXiv preprint. ArXiv:2406.10290 [cs].

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. 2023. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint arXiv:2309.09400.

Nvidia. 2024.07. Minitrion. https://huggingface.co/nvidia/Minitron-4B-Base.

OpenBMB. 2024.04. Minicpm. https://huggingface.co/openbmb/MiniCPM.

OpenBMB. 2024.09. Minicpm3. https://huggingface.co/openbmb/MiniCPM3.

Guilherme Penedo, Hynek Kydliček, Anton Lozhkov, Margaret Mitchell, Čolin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.

Princeton. 2023.11. Sheared-llama. https://huggingface.co/princeton-nlp/Sheared-LLaMA-1.3B.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence.

Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muenninghoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint.

StabilityAI. 2023.11. stabilityai/stablelm-zephyr-3b. https://huggingface.co/stabilityai/stablelm-zephyr-3b.

StabilityAI. 2024.01. stabilityai/stablelm-2-zephyr*. https://huggingface.co/stabilityai/stablelm-2-zephyr.

Alon Talnorn, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

TensorOpera. 2024. Fox-1-1.6b. https://huggingface.co/tensoropera/Fox-1-1.6B.

Toyota. 2024.08. Delm. https://huggingface.co/TRI-ML/DCLM-1B.

Unknown. 2023.12. Tinyllama. https://huggingface.co/tinyllama.
Xiaohongshu. 2023.11. Minima. https://huggingface.co/GeneZC/MiniMA-3B.

Xiaohongshu. 2024.07. Minima2. https://huggingface.co/GeneZC/MiniMA-2-1B.

Weikai Xie, Li Zhang, Shihe Wang, Rongjie Yi, and Mengwei Xu. 2024. Droidcall: A dataset for llm-powered android intent invocation. Preprint, arXiv:2412.00402.

Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu. 2024a. Empowering 1000 tokens/second on-device llm prefilling with mllm-npu. arXiv preprint arXiv:2407.05858.

Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. 2024b. On-device language models: A comprehensive review. arXiv preprint arXiv:2409.00088.

Liang Xu, Anqi Li, Lei Zhu, Hang Xue, Changtai Zhu, Kangkang Zhao, Haonan He, Xuanwei Zhang, Qiyue Kang, and Zhenzhong Lan. 2023. Superclue: A comprehensive chinese large language model benchmark. Preprint, arXiv:2307.15020.

Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. 2024c. A survey of resource-efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092.

Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, et al. 2023. Mobile foundation model as firmware. arXiv preprint arXiv:2308.14363.

Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. 2021. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. AI Open, 2:65–68.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024a. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332.

Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. 2024b. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. arXiv preprint arXiv:2405.20535.

![Framework diagram showing A800 server, Jetson Orin NX, and smartphones connected via SSH and ADB, with cost results and capability evaluation tasks](page_1012_120_420_312.png)

Figure 10: Framework.

Zhengping Zhou, Lezhi Li, Xinxi Chen, and Andy Li. 2023. Mini-giants:" small" language models and open source win-win. arXiv preprint arXiv:2307.08189.

Junyi Zhu, Shuochen Liu, Yu Yu, Bo Tang, Yibo Yan, Zhiyu Li, Feiyu Xiong, Tong Xu, and Matthew B. Blaschko. 2024. Fastmem: Fast memorization of prompt improves context awareness of large language models. Preprint, arXiv:2406.16069.

A Evaluation Suite

The framework of our evaluation suites shown in Figure 10 evaluates SLMs across diverse devices, including smartphones (Google Pixel 7 Pro, Xiaomi 12S, MEIZU 18 Pro) and Jetson Orin NX, focusing on capability evaluation and cost analysis. Capability evaluation is conducted on the A800 server using Lm-Eval/OpenCompass, benchmarking models on various tasks.

Cost analysis measures inference efficiency and resource consumption, with both Jetson and smartphones executing cost tasks via llama.cpp. In the cost analysis, the A800 server serves as the central hub, managing model deployment, execution, and result aggregation. It downloads and transfers models to target devices, where models are quantized, and prompts are generated to prepare for cost evaluation. After testing, the A800 server analyzes cost results, generating a detailed report on latency, memory usage, and performance breakdowns. Communication between A800 and Jetson occurs via SSH, while smartphones connect via ADB, ensuring seamless task distribution and result collection. By integrating hardware diversity,
benchmarking tools, and evaluation metrics, the framework enables a comprehensive analysis of SLM efficiency across different deployment scenarios.

B Model Architecture

While we focus on only decoder-only transformer SLMs, their specific configurations still diversify, as shown in Figure 11(a). The core of Transformer is the multi-head self-attention(MHA) mechanism and the Feed-Forward Neural Network(FFN).

Model architecture analysis. We conduct statistical analysis on the following several components of the model architecture: 1) The type of self-attention; 2) The type of feed-forward neural network; 3) The intermediate ratio of the feed-forward network; 4) The activation function of the feed-forward neural network; 5) The type of layer normalization; 6) The vocabulary size. Figure 11(a) shows the architecture of SLM and the pie chart shows the distribution of six components. Figure 11(b) shows how these distributions change over time.

1) The type of self-attention. The self-attention mechanism is the core of the Transformer model. In general, SLMs mainly use three types of attention mechanism: Multi-Head Attention (MHA), Multi-Query Attention (MQA), Group-Query Attention (GQA) and Multi-Head Latent Attention(MLA). Multi-Head Attention is a mechanism that allows the model to focus on different parts of the input data simultaneously by employing multiple attention heads, which is the most widely used self-attention mechanism in the Transformer models. Multi-Query Attention simplifies multi-head attention by using a single shared query across all heads but allowing different key and value projections. This reduces the complexity in both space and time. Group-Query Attention is a variant of multi-head attention that reduces computational complexity by sharing query representations across multiple heads, while allowing separate key and value representations. The idea is to use fewer query groups but still preserve a level of diversity in the attention mechanism. Multi-Head Latent Attention achieves better results than MHA through low-rank key-value joint compression, and requires much less Key-Value(KV) Cache.

Figure 11(b)(1) shows the changing situation of choosing three self-attention mechanisms during these time periods from 2022 to 2024. We can see that MHA is gradually being phased out and replaced by GQA.

2) The type of feed-forward neural network. Feed-forward network can be summarized into two types: the Standard FFN and the Gated FFN. The Standard FFN is a two-layer neural network with a activation function. The Gated FFN adds an additional gate layer.

The Figure 11(b)(2) shows the changing situation of type of FFN during these time periods from 2022 to 2024. It shows that Standard FFN is gradually being phased out and replaced by Gated FFN.

3) The intermediate ratio of the feed-forward neural network. The intermediate ratio of FFN is the ratio of the intermediate dimension to the hidden dimension. Figure 11(b)(3) shows that the intermediate ratio of the Standard FFN is commonly set to be 4, while the intermediate ratio of the Gated FFN is rather diversified ranging from 2 to 8.

4) The activation function of the feed-forward neural network. There are 4 main kinds of activation functions used in FFN: ReLU (Rectified Linear Unit), GELU (Gaussian Error Linear Unit), GELU\textsubscript{tanh}, SiLU (Sigmoid Linear Unit). Observed from Figure 11(b)(4) , the activation function of FFN was mostly ReLU in 2022, and then changed to GELU and its variants in 2023. For those released in 2024, SiLU becomes the dominant type.

5) The type of layer normalization. There are two main types of layer normalization: LayerNorm and RMSNorm. The Figure 11(b)(5) shows the changing situation of type of the type of layer normalization during these time periods from 2022 to 2024. layer normalization is gradually being phased out and replaced by RMS normalization.

6) The vocabulary size. The vocabulary size is the total number of unique tokens that an SLM can recognize. The Figure 11(b)(6) shows the changing situation of the vocabulary size during these time periods from 2022 to 2024. We can see that the vocabulary size of the model is gradually increasing. The vocabulary of the latest models is often larger than 50k

Model architecture innovations. While the vanilla transformer architecture has been well recognized for its scaling ability, there still exist a few architecture-level innovations in the tested SLMs, namely parameter sharing and layer-wise parameter scaling.

1) Parameter Sharing. Parameter Sharing is a technique used in large language models to reuse the same set of weights across different layers or
Figure 11: The architecture analysis of the SLM, highlighting 6 configurations: attention type, FFN type, FFN ratio, FFN activation, vocabulary size, and normalization type. (a) presents the overall structure of the SLM, and the categorizations with usage frequency of the 6 configurations; (b) analyzes the concrete selections of six configurations over time.
components of the network. This approach allows the model to significantly reduce the number of parameters, leading to more efficient training and inference, while maintaining performance.

Embedding-lm_head sharing. Sharing the weights of the embedding with the final lm_head layer is the most common weight sharing technique. It is the sharing of the word embedding layer and has nothing to do with the rotary position encoding. Models such as Gemma, and Qwen all used this sharing technique.

layer-wise attention/FFN sharing. In this approach, the same set of weights is reused across multiple layers of the model. This is commonly seen in SLM/LLM, where all the transformer layers share the same parameters. For example, Mo-biLLaMA shares the weights of the FFN of all the transformer blocks; MobileLLM shares the weights of the Attention and FFN of two adjacent transformer blocks.

2) Layer-wise parameter scaling. This technique was proposed and used by OpenELM. Traditional SLMs use the same configuration for each transformer layer in the model, resulting in a uniform allocation of parameters across layers. Unlike these models, each transformer layer in OpenELM has a different configuration (e.g., number of heads and feed forward network dimension), resulting in variable number of parameters in each layer of the model. This lets OpenELM to better utilize the available parameter budget for achieving higher accuracies.

3) Nonlinearity compensation. PanGu-π analyzes the state-of-the-art language model architectures and observes the feature collapse problem. PanGu-π adopts two techniques for nonlinearity compensation of language model to solve the feature collapse problem. The series activation function is adapted to FFN, and the augmented shortcuts are integrated into MHA, which effectively introduces more nonlinearity into the Transformer architecture.

Insights: We make two key observations in SLM architectures. (1) As of August 2024, a typical SLM architecture tends to use group-query attention, gated FFN with SiLU activation, an intermediate ratio of FFN between 2 and 8, RMS normalization, and a vocabulary size larger than 50K. However, the choice of such settings is mostly empirical, without strict and public validation on the superiority of such model’s capacity. Instead, the architecture innovations have relative larger impacts on the runtime performance on devices, as shown in §3. (2) The innovations to the transformer architecture is limited in nowaday SLMs. For the few that did contribute architectural innovation (except embedding-lm head sharing), we do not observe strong evidence showing them being significantly superior to the vanilla transformer, and neither are them being generally adopted or studied across different research groups or companies. The significance of those innovations remain to be explored and validated.

C Training Datasets and Costs

We find 12 open-source datasets being used:

• The Pile (Gao et al., 2020) (825B tokens): a combination of smaller corpora in various domains.

• FineWeb-Edu (Penedo et al., 2024) (1.3T tokens): a collection of educational text filtered from FineWeb.

• StarCoder (Li et al., 2023) (35B tokens): Python tokens.

• Cosmopedia (Ben Allal et al., 2024) (25B tokens): a dataset of synthetic textbooks, blogposts, stories, posts and WikiHow articles generated by Mixtral-8x7B-Instruct-v0.1.

• RefinedWeb (Penedo et al., 2023) (5T tokens): despite extensive filtering, high-quality data extracted from the web remains plentiful, obtained from CommonCrawl.

• RedPajama (Computer, 2023) (1.2T tokens): includes over 100B text documents coming from 84 CommonCrawl snapshots and processed using the CCNet pipeline.

• Dolma (Soldaini et al., 2024): a English corpora, which is deduplicated inner corpus and across corpus using MinHash algorithms.

• WuDaoCorpora (Yuan et al., 2021) (4T tokens): a super large-scale Chinese corpora, containing about 3T training data and 1.08T Chinese characters.
<table>
  <tr>
    <th>2022</th>
    <th>2023</th>
    <th>2024</th>
  </tr>
  <tr>
    <td>The Pile<br>4 (25.0%)<br>RoBERTa<br>4 (25.0%)</td>
    <td>WuDaoCorpora<br>4 (25.0%)<br>PushShift.io Reddit<br>4 (25.0%)<br>The Pile<br>11 (68.8%)</td>
    <td>Sinhuoan<br>1 (6.25%)<br>StarCoder<br>2 (12.5%)<br>RedPajama<br>1 (6.25%)<br>RefinedWeb<br>7 (19.4%)<br>Dolma<br>4 (11.1%)<br>CulturaX<br>1 (2.8%)<br>RedPajama<br>8 (22.2%)<br>The Pile<br>5 (13.9%)<br>Cosmogedia<br>3 (8.3%)<br>DCML<br>1 (2.8%)<br>FengShui<br>3 (8.3%)<br>StarCoder<br>4 (11.1%)</td>
  </tr>
</table>

Figure 12: The usage frequency of each open-source pre-training dataset from 2022 to 2024.

![Heatmap plots showing long context capabilities of SLMs](page_370_624_1002_312.png)

Figure 13: Long context capabilities of SLMs.

• RoBERTa (Liu, 2019) CCNewsV2: containing an updated version of the English portion of the CommonCrawl News dataset.

• PushShift (.io Reddit (Baumgartner et al., 2020): a social media data collection, analysis, and archiving platform that since 2015 has collected Reddit data and made it available to researchers.

• DCLM-baseline (Li et al., 2024a) (1.35T tokens): a standardized corpus extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations.

• CulturaX (Nguyen et al., 2023) (6.3T tokens): a substantial multilingual dataset in 167 languages.

The usage preference of pre-training datasets. We then conducted statistics on the usage frequency of the datasets for training SLM from 2022 to 2024. The results are illustrated in Figure 12. It shows that The Pile is the most widely used pre-training dataset especially in 2022 and 2023; yet more recently, more such datasets are proposed and the choice becomes diversified. In fact, The Pile has been abandoned in pre-training SLMs recently, and datasets such as "RefinedWeb" and "RedPajama" have gradually been widely used. It shows the active research and engineering efforts in constructing pre-training datasets with better quality.

Since pretraining typically uses only one epoch, the Training TFLOPs is calculated as: \( \text{Training FLOPs} = 6 \times \text{Model Size} \times \text{Training Tokens} \). Here, the factor 6 accounts for the number of multiplication and addition operations required per token during the model’s forward and backward propagation.

When estimating the actual computation time using Training TFLOPs, we need to consider Model FLOPs Utilization (MFU) (Chowdhery et al., 2023). Using the data from DCLM (Li et al., 2024a), we obtain a typical MFU of approximately 30%.

The estimated training time on H100 (BF16
TFLOPS is 1979) is given by:

\[
\text{Train Hours} = \frac{\text{Training FLOPs}}{\text{MFU} \times \text{H100 TFLOPs}}
\] (1)

As shown in Table 5, we can observe that more tokens are being used to train SLMs over time. The overall training cost has escalated accordingly. And we select 8 models released at 2024 and show that the recent SLMs do not obey the scaling law of training cost and model size. For example, as shown in Table 6, Qwen2 0.5B takes more training tokens and hours than Gemma-2 2B.

Table 5: Token count for SLM pretraining is rising over time.

<table>
  <tr>
    <th>Model</th>
    <th>Size (B)</th>
    <th>Tokens (T)</th>
    <th>TFLOPs</th>
    <th>Date</th>
    <th>H100 h</th>
  </tr>
  <tr>
    <td>OPT</td>
    <td>1.3</td>
    <td>0.18</td>
    <td>1.4</td>
    <td>22.05</td>
    <td>657</td>
  </tr>
  <tr>
    <td>Bloom</td>
    <td>1.1</td>
    <td>0.35</td>
    <td>2.3</td>
    <td>22.11</td>
    <td>1081</td>
  </tr>
  <tr>
    <td>Pythia</td>
    <td>1.4</td>
    <td>0.30</td>
    <td>2.5</td>
    <td>23.03</td>
    <td>1179</td>
  </tr>
  <tr>
    <td>Phi-1.5</td>
    <td>1.5</td>
    <td>0.15</td>
    <td>1.4</td>
    <td>23.09</td>
    <td>632</td>
  </tr>
  <tr>
    <td>TinyLlama</td>
    <td>1.1</td>
    <td>3.0</td>
    <td>19.8</td>
    <td>23.12</td>
    <td>9264</td>
  </tr>
  <tr>
    <td>OpenELM</td>
    <td>1.1</td>
    <td>1.5</td>
    <td>9.9</td>
    <td>24.04</td>
    <td>4632</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>1.5</td>
    <td>7.0</td>
    <td>63.0</td>
    <td>24.06</td>
    <td>29476</td>
  </tr>
</table>

Table 6: The training cost of recent SLMs.

<table>
  <tr>
    <th>Model</th>
    <th>Size (B)</th>
    <th>Tokens (T)</th>
    <th>TFLOPs</th>
    <th>Date</th>
    <th>H100 h</th>
  </tr>
  <tr>
    <td>SmolLM</td>
    <td>0.135</td>
    <td>0.6</td>
    <td>0.49</td>
    <td>24.07</td>
    <td>227</td>
  </tr>
  <tr>
    <td>SmolLM</td>
    <td>0.36</td>
    <td>0.6</td>
    <td>1.30</td>
    <td>24.07</td>
    <td>606</td>
  </tr>
  <tr>
    <td>Qwen 2</td>
    <td>0.5</td>
    <td>12</td>
    <td>36.0</td>
    <td>24.06</td>
    <td>16844</td>
  </tr>
  <tr>
    <td>Qwen 2</td>
    <td>1.5</td>
    <td>7</td>
    <td>63.0</td>
    <td>24.06</td>
    <td>29476</td>
  </tr>
  <tr>
    <td>SmolLM</td>
    <td>1.7</td>
    <td>1</td>
    <td>10.2</td>
    <td>24.07</td>
    <td>4772</td>
  </tr>
  <tr>
    <td>Gemma-2</td>
    <td>2.0</td>
    <td>2</td>
    <td>24.0</td>
    <td>24.07</td>
    <td>11229</td>
  </tr>
  <tr>
    <td>Phi-3</td>
    <td>3.0</td>
    <td>3.3</td>
    <td>59.4</td>
    <td>24.04</td>
    <td>27792</td>
  </tr>
  <tr>
    <td>Qwen 1.5</td>
    <td>4.0</td>
    <td>4</td>
    <td>96.0</td>
    <td>24.02</td>
    <td>44916</td>
  </tr>
</table>

D Long Context Capabilities

We used Needle-In-A-Haystack provided by OpenCompass to explore long-context capabilities of SLMs. The tasks included Single-Needle Retrieval, Multi-Needle Retrieval, and Multi-Needle Reasoning. The scores in Figure 13 are the average of these three tasks. Different models showed large variations in performance. Small models, such as Qwen1.5-0.5B and Qwen2-0.5B, performed less effectively. Qwen1.5-0.5B achieved an average accuracy of 22.13%. Qwen2-0.5B performed slightly better, reaching 43.84%. Qwen1.5-0.5B handled shorter contexts (9k-17k) relatively well. However, its accuracy dropped sharply with longer contexts. This was especially true for middle inserted text (Depth Percent from 20% to 70%). Larger models performed much better. Llama 3.2-3B had an average accuracy of 57.81%. It worked well with shorter contexts but struggled with deeper insertions when contexts exceeded 25k tokens. Qwen2.5-3B achieved an average accuracy of 91.71%. It maintained nearly perfect accuracy across all context lengths and insertion positions. This result highlights its strong ability to handle long contexts and complex scenarios.

Insights: We draw two key insights from the long context capacity of SLMs: (1) Larger parameters are crucial for long-context capabilities. Small models, such as Qwen1.5-0.5B and Qwen2-0.5B, perform adequately on short-context tasks but experience a significant drop in recognition accuracy as the context length increases. In contrast, larger models, such as Qwen2.5-3B, excel with outstanding performance, maintaining near-perfect accuracy across all context lengths and insertion positions. (2) "Lost in the Middle" also occurs in small models. Compared to deep or front insertions, the accuracy of middle-position text (Depth Percent 20%-70%) is significantly lower.

E Tool-use Capabilities

We conducted evaluations on tool-use capabilities with DroidCall(Xie et al., 2024), a dataset for LLM-powered Android intent invocation. Without fine-tuning, SLMs can not catch with LLMs on the tool-use capabilities as shown in Table 7. Qwen2.5-3B, Gemma2, Phi-3.5, and Llama3.2-3B show better performance than GPT-4o after fine-tuning.

Table 7: The tool-use capabilities of SLMs.

<table>
  <tr>
    <th>Model</th>
    <th>Size</th>
    <th>Zero-Shot Acc</th>
    <th>Few-Shot Acc</th>
    <th>FT Acc</th>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct</td>
    <td>1.5B</td>
    <td>61.0</td>
    <td>64.5</td>
    <td>76.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-Instruct</td>
    <td>3B</td>
    <td>62.0</td>
    <td>71.0</td>
    <td>83.0</td>
  </tr>
  <tr>
    <td>Gemma2-it</td>
    <td>2B</td>
    <td>59.0</td>
    <td>67.5</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-instruct</td>
    <td>3.8B</td>
    <td>62.0</td>
    <td>67.5</td>
    <td>83.5</td>
  </tr>
  <tr>
    <td>MiniCPM3-4B</td>
    <td>4B</td>
    <td>67.0</td>
    <td>75.0</td>
    <td>74.5</td>
  </tr>
  <tr>
    <td>Llama2-2-Instruct</td>
    <td>1B</td>
    <td>31.5</td>
    <td>60.5</td>
    <td>75.5</td>
  </tr>
  <tr>
    <td>Llama2-2-Instruct</td>
    <td>3B</td>
    <td>66.5</td>
    <td>72.0</td>
    <td>82.0</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>77.0</td>
    <td>80.5</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>-</td>
    <td>71.5</td>
    <td>76.0</td>
    <td>-</td>
  </tr>
</table>

F Evaluations on RNN-based Models

To explore the runtime cost of RNN-based architecture models, we evaluated Mamba-1.4B on Jetson Orin NX. Compared to Phi-1.5 with a similar size, Mamba does not have an advantage in prefill time. As shown in Figure 14, it takes more time, likely due to the lack of llama.cpp operator optimization for RNN-based models. Decode la-
tency per token typically increases as the number of tokens increases in Transformer-based models because of the \( O(n^2) \) time complexity of the attention mechanism (\( n = \) token count). However, Mamba maintains a very stable decode latency per token, highlighting the advantages of its \( O(n) \) time complexity.

In terms of accuracy performance, we evaluate the Mamba variant: Zamba2. We evaluate Zamba2-1.2B and Zamba2-2.7B on 9 downstream tasks, as shown in Table 8. Zamba2-2.7B demonstrates better performance than Llama2-7B with fewer parameters, while Zamba2-1.2B performs worse than Llama2-7B. Compared to the SLMs released in the same period (May 2024) shown in Figure 1, Zamba2-2.7B performed very well, ranking just behind Phi-3.

Table 8: Capabilities of Zamba2-1.2B and Zamba2-2.7B on 9 downstream tasks.

<table>
  <tr>
    <th>Model</th>
    <th>ARC-E</th>
    <th>ARC-C</th>
    <th>BosQ</th>
    <th>CSPA</th>
    <th>HeliaSwag</th>
    <th>MMLU</th>
    <th>ORQA</th>
    <th>PiQA</th>
    <th>TruthfulQA</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Zamba2-1.2B</td>
    <td>0.44</td>
    <td>0.77</td>
    <td>0.74</td>
    <td>0.52</td>
    <td>0.53</td>
    <td>0.44</td>
    <td>0.52</td>
    <td>0.77</td>
    <td>0.58</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Zamba2-2.7B</td>
    <td>0.48</td>
    <td>0.80</td>
    <td>0.80</td>
    <td>0.75</td>
    <td>0.57</td>
    <td>0.56</td>
    <td>0.52</td>
    <td>0.79</td>
    <td>0.46</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>0.45</td>
    <td>0.74</td>
    <td>0.80</td>
    <td>0.76</td>
    <td>0.76</td>
    <td>0.61</td>
    <td>0.54</td>
    <td>0.79</td>
    <td>0.58</td>
    <td>0.60</td>
  </tr>
</table>

G Ablation Studies

G.1 Impact of Model Architecture on Performance

To systematically examine the impact of model architecture, we trained 18 different models of similar parameter sizes, each on the same dataset containing 20B tokens. Table 9 shows that model architecture has a negligible effect on final training loss (ranging from 3.58 to 3.81), suggesting similar next-token prediction capability across variants. However, inference speeds—both in the prefill and decoding stages—vary significantly among architectures, with up to \( 5.22 \times \) difference in decoding speed.

G.2 Impact of Data Quality on Capabilities

To investigate the effect of data quality, two models with 500M parameters and identical hyperparameters were trained on different datasets: one on RefinedWeb + StarCoder (200B tokens) and one on DCLM (100B tokens). Both models share the same architecture. Evaluation on seven downstream tasks (ARC-Easy, ARC-Challenge, HeliaSwag, TruthfulQA, Winogrande, PIQA, MMLU) yields average accuracies of 0.47 and 0.52, respectively. Despite using fewer tokens, the DCLM-trained model outperforms the model trained on larger but less curated data, indicating the importance of data quality over sheer quantity.

These experiments support our important conclusions:

• Model architecture significantly affects inference latency (§3.1).

• Dataset quality has a crucial impact on model performance (§2.4).

Table 9: Ablation on model architectures. All models trained on 20B tokens with comparable size.

<table>
  <tr>
    <th>ID</th>
    <th>Size(M)</th>
    <th>Bidder</th>
    <th>FTN</th>
    <th>Layers</th>
    <th>Act</th>
    <th>QP</th>
    <th>R/VF</th>
    <th>Loss</th>
    <th>Prefill (ms/b)</th>
    <th>Decode (ms/b)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>106.73</td>
    <td>1280</td>
    <td>2096</td>
    <td>1</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.76</td>
    <td>916.70</td>
    <td>45.52</td>
  </tr>
  <tr>
    <td>2</td>
    <td>106.73</td>
    <td>1280</td>
    <td>2096</td>
    <td>9</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.81</td>
    <td>877.19</td>
    <td>42.48</td>
  </tr>
  <tr>
    <td>3</td>
    <td>101.42</td>
    <td>768</td>
    <td>2048</td>
    <td>1</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.67</td>
    <td>784.94</td>
    <td>26.68</td>
  </tr>
  <tr>
    <td>4</td>
    <td>101.42</td>
    <td>768</td>
    <td>2048</td>
    <td>9</td>
    <td>relu</td>
    <td>4</td>
    <td>16</td>
    <td>3.67</td>
    <td>784.94</td>
    <td>26.68</td>
  </tr>
  <tr>
    <td>5</td>
    <td>101.42</td>
    <td>768</td>
    <td>2048</td>
    <td>9</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.69</td>
    <td>785.95</td>
    <td>26.03</td>
  </tr>
  <tr>
    <td>6</td>
    <td>101.42</td>
    <td>768</td>
    <td>2048</td>
    <td>9</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.69</td>
    <td>785.95</td>
    <td>26.03</td>
  </tr>
  <tr>
    <td>7</td>
    <td>101.42</td>
    <td>768</td>
    <td>2048</td>
    <td>9</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.69</td>
    <td>785.95</td>
    <td>26.03</td>
  </tr>
  <tr>
    <td>8</td>
    <td>101.42</td>
    <td>768</td>
    <td>2048</td>
    <td>9</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.69</td>
    <td>785.95</td>
    <td>26.03</td>
  </tr>
  <tr>
    <td>9</td>
    <td>99.54</td>
    <td>768</td>
    <td>1856</td>
    <td>11</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.65</td>
    <td>853.46</td>
    <td>28.71</td>
  </tr>
  <tr>
    <td>10</td>
    <td>99.54</td>
    <td>768</td>
    <td>1856</td>
    <td>11</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.65</td>
    <td>853.46</td>
    <td>28.71</td>
  </tr>
  <tr>
    <td>11</td>
    <td>100.00</td>
    <td>576</td>
    <td>1536</td>
    <td>18</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>691.56</td>
    <td>18.59</td>
  </tr>
  <tr>
    <td>12</td>
    <td>100.00</td>
    <td>576</td>
    <td>1536</td>
    <td>18</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>691.56</td>
    <td>18.59</td>
  </tr>
  <tr>
    <td>13</td>
    <td>100.00</td>
    <td>576</td>
    <td>1536</td>
    <td>18</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>691.56</td>
    <td>18.59</td>
  </tr>
  <tr>
    <td>14</td>
    <td>100.00</td>
    <td>576</td>
    <td>1536</td>
    <td>18</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>691.56</td>
    <td>18.59</td>
  </tr>
  <tr>
    <td>15</td>
    <td>100.00</td>
    <td>576</td>
    <td>1536</td>
    <td>18</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>691.56</td>
    <td>18.59</td>
  </tr>
  <tr>
    <td>16</td>
    <td>101.06</td>
    <td>448</td>
    <td>1184</td>
    <td>53</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>469.99</td>
    <td>15.48</td>
  </tr>
  <tr>
    <td>17</td>
    <td>101.06</td>
    <td>448</td>
    <td>1184</td>
    <td>53</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.68</td>
    <td>469.99</td>
    <td>15.48</td>
  </tr>
  <tr>
    <td>18</td>
    <td>101.06</td>
    <td>448</td>
    <td>1184</td>
    <td>53</td>
    <td>relu</td>
    <td>16</td>
    <td>16</td>
    <td>3.67</td>
    <td>481.58</td>
    <td>17.70</td>
  </tr>
</table>

(a) Prefill
(b) Decode

Figure 14: Latency of Mamba on Jetson Orin NX.

Reviewer paper summaries: This paper presents a comprehensive study on small language models (SLMs), focusing on their capabilities, runtime performance, and optimization opportunities. The authors benchmark over 60 SLMs, including Phi and Gemma models, and evaluate their performance on 10 datasets. The study identifies key challenges, such as in-context learning limitations and runtime efficiency, and explores potential optimizations like dynamic task-specific routing, architecture-hardware co-design, and vocabulary/KV cache compression. Additionally, an evaluation suite and a leaderboard website are developed to facilitate research in this domain.
Summary of strengths: **Comprehensive Benchmarking** The study covers 68 SLMs from various organizations, making it one of the most extensive benchmarks in this area. Evaluations include both model capabilities (e.g., commonsense reasoning, in-context learning) and runtime performance (e.g., inference latency, memory footprint).

**Useful Insights** The paper provides valuable insights into SLMs' strengths and weaknesses, particularly in how they compare with larger models like LLaMA-7B/8B. It highlights the impact of dataset quality over dataset size, emphasizing the role of model-based data filtering in improving performance.

**Practical Contributions** The creation of an SLM evaluation suite and a leaderboard advances the field by providing a standardized way to compare models. The discussion on optimization opportunities, such as hardware-aware design and quantization techniques, is highly relevant for real-world deployment.
Summary of weaknesses: **Lack of Technical contributions** While the paper provides a comprehensive benchmark and insightful analysis of small language models (SLMs), it lacks significant technical contributions.

**Lack of Discussion on Training Costs** While the paper discusses over-training and dataset quality, it does not explore the computational costs associated with training SLMs. A cost-benefit analysis could be useful for researchers and practitioners interested in training or fine-tuning such models.
Comments, suggestions and typos: see weaknesses.
Overall assessment scores: 2.5