# Enhancing Small Language Models for Graph Tasks Through Graph Encoder Integration
## 2 Related Works  
### Small Language Models (SLMs)  
On-device models refer to AI systems which run directly on local devices, unlike cloud-based models, which rely on external servers. These models are designed to efficiently handle tasks by using pretrained language models, enabling them to operate within the limited resources of a device. Language models with fewer parameters capable of running directly on devices are commonly known as small language models (SLMs). However, deploying SLMs on devices presents several technical challenges [1]. A major challenge is the limited memory, storage, and processing power of most devices [10]. Many consumer devices do not have enough memory to run large language models (LLMs). Additionally, models with billions of parameters require significant storage space, and loading these models can slow down real-time applications, negatively impacting user experience. Limited memory bandwidth further reduces the models' ability to process information quickly, causing delays in responses. These factors make deploying large models on devices difficult, especially when fast and efficient operation is required. To address these challenges, researchers are focusing on developing smaller and more efficient SLMs. One common approach is model compression, which reduces the number of parameters and the memory footprint, making the models lighter and easier to run. While this can sometimes lead to performance trade-offs, architectural improvements, such as sharing parts of the model to reduce redundancy, help maintain accuracy and efficiency. These strategies allow SLMs to function well even with limited device resources [10]. Although LLMs generally perform better in complex tasks [11; 12; 13]. SLMs can still achieve competitive results on certain benchmarks, such as the AI2 Reasoning Challenge (ARC). For specialized applications, SLMs can deliver sufficient performance with fewer parameters, making them a practical choice for devices with resource constraints [2; 14; 15]. Recent advancements in SLM design have significantly improved their performance, allowing them to match or even exceed earlier versions of larger models in some areas. Innovations in attention mechanisms, including multi-head attention (MHA), multi-query attention, and grouped query attention (GQA) [16], have enhanced memory efficiency and processing speeds. By focusing on deeper rather than wider model structures and fine-tuning their parameters, researchers have created lightweight models which maintain strong performance. HuggingFace introduced SmolLM, employing GQA to design models with parameter scales of 135 M, 360 M, and 1.7 B [17]. The 135 M and 360 M models in particular prioritize depth over width, enhancing efficiency through hyperparameter optimization. By sharing input and output embeddings, SmolLM minimizes parameter counts without compromising performance. The models are trained on synthetic datasets generated by Mixtral [11], alongside publicly available datasets such as CommonCrawl [18], FineWebedu [19], and python-edu. Consequently, SmolLM achieves state-of-the-art results across benchmarks including MMLU [20], TriviaQA [21], ARC [22], PIQA [23], and Hellaswag [24], surpassing competing models such as Phi [14] and Qwen [15]. Building on the success of SmolLM, SmolLM2 [25] demonstrates significant advances over its predecessor, particularly in instruction following, knowledge, and reasoning. The 360 M model was trained on 4 trillion tokens using a diverse dataset combination--FineWeb-Edu [19], DCLM [26], and The Stack [27]--along with newly curated filtered datasets which are planned for public release. The instruction tuning version of SmolLM2 was developed through supervised fine-tuning (SFT) using a combination of public datasets and proprietary curated datasets. Direct preference optimization (DPO) was then applied using UltraFeedback [28].