# Enhancing Small Language Models for Graph Tasks Through Graph Encoder Integration
## 1 Introduction  
With the rapid expansion of AI technologies into mobile and edge devices, there has been growing interest in small language models (SLMs) for on-device applications. These models offer several advantages, including enhanced privacy, reduced inference latency, and decreased dependency on network connectivity. By processing user data directly on a device without transmitting it to external servers, SLMs are well suited for privacy-centric computing environments [1, 2]. For example, on-device AI has shown significant potential in healthcare, such as real-time analysis of medical data collected from wearable devices. Processing sensitive data locally allows these systems to deliver actionable insights while maintaining data privacy [3]. However, the data processed in on-device environments often extends beyond simple text or tabular formats. Real-world datasets, such as social networks and system logs, frequently exhibit graph structures where relationships between nodes and edges form complex patterns [4; 5]. These graph data possess structural dependencies which differ from sequential text data, requiring models to capture both local and global relationships effectively. One major limitation of on-device SLMs lies in their inability to handle graph-related tasks effectively [6]. Unlike sequential text, graph data are inherently relational and structured, demanding models to account for both local neighborhood information and global connectivity among nodes and edges. While pretrained language models (PLMs) have demonstrated exceptional performance in text-based tasks, they are not inherently designed for graph data processing. Attempts to linearize graph structures into sequences for autoregressive text prediction often fail to preserve the underlying structural and relational information, resulting in suboptimal performance in graph tasks. This underscores the need for specialized methods to adapt SLMs to graph-based applications. To address this challenge, prompt tuning [6] and fine-tuning [6; 7] have emerged as effective strategies. Prompt tuning involves learning task-specific prompts which enable a model to utilize pretrained knowledge efficiently, requiring fewer trainable parameters and less computational overhead, which is ideal for resource-constrained environments. In contrast, fine-tuning directly updates the selected parameters of a model to enhance performance, offering greater adaptability for tasks requiring precise adjustments. In this work, we propose a novel graph encoder which integrates a graph convolutional network (GCN) [8] and a graph transformer [9] to address these challenges. By combining the strengths of the GCN for local structural modeling and the graph Transformer for capturing global relationships, the proposed encoder enables SLMs to process graph data effectively. Our experimental results demonstrate that this graph encoder significantly improves the performance of SLMs in both prompt tuning and fine-tuning scenarios, achieving results WHICH closely approach the performance of a large language model (LLM). The key contributions of this work are as follows:  
* We propose a novel graph encoder-based approach to enhance the capability of SLMs for graph-based tasks. * We design a graph encoder which combines a GCN and a graph transformer, minimizing information loss in graph structures and optimizing efficiency for resource-constrained environments. * We validate the proposed method in both prompt tuning and fine-tuning scenarios, demonstrating that it significantly improves SLM performance and achieves results comparable to an LLM on graph-related benchmarks. This study highlights the potential of adapting SLMs to graph data processing and contributes to advancing AI scalability in mobile and edge computing environments. The remainder of this paper is organized as follows: Section 2 reviews the related work, focusing on small language models (SLMs) and tuning strategies for pretrained language models (PLMs).