# Enhancing Small Language Models for Graph Tasks Through Graph Encoder Integration
## 2 Related Works
### Training Approaches for Pretrained Language Models (PLMs)  
The training process for pretrained language models (PLMs) typically involves two main steps. First, the models are fine-tuned to perform specific tasks. After that, the models are optimized and adjusted to ensure they can run efficiently on the target systems with limited resources. This process helps the models maintain strong performance while being simplified to fit into environments with fewer computational resources.  
One common way to adapt models for specific tasks is task-oriented fine-tuning [32]. In this approach, a new layer is added to the final part of the pretrained language model, and the entire model is fine-tuned. This method often achieves high accuracy for specific tasks, but it requires a great deal of computing power and large amounts of labeled data. As a result, it can be difficult to use this method for on-device applications or in environments with limited resources.  
To reduce these challenges, prompt-oriented fine-tuning has become a more efficient alternative. Instead of changing the model's internal settings, this method modifies the input by adding specific prompts to guide the model's output. Hard prompt tuning [33; 34] uses manually written text prompts, while soft prompt tuning [35; 36; 37; 38] replaces these text prompts with adjustable vectors that the model learns. This reduces the need to adjust many parts of the model, lowering the computational cost while keeping the model flexible for different tasks. Soft prompt tuning is especially useful when lightweight and scalable solutions are required.  
Prefix-tuning is another efficient fine-tuning method which requires fewer adjustments to the model. In this approach, a short, trainable prefix is added to the input, while the rest of the model remains unchanged. This greatly reduces the number of parameters that need to be trained, resulting in lower memory use. Prefix-tuning is ideal for situations where computational power and memory are limited. Despite its simplicity, it performs well in adapting language models to different tasks while using fewer resources [39; 40; 41].  
Training language models for graph-based tasks adds more complexity because graphs are structured differently from regular text data [6]. Graph tasks, such as classifying nodes or predicting the links between them, require the model to understand the relationships between different points in a graph. Traditional fine-tuning methods often do not perform well for these types of tasks, and thus graph-specific techniques are needed.  
To solve this, methods like skip-gram treat graph nodes like words and create embeddings based on their connections. More advanced methods, such as the graph neural networks (GNNs) [42], gather information from nearby nodes to create these embeddings. Well-known models, such as the graph convolutional networks (GCNs) [8], graph attention networks (GATs) [43], and graph transformer [9], improve graph representations by usingthe structure of the graph to map nodes into vector spaces. This helps with tasks like classifying nodes, predicting connections, and grouping similar nodes.  
By applying graph-specific fine-tuning methods, PLMs can handle structured graph data more effectively. This expands their use to fields which require understanding graphs and structured data, allowing them to perform well in applications beyond standard text-based tasks.