# Enhancing Small Language Models for Graph Tasks Through Graph Encoder Integration
## 4 Experiments
### Comparison of Instruction-Tuned and Non-Instruction-Tuned Models  
Figure 2 presents a comparison of instruction-tuned and non-instruction-tuned models in the SmoILM and SmoILM2 series for node classification tasks on the Citeseer dataset. The evaluation involved four models: SmoILM-360M, SmoILM-360M-Instruct, SmoILM2-360M, and SmoILM2-360M-Instruct. Each model was evaluated using prompt tuning, fine tuning, GCN-T-based prompt tuning, and GCN-T-based fine tuning.  
\begin{table}
\begin{tabular}{l l c} \hline \hline  & \multicolumn{2}{c}{**Citeseer**} \\ \hline
**Model** & **Method** & **Node Classification** \\ \hline \multirow{3}{*}{Nano-Mistral-170M [30]} & Prompt tuning & 31.98 \\  & Fine-tuning & 60.03 \\  & Prompt tuning (with GCN-T) & 55.66 \\  & Fine-tuning (with GCN-T) & 68.78 \\ \hline \multirow{3}{*}{Pythia-160M [31]} & Prompt tuning & 1.06 \\  & Fine-tuning & 24.59 \\ \cline{1-1}  & Prompt tuning (with GCN-T) & 9.05 \\ \cline{1-1}  & Fine-tuning (with GCN-T) & 32.73 \\ \hline \multirow{3}{*}{SmolLM2-135M-Instruct [25]} & Prompt tuning & 34.39 \\ \cline{1-1}  & Fine-tuning & 60.78 \\ \cline{1-1}  & Prompt tuning (with GCN-T) & 57.92 \\ \cline{1-1}  & Fine-tuning (with GCN-T) & 71.34 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Node classification results on the Citeseer dataset using different small language models (SLMs). The table compares standalone prompt tuning and fine-tuning with their GCN-T integrated counterparts. The results highlight the importance of incorporating graph structural information for improved performance in graph-related tasks.  
The results reveal that instruction-tuned models generally perform better in fine-tuning scenarios, especially when integrated with GCN-T. Notably, GCN-T-based fine-tuning consistently achieved the highest performance across all models, leveraging graph structural information to enhance node classification accuracy. In contrast, standard prompt tuning underperformed on instruction-tuned models, such as SmoILM-360M-Instruct and SmoILM2-360M-Instruct, due to the unstructured nature of textual attributes extracted from graph data. These attributes often fail to align with the explicit instruction patterns expected by instruction-tuned models.  
GCN-T-based prompt tuning significantly mitigates this issue by incorporating node relationships into the SLM input through graph embeddings. This method enhances the ability of instruction-tuned models to interpret graph data, resulting in consistent improvements over standard prompt tuning. Fine-tuning, on the other hand, effectively adapts the entire model to the graph data, further improving performance, particularly in instruction-tuned models.  
These findings underscore the importance of integrating graph embeddings via GCN-T for both instruction-tuned and non-instruction-tuned models. By capturing both local and global graph information, GCN-T-based tuning provides a robust solution for  
Figure 2: The graphs compare the node classification accuracy of different tuning methods (prompt tuning, fine-tuning, and GCN-T-based variations) for four models: SmoILM-360M, SmoILM-360M-Instruct, SmoILM2-360M, and SmoILM2-360M-Instruct. Notably, the instruction-tuned models (SmoILM-360M-Instruct and SmoILM2-360M-Instruct) exhibited worse accuracy in prompt tuning compared with their non-instruction-tuned counterparts (SmoILM-360M and SmoILM2-360M). This highlights the limitations of freezing the SLM during prompt tuning for instruction-tuned models, which may be less effective in graph-based tasks without fine-tuning.  
leveraging structural data in graph-based tasks. The results demonstrate that small language models, when enhanced with GCN-T, can achieve competitive performance, even in resource-constrained environments, highlighting the potential of instruction-tuned models for such tasks.