# Enhancing Small Language Models for Graph Tasks Through Graph Encoder Integration
## 3 Proposed Method
### Graph Encoder Architecture  
We propose a graph encoder architecture that combines a graph convolutional network (GCN) [8] and a graph transformer [9] to effectively capture both local structural features and global relationships in graph-structured data. The encoder processes a graph \(G=(\mathcal{V},\mathcal{E},\mathbf{X},\mathbf{E})\), where \(\mathcal{V}\) represents the set of \(N\) nodes, \(\mathcal{E}\) denotes the edge set, \(\mathbf{X}\in\mathbb{R}^{N\times F}\) is the node feature matrix, and \(\mathbf{E}\) optionally contains the edge attributes. The adjacency matrix of the graph is denoted by \(\mathbf{A}\).  
The encoder consists of \(L\) layers. In the initial layer, a GCN is employed to capture the localized structural features of the graph. The operation is defined as:  
\[\mathbf{H}^{(l+1)}=\sigma\Big{(}\tilde{\mathbf{D}}^{-\frac{1}{2}}\tilde{ \mathbf{A}}\tilde{\mathbf{D}}^{-\frac{1}{2}}\mathbf{H}^{(l)}\mathbf{W}^{(l)} \Big{)}, \tag{1}\]  
where \(\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}\) includes self-loops, \(\tilde{\mathbf{D}}\) is the degree matrix of \(\tilde{\mathbf{A}}\), \(\mathbf{W}^{(l)}\) is a trainable weight matrix, and \(\sigma\) denotes the activation function (ReLU). This operation yields intermediate embeddings \(\mathbf{H}^{(1)}\) that encode localized graph features.  
Figure 1: The proposed method for graph prompt tuning combines a graph encoder and a frozen small language model (SLM) to effectively capture both structural and semantic information. The graph encoder, leveraging a GCN and graph transformer layers, is trained (burned) to learn localized and global dependencies, while the frozen SLM processes textual attributes using self-attention mechanisms. This design ensures efficient integration of graph structural information with pretrained semantic embeddings while preserving the SLM’s pretrained knowledge.  
For the subsequent layers (\(l=1,\ldots,L-1\)), the graph transformer is used to capture both local and global dependencies via multi-head attention. The embedding for each node \(i\) is computed as:  
\[\mathbf{H}_{i}^{(l+1)}=\sum_{j\in\mathcal{N}(i)}\alpha_{ij}\mathbf{V}_{j}, \tag{2}\]  
where the attention weight \(\alpha_{ij}\) between nodes \(i\) and \(j\) is computed by  
\[\alpha_{ij}=\text{softmax}\Bigg{(}\frac{\big{(}\mathbf{W}_{Q}\mathbf{H}_{i}^{( l)}\big{)}\big{(}\mathbf{W}_{K}\mathbf{H}_{j}^{(l)}\big{)}^{\top}}{\sqrt{d_{k}}} \Bigg{)}. \tag{3}\]  
Here, \(\mathbf{W}_{Q}\), \(\mathbf{W}_{K}\), and \(\mathbf{W}_{V}\) are trainable parameters used to derive the query, key, and value vectors, respectively. For each node \(i\) and its neighbors \(j\), these vectors are computed as:  
\[\mathbf{Q}_{i}=\mathbf{H}_{i}^{(l)}\mathbf{W}_{Q},\quad\mathbf{K}_{j}= \mathbf{H}_{j}^{(l)}\mathbf{W}_{K},\quad\mathbf{V}_{j}=\mathbf{H}_{j}^{(l)} \mathbf{W}_{V}. \tag{4}\]  
In the final layer, the graph transformer output \(\mathbf{H}^{(L)}\) is passed through a projection layer based on a multi-layer perceptron (MLP) to generate the representation for each node:  
\[\hat{\mathbf{X}}^{i}=\text{MLP}\big{(}\mathbf{H}_{i}^{(L)}\big{)}. \tag{5}\]  
The projection layer, equipped with trainable parameters, ensures that the output dimension aligns with the embedding space of the small language model (SLM), facilitating seamless integration.  
This architecture integrates localized graph convolutional learning with attention-based global aggregation, enabling effective modeling of complex dependencies in graph-structured data. By combining the strengths of the GCN and graph transformer, the model achieves a balance between computational efficiency and expressive power, making it well-suited for tasks requiring joint graph and language understanding.