GNNAVI: Navigating the Information Flow in Large Language Models by Graph Neural Network (2024)

Abstract: Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient finetuning (PEFT) approach. GNNAVI leverages insights into ICL’s information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GNNAVI employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 show GNNAVI surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNAVI with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNAVI enhances information flow and ensures a clear aggregation process. 1

heading: 1 Introduction
text: Large language models (LLMs) show remarkable In-Context-Learning (ICL) capabilities by learning from prompts with demonstrations (Wan et al., 2023; Sun et al., 2023; Patel et al., 2023; Mekala et al., 2023; Ko et al., 2023), with the exponential growth in model sizes. However, fine-tuning LLMs still remains essential for further enhancing their adaptability (Zhang et al., 2023). Prompt-based fine-tuning (Schick and Schütze, 2021a; Ma et al.,
1Our code is available at https://github.com/ ShuzhouYuan/GNNavi.
tion from preceding words ( blue path ), and the final token aggregates information from the label words ( pink path ), resulting in a clearer information aggregation process.
2024), adopting objectives that simulate the language modeling process, emerges as a viable technique, particularly in low-data settings (Gao et al., 2021). Yet, the substantial computational demands of Full-Parameter Fine-Tuning (FPFT), which updates billions of parameters, pose a practical challenge. In fact, optimizing a relatively small subset of an LLM’s parameters can significantly improve its performance (Ding et al., 2023), paving the way for Parameter-Efficient Fine-Tuning (PEFT) methods. These methods include Adapter (Houlsby et al., 2019), Prompt-Tuning (Lester et al., 2021),
3987
Prefix Tuning (Li and Liang, 2021), and LoRA (Hu et al., 2022). They offer alternatives to FPFT but are often not tailored to the prompt-based finetuning of LLMs.
Recent advances in understanding the ICL mechanism offer a new avenue for PEFT of LLMs. ICL’s success in leveraging few-shot demonstrations and prompts (Brown et al., 2020) has motivated the adoption of prompt-based fine-tuning for moderately sized language models in a few-shot learning manner (Ma et al., 2023; Schick and Schütze, 2021b). Recognizing the specific features of finetuning LLMs within the framework of ICL, we propose GNNAVI, a novel PEFT method designed expressly for prompt-based learning. Our method draws inspiration from recent insights into the underlying process of ICL from an information flow perspective, particularly the role of label words in the prompt (Wang et al., 2023). Label words act as anchors with two functions: aggregating information from context words and directing this information to the last token for accurate predictions. GNNAVI incorporates this understanding through the integration of a Graph Neural Network (GNN) layer (Kipf and Welling, 2017; Hamilton et al., 2017) into LLMs, optimizing the promptbased fine-tuning process by navigating the information flow within prompts, as visualized in Figure 1. Following the paths of information flow, we insert a GNN layer into the deep layers2 of the LLM. We treat the input text as a graph, where each token serves as a node, and connect these nodes according to the paths of information flow. The GNN layer aims to guide the information flow by aggregating information from neighbouring nodes.
As a PEFT method, GNNAVI adopts a lightweight fine-tuning strategy, updating only the parameters of the GNN layer. Experimenting with few-shot training examples on GPT2-XL (Radford et al., 2019) and Llama2 (Touvron et al., 2023), GNNAVI achieves remarkable results with just 0.2% of the trainable parameters of the full model, consistently outperforming FPFT and other PEFT methods across various classification tasks. Additionally, we analyze the attention interaction between tokens and find that GNNAVI demonstrates a more stable and clear information aggregation process compared to FPFT.
In summary, our contributions are: i) We pro2We use “deep layers” to refer to the last few layers of the LLM. For instance, in GPT2-XL, there are 48 layers, with the last 12 layers considered as deep layers in our work.
pose a novel PEFT method, GNNAVI, inspired by the information flow perspective of LLMs. GNNAVI effectively navigates the information aggregation process in LLMs. ii) We apply GNNAVI to text classification tasks with few-shot training examples, outperforming baselines while updating only 0.2% to 0.5% of parameters. iii) Our work sheds light on the application of GNNs in NLP and provides novel insights for future research. To the best of our knowledge, we are the first to utilize GNNs to enhance the performance of LLMs from the information flow perspective.

heading: 2 Related Work
text: Prompt-Based Learning GPT-3 (Brown et al., 2020) has sparked interest in prompt-based learning methods, and particularly in the ICL paradigm. This surge in attention has fostered a multifaceted exploration into the factors influencing ICL performance, including input perturbation (Yoo et al., 2022; Min et al., 2022), selection of demonstration (Liu et al., 2022; Nie et al., 2023a), and calibration techniques (Zhao et al., 2021a; Nie et al., 2023b). Concurrently, there has been a deep dive into understanding the underlying mechanism of ICL, employing diverse theoretical frameworks such as gradient descent (Dai et al., 2023), Bayesian inference (Xie et al., 2022) and information flow (Wang et al., 2023). Following the route of ICL, prompt-based fine-tuning has emerged as an effective strategy in scenarios with limited data (Gao et al., 2021; Schick and Schütze, 2021a,b). We leverage insights from these investigations into the ICL mechanism and propose a tailored PEFT method for LLMs.
Parameter-Efficient Fine-Tuning (PEFT) PEFT focuses on enhancing language model performance on downstream tasks by optimizing a small number of parameters, instead of fine-tuning all parameters (Ding et al., 2023). Various PEFT strategies have been explored. Addition-based methods only train modules or parameters added to the model, such as Adapter (Houlsby et al., 2019), Prompt tuning (Lester et al., 2021), and Prefix tuning (Li and Liang, 2021). Specification-based methods selectively fine-tune specific parameters in the original model while keeping the remainder frozen, such as BitFiT (Ben Zaken et al., 2022). Reparameterization-based methods transform existing parameters into a more parameter-efficient form, such as LoRA (Hu et al., 2022). Recent
advancements in PEFT research have increasingly prioritized memory efficiency, aiming to enable the training of LLMs with minimal computational resources, such as MeZO (Malladi et al., 2023) and HiFT (Liu et al., 2024). Our proposed PEFT method is designed specifically for LLMs and draws upon the intricacies of how LLMs process and learn from prompts.
GNN for NLP GNNs are predominantly utilized in NLP tasks involving structural input, such as graph-to-text generation (Gardent et al., 2017) and graph-enhanced question answering (Zhang et al., 2022). Previous approaches employ GNNs to encode complex graph and node representations. For instance, Koncel-Kedziorski et al. (2019) introduced Graph Transformer, which extends graph attention networks (Veličković et al., 2018) for encoding scientific graph inputs, while Li et al. (2021) utilize GNNs to encode knowledge graphs and align them with text embeddings from pretrained language models. Additionally, GNNs serve as auxiliary tools for pretrained language models to encode complex structural information for AMRto-text generation (Ribeiro et al., 2021). Unlike prior work, we leverage GNNs for information aggregation based on the perspective of information flow.

heading: 3 Method

heading: 3.1 Architecture of GNNAVI
text: Intuition Wang et al. (2023) demonstrated that the working mechanism of LLM follows specific paths of information flow. The label words in the input prompt serve two roles for the final predictions: acting as information aggregators by gathering information from their preceding words and propagating the aggregated information to the last token position where the prediction is generated. Building upon their insights, we posit that navigating the flow of information aggregation can enhance both efficiency and effectiveness of LLMs. Leveraging the GNN’s proficiency in information aggregation at the graph level, we explore LLMs from a graph theory perspective and utilize GNN as a tool to guide the information flow.
Working Mechanism We illustrate the working mechanism of GNNAVI in Figure 2. For example, in a sentiment analysis task, the prompt comprises one demonstration from each class and the text to be classified. An LLM processes this prompt layer by layer. The GNN layer is inserted after the l-th decoder layer of the LLM3. Receiving the token
3In our preliminary experiments, GNNAVI performs optimally when the GNN layer is inserted in the last quarter of the layers in LLM. Thus, we add the GNN layer after the 42nd layer of GPT2-XL and after the 28th layer of Llama2-7b in our experiments. A detailed analysis is conducted in §6.1.
representations from the l-th layer, the GNN layer learns node representations by aggregating information from neighboring nodes. Subsequently, the node representations are propagated to the next layer in LLM as hidden states. The nodes are connected following the paths of information flow. As depicted in Figure 2(b), the label words ‘Positive’ and ‘Negative’ aggregate information from their preceding tokens and pass the information to the last token ‘:’ of the prompt. In case the label word is tokenized into subtokens, we use the first subtoken to serve as the label word, following previous work (Zhao et al., 2021b; Wang et al., 2023). We freeze the pretrained parameters of the LLM during training and update only the parameters in the GNN layer.
Graph Neural Network The graph neural network aggregates information from neighboring nodes to model graph and node representations by message passing. To formulate an NLP task on a graph level, we consider the input text as a graph. We define a directed graph G as a triple (V, E ,R) with a set of nodes V = {v1, . . . , vn} (one node for each token), a set of relation types R4, and a set of edges E of the form (v, r, v′) with v, v′ ∈ V , and r ∈ R. Each node vi is associated with a feature vector xi, which is the token representation of the i-th token in the l-th layer. In Figure 2, for instance, the first token ‘Review’ is connected with the label token ‘Positive’. This edge is represented by the triple (Review, aggregate,Positive), where aggregate denotes an edge directed towards a label node.
The node representations in GNN layer are updated by aggregating the information from neighboring nodes. The aggregation algorithms vary across different GNN architectures. For example, the learning process of Graph Convolutional Network (GCN) (Kipf and Welling, 2017) is formulatd as:
hv = σ  W ∑
v′∈N(v)
h (l)
v′
|N(v)|
  (1)
where hv denotes the updated node representation of v, h(l)v′ denotes the token representation of its neighbouring nodes from l-th decoder layer, σ is the activation function, W is the trainable parameter of GNN, N(v) includes all the neighbouring nodes of v.
4In our work, we only consider one relation type: the directed edge from node v to node v′.
We also include another GNN architecture, GraphSAGE (Hamilton et al., 2017), in our studies, which involves a more complex learning process:
hv = σ ( W ( h(l)v ⊕ AGG({h(l)v′ , ∀v ′ ∈ N(v)}) )) (2)
The concatenation function ⊕ concatenates aggregated information with the node current representation, and the aggregation function AGG compiles message passing from neighboring nodes using techniques such as mean, pool and LSTM.5 We visualize the information aggregation process of GNN in Figure 2(c).

heading: 3.2 Task Formulation
text: In our work, we implement prompt-based finetuning for text classification tasks. Our goal is to predict the correct class given a few examples. We reformulate the task as a language modeling problem. Let M be a language model with vocabulary V , and let L be a set of label words. The training set T consists of pairs (s, l), where s is a sequence of tokens from the vocabulary V and l is a label word from the set L. In a sentiment analysis task, for instance, we define a pattern P(s, l) which associates a text s =‘Nice performance’ and a label word l =‘Positive’ as follows:
Review: Nice performance. Sentiment: Positive
For a k-class classification task, we sample one demonstration per class from the training set T , and concatenate them with the text s to be classified to form the prompt X(s):
X(s) = P(s1, l1)⊕ . . .⊕P(sk, lk)⊕P(s, ε) (3)
⊕ denotes the concatenation of the input demonstrations and ε is the empty string. A more intuitive example is shown in Figure 2. The language model reads the prompt X(s) and predicts the next token l, which is the label assigned to s. M is initialized with pretrained parameters ϕ, and fine-tuned by minimizing the cross-entropy loss:
ℓ = − ∑
(s,l)∈T log pϕ(X(s), l) (4)
pϕ(., .) returns the probability which M assigns to the correct label l. In our work, we randomly select one demonstration per class to form the prompt and remove them from T . The training examples are then sampled from the remaining samples in T .
5We apply mean aggregation to GraphSAGE in this work.

heading: 4 Experiments

heading: 4.1 Datasets
text: We implement text classification tasks using five commonly used datasets from different domains, including SST-2: Stanford Sentiment Treebank Binary for sentiment analysis (Socher et al., 2013); EmoC: EmoContext for 4-label emotion classification (Chatterjee et al., 2019); TREC: Text REtrieval Conference Question Classification (TREC) for question type classification containing 6 types (Li and Roth, 2002; Hovy et al., 2001); Amazon: binary classification for Amazon reviews (McAuley and Leskovec, 2013); AGNews: AG’s news topic classification dataset for topic classification with 4 labels (Zhang et al., 2015).

heading: 4.2 Experimental Setting
text: The prompt is designed following the template in Equation 3. We take one demonstration per class to form the prompt6 and append the sample to be predicted at the end of the prompt. Following a fewshot learning setting, we experiment with different numbers of training samples, namely 5, 10, 20, 50, 100, and 200 samples per class. The training samples are randomly selected from the original training set. Another 1000 samples from the original training set are sampled as the validation set, and 1000 samples from the original test set are used for evaluation.7 The accuracy on the validation set is employed to identify the best-performing model, which is subsequently evaluated on the test set. We report the average accuracy over five random seeds. The hyperparameters can be found in Appendix A.

heading: 4.3 Models
text: As GNNAVI is built on the base of decoder-only LLMs, we select two large language models, both with over 1 billion parameters, and equip them with GNNAVI. Specifically, we choose GPT2-XL with 1.6 billion parameters (Radford et al., 2019) and Llama2 with 7 billion parameters (Touvron et al., 2023). For the GNN layer, we opt for GCN and GraphSAGE, denoted as GNNAVI-GCN and GNNAVI-SAGE in the experiments. To integrate GNNAVI with GPT2-XL and Llama2, we modify their source codes from Huggingface (Wolf et al., 2019) and utilize GNN models provided by PyTorch Geometric (Fey and Lenssen, 2019).
6The templates of prompts are presented in Appendix B. 7The original test set of SST-2 contains less than 1000 samples, so we keep the original test set for model evaluation.

heading: 4.4 Baselines
text: ICL one-shot per class: In-context learning (ICL) follows the scenario where the LLM is initialized with pre-trained parameters and instructed by demonstrations to perform text classification tasks. None of the model parameters are updated. We sample one demonstration per class to form the prompt. The demonstrations used to form the prompt are consistent with those used for other methods under the same random seed.
ICL few-shot per class: To compare with the low-data fine-tuning setting, we implement ICL with 5 additional shots per class as the demonstrations. This setting is comparable to a training set with a size of 5 samples per class. Due to the limited input length of GPT2-XL, AGNews and Amazon are set to 4 additional shots per class.
Low-Rank Adaptation (LoRA): LoRA is a PEFT method that reduces the number of trainable parameters by injecting trainable rank decomposition matrices into each layer of the LLM (Hu et al., 2022). We implement LoRA using the Python library PEFT (Mangrulkar et al., 2022).
Prefix-tuning (Prefix): Prefix-tuning utilizes a soft-prompt strategy, incorporating virtual tokens into the LLM and updating only the parameters of the virtual tokens (Li and Liang, 2021). We implement prefix-tuning using the PEFT library (Mangrulkar et al., 2022). The number of virtual tokens8 is set to maintain a comparable size of trainable parameters as for GNNAVI.
Adapter: We insert a standard adapter module after the feed-forward sub-layer of each layer in the LLM (Houlsby et al., 2019). The adapter module is added using AdapterHub (Pfeiffer et al., 2020; Poth et al., 2023).
Full Parameter Fine-tuning (FPFT): Full parameter fine-tuning is implemented as a strong baseline, where all the model parameters are updated during the training process.

heading: 5 Results
text: We report the results with 5 and 200 training examples in Table 1, which reflect the performance under the scenarios where only limited training examples are available and sufficient training examples are provided respectively. Full results are presented in Appendix C.
8The number of virtual tokens can be found in Appendix A.

heading: 5.1 Overall Performance
text: Observing the results of GPT2-XL, GNNAVI remarkably rivals ICL, FPFT, and other parameterefficient baselines. Under the low-data setting of 5 training examples, both GNNAVI-GCN and GNNAVI-SAGE outperform FPFT by over 13%, achieving higher accuracy than other PEFT methods by 0.4% to 21%. Increasing the number of training examples to 200, the average performance of GNNAVI improves to 89.64% and outperforms other baselines.
Similar to GPT2-XL, GNNAVI achieves the best performance with Llama2 among all the baselines. With only 5 training examples, GNNAVI-SAGE achieves 2.77% higher average accuracy than FPFT. Comparing with other PEFT methods, GNNAVI
shows higher average accuracy from 1.8% to 35%. And with 200 training examples, GNNAVI-GCN achieves 92.24% average accuracy, outperforming FPFT, Prefix-tuning, Adapter, and LoRA.

heading: 5.2 Efficiency Analysis
text: GNNAVI significantly reduces the number of trainable parameters compared to the baselines
for both GPT2-XL and Llama2. GNNAVI-GCN for GPT2-XL achieves the highest average accuracy with 5 training examples containing only 2.5 million trainable parameters, which is 615 times smaller than FPFT, six times smaller than Adapter, twice smaller than Prefix, and similar to LoRA. As for Llama2, GNNAVI saves over 6.6 billion trainable parameters compared to FPFT and achieves better results. GNNAVI-GCN also updates fewer parameters than Prefix and Adapter. Although LoRA contains fewer trainable parameters than GNNAVI-GCN in Llama2, the performance of LoRA cannot compete with GNNAVIGCN and GNNAVI-SAGE. Table 2 shows that by saving a significant amount of training parameters, GNNAVI-GCN speeds up the training process by a factor of up to 6 compared to FPFT.
5.3 Influence of Training Examples
Adding more training examples improves the accuracy for most baselines and GNNAVI. As depicted in Figure 3, GNNAVI consistently outperforms other methods as the number of training examples increases. While other methods also show improvement with more training examples,
the extent of improvement is not as consistent as for GNNAVI, particularly for Prefix and Adapter.
Figure 4 shows the performance of GNNAVI for the different tasks as a function of the number of training examples. We observe that the effect of adding training examples is similar for both GPT2XL and Llama2. Notably, adding more training examples yields significant improvements, especially in low-data settings (e.g. with 10, 20, and 50 training examples) where GNNAVI shows a substantial improvement, except for EmoC. However, the significance diminishes when more than 50 training examples are provided, the improvement is not as pronounced here as in low-data settings.

heading: 6 Ablation Study
text: In §6.1 of this section, we delve into the influence of the position where the GNN layer is inserted in the LLM. In §6.2, we investigate the effects of removing one of the information flow paths on performance. All of these studies are conducted using GNNAVI-SAGE with 5 training samples per class under the experimental settings outlined in §4.2.

heading: 6.1 Position of GNN Layer
text: The position where the GNN layer is inserted significantly impacts the model’s performance. Figure 5 illustrates the performance of GNNAVI when the GNN layer is inserted at different locations in GPT2-XL. With the exception of EmoC, all tasks exhibit lower performance when the GNN layer is added in the first 10 layers of GPT2-XL. Performance improves as the GNN is added in deeper layers, reaching peak accuracy around the 44th layer. Subsequently, accuracy declines until the
last layer. This trend may stem from the gradual initiation of the information flow process in the early layers of LLM, where the GNN’s influence is limited due to insufficient token interaction. Conversely, in the final layers, the information flow process is nearly complete, rendering it too late for the GNN to guide effectively. Despite variations in performance changes across tasks, the average performance suggests that the optimal placement for the GNN layer is between the 38th and 42nd layers for GPT2-XL.

heading: 6.2 Removal of Information Flow
text: We conduct an ablation study to investigate how removing specific information flow paths affects the results while retaining others. In our approach, we connect the label words to their preceding words to aggregate information and to the last token to distribute the information from the label words. These connections are referred to as the aggregation and distribution paths in the ablation study. As illustrated in Figure 6, we remove one path and retain another.
As shown in Table 3, both the aggregation and distribution paths contribute significantly to the performance. Removing either of them results in a decrease in the average accuracy across the five tasks. Except for the two binary classification tasks SST-2 and Amazon, removing the distribution path causes a greater drop in performance. Based on these results, we conclude that the distribution path
plays a more significant role in the information flow process, especially for tasks with more than two labels.

heading: 7 Further Discussion: Information Flow
text: While the attention mechanism in LLM offers an information flow perspective for interpreting the model’s working mechanism (Wang et al., 2023), it treats the input text as a fully connected graph. In contrast, GNNAVI explicitly connects the context tokens to the label tokens for information aggregation and the label tokens to the final token for information distribution. Thereby, the correct information flow is hardwired into the GNN. There is no need to learn it by adjusting the attention weights. To further investigate the differences in information flow between FPFT and GNNAVI, we utilize the saliency technique (Simonyan et al., 2013) for interpretation. Following the approach of Wang et al. (2023), we compute the saliency score for each element of the attention matrix using a Taylor expansion (Michel et al., 2019):
Il = ∑
h
∣∣∣∣A⊤h,l ∂L(x)
∂Ah,l
∣∣∣∣ , (5)
where Ah,l represents the attention matrix of the h-th attention head in the l-th layer. x is the input, and L(x) is the loss function. The saliency matrix Il for the l-th layer is obtained by averaging the values across all attention heads. Each element Il(i, j) of the matrix denotes the significance of the information flow from the j-th word to the i-th word in the prompt.
We employ three quantitative metrics to assess the information flow: Sagg measures the information flow of the aggregation path from previous context words to label words, Sdist measures the information distribution from label words to the last token, and Srest accounts for other information flow between remaining words excluding Sagg and Sdist. The average significance of information flow can be formulated as:
S =
∑ (i,j)∈C Il(i, j)
|C| , (6)
where C is the total number of token interactions involved.9
As depicted in Figure 7, the information flow of GNNAVI appears more stable compared to FPFT.
9The full formulas of Sagg , Sdist, and Srest can be found in Appendix D.
In FPFT, without guided navigation, tokens interact with every preceding word, leading to a trend of confusion between the information flow Sdist and Srest. This indicates a struggle to identify the ‘right’ information for final prediction. Conversely, GNNAVI adheres to the information flow guided by the GNN, resulting in stable curves that depict a consistent information aggregation process, aligning with the findings of Wang et al. (2023). Compared to FPFT, the stable curves affirm that GNNAVI serves as a navigator, ensuring the information flows in predefined directions.

heading: 8 Conclusion
text: In this work, we propose a novel PEFT method, GNNAVI, leveraging GNN to navigate information flow within LLMs. Specifically tailored for prompt-based fine-tuning, GNNAVI significantly reduces the number of trainable parameters by simply adding a GNN layer into LLMs to guide the information flow within the prompt. GNNAVI outperforms FPFT and other PEFT methods across various classification tasks, even with few training examples. Our work offers insights into handling LLMs from a graph perspective and presents
a novel application of GNNs in NLP. Future work could explore different token connectivities for GNNs or utilize GNNs to control the information flow in LLMs.
Limitation
Although GNNAVI introduces a novel insight for NLP research, there are several limitations in our work. Firstly, GNNAVI is susceptible to the quality of the demonstrations. We find that its performance heavily relies on the selection of demonstrations when only a few training examples are available. However, this issue is alleviated with an increase in the number of training examples. Secondly, while GNNAVI builds upon the information flow of LLMs, it offers a more transparent working mechanism. However, as a black-box model, the working mechanism of the GNN layer is not investigated in this work. Thirdly, we only evaluated the performance of GNNAVI on text classification tasks, other NLP tasks are not explored in this study. We leave these limitations for future work.
Ethics Statement
This study adhered to the ACM Code of Ethics. The datasets employed in our research are publicly accessible, and we utilized them solely for the purpose of evaluating our models. Any potential inaccuracies in the datasets are beyond our responsibility.

heading: Acknowledgements
text: We want to express our gratitude to all anonymous reviewers for their invaluable contributions and constructive feedback. This work was supported by Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), German Research Foundation (DFG) grant (SCHU 2246/14-1), Munich Center for Machine Learning (MCML), and China Scholarship Council (CSC). Besides, we would like to thank Johanna Reiml for her valuable help in running additional experiments.

heading: A Hyperparameters
text: We present the hyperparameters for GNNAVI and other baselines in Table 4. The models were trained using NVIDIA A100-SXM4-40GB GPUs. Due to limited resources, the batch size was set to 1, and full parameter fine-tuning of Llama2 was implemented using 8 bits. We observed that for Llama2, GNNAVI and other PEFT methods were sensitive to the selection of prompts with very few training samples, and thus could not achieve optimal performance. To address this, we replaced these results by using another random seed to change the demonstrations in the prompt.

heading: B Demonstration Templates and Label Words
text: The templates for the prompt are presented in Table 5. [S] denotes the demonstration selected to form the prompt, [L] represents the label word of the demonstration, and [Si] denotes the sample to be predicted.

heading: C Full Results
text: Due to space constraints, the complete results are provided in Table 6. Each value in the table represents the average accuracy over five experiments conducted with different random seeds.

heading: D Formula of Saliency Score
text: We utilize l to denote the label words, such as ‘Positive’ and ‘Negative’, while f represents the final token, such as ‘:’. Additionally, t denotes other tokens excluding label and final tokens. Sagg calculates the mean significance of information flow from the previous context words to label words:
Sagg =
∑ (i,j)∈Ctl Il(i, j)
|Ctl| ,
Ctl = {(lk, j) : k ∈ [1, C], j < lk} . (7)
Sdist calculates the mean significance of information flow from the label words to the final token:
Sdist =
∑ (i,j)∈Clf Il(i, j)
|Clf | ,
Clf = {(f, lk) : k ∈ [1, C]} . (8)
Srest calculates the mean significance of information flow among the rest words, excluding Sagg
and Sdist:
Srest =
∑ (i,j)∈Ctt Il(i, j)
|Ctt| ,
Ctt = {(i, j) : j < i} − Ctl − Clf . (9)

heading: Future Work
text: Future work could explore different token connectivities for GNNs or utilize GNNs to control the information flow in LLMs. Limitation
Although GNNAVI introduces a novel insight for NLP research, there are several limitations in our work. Firstly, GNNAVI is susceptible to the quality of the demonstrations. We find that its performance heavily relies on the selection of demonstrations when only a few training examples are available. However, this issue is alleviated with an increase in the number of training examples. Secondly, while GNNAVI builds upon the information flow of LLMs, it offers a more transparent working mechanism. However, as a black-box model, the working mechanism of the GNN layer is not investigated in this work. Thirdly, we only evaluated the performance of GNNAVI on text classification tasks, other NLP tasks are not explored in this study. We leave these limitations for future work