RAGraph: A General Retrieval-Augmented Graph Learning Framework
Authors: Xinke Jiang
Rihong Qiu
Yongxin Xu
WentaoZhang
Yichen Zhu
Ruizhe zhang
Yuchen Fang
Xu Chu
Junfeng Zhao
Yasha Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph Neural Networks
Graph Prompt Tuning
Retrieval-Augmented Generation
Primary area: graph_neural_networks

Abstract: Graph Neural Networks (GNNs) have become essential in interpreting relational data across various domains, yet, they often struggle to generalize to unseen graph data that differs markedly from training instances. In this paper, we introduce a novel framework called General Retrieval-Augmented Graph Learning (RAGraph), which brings external graph data into the general graph foundation model to improve model generalization on unseen scenarios. On the top of our framework is a toy graph vector library that we established, which captures key attributes, such as features and task-specific label information. During inference, the RAGraph adeptly retrieves similar toy graphs based on key similarities in downstream tasks, integrating the retrieved data to enrich the learning context via the message-passing prompting mechanism. Our extensive experimental evaluations demonstrate that RAGraph significantly outperforms state-of-the-art graph learning methods in multiple tasks such as node classification, link prediction, and graph classification across both dynamic and static datasets. Furthermore, extensive testing confirms that RAGraph consistently maintains high performance without the need for task-specific fine-tuning, highlighting its adaptability, robustness, and broad applicability.

RAGRAPH: A General Retrieval-Augmented Graph Learning Framework
Xinke Jiang**
Rihhnn Qiu Yongxin Xu Wentao Zhang Yichen Zhu Ruizhe Zhang
Yuchen Fang", Xu Chu Junfeng Zhao Yasha Wang**
(xinkejiang, rihongqiu, xuyx, ruizhezhang) @stu. pku. edu. cn com
{wentaozh2001, yichenzhu2014,
a {chu_xu, zhaojf, wangyashaa edu.
Key Laboratory of High Confidence Software Technologies (Peking University),
School of Computer Science, Peking University, China
No Affiliation, University of Electronic Science and Technology of China
Center Frontiers of Computing Studies, Peking University, Beijing, China
Big Data Technology Research Center, Nanhu Laboratory, Jiaxing, China
* Peking University Information Technology Institute, Tianjin Binhai, China
Q
Puttps//github htttps///gttab..cco//rttess///// com//Artessay//AAraap
Abstract
Graph Neural Networks (GNNs) have become essential in interpreting relational data across various domains, yet, they often struggle to generalize to unseen graph data that differs markedly from training instances. In this paper, we introduce novel framework called General netrieval-Augmenne which brings external graph data into the general graph foundation model improve model generalization on unseen scenarios. On the top of our framework a toy graph vector library that established, which captures key attributes. such features and task-specific label information. During inference, the R RAGRAPH adeptly retrieves similar toy graphs based key similarities in downstream tasks, integrating the retrieved data to enrich the learning context via the message- paangggg prompting mechanism. Our extensive experimental evaluations demonstrate that RAGRAPH significantly outperforms state-of fate-of-the-art the-hrr-tt graph learning methods in multiple tasks such as node classification, link prediction, and graph classification across both dynamic and static datasets. Furthermore, extensive testing confirms that RA- GRAPH consistently maintains high performance without the need for task-specific fine-tuning highlighting its adaptability, robustness, and broad applicability.
Graph Learning (R (RAGRAPH),
Introduction
Graph Neural Networks (GNNs) [5. 50, 98, 65, 129, 128, 127] have recently burgeoned a surge of interest in both academic and industry communities due to their robust capability to model complex, real-world data in diverse domains, including societal [74, 57, 81], biochemical [17, 1 12, 108], and traffic-related [56, 24, 46, 22, 21] fields and etc [55, 38, 70, 15, 26, 25]. Utilizing a message-passing mechanism [50, 30), GNNs have transcended traditional node embedding approaches [29, 80, 96], en- abling the capture of intricate relationships within data through sophisticated architectures and advanced graph representation learning techniques [50, 52, 56, 18, 98]. However, the challenge of generalizing GNNs across different modalities. domains [64, 63], and tasks remains largely unexplored [58, 1 14]. This is in stark contrast to the significant successes of large models such as GPT's [75, 76] in NL.P and Sora [66] in CV, presenting crucial frontier for further research and realms for graph data generalizing.
Indicates equal contribution.
Yasha Wang and Junfeng Zhao are corresponding authors.
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

In graph learning tasks, providing the necessary context is crucial for graph generaliza- tion [132. 53, 39, 1371, i.e, retrieve similar shopping context illustrated in Figure 1 (c). Therefore, our insight is to enhance the model's generalization ability and prediction accuracy by retrieving necessary contexts during graph learning through retrieval. Retrieval-Augmented Generation (RAG) represents prominent methodology, significantly augmenting language model functionalities through the integration of dynamic retrieval mechanism during the generation process [138, 78] (e.g., person asks what animal it is, and we use some visual [141] or text retrieval [2] methods to retrieve more descriptive features or even the wanted category). RAG enriches not only accurate and reliable content but also reduces factual errors, addressing challenges such as incorrect answers, hallucinations, and limited interpretability in knowledge-intense the need for updating model parameters and could be generalized even in unseen scenarios.
tasks [42, 2. 1], obviating
However, how to enable retrieval -aggevaee--gmeeeeedde generation for graph learning, i.e. retrieving the user's historical purchasing behavior to enhance recommendation ability [31, 14, 36] and identifying fraud crimes by searching for similar fraudulent relationship behaviors [86, 65]. still remains unexplored and faces the following challenges CI& C2.
CL. The first challenge is how to leverage the retrieved context i.e, features (X) and labels (Y) into the GNNs model under dynamic changing scenarios. Previous studies, such as PRODIGY [39], have adopted the concept of in-context learning (ICL) by constructing consistent and static task graphs for each specific task or dataset. These task graphs determine labels through the calculation of similarities using hidden vectors, employing few-shot learning approach. However, PRODIGY's reliance on fixed set of examples as rules may not sufficiently address and generalize the variety of scenarios encountered real-world settings, which is particularly problematic in dynamically changing environments, as the system focuses primarily on teaching the direct mapping paradigm from inputs to outputs (X + Y), rather than truly integrate the input (X) and output (Y) data into the analysis. In contrast to RAG, PRODIGY struggles to incorporate external information (X and related to data nodes, which is crucial for enriching the learning process graph- based systems.
C2. Moreover, it is challenging to develop tune-free prompt mechanism to support retrieved knowledge and be applicable to seamlessly switch unseen scenarios and multi-tasks Numerous initiatives have been undertaken in the realm of graph pre-training [34, 1 117, 35, 82, 99, 37, 7, 89, 126]. however, the challenge persists in designing a alug-and-plaa RAG module that can seamlessly interface with already pre-trained models. Insights derived from prior investigations into the graph prompt [9, 27, 91, 67, 114, 20, 95], the knowledge obtained by RAG can facilitated and injected into prompt via plug-and-pen manner.
ad- dress these two challenges previously mentioned, we put forward the General Retrieval- Augmented Graph Learning Frame- work (RAGRApH).
For endeavoring
Query
Query
Query
Which mammal catches mice, enjoys eating fish and has tail?
Please help impaint the picture.
What will this user buy next?
RAG
RAG
RAG
Cat, maammal with L primarily foeds mice.
Similar photos:
Drawing inspiration from the success of RAG on LLMs [138] and the ICL on GNNs [39] (we detail the difference
Answer
Answer
Answer
Maybe
cat.
T-shirt.
between RAG and ICL in Appendix E), we constructed a toy graphs vector library by chunking from resource graphs, where the library key stores key information, including environmental, historical, structural, and semantic details, while
(a) NLP
(b) CV
(0)
GNNS
Figure I: (a) RAG in NLP utilizes retrieval to enhance model responses, based query retrieve related features (e.g., e., tail, primarily feeds on mice) and answers (e.g., Cat). (b) In CV, RAG employs similar photo retrieval to enhance model comprehension, assisting in downstream tasks such as inpainting or image question answering. (c) For GNNs, RAG could leverage retrieval of similar historical subgraphs or scenarios aid in graph-based tasks (e.g.. recommendations or fraud detection).
node features and label information (task-specific output vector) are stored as values. For downstream tasks, the key value of the query node would be leveraged retrieve toy graphs by the key similarities.

---

and the stored features (X) and labels (Y) would be aggregated structurally to provide essential knowledge to the query node, instead of the mapping paradigm, to address challenge CI. In prompt mechanism design, we start by transferring features and task-specifer output from the toy graphs to their master nodes (the central node of the toy graph) via pessage-passing Subsequently, features from the master nodes and the query node's s neighbors are aggregated to the query node, along with the task-specific output from master nodes. This process could be parameter-free, indicating that our model can be applied across different tasks and datasets without the need to fine-tune for downstream tasks, effectively addressing challenge C2.
In summary, our contributions are listed as follows:
To the best of our knowledge, our proposed framework, RAGRAPHI is the first to integrate RAG with pre-trained GNNs. By constructing a key-value vector library for toy graphs, RAGRAPH facilitates explicit plug-and-play access to pre-trained GNNs, achieving commendable performance even without tnnettuning demonstrating its superiority on cross-task and cross-dataset capabilities.
Our RAGRAPII employs classic paasaae--assingg mechanism and introduces well-designed prompt mechanism to integrate knowledge. This approach effectively incorporates the retrieved knowledge X and Y from toy graphs, into the pre-trained GNNs model, enhancing the accuracy and relevance of the model's 's outputs.
We have extensively tested R RAGRAPH on both static and dynamic graphs across multiple levels of graph tasks (node, edge, and graph). The results validate the effectiveness of our model, showing significant improvements
sfate-of-the art baselines in both fine-tuned and tuning- ouning-free scenarios, particularly in cross dataset validations.
2 Related Work
2.1 Retrieval-Augmenned Generation on Large Language Models
RAG integrates an external knowledge retrieval component and through prompt engineering into pre- trained language models to enhance factual consistency, thus improving the reliability and interpretabil- ity of LLM responses [134, 138, 51, 23, 45, 119, 59, 111, 130]. Traditional RAG approaches utilize retriever models to source relevant documents from extensive knowledge corpora [107, 83, 71, 49), which are then processed further by reader models-primarily LLMs [77, 85). Furthermore, several studies focus on fine-tuning reader LLMs by applying prompt-tuning with retrieved knowledge or using RAG API calls [69, 42, 2, 116, 102, 131, 62]. While RAG has seen considerable success in the NLP field, it has also been applied to tasks involving joint visual and text retrieval [141, 61, 60, 8. 125, 10], code retrieval [68, 1 136], audio retrieval [6. 32] and video retrieval [3, 101]. Although there have been applications of RAG on structured data such as KG-RAG for knowledge graphs [48, 45, 87, 93, 94, 40), these primarily leverage the text information of knowledge graph nodes enhance language or graph models. In contrast, there are no significant studies utilizzig RAG on structured graphs without text information enhance pre-trained GNNs. Our work aims to extend this successful approach similarly to graph data, to enhance the capabilities of pre-trained GNNs, and can be adapted to various tasks and across different graphs without additional fine-tuning by integrating a plug-and-play RAG module.
2.2 Graph Prompt Learning
Inspired by the application of pre-training models [75, 76) and prompt learning [103, 136, 43] in NLP, recently, learning on the graph has been divided into pre-training models on large-scale graph data [34, 17,35, 82, 39, 133, 105, 90, 120, 122, 123, 121, 124], with or without labels, followed by fine-tuning model parameters via prompts for diverse downstream tasks [67, 114, 39, 140, 90, 95]. The adoption of prompting mechanisms in graph learning represents a promising avenue to overcome the constraints of traditional graph representation methods, striking balance between flexibility and expressiveness [92]. For instance, VNT [95] utilizes virtual nodes as prompts to refine the application of pre-trained graph models. GraphPrompt [67] introduces a task-specific readout mechanism to tailor models for various tasks, while GraphPro [1 14] implements spatial- and cemporal-based gating mechanisms suited for dynamic recommendation systems. Furthermore, PRODIGY [39] constructs task graphs (prompts) and data graphs to enhance the model's ICL capabilities. Leveraging the successes in graph prompt learning, we aim to inject retrieved knowledge via prompt into pre-trained GNNs to support downstream tasks.
3

---

3 Preliminaries
In RAGRAPH, we focus RAG on multi-level graph tasks. For consistency, we define the graphs as dynamic graphs, considering static graphs as the special cases within this framework. The subsequent definition provides detailed description of toy graphs, including the definitions of keys and values utilized in RAGRAPH. Additionally, inspired by GraphPrompt [67], we have unified node-level, edge- level, and graph-level tasks into cohesive framework, and employ query graphs to tackle downstream tasks with precision.
Definition 1. (Dynamic Graph) Let denote dynamic graph comprising sequence of graph snapshots, each represented as static graph G (V V V== Vi= (v....,, defines the combined set of nodes across all snapshots and 8 U E, CV x V is the edge set, where V2 and F, represent the nodes and edges of the 1-th snapshot, respectively. Feature matrix X; = {x+ E V} E Rnxd contains the feature vectors for the nodes in the t-th snapshot, where d is the feature dimension. A, denotes the edge weight matrix at time t. where edge weight A E (0.1) if U1 E V; and vy) E E,, and 0 otherwise. Furthermore, Y, represents the task-related labels associated with nodes, edges, or the graph at time L. Note that graph is static if T = I and for consistency in terminology, we unify static graphs as particular instance of dynamic graphs.
Definition 2. (Toy Graph Vector Base) Let G !IG JE= denote dynamic resource graph. We chunk G into snapshots and take each node in G as the master node Um of the corresponding toy graph, and then store um with its neighbors within k hops as subgraphs. Data augmentation techniques [56, 135] such as node dropout, edge dropout, and random noise addition are employed on subgraphs to enhance the robustness and variability when generating each toy graph G* (c.f. Section 4. for details). Each toy graph GT CGR
RR is associated with a specific timestamp and master node e. E V. with each toy graph's 's scale being considerably smaller in scale compared to their corresponding GR. O Toy graphs can be retrieved using keys that include the timestamp t, the hidden embedding of the master node hT f (e.g,, embedded by pre-trained GNNs in RAGRAPE APH), the environmental key (e.g., the neighbors set V (( {wt A, [m.i] >> 0,eo EGT) }) and the structure-based position-aware code sn (cf. Appendix C.2 for details). @ By retrieving based on key similarity (c.f. Section 4.2 for details), we can obtain the required values of GT, ..e. task- specific output vector {oī E eRf E GT} and hidden embeddings (hī E ( |e; GT) of the master node and its neighbors, where Si and f2 represent the dimensions. Finally, we denote the key-value vector base for the toy graph as GT.
Definition 3. (A Unified Graph Task Definition) Given dynamic graph G, can be divided into training and testing subsets, i.e. G Gurain Ugvest based on either snapshot or node set partitioning. The label y; of a node v, edge (v,,0y) ) or subgraph G, can be observed only they belong to Girain- The objective of label prediction is to predict test labels Ytest EGest- Following GraphPrompt [67], we unify the three types graph learning tasks (node-level, edge-level, and graph-level) into a single framework via similarity comparison sim(-,) of the task-specifi output vector (abbreviated O where each entry is o) with the ground- -rouh (i.e., the one-hot vector or the prototype embedding under few-shot setting). It's noted that can be either low-dimensions predicted classes) under normal settings [50, 129]. or high- gigh-dimensional under few-shot settings [67] or in link prediction tasks [114, 31]. In our experiment, 1 for node-level and graph-level tasks, the downstream tasks are given in few-shot settings following [67]] For node / graph classification on a node/ / graph set, let C be the set of classes with y C denoting the class label of node / graph. For each node / graph class, the class prototypical output vector is calculated by the mean value of the k-shot set D: d. - ((.. (E. The class y, of the node or graph determined by calculating similarity with the class prototype as: y = argmax.gcsimmo
(with the dimension equal to the number of
@ For edge-level tasks, to predict a link between nodes v; and v,, if (1,.((1,0)) E Eraan E Guain and sim(o > sim(o.,o; +e, we regard (1), as linked. Following PRODIGY [39] and GraphPrompt [67], we also apply query graph Go that includes the center node and its neighbors within & hops. Specifically, for geaph- task, we apply a full-link virtual node as the center node inside the query graph G

---

RAGRAPH Framework
In this section, we introduce RAGRAPH, general and novel aurieval-augmentt graph learning framework that can operate on arbitrary graphs with or without additional fine-tuning, as illustrated in Figure Initially, Section 4.1, we elucidate the methodology for constructing the Resource Toy Graphs. Subsequently, in Section 4.2 we detail the Toy Graphs Retrieval Process. Finally, the Training and Inference processes are elaborated Section 4.3, which utilize retrieved toy graphs from two propagation views-intra and inter-proe preee--ropggaiin---o and handle two types of information: hidden embeddings and task-specific output vectors two techniques (noisy trainable approach or parameter- free approach). The main notations of RAGRAPH For enhanced clarity, the Toy Graph Construction is outlined in Algorithm 1 (cf. Appendix C.5) and the Training and Inference with Toy Graphs Retrieval are detailed in Algorithm 2 (cf. Appendix C.5). Moreover, in Appendix C.4, we thcoretically prove the effectiveness applying RAG on GNNs from the perspective of mutual information gain.
summarized in Table 3, Appendix A.
Toy Graphs Embedding Pipeline
Resource Graph
Graphs
Pretrained Graph Model (Frozen)
Resource Toy Graphs (Key-Values) Base
Task-rpocific Output Vector
Chunk
Feed
(Vilusm)
3000
Hidden
Embeddiing
Enviromment [Histoy __
Poritione aware / Hidden Embodding (Keys)
Store Keys
Toy Graphs Retrieval Process
Learnable
Hidden Emchaldingg
Query Graph
COO
Feed
Propogate
GO
Topk
Joint Search
Q
Retrieve Toy Gruphs Glook
Fetch Keys
History Information (f... Enstroument
OutpustVie.
Nade
Connect
foily e Mie2)
Tyy Greph Isto Propaas
Training Inference
Node
Propegate
Graph-Level
OutoutPes Propogation
O
WildenEmbe
Gragh Protscype
Encoder
1
Propogation
OutputVee.
Oc
Node-Level
Decoder
Protutype OutgutVic..
Dc
Wrighted
Edge-Level
Add
Ouquut
Similarit
Other Nade
0j
0g (Loomoblee
Féc.
OutputVee.
Edge Weighr
(hrn--yy-Crgg Fne Propogale
Figure The overall framework of RAGRAPH. Given resource graph G we chunk it and augment toy graphs (GT), }, and feed them into pre- pre-trained GNNs to generate hidden embeddings via the encoder and task-specific output vectors via decoder, which are stored as values. Keys such as environment, history, position-aware, and hidden embeddings
stored to form the key-value database of toy graphs g For given query graph GO, the keys are fetched to retrieve the lopK toy graphs G from performed to propagate hidden
the database. ① Leveraging GT intra- and inter pitrr-propagation embeddings and task task-specific output vectors to pass retrieved knowledge to center node Ue. Through a weighted fusion, the aggregated output used to perform graph-, node- and edge-level tasks.
4.1 Toy Graphs Embedding Pipeline
In graph-based learning, nodes with higher sannenttiity-typiccaa with higher degrees-ofien hold more significance, meaning their information more extensively learned during graph-pre-traine processes. Conversely, less important nodes- -thse in the long tail oftenften have their features over-

---

looked. This issue is particularly pronounced in LLMs performing RAG, where the predominance of common knowledge overshadows the long-tail knowledge that RAG is meant to leverage. To tackle this, we construct toy graphs using an inverse importance sampling strategy, thereby countering this bias by preferentially sampling and augmenting toy graphs that accentuate the long-tail knowledge.
Inverse Importance Sampling Strategy. To achieve this, we calculate each node's 's importance I (v) for node v E R by combining PageRank (R(v) and Degree Centrality DC(() using the formula I(c)= + a))C(v), where E (0,1) is the balance weight. We reverse the node importance
(((e.)
with / %(v)+e** C O, O, normalize it to obtain node v 's sampling probabilitiess p. Σح "e,,"" and
Pt)) to prioritize nodes with higher sampling probability (lower importance) according to ps. When sampling. for each master node Um we generate its k-hop neighbors, termed an ego net G (vm) Given the constrained size of the resource graph, we adopt data augmentation techniques commonly used contrastive learning [56, 1 118, 117] to enhance the representativeness and diversity of the resultant toy graphs.
perform weighted sampling function WEIGHTEDSAMPLIINGG
Toy Graphs Augmentation Strategy. For augmentation. we first calculate the average reversed importance I' (G )) of the nodes within ego graph as I' (Um)) (ds(v..) Ever, (vm) ('()..
which then determines the number of augmentations aug GG (fm)) = KK- I[G (fm))) ]. where K a scaling constant that adjusts the intensity of the augmentation. For node Ug E G. (um), the augmentation techniques DATAAUGMENTATIONGS (Um), Raug) employed include:
Node Dropout: &s E G. (um) ) has a probability of being dropped: p(u; being dropped) =1
@ Addition of Gaussian Noise: we add gaussian noise to node features as augmentation X' X'(e;)=
X (O,,".)
Node Interpolation: a new node feature X' (Unnew) is created by linearly combining the features of two existing nodes v, and By, calculated as X' (Unew) ) AX(e;) + (1 (1- (1;)),
GT. And the edge weight between the new node Unew and node U is updated to XA[i.j]] and node vy is
(I --))A[iij] accordingly [109].
Edge Rewriting: we alter connections based on the average of the involved nodes' sampling probabilities, expressed as p( u(; being rewired)= = Pe
Key-Value Pairs Construction. After completing the sampling and augmentation procedures, the generated toy graphs are transformed into key-value pairs for storage [1]0]. Specifically, we collect each master node's um historical information (timestamps r), environmental information (neighbors (u )), structural encodings s,, (as described in the Appendix C.2), and the hidden embeddings A (obtained by processing the toy graph through the frozen pre- pre-trained GNNs) and store them as keys the master node Um of the toy graph. Additionally, we store task task-specific output vectors {oī [u G and hidden embeddings (h] c; E G*) as values at each node of the toy graph. For storage of these key-value pairs, we utilize the FAISS vector library [14] to facilitate accelerated retrieval and storage.
4.2 Toy Graphs Retrieval Process
After constructing the key-value toy graphs vector database, we proceed with the retrieval process for sub-tasks according to the four sub-simila aub-similarities between the key values of the master node Um in the toy graph and the center node v2 in the query graph, as detailed in Appendix C.3. The final similarity score a weighted combination of these factors, and the topK toy graphs selected as the retrieval results:
S(Ve,Um) = wxx.k x SSime ), (v,)Um) Snnvirommmn ( S (Vorum))?
(I)
where == =I [ M22 W,, wa] environment, and semantic similarities, respectively. Using this composite similarity, we rank and retrieve the topk toy graphs:
w3] are the
weights attributed to the time, structure,
TTox=Top
where GTog represents the subset of toy graphs that best match the query based on the combined criteria. This process ensures that we retrieve the most relevant toy graphs based on comprehensive similarity measure, incorporating historical, structural, and environmental information.

---

4.3
Training and Inference
In Section 1.3.1, we detail the Knowledge Injection Propagation process, which includes two distinct propagation manners. Next, in Section 4.3.2, we present our approach for combining the retrieved hidden embeddings with the task- task-specific output vectors. Additionally, enhance the robustness of RAGRAPPE noise-based prompt tuning strategy is introduced in Section 4.3.3.
4.3.1 Knowledge Injection Propagation
After retrieving the topK toy graphs Tup knowledge, specifically the task-specific output vectors O and hidden embeddings H, is propagated from these toy graphs to the master nodes (Toy Graph Intra Propagation) and then to the center node ee (Query-Toy Graph Inter Propagation). This propagation utilizes message-passee passing mechanisms via GNNs (cf. Appendix C.I). Each master node um in the toy graphs is connected to the center node 2 of the query graph based on the similarity scores (() computed in Eq.(1) and the connection weights dictate the influence of each toy graph, ensuring that graphs with higher similarity have a more substantial impact. This process can be implemented using either a parameter-fire or learnable approach. Moreover, it is worth noting that for learnable methods, the parameters of GNN are different.
Toy Graph Intra Propagation Within each toy graph, information z is propagated from neighbors the master node using pre-trained GNNs. The task- sees--spcific output vectors o and hidden embeddings h from the neighbors are aggregated and transmitted to the master node. For each node in toy graph GT, the GNN aggregates information from its neighbors (us)) to update the master node em.
Zm = GNN (fz 8 (V(U))))
(3)
where z; and Zm represent the hidden embeddings hi, or task-specific output vectors 0. of the neighbor nodes and master node, respectively. For parameter-free situations, we can prepare Z. in advance when constructing the toy graph to improve inference efficiency.
Query-Toy Graph Inter Propagation Next, information from the toy graphs is aggregated to the query graph. Specifically, during propagation, information z from the neighbors and master node of the toy graph is propagated to the center node using the same pre-trained GNNs. For a center node v. in the query graph G the GNN aggregates hidden embeddings H from its neighbors N (vx) and the master node U. from the toy graph:
h = GNN( , (NN.))
When propagating the task-specific output vector O only the master node's information is passed to the center node:
(5) For scenarios where the propagation mechanism is learnable, attention mechanisms can be adapted on the edges. In parameter- free scenarios where there are no lcarnable weights the attention on the edges is determined based on the edge weights from the previous resource graph.
O oz= GNN((o; 1; E [pm)))
4.3.2 Knowledge Fusion Layer
Finally, at the data fusion layer, the aggregated hidden embeddings H of the center node Ve are processed through the pre-trained GNN's decoder DECODER(-) to obtain output vector O. This output vector is then combined with the aggregated task-specific output vector in a weighted manner to produce the final output for downstream tasks illustrated in Definition 3. The combined output is formulated as follows:
ô 70= - (1-:)=++(())CCDDE((( DECODER(he),
(6)
where 7 is a reweighting graph-, or edge-level tasks via a similarity function.
The resulting vector ô, is then utilized to perform node-,
For the same task, the decoder can be directly used generate outputs. For different tasks, the decoder can be masked, allowing the model to utilize pre-computed embeddings without additional training. Furthermore, the decoder can be fine-tuned to better meet the specific requirements of each task, providing both flexibility and optimized performance. This approach ensures that the model effectively integrates and leverages information from both the toy graphs and the query graph, enhancing its effectiveness in various downstream tasks through the use of the aggregated task-specific output vector.
7

---

4.3.3 Noise-based Graph Prompting Tuning
When prompt tuning, RAGRAPHI employs the same prompt loss function Cpromps the backbone model (e.g., GraphPro, GraphPrompt). However, to mitigate the challenge of noise retrieval- common issue in traditional RAG where highly related but irrelevant data is often retrieved we enhance the training process by incorporating noise data to bolster model robustness, motivated by [55]. Specifically, we implement two types of noise integration strategies;
Toy--rTh Noise: This strategy involves artificially introducing irrelevant nodes (ny & (vm)) into the toy graph during its construction, complementing other augmentation techniques. Toy-Graph Noise: Throughout the training phase, we not only retrieve the top toy graphs that are most relevant but also deliberately include the bottomK toy graphs to incorporate noise knowledge.
The integration of these noise elements is intended to enhance the model's 's ability to distinguish relevant information from irrelevant information, significantly improving its robustness and overall performance in downstream tasks by noise training. However, during the inference stage, we do not incorporate the noise.
5 Experiments
In this section, we conduct series of experiments to evaluate the performance of RAGRAPII against state-of-the-and baselines on three dynamic and five static datasets on three-level graph tasks. Further details and experiment results are provided in Appendix D.
5.1 Experimental Setup
Datasets. We use four static datasets PROTEINS, COX2, ENZYMES and BZR for graph classification and node classification, as well as three dynamic datasets TAOBAO, KOUBEI and AMAZON for link prediction. More details about these datasets can be found in Table 4 in Appendix D.I.
Methods and Baselines. We consider three versions of our proposed framework RAGRAPII: I) RAGRAPI//EE, which indicates we utilize the plug-and-plag RAGRAPII without fine-tuning the train set; 2) RAGRAPH/FT, which employs prompt tuning on the train set with RAG; and 3)
RAGRAPH/NFT, which applies noise prompt tuning on the train set with RAG. For the baseline of the
dynamic graph, we choose LightGCN [31], SGL [104], MixGCF [41], SimGCL [118] GraphPro [114] and CraphPro+
[39]. For the static graph, we choose GCN [50], GraphS [30],GAT [98), GIN [106], GraphPrompt [67], GraphPrompt+PROII
[39] as baselines. In addition, we denote '/NF' and '/FT' respectively represent without fine-tuning and fine-tuning. A detailed description of baselines can be referred to in Appendix D.3.
Settings and Evaluation. We establish training- resource split with the remainder of the data reserved as unseen during fine-tuning. For static graphs, the split is based on node partitioning with the ratio of 50%:30% [67], while for dynamic graphs, it is based on partitioning snapshots with the history snapshots as resource graph [39]. For fair comparisons, for methods employing PRODIGY and fine-tune models using the training while retrieving the resource graph to prevent information leakage and over-fitting: @ when testing, we retrieve the combined training and resource graphs. For other methods, fine-tuning was directly performed the combined train and resource set for faimess. For the evaluation of static graphs, we refer GraphPrompt, utilizing pra-trdi GNNs for both node- and graph-level tasks within a k-shot classification framework. For dynamic graphs, we follow GraphPro to employ pre-trained GNNs on a substantial dataset fraction, with fine-tuning and testing conducted on later snapshots. Moreover, we pre-train GraphPro and GraphPrompt unsupervised on other datasets within the similar domain following [67, 39] to avoid information leakage. For classification tasks, we utilize the accuracy as evaluation matric; For link prediction tasks, we use standard metrics Recall@i @k and nDCG@ @ k at k= =20 in line with existing methodologies [31, 104, 1 118]. The metries used the experiment are detailed in Appendix D.2 and the implementation details of RAGRAPH and baselines are in Appendix D.4.
RAGRAPH,
8

---

Table Accuracy evaluation on node and graph classification. All tabular results (%) are in mean standard deviation across five seeds run, with best bolded and runner- up underlined.
Node Classification
Graph Classification
Methods
PROTEINS ENZYMES PROTEINS
COX2
ENZYMES
BZR
(5-shot)
(5-shot)
(5-shot)
(5-shot)
(5-shot)
(5-shot)
GCN
46.63
12.80
14.80406.64
67.87 103.9
22.67.155.2
58.76
GraphSAGE
08.87 48.75
52.99
67.0 07.02+00... 42 21. 20.17
58.27=04.79
GAT
48. 18.13407.90 90 47.75=00.22 55.82:07.31
04.89+03.23
20.67
57.04=10..70
GIN
19.61101.55 48.82
56.17:08..1
62.77:10.85
19.00
56.54
GraphPrompt+
Vanilla/NF
44.88=13.17 48.81 56.68403.63 53.04:101.1
36.50=03.31
68.77
Vanilla/FT
48.99 88 51.99 47.04
64.04::0..20 20 40.00
69.01 02.221
PRODIGY/NF
47.32408.12 43.80
53.48106.72
53.97
22. 13.884
67. 1 08.8.3
PRODIGY/FT
53.26+06.12
57.98=12..37
57.14410.34
65.31 28
25.94 12 68.08=06..
RAGRAPH/NF
56.12+04.11 75.92=00.72
08348
55.32:14.15
18.17703.33
77.53 05.53:05..24
RAGRAPH/FT
58.74 75.74
t02.52
:02.0
47.71 +66.8
76.79+ 05.002
RAGRAPH/NFT
59.83100.40 76.23 .63 59.08 103.50
71.70+10..2
49.17 104.64
74.81 +04.25
5.2 Retrieval-Augmented Graph Results
As discussed, conduct experiments and report the results of the three graph tasks for static graph and dynamic graph, as illustrated Table and Table 2. From the reported accuracy, we can find the following observations:
Outperforming SOTA Methods. First, our proposed RAGRAPI outperforms almost all the base- lines across the three graph tasks, demonstrating the effectiveness of RAGRAPH in transferring knowledge from the pre-training downstream tasks compared traditional GNNs i.e. GCN and GraphSAGE. achieves the highest average accuracy across almost all tasks on ENZYMES, with an improvement of least 5. 5.19 in the static graph, and up 11.78%c the dynamic graph the best baseline PRODIGY/FT. We argue that by virtue of the integration of hidden embedding and task-specific output vector, RAGRAPP is able comprehend more knowledge than simply learns the paradigm from X + Y. Second, compared with the models of PRODIGY/NE and GGIAPIINNE the introduction of noise training in noise prompt tuning also improves the robustness of the model, avoiding the influence of large amount of noise on the information aggregation inside the query graph.
Strong Retrieval-Augmentte Perfor- mance Unseen Datasets. We ob- serve that PRODIGY/NF and RA- GRAPH/NF better to Vanilla/NF, in- dicating that the retrieval knowledge truly works when testing on unseen datasets. Moreover, the difference be- tween PRODIGY/NF and PRODIGY/FT is much greater than that of RAGRAPH, which also indicates that a simple learn ing paradigm for ICL is not enough and that R RAGRAPI can achieve accept- able results even on unseen downstream datasets without the need for sophisti- cated fine-tuning.
16
Accuracy (96)
Ascuracy
40
tooK
Figure 3: Hyper-parameter parameter study with hopsk (Left) from 1 to 5 andtopk from to 20 (Right) on node classification with PROTEINS, and ENZYMES datasets with the setting Table

---

Table Performance evaluation (%c) on link prediction.
TAOBAO
KOUBEI
AMAZON
Method
Recall
nDCG
Recall
nDCG
Recall
nDCG
LightGCN
22.47:402.33 21.89 30.21 106.45 22.24:105.88 15.07=06.48 06.53+02.66
SGL
22. 22.15+02.20 22.12 09 92.61=01.21 22.36::01..
15.78=07.12
07.90+02..4
MixGCF
22.84:02.15 23.05
32.06=:04.0 22.49:006.9
9.9
05.24=08.98
07.40
SimGCL
22. 18:2.22 23.15
33.07:05.22
23.08:
16. 00
07.58
GraphPro+
Vanilla/NF
20.10+01.:0 20.12::01.00
201.1
15.31 12.56=07.d0 45 06.3 #03.92
Vanilla/FT
23.99:02.11 23.26:t01.42
33.96:00413
24.66::02.78
07.
08.73
PRODIGY/NF
21.67 23.15
21.66403.21
14.82:10.99
I 02.66
05.84
PRODIGY/FT
23.74
23.65+02.31 33,46:+01.70
23.28
16.72=01.28
08.09:02.66
RAGRAPH/NF
20.31 20.45:01.44
22.86:00.44
06.68+00.48
13.78=05.54
06.52:t02.99
RAGRAPH/FT
24.78 24.35
103.93
24.82
18.69 07.45
09.09113.88
RAGRAPH/NFT
24.89 102.01 +01.44
33.27
24.09
08.32=06.91 09.01 +03.67
5.3 Hyper-parameter Study
In this section, we examine the impact of various sypee-pprameters on RAGRAPH. We specifically analyze the effects of varying the number of hops k in toy graphs from the list [1,2,3,4 4,5) and the number linked toy graphs lopK from the list 100, 15,30,50) to verify the sensitive:
Figure 3 (Left) illustrates relationships between accuracy and the toy graph hop k. We observe that increases, the volume of retrieved knowledge grows exponentially. However, an excessive accumulation of knowledge not only fails to enhance accuracy but also introduces increased irrelevant noise that burdens the GNNs. Notably, aceuracy shows trend of initial improvement followed by decline as l is increased. This pattern suggests that at lower k values. the retrieved information tends to consist of isolated, less useful knowledge. In contrast, higher k values, the GNNs struggle to process extensive reasoning chains, leading to the utilization of complex and abundant information that is less effective than even the baseline model's performance. Figure (Right) shows effects on accuracy with different numbers of toy graphs lopK. As with the previous figure, increasing lop demonstrates that an excessive amount of knowledge can hinder the GNNs comprehension capabilities. Conversely, smaller topk results in insufficient knowledge to enhance performance on downstream tasks.
6 Conclusion
We introduced RAGRAPH, a novel and general framework that enhances Graph Neural Networks (GNNs) by integrating Retrieval-Augmented Generation (RAG) techniques. This plug-and-play approach improves GNNs' ability to generalize to unseen data by retrieving relevant information. Experimental results show that RAGRAPH outperforms state-of-the-art methods in various graph learning tasks, demonstrating its adaptability and robustness. While RAGRAPU is currently limited to retrieving subgraphs, future research could explore using more graph-structured data such as nodes, edges, and trees to further enhance its capabilities. In general, our work provides valuable insights and serves as a reference for future Large Graph Models.
Acknowledgments
This work is supported by the National Natural Science Foundation of China NN..U23A20668)
10

---

References
[1] A. Asai, S. Min, Z. Zhong, and D. Chen. Retrieval- language models and applications. In Proceedings the 61 Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41 46, 2023.
[2] A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through reff-teefection. In ICLR, 2024.
[3 S.-V. Bogolin, Croitoru, H. Jin, Y. Liu, and S. Albanic. Cross modal retrieval with querybank normalisation. In CVPR, 2022.
[4] K. M. Borgwardt, C. S. Ong, S. Schönauer, S. Vishwanathan, J. Smola, and H.-P. Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21 [1supp__):44--5556 1):44--i56, 2005.
[5] H. Cai, V. W. Zheng, and K. C.-C. Chang. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE TKDE, 2018.
[6] D. M. Chan, S. Ghosh, A. Rastrow, and B. Hoffmeister. Using external off-policy speech-to-text mappings in contextual end-to-end automated speech recognition, 2023.
[7] M. Chen, W. Zhang, W. Zhang, Q. Chen, and Chen. Meta relational learning for few-shot link prediction in knowledge graphs. arXiv preprint arXiv: 190900155, 2019.
[8] W. Chen, H. Hu, X. Chen, P. Verga, and W. W. Cohen. Murag: Multimodal retrieval aagmannntd generator for open question answering over images and text. In EMNLP, 2022.
[9] X. Chen, S. Zhang, Y. Xiong, X. Wu, J. Zhang, X. Sun, Y. Zhang, Y. Zhao, and Y. Kang. Prompt learning on temporal interaction graphs. arXiv:2402.065 00226.6...6..... 2024.
[10] Z. Cheng, Zhang, X. Xu, G. Trajcevski, T. Zhong, and F. Zhou. Retrieval- -ugmenel hypergraph for multimodal social media popularity prediction. In Proceedings of the 30th ACM SICKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 445-455, New York, NY, USA, 2024. Association for Computing Machinery.
[11] X. Chu, Y. Jin, X. Wang, S. Zhang, Y. Wang, W. Zhu, and H. Mei. Wasserstein barycenter matching for graph size generalization of message passing neural networks. In Proceedings of the 40th International Conference on Machine Learning. ICML'23. JMLR.org, 2023.
[12] F. Cuconasu, G. Trappolini, Siciliano, S. Filice, C. Campagnano, Y. Maarek, N. Tonellotto, and F. Silvestri. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, volume 17 of SIGIR 2024, page 719-729. ACM, July 2024.
[13] H. Cui, Z. Lu, P. Li, and C. Yang. On positional and structural node features for graph neural networks on non ant-attributed graphs, 2021.
[14] M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvas, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou. The faiss library, 2024.
[15] Y. Duan, G. Zhang, S. Wang, X. Peng, Z. Wang, J. Mao, H. Wu, X. Jiang, and K. Wang. Cat- gnn: Enhancing credit card fraud detection via causal temporal graph neural networks. ArXiv, abs/2402.11400 2024.
[16] Y. Duan, J. Zhao, pengcheng, J. Mao. H. Wu, Xu, S. Wang, C. Ma, K. Wang, K. Wang, and X. Li. Causal deciphering and inpainting in spatio-temporal dynamics via diffusion model. 2024.
[17] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In NeurIPS, 2015.
[18] P. Eli Chien, Jianhao Peng and O. Milenkovie, Adaptive universal generalized pagerank graph neural network. In ICLR, 2021.

---

[19] F. Fang, Y. Bai, Ni, M. Yang, X. Chen, and R. Xu. Enhancing noise robustness of retrieval- augmented language models with adaptive adversarial training. 2024.
[20] T. Fang, Y. Zhang. Y. Yang, Wang, and Chen. Universal prompt tuning for graph neural networks, In NeurIPS, 2024.
[21] Y. Fang, Y. Liang. B. Hui, Z. Shao, L.. Deng, X. Liu, X. X.Jiang, and K. Zheng. Efficient large-scale traffic forecasting with transformers: A spatial data management perspective. In Inproceedings of SIGKDD, 2025.
[22] Y. Fang, Y. Qin. H. Luo, F. Zhao, B. Xu, L. Zeng, and C. Wang. When spatio-temporal meet wavelets: Disentangled traffic forecasting via efficient spectral graph attention networks. In ICDE, 2023.
[23] L. Gao, X. Ma, J. Lin, and J. Callan. Precise zero-shot dense retrieval without relevance labels, 2022.
[24] X. Gao, H. Chen, and J. Haworth. A spatiotemporal analysis of the impact of lockdown and coronavirus london's 's bicycle hire scheme: from response recovery to a new normal. GIS, 2023.
[25] X. Gao, J. Haworth, D. Zhuang, H. Chen, and X. Jiang. Uncertainty quantification in the road- level traffic risk prediction by ppepprlltepoora zero-inflated negative binomial graph neural network (stzinb-gnn). GIScience 2023, 2023.
[26] X. Gao,, X. Jiang, J. Haworth, D. Zhuang, S. Wang, H. Chen, and S. Law. Uncertainty-awar probabilistic graph neural networks for road-level traffic crash prediction. Accident Analysis Prevention, 208: 1 108011, 2024.
[27] 7. Gao, X. Sun, 7. Liu, Y. H. Cheng, J. Li. Protein multimer structure prediction via PPI- guided prompt learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
[28] M. Gjoka, M. Kurant, C. T. Butts, and A. Markopoulou. Walking in facebook: A case study of unbiased sampling of osns. In 2010 Proceedings IEEE INFOCOM, pages 1-9, 2010.
[29] A. Grover and Leskovee. node2vec: Scalable feature learning for networks. In SIGKDD, 2016.
[30] W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. In NeurlPS, 2017.
[31] X. He, K. Deng, X. Wang. Y. Li, Y. Zhang, and M. Wang. Lightgon: Simplifying and powering graph convolution network for recommendation. In SIGIR, 2020.
[32] Z. He, W. Hao, W.-T. Lu, C. Chen, K. Lerman, and X. Song. Alcap: Aligament-auggmented music captioner. In ICASSP, 2023.
[33] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models, 2021.
[34] W. Hu, B. Liu, J. Gomes, M. Zitnik. Liang, V. Pande, and J. Leskovec. Strategies for pre-training graph neural networks. arXiv preprint arXiv: 12265,, 2019.
[35] Z. Hu, Y. Dong, K. Wang, K.-W. Chang, and Y. Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857-1867, 2020.
[36] J. Huang, G. Cai, J. Zhu, Z. Dong, R. Tang, W. Zhang, and Y. Yu. Recall-augmented ranking: Enhancing thickk-thrrugh rate prediction accuracy with cross-stage data. WWW, 2024.
[37] Huang and M. Zitnik. Graph meta learning via local subgraphs, 2020.
[38] M. Huang, Liu, X. Ao, K. Li, Chi, J. Feng, H. Yang, and Q. He. Auc-oriented graph neural network for fraud detection. In 2022.
12

---

[39] Q. Huang, H. Ren, P. Chen, G. Kržmanc, D. Zeng, P. Liang, and J. Leskovec. Prodigy: Enabling in-context learning over graphs, 2023.
[40] Q. Huang. H. Ren, and Leskovec. Few-shot relational reasoning via connection subgraph pretraining. In NeurIPS, 2022.
[41] T. Huang, Y. Dong, M. Ding, Z. Yang, W. Feng, X. Wang, and J. Tang. Mixgef: An improved training method for graph neural network-based recommender systems. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 665-674, 2021.
[42] G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, Petroni, Schick, J. Dwivedi- Yu, Joulin, S. Riedel, and H. Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022.
[43] Jiang, Y. Fang, R. Qiu, H. Zhang, Y. Xu, H. Chen, W. Zhang, R. Zhang, Fang, X. Chu, J. Zhao, and Y. Wang. Te-raggturing-comletee rag's 's case study medical Ilm systems. ArXiv, abs/2408.09 099999 2024.
[44] Jiang, Z. Qin, Xu, and X. Ao. Incomplete graph learning via attribute-struccure decoupled variational auto-encoder. In Proceedings of the 17th ACM International Conference Web Search and Data Mining, WSDM '24, page 30, New York, NY, USA, 2024. Association for Computing Machinery.
[45] X. Jiang, R. Zhang, Xu, R. Qiu, Y. Fang, Z. Wang, J. Tang, H. Ding, X. Chu, J. Zhao, and Y. Wang. Hykge: A hypothesis knowledge graph enhanced framework for accurate and reliable medical Ilms responses, 2024.
[46] X. Jiang, D. Zhuang, X. Zhang, H. Chen, J. Luo, and X. Gao. Uncertainty quantification via spatial-temporal tweedie model for zero-inflated and long-tail travel demand prediction. In CIKM, 2023.
[47] R. Y. Jiaxuan You and J. Leskovec. Position-aware graph neural networks. In ICML, 2019.
[48] B. Jin, Y. Zhang, Q. Zhu, and Han. Heterformer: Transformer-based deep node representation learning on heterogeneous text-rich networks. SIGKDD, pages 1020-1031, 2023.
[49] J. Kim, S. M. Jachyun Nam, J. Park, S.-W. M. Seo, J.-W. Ha, and J.iin Sure: Summarizing retrievals using answer candidates for open-domain qa of 1lms. In ICLR, 2024.
[50] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2016.
[51] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäächel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive nlp tasks, 2021.
[52] G. Li, M. Müller, A. Thabet, and B. Ghanem. Deepgens: Can gens go as deep as cnns? In ICCV, 2019,
[53] H. X. Wang, Z. Zhang, and W. Zhu, Out-of-distribution generalization on graphs: A survey, 2022.
[54] Li, Chen, Y. Liu, J. Wang, Q. He, M. Cheng, and X. Ao. Boosting the adversarial robustness of graph neural networks: An ood perspective. In International Conference on Learning Representations, 2024.
[55] K. Li, Y. Liu, X. Ao, J. Chi, Feng. H. Yang, and Q He. Reliable representations make a stronger defender: Unsupervised structure refinement for robust gnn. In SIGKDD, 2022.
[56] R. Li. T. Zhong, X. Jiang, G. Trajcevski, J. Wu, and F. Zhou. Mining spatio-temporal relations via self-paced graph contrastive learning. In SIGKDD, 2022.
[57] L., R. Xie, Y. Zhu, X. Ao, Zhuang, and Q. He. User-centric conversational recommendation with multi-aspect user modeling. In SIGIR, 2022.
13

---

[58] X. Li, D. Lian, Z. Lu, J. Bai. Z. Chen, and X. Wang. Graphadapter: Tuning dision-languee models with dual knowledge graph. In NeurlPS, volume 36, 2024.
[59] X. Li, Zhao, Y. K. Chia, B. Ding, S. Joty, S. Poria, and L. Bing. Chain-of Beain-of-know//e knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources, 2023.
[60] W. Lin and B. Bymne Retrieval augmented visual question answering with outside knowledge. In EMNLP, 2022.
[61] W. Lin, J. Mei, J. Chen, and B. Byrne. Prefimr: Scaling up fine- fine-grained late-interaction multi-modar retrievers, 2024.
[62] X. V. Lin, X. Chen, M. Chen, W. Shi, M. Lomeli, R. James, P. Rodriguez. J. Kahn, G. Szilvasy, M. Lewis, L. Zettlemoyer, and S. Yih. Ra-dit: eetrieval-uugmented dual instruction tuning. In ICLR, 2024.
[63] Y. Liu, X. Ao, F. Feng, and Q. He. Ud-gnn: Uncertainty-aware debiased training on semi- homophilous graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1131-1140, 2022.
[64] Y. Liu, X. Ao, F. Feng, Y. Ma, K. Li, T.-S. Chua, and Q. He. Flood: A flexible invariant learning framework for out-of-distribution generalization on graphs. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1548-1558, 2023.
[65] Y. Liu, X. Ao, Z. Qin, J. Chi, J. Feng, H. Yang, and Q. He. Pick and choose: man-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021, pages 3168-3177 202,31,7,2211 2021.
[66] Y. Liu, K. Zhang, Y. Li, Yan, C. Gao, R. Chen, Z. Yuan, Y. Huang, H. J. Gao, L. He, and I.. Sun. Sora: review on background, technology, limitations, and opportunities of large vision models, 2024.
[67] Z. Liu, X. Yu, Y. Fang, and X. Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW, 2023.
[68] S. Lu, N. Duan, H. Han, D. Guo, S. won Hwang, and A. Svyatkovskiy. Reace: A retrieval- augmented code completion framework. In ACL, 2022.
[69] H. Luo, Y.S.. Chuang. Y. Gong, T. Zhang, Y. Kim, X. Wu, Fox, H. Meng, and J. Glass. Sail: Search-augmented instruction learning, 2023.
[70] J. Luo, W. Zhang, Y. Fang, X. Gao, D. Zhuang, H. Chen, and X. Jiang. Timeseries suppliers allocation risk optimization via deep black litterman model. ArXiv, abs/2401.17350, 2024.
[71] K. Ma, H. Cheng, Y. Zhang, X. Liu, E. Nyberg, and J. Gao. Chain-of-skills: A configurable model for open-domain question answering, 2023.
[72] H. Mao, X. Chen, Q. Fu, L. Du S. Han, and D. Zhang. Neuron campaign for initialization guided by information bottleneck theory. In CIKM, 2021.
[73] S. Maskey,, R. Levie. Lee, and G. Kutyniok. Generalization analysis of message passing neural networks on large random graphs. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 4805 4817. Curran Associates, Inc., 2022.
[74] S. Matsug,, Y. Fujiwara, and H. Shiokawa. Uncovering the largest community in social networks at scale. In ICAI, 2023.
[75] OpenAl. Introducing chatgpt. http::://penna..com/bblg//
2022.
[76] OpenAL. Gpt-4 technical report. ArXiv, absr233333/330 2023.
[77] S. Pan, L. Luo, Wang, C. Chen, Wang, and Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions Knowledge and Data Engineering, 2024.
14

---

[78] Parashhar, Z. Lin. T. Liu, X. Dong, Y. Li, D. Ramanan, J. Caverlee, and S. Kong. The neglected tails of vision- -aiiuge models. In CVPR, 2024.
[79] Pasake, S. Gross, Massa, A. Lerer, J. Bradbury. G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[80] B. Perozzi, R. Al- and S. Skiena. Deepwalk: Online learning of social representations. In SIGKDD, 2014.
[81] Y. Qin, Y. Fang, H. Luo, F. Zhao, and C. Wang. Next point-of- interetts recommendation with auto-correlation enhanced multi-modal transformer network. In SIGIR, 2022.
[82] J. Qiu, Q Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang. Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1150-1160, 2020.
[83] Y. Qu, Y. Ding, J. Liu, K. Liu, R. Ren, W. X. Zhao, D. Dong, H. Wu, and H. Wang. Rocketqa: An optimized training approach dense passage retrieval for open-domain question answering. 2021.
[84] R. A. Rossi and N. K. Ahmed. The network data repository with interactive graph analytics and 2015.
visualization. In AAAI Conference on Artificial Intelligence, pages
[85] P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, and C. D. Manning. Raptor: Recursive abstractive processing for tree-orga retrieval. In ICLR, 2024.
[86] S. Sharma and R. Sharma. Identifying possible rumor spreaders on twitter: A weak supervised learning approach. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2021.
[87] K. Soman, P. W. Rose, J. H. Morris, R. E. Akbas, B. Smith, B. Peetoom, C. Villouta- Reyes, G. Cerono, Y. Shi, A. Rizk-Jackson, S. Israni, C. A. Nelson, S. Huang, and S. E. Baranzini. Biomedical knowledge enahheenhanced prompt generation for large language models, 2023.
[88] Su and T. M. Khoshgoftaar. A survey of collaborative filtering techniques. Adv. in Artif. Intell., 2009, Jan. 2009.
[89] J. Sun, Y. Zhou, and C. Zong. One-shot relation learning for knowledge graphs via neighbor- hood aggregation and paths encoding. Transactions on Asian and Low-Resource Language Information Processing, 21(3):1-19, 2021.
[90] M. Sun, K. Zhou, X. He, Y. Wang, and X. Wang. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In SIGKDD, 2022.
[91] X. Sun, H. Cheng, J. Li, B. Liu, and J. Guan. All in one: Multi-task prompting for graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (K)D'23, page 2120-2131, 2023.
[92] X. Sun, Jhangg X. Wu, H. Cheng, Y. Xiong, and Graph prompt learning: A comprehensive survey and beyond. arXiv:2311.16534 2023.
[93] Y. Tan, H. Lv, X. Huang, J. Zhang, S. Wang, and C. Yang. Musegraph: tuning of large language models for generic graph mining, 2024.
Graph-oriented instruction
[94] Y. Tan, Z. Zhou, H. Lv, W. Liu, and C. Yang. Walkim: A uniform language model fine-tuning framework for attributed graph embedding. A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, NewrlPS, volume 36, pages 13308-13325. Curran Associates, Inc., 2023.
[95] Z. Tan, R. Guo, K. Ding, and Liu. Virtual node tuning for few-shot node classification. In SIGKDD, 2023.
15

---

[96] J. Tang, M. Qu, M. Wang. M. Zhang, J. Yan, and Q. Mei. Line; Large-scuee scale information network embedding. WWW, 2015.
[97] B. Teji, S. Roy, D. S. Dhami, D. Bhandari, and H. Guzzi. Graph embedding techniques for predicting missing links in biological networks: An empirical evaluation. IEEE Transactions Emerging Topics in Computing,
191-
[98] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph attention networks, In ICLR, 2018.
[99] S. Wang, K. Ding, C. Zhang, C. Chen, and Li. Task-adaptive few-shot node classification. Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2022.
[100] S. Wang, Y. Dong, X. Huang, C. Chen, and J. Li. FAITH: Few-shot graph classification with hierarchical task graphs. In International Joint Conference on Artificial Intelligence, 2022.
[101] X. Wang. Zhu, and Y. Yang. T2vlad: Global-local sequence alignment for text-video retrieval. In CVPR, 2021.
[102] Y. Wang, R. Ren, J. Li, W. X. Zhao, J. Liu, and J. Wen. Rear: A relevance aware retrieval augmented framework for open-domain question answering. 2024.
[103] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[104] J. Wu, X. Wang. F. Feng, X. He, Chen, Lian, and X. Xie. Self Sulf-supe graph learning for recommendation. In Proceedings of the 44th international ACM SIGIR conference research and development in information retrieval, pages 726-735, 2021.
[105] L. Xia, B. Kao, and C. Huang. Opengraph: Towards open graph foundation models, 2024.
[106] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In ICLR. 2019.
[107] P. Xu, W. Ping, X. Wu, L. McAfee, C. Zhu, Z. Liu, S. Subramanian, E. Bakhturina, M. Shoeybi, and B. Catanzaro. Retrieval meets long context large language models. In ICLR, 2024.
[108] Y. Xu, X. Chu, K. Yang, Z. Wang, P. Zou, H. Ding. J. Zhao, Y. Wang, and B. Xie. Seqcare: Sequential training with external medical knowledge graph for diagnosis prediction in healthcare data. In Proceedings of the ACM Web Conference 2023, pages 281 2819-2830, 2023.
[109] Y. Xu, X. Jiang, X. Chu, Y. Xiao, C. Zhang, H. Ding, J. Zhao, Y. Wang, and B. Xie. Protomix: Augmenting health status representation learning via prototype- prooetype-based mixup. In Knowledge Discovery and Data Mining, 2024.
[110] Y. Xu, K. Yang, C. Zhang, P. Zou, Z. Wang, H. Ding. J. Zhao, Y. Wang, and B. Xie. Vecocare: Visit sequencss-clinieee cllinccl notes joint learning for diagnosis prediction in healthcare data. In IJCAI, volume 23, pages 4921 4929, 2023.
[111] Y. Xu*, R. Zhang* X. Jiang*, Y. Feng. Y. Xiao, X. Ma, R. Zhu, X. Chu, J. Zhao, and Y. Wang. Parenting: Optimizing knowledge selection of strieval-auggeent language models with parameter decoupling and tailored tuning. arXiv preprint arXiv:2410. 10360, 2024.
[112] K. Yang, Xu, P. Zou, H. Ding, J. Zhao, Y. Wang. and B. Xie. Kerprint: local-global knowledge graph enhanced diagnosis prediction for retrospective and prospective interpretations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 5357-5365. 2023.
[113] S. Yang, X. Jiang, H. Zhao, W. Zeng, H. Liu, and Y. Jia. Faima: Feature-aware in-context learning for multi-domain aspect-based sentiment analysis. COLING, 2024.
[114] Yang, Xia, D. Luo, K. Lin, and C. Huang. Graphpro: Graph pre-training and prompt learning for recommendation. In WWW, 2024.
16

---

[115] J. Yoo. N. Ahn, and K.-A. Sohn. Rethinking data augmentation for image apper-resolution: A comprehensive analysis and new strategy. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8372-8381 2020.
[116] O. Yoran, T. Wolfson, O. Ram, and J. Berant. Making vetrieval-augmennt language models robust to irrelevant context. In /CLR,, 2024.
[117] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823,, 2020.
[118] J. Yu, H. Yin, Xia, T. Chen, L. Cui, and Q. V. H. Nguyen. Are graph augmentations necessary? the 45th international
simple graph contrastive learning for recommendation. In Proceedings the ACM SIGIR conference on research and development in information retrieval, pages 1294-1303, 2022.
[119] W. Yu, H. Zhang, X. Pan, K. Ma, H. Wang, and D. Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models, 2023.
[120] X. Yu, Y. Fang, Z. Liu, Y. Wu, Z. Wen, J. Bo, X. Zhang, and S. C. Hoi. Few-shot learning on graphs: from meta-learning to pre-training and prompting. arXiv preprint 4rXi::2402.01440, 2024.
[121] X. Yu, Z. Liu, Y. Fang. Z. Liu, S. Chen, and X. Zhang. Generalized graph prompt: Toward unification of pre-training and downstream tasks on graphs. IEEE Transactions on Knowledge and Data Engineering. 2024.
[122] X. Yu. Z. Liu, Y. Fang, and X. Zhang, Dygprompt: Learning feature and time prompts on dynamic graphs. arXiv preprint 1331::2700..3337,,. 2024.
[123] X. Yu, J. Zhang, Y. Fang, and R. Jiang. Non-homophilic graph pre-training and prompt learning. arXiv preprint 2408 12594, 2024.
[124] X. Yu, C. Zhou, Y. Fang, and X. Zhang. Multigprompt for multi-task pre-traing pre-training and prompting on graphs. In WWW, 2024.
[125] Z. Yuan, Q. Jin, C. Tan, Z. Zhao, H. Yuan, F. Huang, and S. Huang. Ramm: Retrieval- -ugmeettd biomedical visual question answering with multi-modal pre-trrining. In CVPR, 2023.
[126] C. Zhang, H. Yao, C. Huang, M. Jiang, Z. Li, and N. V. Chawla. Few-shot knowledge graph completion. AnAII, 2020.
[127] H. Zhang and X. Jiang. Conumip: Continuous-time dynamic graph learning via uncertainty masked mix-up on representation space. Knowledge- Based Systems, 306:1 12748, 2024.
[128] H. Zhang, D. Wang, W. Zhao, Z. Lu, and X. Jiang. Imesn: An improved neighborhood aggre- gation interaction strategy for multi-scale contrastive siamese networks. Patlern Recognition, 158: 1052, 2025.
[129] R. Zhang, X. Jiang, Y. Fang, J. Luo, Y. Xu, Y. Zhu, X. Chu, J. Zhao, and Y. Wang. Infinite- horizon graph filters: Leveraging power series enhance sparse information aggregation. ArXiv, abs/2401 .09943, 2024.
[130] R. Zhang, Y. Xu, Y. Xiao, R. Zhu, X. Jiang X. Chu, J. Zhao, and Y. Wang. Kapo: Knowledge- aware preference optimization for controllable knowledge selection in sarieval-augmenter language models. arXiv preprini arXiv:2408. 03297, 2024.
[131] Y. Zhang, Z. Chen, Y. Fang, L. Cheng, Y. Lu, F. Li, W. Zhang, and H. Chen. Knowledgeable preference alignment for llms in domain-specific question answering, 2023.
[132] Z. Zhang, H. Li. Z. Zhang, Y. Qin, X. Wang, and W. Zhu. Graph meets 1lms: Towards large graph models, 2023.
[133] Zhao, A. Chen, X. Sun, H. Cheng, and J. Li. All in one and one for all: A simple yet effective method towards cross-domain graph pretraining, 2024.
17

---

[134] R. Zhao, H. Chen, W. Wang, F. Jiao, X. L. Do, C. Qin, B. Ding, X. Guo, M. Li, X. Li, and S. Joty. Retrieving multimodal information for augmented generation: A survey, 2023.
[135] J. Zhou, C. Xie, Z. Wen, X. Zhao, and Q. Xuan. Data augmentation on graphs: technical survey, 2023.
[136] S. Zhou, U. Alon, F. F. Xu, Z. Wang, Z. Jiang, and G. Neubig. Docprompting: Generating code by retrieving the docs. In ICLR, 2023.
[137] X. Zhou, R. Lumbantoruan, Y. Ren, I.. Chen, X. Yang, and Shao. Dynamic bi-layer graph learning for context-aware sequential recommendation, ACM Trans, Recomm. Syst., 2(2), apr 2024.
[138] F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua. Retrieving and reading: A comprehensive survey on open-domain question answering. 2021.
[139] Zhu, X. Feng, X. Du, Y. Gu, W. Yu, H. Wang, Q. Chen, Chu, Chen, and B. Qin. An information bottleneck perspective for effective noise filtering retrieval retrieval-augneet augmented generation. In L. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1044-1069, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
[140] Y. Zhu, Guo, and S. Tang. Sgl-pt: A strong graph learner with graph prompt tuning, 2023. [141] Y. Zhu, Z. Ou, X. Mou, and J. Tang.
embodied agents. In CVPR, 2024.
18

---

A Notations
The notations in this paper are summarized in Table 3.
Table 3: Notations Tables in RAGRAPH
Notation GIVIE G Go GT Topk G
Definition
The dynamic graph node / edge set with C,/ V, E, as its entry The resource graph with G as its entry
The query graph with ve as its center node The toy graph database The topK retrieved toy graphs
The toy graph k k-hop ego net for node Um
(Um)
X, A,Y,
The feature/ edge weight label matrix at time t
The set of label classes The neighbors of node The hidden embeddings/ task-specific output vector with h; / as its U; vector
N(v)
HIO
D
The k-shot labeled set
t/T
Timestamp
Number of nodes
d
The dimension of node feature The i-th node The master node of toy graph / The center node of toy graph
Vi
/ve
f1/f2
The dimension of hidden embedding/ task-specific output vector
The minimum value
hop
[
The node importance for node TR
PR PR(D)/C(()
The PageRank/ Degree Centrality value
Pi
The sampling probability of node v;
K
The scaling constant
nang (GT))
The number of augmentations of toy graph GT
The weighted similarity between query node v, and master node um The layer of GNN
Or
The balance weight with a E (0,1)
X
The weight of mixup
Wi
The weights the time, structure, semantic, and environment similaritics The reweight syper-parameter
B
More Motivation Details
B.1
Why Toy Graph Augmentation is needed
The reasons for toy graph augmentation:
Expanding toy graph base, enriching the scale of the knowledge repository [115].
Simulating Real-World Scenarios: Real-world graphs often encounter challenges such as missing nodes [44], noisy attributes [54], and unexplored connections [97]. We introduce node dropout, noise injection, and edge removal to simulate these scenarios accurately.
Addressing Graph Domain Shift: To mitigate domain shift between the graph knowledge base and testing graphs, our augmentations employ Mixup techniques such as Node Interpolation and Edge Rewiring. These techniques interpolate between training samples to generate synthetic samples, effectively smoothing decision boundaries embedding and reducing the model's 's sensitivity to minor variations in input data, thereby stabilizing predictions domain shift testing samples [109].
19

---

B.2 Why Noise-based Graph Prompt Tuning is needed
To address inherent challenges in toy graph quality, we introduce Noise- based Graph Prompting Tuning (c.f. Section 4.3.3). This method involves fine-tuning the model with artificially introduced noisy toy graphs Toyner-Tyy-Graphh Noise & Toy-Graph Noise), inspired by poise-tuning techniques in [19, 12, 116]. Our approach enhances the model's robustness against real-world retrieval noise, as evidenced by superior performance compared to traditional tuning methods (in Main Text Tables I and 2). This approach reduces the stringent requirement for an exceptionally high-quality graph vector base, thereby ensuring robust performance across various tasks within our R RAGRAPII, and significantly mitigating data quality impacts.
B.3 Difficulty to construct and maintain high-quality and diverse graph vector base
In RAGRAPH, the toy graph base largely leverages significant prior research datasets in pre-trained GNNs [67, 105, 39, 124], which are trained meticulously curated graph datasets and cover diverse domains, such as biology, chemistry, medicine recommendation tasks, etc. For example, the PROTHINS dataset [4], derived from cryo-electron microscopy and X-ray crystallography, and the ENZYMES dataset [100]. based on EC enzyme classification, are meticulously annotated by medical experts.
B.4
Why Inverse Importance Sampling Strategy is needed
The adoption of the Inverse Importance Sampling strategy is crucial. In RAGRAPH, subgraphs sampled as toy graphs, where nodes with higher degrees (non-long-tall knowledge, extensively learned and embedded into GNN parameters) are more frequently included in subgraphs due their extensive connections with neighbors, resulting in higher frequency in toy graph base [28]. Conversely, nodes with low degrees (long-tail knowledge), are more important but ignored. To mitigate this issue, propose this by prioritizing nodes with lower degrees to capture long-tail knowledge when sampling.
B.5 Why Four Similarities are needed
In practical applications, the four similarities all contribute performance improvement and we state the significance as follows:
Time information is crucial to predict future states or trends [114] via node history, i.e. in social networks, analyzing historical user interaction aids in predicting future behaviors.
Structure pertains to how nodes are interconnected and overall graph topology, vital for capturing similar graph structure patterns [13, 44, 113]. In transportation networks. factories always located on the outer ring the city, sharing similar structural connectivity, aiding in the discovery of spatiotemporal patterns [56, 16).
Sharing similar neighborhoods is essential for evaluating node similarity and correlation. In recom- mendations, shared purchase histories between users and products indicate potential interests, akin to collaborative filtering [88].
Semantic information measures similarity based features [39]. In knowledge graphs, identifying relevant subgraphs to query nodes enhances retrieval accuracy based on semantic similarity.
B.6
Why Knowledge Fusion needed
Fusion and decoder here represent one of the core contributions of RAGRAPH:
Overall Task Perspective: For the same tasks, the decoder can be directly employed to obtain outputs. For different tasks, the decoder can masked and utilize pre-computed embeddings without training or be tuned to better adapt. This underscores our primary contribution, where the decoder functions as a versatile aldg-anddplayy and "tune-free" component.
Integral Fusion Strategy: Fusion Strategy facilitates concurrent information propagation from toy graphs (hidden embeddings) and Y (task-specific output vector) to query graph, aligning with our secondary contribution.
20

---

B.7 How RAGRAPH works on par with RAG in and CV
In NLP, RAG enhances the generation of LLM by retrieving relevant information via prompts. Similarly, in RAGRAPH, enhance downstream graph learning by integrating information from retrieved toy graphs. Using these toy graphs with shared patterns assists the model inference. In our framework, the "generation" involves the retrieval-enhanced Graph Prompt: Toy Graph Intra Propagate & Query-Toy- Graph Inter Propagate to propagate retrieved knowledge (X and Y) into the query graph. To illustrate, analyze this from both experiment and theory.
Experiment We perform a case study to illustrate how "generation" works by displaying specific instances of node vectors in Appendix D.6.
2. Experiment 2: In traditional GNN tasks, GCN, GAT, and GIN typically expand their receptive fields through stacked message- partngggaassinggggge layers or neighborhood subgraph sampling for inference. Patterns learned in these contexts are often localized within the constrained receptive field. In contrast, in RAGRAPH, we observe that subgraphs sharing similar patterns often exhibit properties more aligned with downstream tasks. These subgraphs provide richer information for inference compared to simply enlarging receptive fields. As shown in Main Text Tables 1 and 2, Figure 3, RAGRAPH'S strategy of incorporating toy graphs significantly outperforms baselines.
3. Theory I: Furthermore, we provide a theoretical justification of retrieval augmentation in GNNs (see Appendix B.4). From informationn-hhorreti perspective, introducing RAG knowledge into GNNs enhances the mutual information between input features X and output labels Y, such that:
I(X,
thereby improving the performance of downstream tasks. This is aligned with the information theory of RAG in NLP [73].
4. Theory 2: Recent studies [11, 139] also suggest the generalization error diminishes with an increase in the node number of the graph in Theorem 1.1 [139]: the generalization error between the expected loss Reep ()) E (x (<((()).v) and empirical loss Remp (0) ((O
super bounded:
(Rsp(( (0) Remp (+) S V ((),,
where C' represents the model complexity (e.g., parameters), m denotes the training set size, and g(n) En [n- D+I depends on the average graph size (node number) with as the graph size distribution and D is the metric-measure space dimension. In RAGRAPH, retrieving similar toy graphs significantly increases the number of graph nodes (via Query-Toy- Graph Inter Propagate, linking toy graph nodes to query graph), significantly augmenting n while reducing q(n). Consequently, the upper bound of generalization error decreases, promoting smoother graph learning convergence and enhancing pattern learning.
C Further Methods Details
C.1 Revisiting Graph Neural Networks
The goal of a GNN is to learn node embeddings based on an iterative aggregation of messages from the local network neighborhood. We use embedding matrix (z!")) JUEV denote the embedding for all the nodes after applying an L-layer GNN. The I-th layer of GNN, fz! G ((2()))))))))))))))))))))(( written as:
(4-))),, can be
(1-1) = "ss (z (z((1 _ (-11),,
AGa ((m (uE((c))
(0) where z is the embedding for E V after passing through layers, = x. or h1 or Og. m is the message embedding, and (u)) is the set of direct neighbors of v. Different GNNs can have various definitions of message-passing functions () and aggregation functions AGa and these two functions could be prremeter-free.
21

---

C.2 Key Construction of Position-aware Code
Given a randomly selected node anchor set V/s C V. we calculate the minimal distances, a.k.a. hops between the two node sets. Suppose Vy E V, the Vs. the distance similarity between node Uy and ew can be depicted as dis(Uw;Uw) By normalizing the similarity to [0, 1], distance-to-ccenrrodd Uw)))
if dis( < disg
d2c(vu,tho)= dis(va,Uw)
otherwise
here hyperparameter dis, is the maximum hops, the distance beyond this boundary is considered invalid. The structure feature node v, is d2e(v,. By collecting all distances with anchor-set Vss the structure Se R=*[Val is written as follows:
d2c(e.. (d2c) e E Vs C Vl.
5= [=(dccvv..Vs)lo EV],
where [-] means the concatenation operation. The cestance-to-centron feature converts the non- Euclidean structure to the Euclidean structure. d2c dramatically reduces the size of the matrix and meanwhile contains more structure information instead of identifier information, the size of the anchor set Io follows P-GNNs(47, 44).
C.3 Similarity Functions
For the history key, we adopt an exponential decay function to measure the time similarity values. We smooth the impact of time differences and provide a controlled decay coefficient 0. The time similarity, Sime: between the same node Vi and Uma with different timestamp ((vm)))(()))) defined as:
Stime e (((()))) [[(m
(10)
where is positive parameter that controls the rate of exponential decay.
For the environment key, match the environment of node using Jaccard similarity to compare the sets of neighbors (v.)) in the query graph and N ((um) ) in the toy graph:
WW (N(v)) (vm))
Senviromment (ve)
(11)
For the hidden embedding key, we input the query graph into pre-trained GNNs to obtain the hidden embedding for the query node, with the similarity defined as:
h GNN(Xee), Ssemntic(Ve. (osine((h.hhh)
(12)
For the position-aware code, we denote Sc,8m as the position-aware code of node Un and utilize cosine similarity before, defined as Satueurre (Uc.um) cosine(sc.Sm))
C.4 Proof of the Effectiveness of RAG
In this section, we will theoretically prove that introducing RAG knowledge can significantly improve the predictive performance of the model.
Assume X represents the input features. represents the target output labels, and RAG represents external knowledge related to the input features (or even the output labels). We analyze from the mutual information view, where T(X;Y X;Y) Y) quantifies the dependency between X and Y. which reflects performance of the model, the larger the value, the better the performance of the model [56, 72). By introducing RAG knowledge RAG into GNNs, we can effectively increase the mutual information between the input features X and the output labels Y as [XX, (XXRAAGYY > (X;Y), thereby improve
22

---

the model's downstream task performance. The derivation is as follows: I(X,RAAG)))
P(X,Y p R(A,RYY)
Σx(X, Log ((X,Y)
Σ P(X,RAGY Jog X,RAG,Y
p(X)p((
X.Y
P(X,R = E p|X,
E DXX,RAG, ((X,Y)
P(X,RAG)M( X.RAG,Y
p(X)p(Y)
X,RAG,Y
P(X,RAG,Y)
P(X)
E P(X,RAG,Y
R(X,) p(X,Y))
X,RAG.Y
P(X.RAG,Y)
= E D(X,RAG,YXlo5
(RAGIX)P(XX)
X,RAG,Y
XIX
E |RAA,YYX))(XX
((RAGIX)
X,RAG,Y
-I[RAG]Y|
(13)
where =xx knowledge RAG provides to assist in predicting Y based on X, this term will approach zero when the RAG noise to the prediction task. In summary, the integration of RAG knowledge can enhance the mutual information between X and Y, thereby improving the performance and accuracy of downstream tasks.
Moreover. (YRA; (X) measures that how much additional
C.5 Algorithms
In this section, we will provide a detailed description of the algorithms of Toy Graph Construction (Algorithm 1) and Training and Inference with Toy Graphs Retrieval (Algorithm 2).
We outline the process for constructing toy graphs in Algorithm. 1. In line 1, the toy graph database GJ is initialized.
Algorithm 1.
Lines 2-15 describe the steps construct toy graphs by iterating through each snapshot of the dynamic resource graph g
In more detail, lines 3-5 details calculate the importance and reverse importance of each node within the snapshot. Following this, lines 6-7 involve normalizing the sampling probabilities according to the reverse importance values. The selection of master node and the generation of its k-hop ego network are carried out lines 8-9. Subsequently, line 10 involves augmenting the toy graph through specific data augmentation techniques.
Lines 11-16 detail the generation of key-value pairs for each toy graph. This includes saving the timestamp as the history key, the neighbors of the master node as the environment key, the structural encoding as the structure key, the hidden embedding as the semantic key, and the task-specific output vectors as the value. Each toy graph is then stored in the database GT.
Ultimately, in line 18, the algorithm returns the toy graph database GT.
Algorithm 2. We introduce the algorithm for training and inference with toy graph retrieval in Algorithm. 2. Initially, in line we define the required inputs, including the testing graph Gieu: the toy graph database GJ, and other relevant parameters. The final output is the aggregated result o..
The RETRIEVETOY GRAPHS function, detailed in lines 3-11, initializes an empty similarity list and iterates over each toy graph in the database. Lines 5-6 compute various similarity metrics, and the overall similarity is determined in line 7. This similarity score is then added to the list. After sorting by similarity, the topK toy graphs are retrieved and returned in line 11.
Within the PROPAGATION function (lines 12-17), each retrieved toy graph undergoes intra- propapprppagttion in line 14. The intra-propagation step follows in line 16, ultimately returning the propagated results 2.-
The KNOWL.DGGEFUSE function, found lines 18-21, combines the outputs from previous steps. The final combined outputs o, generated by the decoder and returned in line 21.
23

---

Algorithm 1 Toy Graph Construction
Require: Dynamic Resource Graph G node importance balance weight o, toy graph scaling constant
K. maximum hop k
Ensure: Toy graph embedding key-value database GT
1: Initialize toy graph database G 0-0
2: for each snapshot G G do
Construct Toy Graphs
3:
for each node e E G do
4: Calculate importance [(v)) ( (L)-
5: Reverse node importance ('( T(b)+
6: end for
7:
for each node v E G., do
8:
Normalize sampling probabilities ps (wy)
('().)
9: end for
Sample master node Unn t- WEIGHTEDSAMPPINGGG pi) based on probability Pt
11:
Generate k-hop ego net G (Um) ) for node Um
Augment toy graph (GT) DATAAAUGEEENNTTINNN (Um)), Rang) with Dag (G (Umm))
[K- [G" (mm)
(G*
13:
for each G E(GT) do
Generate keys-values pair
14:
Save timestamp r as history key
15:
Save neighbors.J (N;,,, of master node om environment key
Save structural encoding 8,, of node em via Eq. (9) as structure key
17:
Save hidden embedding h by feeding GT into pre-trained GNNs as semantic key Save the hidden embedding {hī (u E GT) as value
18:
Save specifpccific output vectors {oī Jo, E EGT) by feeding (hş by EGT into decoder as
value
20:
Store toy graph GT into database GT end for
21:
22:
end for
23:
return Toy graph database GT
The main algorithm begins line 22. If fine-tuning enabled and the prompt loss has not converged, lines 23-34 detail the process of toy graph retrieval and propagation for each query graph. This includes the optional addition of noise in lines 26-29. The hidden embedding and task-specific output vectors propagated in lines 30-31, and the aggregated outputs are fused in line 32. The prompt loss is computed, and fine-tuned parameters are updated in lines 33-34.
If fine-tuuing is not required, lines 35-39 describe a similar process of toy graph retrieval and propaga- tion, without the fine-tuning steps. The aggregated outputs are computed directly.
D Further Experiment Details
D.1 Datasets Statics
We employ eight benchmark datasets for evaluation including four public static classification datasets for node- and graph-level tasks.
(L) PROTEINS [4] is collection of protein graphs, including the amino acid sequence, conformation, structure, and features such as active sites of the proteins. Each node represents a secondary structure, while each edge illustrates the neighboring relationship either within the amino acid sequence or in 3D space. The nodes are divided into three categories, and the graphs are classified into two distinct classes. This dataset is used for node and graph classification tasks, containing 1,113 graphs with an average of 39.06 nodes and 72.82 edges per graph, with a density of 4.8e-2.
(2) COX2 [84] is dataset of molecular structures, including 467 cyclooxygenase-2 inhibitors. Each node represents an atom, and each edge signifies chemical bond between atoms, such as single, double, triple, aromatic bonds. All the molecules belong two categories. This dataset is used for
24

---

Algorithm 2 Trainiig and Inference with Toy Graphs Retrieval
Require:
Testing graph Geea, toy graph database GJ, pre-trained GNN model GNNe, (-), number of TopK toy graphs to retrieve, similarity weights W) WW,WW, fine-tunin flag f fine tune, noise prompt-tuning flag add_noise
Ensure: Aggregated output ā.
1: function T GR A ,
2: Initialize similarity list {S}
3: for each toy graph GT E do
4: Calculate time similarity Stimes environment similarity Senviromment structure similarity
Sstructure and semantic similarity Ssemanic
5:
Compute similarity S + Wh SSme W2 Satructure + - Senviroment + W4 Seeantic 6: Add (GT to {S}
7:
end for
8: Sort (.S} by similarity in descending order
9: Retrieve lopk toy graphs Goopk (G (S}) with LopK similarities}
10: return Retrieved toy graphs Gopp
11: end function
12: function PROPAGATION(C GTopk)
for each toy graph GT E &TTpp do
14: Perform Intra Propagation 2m <- INTRAPROPAGATION((")
end for
16: 2. f INTERPROPPAATIIONG,, CCCTp)
return Ze
18: end function
function KNOWLEDOEFUSSIX((h..))
20:
Combined output o, - (1-)))0ot))))CC0 DECODER(Ne)
return Combined outputs o,
22: end function
23: if [ine__ Sine_tnnde & Grrmpp not converged then
for each query graph G Ies do
Toy Graph Retrieval and Propagation
25:
G.ook f- RETRIEVETTYGRRPISSCS (T. if iid__noise then for each toy graph GT T do
26:
28:
Introduce noise GT + ADDNNISE(G")) end for Add bottom toy graphs to GTopx end if
bs Inner- Tiner-Toy-Graph Noise
29:
30:
D Toy-Graph Noise
31:
32:
he - PROPAGATION(G GTTpk)
Propogate hidden embedding Aggregated outputs O, + KXOWLEDGEFUUSONNI..c. Propogate task-specific output vector
33:
0. PROPAGATION GT
34:
35:
Based on task-specific loss function Update fine-tuned parameters θ by minimizing Eprompt
Compute loss Eprompt via o, and 0c
36:
37:
end for
38:
else
39:
for each query graph G Eies do
Toy Graph Retrieval and Propagation
40:
Topp RETRIEVETOYGRAPHSC' GT. TopK)
h, - PROPAGATION GTopk)
Propogate hidden embedding
42:
0. + PROPAGATIONG G'Topk)
Propogate task-specific output vector
43:
Aggregated outputs 0. + KXOWLEDGRFFUSION/I...
44:
end for
45: end
return Aggregated outputs
25

---

Table Statistics of the experimental datasets and summary of datasets.
Statistics
TAOBAO
KOUBEI
AMAZON
PROTEINS
COX2
ENZYMES
BZR
Nodes per Graph
204,168
221,366
238,735
39.06
41.22
32.63
35.75
Edges per Graph
8,795,404
3,986,609
876,237
72.82
43.45
62.14
38.36
Density
8.6e-4
3.3e-4
6.2e-5
4.8e-2
2.6e-2
5.9e-2
3.0e-2
Graphs
1,113
467
405
Graph Classes
Node Features
1
18
Node Classes
Snapshot Granularity
daily Edge
weekly Edge
weekly Edge Dynamic Snepshot
Task
Node, Graph Static Node, Graph
Graph Static Gruph
Node, Graph
Graph
Type
Dynamic Snapshot
Dynamic Snapshot
Static
Static
Dataset Partition
Node, Graph
Gruph
graph classification tasks, with each graph having an average 41.22 nodes and 43.45 edges and density of 2.6e-2.
(3) ENZYMES [100] is dataset of 600 enzymes collected from the BRENDA enzyme database. These enzymes are labeled into 6 categories according to their top-level EC enzyme classification. This dataset is used for node and graph classification tasks, with each graph having average of 32.63 nodes and 62. 14 edges and a density 5.9c-2.
(4) BZR [84] is a collection of 405 ligands for the benzodiazepine receptor. Each ligand is represented by a graph, and all ligands are categorized into two groups. This dataset used for graph classification tasks, with each graph having an average of 35.75 nodes and 38.36 edges and a density of 3.0e-2.
Additionally, we leverage three publicly available datasets encompassing wide array of real-world scenarios in dynamic recommendation (link prediction):
(5) The TAOBAO dataset captures implicit feedback data from Taobao.com. a prominent Chinese e-commerce platform, collected over a span of 10 days. This dataset is used for edge classification tasks, containing 204,168 nodes and .795,404 edges, with density of 8.6c-4.
(6) The KOUBEI dataset records 9 weeks of user interactions with nearby stores on Koubei, a platform integrated within Alipay. This dataset is used for edge classification tasks, containing 221,366 nodes and 3,986,6000 609 edges, with a density of 3.3e-4.
(7) The AMAZON dataset comprises a collection of product reviews sourced from Amazon, spanning duration of 13 weeks. This dataset is used for edge classification tasks, containing 238,735 nodes and 876,237 edges, with density of 6.2e-5.
These datasets' detailed statistics summarized in Table 4. The "Task" column provides information about the type of downstream task conducted on each dataset: "Node" denotes node classification tasks, "Graph" signifies graph classification tasks, and "Edge" indicates tasks related to link prediction. The "Type" column indicates the type of graph dataset: "Dynamic" for dynamic dataset 2 1, and "Static" for static dataset 1. For dynamic datasets, the "Snapshot Granularity" denotes the time granularity for each dataset. In our experimental setup, for dataset partition, dynamic graphs are partitioned according to snapshots, while static graphs are partitioned either by node by the entire graph.
D.2 Evaluation Matrices
Node and Graph classification evaluation. For the node classification, we use the prediction accuracy to measure the model.
Link prediction evaluation. For the link prediction, we evaluate recall and ranking quality the effects of recommendation following previous studies [118, 31J. We use Recall @k and NDCG@K as metries. Note that this task should be binary task. We denote the topk largest value as reliy1 e [1,A) for node U..
26

---

Recall @k measures the ratio of true positive links contained in the top is predicted links for each node:
A
rely
Recall
(14)
==
where rele, if the j-th predicted link for node v, exists, otherwise 0. I[.)) the indicator function, and if A[: ( then [[A[f=]>0) =1
NDCG@k (Normalized Discounted Cumulative Gain) is computed by normalizing DCG@K (Dis- counted Cumulative Gain) which accounts for the position correctly predicted links. DCG@ki is defined as:
reli
DCG
(15)
7
log (j+1)
D.3 Baseline Details
In this section, present the details of baselines.
Table 5: Baseline Code URLs of Github Repository
Baseline
Type
Code Repo URL
GCN
Static
GraphSAGE
Static
attps://githbb..cm/lllllaaaaaa
GAT
Static
https///gitaab.ccm/eeeall///A
GIN
Static
LightGCN
Dynamic
attp:///gghabbbcc////uuudggggggg
SGL
Dynamic
MixGCF
Dynamic
SimGCL
Dynamic
GraphPro
Dynamic
otte:://ggtaabbccoo/////////oppp
GraphPrompt
Dynamic, Static
PRODIGY
Dynamic, Static
GCN [50]: GCN is an end-to-end learning framework for graph-structured data. It utilizes neighborhood aggregation to integrate structural information, which particularly effective in node classification and graph classification tasks.
GraphSAGE [30]: GraphSAGE, a general and inductive framework that leverages node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen data.
GAT [98]: GAT is spatial domain method, which aggregates information through the attention- learned edge weights.
GIN [106]: GIN utilizes multi-layer perceptron sum the results of GNN and learns parameter to control residual connection.
LightGCN [31]: LightGCN learns user and item embeddings by linearly propagating them on the user-item interaction graph, and uses the weighted sum of the embeddings learned all layers the final embedding.
SGL [104] SGL is to supplement the classical supervised task of recommendation with auxiliary self-supervised task, which reinforces node representation learning via self-discriminatiio..
MixGCF [41]: MixGCF generates the synthetic negative by aggregating embeddings from different layers of raw negatives' neighborhoods to perform collaborative filtering.
27

---

SimGCL [1 [1]8] SimGCL applies unsupervised contrastive learning to enhance representation learning. making suitable for link prediction tasks. applied dynamic graphs test its adaptability and performacee.
GraphPro [114]: GraphPro extends GraphPrompt by introducing spatial and temporal prompts tailored for dynamic graph learning, enhancing the ability capture both structural and temporal relationships within graph data.
GraphPrompe [67]: GraphPrompt integrates pre-training and downstream tasks using a unified template approach and employs task-specific prompts to enhance sub-task learning, applicable to both dynamic and static graph contexts.
PRODIGY [39]] PRODIGY focuses facilitating downstream tasks through in-context examples and learning from the Y paradigm. It is implemented to enhance learning in both dynamic and static graphs by leveraging contextual learning strategies.
D.4 Implementation Details
Implementations are done using the Py PyTorch 2.3.0 framework [79] in Python 3. 11 on an Ubuntu server equipped with 1 VV00 GPU and an Intel(R) Xeon(R) CPU.
In node and graph classification tasks: For baseline GCN [50]. we employ 2-layer architecture and set the hidden dimension as 256. For GraphSAGE [30], we utilize the mean aggregator and employ 2-yey architecture. The hidden dimension is also to 256. For GAT [98]. we employ 2-layer architecture and set the hidden dimension as 256. Besides, we apply 8 attention heads in the first GAT layer. Similarly, for GIN [106], we also employ 2-layer architecture and set the hidden dimension as 256. For GraphPrompt, we follow [67] to employ 2-layer GCN as the backbone and set the hidden dimensions as 256.
In the link prediction task: For LightGCN, SGL, MixGCF, SimGCL and GraphPro, we employ a 3-layer GNN architecture and set the hidden dimension 64 with Low-Rank Adaptation (LoRA) [33] rank equals to 16. For GraphPro, the backbone graph encoder is SimGCL.
Moreover, for all three tasks, the hyper-paramette of baselines provided in the paper. In PRODIGY and RAGRAPH, is set to 2, top set to 5, y is set to 0.8 for PROTEINS and 0.5 for ENZYMES in node level, y is set to 0.5 for PROTEINS, 0.6 for COX2, 0.8 for ENZYMES and 0.5 for BZR in graph level, a= A = 0.5, 3,
based on the recommended values
W2 Wy -0.05. we = 0.85.
D.5 Resource Graph Scalability Study
60
70-
Accurscy (%6
60
50
45
50
40-
40
30
GIN
PSOBEGY
FRODIGY
10%
20%
30%
40%
50%
60%
70%
88%
10%
20%
34%
40%
50%
60%
70%
50%
Truin Resource Graph Size
Truln Resource Graph Size (%)
(a) Node Classification on ENZYMES dataset
(b) Node Classification on PROTEINS dataset
Figure 4: Performance comparisons of RAGRAPH and several baselines with different proportions of training and resource data.
We assess the impact of varying amounts of training and resource data on model performance. As illustrated in Figure 4, we vary the proportion of train and resource graph size from 10% to 80%, with increments of 10%, and conduct experiments on node classification tasks using ENZYMES and PROTEINS datasets, respectively. For comparative analysis, we select GIN, GraphPrompt, and PRODIGY as baseline models. To ensure fairness our experiments, we maintain a consistent ratio of train to resource data at 3:5 during fine-tuning, utilizing the sum of these as retrieval database.
28

---

As shown in Figure 4, there is clear trend where the accuracy of the model improves as the proportion of the dataset increases. However, the rate of accuracy improvement starts to plateau once certain dataset proportion is reached (i.e. 30% in PROTHINS for PRODIGY and R AGRAPH, 40% in PROTEINS for GraphPrompt). Among the evaluated models, GIN and GraphPrompt show the slowest convergence rates, whereas PRODIGY converges at a moderate pace, and R AAGAAPH converges the fastest. This rapid convergence in PRODIGY and RAGRAPH is attributed to its ability to engage in effective knowledge retrieval, significantly enhancing the model's 's comprehension abilities. Remarkably, both PRODIGY and R RAGRAPII can achieve commendable results in downstream tasks even with a small proportion of the dataset. Compared to PRODIGY, RAGRAPH exhibits superior performance because while PRODIGY primarily learns a mapping from X to Y, RAGRAPH not only learns this mapping but also integrates additional knowledge into GNNs more effectively. This integration becomes increasingly beneficial as the dataset proportion grows, allowing R RAGRAPH to outperform other models, particularly at higher data volumes where it can better leverage its knowledge integration capabilities.
D.6 Qualitative Analyses of Toy Graphs Retrieving
In this section, we conduct qualitative analyses of the toy graphs retrieving experiment. For the sake of understanding, we conduct experiments under normal settings where the dimensionality of the task-specific output vector is equal the number of classes.
On the ENZYMES dataset, for a 3-class node classification task, regarding node "13984", which belongs to class 3, if we only use the GraphPrompt Backbone, the resulting one-hot encoding is [0.28,0.34.0.38]]
However, since the node is of class 3, we expect the one-hot encoding to be as close as possible to [0,0.1]. In RAGRAPII retrieval, taking the top 3 retrieved graphs as examples, the connection weights for these 3 toy graphs to query graphs are 0.5, 0.7, and 0.1, respectively, and their corresponding label one-hot encodings are [0,0,1], [0,0,1], and [0,1,0]. Therefore, the result obtained by propagating the task-specific output vector through toy graphs is: [0,0.1,1.2], and after normalization, the result is [0.0.08,0.92].
Mcanwhile, the vector obtained by propagating toy graphs hidden embedding and via decoder is [0.37,0.32,0 The retrieval toy graphs notably enhances performance at both the task specifcc output vector and hidden embedding levels. The final vector is obtained through a weighted sum with y 0.5 Eq(6) is [0.185,0.200 0.79), after normalization the result is [0.157,0. 1..1..,00677)) which greatly enhances the model's discriminative ability compared to GraphPrompt (0.28,0.34,0..38)
Output Vector via Decode the Hiddes Embedding
13994 [0.1, 0.3)
[0. 0.1, 1.2]
0.5
[0.3, 0.2,
13984
[0, 1,
Y
[0.57, 0.32, 9.66)
13997 (0.2, 0.7]
Task-specific Outpur Vectur
[0, 1.2. 1.2]
10185, 0.20, 0.79)
[0.0,
0,7
Nommlization
0.157
139R3
(0.3, 0.3, 0.4)
[0,0,1] 1]
Task-qpeeific Outgut Vector
Center Node of the Query Graph
Output Vector Decode the Hidden Fimbadding
Node Toy Graph
Figure 5: Qualitative analyses of toy graphs retrieving - how "generation" works.
29

---

PRODIGY: Learn X- by ICL
Example Graphs
Query Node Embedding
Toy Master Embedding
c80
Query Graph
Queey Graph
Graph
DECY
similarity
Assign the same label Y to query graph
RAGraph Learn X & from RAG
Dynamic Toy Graphs
Tay Groph Intra Propagare
Query Graph
Bu
Thyry-Toy--Genph Propagate
Inject the retrieved X and into query node
Figure 6: Difference Illustration between PRODIGY and RAGRAPH.
E Difference between ICL (PRODIGY) and RAG (RAGRAPH)
In this section, explore the distinctions between PRODIGY and RAGRAPH from several critical perspectives, as illustrated in Figure 6:
This approach utilizes fixed examples rules, which may not be optimal for dynamic and evolving scenarios. PRODIGY primarily focuses on learning direct mappings from X to Y through in-context learning. However, encounters challenges integrating external information that is more pertinent to the query node. This is particularly problematic when the distribution of each node belonging to the same label class varies, and simply learning the mapping based on the prototype node will somehow be misleading.
PRODIGY:
RAGRAPH: In contrast, RAGRAPHI is designed to handle non-static, streaming knowledge, making it well-suited dynamic graph structures and evolving tasks. It actively retrieves relevant knowledge on on-demand, effectively incorporating information about both X and Y from external sources into GNNs. Moreover, RAGRAPH can operate without the need for model fine-tuning, providing substantial flexibility. This adaptability enables RAGRAPH to excel in tasks that require continuous adaptation changing conditions and the integration of external, relevant information.
In summary, we argue that qualified Estrieval-Augmenti Generation (RAG) system for Graph Learning should fulfill several essential criteria to effectively support complex reasoning tasks: I) It should retrieve ample feature and task-related label information, analogous to how attributes gathered in the NLP domain to stimulate the reasoning capabilities of LLM.: 2) The system should adapt new tasks unseen datasets without requiring fine-tuning of model parameters; Knowledge within the system must be dynamically updated and stored, ensuring current and relevant data utilization.
Broader Impacts
Our work builds on the widespread application of Retrieval-Augmented Generation (RAG) in large language models (LLMs) and aims to extend its success to graph data, thereby constructing graph foundation models. This approach allows models to transfer rapidly without requiring learnable parameters, avoiding potential performance degradation from fine-tuning pre-trained models. As result, RAG is particularly effective domains with scarce and long-tail data, such as network anomaly detection, rare discase diagnosis/treatmeent supply chain disruption, and new user recommendations.
30

---

Additionally, our model establishes an excellent paradigm by incorporating retrieved features and label information into the learning process, significantly enhancing the model's understanding capabilities. Our work provides valuable insights and serves as a reference for future Large Graph Models.
G Data Ethics Statement
To evaluate the efficacy of this work, we conducted experiments that only use publicly available datasets, namely, PROTEINS, COX2, ENZYMES, BZR TAOBAO, KOUBEI and AMAZON in accordance to their usage terms and conditions if any. We further declare that no personally identifiable information used, and no human animal subject was involved in this research.
'attps://chrmmrrs. www.giii...taaaaaa...ee
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the introduction section, we delineate the problems addressed by this work and outline our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the conclusion section, we highlight the limitations of the current work and suggest directions for future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech -totteexttttttttt system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification:
We provide the complete theoretical proof' in Appendix C.4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yss]
Justification: We provide detailed experiment settings in Appendix D.4. Besides, code is anonymously available at https: //aooyymouu 4open dopen.scienn science/r/GLM RAG--AG----- 049D/.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. case of closed-source models, it may be that access to the model is limited in some way (e-g., .. to registered users), but it should possible for other researchers have some path to reproducing verifying the results.
the
5. Open access to data and code
33

---

Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is anonymously available htttps:// http:///oooommuuuuuum hanoyymous.oopn...oo. sciencerr GLM- RAG-049D/.
Guidelines:
The answer NA means that paper does not include experiments requiring
requuiiinn code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Co////odiieeeeeeeeee
/nips. wwtppbccc//aaeeccccccccccccceeee /utee//ueueeeeeeeeuamiiii wwwssssoiiigyyoosoo for more details.
The authors should provide instructions on data access and preparation, including how to access the data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide experiment settings in Section 5.1 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each experiment, conducted 5 repeated experiments and reported the standard deviation Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in Appendix D.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics https: //te:r//./uuu.eeeeeee wwtppublcc/Bhiiaaaadeeeeeeeeeeee
Answer: [Yes]
Justification: have read the NeurIPS Code of Ethics and confirm our research in the paper conforms with Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special considera- tion due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential impacts in Appendix F.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The graph neural network framework proposed in our paper does not extend to application domains requiring safeguards. Additionally, the datasets used are widely-used node classification datasets, thus eliminating the need for specific safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide source links for all datasets and baselines in Appendix D, and we have cited all referenced works.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
36

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode www.coo//aac..oc /ataasss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our code anonymously at attttps: .oop: sciencei r/ r/GLM GGMM-RAG-099D/
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then much detail possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
37

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "FPrMtxnN8r", "reviewer": "Reviewer_va3W", "summary": "### Summary:\n\nThe paper presents RAGRAPH, a pioneering framework that integrates Retrieval-Augmented Generation (RAG) with  pretrained Graph Neural Networks (GNN) to bolster their generalizability on unseen graph data. The authors construct a toy graph vector library capturing key attributes, which aids in the retrieval of analogous graphs to enrich the learning context during inference. RAGRAPH demonstrates superior performance over existing methods in tasks such as node classification, link prediction, and graph classification, showcasing its adaptability and robustness across diverse datasets without the need for fine-tuning.\n\n### Contributions:\n\n1. The introduction of RAGRAPH, the first of its kind to merge RAG techniques with pre-trained GNNs, offering a significant leap in model generalization.\n\n2. The work creates a novel library that stores key graph attributes, facilitating the retrieval of similar graphs to enhance learning.\n\n3. The proposed RAGRAPH outperforms state-of-the-art methods across multiple graph learning tasks, highlighting its effectiveness. Besides, the framework maintains high performance across different tasks and datasets, emphasizing its robustness.", "questions": "See the concern above.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. RAGRAPH's strength lies in its innovative use of retrieval mechanisms to enhance graph learning tasks. By retrieving and integrating external graph data, it effectively broadens the context for learning, leading to improved performance and generalization.\n\n2. The whole process is reasonable and solid, with clear presenation.\n\n3. The ability to maintain high performance across various datasets without task-specific fine-tuning is a significant strength. This adaptability makes RAGRAPH a robust choice for diverse real-world applications. Moreover, RAGRAPH's design as a plug-and-play module allows for seamless integration with pre-trained GNNs.", "weaknesses": "I have only one concern about the cons of this work:\n\nThe major weakness is the difficulty to construct and maintain high-quality graph vector base for different tasks, since according to my experiences, the performance is highly dependent on the quality of the external knowledge. This may influence the application of the proposed method in more diverse real-world cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We address your concerns and answer your questions below.\n\n**W1: The difficulty to construct and maintain high-quality and diverse graph vector base for different tasks.**\n\n**A1**: We acknowledge the challenge you mentioned of constructing and maintaining high-quality graph vector bases tailored to diverse tasks. In RAGraph, **our toy graph base largely leverages significant prior research datasets in pre-trained GNNs** [1] [2] [3] [4], which are trained on meticulously curated graph datasets and cover diverse domains, such as biology, chemistry, medicine recommendation tasks, etc. For example, the PROTEINS dataset [5], derived from cryo-electron microscopy and X-ray crystallography, and the ENZYMES dataset [6], based on EC enzyme classification, are meticulously annotated by medical experts. \n\nMoreover, to address inherent challenges in data quality, we introduce Noise-based Graph Prompting Tuning (Section 4.3.3). This method involves fine-tuning the model with artificially introduced noisy toy graphs (Inner-Toy-Graph Noise & Toy-Graph Noise), inspired by noise-tuning techniques in NLP [7] [8] [9]. **Our approach enhances the model's robustness against real-world retrieval noise**, as evidenced by superior performance compared to traditional tuning methods (**in Main Text Tables 1 and 2**). This approach reduces the stringent requirement for an exceptionally high-quality graph vector base, thereby ensuring robust performance across various tasks within our RAGraph, and significantly mitigating data quality impacts.\n\nLastly, to verify the diversity of applications for RAGraph, we also conducted experiments on the time-series encyclopedia TGB Wiki, the paper-cited datasets Arxiv and Cora, and large-scale graphs MAG in **Rebuttal PDF Table 6**.\n\nThe experimental results demonstrate that RAGraph can be applied in diverse real-world cases. In addition, **the effect of noise-based fine-tuning is better than that of direct fine-tuning, which also proves the effectiveness of our NFT approach, effectively addressing the inherent challenge in data quality**.\n\n[1] Liu, Z., Yu, X., Fang, Y., & Zhang, X. 2023. GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks. In WWW.\n\n[2] Xia, L., Kao, B., & Huang, C. (2024). OpenGraph: Towards Open Graph Foundation Models. Retrieved from arXiv preprint arXiv.\n\n[3] Yu, X., Zhou, C., Fang, Y., & Zhang, X. (2023). MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs. In WWW.\n\n[4] Huang, Q., Ren, H., Chen, P., Kržmanc, G., Zeng, D., Liang, P., & Leskovec, J. (2023). PRODIGY: Enabling In-context Learning Over Graphs. In NeurIPS.\n\n[5] Borgwardt, K. M., Ong, C. S., Schönauer, S., Vishwanathan, S. V. N., Smola, A. J., & Kriegel, H.-P. (2005). Predicting protein function through graph kernels. In Bioinformatics.\n\n[6] Wang, S., Dong, Y., Huang, X., Chen, C., & Li, J. (2022). FAITH: Few-shot graph classification with hierarchical task graphs. In IJCAI.\n\n[7] Yoran, O., Wolfson, T., Ram, O., & Berant, J. (2024). Making Retrieval-Augmented Language Models Robust to Irrelevant Context. In ICLR.\n\n[8] Fang, F., Bai, Y., Ni, S., Yang, M., Chen, X., & Xu, R. (2024). Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training.  In ACL.\n\n[9] Cuconasu, F., Trappolini, G., Siciliano, F., Filice, S., Campagnano, C., Maarek, Y., Tonellotto, N., & Silvestri, F. (2024). The Power of Noise: Redefining Retrieval for RAG Systems. In arxiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GbEgjIPyAJ", "reviewer": "Reviewer_3qQ8", "summary": "This paper aims to leverage the retrieval-augmented generation method to improve the generalisation capability of pretrained graph neural networks (GNNs) to unseen data. To this end, a framework named RAGRAPH is proposed. RAGRAPH first constructs a toy graph vector library (key-value pairs) by chunking from resource graphs, where the keys store some features of the master nodes of the toy graphs and the values are the corresponding node representations and task-specific outputs. When making predictions for unseen nodes/graphs, RAGRAPH retrieves top-$K$ most similar toy graphs from the vector library and augment the nodes/graphs with the retrieved toy graphs. Moreover, to mitigate the challenge of retrieving related but irrelevant graphs, the paper also proposes a prompt tuning method to finetune the pretrained GNNs. The main idea of this prompt tuning method is to explicitly inject some noises into the retrieved toy graphs during the finetuning stage.", "questions": "1. What are the justifications for the specific design choices in the proposed RAGRAPH framework? Can they all contribute to the performance improvement?\n2. How to set the hyperparameters in RAGRAPH? It is unclear whether there is a general setting that can achieve decent performance across different datasets, or if the hyperparameters need to be adapted for each specific dataset. \n3. Can you provide some qualitative analyses to provide some insights regarding why retrieving toy graphs can help improve the performance?", "limitations": "The authors have discussed limitations and broader impacts in their paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper propose a novel framework RAGRAPH, which leverages RAG methods to enhance the generalisation capability of pretrained GNNs.  \n2. The paper is well-structured and easy to follow. \n3. Experimental results on three graph learning tasks, i.e., node classification, graph classification and link prediction, demonstrate the effectiveness of the proposed RAGRAPH framework. \n4. The code is provided for reproducibility.", "weaknesses": "1. The RAGRAPH framework is quite complicated, consisting of several steps in the pipeline. However, some design choices within the pipeline are not properly justified. For example:\n\n(1) When constructing the toy graph vector library (section 4.1), RAGRAPH uses an inverse importance sampling strategy and some augmentation strategies to sample toy graphs to be stored in the library. It is currently unclear if we actually need all these steps to achieve satisfactory performance and how different choices would affect the final performance. \n\n(2) In the toy graph retrieval process (section 4.2), RAGRAPH uses four different sources of similarities, i.e., time, structure, environment and semantic similarities, to compute the similarities between the centre node in the query graph and the master nodes in the toy graphs. It is also unclear if all these four similarities all contribute to the performance improvement. Moreover, it is unclear how to set the different weights for these four similarities. \n\n(3) In the knowledge fusion layer (section 4.3.2), RAGRAPH fuses the Decoder output with the aggregated task-specific output vector, which is obtained from the retrieved toy graphs, to make final prediction. However, it is unclear if this fusion method can outperform using the Decoder output only or the aggregated task-specific output vector only. \n\nTherefore, I would recommend conducting some ablation studies to justify the specific choices in the RAGRAPH framework. \n\n2. There are lots of hyperparameters in the RAGRAPH framework, such as the balance weight $\\alpha$, the scaling constant $K$ in toy graph augmentation, the hyperparameters in different augmentation methods, the reweighting hyperparameter $\\gamma$, ect. However, it is unclear how to set these hyperparameters to obtain the best performance. Additionally, the specific settings of these hyperparameters used in the experiments are also absent from the paper. \n\n3. In line 332-333, the paper states that “RAGRAPH outperforms all the baselines across the three graph tasks”. However, this is inaccurate given that RAGRAPH does not always obtain the best performance in the link prediction task (see Table 2). \n\n4. I am confused about the analyses in line 348-353. The paper states that “PRODIGY/NF and RAGRAPH/NF are inferior to Vanilla/NF, indicating that …”. However, the experimental results in Table 1 and Table 2 actually indicate that both PRODIGY/NF and RAGRAPH/NF outperform Vanilla/NF in almost all the cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns and answer the questions below.\n\n**W1 & Q1: Unclear of the contribution of (a) inverse importance sampling, (b) augmentation,  (c) four similarities, and (d) fusion method to performance improvement.**\n\n**A1**:  We conducted four ablation experiments on both node and graph classification tasks with settings in Appendix C.4 with the following variants by removing: (a) inverse important sampling strategy (wo IIS), (b) augmentation (wo AUG),  (c) any one of the four similarities (wo Time, wo Structure, wo Environment, wo Semantic); and (d) Only use X (mask the task-specific output vector, w X) and only use Y (mask the decoder, w Y):\n\n(a) The adoption of Inverse Importance Sampling strategy is crucial. In RAGraph, subgraphs are sampled as toy graphs, where nodes with higher degrees (non-long-tail knowledge, extensively learned and embedded into GNN parameters) are more frequently included in subgraphs due to their extensive connections with neighbors, resulting in higher frequency in toy graph base [1]. Conversely, **nodes with low degrees (long-tail knowledge), are more important but ignored**. To mitigate this issue, we propose this by **prioritizing nodes with lower degrees to capture long-tail knowledge when sampling**. \n\nThe ablation results in **Rebuttal PDF Table 2** show that w (with) IIS significantly outperforms wo (without) IIS. \n\n(b) Furthermore, regarding the rationale for conducting augmentation,  due to word restriction, please refer to  **global response A1, and ablation result is in PDF Table 1**. \n\n(c) In practical applications, the four similarities all contribute to performance improvement and we state the significance as follows:\n\n- Time information is crucial to predict future states or trends [2] via node history, i.e. in social networks, analyzing historical user interaction aids in predicting future behaviors.\n\n- Structure pertains to how nodes are interconnected and overall graph topology, vital for capturing similar graph structure patterns [3]. In transportation networks, factories are always located on the outer ring of the city, sharing similar structural connectivity, aiding in the discovery of spatio-temporal patterns [4].\n\n- Sharing similar neighborhoods is essential for evaluating node similarity and correlation. In recommendations, shared purchase histories between users and products indicate potential interests, akin to collaborative filtering [5].\n\n- Semantic information measures similarity based on features [6]. In knowledge graphs, identifying relevant subgraphs to query nodes enhances retrieval accuracy based on semantic similarity.\n\nThe ablation results in **Rebuttal PDF Table 3** indicate that each type of similarity has a positive impact.\n\n(d) Fusion and decoder here represent one of the core contributions of RAGraph: \n\n- Overall Task Perspective: For same tasks, decoder can be directly employed to obtain outputs. For different tasks, decoder can be masked and utilize pre-computed embeddings without training or be tuned to better adapt. This underscores our primary contribution, where decoder functions as a versatile \"plug-and-play\" and \"tune-free\" component. \n\n- Integral Fusion Strategy: Fusion Strategy facilitates concurrent information propagation from toy graphs X (hidden embeddings) and Y (task-specific output vector) to query graph, aligning with our secondary contribution.\n\nExperimental results in **Rebuttal PDF Table 4** show the effect brought by fusion is enormous, and any deficiency in X / Y has an impact on model performance. \n\n[1] Walking in Facebook: A case study of unbiased sampling of OSNs. In SIGCOMM 2010.\n\n[2] GraphPro: Graph Pre-training and Prompt Learning for Recommendation. In WWW 2024.\n\n[3] Position-aware Graph Neural Networks. In ICML 2019.\n\n[4] Spatiotemporal Multi-Graph Convolution Network for Ride-Hailing Demand Forecasting. In AAAI 2019.\n\n[5] A Survey of Collaborative Filtering Techniques. In AAAI 2009.\n\n[6] PRODIGY: Enabling In-context Learning Over Graphs. In NeurIPS 2023.\n\n---\n**W2 & Q2: Unclear how to set hyperparameters.** \n\n**A2**: For those sensitive $k$ and $topK$, we have presented and analysed in **Figure 3**. For less sensitive hyperparameters, we observed optimal performance across a broad spectrum of values. In our study, for less sensitive parameters, we employed Bayesian optimization using Optuna (https://github.com/optuna/optuna) to achieve hyperparameter tuning.\n\n- Bayesian Optimization: To mitigate the difficulty of manually fine-tuning hyperparameters for each dataset, we adopted a Bayesian optimization technique that concurrently optimizes hyperparameters across multiple datasets on the validation set [7] [8]. This approach automates and streamlines the hyperparameter tuning process across various datasets in RAGraph, eliminating the need for individual dataset-specific fine-tuning, and enhancing both generalization capability and resource utilization.  We apologize for our oversight of the paper and will supplement details into the main text.\n\n- Hyperparameter Settings: The hyperparameter configuration in our study is detailed as follows: $k$ is set to 2, $topK$ is set to 5, $\\alpha=\\lambda=\\gamma=0.5, K=50, w_1=w_2=w_3=0.1, w_4=0.6$, which can be found in Appendix C.4.\n\n[7] Practical Bayesian optimization of machine learning algorithms. In NeurIPS 2012. \n\n[8] Taking the human out of the loop: A review of Bayesian optimization. In TPD 2016.\n\n---\n**W3: Two inaccurate writings.**\n\n**A3**:  Thanks for pointing out this question. We will correct the two inaccurate writings in final version by replacing (1) \"all\" to \"almost all\",  (2) \"inferior\" to \"better\". Additionally, we have thoroughly reviewed the manuscript for any other potential typographical errors to ensure accuracy.\n\n---\n**Q4: Qualitative analyses of toy graphs retrieving.**\n\n**A4**:  Due to word restriction, please refer to the **global response A2 and the Rebuttal PDF Figure 1**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7B6NpsgdHa", "reviewer": "Reviewer_sivC", "summary": "The paper proposes RAGraph, a framework that enhances GNNs with RAG. RAG allows GNNs to utilize unseen data by retrieving relevant information. Extensive experimental results show the effectiveness of RAGraph.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. A general and flexible framework.\n\nS2. Extensive experiments on various tasks (both node, edge, and graph level)\n\nS3. Structured and comprehensive discussion (e.g. details of experiments and comparison with PRODIGY).", "weaknesses": "W1. The diversity of datasets needs to be improved. For link prediction, the paper is evaluated on mostly e-commerce data, where knowledge graph tasks should also be considered. For node classification, both homophilic datasets e.g. Cora, Arxiv; and heterophilic datasets should be evaluated.\n\nW2. Lack of large scale experiments, e.g. those OGB node/link/graph datasets.\n\nW3. The model seems complicated with a big hyperparameter search space, e.g. the weights of time, structure, environment, and semantic similarities in Eq. 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. We address your concerns and answer your questions below.\n\n**W1: The diversity of datasets needs to be improved. For link prediction, the paper is evaluated on mostly e-commerce data, where knowledge graph tasks should also be considered. For node classification, both homophilic datasets e.g. Cora, Arxiv; and heterophilic datasets should be evaluated.**\n\n**A1**:  Thank you very much for your suggestions. \n\n- For the link prediction dataset: we have incorporated the link prediction results on the temporal knowledge graph ICEWS [1] with the backbone SPA [5].\n\n- For the homophilic node classification datasets:  we have introduced the OGBN-Arxiv [2] and Cora [3] datasets for the homophilic setting with backbone GCN. \n\n- For the heterophilic node classification dataset: we have included the OGBN-MAG [4] graph, which contains 244,160,499 nodes and 1,728,364,232 edges with backbone R-GCN.\n \nThe experimental results of the RAGraph and baselines in **Table 6 of the Rebuttal PDF** demonstrate the superiority of the RAGraph, aligning with our analysis and conclusions presented in Section 5 Experiment.\n\n[1] https://www.lockheedmartin.com/en-us/capabilities/research-labs/advanced-technology-labs/icews.html\n\n[2] https://ogb.stanford.edu/docs/nodeprop/#ogbn-arxiv.\n\n[3] https://paperswithcode.com/dataset/cora. \n\n[4] Microsoft Academic Graph: when experts are not enough. Quantitative Science Studies. (2020). https://ogb.stanford.edu/docs/lsc/mag240m/.\n\n[5] Search to Pass Messages for Temporal Knowledge Graph Completion. In ACL 2022.\n\n---\n**W2: Lack of large-scale experiments.**\n\n**A2**:  Thanks for your suggestions!  We have conducted node classification (OGBN-MAG) tasks on large-scale graphs as detailed in **A1 to W1 and shown in Table 6 of the Rebuttal PDF**.\n\nWe will add these experiment results to the final version. In addition, regarding the reproducibility of the experiment, we have also updated the evaluation code for this sub-test in anonymous github.\n\n---\n**W3: The model seems complicated with a big hyperparameter search space.**\n\n**A3**: Thank you for raising this question, indeed, the challenge of dealing with a large search space is inherent and formidable in most deep-learning models. In our RAGraph, for those sensitive hyperparameters $k$ and $topK$, we have presented in **Figure 3**. For less sensitive hyperparameters, we observed optimal performance across a broad spectrum of values and employed Bayesian optimization using Optuna (https://github.com/optuna/optuna) to address hyperparameter tuning.\n\nTo mitigate the difficulty of manually fine-tuning hyperparameters for each dataset, we adopted a Bayesian optimization technique that concurrently optimizes hyperparameters across multiple datasets on the validation set [6] [7]. This approach automates and streamlines the hyperparameter tuning process across various datasets in RAGraph, eliminating the need for individual dataset-specific fine-tuning, and enhancing both generalization capability and resource utilization.  \n\n[6] Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In NeurIPS. \n\n[7] Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., & Freitas, N. (2016). Taking the human out of the loop: A review of Bayesian optimization. In  IEEE Transactions on Power Delivery.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bj5E3yMsGh", "reviewer": "Reviewer_v5NH", "summary": "The paper propose a Retrieval-Augmented method to further assist graph in context learning. With the advancement of graph In-context learning and graph prompting, RAG is a natural technique that could be built upon them. The main contribution of the paper is to develop such a RAG pipeline on graph learning scenario by defining the graph database, graph retrieval pipeline, and training and inference of the model.", "questions": "1. Why do we need Toy Graphs Augmentation Strategy in 4.2\n2. Why noise-based graph prompting is required?\n3. If the model is trained to able to utilize retrieve data/embedding, is it still able to perform zero-short inference?", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea is novel in the field. And the implementation is clearly stated. Experiments show the effectiveness of the propose method.", "weaknesses": "While the idea is novel. It is unclear why it works and how it is in par with RAG in NLP/CV.\n\nSpecifically, it not sure how the “generation” in RAG works in the proposed framework. And the author did not explain why the proposed method would work — what pattern the model has learned from the retrieved graph pattern?\n\nBased on my limited knowledge, the key contributions of the paper would be how to construct the query data base and how to retrieve them. The methodology part doesn’t explain through why it’s design like so. Motivation of the adopted construction is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns and answer your questions below.\n\n**W1: Why RAGraph works on par with RAG in NLP/CV, and how “generation” works. What pattern was learned from retrieved graph pattern?**\n\n**A1**: In NLP, RAG enhances the generation of LLM by retrieving relevant information via prompts. Similarly, in RAGraph, we enhance downstream graph learning by integrating information from retrieved toy graphs. Using these toy graphs with shared patterns assists the model inference. In our framework, the \"generation\" involves the retrieval-enhanced Graph Prompt: Toy Graph Intra Propagate & Query-Toy-Graph Inter Propagate to propagate retrieved knowledge (X and Y) into query graph. To illustrate, we analyze this from both experiment and theory.\n\n1. Experiment 1: We perform a case study to illustrate how \"generation\" works by displaying specific instances of node vectors. Due to word restriction, please refer to the **global response A2 and Rebuttal PDF Figure 1**. \n\n2. Experiment 2: In traditional GNN tasks, GCN, GAT, and GIN typically expand their receptive fields through stacked message-passing layers or neighborhood subgraph sampling for inference. Patterns learned in these contexts are often localized within the constrained receptive field. In contrast, in RAGraph, we observe that subgraphs sharing similar patterns often exhibit properties more aligned with downstream tasks. These subgraphs provide richer information for inference compared to simply enlarging receptive fields. As shown in **Main Text Tables 1 and 2, Figure 3**, RAGraph's strategy of incorporating toy graphs significantly outperforms baselines.\n\n3. Theory 1: Furthermore, we provide a theoretical justification of retrieval augmentation in GNNs (see Appendix B.4). From an information-theoretic perspective, introducing RAG knowledge into GNNs enhances the mutual information between input features X and output labels Y, such that $I(X, RAG; Y) \\geq I(X; Y)$, thereby improving the performance on downstream tasks. This is aligned with the information theory of RAG in NLP [1].\n\n4. Theory 2: Recent studies [2] [3] also suggest the generalization error diminishes with an increase in the node number of the graph in Theorem 1.1 [4]: the generalization error between the expected loss $R_{exp}(\\Theta)=\\mathbb{E}_{(x,y)\\sim \\mu_G}[\\mathcal{L}(\\Theta(x), y)]$ and\n\n empirical loss $R_{emp}(\\Theta)=\\frac{1}{m}\\sum_{i=1}^m[\\mathcal{L}(\\Theta(x^i), y^i)]$ are supper bounded: $|R_{exp}(\\Theta)-R_{emp}(\\Theta)| \\leq \\sqrt{\\frac{C}{m}q(N)}$, where C represents the model complexity (e.g., parameters), m denotes the training set size, and $q(N) = \\mathbb{E}_{N\\sim v} [N^{-\\frac{1}{D+1}}]$ depends on the average graph size N (node num) with v is the graph size distribution and D is the metric-measure space dimension. In RAGraph, retrieving similar toy graphs significantly increases the number of graph nodes (via Query-Toy-Graph Inter Propagate, linking toy graph nodes to query graph), significantly augmenting N while reducing q(N). Consequently, upper bound of generalization error decreases, promoting smoother graph learning convergence and enhancing pattern learning.\n\n[1] Generalization Analysis of Message Passing Neural Networks on Large Random Graphs. In NeurIPS 2022.\n\n[2] Wasserstein Barycenter Matching for Graph Size Generalization of Message Passing Neural Networks. In ICML 2023.\n\n[3] An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation. In ACL 2024\n\n---\n**W2:  Lack motivation to construct toy database and retrieve.**\n\n**A2**: The motivation behind initially constructing the toy graph database is to identify similar knowledge patterns in graphs (where the toy graph serves as a repository of such patterns, including X and Y). Therefore, establishing such a database is necessary to store potential candidate sets for downstream tasks. However, during the construction of toy graphs, if the toy graph size is too large, it may introduce excessive noise and adversely affect model; conversely, if it is too small, the introduced knowledge may be insufficient. Thus, we employ a method of chunking the resource graph given hop $k$. Moreover, to better store long-tail knowledge and simulate real world scenarios, we also introduce inverse importance sampling and leverage augmentation to expand toy graph database.\n\nIn retrieval process, the motivation is to comprehensively retrieve knowledge X and Y that can enhance downstream tasks, and we evaluate similarity across four dimensions: time, structure, environment, and semantic relevance in Appendix B.3. \n\nThank you for your suggestions, and we will incorporate these clarifications into the final version.\n\n---\n**Q1: Why do we need Toy Graphs Augmentation.**\n\n**A3**: We have provided an analysis and ablation study to illustrate the importance of augmentations. Due to word restrictions, please refer to **global response A1 and Rebuttal PDF Table 1**.\n\n---\n**Q2: Why noise-based graph prompting is required？**\n\n**A4**: The necessity arises because ensuring the high quality of the graph vector base is challenging, which heavily depends on the quality of external knowledge. In RAGraph, we introduce Noise-based Graph Prompting Tuning (outlined in Section 4.3.3) to address this challenge. Due to word restriction, please refer to **global response A3**. \n\n---\n**Q3: If RAGraph still has the zero-short inference ability.**\n\n**A5**: Your suggestion is very insightful. To assess whether fine-tuned model maintains zero-shot inference capability when tested without knowledge, we conducted experiments comparing PRODIGY and RAGraph:\n\nResults in **Rebuttal PDF Table 5** indicate that the performance without knowledge shows a decrease compared to injected knowledge due to the absence of knowledge. However, compared to PRODIGY, RAGraph is more robust and we argue that the trained model still retains its zero-shot knowledge inference capability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to all reviewers for their insightful comments and acknowledging the strengths of RAGraph. We have addressed all the concerns raised and provided comprehensive answers in this rebuttal.

In the attached PDF, we present:
 
**(1) More ablation experiments mentioned by Reviewer 3qQ8 & Reviewer v5NH;**

**(2) Added datasets experiments pointed out by Reviewer sivC;**

**(3) Qualitative analyses of toy graphs retrieved suggested by Reviewer 3qQ8 & Reviewer v5NH.**

---
Regarding commonly asked questions:  (1) The effect of augmentation;  (2) Qualitative analyses of toy graphs retrieval, and (3) The effect of Noise-based Graph Prompt Tuning;  we give detailed explanations in the next parts and we will add the explanation in our future submission: 

**1. For Reviewer 3qQ8 & Reviewer v5NH: Answers to the effect of augmentation.**

**A1**:  The reasons for toy graph augmentation:

- **Expanding toy graph base**, enriching the scale of the knowledge repository [1].

- **Simulating Real-World Scenarios**: Real-world graphs often encounter challenges such as missing nodes [2], noisy attributes [3], and unexplored connections [4]. We introduce node dropout, noise injection, and edge removal to simulate these scenarios accurately. 

- **Addressing Graph Domain Shift**: To mitigate domain shift between the graph knowledge base and testing graphs, our augmentations employ Mixup techniques such as Node Interpolation and Edge Rewiring. These techniques interpolate between training samples to generate synthetic samples, effectively smoothing decision boundaries in embedding and reducing the model's sensitivity to minor variations in input data, thereby stabilizing predictions on domain shift testing samples [5].

To validate this strategy, we conducted additional experiments on node and graph classification tasks described in Appendix C.4.  For simplicity, we abbreviate “Augmentation strategy” as “AUG”. **The ablation results presented in PDF Table 1 indicate that tw (with) AUG significantly outperforms wo (without) AUG on both node and graph classification tasks.** 

[1] Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy. In TIP 2020.

[2] Incomplete Graph Learning via Attribute-Structure Decoupled Variational Auto-Encoder. In WSDM 2023.

[3] Boosting the adversarial robustness of graph neural networks: An OOD perspective. In ICLR 2024.

[4] Graph embedding techniques for predicting missing links in biological networks: An empirical evaluation. In TETP 2024.

[5] ProtoMix: Augmenting health status representation learning via prototype-based mixup. In SIGKDD 2024.

---
**2. For Reviewer 3qQ8 & Reviewer v5NH: Qualitative analyses of toy graphs retrieving -- how “generation” works.**

**A2**: We conduct qualitative analyses of how "generation" works while learning graphs through a case study in **Rebuttal PDF Figure 1**.

On the ENZYMES dataset, for a 3-class node classification task, regarding node "13984", which belongs to class 3, if we only use the GraphPrompt Backbone, the resulting one-hot encoding is: [0.28, 0.34, 0.38]. 

However, since the node is of class 3, we expect the one-hot encoding to be as close as possible to [0,0,1]. In RAGraph retrieval, taking the top 3 retrieved graphs as examples, the connection weights for these 3 toy graphs to query graphs are 0.5, 0.7, and 0.1, respectively, and their corresponding label one-hot encodings are [0,0,1], [0,0,1], and [0,1,0]. Therefore, the result obtained by propagating the task-specific output vector through toy graphs is: [0, 0.1, 1.2], and after normalization, the result is [0, 0.08, 0.92]. 

Meanwhile, the vector obtained by propagating toy graphs hidden embedding and via decoder is: [0.37, 0.32, 0.66].  
The retrieval of toy graphs notably enhances performance at both the task-specific output vector and hidden embedding levels. The final vector is obtained through a weighted sum with $\gamma=0.5$ in Eq(6) is [0.185, 0.20, 0.79], after normalization the result is [0.157, 0.170, 0.673], which **greatly enhances the model's discriminative ability** compared to GraphPrompt [0.28, 0.34, 0.38].

---
**3. For Reviewer va3W & Reviewer v5NH: The effect of Noise-based Graph Prompt Tuning.**

**A3**: To address inherent challenges in toy graph quality, we introduce Noise-based Graph Prompting Tuning (Section 4.3.3). This method involves fine-tuning the model with artificially introduced noisy toy graphs (Inner-Toy-Graph Noise & Toy-Graph Noise), inspired by noise-tuning techniques in NLP [6] [7] [8]. Our approach enhances the model's robustness against real-world retrieval noise, as evidenced by superior performance compared to traditional tuning methods (**in Main Text Tables 1 and 2**). This approach reduces the stringent requirement for an exceptionally high-quality graph vector base, thereby ensuring robust performance across various tasks within our RAGraph, and significantly mitigating data quality impacts. 

Lastly, to verify the diversity of applications for RAGraph, we also conducted experiments on the time-series knowledge graph Crisis Warning ICEWS, the paper-cited datasets Arxiv and Cora, and large-scale graphs MAG to test Noise-based Graph Prompt Tuning in **Rebuttal PDF Table 6**.
**Experiments show RAGraph's applicability in diverse real-world cases. In addition, the effect of noise-based fine-tuning is better than fine-tuning. This further demonstrates the effectiveness of the NFT approach in tackling the inherent challenges related to data quality.**

[6] Making Retrieval-Augmented Language Models Robust to Irrelevant Context. In ICLR 2024.

[7] Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training.  In ACL 2024.

[8] The Power of Noise: Redefining Retrieval for RAG Systems. In arxiv 2024.

---
$\downarrow \downarrow$ **is the Rebuttal PDF, which contains the supplementary experiments and figures.**

Meta-review: This work proposes a framework named General Retrieval-Augmented Graph Learning (RAGraph) to utilize external graph data for the general graph foundation model to improve generalization on unseen scenarios. Reviewers acknowledged the merits of this work for integrating RAG techniques with pre-trained GNNs.

The authors should include the content they added during the rebuttal when they prepare the final version of the paper. Those include:
1. The responses to Reviewer va3W's point: the difficulty of constructing and maintaining high-quality and diverse graph vector bases for different tasks
2. Justifications for the specific design choices in the proposed RAGRAPH framework, Details about setting hyperparameters, and Qualitative analyses of toy graphs retrieving (Reviewer 3qQ8)
3. Adding the results on heterogeneous graphs and large-scale graphs, as well as the experiments without hyperparameters (Reviewer sivC)
4. Adding motivation to construct a toy database and retrieve & adding the zero-shot results (Reviewer v5NH)
Decision: Accept (poster)